Combined visual question and answer method based on core-to-global semantic fusion reasoning

Abstract

The invention discloses a combined visual question and answer method based on core-to-global semantic fusion reasoning, and belongs to the technical field of image and text processing, and the method comprises the steps: obtaining image data and text question data; inputting the image data into the semantic feature extraction model to obtain image global features and visual core features; inputting the text problem data into the semantic feature extraction model to obtain text problem global features and text core features; transmitting the image global feature, the visual core feature, the text problem global feature and the text core feature to a preset multi-modal semantic fusion model to execute multi-modal semantic learning of different levels; fusing the learning results of the single-level language; and according to the target answer prediction vector, calculating to obtain a target prediction answer. According to the method and the device, the technical problems of inaccuracy in reasoning and model instability caused by inaccurate semantic alignment, feature redundancy and noise interference of multi-modal information in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image and text processing, and in particular to a combined visual question answering method based on core-to-global semantic fusion reasoning. Background Art

[0002] Combined Visual Question Answering (VQA) is an important task that combines computer vision and natural language processing. Its goal is to generate corresponding answers based on the input image content and text questions. VQA technology has broad application prospects in smart assistants, smart homes, autonomous driving and other fields. In these applications, the system needs to understand the visual content in the image, such as objects, attributes and their relationships, and perform semantic reasoning in combination with text questions to generate accurate answers.

[0003] Existing VQA models are mainly divided into holistic and modular methods. The holistic method tends to use a single model to process all input features, such as a neural network to learn multiple semantic features. However, it ignores the semantic inconsistency between multimodal information and easily leads to information confusion. The modular method decomposes the problem into multiple subtasks and uses a dynamic network for reasoning. Although it improves the interpretability of the model to a certain extent, it is still not refined enough for the extraction of core semantic features in multimodal information. In addition, most existing models focus on extracting rich image features, but ignore the filtering of redundant information and noise in these features, resulting in semantic gaps and model instability in the reasoning process. In view of these current situations, there is an urgent need for a VQA method that can effectively extract and fuse multimodal core semantic features and maintain semantic consistency during the reasoning process. Summary of the invention

[0004] This application provides a combined visual question answering method based on core-to-global semantic fusion reasoning, aiming to solve the technical problems in the prior art of inaccurate reasoning and model instability caused by inaccurate semantic alignment of multimodal information, feature redundancy and noise interference.

[0005] In view of the above problems, the present application provides a combined visual question answering method based on core-to-global semantic fusion reasoning.

[0006] The present application provides a combined visual question-answering method based on core-to-global semantic fusion reasoning, the method comprising obtaining image data to be processed and corresponding text question data; inputting the image data into a pre-built semantic feature extraction model to obtain image global features and visual core features; inputting the text question data into the semantic feature extraction model to obtain text question global features and text core features; passing the image global features, the visual core features, and the text question global features and the text core features to a preset multimodal semantic fusion model to perform multimodal semantic learning at different levels to obtain single-level language learning results; fusing the single-level language learning results to form a target answer prediction vector; and calculating the target predicted answer based on the target answer prediction vector.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0008] By adopting a technical solution of core-to-global semantic fusion reasoning, including extracting global features and core features from images and texts, and performing hierarchical semantic learning through a multimodal semantic fusion model, the problems of inaccurate reasoning and model instability caused by inaccurate semantic alignment of multimodal information, feature redundancy and noise interference in the prior art are solved, achieving the technical effect of improving the semantic understanding ability and answer prediction accuracy of the visual question and answer system in multimodal data.

[0009] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flowchart of a combined visual question answering method based on core-to-global semantic fusion reasoning is provided for an embodiment of the present application.

[0011] Figure 2 A schematic diagram of a semantic feature extraction model in a combined visual question answering method based on core-to-global semantic fusion reasoning is provided for an embodiment of the present application.

[0012] Figure 3 A schematic diagram of visual core features in a combined visual question answering method based on core-to-global semantic fusion reasoning is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0013] The overall idea of ​​the technical solution provided by this application is as follows:

[0014] The embodiment of the present application provides a combined visual question-answering method based on core-to-global semantic fusion reasoning. By extracting global features and core features from image and text data respectively, these features are then input into a multimodal semantic fusion model for hierarchical semantic learning, and finally the learning results of each level are fused to form a target answer prediction vector, thereby calculating the most likely answer. This method aims to enhance the accuracy and stability of the visual question-answering system through core-to-global semantic fusion.

[0015] After introducing the basic principles of the present application, various non-limiting implementation methods of the present application will be specifically described below in conjunction with the drawings in the specification.

[0016] Examples, such as Figure 1 As shown, the embodiment of the present application provides a combined visual question answering method based on core-to-global semantic fusion reasoning, the method comprising:

[0017] Step S100: Obtain image data to be processed and corresponding text question data.

[0018] Specifically, image data refers to pictures or sets of images input to the visual question answering system, which contain visual information based on the questions that need to be understood, analyzed, or answered. Image data can be any file containing visual information, such as photos, drawings, or screenshots. For example, image data may be a photo showing a sofa and coffee table in a living room. Text question data refers to natural language questions related to images, which are usually described in text and require the system to generate a corresponding answer based on the image content. Text question data can be a simple question or may contain multiple sub-questions.

[0019] Image data is usually acquired through image acquisition devices (such as digital cameras, smartphone cameras), or can be extracted from image databases or online resources. These image files may be in formats such as JPEG, PNG, etc., and contain the visual content that needs to be analyzed.

[0020] Once these two data are acquired, the system is ready for the next step of processing, which is feature extraction. The image data is fed into a pre-trained deep learning model (such as object detection using a convolutional neural network CNN), while the text question data may be analyzed by a natural language processing model (such as BERT or LSTM).

[0021] This step is the starting point of the entire visual question answering process, ensuring that there is enough information for subsequent feature extraction and answer reasoning.

[0022] Step S200: inputting the image data into a pre-built semantic feature extraction model to obtain the global features and visual core features of the image.

[0023] Specifically, if Figure 2 As shown in the figure, the semantic feature extraction model is a pre-trained deep learning model used to extract important features from images. The semantic feature extraction model can identify and extract key elements in an image, such as objects, colors, shapes, and the relationships between these elements. Common models include convolutional neural networks (CNN), Mask RCNN, etc., which can generate semantic features of images. Image global features refer to feature information that describes the entire image and reflects the overall content and structure of the image. Visual core features refer to the most important and critical detail features extracted from an image. These features are usually associated with the most important objects or scenes in the image.

[0024] When image data is input, the semantic feature extraction model first processes the image to identify important elements and structures in the image. These elements and structures are extracted as global image features, which represent the overall content of the image, such as the main background of the image, the overall tone, and the position and relationship of the main objects. At the same time, the model also extracts visual core features, which are more refined and focus on the most critical objects or scenes in the image.

[0025] This step lays the foundation for accurately answering questions related to the image. In this way, meaningful features can be extracted from complex visual information, not only to understand the overall structure of the image, but also to capture the most critical details in the image.

[0026] Step S300: inputting the text question data into the semantic feature extraction model to obtain the text question global features and text core features.

[0027] Specifically, the global features of a text question describe the feature information of the entire question text, and usually reflect the overall semantic structure of the question. For example, the global features of a text question can represent the main intent and context of the question, such as "Is this question asking about quantity, location, or color?". The core features of a text question are the most critical words or phrases extracted from the question text, which are directly related to the core meaning of the question. For example, in "How many cats are there in the picture?", "how many" and "cats" are the core features of the text question because they determine the focus of the question.

[0028] First, the text question data is input into the semantic feature extraction model to extract useful semantic features from it. The text question data is converted through user input, speech recognition system (such as Siri, Google Assistant), or obtained from existing data sets.

[0029] Semantic feature extraction models usually use natural language processing (NLP) techniques such as recurrent neural networks (RNN), long short-term memory networks (LSTM), or Transformer-based models (such as BERT). These models have been pre-trained on large amounts of text data and are able to understand and analyze the complex structure of natural language. When text question data is input, the semantic feature extraction model first converts the text into word vectors (such as using Word2Vec or GloVe), which represent the semantic information of the words. On this basis, the semantic feature extraction model extracts the global features of the text through layer-by-layer processing, which reflect the overall intent and semantics of the question. At the same time, the semantic feature extraction model also extracts the core features of the text, which are the most critical parts of the question, usually some keywords or phrases.

[0030] Through this step, complex natural language questions can be effectively understood and processed, not only capturing the overall semantics of the question, but also focusing on the most critical parts, thereby providing more precise answers.

[0031] Step S400: passing the image global features, the visual core features, the text question global features, and the text core features to a preset multimodal semantic fusion model to perform multimodal semantic learning at different levels to obtain single-level language learning results.

[0032] Specifically, multimodal semantic learning refers to the process of learning features at multiple levels by fusing information from different modalities (such as images and text). Single-level semantic learning results: This refers to the learning results obtained from a specific level (such as the global level or the core level) during the process of multimodal semantic learning.

[0033] First, the extracted image global features, visual core features, and text question global features and text core features are passed to the multimodal semantic fusion model. In this model, features at different levels will be processed separately. For example, the model may first focus on the global level, at which the model will use the attention mechanism to understand the relationship between the overall background of the image and the overall intent of the question. Next, the model may process the core level, at which the model focuses on the most important detailed features in the image and text, such as the correspondence between a specific object in the image and the core keywords in the question.

[0034] Multimodal semantic fusion models usually adopt deep learning architectures, such as Transformer-based models or models using bilinear attention mechanisms. These models can gradually understand and fuse multimodal information at different levels.

[0035] Through multi-level processing, the relationship between images and texts can be understood from different angles and levels, and finally a single-level semantic learning result is generated. This result reflects the system's comprehensive judgment of image and text information at this level. This learning result will be used in the subsequent answer prediction process to help the system generate more accurate answers based on learning at different levels.

[0036] Through this step, the system can effectively integrate images and text information to accurately answer complex questions. It can not only understand the overall and local details of the image, but also accurately capture the key elements of the question, thereby greatly improving the performance and accuracy of the visual question answering system.

[0037] Step S500: Fusing the single-level language learning results to form a target answer prediction vector.

[0038] Specifically, the single-level language learning results are the semantic learning results obtained from a specific level in the multimodal semantic learning model. It represents the system's understanding of images and texts at this specific level, such as the learning results of the global feature level or the core feature level. Fusion is the process of combining multiple learning results or features together. The fusion process can be performed by weighted averaging, splicing, attention mechanism, etc. The purpose is to integrate the learning results of different levels to form a more accurate comprehensive representation. The target answer prediction vector is the final fused vector, which contains the model's prediction information for the answer. Each element of this vector usually corresponds to a possible answer option, and the value of the element represents the model's confidence or probability for the option.

[0039] The single-level language learning results come from the global level and the core level. The global level results reflect the model's understanding of the overall structure of the image and text, such as the relationship between the overall background of the image and the general meaning of the question. The core level results focus on more detailed features, such as the correspondence between a specific object in the image and the keywords in the question.

[0040] Using the weighted average method, the learning results of each layer are weighted by importance and then averaged. You can also use the concatenation method to concatenate the feature vectors of different layers into a larger vector that contains comprehensive information from each layer. Using the attention mechanism, the model automatically selects the most important layers and features and assigns them higher weights.

[0041] The learning results of different layers are fused to generate a target answer prediction vector. This vector is usually implemented through a softmax layer or other classifier, which converts the fused feature vector into a probability distribution, indicating the confidence of each possible answer.

[0042] Through this step, the system can make comprehensive use of information from different levels to make more accurate and robust answer predictions, which not only improves the accuracy of predictions, but also enhances the performance of the model in handling complex multimodal tasks.

[0043] Step S600: Calculate and obtain a target predicted answer based on the target answer prediction vector.

[0044] Specifically, the target predicted answer is the answer that the system finally generates, based on the highest confidence value in the target answer prediction vector.

[0045] First, the target answer prediction vector is generated by fusing the results of multiple levels of semantic learning. Each element in this vector represents the system's confidence in a possible answer. The process of calculating the target prediction answer usually uses the argmax operation, which finds the element with the largest value in the target answer prediction vector. This step is performed after the softmax layer, which is a common activation function used to convert the output of the model into a probability distribution. Softmax ensures that the sum of all predicted values ​​is 1, thereby representing the relative probabilities between different answers. For multi-classification problems, softmax is very effective because it can convert the unnormalized output of the model (called logits) into easy-to-interpret probability values.

[0046] Finally, after this step, the target prediction answer obtained will be used as the final output to answer the question raised by the user. This ensures that the system can select the most likely answer based on the fused features and learning results and provide it to the user, which is suitable for various application scenarios.

[0047] Further, such as Figure 3 As shown, the image data is input into a pre-built semantic feature extraction model to obtain image global features and visual core features, including: extracting the image data based on the MaskRCNN network of the semantic feature extraction model to obtain the image global features; obtaining image RoI features and object candidate frames through the MaskRCNN network; using the image RoI features as image embedding, that is, using the image RoI features as image data features; constructing a BiTreeLSTM network using the Prim algorithm; using the BiTreeLSTM network to process the image data features to generate object features; based on the object features, further generating relationship features through the BiTreeLSTM network; adding an attribute branch in the object candidate frame, the attribute branch is composed of a multi-layer perceptron, the attribute branch takes the image data features as input, and performs attribute classification prediction through a cross entropy loss function to obtain attribute features; combining the object features, the relationship features, and the attribute features to obtain the visual core features.

[0048] Specifically, the MaskRCNN network is an object detection model that can not only detect objects in an image, but also generate a segmentation mask for each object. The network consists of two main parts: a Region Proposal Network (RPN) for generating object candidate boxes, and a CNN-based subnetwork for generating object classification and location. RoI features are features extracted from specific regions in an image, representing objects or key regions detected in the image. Object candidate boxes are boxes generated by RPN, representing areas in the image that may contain objects. Embedding refers to converting high-dimensional data into low-dimensional vector representations to capture the semantic information of the data. Here, the RoI features of the image are used as embeddings for image data. Prim's algorithm is a classic algorithm for generating minimum spanning trees, which is used here to construct the structure of the BiTreeLSTM network to form optimal connections between nodes. The BiTreeLSTM network is a bidirectional tree-like long short-term memory network, which is a special LSTM (long short-term memory) network used to process tree-structured data. In image processing, BiTreeLSTM can capture hierarchical relationships between objects. Object features are feature information of specific objects extracted from an image, such as location, category, shape, etc. Relational features are features that describe the relationship between different objects in an image. Attribute features are features that describe the attributes of the object itself. Multilayer Perceptron is a common feedforward neural network, usually composed of multiple hidden layers, used for classification or regression tasks of input data.

[0049] First, the input image data is processed using the MaskRCNN network to extract global image features. MaskRCNN generates multiple object candidate boxes (RoIs) through its Region Proposal Network (RPN), which represent image regions containing important objects.

[0050] Next, RoI features are further extracted from these candidate boxes. These features represent the detailed information of specific objects in the image. These RoI features are used as image embedding, which actually converts high-dimensional image information into low-dimensional vector representation for subsequent processing. In order to capture the complex relationship between objects in the image, the system uses the Prim algorithm to construct a BiTreeLSTM network.

[0051] The BiTreeLSTM network first processes the embedding of the image (i.e., RoI features) to generate object features that represent specific objects. These object features include the category, location, and shape information of each object. Then, based on these object features, BiTreeLSTM further generates relational features. In order to enhance the understanding of object attributes, the system adds an attribute branch to the object candidate box. This branch consists of a multi-layer perceptron (MLP), which receives image data features as input and performs attribute classification prediction through a cross-entropy loss function to obtain attribute features. Attribute features may include the color, material, or state of an object. For example, "red" in "red car" is an attribute feature.

[0052] Finally, the system combines object features, relationship features, and attribute features to obtain visual core features. This combination process integrates the key information in the image into a comprehensive feature representation, which can better understand and utilize image information in subsequent tasks. Through this step, the most meaningful visual information can be extracted from the image, laying a solid foundation for subsequent multimodal semantic fusion and question answering.

[0053] Furthermore, the text question data is input into the semantic feature extraction model to obtain global features of the text question and core features of the text, including: mapping word information in the text question data to a low-dimensional feature space through word embedding technology to form a word embedding vector; sending the word embedding vector in sequence to a GRU network unit for sequential feature learning to obtain the global features of the text question; using a stop word filter to process the text question data, and obtaining core words in the text question data after screening, wherein the stop word filter includes a stop word list based on NLTK and rare words whose frequency in the entire text question set is lower than a set threshold; and mapping the core words to core features of the text through word embedding technology.

[0054] Specifically, word embedding technology is a method of mapping words in natural language into low-dimensional vector representations. Word embedding converts high-dimensional word information into low-dimensional feature vectors, capturing the semantic relationships between words. Commonly used word embedding techniques include Word2Vec, GloVe, and FastText. Word embedding vectors are vectors generated by word embedding technology, representing the positions of words in the low-dimensional feature space. Word embedding vectors are the basis for the model to learn semantic features, and the position of each word in this vector space reflects its semantic information. GRU (Gated Recurrent Unit) is a variant of the recurrent neural network (RNN) used to process sequential data. GRU can effectively capture the context information in the sequence and avoid the problem of gradient disappearance in long sequences. It is commonly used for feature extraction and sequence learning in natural language processing. Stop word filters are tools used to filter out unimportant or common words. Stop word filters remove these words and retain the keywords in the text. NLTK (Natural Language Toolkit) is a commonly used natural language processing library that provides a list of stop words and other language processing tools.

[0055] First, convert the word information in the text question data into a form that can be understood and processed by the model. This process starts with the use of word embedding technology. Word embedding techniques such as Word2Vec or GloVe can map each word to a low-dimensional feature space, forming corresponding word embedding vectors. These vectors are low-dimensional numerical representations that capture the semantic similarities between words.

[0056] Next, feed these word embedding vectors into the GRU network units in sequence for sequential feature learning. GRU, through its gating mechanism, can effectively retain and transmit the context information in the sequence, thereby capturing the semantic structure in the question.

[0057] To further extract the key semantic information in the text, use a stop word filter to process the text question data. The stop word filter will remove those common words that are semantically unimportant, such as "of", "is", "in", etc. These words usually do not affect the core meaning of the question. Here, the filter is based on the stop word list of NLTK (a widely used natural language processing library), and it will also filter out rare words with a frequency lower than a set threshold in the entire text question set. Through this screening, the system can retain the core words in the question, which are usually nouns, verbs, or other key phrases. Once the core words are screened out, use word embedding technology again to map these core words into text core features. These core features are the most important part of the question and directly affect the system's understanding and subsequent processing of the question.

[0058] Through this deficiency, global features and core features containing rich semantic information are extracted from the original text question data, laying the foundation for subsequent semantic understanding and answer generation. In this way, not only can the context of the entire question be understood, but the key elements of the question can also be accurately captured, thereby improving the effect and accuracy of natural language processing tasks.

[0059] Furthermore, the method also includes: combining the object features, the relationship features, and the attribute features in the manner of object, object-attribute, object-relationship-object to generate a visual core semantic matrix; using the text core features as supervision information, calculating the importance vector of each visual semantic feature in the visual core semantic matrix, and matching it with the text core features to identify visual core semantic feature instances that are directly related to text issues.

[0060] Specifically, the visual core semantic matrix is ​​a matrix that integrates objects in an image, the relationships between objects, and the attributes of objects, reflecting the most important semantic information in the image. Each row or column in the matrix may correspond to a specific object, attribute, or relationship. Supervisory information is guidance information used in the model training or feature matching process to optimize the learning direction of the model. Here, the text core features are used as supervisory information to help identify important features in the visual core semantic matrix. The importance vector is a vector that indicates the degree of importance of each feature in the visual core semantic matrix. The higher the value of the importance vector, the stronger the relevance of the feature in answering the question. Visual core semantic feature instances are features directly related to the text question identified from the visual core semantic matrix, which are key information used by the model to generate answers.

[0061] First, various features extracted from the image are effectively combined to form a comprehensive semantic representation, combining object features, relationship features, and attribute features in a certain logical order. Specifically, the system will first combine these features according to the objects in the image to form a visual core semantic matrix. This matrix is ​​constructed by integrating the features of the object itself, the relationship between the object and its attributes (object-attribute), and the relationship between objects (object-relationship-object). After the matrix is ​​generated, the text core features are used as supervisory information to guide the analysis of the visual core semantic matrix. The text core features are key information extracted from the text question. These text core features help the system focus on the visual features that are most relevant to the question.

[0062] Next, the importance vector of each visual semantic feature in the visual core semantic matrix is ​​calculated. This vector represents the importance of each feature in answering the question. For example, if the question is about "color", the importance of the attribute feature representing the color of the object may be very high.

[0063] Through this step, the core information directly related to the question can be effectively extracted from complex visual and textual data, ensuring that the most critical features can be accurately captured when dealing with visual question answering tasks, and accurate answers can be generated based on these features.

[0064] Furthermore, the text core features are used as supervisory information to calculate the importance vector of each visual semantic feature in the visual core semantic matrix, including: given the visual core semantic feature matrix and the core features of the text Where n ik and n qk represents the number of feature instances, d ik and d qk Represents the feature dimension of each instance; the importance of the visual core semantic feature instance and the text core feature instance can be calculated as:

[0065]

[0066] Among them, τ qk () and τ ik () is a learnable linear projection vector, and Projection and and For the importance matrix a qi The importance vector can be obtained by row summing By sorting the values ​​in the importance vector, the instances corresponding to the top K maximum values ​​are selected and combined into a new visual core feature. Among them, the value of K is: K = λ × mean (d qk ), where λ is a hyperparameter, mean(d qk ) represents the average number of core words extracted from all text questions, λ=6.

[0067] Specifically, the importance vector a qi is a vector that measures the correlation between visual features and text features. Each element of it represents the importance of a feature instance in the visual feature matrix in answering the question. The linear projection vector τ qk () and τ ik() is a learnable vector used to project visual features and text features into the same feature space for similarity calculation. The purpose of projection is to enable features of different modalities (such as images and text) to be aligned and compared in the same space. Top-K screening is the process of selecting the top K most important feature instances from the importance vector. The value of K is determined by the number of core words in the text question and a hyperparameter.

[0068] First, the object, relation, and attribute features of the image are combined into a visual core semantic feature matrix f ik This matrix contains all the important feature instances extracted from the image. Next, the system takes the text core feature vector f in the text question qk As supervision information, to guide the analysis of visual features. Then, the system uses the linear projection vector τ qk and τ ik The visual features and text features are projected into the same feature space so that they can be compared and aligned in the same dimension. Then, the system uses the softmax function to calculate the importance vector a qi , which represents the importance of each feature instance in the visual core semantic feature matrix in answering the question. The softmax function ensures that these importance values ​​are normalized to a probability distribution, so that it is more intuitive to understand which feature is most critical to answering the question. The values ​​in the importance vector are then sorted and the top-K most important feature instances are selected.

[0069] Finally, through this selection, the system generates a new, streamlined visual core semantic feature matrix that contains only those features that are most helpful in answering the question. This approach ensures that the system can efficiently focus on the most relevant information when dealing with complex visual question answering tasks, thereby improving the accuracy of the answer.

[0070] Furthermore, the method further comprises: based on the image core semantic feature matrix f ik and image features f i , calculate the weight graph ψ i :

[0071]

[0072] where τ i () is a learnable linear projection vector, Projection Use the weight map to classify the image feature f i Weighted filtering can be performed to obtain the global features of the filtered image:

[0073]

[0074] in, is the channel normalized vector, ⊙ represents the Hadamard product, and similarly, the text core feature f' qk For supervision information, the weight map ψ can be calculated q And the global features of the filtered text question are:

[0075]

[0076]

[0077] Among them, τ q () is a learnable linear projection vector, Projection

[0078] Specifically, we first use the image core semantic feature matrix f iq and the image feature vector f i To calculate the feature weights. Specifically, a learnable linear projection vector τ is used i () Project the image features into another feature space, and then use the softmax function to calculate a weight map ψ i , this weight map represents the importance of each feature in the entire image feature. Once the weight map is calculated, the system will apply the weight to the corresponding image feature f through the Hadamard product i This can adjust the contribution of each feature so that important features have a greater weight in the result. Then, the system combines the weighted features with the initial image features f i Combine and obtain the filtered image global features through addition operation This step ensures that the system can accurately extract the global image information that is most relevant to the question.

[0079] A similar process is also applied to the text core feature f qk The system also uses a linear projection vector τ q () to project the text features, and then calculate a text weight map ψ through softmax q This weight map will be used to weight the text features and generate the filtered text question global features Ensure that text features can be more focused on the core information corresponding to visual features.

[0080] Through this step, the system can effectively integrate and filter multimodal information, combine complex image features and text features to form a clear, problem-oriented feature representation, which can significantly improve the accuracy and performance of the model.

[0081] Furthermore, the image global features, the visual core features, the text question global features, and the text core features are passed to a preset multimodal semantic fusion model to perform multimodal semantic learning at different levels to obtain a single-level language learning result, including: the multimodal semantic learning at different levels includes multimodal core semantic learning and multimodal global semantic learning; the visual core features and the text core features are used as inputs of the multimodal core semantic learning, and a bilinear attention mechanism is used to fuse the visual core features and the text core features; the output of the multimodal core semantic learning is a dimension d lk Joint expression vector for The mth element in , its corresponding value can be calculated as:

[0082]

[0083] in, and is the learnable parameter matrix, is a bilinear attention map, which is calculated as:

[0084]

[0085] in, and is a learnable parameter matrix; with the image global features, the text question global features and the joint expression vector As the input of the multimodal global semantic learning, wherein, in the multimodal global semantic learning, the global feature of the text question and the joint expression vector Together as supervision information, the supervision information for multimodal global semantic learning is Where || represents matrix concatenation, and the same bilinear attention mechanism as the multimodal core semantic learning is adopted for multimodal feature fusion; the output of the global overall learning is a dimension d gc Joint expression vector in, The jth element in , its corresponding value can be calculated as:

[0086]

[0087] in, and is the learnable parameter matrix, is a bilinear attention map, which is calculated as:

[0088]

[0089] in, and is a learnable parameter matrix; the joint expression vector and the joint expression vector As a single-level language learning outcome.

[0090] Specifically, multimodal semantic learning is the process of learning and understanding data by fusing information from different modalities (such as images and text). Multimodal learning can combine information from images and text to form a richer semantic representation. Modal core semantic learning is semantic learning within a single modality (such as only images or only text), focusing on extracting core information in the modality. Modal global semantic learning is the learning of global features in the modality, aiming to understand the overall structure and semantics of the entire image or text. The bilinear attention mechanism is a method for calculating the attention distribution between two inputs. It can calculate the correlation between them based on inputs from different modalities. The attention weight matrix is ​​a matrix calculated by the bilinear attention mechanism, which represents the attention distribution between different features. The higher the weight, the stronger the correlation between the corresponding features.

[0091] First, the image global features and visual core features extracted from the image are combined with the text global features and text core features extracted from the text, and a multimodal semantic fusion model is constructed to perform semantic learning at different levels. In this process, the system performs modal core semantic learning and modal global semantic learning. Modal core semantic learning focuses on extracting the most important core information in each modality, such as the main objects in the image or the key phrases in the text. Modal global semantic learning focuses on understanding the overall structure of the entire image or text, such as the overall layout of the image or the main context of the text.

[0092] In order to effectively integrate these features, a bilinear attention mechanism is used. This mechanism measures the correlation between image features and text features by calculating the attention distribution between them. For example, the system calculates the attention weight between each text core feature and all visual features in the image to generate an attention weight matrix. This matrix represents the correlation between each text feature and image feature.

[0093] Next, the system weights the image features and text features through attention weights to generate a multimodal feature representation that combines image and text information. Specifically, the system applies these weights to the image and text features and combines these features into a multimodal feature vector, which is used for further answer generation or decision-making process.

[0094] Finally, the multimodal feature vector is passed to the softmax function classifier to generate the final answer or decision result. In this way, image and text information can be accurately combined in visual question answering tasks to generate high-quality semantic understanding and answers.

[0095] Furthermore, the single-level language learning results are fused to form a target answer prediction vector, including: taking the single-level language learning results as input, predicting the probability distribution in the answer set through a softmax function classifier, and selecting the candidate answer with the highest probability as the target answer prediction vector:

[0096]

[0097] Among them, W lk and W gc is the learnable weight parameter, τ lk and τ gc For a learnable linear projection function, and The dimension of the vector is converted to d A , where A is the number of all answer candidates.

[0098] Specifically, a classifier is a machine learning model that maps input features to specific categories. In this context, the classifier predicts the probability of each candidate answer through a softmax function. The weight parameter W is learned lk and W gc It is a parameter obtained through model training and is used to weight the semantic learning results at different levels to better reflect the importance of each feature when generating the final answer.

[0099] After multimodal semantic learning is completed, multiple single-level semantic learning results are obtained, which reflect the understanding of the semantic relationship between images and texts at different levels. Next, the single-level learning results are fused to form the final answer prediction. This is achieved by using the softmax function, which can convert the linear combination of multiple-level learning results into the probability distribution of candidate answers.

[0100] Specifically, the semantic learning results of each single layer are first passed as input to a classifier. This classifier contains learnable weight parameters W lk and W gc , which are used to weight the learning results of different levels. These weight parameters are obtained through model training and represent the contribution of learning results of different levels to the final answer.

[0101] During the calculation, the linear projection vector τ is applied lk and τgc , these vectors are used to transform the learning results into a new feature space so that the classifier can better understand and process these features. Next, the system uses the softmax function to calculate the probability distribution of each candidate answer. Softmax converts the scores of all candidate answers into a probability distribution, ensuring that the probability of each answer is between 0 and 1, and the sum of the probabilities of all candidate answers is 1. Finally, the system selects the candidate answer with the highest probability as the final target answer prediction vector. For example, when answering the question "What color is the cat in the picture?", the system may calculate multiple possible answers. After passing the softmax function, the system will select the answer with the highest probability as the final output.

[0102] Through this step, the learning results from different levels can be effectively integrated together to generate an accurate answer prediction.

[0103] In summary, the combined visual question answering method based on core-to-global semantic fusion reasoning provided by the embodiment of the present application has the following technical effects:

[0104] 1. By extracting and fusing core and global semantic features, the semantic understanding ability of the visual question answering system in multimodal data is improved, thereby improving the accuracy of answer prediction.

[0105] 2. By introducing MaskRCNN and BiTreeLSTM networks, the global and detail features of the image are processed more finely, which enhances the ability to capture object relationships and attributes in the image and improves the expression effect of visual features.

[0106] 3. The use of word embedding and GRU network to process text questions enables the system to better understand the semantic structure of the questions, and extract the core features of the text through stop word filtering, thereby improving the accuracy of the text semantic features.

[0107] 4. By calculating the importance vector of the visual core semantic matrix and screening key feature instances, it ensures that the system can focus on the most relevant features when processing complex semantics, thereby improving the efficiency of feature extraction.

[0108] 5. By weighting and normalizing image and text features, the system can better filter and integrate multimodal information and improve the accuracy and expression of global semantic features.

[0109] Any step of the method described above can be stored as a computer instruction or program in an unlimited computer memory, and can be called and recognized by an unlimited computer processor to implement any method in the embodiments of the present application, without any unnecessary restrictions.

[0110] Furthermore, the first or second mentioned above may not only represent an order relationship, but may also represent a specific concept, and / or refer to multiple elements that can be selected individually or in full. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A combined visual question answering method based on core-to-global semantic fusion reasoning, characterized in that: The method comprises: Obtain image data to be processed and corresponding text question data; Inputting the image data into a pre-built semantic feature extraction model to obtain image global features and visual core features; Inputting the text question data into the semantic feature extraction model to obtain the text question global features and text core features; Passing the image global features, the visual core features, the text question global features, and the text core features to a preset multimodal semantic fusion model to perform multimodal semantic learning at different levels to obtain a single-level language learning result; Fusing the single-level language learning results to form a target answer prediction vector; The target prediction answer is calculated based on the target answer prediction vector.

2. The method according to claim 1, characterized in that: The step of inputting the image data into a pre-built semantic feature extraction model to obtain the global features and visual core features of the image includes: Extracting the image data based on the MaskRCNN network of the semantic feature extraction model to obtain the global features of the image; Obtain image RoI features and object candidate boxes through the MaskRCNN network; Using the image RoI feature as an embedding of the image, that is, using the image RoI feature as an image data feature; Use Prim's algorithm to build a BiTreeLSTM network; Using the BiTreeLSTM network, processing the image data features to generate object features; Based on the object features, further generating relationship features through the BiTreeLSTM network; Adding an attribute branch in the object candidate frame, the attribute branch is composed of a multi-layer perceptron, the attribute branch takes the image data feature as input, and performs attribute classification prediction through a cross entropy loss function to obtain attribute features; The object feature, the relationship feature, and the attribute feature are combined to obtain the visual core feature.

3. The method according to claim 1, characterized in that: The step of inputting the text question data into the semantic feature extraction model to obtain the text question global features and text core features includes: Mapping the word information in the text question data to a low-dimensional feature space through word embedding technology to form a word embedding vector; The word embedding vectors are sequentially sent to the GRU network unit for sequential feature learning to obtain the global features of the text question; Processing the text question data using a stop word filter to obtain core words in the text question data after screening, wherein the stop word filter includes a stop word list based on NLTK and rare words whose frequency in the entire text question set is lower than a set threshold; The core words are mapped to core features of the text through word embedding technology.

4. The method according to claim 2, characterized in that: The method further comprises: The object features, the relationship features, and the attribute features are combined in the manner of object, object-attribute, object-relationship-object to generate a visual core semantic matrix; The text core features are used as supervisory information, the importance vector of each visual semantic feature in the visual core semantic matrix is ​​calculated, and matched with the text core features to identify the visual core semantic feature instances that are directly related to the text problem.

5. The method according to claim 4, characterized in that: The method of using the text core features as supervision information and calculating the importance vector of each visual semantic feature in the visual core semantic matrix includes: Given the visual core semantic feature matrix and the core features of the text where n ik and n qk represents the number of feature instances, d ik and d qk Represents the feature dimension of each instance; The importance of the visual core semantic feature instance and the text core feature instance can be calculated as: Among them, τ qk ( ) and τ ik ( ) is a learnable linear projection vector, and Projection and and For the importance matrix a qi The importance vector can be obtained by row summing By sorting the values ​​in the importance vector, the instances corresponding to the top K maximum values ​​are selected and combined into a new visual core feature. Among them, the value of K is: K = λ × mean (d qk ), where λ is a hyperparameter, mean(d qk ) represents the average number of core words extracted from all text questions, λ=6.

6. The method according to claim 1, characterized in that: The method further comprises: Based on the image core semantic feature matrix and image features f i , calculate the weight graph ψ i : where τ i ( ) is a learnable linear projection vector, Projection By weighting the image feature fi using the weight map, the filtered image global features can be obtained: in, is the channel normalized vector, ⊙ represents the Hadamard product, and similarly, the text core feature f′ qk For supervision information, the weight map ψ can be calculated q And the global features of the filtered text question are: Among them, τ q ( ) is a learnable linear projection vector, Projection 7. The method according to claim 1, characterized in that: The method of passing the image global features, the visual core features, the text question global features, and the text core features to a preset multimodal semantic fusion model to perform multimodal semantic learning at different levels to obtain a single-level language learning result includes: The multimodal semantic learning at different levels includes multimodal core semantic learning and multimodal global semantic learning; The visual core features and the text core features are used as inputs of the multimodal core semantic learning, and a bilinear attention mechanism is adopted to fuse the visual core features and the text core features; The output of the multimodal core semantic learning is a dimension d lk Joint Expression Vector for The mth element in , its corresponding value can be calculated as: in, and is the learnable parameter matrix, is a bilinear attention map, which is calculated as: in, and is the learnable parameter matrix; The image global feature, the text question global feature and the joint expression vector As the input of the multimodal global semantic learning, wherein, in the multimodal global semantic learning, the global feature of the text question and the joint expression vector Together as supervision information, the supervision information for multimodal global semantic learning is Where || represents matrix concatenation, and the same bilinear attention mechanism as the multimodal core semantic learning is adopted to perform multimodal feature fusion; The output of global overall learning is a dimension d gc Joint Expression Vector in, The jth element in , its corresponding value can be calculated as: in, and is the learnable parameter matrix, is a bilinear attention map, which is calculated as: in, and is the learnable parameter matrix; The joint expression vector and the joint expression vector As a single-level language learning outcome.

8. The method according to claim 1, characterized in that: The step of fusing the single-level language learning results to form a target answer prediction vector includes: Taking the single-level language learning result as input, the probability distribution in the answer set is predicted through the softmax function classifier, and the candidate answer with the highest probability is selected as the target answer prediction vector: Among them, W lk and W gc is the learnable weight parameter, τ lk and τ gc For a learnable linear projection function, and The dimension of the vector is converted to d A , where A is the number of all answer candidates.

Images