Visual question and answer method based on environmental perception and knowledge base

Abstract

The invention discloses a visual question and answer method based on environmental perception and a knowledge base, and aims to realize an accurate visual question and answer function through a reasoning technology of integrating field specific environmental information and knowledge base information. The method comprises the following steps: firstly, extracting image features by utilizing a ResNet model in visual feature extraction, extracting text features by utilizing a BERT model in text feature extraction, and comprehensively processing the features and context information acquired by an environment perception module through multi-modal fusion; and then, on the basis of the fused multi-modal features, through combination of a Wikdata knowledge base and Seq2Seq, an answer is output in a text form, and visual feedback can be provided. Through environment perception, the system can adjust answers in real time to ensure that the system highly adapts to the current scene, and is applied to the fields of intelligent assistants, aided driving, household robots and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual question answering, and in particular relates to a visual question answering method based on environment perception and a knowledge base. Background Art

[0002] Visual Question Answering (VQA) is an interdisciplinary task that integrates computer vision and natural language processing. Its main purpose is to enable machines to answer natural language questions based on image or video content. First, traditional visual question answering systems show certain limitations when facing complex or dynamic environments. In 2020, Garderes et al. used ConceptNet as a knowledge source and embedded entity information into the representation of language. In 2023, Cheng et al. used the bilinear pooling method to embed external knowledge into image expression. In 2023, Liu et al. reasonably used the retrieved external knowledge and the implicit internal knowledge of the visual question answering model itself to enhance the effect. In 2024, Wu et al. obtained the unbiased scene graph of the corresponding image in the visual question answering dataset and the knowledge subgraph of the external knowledge base for fusion. However, they usually rely on pre-trained datasets and ignore the ability of real-time environmental perception, making it difficult to handle information in environmental changes or dynamic scenes. Secondly, existing systems mainly rely on the association between visual features and language features, and lack effective use of background knowledge. Many visual question answering tasks require not only an understanding of visual content, but also a mastery of common sense knowledge and reasoning ability. Existing systems are still insufficient in terms of knowledge base integration and reasoning, resulting in answers to questions that may not be accurate or complete.

[0003] Therefore, how to combine environmental perception capabilities with knowledge base systems to improve the performance of visual question answering systems in complex scenarios has become one of the urgent problems to be solved in this field. Summary of the invention

[0004] The purpose of the present invention is to solve the above-mentioned defects in the prior art and to provide a visual question answering method based on environmental perception and knowledge base to improve the environmental adaptability and answer accuracy of the system.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A visual question answering method based on environment perception and knowledge base, including:

[0007] Step 1: Through image preprocessing, ResNet feature extraction and global average pooling, the spatial attention mechanism is introduced to extract visual features and generate a visual feature vector v for multimodal fusion for question and answer generation;

[0008] Step 2: Obtain contextual text representation through text normalization, BERT word segmentation, and pre-training model, and use Transformer encoding layer and average pooling to generate a fixed-length text feature vector t for multimodal fusion for question and answer generation;

[0009] Step 3: Collect environmental data through the sensor network and perform preprocessing and feature extraction, and finally fuse them into a unified environmental feature vector e for multimodal fusion and visual and text features to generate answers;

[0010] Step 4: Project the visual features, text features, and environmental features to the same dimension through linear transformation and assign different weights for weighted summation. Then, process them through a fully connected layer and nonlinear activation, and finally apply Dropout regularization to generate a comprehensive feature vector for subsequent answer generation.

[0011] In step 5, a structured vector is constructed by extracting matching knowledge information from Wikidata, and then a bidirectional LSTM is used to encode the input feature vector and knowledge vector. The LSTM decoder is then used to generate the output answer, and the context vector and attention mechanism are combined to generate the probability distribution of each word, thereby generating the final answer.

[0012] Furthermore, step 1 is as follows:

[0013] The input image is resized to a fixed size (224×224), which includes cropping, scaling, and padding. The pixel values ​​of the resized image are normalized to a fixed range [-1,1]. The pre-trained ResNet deep convolutional neural network model is used to extract features from the processed image to obtain a high-dimensional feature vector. After the convolution layer of ResNet, a global average pooling method is used to compress the high-dimensional feature vector into a fixed-length feature representation. The spatial attention mechanism is introduced to enhance the influence of key areas in the final decision by applying attention weights to the feature map. The output visual feature vector v will be used as the visual feature input for multimodal fusion for question-answer generation.

[0014] Furthermore, step 2 is as follows:

[0015] The questions input by the user are normalized to remove extra spaces and standardized punctuation, and the text is unified into lowercase. The input questions are decomposed into subword units using BERT's tokenizer to handle unknown words and affixes. The pre-trained BERT model is loaded to obtain context-rich text representations, and the input subwords are mapped to their corresponding embedding vectors. Context encoding is performed through the Transformer encoding layer to generate context representations for each subword, generate special [CLS] tag representations, and extract feature representations of the entire input sequence to capture detailed information in the question. Pooling operations are performed to generate fixed-length text feature vectors, and the pooling operation is average pooling. The attention mechanism is introduced to enhance the feature representation of the key parts of the question. The output text feature vector t will be used as text input for multimodal fusion for question-answer generation.

[0016] Furthermore, step 3 is as follows:

[0017] Use sensor networks to obtain environmental data, including time information, location information, lighting conditions, weather information, temperature and humidity information, and sound environment. Preprocess and extract features from the collected environmental data, periodically encode time information through sine and cosine functions, extract lighting conditions through brightness analysis, obtain location information through GPS positioning or Wi-Fi, obtain weather information through weather API interfaces, obtain temperature and humidity information through temperature and humidity sensors, and extract sound environment through audio signal processing;

[0018] All environmental features are concatenated into a unified environmental feature vector through a feature fusion mechanism, and a simple fully connected layer is used to encode the environmental features to generate a vector representation compatible with visual and text features. This vector representation is used as the input of multimodal fusion and together with visual and text features for answer generation.

[0019] Furthermore, step 4 is as follows:

[0020] It receives visual features, text features, and environmental features and converts them into feature vectors of the same dimension. The model then fuses these feature vectors together by weighted summation, assigning different weights to produce a comprehensive feature vector.

[0021] During the fusion process, the model uses a linear transformation to project each feature vector to the same dimension, and then uses different weights for weighted summation. The fused feature vector undergoes a linear transformation and a ReLU activation function in the fully connected layer to introduce nonlinear characteristics.

[0022] Finally, the model uses the Dropout technique for regularization to reduce overfitting and improve the generalization ability of the model. The output comprehensive feature vector contains visual, textual, and environmental information for subsequent answer generation.

[0023] Furthermore, step 5 is as follows:

[0024] First, it receives a comprehensive feature vector of multimodal fusion, including visual, textual and environmental information. Then, the system uses textual features to match relevant entities from Wikidata and extracts relevant attributes and relationships to form a structured knowledge vector.

[0025] Next, the system uses a bidirectional LSTM model to encode the input feature vector and knowledge vector to generate hidden states. The decoder uses the encoder's hidden state to generate the output answer and uses the attention mechanism to calculate the context vector, which is combined with the decoder's hidden state to generate the final output.

[0026] Finally, the system processes the output of the decoder using a fully connected layer and a softmax layer to generate a probability distribution for each word in the final answer. The entire system aims to combine multimodal features and knowledge graphs to generate accurate answers.

[0027] The beneficial effects of the present invention are:

[0028] Based on the above technical solutions, the present invention provides a visual question-answering method based on environmental perception and knowledge base, which solves the problems of insufficient environmental perception and inaccurate answers in traditional visual question-answering systems through the multimodal fusion of visual features, text features and environmental features, and the introduction of the Wikidata knowledge base. By introducing the environmental perception link, the present invention can acquire and process environmental data in real time, so that the visual question-answering system can automatically adjust its answers according to changes in the environment, thereby improving the adaptability and accuracy of the system. At the same time, in combination with the Wikidata knowledge base, by obtaining the attributes and relationships of related entities from an external knowledge base, the present invention can provide richer and more accurate answers, significantly improving the application effect of the visual question-answering system in complex scenes. Finally, the present invention makes up for the limitations of the existing visual question-answering system under a single data source through the fusion of multimodal data, and improves the accuracy of the answers and the intelligence level of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the overall structure of a specific embodiment of the present invention;

[0030] Figure 2 is a schematic diagram of a visual feature extraction process in a specific embodiment of the present invention;

[0031] Figure 3It is a flowchart of text feature extraction in a specific embodiment of the present invention;

[0032] Figure 4 It is a schematic diagram of generating answers by combining the structure of text, visual and environmental information and connecting the knowledge base in a specific embodiment of the present invention; DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0034] The technical solution of the present invention includes five main steps: visual feature extraction, text feature extraction, environmental perception processing, multimodal fusion and answer generation. By combining the knowledge base, a multimodal visual question answering system is realized. The overall model framework is as follows Figure 1 As shown, the details are as follows:

[0035] The step 1 visual feature extraction, such as Figure 2 As shown in the figure, the input image is normalized. First, in the process of visual feature extraction, the input image is cropped, scaled, and padded to adjust the image to a fixed size of 224×224 pixels. Then, the pixel values ​​of the image are normalized using the normalization formula:

[0036] I norm =

[0037] I is the pixel value matrix, μ is the mean of each channel (RGB), and σ is the standard deviation. The mean μ is [0.485, 0.456, 0.406]; the standard deviation σ is [0.229, 0.224, 0.225]; the three values ​​correspond to the R, G, and B channels respectively. The normalized pixel values ​​are normalized to the range [-1, 1].

[0038] Next, the pre-trained ResNet deep convolutional neural network model is used to extract high-dimensional features from the image. A 3×3 convolution kernel is used, the convolution stride is set to 1, and the output feature tensor of size 7×7×2048 is obtained. Then a global average pooling operation is applied to each feature map, averaging all elements of each feature map and outputting a 2048-dimensional vector v;

[0039] In addition, the system introduces a spatial attention mechanism to enhance the influence of key areas in the final decision by applying attention weights to feature maps.

[0040] The step 2 text feature extraction, such as Figure 3As shown in the figure, a regular expression is used to eliminate consecutive spaces in the text and normalize it into a single space. Then remove symbols such as commas and exclamation marks that are irrelevant to the semantics of the question, and retain punctuation question marks that are related to the meaning of the question. Convert the text to lowercase uniformly to ensure that it is case-insensitive during processing. Convert the input normalized text into subword units, use the BERT tokenizer to insert a special tag [CLS] at the beginning of the text sequence as a global representation of the sentence, and add a [SEP] tag at the end of the sentence to indicate the end of the sentence. Set the input length of the BERT model to a fixed value of 512 subwords. Each subword unit is mapped to a word vector and combined with a position vector and a paragraph vector. Set the BERT-Base model to 12 encoding layers and 768-dimensional hidden layers; the BERT-Large model to 24 encoding layers and 1024-dimensional hidden layers. Each layer has a multi-head self-attention mechanism that can process the dependencies of the input sequence in parallel. The input embedding vector passes through multiple Transformer encoding layers in sequence. The self-attention mechanism in each encoding layer uses 16 attention heads to calculate the relationship between each subword in the input sequence and other subwords. Each layer uses residual connection and normalization operation after each layer. Then, in the context representation of all subwords, the vector representation of the [CLS] tag is extracted, the global feature representation of the [CLS] tag is averaged and pooled, the context representation of each subword is averaged, and the representation of all subwords is integrated into a fixed-length text feature vector. The importance score of each word in the input sequence is calculated through the attention mechanism, and higher weights are assigned to words of high importance. The weighted feature representation is combined with the global feature vector to generate an enhanced text feature vector t.

[0041] The step 3 of environmental perception processing uses a sensor network to obtain surrounding environment information: the system collects a variety of environmental information through the sensor network, including time, location, lighting conditions, weather, temperature, humidity and sound environment. The time data is encoded through sine and cosine functions, and the generated values ​​represent the angle of time within a 24-hour period.

[0042]

[0043] t is the encoded value; Hour is the corresponding time data in hours.

[0044] Location information is obtained through GPS positioning or Wi-Fi, weather information is obtained through the weather API interface, temperature and humidity information is obtained through temperature and humidity sensors, and sound environment is extracted through audio signal processing; the input environmental feature vector is encoded through a fully connected layer, and a vector representation compatible with visual and text features is output. The activation function ReLU is applied to improve nonlinear expression capabilities and prevent overfitting.

[0045] Fe =ReLU(W⋅Input+b)

[0046] Input is the environment feature vector after preprocessing and concatenation; w is the weight matrix of the fully connected layer; b is the bias vector; F e It is the output vector after the fully connected layer and the ReLU activation function, which will be used as the input of multimodal feature fusion and combined with visual and text features.

[0047] The multimodal fusion of step 4 combines visual features, text features and environmental features: the system receives features extracted from vision, text and environment,

[0048] Visual features: F v ∈R dv

[0049] Text features: F t ∈R dt

[0050] Environmental characteristics: F e ∈R de

[0051] where d v , d t , d e Dimensions corresponding to vision, text, and environment.

[0052] And project these features to the same dimension through a linear transformation:

[0053] F v ′=W v F v +b v , where W v ∈R df×dv , b v ∈R df

[0054] F t ′=W t F t +b t , where W t ∈R df×dt , b t ∈R df

[0055] F e ′=W e F e +b e , where W e ∈R df×de , b e ∈R df

[0056] W v ∈R df×dv is the weight matrix of visual features; W t ∈R df×dt is the weight matrix of text features; W e ∈R df×de is the weight matrix of environmental features; b v ∈R df is the bias vector of the visual feature; b t ∈R df is the bias vector of text features; b e ∈R df is the bias vector of environmental features; F v ′ is the visual feature after linear transformation; F t ′ is the text feature after linear transformation; F e ′Environmental features after linear transformation.

[0057] The system then assigns different weights to the features of each modality, and fuses the visual, textual, and environmental features together through weighted summation to generate a comprehensive feature representation:

[0058] F sum =α⋅F v ′+β⋅F t ′+γ⋅F e '

[0059] F sum It is the fusion feature vector after weighted summation; the weighting coefficients of each modal feature of α, β, and γ.

[0060] In order to further process the fused features, the system introduces a fully connected layer for linear transformation.

[0061] F dense =W f F sum +b f

[0062] W f is the weight matrix of the fully connected layer; b f is the bias term of the fully connected layer; F dense is the output of the fully connected layer.

[0063] And introduce nonlinear characteristics through the ReLU activation function:

[0064] F relu =ReLU(F dense )=max(0,F dense )

[0065] F reluis the output after applying the ReLU activation function.

[0066] Regularize the output of the fully connected layer and randomly discard the output of some neurons;

[0067] F dropout =Dropout(F relu ,p)

[0068] F dropout is the output after applying Dropout; p is the dropout probability.

[0069] The answer to step 5 is generated, such as Figure 4 As shown, the final answer is generated according to the feature vector of multimodal fusion, and the comprehensive feature vector of the receiving multimodal fusion step is set to:

[0070] x=[x v ,x t ,x e ]

[0071] x v 、x t 、x e Feature vectors representing visual, textual and environmental information respectively.

[0072] The system matches the fused comprehensive feature vector with the Wikidata knowledge base to extract the attributes and relationships of related entities. Through SPARQL query, the system extracts the relevant knowledge triples (h, r, t), where h represents the head entity, r represents the relationship, and t represents the tail entity; the knowledge information extracted from Wikidata can be represented as a structured vector:

[0073] z=f Wikidata (x)

[0074] z is the knowledge information extracted from the Wikidata knowledge base through matching, represented as a structured vector.

[0075] These triplets are encoded by a bidirectional LSTM model to generate hidden states:

[0076] h t =BiLSTM(x t ,z t ,h t−1 )

[0077] h t represents the hidden state at time step t, x t is the input feature, z t is a knowledge vector.

[0078] The decoder uses the encoder’s hidden state h to generate the output answer,

[0079] o t =LSTM(h t ,h t−1 )

[0080] o t is the output generated at time step t.

[0081] The decoder gradually generates the answer sequence through the LSTM model and introduces the attention mechanism to calculate the similarity between the hidden state of the decoder and all the hidden states of the encoder to generate the context vector.

[0082]

[0083]

[0084] α t,i is the attention weight; c t is the context vector.

[0085] Finally, the output of the decoder passes through the fully connected layer and the softmax layer to calculate the probability distribution of the vocabulary corresponding to each time step, thereby generating the final answer:

[0086] y t =softmax(W o o t +b o )

[0087] y t is the probability distribution of each word when the answer is generated; W o is the weight matrix of the fully connected layer; b o is the bias term of the fully connected layer.

[0088] The scheme described in this embodiment strengthens the effective use of background knowledge and increases the mastery and reasoning ability of common sense knowledge. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0089] The technical contents not described in detail in the present invention are all known technologies.

Claims

1. A visual question answering method based on environment perception and knowledge base, characterized by comprising the following steps: Step 1: Through image preprocessing, ResNet feature extraction and global average pooling, the spatial attention mechanism is introduced to extract visual features and generate a visual feature vector v for multimodal fusion for question and answer generation; Step 2: Obtain contextual text representation through text normalization, BERT word segmentation, and pre-training model, and use Transformer encoding layer and average pooling to generate a fixed-length text feature vector t for multimodal fusion for question and answer generation; Step 3: Collect environmental data through the sensor network and perform preprocessing and feature extraction, and finally fuse them into a unified environmental feature vector e for multimodal fusion and visual and text features to generate answers; Step 4: Project the visual features, text features, and environmental features to the same dimension through linear transformation and assign different weights for weighted summation. Then, process them through a fully connected layer and nonlinear activation, and finally apply Dropout regularization to generate a comprehensive feature vector for subsequent answer generation. In step 5, a structured vector is constructed by extracting matching knowledge information from Wikidata, and then a bidirectional LSTM is used to encode the input feature vector and knowledge vector. The LSTM decoder is then used to generate the output answer, and the context vector and attention mechanism are combined to generate the probability distribution of each word, thereby generating the final answer.

2. A visual question answering method based on environment perception and knowledge base according to claim 1, characterized in that: In step 1, the input image is adjusted to a fixed size (224×224), and the process includes cropping, scaling and padding of the image; the pixel values ​​of the adjusted image are normalized to a fixed range of [-1, 1]; Use the pre-trained ResNet deep convolutional neural network model to extract high-dimensional features from the image and obtain high-dimensional feature vectors; After the convolution layer of ResNet, a global average pooling method is used to compress the high-dimensional feature vector into a fixed-length feature representation; a spatial attention mechanism is introduced to enhance the influence of key areas in the final decision by applying attention weights to the feature map; The output visual feature vector v will be used as the visual feature input and passed to the multimodal fusion step for question and answer generation.

3. The visual question answering method based on environment perception and knowledge base according to claim 1, characterized in that: In step 2, the question input by the user is subjected to text normalization processing to remove redundant spaces and standardized punctuation marks, and the text is unified into lowercase; the input question is decomposed into sub-word units using the BERT tokenizer to process unregistered words and affixes; Load the pre-trained BERT model to obtain context-rich text representation, mapping the input subwords to their corresponding embedding vectors, which include word embedding, position embedding, and paragraph embedding; Context encoding is performed through the Transformer encoding layer to generate a context representation of each subword; a special [CLS] tag representation is generated, and a feature representation of the entire input sequence is extracted to capture detailed information in the question; a pooling operation is performed to generate a fixed-length text feature vector, and the pooling operation is average pooling; Introduce the attention mechanism to enhance the feature representation of the key parts of the problem; The output text feature vector t will be used as text input and passed to the multimodal fusion step for question and answer generation.

4. The visual question answering method based on environment perception and knowledge base according to claim 1, characterized in that: In step 3, a sensor network is used to obtain environmental data, including time information, location information, lighting conditions, weather information, temperature and humidity information, and sound environment; the collected environmental data is preprocessed and feature extracted, the time information is periodically encoded through sine and cosine functions, the lighting conditions are extracted through brightness analysis, the location information is obtained through GPS positioning or Wi-Fi, the weather information is obtained through the weather API interface, the temperature and humidity information is obtained through the temperature and humidity sensor, and the sound environment is extracted through audio signal processing; All environmental features are spliced ​​into a unified environmental feature vector through a feature fusion mechanism, and a simple fully connected layer is used to encode the environmental features to generate a vector representation compatible with visual and text features. This vector representation serves as the environmental information input for multimodal fusion and is used together with visual and text features for answer generation.

5. The visual question answering method based on environment perception and knowledge base according to claim 1, characterized in that: In step 4, visual features, text features and environmental features are received, and their features are represented by the following vectors: Visual features: F v ∈R dv Text features: F t ∈R dt Environmental characteristics: F e ∈R de where d v , d t , d e Dimensions corresponding to vision, text, and environment; Before performing weighted summation, each feature needs to be projected to the same dimension d through linear transformation f ; For visual features: F v ′=W v F v +b v , where W v ∈R df×dv , b v ∈R df For text features: F t ′=W t F t +b t , where W t ∈R df×dt , b t ∈R df Environmental characteristics: F e ′=W e F e +b e , where W e ∈R df×de , b e ∈R df W v ∈R df×dv is the weight matrix of visual features; W t ∈R df×dt is the weight matrix of text features; W e ∈R df×de is the weight matrix of environmental features; b v ∈R df is the bias vector of the visual feature; b t ∈R df is the bias vector of text features; b e ∈R df is the bias vector of environmental features; F v ′ is the visual feature after linear transformation; F t ′ is the text feature after linear transformation; F e ′Environmental characteristics after linear transformation; During the fusion process, different weights are assigned to each modal feature, namely α, β and γ. The fused feature F sum It is expressed as: F sum =α⋅F v ′+β⋅F t ′+γ⋅F e ′ F sum is the fusion feature vector after weighted summation; the weighting coefficients of each modal feature of α, β, and γ; The fused feature F sum In the fully connected layer, linear transformation is used for further processing: F dense =W f F sum +b f W f is the weight matrix of the fully connected layer; b f is the bias term of the fully connected layer; F dense is the output of the fully connected layer; The ReLU activation function is used in the fully connected layer to introduce nonlinear characteristics; F relu =ReLU(F dense )=max(0,F dense ) F relu is the output after applying the ReLU activation function; Regularize the output of the fully connected layer and randomly discard the output of some neurons; F dropout =Dropout(F relu ,p) where F dropout is the output after applying Dropout; p is the dropout probability; Output a comprehensive feature vector containing visual, textual, and environmental information for subsequent answer generation.

6. The visual question answering method based on environment perception and knowledge base according to claim 1, characterized in that: In step 5, the comprehensive feature vector of multimodal fusion is first received: x=[x v ,x t ,x e ] Among them, x v 、x t 、x e Feature vectors representing visual, textual and environmental information respectively; Use the input text features to find matching entities from Wikidata; for matching entities, extract related attributes and relationships through SPARQL; these attributes and relationships form a triple (h, r, t), where h represents the head entity, r represents the relationship, and t represents the tail entity; the knowledge information extracted from Wikidata can be represented as a structured vector: z=f Wikidata (x) z is the knowledge information extracted from the Wikidata knowledge base through matching, represented as a structured vector; The encoder encodes the input feature vector x and knowledge vector z through a bidirectional LSTM model to generate a hidden state h t , the encoding process is: h t =BiLSTM(x t ,z t ,h t−1 ) Among them, h t represents the hidden state at time step t, x t is the input feature, z t is the knowledge vector; The decoder uses the encoder's hidden state h to generate the output answer. The decoding process is: o t =LSTM(h t ,h t−1 ) Among them, t is the output generated at time step t; The decoder calculates the similarity between the current decoder hidden state and all encoder hidden states through the LSTM model, and performs weighted summation to generate the context vector c t : ； ； α t,i is the attention weight; c t is the context vector; The context vector is then combined with the decoder hidden state to generate the final output; The output of the decoder is passed through a fully connected layer and a softmax layer to generate the probability distribution of each word of the final answer: y t =softmax(W o o t +b o ) Among them, y t is the probability distribution of each word when the answer is generated; W o is the weight matrix of the fully connected layer; b o is the bias term of the fully connected layer.

Images