Multimodal question answering methods, devices, and equipment based on multi-stage scene graph enhancement

Abstract

This application discloses a multimodal question-answering method, apparatus, and device based on multi-stage scene graph enhancement, relating to the field of artificial intelligence technology. It generates a structured scene graph of an image through a visual language model, and performs multi-stage alignment of the visual features corresponding to the scene graph and the text features corresponding to the natural language question, achieving cross-modal embedding fusion. Then, it utilizes knowledge retrieval and prompting learning mechanisms, combined with external knowledge, to generate accurate question-answering results. Thus, with multimodal semantic understanding and knowledge reasoning as its core, through the synergistic optimization of graph structure modeling, semantic alignment, and knowledge enhancement, it achieves deep semantic fusion and logically consistent question answering between images and text, thereby significantly improving the accuracy and robustness of question answering in complex scenarios.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a multimodal question answering method, apparatus and device based on multi-stage scene graph enhancement. Background Technology

[0002] With the rapid development of the internet and artificial intelligence technologies, the scale of multimedia information worldwide has grown exponentially. Information carriers have evolved from traditional plain text to multimodal knowledge forms integrating images, videos, audio, and text. This trend has driven the rise of cross-modal understanding and reasoning tasks, among which Visual Question Answering (VQA) technology, as an important branch of visual semantic understanding, has become a research hotspot in the field of artificial intelligence. The goal of VQA is to enable computers to comprehensively understand input image content and natural language questions, just like humans, and generate semantically reasonable, logically consistent, and factually accurate natural language answers. However, traditional VQA suffers from insufficient cross-modal semantic fusion and a lack of semantic interpretability. Summary of the Invention

[0003] This application provides a multimodal question answering method, apparatus, and device based on multi-stage scene graph enhancement, which at least solves the problems of insufficient cross-modal semantic fusion and lack of semantic interpretability in related technologies.

[0004] In a first aspect, embodiments of this application provide a multimodal question-answering method based on multi-stage scene graph enhancement, including: Acquire the image data to be processed and the user's natural language questions; The image data is input into a pre-trained visual language model to obtain a scene graph. The scene graph is represented as a triple, including nodes, connecting edges, and attribute features. The nodes are detected object instances in the image, the attribute features are the visual attributes and semantic labels of the objects, and the connecting edges are the spatial and semantic relationships between objects. Visual features corresponding to scene graphs of different modalities and text features corresponding to natural language problems are mapped to vector spaces of the same dimension, and cross-modal feature fusion is achieved through a multi-stage semantic alignment network to obtain a unified semantic vector after semantic alignment. Based on the unified semantic vector, retrieve matching target knowledge fragments from an external knowledge base; Based on the target knowledge fragments and natural language questions, an input sequence containing multi-source information is constructed through a dynamic prompting learning mechanism, and a context-aware generative model is used to model semantic dependencies to generate logical question-and-answer results.

[0005] Secondly, embodiments of this application provide a multimodal question-answering device based on multi-stage scene graph enhancement, including: The acquisition module is used to acquire the image data to be processed and the user's natural language questions; The input module is used to input the image data into a pre-trained visual language model to obtain a scene graph. The scene graph is represented as a triple, including nodes, connecting edges, and attribute features. The nodes are detected object instances in the image, the attribute features are the visual attributes and semantic labels of the objects, and the connecting edges are the spatial and semantic relationships between objects. The alignment module is used to map the visual features corresponding to scene graphs of different modalities and the text features corresponding to natural language problems to a vector space of the same dimension, and to achieve cross-modal feature fusion through a multi-stage semantic alignment network to obtain a unified semantic vector after semantic alignment. The retrieval module is used to retrieve matching target knowledge fragments from an external knowledge base based on the unified semantic vector. The generation module is used to construct an input sequence containing multi-source information based on the target knowledge fragment and natural language question through a dynamic prompting learning mechanism, and to model semantic dependencies using a context-aware generative model to generate logical question-and-answer results.

[0006] Thirdly, embodiments of this application provide an electronic device, which includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the steps of the multimodal question answering method based on multi-stage scene graph enhancement as described in any embodiment of the first aspect.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the steps of the multimodal question-answering method based on multi-stage scene graph enhancement as described in any embodiment of the first aspect.

[0008] Fifthly, embodiments of this application provide a computer program product, which is stored in a storage medium and executed by at least one processor to implement the steps of the multimodal question answering method based on multi-stage scene graph enhancement provided in the first aspect of embodiments of this application.

[0009] The multimodal question-answering method, apparatus, and device based on multi-stage scene graph enhancement in this application generate a structured scene graph of an image through a visual language model. It then performs multi-stage alignment of the visual features corresponding to the scene graph and the text features corresponding to the natural language question, achieving cross-modal embedding fusion. Finally, it utilizes knowledge retrieval and prompting learning mechanisms, combined with external knowledge, to generate accurate question-answering results. Thus, with multimodal semantic understanding and knowledge reasoning at its core, and through the synergistic optimization of graph structure modeling, semantic alignment, and knowledge enhancement, it achieves deep semantic fusion and logically consistent question-answering between images and text, significantly improving the accuracy and robustness of question answering in complex scenarios. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating a multimodal question-answering method based on multi-stage scene graph enhancement provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a question-and-answer scenario regarding the construction commencement and construction time provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a multimodal question-answering device based on multi-stage scene graph enhancement provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0012] Figure label: A multimodal question-answering device 300 based on multi-stage scene graph enhancement includes an acquisition module 301, an input module 302, an alignment module 303, a retrieval module 304, and a generation module 305. Electronic device 400, processor 401, memory 402, communication interface 403, bus 410. Detailed Implementation

[0013] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0014] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0015] It should be noted that the acquisition, storage, use, and processing of data in this application embodiment all comply with the relevant provisions of national laws and regulations.

[0016] With the rapid development of the internet and artificial intelligence technologies, the scale of multimedia information worldwide is growing exponentially. Information carriers have gradually evolved from traditional plain text to multimodal knowledge forms integrating images, videos, audio, and text. This trend has driven the rise of cross-modal understanding and reasoning tasks, among which Visual Question Answering (VQA) technology, as an important branch of visual semantic understanding, has become a research hotspot in the field of artificial intelligence. The goal of VQA is to enable computers to comprehensively understand input image content and natural language questions, just like humans, and generate semantically reasonable, logically consistent, and factually accurate natural language answers.

[0017] Currently, visual question answering technology is typically based on deep neural network frameworks. It extracts visual features from images using convolutional neural networks (CNNs) and encodes natural language questions using recurrent neural networks (RNNs) or transformer structures, then performs feature fusion and matching in a joint representation space. However, most of these models rely on the overall alignment of visual and linguistic features, failing to fully capture fine-grained semantic cues in images. For example, for images containing multiple objects, complex interactions, or implicit scene logic, the models often fail to identify key visual regions and struggle to understand semantically unexpressed relationships, such as "what are the people in the picture doing?" or "the spatial relationship between vehicles and buildings." These problems lead to vague, one-sided, or even erroneous answers generated by the models, limiting their application in scenarios requiring high accuracy and interpretability.

[0018] In professional fields such as public security, traffic safety, and counter-terrorism, visual question answering (VQA) tasks face more complex challenges. Images in these scenarios typically contain multi-subject, multi-scale, and high semantic density information, such as faces, vehicles, scene signs, timestamps, and actions. Traditional VQA models based on end-to-end feature fusion perform poorly in these tasks, mainly due to the following shortcomings: First, they lack the ability to understand professional knowledge and cannot utilize external domain knowledge for semantic completion; second, they lack cross-modal reasoning mechanisms and do not adequately model the logical connection between visual evidence and linguistic reasoning; third, the models have poor interpretability, making it difficult to clearly demonstrate the reasoning basis during the decision-making process, which is detrimental to the traceability and security requirements of public security systems.

[0019] To address the aforementioned issues, academia and industry have proposed Knowledge-based Visual Question Answering (KB-VQA) technology. This approach introduces external knowledge graphs or structured knowledge bases into traditional VQA, aiming to achieve cross-modal enhancement from "visual perception" to "knowledge reasoning" by fusing image perception information with symbolic knowledge representation. By linking visual object recognition results with entities and relationships in the knowledge graph, the KB-VQA system can utilize external knowledge to complete factual completion and logical reasoning, thereby effectively improving the accuracy and semantic richness of question answering. For example, when questions involve the function, category, or social attributes of objects in an image, the knowledge graph can provide supplementary semantic explanations, making the model-generated answers more logical and consistent.

[0020] However, existing KB-VQA methods still face significant bottlenecks in cross-modal semantic alignment and multi-scale feature fusion. First, at the semantic alignment level, there are distributional differences between the semantic spaces of visual features and knowledge graph entities, making it difficult to achieve true semantic unification using traditional feature concatenation or attention weighting. Second, in terms of multi-scale fusion, current models often exhibit information redundancy or loss when integrating local image features, global context, and external knowledge, leading to inference results that deviate from the semantics of the question. Furthermore, during knowledge retrieval and linking, the model needs to quickly locate the most relevant entities and relationships to visual elements within a vast knowledge base, placing higher demands on the system's real-time performance and accuracy.

[0021] With the rise of large-scale visual language pre-trained models (such as CLIP, BLIP, and LLaVA), visual question answering technology is gradually evolving from feature-level fusion to semantic-level reasoning. These models, pre-trained on large-scale image-text aligned data, possess a certain degree of cross-modal generalization ability, but still rely on high-quality domain knowledge for supplementation in specific tasks. When facing specialized scenarios (such as public safety, military reconnaissance, and medical image analysis), the knowledge coverage and factual accuracy of pre-trained models are limited, making it difficult to handle complex semantic reasoning requirements. Therefore, embedding knowledge graphs into VQA systems, enabling models to perform knowledge-driven logical reasoning based on visual understanding, has become an important direction for the next generation of intelligent question answering systems.

[0022] While current visual question answering technology has made some progress in general scenarios, it still has significant shortcomings in knowledge enhancement, semantic alignment, and multimodal fusion. In particular, in complex visual semantic understanding tasks in professional fields such as public security, the robustness and interpretability of existing methods need to be improved.

[0023] To address the problems in related technologies, this application provides a method, apparatus, and device for multi-stage scene graph enhanced question answering (MSG-RAG), which integrates visual perception, knowledge reasoning, and semantic optimization to overcome the limitations of traditional VQA in cross-modal reasoning and knowledge utilization, and achieve more accurate, interpretable, and efficient visual question answering.

[0024] The multimodal question answering method based on multi-stage scene graph enhancement provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0025] It should be noted that the multimodal question answering method based on multi-stage scene graph enhancement in this application embodiment can be applied to fields such as intelligent monitoring, decision support, education and training, intelligent customer service, public security investigation, and medical diagnosis.

[0026] Figure 1 A flowchart illustrating a multimodal question-answering method based on multi-stage scene graph enhancement, according to an embodiment of this application, is shown. Figure 1 As shown, this multimodal question answering method based on multi-stage scene graph enhancement may specifically include the following steps: S101. Obtain the image data to be processed and the user's natural language question; S102. Input the image data into a pre-trained visual language model to obtain a scene graph. The scene graph is represented as a triple, including nodes, connecting edges, and attribute features. The nodes are detected object instances in the image, the attribute features are the visual attributes and semantic labels of the objects, and the connecting edges are the spatial and semantic relationships between objects. S103. Map the visual features corresponding to scene graphs of different modalities and the text features corresponding to natural language problems to vector spaces of the same dimension, and achieve cross-modal feature fusion through a multi-stage semantic alignment network to obtain a unified semantic vector after semantic alignment. S104. Based on the unified semantic vector, retrieve matching target knowledge fragments from the external knowledge base; S105. Based on the target knowledge fragment and the natural language question, construct an input sequence containing multi-source information through a dynamic prompting learning mechanism, and use a context-aware generative model to model semantic dependencies to generate logical question-and-answer results.

[0027] In other words, the multimodal question answering method based on multi-stage scene graph enhancement in this application firstly performs content understanding on the input image through a visual language model to generate a structured scene graph to describe the entities, attributes, and relationships in the image; secondly, a multi-stage fusion strategy is adopted in the retrieval stage to align and embed visual features and text features at both macro and micro levels; thirdly, cross-modal retrieval is performed based on the aligned feature space to obtain external knowledge related to the semantics of the question; finally, through prompt learning and context generation mechanisms, the retrieved knowledge is combined with the input question to generate accurate question answering results.

[0028] Therefore, visual content is elevated from pixel-level features to a structured representation at the semantic level. Through multi-stage semantic graph enhancement and cross-modal fusion mechanisms, visual elements, textual semantics, and external knowledge form a unified embedding space, ultimately driving a large language model to complete knowledge-driven multimodal question answering tasks. This includes four core stages: scene graph generation, semantic alignment, knowledge retrieval, and answer generation. Each stage is connected through semantically consistent intermediate representations, forming an end-to-end multimodal reasoning chain.

[0029] As can be seen, compared with related technologies, this application has the following beneficial effects: it adopts multi-stage scene graph modeling to realize hierarchical expression of image semantics; it introduces cross-modal alignment network to enhance the interaction capability of visual and linguistic features; and it realizes fine-grained semantic matching through multi-dimensional embedding mechanism to improve question answering accuracy.

[0030] The specific implementation methods for each of the above steps are described below.

[0031] In some embodiments, in S101, image data to be processed is acquired by an image acquisition device; and natural language questions from the user are received by a speech recognition system or a text input interface.

[0032] Optionally, after S101, the input image is enhanced, including brightness adjustment, contrast normalization, image size normalization, color space conversion, and noise reduction. At the same time, the natural language question is grammatically corrected and semantically normalized to achieve standardized preprocessing of the input data, resulting in preprocessed image data and natural language question text, so as to ensure the quality and consistency of the input data.

[0033] In some embodiments, in S102, a visual-language model (VLM) is used to perform multi-layer semantic parsing on the input image, extract key objects, attributes and relationships, and construct a structured scene graph.

[0034] In practice, advanced visual language models (such as CLIP and BLIP) are used to encode the input image at multiple levels. This not only extracts the global visual features of the image but also enhances local features in key regions, resulting in a rich visual representation. Simultaneously, a pre-trained visual language model generates a preliminary semantic description of the image. These descriptions cover core information such as the main objects, scene layout, and key attributes, providing a crucial textual foundation for subsequent scene graph construction. In other words, a vectorized representation of the image and a preliminary semantic description are obtained.

[0035] In practice, visual elements in an image (such as objects, people, and backgrounds) and their relationships and attributes are represented in the form of a graph structure, thus constructing a structured visual scene graph. First, entity recognition technology extracts key objects and concepts from the initial semantic description, forming a set of nodes in the graph. Then, relation extraction technology identifies spatial, action, and attribute relationships between these objects, forming a set of edges in the graph. Finally, attribute annotation technology adds detailed attribute information, such as color, shape, size, and state, to each object node. The entire scene graph is organized in the form of triplets (subject-relationship-object), forming a semantically rich and clearly structured graph. This structured representation not only reduces redundancy and noise in natural language descriptions but also enhances the deep understanding and semantic perception of image content, providing strong support for subsequent knowledge retrieval and question answering generation.

[0036] As an optional implementation, the model extracts the low-level visual features of the image through a convolutional neural network; captures the global contextual information of the image through a transformer architecture; and constructs a semantic representation of the image through a graph neural network to obtain a scene graph.

[0037] The generated scene graph is represented as a set of triples as follows: ; in, Represents a set of entity nodes. Let i represent the set of relationships, where i represents the source entity node (the i-th entity node in the set of entity nodes), and j represents the target entity node (the j-th entity node in the set of entity nodes).

[0038] That is, the scene graph includes: object instances detected in the image as nodes, visual attributes and semantic labels of objects as attribute features, and spatial and semantic relationships between objects as connecting edges.

[0039] In this way, the scene graph not only preserves the explicit object information of the image, but also expresses the structural semantics of the image through relational edges, providing structural support for subsequent multimodal alignment. Compared with traditional global feature vector representation, scene graph modeling can achieve a hierarchical understanding from local objects to overall semantics, significantly improving the interpretability and transferability of visual semantics.

[0040] As an optional implementation, the model first locates the main entities in the image using an object detection network (such as Faster R-CNN or DETR), and then uses a region feature encoder to convert the visual representation of each entity into a semantic vector; at the same time, it uses a relationship detection module to identify the semantic relationships between entities, such as "person-riding-horse" and "car-parked-on-the-roadside".

[0041] In another embodiment, the model includes a visual encoder, a text encoder, and a cross-modal fusion unit to extract multi-scale visual information from images, including object-level bounding box features obtained through object detection algorithms and relation-level interaction features obtained through graph convolutional networks. Object-level features generate candidate regions through a region proposal network and extract features through ROI pooling. Relation-level features propagate information between objects through a message passing mechanism and enhance the integrity of scene semantic expression through a multi-level feature fusion mechanism. This fusion mechanism includes a combination of various methods such as feature concatenation, attention weighting, and gated recurrent units, ultimately outputting a scene graph representation with rich semantics.

[0042] Furthermore, in some embodiments, after S102, the local details and global visual features of the image are extracted by a visual encoder using a multi-scale feature pyramid structure; and the word-level and sentence-level representations of natural language problems are obtained by a text encoder using a pre-trained language model to obtain the corresponding text features.

[0043] In some embodiments, in S103, firstly, a pre-designed multi-stage semantic alignment network is used to project the visual features of the image, the structured text features of the scene graph, and the natural language question features into the same vector space. During the projection process, the alignment network uses contrastive learning and metric learning techniques to shorten the distance between semantically related visual and text features and widen the distance between unrelated features, thereby achieving cross-modal semantic alignment.

[0044] Furthermore, the multi-stage semantic alignment network adopts an end-to-end trainable hierarchical architecture, including a macro-level semantic alignment layer and a micro-level semantic alignment layer. The macro-level semantic alignment layer uses a scaled dot product-based global attention mechanism to align visual features corresponding to the scene graph and text features corresponding to the natural language question; it also calculates the similarity matrix between the global image description vector and the question semantic vector to achieve cross-modal association. The micro-level semantic alignment layer uses a cross-attention-based local interaction mechanism to align features in specific regions of the image with the embedded representations of keywords in the question; it also establishes a fine-grained correspondence between image region feature maps and question word embedding sequences to achieve accurate semantic matching. The outputs of the macro-level and micro-level semantic alignment layers are adaptively weighted through a gated fusion unit to obtain a unified semantic vector that retains both global consistency and local details. In other words, this multi-stage semantic alignment network first performs linear projection and normalization at the feature level, then performs cross-attention calculation in the semantic space, and finally obtains a semantically consistent unified embedding representation through residual connections and layer normalization.

[0045] Optionally, the weights of each modal feature can be adaptively adjusted according to the current problem and image content to achieve more refined cross-modal interaction.

[0046] In this way, feature representations from different modalities are mapped to a unified semantic space, and cross-modal semantic alignment and feature fusion are achieved, resulting in a unified, multimodal semantic vector. This vector integrates visual information from images, structured information from scene graphs, and textual information from questions, providing a high-quality query representation for subsequent knowledge retrieval.

[0047] In another embodiment, semantic initialization of the question and image is performed globally, mapping text embeddings and scene graph node features to a shared semantic space. Subsequently, an attention-based cross-modal interaction network is introduced at the local level to match keywords in the text with image nodes, capturing fine-grained semantic relationships. The goal of this stage is to minimize the semantic distance between visual and linguistic representations through a joint optimization function, thereby obtaining a unified cross-modal representation vector. ; in, The image representation output by the visual scene graph encoder. The language representation output by the text encoder. This represents a semantic alignment mapping function.

[0048] Through this mechanism, the model can achieve hierarchical fusion of visual structure and linguistic semantics, allowing implicit semantic relationships in images to be explicitly activated in the linguistic space. Compared with traditional single-stage fusion methods, multi-stage alignment has significant advantages at both macroscopic and microscopic semantic levels, capturing global semantic consistency while recognizing local detail differences.

[0049] Furthermore, in some embodiments, the retrieval process in S104 can be divided into two stages: coarse-grained retrieval and fine-grained retrieval. Specifically, in the coarse-grained retrieval stage, a unified semantic vector is used to perform preliminary screening in the knowledge base, returning a relatively large set of candidate knowledge (typically containing dozens to hundreds of relevant documents or knowledge fragments). The goal of this stage is to ensure the recall rate of the retrieval and avoid missing any potentially relevant information. In the fine-grained retrieval stage, the candidate set returned by the coarse-grained retrieval is sorted and screened more finely. By calculating more complex similarity metrics (such as interactive computation based on attention mechanisms) and introducing more contextual information (such as relational paths in the scene graph), the most relevant and accurate small number of knowledge fragments (typically 3-5) are selected from the candidate set. This multi-stage retrieval strategy ensures both the coverage and accuracy of the retrieval, effectively addressing the challenges of information diversity and complexity in multimodal question answering tasks.

[0050] In practice, knowledge retrieval is based on similarity calculation using dense vector embedding. In the coarse retrieval stage, an efficient approximate nearest neighbor search algorithm is used to quickly calculate the semantic relevance between the question representation and all document representations in the knowledge base. Top-K candidate documents are selected from the large-scale knowledge base using inner product or cosine similarity measures. In the fine ranking stage, a deep matching model based on an interactive attention mechanism is used to finely calculate the semantic association strength between the question and each candidate document. The complex interaction between the question and the document is modeled through a multi-layer Transformer encoder, and the ranking learning algorithm is used to rank the candidate knowledge from high to low according to the relevance score to select the optimal candidate knowledge. The ranking process also considers the diversity and coverage of the results to ensure that the knowledge retrieved in the end is both relevant and comprehensive.

[0051] In other words, based on a unified semantic vector, a multi-stage knowledge retrieval process is performed, including: querying multiple candidate documents from all documents in an external knowledge base that meet the similarity condition of the unified semantic vector; using a deep matching model based on an interactive attention mechanism to calculate the semantic relevance between the unified semantic vector and each of the candidate documents; and using a ranking learning algorithm to sort the multiple candidate documents from high to low semantic relevance to obtain the optimal target knowledge fragment.

[0052] In this way, by performing external knowledge retrieval in a semantically aligned unified representation space, the semantic gaps in visual and linguistic information can be compensated for. This stage employs a semantic similarity-based retrieval strategy, calculating the semantic relevance between the question representation and entity relationships in the knowledge graph, and selecting the knowledge fragments that best match the question from the external knowledge base. The retrieved knowledge content may include factual triples, attribute definitions, relational chains, and semantic context descriptions. To ensure the accuracy and efficiency of the retrieval, a hierarchical index structure is adopted at the bottom layer, dividing the knowledge graph into two levels: a topic domain index and a local entity index, thereby achieving fast approximate nearest neighbor search. The retrieved knowledge fragments are structured and encoded, then fused with the scene graph semantics to form an enhanced semantic representation. Its function is to provide the model with the supplementary information needed for implicit reasoning, so that the system has stronger factual reasoning ability and semantic completeness.

[0053] Furthermore, in some embodiments, in S105, a context generation mechanism based on prompt learning is introduced to unify the modeling of image semantics, textual questions, and external knowledge. Specifically, the model integrates the above information into a multimodal context input large language model in natural language form through dynamic prompt templates. That is, it constructs a structured input template, which includes the user's question, retrieved relevant knowledge, scene graph description of the image, and possible instruction prompts, thereby generating a natural language answer.

[0054] Optionally, to improve the accuracy and consistency of the output results, a beam search strategy is used in the reasoning stage to explore multiple candidate answer paths, and the results are filtered based on confidence scores and logical consistency.

[0055] Optionally, a self-verification mechanism can be introduced during the generation process. After generating an initial answer, the model automatically performs a consistency check against the input context. If a logical conflict or semantic mismatch is found, a self-feedback correction is triggered, thereby improving the reliability of the final output and ensuring that the output answer is not only accurate but also conforms to human expression habits and logical norms.

[0056] Optionally, it can support multi-turn dialogue and dynamic context management, handle complex interactive question-and-answer scenarios, record historical question-and-answer records and dialogue topics, and maintain semantic coherence and topic relevance in multi-turn interactions.

[0057] Optionally, for the prompt template, the composition structure and information density can be adaptively adjusted according to the question type and difficulty. Multi-source heterogeneous knowledge can be embedded into the input sequence of the question-answering generation model through special delimiters and positional encoding. The embedding strategies include various methods such as prefix concatenation, intermediate insertion, and hierarchical organization to enhance the generation model's ability to understand complex contextual semantics. This understanding ability is specifically reflected in multiple cognitive dimensions such as semantic disambiguation, logical reasoning, multi-hop reasoning, and common sense reasoning, which significantly improves the reasoning ability and answer quality of the question-answering system.

[0058] In some optional embodiments, parameters such as the confidence threshold of the visual language model, the fusion weights of the multimodal alignment network, and the recall quantity of knowledge retrieval can be dynamically adjusted based on user feedback and scoring metrics. Specifically, the generated question-and-answer results can be automatically scored using multiple quality metrics, including semantic accuracy, logical consistency, and language fluency, and the answers can be filtered and reordered based on the scoring results to ensure the reliability of the output quality.

[0059] In some optional embodiments, an anomaly handling mechanism is designed that can detect abnormal input data, processing errors, and abnormal generated results, and ensure basic service quality in abnormal situations through degradation strategies and backup plans.

[0060] In some optional embodiments, the scene graph structure, retrieval process, and generated results can be presented graphically to achieve a visual display; and, it supports interactive user exploration and result verification, enhancing interpretability and user experience.

[0061] In addition, it should be noted that for computationally intensive tasks, such as S102 and S104, execution is distributed across multiple computing nodes. Resource utilization is optimized through load balancers and task schedulers to ensure efficient operation under large-scale data.

[0062] Below, in conjunction with Figure 2 Taking the question-and-answer scenario of building commencement and construction time as an example, this paper will specifically explain the multimodal question-and-answer method based on multi-stage scene graph enhancement.

[0063] like Figure 2 As shown, firstly, high-quality input image data is acquired through image acquisition equipment; simultaneously, natural language questions from users, such as "When did the building in the picture start construction?", are received through a speech recognition system or text input interface. Then, the input data undergoes standardized preprocessing, including image size normalization, color space conversion, and text segmentation.

[0064] Furthermore, a visual language model pre-trained on a large-scale multimodal dataset is used to perform deep semantic parsing on the input image. This model extracts the low-level visual features of the image through a convolutional neural network, captures the global contextual information of the image through a transformer architecture, and finally constructs the semantic representation of the image through a graph neural network, generating a scene graph representation containing rich structured information. The scene graph is organized in the form of triplets, which include the detected object instances in the image as nodes, the visual attributes and semantic labels of the objects as attribute features, and the spatial and semantic relationships between objects as connecting edges.

[0065] Next, dedicated visual encoders and text encoders are used to process the image and question inputs respectively. The visual encoder uses a multi-scale feature pyramid structure to extract local details and global features of the image, while the text encoder uses a pre-trained language model to obtain word-level and sentence-level representations of the question. Finally, the features of different modalities are mapped to a vector space of the same dimension through a feature projection layer. Subsequently, cross-modal feature fusion is achieved through a multi-stage semantic alignment network. This network first performs linear projection and normalization at the feature level, then performs cross-attention calculation in the semantic space, and finally obtains a semantically consistent unified embedding representation through residual connections and layer normalization.

[0066] Furthermore, a two-stage knowledge retrieval process is executed based on a unified embedding representation. In the first stage, an approximate nearest neighbor search algorithm is used to quickly recall candidate documents from a large-scale knowledge base. In the second stage, a deep interaction model is used to finely reorder the candidate documents in order to obtain the external knowledge most relevant to the semantics of the question from both the structured knowledge base and the unstructured document library.

[0067] Finally, based on the retrieved knowledge fragments and the original input question, an input sequence containing multi-source information is constructed through a dynamic prompting learning mechanism. A context-aware generative model is used to model semantic dependencies, generate logical question-and-answer results, and automatically verify and post-process the generated content for optimization.

[0068] As can be seen, the multimodal question answering method based on multi-stage scene graph enhancement in this application has achieved systematic innovation in multimodal information understanding, semantic alignment and knowledge-driven reasoning. It not only overcomes the semantic ambiguity and logical incoherence problems of traditional VQA models, but also significantly improves the interpretability and robustness of the question answering system in complex visual semantic scenarios, providing a solution with broad application potential for intelligent image and text understanding and cognitive reasoning tasks.

[0069] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0070] Based on the same technical concept, corresponding to any of the above embodiments, this application also provides a multimodal question-answering device 300 based on multi-stage scene graph enhancement.

[0071] like Figure 3 As shown, the multimodal question-answering device 300 based on multi-stage scene graph enhancement may include: The acquisition module 301 is used to acquire the image data to be processed and the user's natural language questions; The input module 302 is used to input the image data into a pre-trained visual language model to obtain a scene graph. The scene graph is represented as a triple, including nodes, connecting edges, and attribute features. The nodes are detected object instances in the image, the attribute features are the visual attributes and semantic labels of the objects, and the connecting edges are the spatial and semantic relationships between objects. Alignment module 303 is used to map the visual features corresponding to scene graphs of different modalities and the text features corresponding to natural language problems to vector spaces of the same dimension, and to achieve cross-modal feature fusion through a multi-stage semantic alignment network to obtain a unified semantic vector after semantic alignment. The retrieval module 304 is used to retrieve matching target knowledge fragments from an external knowledge base based on the unified semantic vector. The generation module 305 is used to construct an input sequence containing multi-source information based on the target knowledge fragment and the natural language question through a dynamic prompting learning mechanism, and to model semantic dependencies using a context-aware generative model to generate logical question-and-answer results.

[0072] It should be noted that the device architecture of this application supports both modular and distributed deployment. Each module can operate independently, achieving data communication and intermediate feature transfer through standardized interfaces. This design not only improves the scalability and maintainability of the device but also facilitates functional tailoring and customized expansion in different application scenarios. For example, in public security image analysis scenarios, the retrieval module 304 can be emphasized to enhance the ability to interpret complex behavioral scenarios; in education and multimedia retrieval scenarios, the alignment module 303 can be enhanced to achieve a more natural interactive experience.

[0073] Therefore, this device achieves hierarchical expression of image semantics through multi-stage scene graph modeling, enabling it to understand visual information at the structural level rather than relying solely on surface features. The introduction of a cross-modal alignment network enhances the interaction between visual and linguistic features, achieving fine-grained alignment from the semantic level to the relational level. An enhancement mechanism based on knowledge retrieval allows the model to combine external knowledge for semantic completion and logical reasoning, significantly improving the accuracy and consistency of question-answering results. The overall architecture is clear and training is stable, making it widely applicable in fields such as public security visual question answering, image content analysis, intelligent retrieval, and multimodal knowledge question answering, demonstrating good versatility and scalability.

[0074] It should be noted that, for ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0075] The apparatus described above is used to implement the corresponding multimodal question answering method based on multi-stage scene graph enhancement in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0076] Based on the same technical concept, corresponding to any of the above embodiments, this application also provides an electronic device.

[0077] Figure 4 A schematic diagram of a more specific electronic device hardware structure provided in this embodiment is shown.

[0078] The electronic device 400 may include a processor 401 and a memory 402 storing computer program instructions.

[0079] Specifically, the processor 401 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0080] Memory 402 may include mass storage for data or instructions. For example, and not limitingly, memory 402 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 402 may include removable or non-removable (or fixed) media. Where appropriate, memory 402 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 402 is non-volatile solid-state memory.

[0081] In certain embodiments, the memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Thus, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this application.

[0082] The processor 401 reads and executes computer program instructions stored in the memory 402 to implement any of the multimodal question answering methods based on multi-stage scene graph enhancement in the above embodiments.

[0083] In some examples, electronic device 400 may also include communication interface 403 and bus 410. For example, Figure 4 As shown, the processor 401, memory 402, and communication interface 403 are connected through bus 410 and complete communication with each other.

[0084] The communication interface 403 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0085] Bus 410 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not as a limitation, bus 410 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 410 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0086] For example, the electronic device 400 can be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc.

[0087] Based on the same technical concept, corresponding to any of the methods in the above embodiments, this application also provides a non-transitory computer-readable storage medium. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the multimodal question-answering methods based on multi-stage scene graph enhancement in the above embodiments. Examples of computer-readable storage media include non-transitory computer-readable storage media, such as portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, etc.

[0088] Based on the same technical concept, corresponding to any of the above embodiments, this application also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processor to execute the multimodal question answering method based on multi-stage scene graph enhancement. Corresponding to the execution entity for each step in each embodiment of the multimodal question answering method based on multi-stage scene graph enhancement, the processor executing the corresponding step can belong to the corresponding execution entity.

[0089] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0090] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0091] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0092] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0093] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A multimodal question answering method based on multi-stage scene graph enhancement, characterized in that, include: Acquire the image data to be processed and the user's natural language questions; The image data is input into a pre-trained visual language model to obtain a scene graph. The scene graph is represented as a triple, including nodes, connecting edges, and attribute features. The nodes are detected object instances in the image, the attribute features are the visual attributes and semantic labels of the objects, and the connecting edges are the spatial and semantic relationships between objects. Visual features corresponding to scene graphs of different modalities and text features corresponding to natural language problems are mapped to vector spaces of the same dimension, and cross-modal feature fusion is achieved through a multi-stage semantic alignment network to obtain a unified semantic vector after semantic alignment. Based on the unified semantic vector, retrieve matching target knowledge fragments from an external knowledge base; Based on the target knowledge fragments and natural language questions, an input sequence containing multi-source information is constructed through a dynamic prompting learning mechanism, and a context-aware generative model is used to model semantic dependencies to generate logical question-and-answer results.

2. The method according to claim 1, characterized in that, The multi-stage semantic alignment network includes a macro-semantic alignment layer and a micro-semantic alignment layer; The method of achieving cross-modal feature fusion through a multi-stage semantic alignment network to obtain a unified semantic vector after semantic alignment includes: A macro-semantic alignment layer is used to align the visual features corresponding to the scene graph and the text features corresponding to the natural language question using a global attention mechanism based on scaling dot product; and a similarity matrix between the global description vector of the image and the semantic vector of the question is calculated to achieve cross-modal association. The micro-semantic alignment layer uses a local interaction mechanism based on cross-attention to align the features of specific regions in the image with the embedded representations of keywords in the question; and establishes a fine-grained correspondence between the image region feature map and the question word embedding sequence to achieve accurate semantic matching. The outputs of the macro-semantic alignment layer and the micro-semantic alignment layer are adaptively weighted by a gated fusion unit to obtain a unified semantic vector.

3. The method according to claim 1, characterized in that, The step of retrieving matching target knowledge fragments from an external knowledge base based on the unified semantic vector includes: Based on the unified semantic vector, query multiple candidate documents from all documents in the external knowledge base that meet the similarity condition with the unified semantic vector; The semantic relevance between the unified semantic vector and each candidate document is calculated using a deep matching model based on an interactive attention mechanism. Then, a ranking learning algorithm is used to sort the multiple candidate documents from high to low semantic relevance to obtain the optimal target knowledge fragment.

4. The method according to claim 1, characterized in that, The acquisition of the image data to be processed and the user's natural language questions includes: Image data to be processed is acquired through image acquisition equipment; The system receives users' natural language questions through a speech recognition system or text input interface.

5. The method according to claim 1, characterized in that, The step of inputting the image data into a pre-trained visual language model to obtain a scene map includes: Extracting low-level visual features of images using convolutional neural networks; Capture global contextual information of images using the transformer architecture; A scene graph is obtained by constructing a semantic representation of the image using a graph neural network.

6. The method according to claim 1, characterized in that, After inputting the image data into a pre-trained visual language model to obtain a scene map, and before mapping the visual features corresponding to the scene maps of different modalities and the text features corresponding to the natural language questions to a vector space of the same dimension, the method further includes: The image's local details and global visual features are extracted using a multi-scale feature pyramid structure via a visual encoder. By using a pre-trained language model through a text encoder, word-level and sentence-level representations of natural language problems are obtained, and corresponding text features are derived.

7. A multimodal question-answering device based on multi-stage scene graph enhancement, characterized in that, include: The acquisition module is used to acquire the image data to be processed and the user's natural language questions; The input module is used to input the image data into a pre-trained visual language model to obtain a scene graph. The scene graph is represented as a triple, including nodes, connecting edges, and attribute features. The nodes are detected object instances in the image, the attribute features are the visual attributes and semantic labels of the objects, and the connecting edges are the spatial and semantic relationships between objects. The alignment module is used to map the visual features corresponding to scene graphs of different modalities and the text features corresponding to natural language problems to a vector space of the same dimension, and to achieve cross-modal feature fusion through a multi-stage semantic alignment network to obtain a unified semantic vector after semantic alignment. The retrieval module is used to retrieve matching target knowledge fragments from an external knowledge base based on the unified semantic vector. The generation module is used to construct an input sequence containing multi-source information based on the target knowledge fragment and natural language question through a dynamic prompting learning mechanism, and to model semantic dependencies using a context-aware generative model to generate logical question-and-answer results.

8. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; when the processor invokes the computer program instructions, it implements the multimodal question answering method based on multi-stage scene graph enhancement as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when invoked by a processor, implement the multimodal question-answering method based on multi-stage scene graph enhancement as described in any one of claims 1-6.

10. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the multimodal question answering method based on multi-stage scene graph enhancement as described in any one of claims 1-6.

Images