A multi-language knowledge question answering method based on a text-image recognition fusion model

Abstract

The application discloses a kind of multi-language knowledge question and answer method based on image-text recognition fusion model, comprising the following steps: receiving and processing multi-language question and answer request containing image information and text information;Improved heterogeneous graph attention network is introduced, and a multi-modal semantic fusion model is constructed;A semantic role alignment graph is constructed, and a semantic role consistency connection relationship is established between image area nodes and text word nodes;A heterogeneous semantic graph structure is constructed, and a multi-type edge connection relationship is established;Perform structure-guided attention propagation operation, filter effective path using mask mechanism, and output fusion semantic representation vector;Extract the semantic focus in user's question, obtain knowledge matching result and generate candidate answer set;Based on knowledge matching result and candidate answer set, output final answer result and update multi-modal semantic fusion model.The application uses improved heterogeneous graph attention network, realizes multi-language image-text fusion semantic understanding and intelligent question and answer generation.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, and in particular to a multilingual knowledge question answering method based on a text-image recognition fusion model. Background Technology

[0002] Against the backdrop of the continuous evolution of intelligent question-answering systems, the fusion of multimodal information (such as images and text) to improve question-answering accuracy and semantic understanding has become a key research focus in the interdisciplinary field of natural language processing and computer vision. Most existing image-text fusion question-answering technologies are based on joint embedding spaces or encoder-decoder architectures, encoding images and text separately before splicing and fusion, or using attention-guided information interaction. These methods are effective in scenarios where structured text and accompanying images are consistent, and can achieve preliminary semantic matching in general knowledge reasoning and simple question-answering tasks. However, due to the significant differences in semantic structure and expression between images and text, existing fusion strategies often lack the ability to uniformly model the semantic roles of images and text, resulting in an inability to accurately characterize the deep semantic relationships between image regions and text components, thus limiting the model's understanding depth and reasoning breadth.

[0003] In multilingual question answering tasks, significant differences exist in lexical structure, semantic logic, and contextual habits among different languages. Existing multilingual question answering methods mainly rely on pre-trained multilingual encoders for semantic vector extraction, followed by alignment matching using a unified language-independent space. While these methods possess some cross-language transfer capabilities, they often struggle to handle structural differences between image region semantics and specific linguistic expressions in mixed text-image input scenarios. Especially when it comes to fine-grained entity matching and knowledge path retrieval, conventional text-image fusion mechanisms are ill-suited to support context-adaptive question answer generation.

[0004] Existing research has attempted to introduce graph neural networks to model multimodal structural information, constructing nodes from image regions, text words, and semantic labels, and using graph convolution or attention propagation mechanisms for semantic fusion. However, these methods generally fail to distinguish between different types of semantic relationship edges, easily leading to chaotic information propagation paths in the graph structure and making it difficult to effectively constrain the semantic consistency between nodes. Furthermore, most graph structure construction processes are static, lacking mechanisms to dynamically optimize the graph structure by incorporating user language preferences, semantic focus changes, and interactive feedback, thus limiting the system's adaptability to complex contexts.

[0005] Therefore, how to provide a multilingual knowledge question answering method based on image and text recognition fusion model is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a multilingual knowledge question answering method based on a graph-text recognition fusion model. This invention fully utilizes an improved heterogeneous graph attention network, a multimodal semantic alignment mechanism, and a multilingual context fusion technology. It describes in detail the construction of a semantic role alignment graph, a heterogeneous semantic graph structure, and an attention propagation method controlled by a structure mask. It has the advantages of high cross-modal alignment accuracy, sufficient semantic information fusion, and strong adaptability to multilingual question answering.

[0007] A multilingual knowledge question answering method based on a text-image recognition fusion model according to an embodiment of the present invention includes the following steps:

[0008] It receives and processes multilingual question-and-answer requests containing image and text information to generate a multimodal semantic mapping base set;

[0009] An improved heterogeneous graph attention network is introduced to construct a multimodal semantic fusion model;

[0010] A semantic role alignment graph is constructed based on a multimodal semantic fusion model, and a semantic role consistency connection relationship is established between image region nodes and text word nodes;

[0011] A heterogeneous semantic graph structure is constructed based on a multimodal semantic fusion model, and multi-type edge connection relationships are established;

[0012] In a heterogeneous semantic graph structure, the structure-guided attention propagation operation is performed based on edge type, effective paths are filtered using a masking mechanism, cross-modal attention weights between nodes are calculated, and a fused semantic representation vector is output.

[0013] The fused semantic representation vector is input into the question-answering intent recognition module to extract the semantic focus in the user's question, obtain the knowledge matching results, and generate a set of candidate answers.

[0014] Based on the knowledge matching results and the candidate answer set, the candidate answers are processed by language style generation or translation, combined with language tag node information, and the final answer result is output and the multimodal semantic fusion model is updated.

[0015] Optionally, the multimodal semantic mapping base set includes an image region feature set, a text semantic vector set, a word semantic role label set, an image semantic role label set, and language labels.

[0016] Optionally, the improved heterogeneous graph attention network includes a semantic role alignment graph construction module, a heterogeneous graph structure modeling module, a structure-guided attention propagation module, and a multi-layer heterogeneous attention aggregation module;

[0017] The semantic role alignment graph construction module constructs a semantic role alignment graph based on the word semantic role label set and the image semantic role label set in the multimodal semantic graph construction base set;

[0018] The heterogeneous graph structure modeling module completes the initialization operation of multiple types of nodes while constructing the heterogeneous semantic graph structure;

[0019] The structure-guided attention propagation module performs a structural masking operation on the text-image alignment edge based on the semantic role alignment graph, shielding connection paths that do not meet semantic consistency, and performs attention propagation operations under each edge type to extract semantic information transmission paths;

[0020] The multi-layer heterogeneous attention aggregation module aggregates the attention propagation results under various edge types, and fuses semantic features based on edge type weights or semantic gating mechanisms to output a fused semantic representation vector.

[0021] Optionally, constructing the semantic role alignment graph specifically includes:

[0022] The semantic alignment process is performed by the semantic role alignment graph construction module in the multimodal semantic fusion model, which extracts the word semantic role label set and the image semantic role label set from the multimodal semantic graph base set;

[0023] Perform a semantic role label comparison operation between each image region node and each text word node. When the semantic role label of the image region node matches the semantic role label of the text word node, establish a semantic alignment edge between the image region node and the text word node.

[0024] Combine all image region nodes, all text word nodes, and all semantic alignment edges to construct a semantic role alignment graph.

[0025] Optionally, the construction of the heterogeneous semantic graph structure specifically includes:

[0026] In the heterogeneous graph structure modeling module of the multimodal semantic fusion model, five types of heterogeneous nodes are initialized, including image region nodes, text word nodes, semantic role nodes, knowledge graph entity nodes, and language tag nodes.

[0027] Based on the multiple semantic relationships between five types of heterogeneous nodes, four types of edge connection relationships are established in the heterogeneous semantic graph structure;

[0028] The image region nodes, text word nodes, semantic role nodes, knowledge graph entity nodes, and language tag nodes are organized into a complete heterogeneous semantic graph structure along with their corresponding image-text alignment edges, semantic role edges, text dependency edges, and entity link edges.

[0029] Optionally, the four types of edge connections include image-text alignment edges, semantic role edges, text dependency edges, and entity link edges;

[0030] The image-text alignment edge connects image region nodes and text word nodes; the semantic role edge connects image region nodes or text word nodes and semantic role nodes; the text dependency edge connects the syntactic dependency relationships between text word nodes; and the entity link edge connects text word nodes or image region nodes and knowledge graph entity nodes.

[0031] Optionally, the calculation of cross-modal attention weights between nodes and the output of the fused semantic representation vector specifically includes:

[0032] The heterogeneous semantic graph structure, consisting of image region nodes, text word nodes, semantic role nodes, knowledge graph entity nodes, and language tag nodes, is input into the structure-guided attention propagation module.

[0033] Perform edge-type-specific attention propagation operations for each edge type, including graph-text alignment edges, semantic role edges, text dependency edges, and entity link edges.

[0034] In the structure-guided attention propagation module, a structure mask control mechanism is set for each type of edge, and the edge connection legality information provided by the semantic role alignment graph is used to shield connection paths that do not meet the semantic consistency condition.

[0035] On the edge connections of the preserved path, calculate the cross-modal attention weights between the connected nodes;

[0036] The multi-layer heterogeneous attention aggregation module performs heterogeneous information fusion based on the attention weights of multi-class edge connections, and aggregates the structural attention representations of all nodes in the graph to generate a fused semantic representation vector.

[0037] Optionally, the step of setting a structure mask control mechanism for each edge type to block connection paths that do not meet the semantic consistency condition specifically includes:

[0038] In the structure-guided attention propagation module, structure mask control parameters are configured for image-text alignment edges, semantic role edges, text dependency edges, and entity link edges to form a structure mask set.

[0039] Based on the semantic consistency information in the semantic role alignment graph, a connection mask for image-text alignment edges is constructed to shield the connection between image region nodes and text word nodes with inconsistent semantic roles, while retaining the image-text alignment edge paths with consistent semantic role labels.

[0040] For semantic role edges, a connection mask is constructed based on the semantic label matching relationship between image region nodes or text word nodes and semantic role nodes to shield connection edges with label conflicts or redundancy, while retaining valid connection paths with consistent semantic roles.

[0041] For text dependency edges, a structure mask is constructed based on the strength of syntactic dependency relations between text word nodes to shield dependency connections with semantic coupling strength below a threshold, while retaining dependency edge paths with high semantic coupling strength and representing core syntactic structures.

[0042] For entity link edges, a connection mask is constructed based on the link confidence between the node and the knowledge graph entity to block entity link edges with confidence below a preset threshold, while retaining entity association paths with high semantic fit.

[0043] Optionally, obtaining the knowledge matching results and generating a candidate answer set specifically includes:

[0044] The fused semantic representation vector is input into the question-answering intent recognition module to extract semantic focus information from the user's question and generate a semantic focus representation vector.

[0045] Based on the semantic focus representation vector, we call the multilingual knowledge subgraph that matches the language tag node in the knowledge base, limit the scope of candidate knowledge, and extract the subgraph structure containing entities related to semantic focus;

[0046] Matching calculations are performed on entity nodes in the multilingual knowledge subgraph. Based on image region features, text semantic information, and semantic role labels in the fused semantic representation vector, an entity candidate set is generated, and the semantic matching score of each entity is calculated.

[0047] Based on the entity candidate set, the association paths between candidate entities are mined, and a semantic path candidate set and confidence score are generated according to the path structure relevance and entity matching score.

[0048] A knowledge matching result is generated based on the entity candidate set and the semantic path candidate set. The knowledge matching result includes the structured knowledge content represented by high-confidence entities and associated semantic paths.

[0049] Based on the knowledge matching results, and combining the semantic focus representation vector and language tag node information, a set of candidate answers that are semantically relevant to the user's question and context-appropriate is generated.

[0050] Optionally, the step of outputting the final answer and updating the multimodal semantic fusion model specifically includes:

[0051] A set of candidate answers is generated based on the knowledge matching results and the current context information. Each candidate answer in the set is generated by combining the semantic path representation of the matching entity and the current semantic focus representation vector.

[0052] The candidate answer set is processed by combining language tag node information to generate language styles. The language style generation process includes natural language expression template selection, language style adjustment and multilingual translation module invocation.

[0053] The candidate answer set after language style processing is input into the answer scoring module. It is sorted based on semantic relevance score, context consistency score and historical feedback score, and the candidate answer with the highest confidence is output as the final answer result.

[0054] Collect user interaction feedback information and construct a user interaction feedback sample set. The feedback information includes answer click rate, user evaluation tags, multi-turn dialogue response data and user language preferences.

[0055] Based on the user interaction feedback sample set, the parameters of the semantic role alignment graph construction module and the structure-guided attention propagation module are jointly optimized and updated, the graph-text semantic alignment strategy and the structure propagation path filtering mechanism are dynamically adjusted, and the multimodal semantic fusion model is updated.

[0056] The beneficial effects of this invention are:

[0057] This invention proposes a multilingual knowledge question-answering method based on an image-text recognition fusion model, achieving deep fusion of image and text information in multilingual question-answering scenarios and effectively improving cross-modal semantic representation and structured knowledge retrieval capabilities. A semantic role alignment graph construction mechanism is adopted, ensuring that semantic connections between image region nodes and text word nodes are established on the basis of semantic role label consistency, thereby enhancing the structural consistency of the image-text semantic graph. An improved heterogeneous graph attention network is introduced, unifying the modeling of image regions, text words, semantic roles, knowledge graph entities, and language labels as multi-type nodes. Multi-dimensional connection paths are constructed through image-text alignment edges, semantic role edges, text dependency edges, and entity link edges, enhancing the breadth and fusion capability of semantic propagation between nodes.

[0058] During the structure-guided attention propagation process, a structure mask control mechanism is configured for different edge types to achieve attention propagation path filtering under semantic consistency constraints. This effectively suppresses the interference of redundant or semantically conflicting paths on the fused representation, improving the effectiveness of multimodal information fusion. After the fused semantic representation vector is input into the question-answering intent recognition module, combined with user language tag node information, it can accurately extract semantic focus in multilingual contexts, and call the matched multilingual knowledge subgraph for entity recognition and semantic path retrieval, improving the relevance and structural integrity of knowledge matching results. In the question-answering result generation stage, combined with natural language expression templates, language style adaptation, and translation processing mechanisms, it can output answer content that conforms to user language preferences, enhancing the system's multilingual adaptability.

[0059] Furthermore, this invention incorporates user interaction feedback to dynamically optimize semantic role alignment strategies and graph structure propagation parameters, achieving adaptive evolution of heterogeneous semantic graph structures and enabling structural updates and semantic adjustments. Overall, this method improves cross-modal semantic alignment accuracy, structured information fusion efficiency, and multilingual context adaptability in multilingual knowledge question answering tasks, making it suitable for intelligent question answering scenarios that require high accuracy and semantic consistency from mixed text and graph inputs. Attached Figure Description

[0060] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0061] Figure 1 This is a flowchart of a multilingual knowledge question answering method based on a text-image recognition fusion model proposed in this invention;

[0062] Figure 2 This is a schematic diagram of the structure of the improved heterogeneous graph attention network in the multilingual knowledge question answering method based on a graph-text recognition fusion model proposed in this invention;

[0063] Figure 3 This is a schematic diagram of the structure-guided attention propagation and semantic information fusion process in a multilingual knowledge question answering method based on a text-image recognition fusion model proposed in this invention. Detailed Implementation

[0064] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0065] refer to Figure 1-3 A multilingual knowledge question answering method based on a text-image recognition fusion model includes the following steps:

[0066] It receives and processes multilingual question-and-answer requests containing image and text information to generate a multimodal semantic mapping base set;

[0067] An improved heterogeneous graph attention network is introduced to construct a multimodal semantic fusion model;

[0068] A semantic role alignment graph is constructed based on a multimodal semantic fusion model, and a semantic role consistency connection relationship is established between image region nodes and text word nodes;

[0069] A heterogeneous semantic graph structure is constructed based on a multimodal semantic fusion model, and multi-type edge connection relationships are established;

[0070] In a heterogeneous semantic graph structure, the structure-guided attention propagation operation is performed based on edge type, effective paths are filtered using a masking mechanism, cross-modal attention weights between nodes are calculated, and a fused semantic representation vector is output.

[0071] The fused semantic representation vector is input into the question-answering intent recognition module to extract the semantic focus in the user's question, obtain the knowledge matching results, and generate a set of candidate answers.

[0072] Based on the knowledge matching results and the candidate answer set, the candidate answers are processed by language style generation or translation, combined with language tag node information, and the final answer result is output and the multimodal semantic fusion model is updated.

[0073] This invention introduces an improved heterogeneous graph attention network, combined with a semantic role alignment mechanism and a structure mask control strategy, to achieve deep semantic fusion of images and text in multilingual question-answering tasks. The system can dynamically construct a heterogeneous semantic graph structure based on the semantic consistency of images and text, and guide the attention propagation path based on edge type, effectively improving the accuracy of cross-modal information interaction and the quality of semantic representation. Through semantic focus extraction and multilingual knowledge graph matching mechanisms, the system can accurately locate user questions and generate structured candidate answers in multilingual contexts. Combined with language tag information, it completes language style generation and translation processing, improving the linguistic adaptability and contextual consistency of the answer content. The overall method possesses strong cross-language universality, image-text semantic alignment capability, and question-answering response accuracy, making it suitable for intelligent question-answering systems with mixed multilingual image-text input.

[0074] In this embodiment, the multimodal semantic mapping base set includes an image region feature set, a text semantic vector set, a word semantic role label set, an image semantic role label set, and language labels;

[0075] The language tag generation method is as follows: receive a multilingual question and answer request containing image information and text information, extract the original image data and original text data, obtain the language type used by the user in asking the question, and mark it as a language tag;

[0076] The image region feature set is generated as follows: target detection and region segmentation are performed on the original image data, a multi-scale region extraction algorithm is used to extract several regions containing the main semantic entities in the image, an image region set is constructed, and visual feature encoding is performed on each image region to generate an image region feature vector set.

[0077] The text semantic vector set is generated by performing language recognition, word segmentation, part-of-speech tagging and syntactic analysis on the original text data to generate a word sequence, and then encoding the word sequence based on a multilingual semantic encoder to generate a text semantic vector set.

[0078] The word semantic role label set is generated by performing semantic role recognition on the word sequence using a semantic role labeling model to obtain the word semantic role label set.

[0079] The method for generating the image semantic role label set is as follows: using a semantic role annotation model to perform semantic classification and entity recognition on the image region set to obtain the image semantic role label set.

[0080] This invention achieves unified semantic modeling of images and text at the semantic level by constructing a multimodal semantic graph foundation set, thereby improving the structural integrity and expressive accuracy of image-text information fusion. Through multi-scale region extraction and semantic role labeling methods, key semantic components in images and text are accurately extracted, and language type identification and adaptation are performed in conjunction with language tags, providing a high-quality semantic foundation for subsequent graph structure construction and cross-language question answering reasoning. This method effectively enhances the image-text understanding capability and semantic alignment effect of multilingual question answering systems.

[0081] In this embodiment, the improved heterogeneous graph attention network includes a semantic role alignment graph construction module, a heterogeneous graph structure modeling module, a structure-guided attention propagation module, and a multi-layer heterogeneous attention aggregation module;

[0082] The semantic role alignment graph construction module is based on the word semantic role label set and the image semantic role label set in the multimodal semantic graph base set. It identifies node pairs with consistent semantic roles and establishes semantic role consistency connection relationships between image region nodes and text word nodes to construct a semantic role alignment graph.

[0083] The heterogeneous graph structure modeling module completes the initialization operation of multiple types of nodes while constructing the heterogeneous semantic graph structure;

[0084] The structure-guided attention propagation module performs a structural masking operation on the text-image alignment edge based on the semantic role alignment graph, shielding connection paths that do not meet semantic consistency, and performs attention propagation operations under each edge type to extract semantic information transmission paths;

[0085] The multi-layer heterogeneous attention aggregation module aggregates the attention propagation results under various edge types, and fuses semantic features based on edge type weights or semantic gating mechanisms to output a fused semantic representation vector.

[0086] This invention employs an improved heterogeneous graph attention network to structurally model and guide the propagation of semantic relationships between image and text nodes, thereby enhancing the accuracy and path control capabilities of cross-modal semantic fusion. The semantic role alignment graph construction module achieves semantic consistency alignment between image and text nodes, the structure-guided attention propagation module utilizes a masking mechanism to filter effective paths and avoid interference from irrelevant connections, and the multi-layer heterogeneous attention aggregation module fuses multi-source semantic features based on edge types. The overall method enhances the structural expressiveness of semantic representation and the accuracy of question-answering reasoning.

[0087] In this embodiment, constructing the semantic role alignment graph specifically includes:

[0088] The semantic alignment process is performed by the semantic role alignment graph construction module in the multimodal semantic fusion model, which extracts the word semantic role label set and the image semantic role label set from the multimodal semantic graph base set;

[0089] For each image region node and each text word node, a semantic role label comparison operation is performed. When the semantic role label of the image region node is consistent with the semantic role label of the text word node, a semantic alignment edge is established between the image region node and the text word node to represent the consistency relationship between the image region and the text word in semantic function.

[0090] By combining all image region nodes, all text word nodes, and all semantic alignment edges, a semantic role alignment graph is constructed to represent the structural correspondence between images and text in the semantic role dimension.

[0091] This invention constructs a semantic role alignment graph by establishing alignment edges based on semantic role label consistency between image region nodes and text word nodes, thus achieving structural correspondence between image and text information in the semantic function dimension. This method can accurately identify node pairs with consistent semantic roles across modalities, effectively improving the semantic alignment accuracy between images and text. It provides a structural constraint basis for subsequent heterogeneous graph construction and attention propagation, enhancing the interpretability of multimodal semantic graph construction and the semantic understanding capability of question-answering systems.

[0092] In this embodiment, the construction of the heterogeneous semantic graph structure specifically includes:

[0093] In the heterogeneous graph structure modeling module of the multimodal semantic fusion model, five types of heterogeneous nodes are initialized, including image region nodes, text word nodes, semantic role nodes, knowledge graph entity nodes, and language tag nodes.

[0094] Based on the multiple semantic relationships between five types of heterogeneous nodes, four types of edge connection relationships are established in the heterogeneous semantic graph structure;

[0095] The image region nodes, text word nodes, semantic role nodes, knowledge graph entity nodes, and language tag nodes are organized into a complete heterogeneous semantic graph structure along with their corresponding image-text alignment edges, semantic role edges, text dependency edges, and entity link edges.

[0096] This invention unifies the initialization of image region nodes, text word nodes, semantic role nodes, knowledge graph entity nodes, and language tag nodes in a heterogeneous graph structure modeling module, and constructs four types of edge connections based on multiple semantic relationships to form a structurally complete heterogeneous semantic graph. This structure can comprehensively express the semantic dependencies and association paths between multimodal elements, providing a clear and controllable information path for subsequent attention propagation and semantic fusion, effectively improving the structural support capabilities for cross-modal information organization and knowledge retrieval.

[0097] In this embodiment, the four types of edge connections include image-text alignment edges, semantic role edges, text dependency edges, and entity link edges;

[0098] The image-text alignment edge connects image region nodes and text word nodes; the semantic role edge connects image region nodes or text word nodes and semantic role nodes; the text dependency edge connects the syntactic dependency relationships between text word nodes; and the entity link edge connects text word nodes or image region nodes and knowledge graph entity nodes.

[0099] This invention comprehensively characterizes the semantic relationship structure between images and text by constructing image-text alignment edges, semantic role edges, text dependency edges, and entity link edges. This edge connection enhances the information expression capability between nodes in heterogeneous graphs, providing stable structural support for cross-modal semantic fusion and accurate knowledge localization, and improving the semantic understanding accuracy of question-answering systems.

[0100] In this embodiment, the calculation of cross-modal attention weights between nodes and the output of the fused semantic representation vector specifically includes:

[0101] The heterogeneous semantic graph structure, consisting of image region nodes, text word nodes, semantic role nodes, knowledge graph entity nodes, and language tag nodes, is input into the structure-guided attention propagation module.

[0102] Perform edge-type-specific attention propagation operations for each edge type, including graph-text alignment edges, semantic role edges, text dependency edges, and entity link edges.

[0103] In the structure-guided attention propagation module, a structure mask control mechanism is set for each type of edge, and the edge connection legality information provided by the semantic role alignment graph is used to shield connection paths that do not meet the semantic consistency condition.

[0104] On the edge connections of the preserved path, cross-modal attention weights are calculated between the connected nodes. These attention weights are used to measure the semantic association strength between heterogeneous nodes such as image region nodes and text word nodes.

[0105] The multi-layer heterogeneous attention aggregation module performs heterogeneous information fusion based on the attention weights of multi-class edge connections, and aggregates the structural attention representations of all nodes in the graph to generate a fused semantic representation vector.

[0106] This invention introduces a structure-guided attention propagation mechanism into heterogeneous semantic graph structures, combined with a structure masking control strategy, to achieve attention computation and path selection under semantic consistency constraints across multiple edge types. The attention propagation results for each edge type are fused by a multi-layer aggregation module to generate a unified semantic representation vector, accurately measuring the semantic association strength between heterogeneous nodes. This mechanism effectively improves the expressive accuracy and structural control capabilities of cross-modal semantic fusion, providing a high-quality semantic foundation for downstream question-answering reasoning.

[0107] In this embodiment, the step of setting a structure mask control mechanism for each type of edge to block connection paths that do not meet the semantic consistency condition specifically includes:

[0108] In the structure-guided attention propagation module, structure mask control parameters are configured for image-text alignment edges, semantic role edges, text dependency edges, and entity link edges to form a structure mask set.

[0109] Based on the semantic consistency information in the semantic role alignment graph, a connection mask for image-text alignment edges is constructed to shield the connection between image region nodes and text word nodes with inconsistent semantic roles, while retaining the image-text alignment edge paths with consistent semantic role labels.

[0110] For semantic role edges, a connection mask is constructed based on the semantic label matching relationship between image region nodes or text word nodes and semantic role nodes to shield connection edges with label conflicts or redundancy, while retaining valid connection paths with consistent semantic roles.

[0111] For text dependency edges, a structure mask is constructed based on the strength of syntactic dependency relations between text word nodes to shield dependency connections with semantic coupling strength below a threshold, while retaining dependency edge paths with high semantic coupling strength and representing core syntactic structures.

[0112] For entity link edges, a connection mask is constructed based on the link confidence between the node and the knowledge graph entity to block entity link edges with confidence below a preset threshold, while retaining entity association paths with high semantic fit.

[0113] This invention implements a structural masking control mechanism for four edge types, enabling the filtering of connection legitimacy for graph-text aligned edges, semantic role edges, text dependency edges, and entity link edges. A masking strategy is constructed using semantic role consistency, syntactic dependency strength, and entity link confidence to accurately shield semantically inconsistent or weakly related paths while preserving highly relevant connection pathways. This mechanism enhances the effectiveness of semantic propagation paths and the accuracy of information fusion, improving the discriminative power and generalization performance of graph structures in multimodal semantic modeling.

[0114] In this embodiment, obtaining the knowledge matching results and generating a candidate answer set specifically includes:

[0115] The fused semantic representation vector is input into the question-answering intent recognition module to extract semantic focus information from the user's question and generate a semantic focus representation vector.

[0116] Based on the semantic focus representation vector, we call the multilingual knowledge subgraph that matches the language tag node in the knowledge base, limit the scope of candidate knowledge, and extract the subgraph structure containing entities related to semantic focus;

[0117] Matching calculations are performed on entity nodes in the multilingual knowledge subgraph. Based on image region features, text semantic information, and semantic role labels in the fused semantic representation vector, an entity candidate set is generated, and the semantic matching score of each entity is calculated.

[0118] Based on the entity candidate set, the association paths between candidate entities are mined, and a semantic path candidate set and confidence score are generated according to the path structure relevance and entity matching score.

[0119] A knowledge matching result is generated based on the entity candidate set and the semantic path candidate set. The knowledge matching result includes the structured knowledge content represented by high-confidence entities and associated semantic paths.

[0120] Based on the knowledge matching results, and combining the semantic focus representation vector and language tag node information, a set of candidate answers that are semantically relevant to the user's question and context-appropriate is generated.

[0121] This invention introduces a semantic focus recognition mechanism, combined with fused semantic representation vectors, to accurately extract core semantic targets from user queries. It also limits the scope of multilingual knowledge subgraphs based on language tags, achieving efficient cross-lingual knowledge retrieval. In the entity matching stage, image features, text semantics, and semantic role tags are fused to improve the accuracy and relevance of entity recognition. By mining the path structure between entities and calculating confidence, a semantic path candidate set is constructed, generating structured knowledge matching results. Finally, the semantic focus and language tag information are combined to generate a context-appropriate candidate answer set, effectively improving the semantic accuracy, linguistic consistency, and multilingual responsiveness of the question-and-answer content.

[0122] In this embodiment, the step of outputting the final answer and updating the multimodal semantic fusion model specifically includes:

[0123] A set of candidate answers is generated based on the knowledge matching results and the current context information. Each candidate answer in the set is generated by combining the semantic path representation of the matching entity and the current semantic focus representation vector.

[0124] The candidate answer set is processed by combining language tag node information to generate language styles. The language style generation process includes natural language expression template selection, language style adjustment and multilingual translation module invocation.

[0125] The candidate answer set after language style processing is input into the answer scoring module. It is sorted based on semantic relevance score, context consistency score and historical feedback score, and the candidate answer with the highest confidence is output as the final answer result.

[0126] Collect user interaction feedback information and construct a user interaction feedback sample set. The feedback information includes answer click rate, user evaluation tags, multi-turn dialogue response data and user language preferences.

[0127] Based on the user interaction feedback sample set, the parameters of the semantic role alignment graph construction module and the structure-guided attention propagation module are jointly optimized and updated, the graph-text semantic alignment strategy and the structure propagation path filtering mechanism are dynamically adjusted, and the multimodal semantic fusion model is updated.

[0128] This invention generates a candidate answer set by combining knowledge matching results with semantic focus information, and performs language style generation processing based on language tag nodes, achieving style adaptation and natural expression of answer content in a multilingual environment. By introducing an answer scoring mechanism, which ranks answers based on semantic relevance, contextual consistency, and user feedback, the accuracy and user acceptance of the final answers are improved. After outputting the results, the system further collects user interaction feedback data, jointly optimizes the semantic role alignment strategy and the structural propagation path filtering mechanism, and dynamically adjusts the structure and parameter configuration of the multimodal semantic fusion model. This method enhances the system's context adaptability and model self-learning ability, improving the intelligence and response quality of the question-answering service.

[0129] Example 1:

[0130] To verify the feasibility of this invention in practice, it was applied to a multilingual intelligent customer service system of a cross-border e-commerce platform. This platform provides daily product consultation, after-sales service, and logistics status inquiry support to users from multiple countries and regions, involving mixed text and image question-and-answer requests in multiple languages, including Chinese, English, French, German, and Arabic. Users often upload product screenshots, invoice images, and logistics screenshots, along with brief text descriptions, to ask system-related questions, such as "How do I return this product?", "Is the product in the picture still in stock?", and "What does this code on the invoice mean?". Existing systems primarily rely on text question-and-answer engines based on pre-trained multilingual models to process pure text content. Image information cannot be effectively used for reasoning, and the entity matching accuracy between different languages ​​is low. Especially in non-English language scenarios, there are problems such as incomplete contextual understanding, inconsistent answer styles, and ambiguous knowledge positioning, which seriously affect the quality of question-and-answer services.

[0131] To address the aforementioned issues, the multilingual knowledge-based question-answering method based on a text-image recognition fusion model proposed in this invention is fully integrated into this intelligent customer service system. In practical application, the system first extracts image region nodes from the user-uploaded image input target detection and region segmentation module. Simultaneously, it performs language recognition and grammatical analysis on the user's text request to generate text word nodes, and labels all nodes with semantic role tags using a semantic role labeling model. During the graph construction phase, the system automatically constructs a text-image semantic role alignment graph based on semantic consistency, forming a heterogeneous semantic graph structure with image region nodes, text word nodes, semantic role nodes, knowledge graph entity nodes, and language tag nodes, creating a multimodal graph structure with four types of edge connections. During attention propagation, the model masks invalid edge paths based on semantic role alignment edge mask information, retaining only semantically consistent semantic propagation channels. In the question-answering reasoning phase, the fused semantic representation vector is input into the question-answering intent recognition module to accurately locate the user's question focus, and cross-language entity matching and semantic path retrieval are completed based on the graph subgraph structure. The final answer is output by a language style generation module guided by language tags, consistent with the user's original language, and possesses characteristics of natural language expression, clear logic, and high semantic relevance.

[0132] To verify the effectiveness of the proposed method, an evaluation dataset was constructed using historical real-world question-and-answer data from the platform. This dataset included 12,000 text-and-image question-and-answer requests, covering 5 languages ​​and 8 product categories. The system was used to deploy a traditional baseline model and the proposed model for comparative experiments. Evaluation metrics included question-and-answer accuracy, entity matching accuracy, multilingual adaptation rate, average response time, and user click-through rate. Test results show that the proposed method outperforms the traditional method in all metrics, particularly in terms of text-and-image fusion question-and-answer accuracy and entity matching capabilities in low-resource language scenarios.

[0133] The table below shows some of the comparative experimental data. The table is titled "Performance Evaluation Results of Text-Image Fusion Multilingual Question Answering":

[0134] Table 1 Performance Evaluation Results of Multilingual Question Answering with Image and Text Fusion

[0135] Language type Question and Answer Type Sample size Accuracy improvement (%) Entity matching accuracy (%) Average response time (ms) User click-through rate increased (%) Chinese After-sales return category 2200 11.3 91.2 420 18.7 English Product Inquiry 2500 8.5 89.7 410 15.4 French Payment and logistics 1800 12.6 88.1 430 20.2 German Order Modification Class 1500 10.1 87.6 435 17.9 Arabic Invoice recognition 1300 14.8 90.5 450 22.3 Chinese Complex Q&A with Mixed Images and Text 1700 15.2 93.3 460 24.1 English Complex Q&A with Mixed Images and Text 1000 13.7 92.4 445 19.5

[0136] As can be seen from the table above, this invention significantly improves upon traditional methods in several key performance indicators of image-text fusion multilingual question answering tasks. Firstly, regarding question answering accuracy, the average accuracy improvement of this invention in five languages—Chinese, English, French, German, and Arabic—is 11.3%, 8.5%, 12.6%, 10.1%, and 14.8%, respectively. The Arabic scenario shows the largest improvement, demonstrating the superior semantic understanding and question answer generation capabilities of this method in low-resource language environments. In complex image-text mixed question answering scenarios, the accuracy improvement of this invention reaches 15.2% and 13.7% in Chinese and English, respectively, further validating the effectiveness of the structure-guided attention mechanism in processing image and text information fusion.

[0137] In terms of entity matching accuracy, this invention maintains an accuracy rate above 87% across all languages ​​and question-answering types, reaching a maximum of 93.3%, significantly higher than the average level of traditional baseline methods. This indicates that the proposed heterogeneous semantic graph structure and semantic role alignment mechanism can more accurately locate key semantic entities in user questions, improving the efficiency of knowledge graph retrieval and relevance matching capabilities. Regarding response speed, although the fusion structure introduces a graph neural network propagation process, the average response time is still controlled between 420ms and 460ms, maintaining good system real-time performance and meeting the application requirements of large-scale online question-answering systems.

[0138] Furthermore, the increased click-through rate also demonstrates the advantages of this invention in terms of user experience. Click-through rates increased by over 15% across all languages, with Arabic and Chinese mixed text and image question-and-answer scenarios reaching 22.3% and 24.1% respectively, indicating that the answer content is more closely aligned with user needs in terms of language style generation and semantic relevance. Overall, this invention achieves comprehensive optimization in accuracy, matching precision, response speed, and user interaction feedback, demonstrating promising prospects for practical application and promotion.

[0139] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multilingual knowledge question answering method based on a text-image recognition fusion model, characterized in that, Includes the following steps: It receives and processes multilingual question-and-answer requests containing image and text information to generate a multimodal semantic mapping base set; An improved heterogeneous graph attention network is introduced to construct a multimodal semantic fusion model; A semantic role alignment graph is constructed based on a multimodal semantic fusion model, and a semantic role consistency connection relationship is established between image region nodes and text word nodes; A heterogeneous semantic graph structure is constructed based on a multimodal semantic fusion model, and multi-type edge connection relationships are established; In a heterogeneous semantic graph structure, the structure-guided attention propagation operation is performed based on edge type, effective paths are filtered using a masking mechanism, cross-modal attention weights between nodes are calculated, and a fused semantic representation vector is output. The fused semantic representation vector is input into the question-answering intent recognition module to extract the semantic focus in the user's question, obtain the knowledge matching results, and generate a set of candidate answers. Based on the knowledge matching results and the candidate answer set, the candidate answers are processed by language style generation or translation, combined with language tag node information, and the final answer result is output and the multimodal semantic fusion model is updated.

2. The multilingual knowledge question answering method based on a text-image recognition fusion model according to claim 1, characterized in that, The multimodal semantic mapping base set includes an image region feature set, a text semantic vector set, a word semantic role label set, an image semantic role label set, and language labels.

3. The multilingual knowledge question answering method based on a text-image recognition fusion model according to claim 1, characterized in that, The improved heterogeneous graph attention network includes a semantic role alignment graph construction module, a heterogeneous graph structure modeling module, a structure-guided attention propagation module, and a multi-layer heterogeneous attention aggregation module; The semantic role alignment graph construction module constructs a semantic role alignment graph based on the word semantic role label set and the image semantic role label set in the multimodal semantic graph construction base set; The heterogeneous graph structure modeling module completes the initialization operation of multiple types of nodes while constructing the heterogeneous semantic graph structure; The structure-guided attention propagation module performs a structural masking operation on the text-image alignment edge based on the semantic role alignment graph, shielding connection paths that do not meet semantic consistency, and performs attention propagation operations under each edge type to extract semantic information transmission paths; The multi-layer heterogeneous attention aggregation module aggregates the attention propagation results under various edge types, and fuses semantic features based on edge type weights or semantic gating mechanisms to output a fused semantic representation vector.

4. The multilingual knowledge question answering method based on a text-image recognition fusion model according to claim 1, characterized in that, The construction of the semantic role alignment graph specifically includes: The semantic alignment process is performed by the semantic role alignment graph construction module in the multimodal semantic fusion model, which extracts the word semantic role label set and the image semantic role label set from the multimodal semantic graph base set; Perform a semantic role label comparison operation between each image region node and each text word node. When the semantic role label of the image region node matches the semantic role label of the text word node, establish a semantic alignment edge between the image region node and the text word node. Combine all image region nodes, all text word nodes, and all semantic alignment edges to construct a semantic role alignment graph.

5. The multilingual knowledge question answering method based on a text-image recognition fusion model according to claim 1, characterized in that, The construction of the heterogeneous semantic graph structure specifically includes: In the heterogeneous graph structure modeling module of the multimodal semantic fusion model, five types of heterogeneous nodes are initialized, including image region nodes, text word nodes, semantic role nodes, knowledge graph entity nodes, and language tag nodes. Based on the multiple semantic relationships between five types of heterogeneous nodes, four types of edge connection relationships are established in the heterogeneous semantic graph structure; The image region nodes, text word nodes, semantic role nodes, knowledge graph entity nodes, and language tag nodes are organized into a complete heterogeneous semantic graph structure along with their corresponding image-text alignment edges, semantic role edges, text dependency edges, and entity link edges.

6. The multilingual knowledge question answering method based on a text-image recognition fusion model according to claim 5, characterized in that, The four types of edge connections include image-text alignment edges, semantic role edges, text dependency edges, and entity link edges; The image-text alignment edge connects image region nodes and text word nodes; the semantic role edge connects image region nodes or text word nodes and semantic role nodes; the text dependency edge connects the syntactic dependency relationships between text word nodes; and the entity link edge connects text word nodes or image region nodes and knowledge graph entity nodes.

7. The multilingual knowledge question answering method based on a text-image recognition fusion model according to claim 1, characterized in that, The cross-modal attention weights between computing nodes, and the output fused semantic representation vector specifically include: The heterogeneous semantic graph structure, consisting of image region nodes, text word nodes, semantic role nodes, knowledge graph entity nodes, and language tag nodes, is input into the structure-guided attention propagation module. Perform edge-type-specific attention propagation operations for each edge type, including graph-text alignment edges, semantic role edges, text dependency edges, and entity link edges. In the structure-guided attention propagation module, a structure mask control mechanism is set for each type of edge, and the edge connection legality information provided by the semantic role alignment graph is used to shield connection paths that do not meet the semantic consistency condition. On the edge connections of the preserved path, calculate the cross-modal attention weights between the connected nodes; The multi-layer heterogeneous attention aggregation module performs heterogeneous information fusion based on the attention weights of multi-class edge connections, and aggregates the structural attention representations of all nodes in the graph to generate a fused semantic representation vector.

8. A multilingual knowledge question answering method based on a text-image recognition fusion model according to claim 7, characterized in that, The aforementioned structural mask control mechanism for each edge type, which masks connection paths that do not meet semantic consistency conditions, specifically includes: In the structure-guided attention propagation module, structure mask control parameters are configured for image-text alignment edges, semantic role edges, text dependency edges, and entity link edges to form a structure mask set. Based on the semantic consistency information in the semantic role alignment graph, a connection mask for image-text alignment edges is constructed to shield the connection between image region nodes and text word nodes with inconsistent semantic roles, while retaining the image-text alignment edge paths with consistent semantic role labels. For semantic role edges, a connection mask is constructed based on the semantic label matching relationship between image region nodes or text word nodes and semantic role nodes to shield connection edges with label conflicts or redundancy, while retaining valid connection paths with consistent semantic roles. For text dependency edges, a structure mask is constructed based on the strength of syntactic dependency relations between text word nodes to shield dependency connections with semantic coupling strength below a threshold, while retaining dependency edge paths with high semantic coupling strength and representing core syntactic structures. For entity link edges, a connection mask is constructed based on the link confidence between the node and the knowledge graph entity to block entity link edges with confidence below a preset threshold, while retaining entity association paths with high semantic fit.

9. A multilingual knowledge question answering method based on a text-image recognition fusion model according to claim 1, characterized in that, The process of obtaining knowledge matching results and generating a candidate answer set specifically includes: The fused semantic representation vector is input into the question-answering intent recognition module to extract semantic focus information from the user's question and generate a semantic focus representation vector. Based on the semantic focus representation vector, we call the multilingual knowledge subgraph that matches the language tag node in the knowledge base, limit the scope of candidate knowledge, and extract the subgraph structure containing entities related to semantic focus; Matching calculations are performed on entity nodes in the multilingual knowledge subgraph. Based on image region features, text semantic information, and semantic role labels in the fused semantic representation vector, an entity candidate set is generated, and the semantic matching score of each entity is calculated. Based on the entity candidate set, the association paths between candidate entities are mined, and a semantic path candidate set and confidence score are generated according to the path structure relevance and entity matching score. A knowledge matching result is generated based on the entity candidate set and the semantic path candidate set. The knowledge matching result includes the structured knowledge content represented by high-confidence entities and associated semantic paths. Based on the knowledge matching results, and combining the semantic focus representation vector and language tag node information, a set of candidate answers that are semantically relevant to the user's question and context-appropriate is generated.

10. A multilingual knowledge question answering method based on a text-image recognition fusion model according to claim 1, characterized in that, The process of outputting the final answer and updating the multimodal semantic fusion model specifically includes: A set of candidate answers is generated based on the knowledge matching results and the current context information. Each candidate answer in the set is generated by combining the semantic path representation of the matching entity and the current semantic focus representation vector. The candidate answer set is processed by combining language tag node information to generate language styles. The language style generation process includes natural language expression template selection, language style adjustment and multilingual translation module invocation. The candidate answer set after language style processing is input into the answer scoring module. It is sorted based on semantic relevance score, context consistency score and historical feedback score, and the candidate answer with the highest confidence is output as the final answer result. Collect user interaction feedback information and construct a user interaction feedback sample set. The feedback information includes answer click rate, user evaluation tags, multi-turn dialogue response data and user language preferences. Based on the user interaction feedback sample set, the parameters of the semantic role alignment graph construction module and the structure-guided attention propagation module are jointly optimized and updated, the graph-text semantic alignment strategy and the structure propagation path filtering mechanism are dynamically adjusted, and the multimodal semantic fusion model is updated.

Images