Multimodal collaborative memory large model inference system fusing cross-modal graph neural networks

Abstract

The application provides a multi-modal collaborative memory large model inference system fusing a cross-modal graph neural network, structured representation and relationship modeling of multi-modal data are realized by introducing a graph neural network, and multi-modal memory enhancement and inference scheduling mechanisms are combined to realize collaborative fusion and high-precision inference of multi-source heterogeneous information, the system faces multi-modal input scenes, and unified modeling and joint inference are realized on text, images and structured data, the overall architecture adopts hierarchical modular design, and includes a data access module, a cross-modal semantic alignment module, a graph structure construction module, a graph inference module, a multi-modal memory management module, an inference scheduling control module and a result generation module. The multi-modal collaborative memory large model inference system fusing the cross-modal graph neural network realizes efficient fusion and high-precision inference of multi-source heterogeneous information.

Description

Technical Field

[0001] This invention relates to a large model inference system, and more particularly to a multimodal collaborative memory large model inference system that integrates cross-modal graphical neural networks. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, intelligent systems centered on large language models have made significant progress in fields such as natural language processing, knowledge-based question answering, and intelligent decision-making. At the same time, the information processing capabilities of single modalities are gradually becoming insufficient to meet actual business needs, and multimodal learning technology has gradually become a research hotspot. Multimodal models, by integrating multiple data formats such as text, images, and speech, achieve a comprehensive understanding of complex information and are widely used in scenarios such as intelligent customer service, content generation, and intelligent retrieval.

[0003] In multimodal modeling, achieving effective alignment and fusion between different modalities is a key issue. Current mainstream methods typically fuse features from different modalities based on shared representation spaces or attention mechanisms, such as achieving semantic alignment through cross-modal encoders. However, these methods often rely on implicit modeling, lack explicit expression of the structural relationships between multimodal data, and struggle to capture complex semantic associations.

[0004] To enhance the relational modeling capabilities of models, researchers have introduced graph neural networks (GNNs) for structured representation learning. GNNs can model entities and their relationships as graph structures, effectively characterizing the connections between data through node feature propagation and neighborhood aggregation mechanisms. GNNs have demonstrated good performance in knowledge graphs, recommender systems, and complex relational reasoning. However, in multimodal scenarios, how to uniformly map information from different modalities into graph structures and perform joint modeling remains an exploratory stage.

[0005] On the other hand, to improve the reasoning ability of large models in complex tasks, memory enhancement mechanisms have been gradually introduced into model design, such as Retrieval-Augmented Generation (RAG) methods based on vector databases. These methods supplement the model's contextual information through external knowledge retrieval, thereby improving the accuracy and timeliness of responses. However, existing memory mechanisms mostly employ a flat storage approach, lacking modeling of the relationships between multimodal information, resulting in limited memory utilization efficiency and difficulty in supporting complex reasoning tasks.

[0006] Furthermore, in practical applications, the processing of multimodal information often involves multiple sub-modules (such as encoding, retrieval, and reasoning). Existing systems typically use a loosely coupled approach to combine these modules, resulting in a lack of a unified coordination mechanism between them. This leads to semantic loss and inconsistencies in reasoning during the information flow process.

[0007] To address the problem of multimodal information understanding and reasoning, various improvement schemes have been proposed in existing technologies, which can be mainly summarized into the following categories:

[0008] (1) Fusion schemes based on cross-modal pre-trained models; represented by CLIP models and large multimodal models (such as GPT-4V models), these methods construct a unified multimodal coding space, mapping different modalities such as text and images to the same semantic vector space, and achieving cross-modal alignment through contrastive learning or attention mechanisms. During inference, the model can perform content understanding and generation based on joint representations. However, these methods mainly rely on implicit semantic alignment mechanisms and lack the ability to explicitly model entity relationships and structural information, which limits their effectiveness when dealing with complex relationship reasoning.

[0009] (2) Retrieval-enhanced multimodal reasoning schemes; represented by retrieval-enhanced generation, this type of method improves the accuracy and knowledge coverage of the model's answers by constructing an external knowledge base (such as a vector database) and dynamically retrieving relevant information and injecting it into the context during the reasoning process. Some extended schemes have already supported the retrieval and fusion of multimodal data. However, this type of method usually adopts a flat vector storage structure, lacks modeling of the relationships between multimodal data, and lacks structured organization among the retrieval results, making it difficult to support complex reasoning chains.

[0010] (3) Reasoning schemes based on graph structure modeling; In the field of structured reasoning, some methods introduce graph neural networks to model knowledge graphs or relational data, and realize relational reasoning through the representation learning of nodes and edges. These methods perform well in tasks such as entity relation inference and path reasoning. However, these methods are usually designed for a single modality (such as structured data or text extraction results), and are difficult to apply directly in multimodal scenarios, especially in the unified modeling of heterogeneous data such as images and text.

[0011] (4) Large-scale model reasoning scheme based on memory enhancement; Some existing systems introduce long-term memory modules (such as external storage or context caching mechanisms) to record historical interaction information or knowledge fragments and call them during reasoning to improve the context continuity and reasoning ability of the model. However, such methods mostly organize memory in a linear or key-value pair form, lacking structured association modeling, which makes it impossible to form effective connections between memories, thus limiting the efficiency of information utilization in complex reasoning tasks.

[0012] In summary, based on the existing multimodal reasoning schemes mentioned above, although some progress has been made in cross-modal fusion, retrieval enhancement, and structured modeling, the following shortcomings still exist in complex multimodal reasoning scenarios:

[0013] (1) The cross-modal fusion method is shallow and the semantic alignment is insufficient. Most existing methods are based on a unified vector space or attention mechanism to achieve cross-modal alignment (such as CLIP model-like methods). Essentially, they belong to implicit feature fusion and lack explicit modeling of cross-modal entities and their relationships. This leads to semantic offset or inconsistency in understanding when dealing with complex semantic associations or multi-hop reasoning.

[0014] (2) Lack of a unified structured relationship modeling mechanism; Although some methods introduce graph neural networks for relationship modeling, they are usually limited to a single modality (such as knowledge graphs for text extraction), making it difficult to express multimodal information such as images and text in a unified graph structure, resulting in the system being unable to fully utilize the complex relationships in multimodal data.

[0015] (3) The memory mechanism is flat and lacks the ability to organize associations. Existing methods based on retrieval enhancement usually adopt vectorized storage and similarity retrieval mechanisms. Their memory structures are mostly flat data sets, lacking modeling and organization of the relationships between memory units, resulting in a lack of logical association between retrieval results and difficulty in supporting complex reasoning links.

[0016] (4) The multimodal reasoning process lacks a collaborative mechanism. In the existing system, different modal information is usually processed by independent modules. The reasoning process often adopts a serial or weakly coupled fusion method, lacking a unified collaborative reasoning framework. This leads to the fragmentation of information during transmission, affecting the overall consistency and stability of reasoning.

[0017] (5) It is difficult to support long-link reasoning in complex scenarios. When it involves multi-round interaction or complex decision-making tasks, existing methods are difficult to integrate multimodal information, structured relationships and historical memory at the same time, resulting in a lack of continuity and global consistency in the reasoning process, which in turn affects the accuracy and reliability of the final result.

[0018] It is evident that while existing technologies have made some progress in multimodal modeling, graph structure learning, and memory enhancement, they still have significant shortcomings in cross-modal deep fusion, structured relationship modeling, and multimodal collaborative reasoning. There is an urgent need for an efficient reasoning system that can unify and integrate the above capabilities.

[0019] Compared with the prior art, the difference of this invention is not that it uses cross-modal coding alone, graph neural networks alone, or retrieval enhancement alone, but that it explicitly transforms the cross-modal alignment results into a weighted heterogeneous graph structure and uses historical memory as an associative and reasonable graph node or evidence subgraph to participate in reasoning scheduling, so that multimodal understanding, relational reasoning and memory retrieval are completed collaboratively under the same structured framework. Summary of the Invention

[0020] This invention provides a multimodal collaborative memory large-scale model inference system that integrates cross-modal graph neural networks, solving the problems existing in current multimodal large-scale models in terms of cross-modal semantic alignment, structural relationship modeling, multimodal memory utilization, and collaborative reasoning capabilities. The technical solution is as follows:

[0021] A multimodal collaborative memory large-scale model inference system integrating cross-modal graphical neural networks includes:

[0022] Data access module: Acquires input information from different data sources and performs preprocessing and standardized representation;

[0023] Cross-modal semantic alignment module: maps data from different modalities to a unified semantic space to achieve initial semantic consistency;

[0024] Graph structure construction module: Abstracts entities, attributes and their relationships in multimodal data into a graph structure, and constructs a cross-modal association graph, where nodes represent multimodal semantic units and edges represent the association relationships between different entities or different modalities;

[0025] Graph reasoning module: It uses graph neural networks to perform feature propagation and relation modeling on the graph structure, and achieves cross-modal semantic fusion and complex relation reasoning through multi-round neighborhood information aggregation, thereby obtaining an enhanced structured semantic representation;

[0026] Multimodal memory management module: Constructs multimodal collaborative memory units to structure and store historical inputs, multimodal information and intermediate reasoning results, and realizes dynamic retrieval and fusion of memory information through an association retrieval mechanism to support long context and multi-turn reasoning tasks;

[0027] The reasoning scheduling and control module: performs unified scheduling and fusion of graph reasoning results and memory information, and dynamically selects relevant information to participate in the reasoning process according to the current task requirements;

[0028] Results generation module: Inputs the fusion results into the generation module based on the large language model to generate the final reasoning results or decision output;

[0029] The functional units are connected through a unified data representation and interface. By introducing graph neural networks to perform structured representation and relation modeling of multimodal data, and combining multimodal memory enhancement and inference scheduling mechanisms, the collaborative fusion of multi-source heterogeneous information and high-precision inference are achieved.

[0030] Furthermore, the data access module includes a modality recognition unit, a data cleaning unit, a feature extraction unit, and a unified representation generation unit. The modality recognition unit identifies the source, format, and modality type of the input data. The data cleaning unit performs word segmentation, entity recognition, stop information filtering, and semantic vectorization on the text. The feature extraction unit performs OCR recognition, object detection, region feature extraction, and image-text correspondence extraction on the image, and performs field mapping, unit normalization, primary key alignment, and missing value processing on the structured data. The unified representation generation unit expresses the processing results as a data object containing modality identifiers, semantic vectors, entity sets, attribute sets, time information, source information, confidence levels, and permission constraints.

[0031] Furthermore, the cross-modal semantic alignment module uses a text encoder, an image encoder, and a structured data encoder to generate feature vectors for each modality, and maps the features of different modalities to the same semantic space through contrastive learning, attention mapping, or projection matrix. For the representation of the same entity in different modalities, a cross-modal matching score is calculated based on the entity name, location region, contextual semantics, field identifier, and temporal relationship. When the matching score is higher than a preset threshold, a semantic alignment relationship is established.

[0032] Furthermore, the graph structure construction module maps text entities, image targets, table fields, business objects, historical memory fragments, and intermediate inference results as nodes, and maps entity subordination relationships, spatial relationships, temporal relationships, field association relationships, semantic similarity relationships, causal relationships, and cross-modal reference relationships as edges, and sets the relationship type, direction, weight, and confidence for each edge; the cross-modal association graph can be represented as G=(V,E,A,X), where V is the set of nodes, E is the set of edges, A is the weighted adjacency matrix, and X is the node feature matrix.

[0033] Furthermore, the multimodal memory management module writes historical inputs, user feedback, cross-modal graph nodes, inference paths, intermediate conclusions, and final outputs into the memory bank. Each memory unit includes memoryId, content, modality, embedding, linkedEntity, linkedNode, timeStamp, confidence, usageCount, and expireCondition, and establishes memory edges through relationships such as the same object, the same topic, causal dependence, temporal continuity, or user session. During memory retrieval, the system first performs vector similarity retrieval based on the current task representation, and then performs relation expansion retrieval based on memory edges to obtain a set of memories that are semantically and structurally related to the current task. memoryId represents the memory ID; content represents the memory content; modality represents the modality, the representation of this memory; embedding represents the content vector; linkedEntity represents the associated entity, i.e., the key information points contained or pointed to by the memory; linkedNode represents the associated node; timeStamp represents the timestamp; confidence represents the confidence level; usageCount represents the number of times it is used; and expireCondition represents the expiration rule.

[0034] A method for inference in a large multimodal collaborative memory model that integrates cross-modal graphical neural networks includes the following steps:

[0035] Step S1: Acquire multimodal input data from different data sources and form a unified data input representation through preprocessing; Step S2: Achieve cross-modal semantic alignment by cross-modal encoding on the preprocessed multimodal data; Step S3: Extract entities, attributes, and their relationships from the multimodal data and construct a cross-modal association graph structure; Step S4: Perform relationship modeling and feature propagation on the graph structure based on graph neural networks; Step S5: Construct a multimodal collaborative memory bank and establish associations between memories;

[0036] Step S6: Through the reasoning scheduling mechanism, the graph reasoning results and memory retrieval results are uniformly scheduled and fused; Step S7: The fused multi-source information is input into the large language model, and the final output results, including text answers, decision results or structured information, are obtained through generative reasoning.

[0037] Furthermore, in step S1, the multimodal input data is format-standardized, noise-filtered, permission-verified, and preliminarily extracted. The text, images, and structured data are then converted into a unified input object, which includes at least the data source, modality type, original content, extracted text, entity set, attribute set, semantic vector, confidence level, timestamp, and access permissions.

[0038] Furthermore, in step S2, text vectors, image region vectors, and structured field vectors are obtained using different modal encoders, mapped to a unified semantic space through a projection network, and a cross-modal alignment matrix is ​​calculated based on cosine similarity, attention weights, and entity matching rules to obtain a unified feature representation of multiple modalities.

[0039] Furthermore, in step S3, a cross-modal association graph is constructed based on the entity extraction results, image target recognition results, and structured field relationships. The nodes include text entity nodes, image region nodes, field nodes, business object nodes, and memory nodes, and the edges include same-reference edges, subordinate edges, temporal relation edges, spatial relation edges, semantic similarity edges, and inference dependency edges. Weights are assigned to the edges based on matching confidence or business rules.

[0040] Furthermore, in step S4, a graph neural network is used to perform multi-round message passing on the cross-modal association graph. In the k-th round, any node updates its node representation by aggregating the features of its neighboring nodes, edge type embeddings, and edge weights, and controls the contribution of different modal neighbors to the current node through attention weights. After multiple rounds of iteration, a node representation, subgraph representation, or full graph representation that integrates structural relationships and cross-modal semantics is obtained for subsequent inference.

[0041] The multimodal collaborative memory large-scale model inference system integrating cross-modal graphical neural networks achieves efficient fusion and high-precision inference of multi-source heterogeneous information through structured modeling and collaborative inference methods. Specific objectives include:

[0042] (1) Enhance cross-modal semantic alignment capability; construct a unified cross-modal representation space, introduce a structured relationship modeling mechanism, enable deep alignment of text, images and other modal data at the semantic level, reduce information bias between modalities, and improve the accuracy of multimodal understanding.

[0043] (2) Enhance the ability to model complex relationships; by introducing graph structure representation, entities, attributes and their relationships in multimodal data are abstracted into graph structures, and graph neural networks are used for relationship propagation and feature aggregation, thereby improving the system's ability to model and reason about complex semantic relationships.

[0044] (3) Construct a multimodal collaborative memory mechanism; design a multimodal memory module that supports structured storage and associated retrieval, organize historical context and multi-source data in a graph, realize the association modeling and efficient retrieval between memories, and improve the consistency and stability of long context reasoning.

[0045] (4) Implement a multimodal collaborative reasoning framework; construct a unified reasoning scheduling mechanism so that multimodal information can dynamically interact and make collaborative decisions during the reasoning process, avoiding the information fragmentation problem caused by the independent processing of each modality in traditional methods.

[0046] (5) Improve the overall reasoning performance and generalization ability of the system; by integrating cross-modal modeling, graph structure reasoning and memory enhancement mechanisms, improve the reasoning accuracy and stability of the system in complex scenarios (such as multi-turn question answering, knowledge reasoning, decision analysis, etc.) and enhance its adaptability to different task scenarios.

[0047] The present invention has the following advantages:

[0048] First, by constructing a cross-modal graph structure, this invention achieves an explicit structured representation of multimodal information. Compared with existing methods that rely solely on implicit alignment using vector space, this invention can more accurately characterize the semantic relationships between different modalities, thereby significantly improving the accuracy and consistency of cross-modal understanding.

[0049] Secondly, this invention utilizes graph neural networks for relation propagation and multi-hop reasoning, enabling the relationships between multimodal data to be effectively propagated within the graph structure. Compared to the shallow fusion approach in traditional methods, this invention better supports complex semantic relation modeling and reasoning tasks, improving reasoning depth and logicality.

[0050] Furthermore, the multimodal collaborative memory mechanism proposed in this invention enables memory units to be connected and reasonable by structuring and modeling memory information. This overcomes the information fragmentation problem existing in the flat retrieval method based on RAG in the prior art, thereby significantly improving the efficiency of information utilization and the continuity of reasoning in long context scenarios.

[0051] Furthermore, this invention achieves dynamic collaborative fusion of graph reasoning results and memory information through a reasoning scheduling mechanism, avoiding the information inconsistency problem caused by the independent operation of each module in the prior art, enabling multimodal information to participate in the reasoning process under a unified framework, thereby improving the overall reasoning stability and consistency.

[0052] Furthermore, this invention constructs a closed-loop reasoning mechanism that supports multiple iterations. During the reasoning process, graph structure reconstruction or memory re-retrieval can be dynamically triggered according to task requirements. Compared with the traditional one-time reasoning process, it can effectively support multi-hop reasoning and complex task processing, and improve the accuracy and robustness of reasoning results.

[0053] Finally, by unifying and integrating cross-modal modeling, structured reasoning, and memory enhancement mechanisms, this invention forms a complete multimodal collaborative reasoning system. Compared with existing technical solutions that only optimize in a single dimension, it has stronger system integrity and generalization ability, and can be applied to a variety of complex application scenarios. Attached Figure Description

[0054] Figure 1This is a schematic diagram of the processing of the multimodal collaborative memory large model inference system that integrates cross-modal graphical neural networks. Detailed Implementation

[0055] like Figure 1 As shown, the multimodal collaborative memory large model inference system that integrates cross-modal graph neural networks introduces graph neural networks to perform structured representation and relational modeling of multimodal data, and combines multimodal memory enhancement and inference scheduling mechanisms to achieve collaborative fusion of multi-source heterogeneous information and high-precision inference.

[0056] The system is designed for multimodal input scenarios, performing unified modeling and joint reasoning on text, images, and structured data. Its overall architecture adopts a layered, modular design, including:

[0057] 1. Data Access Module: Acquires input information from data sources of different multimodal data, and performs preprocessing and standardized representation. Multimodal data includes text data, image data, and structured data. By standardizing the format, filtering noise, and preliminarily extracting features from the data, a unified data input representation is formed.

[0058] The data access module includes a modality recognition unit, a data cleaning unit, a feature extraction unit, and a unified representation generation unit. The modality recognition unit identifies the source, format, and modality type of the input data. The data cleaning unit performs word segmentation, entity recognition, stop information filtering, and semantic vectorization on the text. The feature extraction unit performs OCR recognition, object detection, region feature extraction, and image-text correspondence extraction on the image, and performs field mapping, unit normalization, primary key alignment, and missing value processing on the structured data. The unified representation generation unit expresses the processing results as a data object containing modality identifiers, semantic vectors, entity sets, attribute sets, time information, source information, confidence levels, and permission constraints.

[0059] Specifically, the data access module receives multimodal data from user-uploaded files, business system interfaces, image acquisition devices, databases, or historical sessions, and identifies the modality type, data source, business object, and permission scope of each input. For text data, it performs word segmentation, entity recognition, syntactic dependency analysis, and semantic vectorization; for image data, it performs OCR, object detection, region segmentation, image description generation, and image-text location association; for structured data, it performs field mapping, primary key alignment, unit conversion, outlier filtering, and missing value imputation. The standardized input can be represented as U={uid,modality, source, rawContent, extractedText, featureVector, entitySet, attributeSet, timestamp, confidence, permissionScope}, providing a unified data interface for data from different sources and modalities during subsequent alignment, mapping, and inference processes.

[0060] Wherein, uid represents the user ID, a unique identifier for the user who generated or associated this data; modality represents the modality, i.e., the original form of the content; source represents the origin; rawContent represents the original content, the raw data without any processing; extractedText represents the extracted text, the plain text parsed or converted from the original content; featureVector represents the feature vector; entitySet represents the entity set, i.e., the key information points automatically identified from the content; attributeSet represents the attribute set, the additional descriptive tags for the content or entities; timestamp represents the timestamp; confidence represents the confidence level; and permissionScope represents the permission scope.

[0061] 2. Cross-modal semantic alignment module: Maps data from different modalities to a unified semantic space to achieve initial semantic consistency.

[0062] Specifically, the cross-modal semantic alignment module generates feature vectors for each modality through text encoder, visual encoder, and structured data encoder, respectively, and maps the features of different modalities to the same semantic space through contrastive learning, attention mapping, or projection matrix. For entities in text, target regions in images, and field values ​​in tables, the system calculates multiple matching factors such as name matching, contextual similarity, spatial position relationship, field semantics, and temporal consistency to obtain a cross-modal alignment score.

[0063] For the representation of the same entity in different modalities, a cross-modal matching score is calculated based on the entity name, location region, contextual semantics, field identifier, and temporal relationship. When the matching score is higher than a preset threshold, a semantic alignment relationship is established. When the alignment score is higher than the preset threshold, semantic units in different modalities are marked as the same business object or related business objects. When multiple candidate objects have similar scores, they are retained as nodes to be confirmed, and further disambiguation is performed during the graph reasoning stage in conjunction with contextual relationships. This solves the semantic offset problem caused by relying solely on vector nearest neighbors.

[0064] 3. Graph Structure Construction Module: Abstracts entities, attributes and their relationships in multimodal data into a graph structure to construct a cross-modal association graph, where nodes represent multimodal semantic units and edges represent the association relationships between different entities or different modalities.

[0065] Specifically, the graph structure construction module maps text entities, image regions, table fields, business objects, historical memories, and intermediate inference conclusions to nodes of different types; and maps entity subordination relationships, spatial relationships, temporal relationships, field association relationships, causal relationships, semantic similarity relationships, and cross-modal reference relationships to edges of different types. Each edge records the relationship type, direction, weight, confidence level, and source evidence. The cross-modal association graph can be represented as G=(V,E,A,X), where V represents the set of nodes, E represents the set of edges, A represents the weighted adjacency matrix, and X represents the node feature matrix. For the "invoice amount" region in the image, the "amount payable" description in the text, and the "amount field" in the structured table, the system can establish nodes respectively and connect them through referential edges or field mapping edges, thereby forming a computable and traceable structured expression.

[0066] 4. Graph Reasoning Module: Utilizes graph neural networks to perform feature propagation and relation modeling on the graph structure, and achieves cross-modal semantic fusion and complex relation reasoning through multi-round neighborhood information aggregation, thereby obtaining an enhanced structured semantic representation.

[0067] Specifically, the graph reasoning module employs graph attention networks, relational graph convolutional networks, or heterogeneous graph neural networks to perform multi-round message passing on the cross-modal association graph. In the k-th round of propagation, the target node receives the features, edge type embeddings, and edge weights of its neighboring nodes, calculates the contributions of different neighboring nodes to the target node based on the attention weights, and then updates the target node's representation. In multi-hop reasoning scenarios, the system uses multi-round neighborhood aggregation to gradually propagate information from distantly related nodes to the target node. For example, when a user question is associated with a text entity, the text entity is associated with an image region, and the image region is then associated with a structured field, the graph neural network can propagate evidence along the cross-modal path, obtaining a structured semantic representation that integrates multi-source evidence.

[0068] 5. Multimodal Memory Management Module: Constructs a multimodal collaborative memory unit to structure and store historical inputs, multimodal information, and intermediate reasoning results. It also enables dynamic retrieval and fusion of memory information through an association retrieval mechanism to support long context and multi-turn reasoning tasks.

[0069] Specifically, the multimodal memory management module writes historical inputs, user feedback, cross-modal graph nodes, reasoning paths, intermediate conclusions, and final outputs into the memory bank. Each memory unit includes memoryId, content, modality, embedding, linkedEntity, linkedNode, timeStamp, confidence, usageCount, and expireCondition, and establishes memory edges based on the same object, the same topic, causal dependence, temporal continuity, or user session relationships. During memory retrieval, the system first performs vector similarity retrieval based on the current task representation, and then performs relation expansion retrieval based on memory edges to obtain a set of memories that are semantically and structurally related to the current task. This set of memories is not a simple text concatenation, but rather participates in subsequent graph reasoning and generation as candidate nodes or evidence subgraphs.

[0070] Among them, memoryId represents the memory ID; content represents the memory content; modality represents the modality, the representation of this memory; embedding represents the content vector; linkedEntity represents the associated entity, that is, the key information points contained or pointed to by the memory; linkedNode represents the associated node; timeStamp represents the timestamp; confidence represents the confidence level; usageCount represents the number of times it is used; and expireCondition represents the expiration rule.

[0071] 6. Reasoning Scheduling and Control Module: This module performs unified scheduling and fusion of graph reasoning results and memory information, and dynamically selects relevant information to participate in the reasoning process based on the current task requirements.

[0072] Specifically, the inference scheduling and control module determines the inference path based on task type, problem complexity, graph inference confidence, memory retrieval score, and consistency of generated results. For fact-based question-and-answer tasks, high-confidence graph nodes and directly associated memories are prioritized; for complex decision-making tasks, multi-hop graph inference and relation expansion retrieval are added; for long-context tasks, memory units under the same session, the same business object, or the same inference chain are prioritized. When there is a conflict between the graph inference result and the memory retrieval result, the scheduling and control module calculates the evidence weight based on source credibility, time freshness, business rule priority, and user confirmation information; when the fusion confidence is lower than the threshold, graph structure reconstruction, memory re-retrieval, or supplementary question generation are triggered, thereby forming a cyclically optimizeable inference process.

[0073] 7. Result Generation Module: Input the fusion result into the generation module based on the large language model to generate the final reasoning result or decision output.

[0074] Specifically, the result generation module organizes the key nodes, reasoning paths, evidence weights, relevant memory summaries, and task constraints obtained from graph reasoning into structured prompts, which are then input into the large language model to generate textual answers, decision suggestions, or structured information. After generation, the system performs factual consistency checks, cited evidence checks, format checks, and risk rule checks on the output, and writes the results that pass the checks, along with the reasoning paths, back to the memory bank.

[0075] The functional units are connected through a unified data representation and interface to form a complete multimodal collaborative reasoning closed loop.

[0076] Through the above technical solutions, the present invention realizes the organic integration of structured expression of multimodal information, relation modeling and memory-enhanced reasoning, and overcomes the problems of shallow cross-modal fusion, insufficient relation modeling ability and low memory utilization efficiency in the prior art, thereby improving the reasoning accuracy and stability of the system in complex multimodal scenarios.

[0077] Based on the system, the corresponding large-scale model inference method that integrates cross-modal graph structure modeling and multimodal collaborative memory mechanism includes the following steps in its implementation process:

[0078] Step S1: Acquire multimodal input data from different data sources and form a unified data input representation through preprocessing;

[0079] Multimodal input data from different data sources is acquired, including text data, image data, and structured data. The multimodal input data is then standardized in format, filtered for noise, validated for permissions, and preliminarily extracted for features. The text, images, and structured data are then converted into a unified input object, which includes at least the data source, modality type, original content, extracted text, entity set, attribute set, semantic vector, confidence level, timestamp, and access permissions.

[0080] Step S1 specifically includes the following steps:

[0081] S11, identify the input source and modality type;

[0082] S12 performs cleaning, extraction, and encoding on text, images, and structured data, respectively;

[0083] S13 converts the processing results of different modalities into a unified input object;

[0084] S14, mark the input object with permissions, confidence level and source.

[0085] This process ensures that data from different modalities participate in subsequent calculations using a unified field, avoiding information fragmentation caused by different formats used by different modules.

[0086] Step S2: Cross-modal semantic alignment is achieved by cross-modal encoding of the preprocessed multimodal data. The preprocessed multimodal data is encoded by using different modal encoders to obtain text vectors, image region vectors and structured field vectors. These are then mapped to a unified semantic space through a projection network. A cross-modal alignment matrix is ​​calculated based on cosine similarity, attention weights and entity matching rules to obtain a unified feature representation of the multimodal data, thus achieving cross-modal semantic alignment and obtaining a unified feature representation of the multimodal data.

[0087] Step S2 specifically includes the following steps:

[0088] S21, use a text encoder to extract sentence vectors, entity vectors and context vectors;

[0089] S22, using a visual encoder to extract global vectors, region vectors, and OCR text vectors of the image;

[0090] S23, use a structured encoder to extract field names, field values, and table relationship vectors;

[0091] S24, the modal vectors are mapped to a unified semantic space through a projection network;

[0092] S25, calculate the cross-modal alignment matrix and determine the same object or related objects based on similarity thresholds and business rules.

[0093] Step S3: Extract entities, attributes and their relationships from the multimodal data, and construct a cross-modal association graph structure; where multimodal semantic units are used as nodes and entity relationships or intermodal associations are used as edges to form a structured graph representation.

[0094] A cross-modal association graph is constructed based on entity extraction results, image target recognition results, and structured field relationships. Nodes include text entity nodes, image region nodes, field nodes, business object nodes, and memory nodes. Edges include edges representing same-reference relationships, subordinate relationships, temporal relationships, spatial relationships, semantic similarity, and inference dependencies. Weights are assigned to edges based on matching confidence or business rules. This invention maps different modal data into a unified graph structure, explicitly expressing multimodal semantic units and their relationships through nodes and edges, thereby achieving a structured representation of multimodal information.

[0095] Step S3 specifically includes the following steps:

[0096] S31, Extract entities, attributes, events, and constraints from text;

[0097] S32, extract target regions, OCR text, spatial relationships, and visual attributes from images;

[0098] S33 extracts fields, primary keys, foreign keys, and business relationships from structured data;

[0099] S34, map the above semantic units to nodes, and map the same, dependent, temporal, spatial, similar, causal and dependency relationships to edges;

[0100] S35 calculates edge weights based on matching confidence, business rule strength, and source credibility.

[0101] Step S4: Based on the graph neural network, perform relation modeling and feature propagation on the graph structure; use the graph neural network to perform multiple rounds of feature propagation and neighborhood information aggregation on the graph structure, learn the relational representation between nodes, perform feature propagation and neighborhood aggregation on the cross-modal graph structure, realize cross-modal information fusion and complex relational reasoning, and obtain an enhanced structured semantic representation.

[0102] A graph neural network is used to perform multi-round message passing on the cross-modal association graph. In the k-th round, any node updates its representation by aggregating the features of its neighboring nodes, edge type embeddings, and edge weights. Attention weights are used to control the contribution of different modal neighbors to the current node. After multiple rounds of iteration, node representations, subgraph representations, or full graph representations that fuse structural relationships and cross-modal semantics are obtained for subsequent inference.

[0103] Specifically, the computational process of the graph neural network includes: initializing the node feature matrix X and the adjacency matrix A; in each round of propagation, node v receives the features h_u, edge type r_uv, and edge weight w_uv of its neighbor node u, calculates the message weights through an attention function, and then performs weighted aggregation on the neighborhood messages to obtain the updated node representation h_v; after multiple rounds of propagation, node representations, path representations, and full graph representations containing cross-modal structural relationships are obtained. For different tasks, the system can select the target node representation, the relevant subgraph representation, or the full graph pooling representation to participate in subsequent inference.

[0104] Step S5: Construct a multimodal collaborative memory bank and establish relationships between memories;

[0105] The memory can be represented as a combination of vector indexes, graph databases, and structured metadata tables. Vector indexes are used for similarity retrieval, graph databases are used to store relationships between memory units, and metadata tables are used to store time, source, permissions, credibility, and expiration conditions. When writing to the memory, the system simultaneously saves the memory content, semantic vectors, associated entities, associated graph nodes, and reasoning paths, enabling the memory to be both retrieved and used as graph nodes in relational reasoning.

[0106] The multimodal collaborative memory bank is organized according to memory units. Each memory unit includes a memory identifier, source modality, content summary, semantic vector, associated entity, graph node reference, time information, number of uses, credibility, and expiration condition. During the reasoning process, similarity retrieval is first performed based on the current task vector, and then extended retrieval is performed based on the relation edges in the cross-modal association graph, thereby obtaining a set of memories that are related to the current task and have structural association.

[0107] A multimodal collaborative memory bank is constructed to structure and store historical input data, graph structure information, and intermediate reasoning results. During the reasoning process, relevant memory units are retrieved from the memory bank according to the current task context, and the association between memories is established, so that the memory not only supports similarity retrieval, but also relationship-based association retrieval, thereby improving the ability of complex reasoning.

[0108] Step S6: Through the reasoning scheduling mechanism, the graph reasoning results and memory retrieval results are uniformly scheduled and integrated; through the reasoning scheduling mechanism, the graph reasoning results and memory retrieval results are uniformly scheduled and integrated, and relevant information is dynamically selected to participate in the reasoning process according to the task requirements, so as to realize the collaborative reasoning of multimodal information and historical memory.

[0109] Specifically, the inference scheduling mechanism first generates a query vector and a set of target nodes based on the current task, and then obtains the graph inference results and memory retrieval results respectively. Subsequently, it calculates the fusion weight based on evidence relevance, source credibility, time freshness, graph path length, and business rule priority, and organizes high-weight evidence into an inference context. When the graph inference results conflict with the memory results, the system selects credible evidence based on confidence level and business rules, or triggers graph reconstruction, memory re-retrieval, supplementary input, or manual confirmation.

[0110] Step S7: Input the fused multi-source information into the large language model, and obtain the final output results through generative reasoning, including text answers, decision results, or structured information.

[0111] The information input to the large language model is not unordered concatenated text, but a structured context including task objectives, key nodes, reasoning paths, evidence sources, memory summaries, constraints, and output format. After the large language model generates the final output based on this structured context, the system performs consistency checks and evidence citation checks; if the output is inconsistent with graph evidence or memory evidence, it returns to step S6 to readjust the evidence weights or trigger supplementary retrieval.

[0112] The large-model reasoning method proposed in this invention integrates cross-modal graph structure modeling and multimodal collaborative memory mechanisms. Through structured modeling and collaborative reasoning mechanisms, it achieves unified expression and efficient reasoning of multi-source heterogeneous information. Its core lies in the following aspects:

[0113] In this invention, "cooperative memory" should be understood as memory units having not only similarity relationships, but also object relationships, event relationships, time relationships, source relationships, and reasoning dependencies; "cross-modal graph neural network" should be understood as a network structure that performs message passing and relationship aggregation on a heterogeneous graph containing text nodes, image nodes, structured field nodes, and memory nodes.

[0114] First, this invention constructs a cross-modal unified graph structure representation mechanism, which maps text, images, and structured data into a unified graph structure. It explicitly models multimodal semantic units and their relationships through nodes and edges, thereby breaking through the traditional implicit fusion method based on vector space and realizing the structured expression of cross-modal semantic relationships.

[0115] Secondly, this invention introduces a relational reasoning mechanism based on graph neural networks, which performs multi-round feature propagation and neighborhood information aggregation on the cross-modal graph structure to achieve deep fusion and multi-hop relational reasoning between multimodal information, thereby improving the system's ability to model complex semantic relationships.

[0116] Furthermore, this invention designs a multimodal collaborative memory mechanism to structurally store historical inputs, multimodal information, and intermediate reasoning results, and establishes associations between memory units, so that memory information can not only be retrieved, but also participate in the structured reasoning process, thereby improving the efficiency of information utilization in long context scenarios.

[0117] Furthermore, this invention proposes a collaborative fusion mechanism for graph reasoning results and memory retrieval results. Through a unified reasoning scheduling method, structured reasoning results and historical memory information are dynamically fused to achieve collaborative participation and joint decision-making of multi-source information in the reasoning process.

[0118] Finally, this invention constructs a closed-loop reasoning mechanism that supports multiple iterations. During the reasoning process, graph structure reconstruction or memory re-retrieval operations are dynamically triggered according to task requirements, forming a cyclically optimizable reasoning process, thereby supporting multi-hop reasoning and continuous optimization in complex scenarios.

[0119] In addition, the present invention also protects a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the steps of the above-described multimodal reasoning method.

[0120] Compared with existing multimodal reasoning techniques (such as cross-modal pre-trained models, retrieval enhancement generation, and single structured modeling methods based on graph neural networks), this invention has the following advantages in multimodal fusion and reasoning capabilities by introducing cross-modal graph structure modeling, multimodal collaborative memory mechanisms, and a unified reasoning scheduling framework:

[0121] (1) First, by constructing a cross-modal graph structure, the present invention realizes the explicit structured expression of multimodal information. Compared with the existing method that only relies on the vector space for implicit alignment, it can more accurately characterize the semantic relationship between different modalities, thereby significantly improving the accuracy and consistency of cross-modal understanding.

[0122] (2) Secondly, the present invention utilizes graph neural networks for relation propagation and multi-hop reasoning, enabling the correlation between multimodal data to be effectively propagated in the graph structure. Compared with the shallow fusion method in traditional methods, it can better support complex semantic relation modeling and reasoning tasks, and improve reasoning depth and logic.

[0123] (3) Furthermore, the multimodal collaborative memory mechanism proposed in this invention enables the memory units to be connected and reasonable by structuring and modeling the memory information, thus overcoming the information fragmentation problem of the existing RAG-based flat retrieval method and significantly improving the information utilization efficiency and reasoning continuity in long context scenarios.

[0124] (4) Furthermore, the present invention achieves dynamic collaborative fusion of graph reasoning results and memory information through reasoning scheduling mechanism, avoiding the information inconsistency problem caused by the independent operation of each module in the prior art, enabling multimodal information to participate in the reasoning process under a unified framework, thereby improving the overall reasoning stability and consistency.

[0125] (5) In addition, the present invention constructs a closed-loop reasoning mechanism that supports multiple rounds of iteration. During the reasoning process, graph structure reconstruction or memory re-retrieval can be dynamically triggered according to task requirements. Compared with the traditional one-time reasoning process, it can effectively support multi-hop reasoning and complex task processing, and improve the accuracy and robustness of reasoning results.

[0126] (6) Finally, the present invention integrates cross-modal modeling, structured reasoning and memory enhancement mechanisms to form a complete multimodal collaborative reasoning system. Compared with the existing technical solutions that only optimize in a single dimension, it has stronger system integrity and generalization ability and can be applied to a variety of complex application scenarios.

Claims

1. A multimodal collaborative memory large-scale model inference system integrating cross-modal graphical neural networks, characterized in that, include: Data access module: Acquires input information from different data sources and performs preprocessing and standardized representation; Cross-modal semantic alignment module: maps data from different modalities to a unified semantic space to achieve initial semantic consistency; Graph structure construction module: Abstracts entities, attributes and their relationships in multimodal data into a graph structure, and constructs a cross-modal association graph, where nodes represent multimodal semantic units and edges represent the association relationships between different entities or different modalities; Graph reasoning module: It uses graph neural networks to perform feature propagation and relation modeling on the graph structure, and achieves cross-modal semantic fusion and complex relation reasoning through multi-round neighborhood information aggregation, thereby obtaining an enhanced structured semantic representation; Multimodal memory management module: Constructs multimodal collaborative memory units to structure and store historical inputs, multimodal information and intermediate reasoning results, and realizes dynamic retrieval and fusion of memory information through an association retrieval mechanism to support long context and multi-turn reasoning tasks; The reasoning scheduling and control module: performs unified scheduling and fusion of graph reasoning results and memory information, and dynamically selects relevant information to participate in the reasoning process according to the current task requirements; Results generation module: Inputs the fusion results into the generation module based on the large language model to generate the final reasoning results or decision output; The functional units are connected through a unified data representation and interface. By introducing graph neural networks to perform structured representation and relation modeling of multimodal data, and combining multimodal memory enhancement and inference scheduling mechanisms, the collaborative fusion of multi-source heterogeneous information and high-precision inference are achieved.

2. The multimodal collaborative memory large model inference system integrating cross-modal graphical neural networks according to claim 1, characterized in that: The data access module includes a modality recognition unit, a data cleaning unit, a feature extraction unit, and a unified representation generation unit. The modality recognition unit identifies the source, format, and modality type of the input data. The data cleaning unit performs word segmentation, entity recognition, stop information filtering, and semantic vectorization on the text. The feature extraction unit performs OCR recognition, object detection, region feature extraction, and image-text correspondence extraction on the image, and performs field mapping, unit normalization, primary key alignment, and missing value processing on the structured data. The unified representation generation unit expresses the processing results as a data object containing modality identifiers, semantic vectors, entity sets, attribute sets, time information, source information, confidence levels, and permission constraints.

3. The multimodal collaborative memory large model inference system integrating cross-modal graphical neural networks according to claim 1, characterized in that: The cross-modal semantic alignment module uses a text encoder, an image encoder, and a structured data encoder to generate feature vectors for each modality, and maps the features of different modalities to the same semantic space through contrastive learning, attention mapping, or projection matrix. For the representation of the same entity in different modalities, cross-modal matching scores are calculated based on entity name, location region, contextual semantics, field identifier, and temporal relationship. When the matching score is higher than a preset threshold, a semantic alignment relationship is established.

4. The multimodal collaborative memory large model inference system integrating cross-modal graphical neural networks according to claim 1, characterized in that: The graph structure construction module maps text entities, image targets, table fields, business objects, historical memory fragments, and intermediate inference results to nodes, respectively, and maps entity subordination relationships, spatial relationships, temporal relationships, field association relationships, semantic similarity relationships, causal relationships, and cross-modal reference relationships to edges, and sets the relationship type, direction, weight, and confidence for each edge; the cross-modal association graph can be represented as G=(V,E,A,X), where V is the set of nodes, E is the set of edges, A is the weighted adjacency matrix, and X is the node feature matrix.

5. The multimodal collaborative memory large model inference system integrating cross-modal graphical neural networks according to claim 1, characterized in that: The multimodal memory management module writes historical inputs, user feedback, cross-modal graph nodes, inference paths, intermediate conclusions, and final outputs into the memory bank. Each memory unit includes memoryId, content, modality, embedding, linkedEntity, linkedNode, timeStamp, confidence, usageCount, and expireCondition, and establishes memory edges through relationships such as the same object, the same topic, causal dependence, temporal continuity, or user session. During memory retrieval, the system first performs vector similarity retrieval based on the current task representation, and then performs relation expansion retrieval based on memory edges to obtain a set of memories that are semantically and structurally related to the current task. memoryId represents the memory ID; content represents the memory content; modality represents the modality, the representation of this memory; embedding represents the content vector; linkedEntity represents the associated entity, i.e., the key information points contained or pointed to by the memory; linkedNode represents the associated node; timeStamp represents the timestamp; confidence represents the confidence level; usageCount represents the number of times it is used; and expireCondition represents the expiration rule.

6. A method for inference in a large multimodal collaborative memory model that integrates cross-modal graphical neural networks, comprising the following steps: Step S1: Acquire multimodal input data from different data sources and form a unified data input representation through preprocessing; Step S2: Achieve cross-modal semantic alignment by performing cross-modal coding on the preprocessed multimodal data; Step S3: Extract entities, attributes, and their relationships from the multimodal data, and construct a cross-modal association graph structure; Step S4: Based on graph neural networks, perform relation modeling and feature propagation on the graph structure; Step S5: Construct a multimodal collaborative memory bank and establish relationships between memories; Step S6: Through the reasoning scheduling mechanism, the graph reasoning results and memory retrieval results are uniformly scheduled and fused; Step S7: Input the fused multi-source information into the large language model, and obtain the final output results through generative reasoning, including text answers, decision results, or structured information.

7. The multimodal collaborative memory large model inference method fused with cross-modal graphical neural networks according to claim 6, characterized in that: In step S1, the multimodal input data is format-standardized, noise-filtered, permission-verified, and features are initially extracted. The text, images, and structured data are then converted into a unified input object, which includes at least the data source, modality type, original content, extracted text, entity set, attribute set, semantic vector, confidence level, timestamp, and access permissions.

8. The multimodal collaborative memory large model inference method fused with cross-modal graphical neural networks according to claim 6, characterized in that: In step S2, text vectors, image region vectors, and structured field vectors are obtained using different modal encoders. These vectors are then mapped to a unified semantic space through a projection network. Finally, a cross-modal alignment matrix is ​​calculated based on cosine similarity, attention weights, and entity matching rules to obtain a unified feature representation for multiple modalities.

9. The multimodal collaborative memory large model inference method fused with cross-modal graphical neural networks according to claim 6, characterized in that: In step S3, a cross-modal association graph is constructed based on the entity extraction results, image target recognition results, and structured field relationships. The nodes include text entity nodes, image region nodes, field nodes, business object nodes, and memory nodes. The edges include same-reference edges, subordinate edges, temporal relationship edges, spatial relationship edges, semantic similarity edges, and inference dependency edges. Weights are assigned to the edges based on matching confidence or business rules.

10. The multimodal collaborative memory large model inference method fused with cross-modal graphical neural networks according to claim 6, characterized in that: In step S4, a graph neural network is used to perform multi-round message passing on the cross-modal association graph. In the kth round, any node updates its node representation by aggregating the features of its neighboring nodes, edge type embeddings, and edge weights, and controls the contribution of different modal neighbors to the current node through attention weights. After multiple rounds of iteration, node representations, subgraph representations, or full graph representations that fuse structural relationships and cross-modal semantics are obtained for subsequent inference.

Images