Multi-modal relation extraction method and system based on multi-agent cooperation

Abstract

The invention provides a multi-modal relation extraction method and system based on multi-agent cooperation, and belongs to the technical field of entity relation recognition. Processing the obtained image by using a pre-trained processing model to obtain a relationship of entities contained in the image; wherein training of the processing model comprises the steps that multiple sets of data are obtained to serve as a training set, and each set of data comprises an image and a corresponding text to serve as a sample; respectively coding the image and the text to obtain a visual feature vector and a text feature vector; respectively training a Gaussian mixture model for the image modality and the text model, introducing a cross-view supervision mechanism, and guiding the model parameter optimization of the image modality by using the high confidence of the text modality; according to the method, fast or slow reasoning is dynamically selected, uncertainty is decomposed into accidental uncertainty and cognitive uncertainty, and the sample confidence coefficient is calculated through weighted entropy of multi-view Gaussian mixture clustering and model divergence.

Description

Technical Field

[0001] This invention relates to the field of entity relationship recognition technology, specifically to a multimodal relationship extraction method and system based on multi-agent collaboration. Background Technology

[0002] Multimodal relation extraction (MRE) is a fundamental task in information extraction, aiming to identify relationships between entities from given image and text cues. It is a key technology for constructing multimodal knowledge graphs and is widely used in downstream tasks such as multimodal knowledge graph construction, intelligent question answering systems, recommendation systems, and automated reasoning. Current mainstream MRE methods mostly employ a fully supervised learning paradigm, relying on a large number of labeled samples to achieve high-precision relation recognition. Although these methods achieve significant results on specific datasets, their heavy reliance on manually labeled data resources limits their generalization capabilities in open domains and various scenarios.

[0003] To reduce reliance on manual annotation, large language models (LLMs) have demonstrated strong generalization capabilities under zero-shot and few-shot conditions in recent years, sparking interest in their application in MRE tasks. Sun et al. fine-tuned a model using instructions to handle multiple MIE tasks, creating a unified multimodal information extractor and unifying three MIE tasks into a generation problem. This method requires a large amount of labeled data to construct the instruction dataset, enhancing generalization ability, but sometimes performs poorly. Yuan et al. guided ChatGPT to generate relevant knowledge based on semantic similarity and employed self-reflection to refine the knowledge. This method can address the problem of insufficient information in few-shot environments in joint multimodal relation extraction tasks. Zhang et al. proposed a multi-faceted contrastive learning-enhanced multimodal large model. This model uses the LoRA fine-tuning method to inject task-specific knowledge into InstructBLIP and then enhances the multimodal representation to capture relationships between samples using multi-faceted contrastive learning algorithms. In general, these methods still face many challenges in practical reasoning, such as strong dependence on specific instruction samples, limited generalization, and difficulty in controlling inference depth.

[0004] As the underlying model architecture evolves, the reasoning paradigm is gradually shifting from fast, intuitive processing (System 1) to slow, deep reasoning (System 2). Experiments show that large inference models (LRMs) can achieve better performance in complex relationship scenarios through long-chain reasoning, but their computational cost is significant and their reasoning efficiency is low. Conversely, System 1 models such as multimodal large models (LVLMs), while possessing fast response capabilities, lack accuracy when facing intermodal ambiguity or complex semantic relationships. Furthermore, some deep inference models (LRMs) also suffer from the problem of "over-reasoning," generating false relationship judgments on simple relationships or irrelevant entity pairs, leading to performance degradation.

[0005] In summary, existing multimodal relation extraction methods generally employ a fully supervised learning paradigm. While achieving significant performance improvements, these methods heavily rely on large amounts of manually labeled data. In practical applications, labeled data is often limited to specific domains, making data acquisition difficult and significantly restricting the generalization and adaptability of multimodal relation extraction models across different domains. Given that large language models (LLMs) exhibit superior generalization capabilities under zero-shot or few-shot learning conditions, they are widely used in various tasks. Existing techniques employ, on the one hand, instruction fine-tuning methods based on multimodal models (LVLMs) to achieve multimodal relation extraction tasks; on the other hand, they use knowledge generation methods combined with semantic similarity to support few-shot MRE tasks. However, current methods still rely on the construction of large amounts of labeled instruction data or small-shot datasets, and have not yet fully explored the potential reasoning capabilities of large multimodal models in zero-shot MRE tasks.

[0006] The root of the aforementioned problems lies in the current methods' lack of explicit dynamic modeling of fast and slow thinking during the inference process, especially in the face of varying sample complexity, where they lack the ability to adaptively adjust the depth and pattern of inference. Therefore, a new multimodal inference framework is urgently needed that can dynamically balance inference efficiency and effectiveness based on sample characteristics, achieving a collaboration between "fast thinking" and "slow thinking" processes similar to those in human thought, thereby improving the practicality and scalability of large models in real-world multimodal information extraction tasks. Summary of the Invention

[0007] The purpose of this invention is to provide a multimodal relationship extraction method and system based on multi-agent collaboration, so as to solve at least one of the technical problems existing in the background art.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a multimodal relationship extraction method based on multi-agent cooperation, comprising:

[0010] Obtain the image to be processed;

[0011] The acquired image is processed using a pre-trained processing model to obtain the relationships between entities contained in the image. The training of the processing model includes: acquiring multiple sets of data as training sets, each set of data including an image and the corresponding text as a sample; encoding the image and text respectively to obtain visual feature vectors and text feature vectors; training a Gaussian mixture model for the image modality and the text model respectively, while introducing a cross-view supervision mechanism, using the high confidence of the text modality to guide the optimization of the model parameters of the image modality.

[0012] As a further limitation of the first aspect of the present invention, a probability matrix for text and image modalities is initialized; the entropy of each sample is calculated; the Gaussian mixture model of the image modality and the text modality is updated using the expectation-maximization algorithm; with the aid of a cross-view supervision mechanism, a weight is calculated using the high confidence of the text modality, and the Gaussian mixture model of the image modality is updated using this weight; after multiple rounds of training iterations, the condition for training termination is when the Jaccard similarity between the current posterior probability matrix and the previous posterior probability matrix is ​​greater than 0.8; the final uncertainty of the sample is obtained by weighting the uncertainties of the image modality and the text modality.

[0013] As a further limitation of the first aspect of the present invention, for the image modality, CLIP-ViT-base encoding of visual feature vectors is used, and the optimal number of clusters is determined by maximizing the contour coefficient; for the text modality, BGE-M3 encoding of text feature vectors is used, and the optimal number of clusters is also determined by maximizing the contour coefficient.

[0014] As a further limitation of the first aspect of the present invention, for image modalities, visual feature vectors are extracted using CLIP-ViT-base. ∈ For text modalities, BGE-M3 is used to obtain the text embedding, i.e. ∈ :

[0015]

[0016]

[0017] Where I and T represent the input image and text, respectively. and It is the corresponding pre-trained encoder.

[0018] As a further limitation of the first aspect of the invention, in order to determine the optimal number of clusters for each modality, the silhouette coefficient is used to evaluate the compactness and separability of the clustering results. For a given sample i, the silhouette score s(i) is defined as:

[0019]

[0020] Where a(i) represents the average distance between sample i and all other points in the same cluster, and b(i) is the minimum average distance between sample i and the midpoint of the nearest neighbor cluster;

[0021] To identify the optimal number of clusters , in K∈[ , Perform a grid search on the [database name] and select the option that maximizes the average profile coefficient. The value of .

[0022] As a further limitation of the first aspect of the invention, the final uncertainty is obtained by combining the entropy of the two views, and an adaptive weighting mechanism is used to dynamically favor the view with lower uncertainty, thereby enhancing the robustness of the estimation:

[0023]

[0024] The weighting coefficient λ(i)∈[0,1] is automatically adjusted according to modal importance, and λ is calculated through the text-image cosine similarity score.

[0025] Secondly, the present invention provides a multimodal relationship extraction system based on multi-agent cooperation, comprising:

[0026] The acquisition module is used to acquire the image to be processed;

[0027] The processing module is used to process the acquired image using a pre-trained processing model to obtain the relationships between entities contained in the image. The training of the processing model includes: acquiring multiple sets of data as training sets, each set of data including an image and the corresponding text as a sample; encoding the image and text to obtain visual feature vectors and text feature vectors respectively; training a Gaussian mixture model for the image modality and the text model respectively, while introducing a cross-view supervision mechanism, using the high confidence of the text modality to guide the optimization of the model parameters of the image modality, and obtaining the relationships between entities in each sample.

[0028] Thirdly, the present invention provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the multimodal relationship extraction method based on multi-agent cooperation as described in the first aspect.

[0029] Fourthly, the present invention provides a computer device including a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the multimodal relationship extraction method based on multi-agent cooperation as described in the first aspect.

[0030] Fifthly, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the multimodal relationship extraction method based on multi-agent cooperation as described in the first aspect.

[0031] Terminology Explanation: Dual-system Cognitive Theory: This theory posits that human thought consists of two systems: System 1, a fast, intuitive, and automatic system for everyday judgment and immediate reactions; and System 2, a slow, rational, and deliberate system for handling complex problems and deep reasoning. These two systems work together to influence human decision-making and behavior. Agent: An intelligent entity built on a large language model, possessing environmental awareness, autonomous understanding, decision-making, and action-taking capabilities. Agents can simulate independent thinking processes, flexibly utilize various tools, and gradually achieve preset goals, aiming to complete complex tasks through the close integration of perception, thinking, and action. Multi-modal Relation Extraction: This technique integrates visual information to enrich textual relation analysis, using textual and visual cues to detect and classify semantic relationships between entities. For each potential relation... , configuration reliability score Relationship set This includes predefined relation types, where "None" indicates that a specific relation is missing.

[0032] The beneficial effects of this invention are as follows: It proposes a "Talker-Thinker" agent collaboration framework that can be seamlessly integrated with existing multimodal large model (LVLMs) and large inference model (LRMs). By dynamically switching between the two agents, it significantly balances their performance and efficiency in zero-shot multimodal relation extraction tasks. An uncertainty-guided switching adapter is introduced to dynamically select fast or slow inference. Uncertainty is decomposed into accidental uncertainty and cognitive uncertainty. Sample confidence is calculated using weighted entropy of multi-view Gaussian mixture clustering and model divergence. High-confidence samples are assigned to the Talker agent, and low-confidence samples are assigned to the Thinker agent.

[0033] The advantages of additional aspects of the invention will be set forth more clearly in the following description or will be learned by practice of the invention. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is an overall framework diagram of the Talker-Thinker intelligent agent collaborative system described in an embodiment of the present invention.

[0036] Figure 2 This is a flowchart of the multimodal relationship extraction method based on multi-agent collaboration according to an embodiment of the present invention. Detailed Implementation

[0037] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0038] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0039] It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as here.

[0040] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.

[0041] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0042] To facilitate understanding of the present invention, the present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments. However, the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0043] Those skilled in the art should understand that the accompanying drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily essential for implementing the present invention.

[0044] This invention proposes a multi-agent (“Talker-Thinker” Agent) collaborative framework inspired by dual-system cognitive theory. This framework decouples the multimodal relation extraction process through structural decoupling and introduces three modules to achieve efficient, robust, and reasoning-capable relation extraction: First, the Talker agent, possessing fast, intuitive response capabilities, primarily handles multimodal input and generates immediate feedback; second, the Thinker agent, possessing slow, deep reasoning capabilities, can invoke external tools and is suitable for handling complex, ambiguous, or semantically ambiguous relation scenarios; third, a dynamic switching adapter, which dynamically schedules the collaboration process between the Talker and Thinker agents based on the uncertainty of the samples.

[0045] Example 1

[0046] In this embodiment 1, a multimodal relation extraction system based on multi-agent collaboration is first provided, including: an acquisition module for acquiring images to be processed; and a processing module for processing the acquired images using a pre-trained processing model to obtain the relationships between entities contained in the images. The training of the processing model includes: acquiring multiple sets of data as training sets, each set including an image and corresponding text as a sample; encoding the image and text to obtain visual feature vectors and text feature vectors respectively; training a Gaussian mixture model for the image modality and the text model respectively, while introducing a cross-view supervision mechanism, using the high confidence of the text modality to guide the optimization of the model parameters of the image modality, thereby obtaining the relationships between entities in each sample.

[0047] In this embodiment, the above-described system is used to implement a multimodal relation extraction method based on multi-agent collaboration, including: using an acquisition module to acquire an image to be processed; using a processing module to process the acquired image using a pre-trained processing model to obtain the relationships between entities contained in the image; wherein, training the processing model includes: acquiring multiple sets of data as training sets, each set of data including an image and the corresponding text as a sample; encoding the image and text respectively to obtain visual feature vectors and text feature vectors; training a Gaussian mixture model for the image modality and the text model respectively, while introducing a cross-view supervision mechanism, using the high confidence of the text modality to guide the optimization of the model parameters of the image modality.

[0048] The process involves initializing probability matrices for both text and image modalities; calculating the entropy of each sample; updating the Gaussian mixture model (GMM) for both image and text modalities using the expectation-maximization algorithm; calculating a weight using the high confidence of the text modalities through cross-view supervision, and using this weight to update the GMM for the image modalities; training ending when the Jaccard similarity between the current posterior probability matrix and the previous posterior probability matrix is ​​greater than 0.8 after multiple training iterations; and finally obtaining the final uncertainty of the samples by weighting the uncertainties of the image and text modalities.

[0049] For the image modality, CLIP-ViT-base encoding of visual feature vectors is used, and the optimal number of clusters is determined by maximizing the silhouette coefficient; for the text modality, BGE-M3 encoding of text feature vectors is used, and the optimal number of clusters is also determined by maximizing the silhouette coefficient.

[0050] For image modalities, visual feature vectors are extracted using CLIP-ViT-base. ∈ For text modalities, BGE-M3 is used to obtain the text embedding, i.e. ∈ :

[0051]

[0052]

[0053] Where I and T represent the input image and text, respectively. and It is the corresponding pre-trained encoder.

[0054] To determine the optimal number of clusters for each modality, the silhouette coefficient is used to evaluate the compactness and separability of the clustering results. For a given sample i, the silhouette score s(i) is defined as:

[0055]

[0056] Where a(i) represents the average distance between sample i and all other points in the same cluster, and b(i) is the minimum average distance between sample i and the midpoint of the nearest neighbor cluster;

[0057] To identify the optimal number of clusters , in K∈[ , Perform a grid search on the [database name] and select the option that maximizes the average profile coefficient. The value of .

[0058] The final uncertainty is obtained by combining the entropy of the two views, and an adaptive weighting mechanism is used to dynamically favor the view with lower uncertainty, thereby enhancing the robustness of the estimate:

[0059]

[0060] The weighting coefficient λ(i)∈[0,1] is automatically adjusted according to modal importance, and λ is calculated through the text-image cosine similarity score.

[0061] Example 2

[0062] like Figure 1 As shown, this embodiment proposes a collaborative “Talker-Thinker” agent system based on the dual-process theory of human cognition. The two agents exhibit complementary reasoning abilities: the talker agent, modeled as a System 1 model (LVLMs), makes fast and heuristic decisions based on context learning and surface-level alignment; the thinker agent, modeled as a System 2 model (LRMs), performs slower but deeper reasoning based on structured knowledge and external information sources. We also introduce an uncertainty-guided switching adapter to determine whether to use the talker and thinker agents, enabling them to dynamically collaborate to handle tasks of varying complexity and ambiguity.

[0063] The methods and modules designed and used in this embodiment will be explained in detail below.

[0064] The Talker agent is a fast-acting, language-centric agent built on a multimodal big model. It perceives the world, retrieves queries from users, generates dialogue responses, and communicates with users.

[0065] Given an image-text pair query The agent constructs a task-specific cue template and makes predictions through context learning. The speaker receives relational queries in natural language and uses the zero-shot cue template. Integrate visual and textual content into pre-trained frozen multimodal large models (LVLMs) and directly output predicted relation labels:

[0066] (1)

[0067] in, The output distribution of the visual language model integrates textual and visual features through learned multimodal embeddings. The speaker is efficient and well-suited for handling standard or explicit instances. Furthermore, the speaker maintains short-term memory in the form of recent cue-response pairs, ensuring dialogue consistency when deployed in a conversational setting. R represents the Talker's prediction of the query, i.e., the "relation" label it deems most likely to exist. This represents a prompt template for Talker to query a large model. It is a pre-designed natural language instruction query template used to guide a multimodal large model to understand the task and make predictions without additional task-specific training.

[0068] This represents the user's query (q), where text is the text content and img is the image content.

[0069] The Thinker agent is a slower, more deliberate reasoning agent built on large reasoning models (LRMs). It involves long chains of thinking and taking actions to enhance its knowledge from the real world, such as invoking tools (e.g., Google search) or retrieving information from external databases (e.g., Wikipedia).

[0070] Thinkers are typically goal-oriented, prepared to solve specific problems or achieve specific objectives. In this task, the thinker is responsible for reasoning step-by-step through difficult or ambiguous samples. Specifically, when the Talker exhibits samples with high uncertainty, these are passed to the thinker agent. The thinker then uses richer reasoning templates. It may contain external databases, chained reasoning formats, or instructions that explicitly invoke tools (e.g., “Search entity X in Wikipedia” or “Analyze entity in image”).

[0071] In form, the thinker's decision-making process can be represented as follows:

[0072] (2)

[0073] in, This represents limited prior knowledge retrieved from external sources (such as Wikipedia or search engines) to enhance the model. Thinker also maintains and updates long-term memory, storing structured beliefs and factual evidence to inform future reasoning or interpretation. This ability to retrieve and integrate external information allows Thinker to overcome knowledge blind spots and address the limitations of Talker. This represents the prompt template for Thinker's query to the large model.

[0074] In this embodiment, a system switching adapter is introduced between Talker and Thinker within the aforementioned framework. This dynamic switching mechanism automatically selects between slow and fast inference methods by evaluating the uncertainty of each sample, thereby optimizing efficiency and effectiveness. The method of guiding the quantification of whether a sample has high or low confidence through uncertainty is inspired by classic uncertainty decomposition. Therefore, the total uncertainty of the sample is divided into two parts: the inherent random uncertainty of the data and the cognitive uncertainty of model divergence.

[0075] For random uncertainty, it is quantified by estimating the posterior distribution of a given dataset and calculating the entropy of each sample. In this embodiment, a Gaussian mixture model (GMM) is used for multi-view collaborative clustering.

[0076] The detailed steps for multi-view collaborative clustering using Gaussian Mixture Model (GMM) are as follows.

[0077] 1. Multimodal feature extraction: For image modalities, CLIP-ViT-base is used to extract visual feature vectors. ∈ For text modalities, BGE-M3 is used to obtain the text embedding, i.e. ∈ The process can be formalized as follows:

[0078] (3)

[0079] (4)

[0080] Where I and T represent the input image and text, respectively. and It is the corresponding pre-trained encoder.

[0081] 2. Optimal Number of Clusters: To determine the optimal number of clusters for each modality, the silhouette coefficient is used to evaluate the compactness and separability of the clustering results. For a given sample i, the silhouette score s(i) is defined as:

[0082] (5)

[0083] Where a(i) represents the average distance between sample i and all other points in the same cluster (intra-cluster distance), and b(i) is the minimum average distance between sample i and the midpoint of the nearest neighbor cluster (inter-cluster distance).

[0084] To identify the optimal number of clusters , in K∈[ , Perform a grid search on the [database name] and select the option that maximizes the average profile coefficient. The value. This optimization process is performed independently for each modal view, and the result is:

[0085] (6)

[0086] in, The contour coefficients represent the image samples; Represents the contour coefficient of an image sample.

[0087] 3. Multi-view clustering optimization: An iterative multi-view Gaussian mixture model (GMM) cooperative training framework is proposed, described as follows:

[0088] Train Gaussian Mixture Models (GMMs) independently on image and text views:

[0089] (7)

[0090] (8)

[0091] This generates the initial membership matrix. and ,in Let represent the posterior probability that sample i belongs to cluster j. Based on information entropy theory, samples with low uncertainty are selected. Represents the image feature vector matrix; Represents the text feature vector matrix; The optimal number of clusters for representing image data; The optimal number of clusters for text data.

[0092] In the t-th iteration, the sample-level entropy is calculated from the member matrix:

[0093] (9)

[0094] in, This represents the probability that the i-th sample belongs to the k-th category (cluster) after the t-th iteration.

[0095] A cross-view supervision mechanism has been introduced. This is applied when updating the image view. At that time, Weight enhancement is applied to high-confidence samples. Specifically:

[0096] (10)

[0097] Where N represents the total number of samples in the dataset; This represents the weight of the i-th sample in the t-th iteration; This expresses the expectation calculation: based on the best current guess, calculate the average score for all possible classification scenarios; This indicates that the category label is hidden; Indicates model parameters; This represents the parameter estimate of the image model at iteration t+1; This represents the probability density function, which describes the likelihood of a random variable (or a set of variables) taking a specific value.

[0098] Weight here Defined as:

[0099] (11)

[0100] Here, γ is the enhancement coefficient, which is set to γ=2.0 by default. This mechanism ensures that high-confidence samples in the text view have a greater impact on the image clustering process.

[0101] Let J(i) represent the information entropy of the i-th sample in the t-th iteration of the text sample. Iteration stops when cluster allocation consistency satisfies J(i). When )>τ, where J( ) represents the Jaccard similarity between the current and previous member matrices, and τ=0.8 is the threshold for Jaccard similarity.

[0102] 4. Dynamically Weighted Uncertainty Estimation: The final uncertainty is obtained by combining the entropy of the two views, and an adaptive weighting mechanism is used to dynamically favor the view with lower uncertainty, thereby enhancing the robustness of the estimation.

[0103] (12)

[0104] The weighting coefficient λ(i)∈[0,1] is automatically adjusted based on modal importance. λ is calculated using the text-image cosine similarity score; where... Entropy represents the image modality; Entropy represents the text modality.

[0105] λ is calculated using the cosine similarity score between text and image vectors:

[0106] .

[0107] In this embodiment, the top α% of samples with the highest entropy scores (α=10 in this work) are selected to form a low-confidence sample set. .

[0108] Cognitive uncertainty stems from the model's insufficient understanding of objective facts (e.g., domain knowledge, grammatical rules, etc.). It reflects cognitive uncertainty within the model's knowledge boundaries and is a key factor in the illusion generated by LLMs.

[0109] Here, cognitive uncertainty is defined as the discrepancy in the predicted distributions of different models for the same sample due to the knowledge limitations of the models. The process is summarized below:

[0110] Given an input instance x, use an ensemble of N models. Each model k produces a predicted label, denoted as . , where Y is the set of candidate labels (e.g., relation types in an MRE task).

[0111] (13)

[0112] Calculate the number of votes received for each candidate label yi∈Y:

[0113] (14)

[0114] The indicator function I(*) here is defined as:

[0115] (15)

[0116] Define a sample x as uncertain. If the maximum number of votes V is less than a threshold Tv, use majority voting to assign the final predicted label as follows:

[0117] (16)

[0118] In form, the uncertain set Defined as:

[0119] (17)

[0120] In this embodiment, N=3 is set and defined =2 to capture complete divergence, meaning all models choose different labels. This effectively identifies samples where models hold divergent beliefs—an indicator of cognitive uncertainty that may stem from gaps in learned knowledge.

[0121] Finally, the switcher adapter can be represented as a router, defined as:

[0122] (18)

[0123] This mechanism ensures that simple examples are handled efficiently by the Talker agent, while challenging cases (such as samples with insufficient contextual information or modal conflicts) are escalated to the Thinker agent for deeper reasoning and tool calls.

[0124] like Figure 2 As shown, the model proposed in this embodiment takes image-text pairs as input: an image and its corresponding text constitute a sample. The algorithm outputs the relationships between entities in each sample.

[0125] The specific processing steps include the following:

[0126] 1. Encode images and text separately: For the image modality, use CLIP-ViT-base to encode the visual feature vector, and determine the optimal number of clusters by maximizing the silhouette coefficient. For the text modality, use BGE-M3 to encode the text feature vector. Similarly, determine the optimal number of clusters by maximizing the silhouette coefficient.

[0127] 2. Calculate sample uncertainty: Train a Gaussian mixture model for the image modality and the text model respectively, and introduce a cross-view supervision mechanism to use the high confidence of the text modality to guide the optimization of the model parameters of the image modality.

[0128] Step 1: Initialize a probability matrix for text and image modalities.

[0129] Step 2: Calculate the entropy of each sample.

[0130] Step 3: Use the expectation-maximization algorithm to update the Gaussian mixture model for the image and text modalities.

[0131] Step 4: Using a cross-view supervision mechanism, calculate a weight using the high confidence of the text modality, and update the Gaussian mixture model of the image modality using this weight.

[0132] Step 5: After multiple rounds of training iterations, the condition for ending training is when the Jaccard similarity between the current posterior probability matrix and the previous posterior probability matrix is ​​greater than 0.8.

[0133] Step 6: Obtain the final uncertainty of the sample by weighting the uncertainties of the image modality and the text modality.

[0134] 3. Selecting an agent based on the switching adapter: If a sample has a high confidence level, it is directly handed over to the Talker agent, which includes multiple models for prediction. If the predictions from multiple models are consistent, the answer is directly output. Otherwise, the Thinker agent is used to think and arrive at the answer. If a sample has a low confidence level, it is handed over to the Thinker agent to think and arrive at the answer. The uncertainty of the sample is calculated using formula (12). The higher the uncertainty, the lower the confidence level. A threshold is set to distinguish between high-confidence samples and low-confidence samples.

[0135] Example 3

[0136] This embodiment 3 provides a non-transitory computer-readable storage medium for storing computer instructions. When executed by a processor, the computer instructions implement the multi-modal relationship extraction method based on multi-agent cooperation as described above. The method includes:

[0137] The process involves: acquiring an image to be processed; processing the acquired image using a pre-trained processing model to obtain the relationships between entities contained in the image; wherein, training the processing model includes: acquiring multiple sets of data as training sets, each set of data including an image and the corresponding text as a sample; encoding the image and text respectively to obtain visual feature vectors and text feature vectors; training a Gaussian mixture model for the image modality and the text model respectively, while introducing a cross-view supervision mechanism, using the high confidence of the text modality to guide the optimization of the model parameters of the image modality.

[0138] Example 4

[0139] This embodiment 4 provides a computer device, including a memory and a processor, wherein the processor and the memory communicate with each other, and the memory stores program instructions that can be executed by the processor. The processor calls the program instructions to execute the multimodal relation extraction method based on multi-agent cooperation as described above, the method including:

[0140] The process involves: acquiring an image to be processed; processing the acquired image using a pre-trained processing model to obtain the relationships between entities contained in the image; wherein, training the processing model includes: acquiring multiple sets of data as training sets, each set of data including an image and the corresponding text as a sample; encoding the image and text respectively to obtain visual feature vectors and text feature vectors; training a Gaussian mixture model for the image modality and the text model respectively, while introducing a cross-view supervision mechanism, using the high confidence of the text modality to guide the optimization of the model parameters of the image modality.

[0141] Example 5

[0142] This embodiment 5 provides an electronic device, including: a processor, a memory, and a computer program; wherein, the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the multimodal relationship extraction method based on multi-agent cooperation as described above. The method includes:

[0143] The process involves: acquiring an image to be processed; processing the acquired image using a pre-trained processing model to obtain the relationships between entities contained in the image; wherein, training the processing model includes: acquiring multiple sets of data as training sets, each set of data including an image and the corresponding text as a sample; encoding the image and text respectively to obtain visual feature vectors and text feature vectors; training a Gaussian mixture model for the image modality and the text model respectively, while introducing a cross-view supervision mechanism, using the high confidence of the text modality to guide the optimization of the model parameters of the image modality.

[0144] In summary, this invention proposes a "Talker-Thinker" agent collaborative framework to simulate the fast and slow thinking mechanisms of humans, thereby improving the performance of language models in zero-shot multimodal relation extraction (MRE) tasks. This framework dynamically adapts to task difficulty by introducing an uncertainty-guided thinking mode switching mechanism: a fast thinking strategy is employed for simple, highly deterministic samples to improve reasoning efficiency; while slow thinking is triggered for complex or ambiguous samples to ensure reasoning quality. This invention combines random uncertainty with model cognitive uncertainty, utilizes a multi-view Gaussian mixture model for low-confidence sample identification, and delegates the processing to a thinker agent with memory, reflection, and external interaction capabilities. This significantly enhances the reasoning ability and versatility of large language models in complex information environments, achieving a dual optimization of efficiency and effectiveness.

[0145] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0149] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solutions disclosed in the present invention, various modifications or variations that can be made by those skilled in the art without creative effort should be included within the scope of protection of the present invention.

Claims

1. A multimodal relation extraction method based on multi-agent collaboration, characterized in that, include: Obtain the image to be processed; The acquired image is processed using a pre-trained processing model to obtain the relationships between entities contained in the image. The training of the processing model includes: acquiring multiple sets of data as training sets, each set of data including an image and the corresponding text as a sample; encoding the image and text respectively to obtain visual feature vectors and text feature vectors; training a Gaussian mixture model for the image modality and the text model respectively, while introducing a cross-view supervision mechanism, using the high confidence of the text modality to guide the optimization of the model parameters of the image modality.

2. The multimodal relationship extraction method based on multi-agent cooperation according to claim 1, characterized in that, Initialize a probability matrix for both text and image modalities; calculate the entropy of each sample; update the Gaussian mixture model for both image and text modalities using the expectation-maximization algorithm; with cross-view supervision, calculate a weight using the high confidence of the text modality, and update the Gaussian mixture model for the image modality using this weight; after multiple training iterations, the training ends when the Jaccard similarity between the current posterior probability matrix and the previous posterior probability matrix is ​​greater than 0.8; obtain the final uncertainty of the samples by weighting the uncertainties of the image and text modalities.

3. The multimodal relation extraction method based on multi-agent cooperation according to claim 2, characterized in that, For the image modality, CLIP-ViT-base encoding of visual feature vectors is used, and the optimal number of clusters is determined by maximizing the silhouette coefficient; for the text modality, BGE-M3 encoding of text feature vectors is used, and the optimal number of clusters is also determined by maximizing the silhouette coefficient.

4. The multimodal relationship extraction method based on multi-agent cooperation according to claim 3, characterized in that, For image modalities, visual feature vectors are extracted using CLIP-ViT-base. ∈ ; For text modalities, BGE-M3 is used to obtain the text embedding, i.e. ∈ : ； ； Where I and T represent the input image and text, respectively. and It is the corresponding pre-trained encoder.

5. The multimodal relationship extraction method based on multi-agent cooperation according to claim 4, characterized in that, To determine the optimal number of clusters for each modality, the silhouette coefficient is used to evaluate the compactness and separability of the clustering results. For a given sample i, the silhouette score s(i) is defined as: ； Where a(i) represents the average distance between sample i and all other points in the same cluster, and b(i) is the minimum average distance between sample i and the midpoint of the nearest neighbor cluster; To identify the optimal number of clusters , in K∈[ , Perform a grid search on the [database name] and select the option that maximizes the average profile coefficient. The value of .

6. The multimodal relationship extraction method based on multi-agent cooperation according to claim 5, characterized in that, The final uncertainty is obtained by combining the entropy of the two views, and an adaptive weighting mechanism is used to dynamically favor the view with lower uncertainty, thereby enhancing the robustness of the estimate: ； The weighting coefficient λ(i)∈[0,1] is automatically adjusted according to modal importance, and λ is calculated through the text-image cosine similarity score.

7. A multimodal relationship extraction system based on multi-agent cooperation, characterized in that, include: The acquisition module is used to acquire the image to be processed; The processing module is used to process the acquired image using a pre-trained processing model to obtain the relationships between entities contained in the image. The training of the processing model includes: acquiring multiple sets of data as training sets, each set of data including an image and the corresponding text as a sample; encoding the image and text to obtain visual feature vectors and text feature vectors respectively; training a Gaussian mixture model for the image modality and the text model respectively, while introducing a cross-view supervision mechanism, using the high confidence of the text modality to guide the optimization of the model parameters of the image modality, and obtaining the relationships between entities in each sample.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the multimodal relationship extraction method based on multi-agent cooperation as described in any one of claims 1-6.

9. A computer device, characterized in that, The method includes a memory and a processor, the processor and the memory communicating with each other, the memory storing program instructions that can be executed by the processor, and the processor calling the program instructions to execute the multimodal relation extraction method based on multi-agent cooperation as described in any one of claims 1-6.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions that implement the multimodal relation extraction method based on multi-agent cooperation as described in any one of claims 1-6.

Images