A visual reasoning method and device based on GW distance improved MLLM model, and a medium

Abstract

The application relates to the technical field of visual reasoning, and discloses a visual reasoning method and device based on an improved MLLM model of GW distance and a medium, which creatively regards a token as a basic element and applies the token to GW distance through a text token and a visual token, and optimizes an MLLM model in the idea of reverse GW solving to obtain an ideal visual reasoning result taking a text relationship as priori. First, a first matrix is constructed by taking the semantic correlation degree between text tokens as an element, and a second matrix is constructed by taking the semantic correlation degree between the text token and the visual token as an element, so that the first matrix serves as a first distance matrix in GW distance, and the second matrix serves as a coupling matrix in GW distance. Then, the GW objective function minimization algorithm is adopted to reversely deduce the second distance matrix while fixing the two kinds of matrices. Finally, the MLLM model is optimized to improve the precision of semantic relationship understanding in the MLLM model migrating to visual relationship understanding, so that the result obtained by the MLLM model for visual reasoning is more accurate.

Description

Technical Field

[0001] This invention relates to the field of visual reasoning technology, and in particular to a visual reasoning method, apparatus, and medium based on an improved MLLM model using GW distance. Background Technology

[0002] In recent years, multimodal large language models (MLLMs) have made significant progress in visual understanding. Models such as GPT-4V, LLaVA, and Qwen-VL are capable of complex visual question answering, scene relationship understanding, and multi-turn visual reasoning. These capabilities largely stem from their powerful language model backbone, which encodes rich world knowledge and structured relational reasoning abilities. Current mainstream MLLMs typically employ a "visual encoder + projection adapter + large language model" architecture: the visual encoder (such as ViT) processes images into a set of visual tokens, which are then mapped to the language embedding space through a learnable projection adapter. These tokens, along with text tokens, are then input into the large language model for training with an autoregressive language modeling objective.

[0003] However, a challenge arises when these models need to apply pre-existing linguistic relational knowledge to visual scenes. For example, a large language model can correctly infer the "tool-function" relationship of "a hammer is used for striking, and scissors are used for cutting"; but the same multimodal model, upon seeing an image of a hammer and paper, might incorrectly answer "a hammer can be used to cut paper." Similarly, a language model may know that "Huskies and Alaskan Malamutes differ in ear shape and body proportions," but a multimodal model may still visually confuse the two breeds. Therefore, existing technologies suffer from inaccurate inference results when MLLM models perform visual reasoning. Summary of the Invention

[0004] This invention provides a visual reasoning method, apparatus, and medium based on GW distance to improve the visual reasoning results of MLLM models, so as to make the results of visual reasoning by MLLM models more accurate.

[0005] The first aspect of this invention discloses a visual reasoning method based on an improved MLLM model using GW distance, the method comprising: Obtain a text token and a visual token. The text token is a unit used to represent the semantic features of word segmentation after encoding the original text, and the visual token is a unit used to represent the region features after encoding the original image. Construct a first matrix to represent the dependency score between different text tokens, whereby the dependency score represents the degree of semantic relevance between different tokens; Construct a second matrix to represent the dependency score between the text token and the visual token; The first matrix is ​​determined to be the first distance matrix in the GW objective function, and the second matrix is ​​determined to be the coupling matrix in the GW objective function. A minimization operation is performed on the GW objective function to obtain the second distance matrix. Based on the second distance matrix, the preset MLLM model is optimized to obtain an optimized MLLM model, which is used to generate visual reasoning results based on images and text.

[0006] As an optional implementation, in the first aspect of the present invention, the method for obtaining the optimized MLLM model includes: Construct a third matrix, the elements of which are used to represent the dependency scores between different visual tokens; Based on the second distance matrix and the third matrix, a difference parameter is obtained, which is used to measure the difference between the second distance matrix and the third matrix; Based on the difference parameters, the loss function corresponding to the preset MLLM model is optimized to obtain an optimized loss function. The optimized loss function is used to constrain the training process of the MLLM model to obtain an optimized MLLM model.

[0007] As an optional implementation, in the first aspect of the present invention, the construction method of the second matrix includes: Input the text token and the visual token into a preset MLLM model to obtain the attention distribution corresponding to each attention head in the MLLM model. For each attention head: obtain the information entropy based on the attention distribution. Sort the information entropy corresponding to all attention heads in ascending order, and determine the attention heads whose sorting results are before a preset first threshold as candidate attention heads; aggregate the attention scores corresponding to all candidate attention heads to construct a second matrix.

[0008] As an optional implementation, in the first aspect of the invention, before performing a minimization operation on the GW objective function to obtain the second distance matrix, the method further includes: Based on the second matrix, a first suppression parameter is obtained. The first suppression parameter is used to measure the coupling quality of each text token. The coupling quality is obtained based on all elements corresponding to the text token in the second matrix. The encoded sequence corresponding to the text token is determined to be the first sequence; The second suppression parameter is obtained based on the position of the text token in the first sequence; Based on the first suppression parameter and the second suppression parameter, a weight parameter is obtained. The weight parameter is used to perform weighted optimization on the elements in the second matrix to obtain an optimized second matrix. The optimized second matrix is ​​determined to be the coupling matrix in the GW objective function.

[0009] As an optional implementation, in the first aspect of the present invention, the MLLM model includes multiple attention layers, which are used to analyze the dependencies between different tokens in a hierarchical order, and the method further includes: A first self-attention layer, a second self-attention layer, and a cross-attention layer are defined. The first self-attention layer is used to analyze the dependency relationship between different text tokens. The second self-attention layer is used to analyze the dependency relationship between different visual tokens. The cross-attention layer is used to analyze the dependency relationship between the text token and the visual token. Furthermore, the construction methods of the first matrix, the second matrix, and the third matrix also include: Input the text token and the visual token into the MLLM model; Based on the analysis results of the first layer in all the first self-attention layers, construct the first matrix; Based on the analysis results of the last layer in all second self-attention layers, a third matrix is ​​constructed; A second matrix is ​​constructed based on the analysis results of all intermediate layers across the attention layer.

[0010] As an optional implementation, in the first aspect of the present invention, the first matrix The calculation methods include:

[0011] In the above formula, For the initial first matrix, The adjustment parameters are used to stabilize the calculation results, wherein the initial first matrix is ​​obtained based on the analysis results of the first layer among all first self-attention layers.

[0012] As an optional implementation, in the first aspect of the invention, the second distance matrix The calculation methods include:

[0013] In the above formula, Let be the coupling matrix. , The index for the text token. , is the edge vector of the coupling matrix.

[0014] A second aspect of the present invention discloses a visual reasoning device based on a GW distance-improved MLLM model, the device comprising: The unit acquisition module is used to acquire text tokens and visual tokens. The text tokens are units obtained after embedding the original text into words to represent the semantic features of word segmentation, and the visual tokens are units obtained after encoding the original image to represent the region features. A matrix construction module is used to construct a first matrix, which represents the dependency score between different text tokens, and the dependency score represents the semantic relevance between different tokens; and to construct a second matrix, which represents the dependency score between the text token and the visual token. The function derivation module is used to determine that the first matrix is ​​the first distance matrix in the GW objective function, determine that the second matrix is ​​the coupling matrix in the GW objective function, perform a minimization operation on the GW objective function, and obtain the second distance matrix. The model optimization module is used to optimize the preset MLLM model according to the second distance matrix to obtain an optimized MLLM model. The optimized MLLM model is used to generate visual reasoning results based on images and text.

[0015] A third aspect of the present invention discloses another device comprising a memory and a processor, the device comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute some or all of the steps in the visual reasoning method based on the GW distance improved MLLM model according to any of the first aspects of the present invention.

[0016] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked by a processor, are used to execute some or all of the steps in the visual reasoning method based on the GW distance-improved MLLM model described in any of the first aspects of the present invention.

[0017] Compared with the prior art, the present invention has the following beneficial effects: This invention creatively treats text tokens and visual tokens as fundamental elements and applies them to the Geometric Logic Analyzer (GW) distance calculation. First, a first matrix is ​​constructed using the semantic relevance between text tokens as elements, and a second matrix is ​​constructed using the semantic relevance between text and visual tokens as elements. The first matrix serves as the first distance matrix in the GW distance calculation, and the second matrix serves as the coupling matrix. Then, based on the fixed first and coupling matrices, the second distance matrix is ​​derived in reverse using the GW objective function minimization algorithm. Finally, the MLLM model is optimized based on the second distance matrix to improve the accuracy of transferring semantic relationship understanding from the MLLM model to visual relationship understanding, thereby making the results obtained by the MLLM model for visual reasoning more accurate. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a visual reasoning method based on an improved MLLM model using GW distance, as disclosed in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a visual inference device based on an improved MLLM model using GW distance, as disclosed in an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of a device including a memory and a processor disclosed in an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] It should be noted that the Gromov–Wasserstein (GW) distance mentioned in this paper applies to general metric spaces. Therefore, the definition of "graph" in this paper is not limited to regular pixel grids like images, but should also include any structure that can be constructed using the relationships between nodes and edges, that is, various structural graphs that can be generated using such structures as elements. Therefore, the technical solutions adopted in the following embodiments can be applied to the fields of social network analysis (user alignment), bioinformatics (protein interaction network alignment, brain network alignment), chemical information (molecular graph matching), knowledge graphs (entity alignment), computer vision (scene graph matching, point cloud registration), natural language processing (syntactic tree alignment), transportation networks (road network matching), and finance (transaction object alignment), etc.

[0024] Furthermore, the graph alignment method described in this paper aims to establish a correspondence between nodes in two graphs, ensuring that the intrinsic structure of the nodes remains as consistent as possible under this correspondence. This correspondence includes matching and mapping; therefore, this graph alignment method can align nodes / structures of the same type / domain or perform cross-modal alignment, establishing such a correspondence at the semantic or structural level for information from different data formats. For example, given an image containing a target object and a text description containing the target object, feature vectors are extracted from the image using a CNN model and from the text using BERT. Matching / aligning these two feature vectors is then performed. The GW distance can be used to achieve alignment by comparing the consistency of the graph's internal structure.

[0025] Example 1 Please see Figure 1 , Figure 1This is a flowchart illustrating a visual reasoning method based on an improved MLLM model using GW distance, as disclosed in an embodiment of the present invention. Figure 1 The described visual reasoning method based on the GW distance-improved MLLM model can be applied to visual reasoning devices based on the GW distance-improved MLLM model. For example... Figure 1 As shown, this graph alignment method based on GW distance can include the following operations: Step 101: Obtain the text token and visual token.

[0026] A token is the basic unit for a model to process and understand text. In the Transformer encoder, the input sentence is converted into a token sequence, and each token is a vector. In this embodiment of the invention, a text token is a unit used to represent the semantic features of word segmentation after encoding the original text. Specifically, a text token is a discrete unit obtained after word segmentation encoding of the original text. Each unit corresponds to a word segment and is used to represent the semantic features of that word segmentation. The word segmenter can divide the original text into the smallest semantic units that the model can process. The original text may include natural language text (such as sentences, paragraphs, articles, numbers, punctuation, special symbols), special format text (code, program source code, markup languages ​​such as HTML and JSON, mathematical formulas such as LaTeX), text in multimodal models (text prompts input to the model, video subtitles, text obtained after speech recognition), and non-natural language sequences (DNA sequences, binary data). This embodiment of the invention does not limit this.

[0027] Specifically, methods for obtaining a text token may include: 1. BPE (Byte-Pair Encoding): Starting with a set of individual characters, iteratively finds and merges the most frequent adjacent character pairs in the corpus to form new subwords; 2. WordPiece: Unlike BPE, WordPiece selects the character pairs that maximize the likelihood of the training data after merging. 3. SentencePiece: It integrates algorithms such as BPE / Unigram, does not rely on space segmentation, and treats all text, including spaces, as a sequence of Unicode characters.

[0028] In this embodiment of the invention, a visual token is a unit used to represent regional features after encoding the original image. Specifically, a visual token is a structured discrete or continuous representation unit obtained after encoding the original image. Each token represents the semantic or structural information of a local region (such as an image block) in the image. The original image can be represented as a signal of a two-dimensional (or three-dimensional) pixel array, including natural photographs, medical images, screenshots, and spectrograms converted from sound. This embodiment of the invention does not limit this.

[0029] Specifically, methods for obtaining visual tokens may include: 1. ViT (Vision Transformer): The original image is divided into fixed-size, non-overlapping small squares (patch). Each patch is flattened and mapped into a fixed-dimensional continuous vector, i.e., a visual token, through a learnable linear layer (or convolutional layer). 2. VQ-VAE: The autoencoder learns a visual vocabulary (Codebook). Each entry in the vocabulary represents a visual primitive. The original image is input into the encoder, and a feature map is output. For each cell in the feature map, the most similar word is found in the Codebook, and the cell itself is replaced with the number of that word. The entire image is encoded into a sequence of integers, and the vectors in the sequence are the visual tokens.

[0030] It should be noted that the ViT module can be integrated into MLLM, therefore, the process of obtaining visual tokens from the original image encoding can also be implemented in the MLLM model.

[0031] Step 102: Construct the first matrix; construct the second matrix.

[0032] In this embodiment of the invention, the first matrix can be used to represent the dependency score between different text tokens. The dependency score can be used to represent the semantic relevance between different tokens. The first matrix is ​​a form that simulates the distance matrix (relation matrix) in the GW objective function, and its construction method is the same as that of the distance matrix. Specifically, each text token is treated as a node, a similarity matrix is ​​constructed, and then the similarity matrix is ​​transformed into a distance matrix. For example, for a similarity matrix S, the corresponding distance matrix C can be calculated as follows:

[0033] or

[0034] The specific form of the transformation process is not limited in this embodiment of the invention. The resulting distance matrix C is the first matrix. Each element in the matrix can correspond to a (i,j) coordinate, where i and j are the indices of the text tokens. Each coordinate corresponds to a dependency score, which is used to represent the semantic relevance between the i-th text token and the j-th text token.

[0035] In this embodiment of the invention, the second matrix can be used to represent the dependency score between text tokens and visual tokens. The second matrix is ​​a form that simulates the coupling matrix (transmission plan) in the GW objective function, and its construction method is the same as that of the coupling matrix. Specifically, each text token is considered as a node in the source domain, and each visual token is considered as a node in the target domain. Assuming... and These are two discrete probability distributions, defined on two sets of nodes in two spaces, respectively. Each element in the table represents the quality / weight assigned to each node in the source domain. The elements in the target domain are the quality of each node. Here, quality can be understood as the importance or available matching budget of each node in the semantic matching task, which is used to control the capacity or priority of each node in the alignment task.

[0036] The coupling matrix, i.e., the second matrix, is defined in... and Given the joint probability distribution under marginal constraints, the dependency fractions in this matrix represent the distribution... The quality of a node in the source domain being transmitted to the corresponding node in the target domain can be interpreted as the joint probability of selecting node i from the source domain and node k from the target domain, i.e. the probability of matching the text token and visual token corresponding to the two nodes. Under the premise of maintaining their respective marginal importance, the greater this probability, the more likely the two nodes are to be paired.

[0037] Specifically, the initial state of the coupling matrix is ​​determined by the initial state of the matching score at each position, and the initial state of the matching score is determined by the weights of the nodes. Therefore, in the initial state, each graph node needs to be weighted to satisfy... and The initial weight allocation can be configured as follows: It can be an average allocation, where all nodes are considered equally important by default; it can be a degree-centrality-based allocation method, incorporating structural importance—for example, nodes with more connections are considered structurally more important and given more weight; or it can be an attribute-based allocation method, where additional attribute information of a node implies importance and can be directly encoded into the weights. This embodiment of the invention does not limit the specific methods used.

[0038] Therefore, the initial state of the coupling matrix can be obtained through product coupling, where the dependency score at each position in the matrix is ​​obtained by multiplying the weights of nodes in the source domain and the target domain; it can also be obtained using the K-Means method, which first clusters the nodes in both spaces to obtain multiple clusters, calculates the total weight of the clusters, matches them according to the clusters, and then refines the result to each point; it can also be obtained using a random initialization method, which randomly assigns a number between 0 and 1 to the value at each position in the matrix; or it can be obtained using a locally linear method, which calculates a better initial state using a simplified formula in a small neighborhood of the initial prediction. The embodiments of this invention do not limit this approach.

[0039] It should be noted that the semantic correlation between the text tokens and visual tokens required for the second matrix can also be constructed by extracting cross-attention layers in the MLLM model.

[0040] Step 103: Determine the first matrix as the first distance matrix in the GW objective function, determine the second matrix as the coupling matrix in the GW objective function, perform a minimization operation on the GW objective function, and obtain the second distance matrix.

[0041] In this embodiment of the invention, the specific operations and meanings of determining the first matrix as the first distance matrix and the second matrix as the second distance matrix have been described in the above embodiments, and will not be repeated here. Performing a minimization operation on the GW objective function is a process of iteratively updating the coupling matrix until the optimal coupling matrix is ​​reached. Minimization is used to constrain the function to achieve the most suitable match with the minimum structural cost. Specifically, the methods for solving the GW distance can include conditional gradient method, proximal gradient method, entropy regularization and Sinkhorn algorithm, and MM algorithm (Majorization-Minimization). This embodiment of the invention does not limit the process of solving the GW distance.

[0042] It should be noted that, based on the aforementioned method for solving the GW distance, this embodiment of the invention innovatively employs a reverse solution method. The original GW distance solution fixes two distance matrices and iteratively updates to obtain the optimal coupling matrix. This embodiment aims to use the relational geometry in the language space as a reference to deduce the ideal relational geometry that the visual space should possess, and use this as a regularization objective to constrain training. Specifically, through matrix multiplication and element-wise division, the GW distance with a fixed distance matrix and coupling matrix is ​​efficiently calculated, finding the closed-form solution to this inverse geometry problem, which is the second distance matrix. This matrix can represent, for example, the optimal visual representation generated from a text using the GW distance, i.e., the corresponding image.

[0043] Step 104: Optimize the preset MLLM model based on the second distance matrix to obtain the optimized MLLM model.

[0044] In this embodiment of the invention, the method of optimizing the model based on the matrix may include: I. Data-level optimization: Using this matrix as a reference, oversample, weight, or augment the data related to the matrix, or during the model training process, make the model use this matrix as an anchor point to perform a series of comparative learning.

[0045] 2. Loss function level optimization: Based on the matrix, change the optimization direction of the model, use focus loss to automatically increase the gradient contribution corresponding to the matrix, and make the model focus on it; or based on the example regularization term, add an extra term after the original loss function to penalize the model's performance on the data corresponding to this matrix, and force the model to increase its confidence in it.

[0046] 3. Model structure optimization: Assign a preferred label to the matrix and store it as external knowledge in the memory module. The model will prioritize retrieving and referencing this memory during inference. Alternatively, train a lightweight adapter module for this matrix and directly adjust the embedding of the last layer of the model. This can obtain a better representation for the matrix without affecting the overall model structure.

[0047] It should be noted that the MLLM model can repeatedly obtain multiple second distance matrices as references based on the method in the embodiments of the present invention, construct the corresponding sample set, and then combine weighted loss or fine-tuning to optimize the model so that the model reaches the optimization goal. That is, the optimized MLLM model is used to generate visual reasoning results based on images and text.

[0048] In existing technologies, a large language model can correctly infer the tool-function relationship of "a hammer is used for striking, and scissors are used for cutting"; however, the same multimodal model, upon seeing images of a hammer and paper, might incorrectly answer "a hammer can be used to cut paper." Similarly, a language model may know that "Huskies and Alaskan Malamutes differ in ear shape and body proportions," but a multimodal model may still visually confuse the two breeds. This raises a fundamental problem: semantic relationships (such as spatial configuration, logical dependencies, and functional associations) already encoded in a language model cannot be transferred to visual relationship understanding.

[0049] Specifically, the training objective described above focuses on whether each visual token can independently carry the correct semantic information; in other words, it performs "identity-level alignment"—ensuring that each concept itself is correct. However, it does not explicitly constrain whether the relationship structure between multiple visual tokens is consistent with the relationship structure between corresponding concepts in the language space. Even if each concept is correctly identified, the pairwise relationships between them may still be lost during cross-modal transmission.

[0050] Most existing cross-modal alignment methods focus on token-level or identity-level semantic mapping, lacking a method for explicit transfer and constraint of relation-level geometry. While GW distance is a classic tool for comparing the structural differences between two metric spaces, the standard GW problem requires simultaneous optimization of the coupling matrix and alignment of two geometries, resulting in high computational costs. More importantly, in MLLM, cross-attention mechanisms naturally provide a soft correspondence between text tokens and visual tokens—what we lack is not a correspondence, but a precise specification, namely, what relational structure the visual space should present given the linguistic geometry and this correspondence.

[0051] Therefore, the core idea of ​​the method proposed in this invention is to directly utilize the relationship between text tokens and visual tokens as a coupling matrix, use the relational geometry in the language space as the teacher structure, deduce the ideal relational geometry that the visual space should possess, and use this as a regularization objective to constrain training. This method proves that this inverse geometry problem has a unique closed-form solution, which can be efficiently computed through matrix multiplication and element-wise division.

[0052] Existing MLLM training objectives only guarantee the semantic correctness of each concept, but do not guarantee the consistency of relations between concepts. This method aims to propose a way to explicitly transfer relational priors from the language side to the visual side, enabling multimodal models to achieve better performance on tasks such as relational reasoning, spatial understanding, and fine-grained discrimination, without compromising general visual-language capabilities or increasing the computational burden during reasoning.

[0053] As can be seen, this invention creatively treats text tokens and visual tokens as basic elements and applies them to the GW distance. It optimizes the MLLM model using a reverse GW solution approach to obtain ideal visual reasoning results with textual relationships as prior knowledge. First, a first matrix is ​​constructed using the semantic correlation between text tokens as elements, and a second matrix is ​​constructed using the semantic correlation between text tokens and visual tokens as elements. The first matrix serves as the first distance matrix in the GW distance, and the second matrix serves as the coupling matrix. Then, fixing the two matrices, the second distance matrix is ​​derived in reverse using the GW objective function minimization algorithm. Finally, the MLLM model is optimized to improve the accuracy of transferring semantic relationship understanding to visual relationship understanding, thereby making the results obtained by the MLLM model for visual reasoning more accurate.

[0054] In an optional embodiment, the method for obtaining the optimized MLLM model may include: Construct a third matrix; The difference parameters are obtained based on the second and third distance matrices; Based on the difference parameters, the loss function corresponding to the preset MLLM model is optimized to obtain an optimized loss function. The optimized loss function is used to constrain the training process of the MLLM model in order to obtain an optimized MLLM model.

[0055] In this optional embodiment, the elements in the third matrix are used to represent the dependency scores between different visual tokens. The third matrix is ​​constructed in the same way as the first and second matrices. Taking the matrix coordinate (k, l) as an example, the dependency score at this coordinate is used to represent the semantic relevance between the k-th visual token and the l-th visual token.

[0056] In this optional embodiment, the difference parameter is used to measure the difference between the second distance matrix and the third matrix. The second distance matrix defaults to using the same coordinate system as the third matrix. Therefore, on the same matrix coordinate system, the second distance matrix represents the visual semantic relationship derived from the textual semantic relationship using the GW distance algorithm, while the third matrix represents the semantic relationship directly obtained from the visual token based on image encoding and mapping information. Since there is a discrepancy between the two types of information, the difference parameter preserves this discrepancy and optimizes the loss function so that after training with this parameter, the model can approximate the inference result of the textual semantic relationship as closely as possible when inferring visual relationships based on new images.

[0057] It should be noted that the third matrix can be the result obtained by encoding the original image and performing corresponding analysis through the ViT module in the MLLM model, and this embodiment of the invention does not limit this.

[0058] As can be seen, this optional embodiment optimizes the loss function and the training process of the MLLM model by obtaining the difference between the second distance matrix and the third matrix, so that the model can obtain more accurate visual reasoning results.

[0059] In another alternative embodiment, the second matrix is ​​constructed in the following ways: Input the text token and visual token into the preset MLLM model to obtain the attention distribution corresponding to each attention head in the MLLM model. For each attention head: obtain the information entropy based on the attention distribution. Sort the information entropy corresponding to all attention heads in ascending order, and determine the attention heads whose sorting results are below a preset first threshold as candidate attention heads; aggregate the attention scores corresponding to all candidate attention heads to construct a second matrix.

[0060] In this alternative embodiment, comparing the geometry of two different spaces requires a coupling to establish a correspondence between tokens. For example, in MLLM, text-to-visual cross-attention naturally provides this soft correspondence. If the mutual attention between two tokens is high, it indicates that they are semantically close; if the mutual attention is low, it indicates that they are semantically far apart. Suppose that the multi-head cross-attention of a certain intermediate layer of a large language model generates H attention maps, where the (i,j) element of each attention map represents the attention score of the i-th text token to the j-th visual token.

[0061] Therefore, the second matrix can be achieved by aggregating multiple attention maps corresponding to attention heads in the MLLM model. However, existing research shows that the importance of attention heads is highly uneven: many heads produce a nearly uniform distribution, and simply averaging all heads will dilute those sharp correspondences that are truly informative.

[0062] In this optional embodiment, information entropy is used to measure the concentration and dispersion of the attention head in the attention mode. The lower the entropy, the more concentrated the attention of the head and the stronger its discrimination ability. The intermediate layer achieves a good balance between entropy and peak attention, while the head aggregation strategy based on low entropy selection produces the clearest cross-modal correspondence. Therefore, this optional embodiment sorts the information entropy in ascending order and selects low-entropy attention heads according to a preset first threshold. Preferably, the aggregation process can be carried out by averaging the scores of each head to obtain the elements on each matrix coordinate.

[0063] As can be seen, this optional embodiment can directly extract semantic information between text tokens and visual tokens through the cross-attention layer of the MLLM model, which simplifies the method used in the above embodiments. At the same time, it only selects low-entropy attention heads to simplify the computational cost.

[0064] In yet another alternative embodiment, before performing the minimization operation on the GW objective function to obtain the second distance matrix, the method may further include: Based on the second matrix, the first suppression parameter is obtained. The first suppression parameter is used to measure the coupling quality of each text token. The coupling quality is obtained based on all the elements corresponding to the text token in the second matrix. The encoded sequence corresponding to the text token is determined to be the first sequence; The second suppression parameter is obtained based on the position of the text token in the first sequence; Based on the first suppression parameter and the second suppression parameter, the weight parameter is obtained. The weight parameter is used to perform weighted optimization on the elements in the second matrix to obtain the optimized second matrix. The optimized second matrix is ​​determined to be the coupling matrix in the GW objective function.

[0065] In this optional embodiment, the first suppression parameter is used to measure the coupling quality of each text token. That is, based on the coupling score between each text token and all visual tokens in the second matrix, the coupling quality of the text token is determined. Specifically, the method for determining the coupling quality may include: filtering based on the global total coupling degree, i.e., the total dependency score, where a higher score indicates a more active node in the matching process; observing whether the distribution of coupling scores is concentrated on a single node or a few nodes; or using local coupling degree, which is not limited in this optional embodiment.

[0066] In this optional embodiment, the first sequence is the sequence obtained after the text segmenter transforms the original text into words. This sequence contains multiple text tokens with a specific order. Therefore, the importance of a text token in the original text segment can be determined based on its position in the first sequence. For example, system tokens are usually in the early part of the sequence. It should be noted that the purpose of setting the first and second suppression parameters is to filter out tokens in the text stream that do not carry visual semantics, such as system prompts, image placeholders, and formatting tags. These tokens occupy positions in the matrix but do not contribute to the relational structure; instead, they introduce noise.

[0067] In this optional embodiment, a weighted optimization approach combining two suppression parameters can achieve soft suppression rather than hard deletion of such tokens. The weight parameters are derived based on the quality of this cross-modal coupling and its position in the sequence. The weighted optimization may include weighting and then normalizing the dependency score. This preserves potentially useful contextual information while reducing the interference of non-semantic tokens on the distance structure.

[0068] In yet another optional embodiment, the MLLM model includes multiple attention layers, which are used to analyze the dependencies between different tokens in a hierarchical order. The method may also include: The first self-attention layer, the second self-attention layer, and the cross-attention layer are defined. The first self-attention layer is used to analyze the dependency relationship between different text tokens. The second self-attention layer is used to analyze the dependency relationship between different visual tokens. The cross-attention layer is used to analyze the dependency relationship between text tokens and visual tokens. Furthermore, the construction methods of the first, second, and third matrices also include: Input the text token and visual token into the MLLM model; Based on the analysis results of the first layer in all the first self-attention layers, construct the first matrix; Based on the analysis results of the last layer in all second self-attention layers, a third matrix is ​​constructed; A second matrix is ​​constructed based on the analysis results of all intermediate layers across the attention layers.

[0069] In this optional embodiment, the attention layer is the Transformer layer in the MLLM model. This layer can be a class or module directly implemented in the code. Each Transformer layer can include three sub-layers: a first self-attention layer, which calculates self-attention only between text tokens; a second self-attention layer, which calculates self-attention only between visual tokens; and a cross-attention layer, which uses text tokens as Q (Query sequence) and visual tokens as K and V (Key-Value sequence), or vice versa, to achieve cross-modal alignment. These three sub-layers are executed sequentially and can have residual connections between them.

[0070] Among them, the bottom layer, namely the first and second layers, are close to the input end, and the attention head mainly focuses on local, short-distance, and low-level patterns. Based on the analysis results of the first layer among all the first self-attention layers, the purpose of constructing the first matrix is ​​to focus on the collocation between adjacent words, and the text relationship is relatively pure. The middle layer begins to capture combination patterns, and the representation achieves a balance between semantic richness and concentration, so it is suitable for the cross-modal relationship of the second matrix. The high-level and last few layers focus on global semantics and long-distance dependencies, and are suitable for the overall features of objects in the image.

[0071] It should be noted that in the decoder architecture of the MLLM model, text self-attention and cross-modal attention can be submatrices of a causal attention map, jointly normalized by the same softmax. If both are extracted from the same layer, their gradients will be coupled together, leading to instability during training. Therefore, this alternative embodiment employs this extraction method, which also avoids this situation.

[0072] As can be seen, this optional embodiment can use a hierarchical attention extraction method to enable each matrix to find its own suitable relational representation, thereby further refining the model training and optimization process.

[0073] In yet another alternative embodiment, the first matrix The calculation method may include:

[0074] In the above formula, For the initial first matrix, The adjustment parameters are used to stabilize the calculation results, wherein the initial first matrix is ​​obtained based on the analysis results of the first layer among all first self-attention layers.

[0075] In this alternative embodiment, since the raw attention scores are typically concentrated in a very narrow range close to zero, most token pairs would be mapped to an indistinguishable region if used directly. Therefore, a negative logarithmic transformation is applied to amplify the differences. The value >0 prevents numerical instability. After this transformation, semantically related token pairs correspond to smaller distances, while irrelevant token pairs correspond to larger distances. Before calculation, the first and third matrices are symmetricized to reduce the impact of unidirectional attention fluctuations.

[0076] In yet another optional embodiment, the second distance matrix The calculation methods include:

[0077] In the above formula, Let be the coupling matrix. , The index for the text token. , is the edge vector of the coupling matrix.

[0078] In this optional embodiment, the closed-form solution has a very intuitive explanation: This is equivalent to "focusing on each view token separately". and "The expected text distance between those text tokens." Specifically, The description is: if all relational structures in a language are faithfully inherited into the visual space through cross-attention, the distance structure that should be presented between visual tokens.

[0079] To facilitate the implementation of the technical solutions in the above embodiments, this optional embodiment provides examples and explanations of the application of the above embodiments: This alternative embodiment addresses the problem of transferring existing concept-to-concept relation structures from a language model to a visual representation. In the current MLLM architecture, an image is processed by a visual encoder to generate a set of visual tokens. These tokens are then mapped to the language space via a projection adapter and fed into a large language model along with text tokens. The training loss is a standard autoregressive language modeling objective. This objective encourages each visual token to independently carry the correct semantics, but does not constrain whether the pairwise relationships between visual tokens are consistent with the pairwise relationships of corresponding concepts in the language space.

[0080] The basic idea behind this optional embodiment is to leverage two observations: first, language representations already contain relatively stable relational structures; second, cross-attention naturally provides soft correspondences between text tokens and visual tokens. Based on this, an ideal visual relational geometry can be derived from linguistic geometry and cross-modal coupling, and then this ideal geometry can be used to constrain training. The overall training objective is:

[0081] in The overall training objective is the optimized loss function. The loss function corresponding to the preset MLLM model. The difference parameters proposed in this optional embodiment, This is the weighting coefficient (set to 0.002 in the experiment). The following sections will explain in detail. The construction process.

[0082] In the context of MLLM, the standard GW problem has two limitations. First, the standard GW requires simultaneous optimization of the coupling matrix, which is computationally expensive (fourth-order complexity). Second, in MLLM, the cross-attention mechanism naturally provides a soft correspondence between text tokens and visual tokens—there is no need to actually "find" the coupling. What is truly lacking is: given the known coupling, what kind of relational structure should the visual space possess to maintain consistency with the relational structure of the language space?

[0083] Relational alignment first requires establishing a distance metric within each modality. Self-attention provides a natural proxy: if the mutual attention between two tokens is high, they are semantically close; if the mutual attention is low, they are semantically far apart.

[0084] The complete training process is as follows: For each mini-batch of data, standard forward propagation is first performed using ViT and LLM. Then, data is extracted from different layers. , and multiple attention spans, The self-attention matrix (size) extracted from a certain layer of the text branch of a large language model × ), The self-attention matrix extracted from the last layer of the visual encoder (size) × ),right and Perform symmetry transformation. Next, calculate the entropy of each head, and select the low-entropy heads to aggregate into a coupling matrix C. Non-semantic tokens in the dataset are subjected to soft suppression. Then, a negative logarithmic transformation is applied to obtain... and Calculate the target geometry using closed-form solutions:

[0085] Final calculation The parameters are updated by backpropagation after being weighted and summed with the language modeling loss.

[0086] It should be noted that the entire The computations utilize the attention matrix already generated during forward propagation, without introducing any additional parameters. These computations are performed only during training—during inference, the model architecture and computational overhead are exactly the same as the baseline MLLM.

[0087] In the experiment, The weight λ was set to 0.002, the number of head selections k was set to 8, and the non-semantic token decay strength α was set to 0.5. Experiments were conducted on two model families, LLaVA-1.5 and LLaVA-NeXT, covering 7B and 13B scales respectively. Fine-tuning was performed for one epoch on eight NVIDIA RTX 6000 Ada GPUs (48GB), following the original training protocols of each family.

[0088] The following section presents a detailed comparison of our method with the baseline model.

[0089]

[0090] Table 1 shows the performance of this optional implementation on the GQA structured relational reasoning task. GQA is a large-scale scene graph reasoning dataset that requires multi-hop reasoning and spatial relation understanding, encompassing five semantic categories: Attr., Category., Global., Object., and Relation. It can be observed that our method achieves stable overall improvements across all four model configurations. The greatest improvement is seen in the Global task (+1.04% to +3.16%), which requires integrating relationships between multiple objects (e.g., "How many red objects are on the table?"). Significant improvements are also observed in the Category task (involving fine-grained category discrimination). The Object and Attribute tasks show only modest improvements, which is expected—these tasks rely more on local visual features than on the relational structure between concepts.

[0091]

[0092] Table 2 illustrates the performance on the general visual language task. On VQAv2, LLaVA-1.5-7B improved by +1.8 points, while LLaVA-NeXT remained stable. On POPE (Phantom Evaluation), all configurations showed improvements (+0.3 to +0.6), indicating that a better cross-modal distance structure helps reduce object illusions. Overall, the enhanced relational structure did not come at the expense of generality.

[0093]

[0094] Table 3 shows the component ablation results. Starting from the baseline, the gradual addition of layer decoupling, low-entropy head selection, and non-semantic token filtering resulted in a cumulative improvement of +0.7 on GQA and +2.4 on BLINK. Each component contributed to the gain, indicating that the three components are complementary, targeting different noise sources.

[0095] Example 2 Please see Figure 2 , Figure 2This is a schematic diagram of the structure of a visual inference device based on a GW distance-improved MLLM model disclosed in an embodiment of the present invention. Figure 2 As shown, the visual inference device based on the GW distance-improved MLLM model may include: The unit acquisition module 201 is used to acquire text tokens and visual tokens. The text token is a unit obtained after embedding the original text into words to represent the semantic features of word segmentation, and the visual token is a unit obtained after encoding the original image to represent the region features. Matrix construction module 202 is used to construct a first matrix, which represents the dependency score between different text tokens, and the dependency score represents the semantic relevance between different tokens; and to construct a second matrix, which represents the dependency score between text tokens and visual tokens. The function derivation module 203 is used to determine that the first matrix is ​​the first distance matrix in the GW objective function, determine that the second matrix is ​​the coupling matrix in the GW objective function, perform a minimization operation on the GW objective function, and obtain the second distance matrix. The model optimization module 204 is used to optimize the preset MLLM model according to the second distance matrix to obtain the optimized MLLM model. The optimized MLLM model is used to generate visual reasoning results based on images and text.

[0096] Example 3 Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a device including a memory and a processor, as disclosed in an embodiment of the present invention. Figure 3 As shown, the device may include: Memory 301 storing executable program code; Processor 302 coupled to memory 301; The processor 302 calls the executable program code stored in the memory 301 to execute some or all of the steps in any of the visual reasoning methods based on the GW distance improved MLLM model in Embodiment 1 of the present invention.

[0097] Example 4 This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked by a processor, they are used to execute some or all of the steps in any of the visual reasoning methods based on the GW distance-improved MLLM model disclosed in Embodiment 1 of this invention.

[0098] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0099] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0100] Finally, it should be noted that the above embodiments are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual reasoning method based on GW distance improved MLLM model, characterized in that, The method includes: Obtain a text token and a visual token. The text token is a unit used to represent the semantic features of word segmentation after encoding the original text, and the visual token is a unit used to represent the region features after encoding the original image. Construct a first matrix to represent the dependency score between different text tokens, whereby the dependency score represents the degree of semantic relevance between different tokens; Construct a second matrix to represent the dependency score between the text token and the visual token; The first matrix is ​​determined to be the first distance matrix in the GW objective function, and the second matrix is ​​determined to be the coupling matrix in the GW objective function. A minimization operation is performed on the GW objective function to obtain the second distance matrix. Based on the second distance matrix, the preset MLLM model is optimized to obtain an optimized MLLM model, which is used to generate visual reasoning results based on images and text.

2. The visual reasoning method based on the GW distance-improved MLLM model according to claim 1, characterized in that, The methods for obtaining the optimized MLLM model include: Construct a third matrix, the elements of which are used to represent the dependency scores between different visual tokens; Based on the second distance matrix and the third matrix, a difference parameter is obtained, which is used to measure the difference between the second distance matrix and the third matrix; Based on the difference parameters, the loss function corresponding to the preset MLLM model is optimized to obtain an optimized loss function. The optimized loss function is used to constrain the training process of the MLLM model to obtain an optimized MLLM model.

3. The visual reasoning method based on the GW distance-improved MLLM model according to claim 2, characterized in that, The second matrix is ​​constructed in the following ways: Input the text token and the visual token into a preset MLLM model to obtain the attention distribution corresponding to each attention head in the MLLM model. For each attention head: obtain the information entropy based on the attention distribution. Sort the information entropy corresponding to all attention heads in ascending order, and determine the attention heads whose sorting results are before a preset first threshold as candidate attention heads; aggregate the attention scores corresponding to all candidate attention heads to construct a second matrix.

4. The visual reasoning method based on the GW distance-improved MLLM model according to claim 3, characterized in that, Before performing the minimization operation on the GW objective function to obtain the second distance matrix, the method further includes: Based on the second matrix, a first suppression parameter is obtained. The first suppression parameter is used to measure the coupling quality of each text token. The coupling quality is obtained based on all elements corresponding to the text token in the second matrix. The encoded sequence corresponding to the text token is determined to be the first sequence; The second suppression parameter is obtained based on the position of the text token in the first sequence; Based on the first suppression parameter and the second suppression parameter, a weight parameter is obtained. The weight parameter is used to perform weighted optimization on the elements in the second matrix to obtain an optimized second matrix. The optimized second matrix is ​​determined to be the coupling matrix in the GW objective function.

5. The visual reasoning method based on the GW distance-improved MLLM model according to claim 4, characterized in that, The MLLM model includes multiple attention layers, which are used to analyze the dependencies between different tokens in a hierarchical order. The method also includes: A first self-attention layer, a second self-attention layer, and a cross-attention layer are defined. The first self-attention layer is used to analyze the dependency relationship between different text tokens. The second self-attention layer is used to analyze the dependency relationship between different visual tokens. The cross-attention layer is used to analyze the dependency relationship between the text token and the visual token. Furthermore, the construction methods of the first matrix, the second matrix, and the third matrix also include: Input the text token and the visual token into the MLLM model; Based on the analysis results of the first layer in all the first self-attention layers, construct the first matrix; Based on the analysis results of the last layer in all second self-attention layers, a third matrix is ​​constructed; A second matrix is ​​constructed based on the analysis results of all intermediate layers across the attention layer.

6. The visual reasoning method based on the GW distance-improved MLLM model according to claim 5, characterized in that, The first matrix The calculation methods include: In the above formula, For the initial first matrix, The adjustment parameters are used to stabilize the calculation results, wherein the initial first matrix is ​​obtained based on the analysis results of the first layer among all first self-attention layers.

7. The visual reasoning method based on the GW distance-improved MLLM model according to claim 4, characterized in that, Second distance matrix The calculation methods include: In the above formula, Let be the coupling matrix. , The index for the text token. , is the edge vector of the coupling matrix.

8. A visual reasoning device based on an improved MLLM model using GW distance, characterized in that, The device includes: The unit acquisition module is used to acquire text tokens and visual tokens. The text tokens are units obtained after embedding the original text into words to represent the semantic features of word segmentation, and the visual tokens are units obtained after encoding the original image to represent the region features. A matrix construction module is used to construct a first matrix, which represents the dependency score between different text tokens, and the dependency score represents the semantic relevance between different tokens; and to construct a second matrix, which represents the dependency score between the text token and the visual token. The function derivation module is used to determine that the first matrix is ​​the first distance matrix in the GW objective function, determine that the second matrix is ​​the coupling matrix in the GW objective function, perform a minimization operation on the GW objective function, and obtain the second distance matrix. The model optimization module is used to optimize the preset MLLM model according to the second distance matrix to obtain an optimized MLLM model. The optimized MLLM model is used to generate visual reasoning results based on images and text.

9. An apparatus comprising a memory and a processor, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the visual reasoning method based on the GW distance improved MLLM model as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked by the processor, execute the visual reasoning method based on the GW distance-improved MLLM model as described in any one of claims 1-7.

Images