Multimodal large language model visual understanding enhancement method and system based on perception-cognition alignment

Abstract

The application discloses a kind of based on perception-cognition alignment multimodal large language model visual understanding enhancement method and system, in the multimodal large language model Transformer decoding layer to be enhanced according to fixed interval preset trigger layer. By implementing two-stage probe in inference stage: first, the model is guided to capture deep cognitive attention graph using cognitive-oriented prefix, then the model is guided to capture surface perception attention graph using perception-oriented prefix. When the subsequent normal inference without prefix encounters trigger layer, a soft mask is generated based on the difference between the two attention graphs, and the modulation strength is determined according to the cross-modal redundancy of image and text tokens. The hidden state of image tokens is dynamically updated through a joint residual modulation mechanism. The application effectively compensates for the loss of low-level visual details during high-level semantic reasoning in the model, aligns perception and cognition, and significantly improves the performance of multimodal large models in complex visual understanding tasks.

Description

Technical Field

[0001] This invention relates to the field of multimodal large language model and visual understanding technology, and in particular to a method and system for enhancing visual understanding of multimodal large language models based on perception and cognition. Background Technology

[0002] In recent years, multimodal large language models (MLLMs) have made significant progress in tasks such as visual question answering and image captioning. However, existing models generally suffer from a fragmentation phenomenon: while possessing strong low-level visual perception capabilities and accurately extracting local features, they often struggle to transform fragmented visual features into logically consistent internal representations when faced with tasks requiring high-level semantic integration. This deficiency makes the models prone to bias when dealing with scenarios involving complex reasoning. In other words, perfect statistical alignment across modalities on first-order physical features does not spontaneously generate true reasoning ability; "seeing ≠ understanding."

[0003] To address the weaknesses of multimodal large language models in visual understanding, numerous evaluation benchmarks have emerged in the industry, such as MMBEC, MMMU, II-Bench, MVP-Bench, BLINK, DeepEval, and PCA-Bench. However, these benchmarks are somewhat vague in their delineation of the boundary between perception and cognition, making it difficult to independently quantify the model's failure rate in true "understanding." Meanwhile, existing reasoning enhancement capabilities primarily rely on explicit chain-of-thought (CoT) cues or pure text-based reasoning models on the language side (i.e., "thinking" variants). It should be noted, however, that pure text-based CoTs have limited effectiveness in mitigating the cross-modal semantic gap and may even introduce language illusions and exacerbate overthinking problems.

[0004] Furthermore, existing alignment methods during the training phase (such as reinforcement learning based on human feedback and supervised fine-tuning) while improving the cognitive abilities of multimodal large language models to some extent come with high computational costs and the need for massive amounts of labeled corpora. More importantly, these methods require retraining with each model iteration, resulting in extremely high generalization costs. Therefore, how to structurally intervene in the internal visual representation of multimodal large language models without modifying any model parameters or introducing additional training data, thereby narrowing the perception-cognition gap during the inference phase, has become a crucial research topic with significant industrial implications. Summary of the Invention

[0005] This invention aims to address the problem of "seeing ≠ understanding" and the significant gap between perception and cognition in multimodal large language models. It overcomes the shortcomings of existing methods that rely on parameter fine-tuning and provides a method and system for enhancing visual understanding of multimodal large language models based on perception-cognition alignment. By implementing a two-stage probe under the guidance of cognitive-guided prefixes and basic perception-guided prefixes respectively, it captures the attention distribution within the model. Furthermore, it jointly modulates the hidden state of image tokens on the Transformer decoding trigger layer using soft masks and cross-modal redundancy penalties. Without introducing additional training parameters, it significantly improves the high-level cognitive reasoning ability of the model.

[0006] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, the present invention implements a method for enhancing visual understanding of a multimodal large language model based on perception-cognition alignment, comprising the following steps: Step 1: Construct a hierarchical visual understanding evaluation benchmark and select the multimodal large language model to be enhanced as the backbone network. The hierarchical visual understanding evaluation benchmark divides the perceptual task and cognitive task into three levels: low, medium and high. Each multimodal sample corresponds to six progressive questions, covering the entire process from low-level visual signal recognition to high-level causal and social relationship reasoning. Step 2: In the Transformer decoding layer of the backbone network, a number of trigger layers are preset at fixed intervals to dynamically intervene in the hidden state of the image token during the inference phase. Step 3: Enter the cognitive probe stage. In the cognitive probe stage, a cognitive guidance prefix is ​​appended before the user's original question to guide the multimodal large language model to be enhanced to associate potential causal and logical cues during the visual encoding process, and to capture the cognitive attention map at the trigger layer. ; Step 4: Enter the perception probe stage, replace the cognitive guidance prefix with the perception guidance prefix, and reconstrain the attention of the multimodal large language model to be enhanced to pixel-level and object-level surface features, capturing the basic perception attention map at the same trigger layer. ; Step 5: Enter the enhanced inference phase. Input the original user question without prefixes into the model. When the forward propagation of the network reaches any of the aforementioned triggering layers, activate the perception-cognition alignment mechanism based on the cognitive attention map. With the aforementioned basic perceptual attention map The difference is used to generate a soft mask m for adjusting the attention distribution, and a cross-modal redundancy score is generated based on the cross-modal redundancy between image tokens and text tokens. Then, based on the redundancy score, the modulation strength used to suppress redundant information is determined, and finally the hidden state of the image token is updated by joint residual modulation to obtain the enhanced image token representation. Step 6: Input the image token representation enhanced by perception-cognition alignment into the remaining network layers after the trigger layer to participate in the subsequent reasoning and decoding process, and output the final cognitive reasoning result.

[0007] In some implementations, the process of constructing a graded visual understanding assessment benchmark in step S1 includes: Image and text samples are crawled from multiple internet communities, and the samples meet a preset social activity threshold; The perception task is divided into three levels: low, medium, and high, along the semantic granularity axis. Cognitive tasks are divided into three levels—low, medium, and high—along a causal ladder. For each sample, a set of progressive questions is designed, covering the entire process from low-level visual signal recognition to high-level causal and social relationship reasoning.

[0008] In some implementations, the process of generating the soft mask m in step 5 includes: Calculate the cognitive attention map With the aforementioned basic perceptual attention map The difference is used to obtain the attention difference; Apply the ReLU activation function to the attentional differences to filter out negative cognitive biases; The result after ReLU activation is divided by a preset absolute threshold, and the result is restricted to between 0 and 1 to obtain the soft mask m.

[0009] In some implementations, the process of generating the cross-modal redundancy score ρ in step 5 includes: Calculate the maximum cosine similarity between the hidden state of each image token and the hidden states of all text tokens; The maximum cosine similarity is linearly mapped to the interval between 0 and 1 to obtain the cross-modal redundancy score. .

[0010] In some implementations, the process of calculating the redundancy penalty coefficient γ in step 5 includes: The redundancy penalty coefficient γ is calculated based on the cross-modal redundancy score ρ, wherein the redundancy penalty coefficient γ is inversely proportional to the cross-modal redundancy score ρ.

[0011] In some implementations, the joint residual modulation method in step 5 is as follows: The element-wise product of the soft mask m and the redundancy penalty coefficient γ is used as the modulation intensity; The modulation intensity is multiplied by the hidden state of the original image token to obtain the residual term; The residual term is added back to the hidden state of the original image token to obtain the enhanced image token representation.

[0012] In some implementations, the trigger layer interval in step 2 is an integer greater than or equal to 6 and less than or equal to 12.

[0013] In some implementations, a redundancy penalty coefficient is further set for the redundancy score in step 5. , where λ is the preset maximum magnification scale.

[0014] Secondly, this invention implements a multimodal large language model visual understanding enhancement system based on perception-cognition alignment, used to implement the method described in any one of the claims, including: The benchmark construction module is used to construct a hierarchical visual understanding evaluation benchmark, and selects the multimodal large language model to be enhanced as the backbone network. The hierarchical visual understanding evaluation benchmark divides the perceptual task and the cognitive task into three levels: low, medium and high. Each multimodal sample corresponds to six progressive questions, covering the entire process from low-level visual signal recognition to high-level causal and social relationship reasoning. The trigger layer preset module is used to preset a number of trigger layers at fixed intervals in the Transformer decoding layer of the backbone network, and to dynamically intervene in the hidden state of the image token during the inference phase. The cognitive probe module is used to enter the cognitive probe stage. In this stage, a cognitive guidance prefix is ​​appended before the user's original question to guide the multimodal large language model to be enhanced to associate potential causal and logical cues during visual encoding. The cognitive attention map is then captured at the trigger layer. ; The perception probe module is used to enter the perception probe stage, replace the cognitive guidance prefix with the perception guidance prefix, reconstrain the attention of the multimodal large language model to be enhanced to pixel-level and object-level surface features, and capture the basic perception attention map at the same trigger layer. ; The enhanced inference module is used to enter the enhanced inference phase. It inputs the original user question without prefixes into the model. When the network's forward propagation reaches any of the trigger layers, it initiates the perception-cognition alignment mechanism based on the cognitive attention graph. With the aforementioned basic perceptual attention map The difference is used to generate a soft mask m for adjusting the attention distribution, and a cross-modal redundancy score is generated based on the cross-modal redundancy between image tokens and text tokens. Then, based on the redundancy score, the modulation strength used to suppress redundant information is determined, and finally the hidden state of the image token is updated by joint residual modulation to obtain the enhanced image token representation. The output module is used to input the image token representation enhanced by perception-cognition alignment into the remaining network layers after the trigger layer, participate in the subsequent reasoning and decoding process, and output the final cognitive reasoning result.

[0015] Thirdly, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements any of the methods for enhancing visual understanding of multimodal large language models based on perception-cognition alignment.

[0016] Fourthly, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the visual understanding enhancement method for multimodal large language models based on perception-cognition alignment as described in any one of the claims.

[0017] Compared with the prior art, the present invention achieves the following significant technical effects: 1. A hierarchical evaluation benchmark with explicit decoupling of perception and cognition was constructed, enabling precise quantification of "understanding" ability: The hierarchical visual understanding evaluation benchmark constructed in this invention divides the perception task and the cognitive task into three levels—low, medium, and high—along semantic granularity and causal ladder, respectively. Each sample corresponds to six progressive questions, covering the entire process from pixel-level signal recognition to high-level causal reasoning. This benchmark can independently quantify the failure rate of multimodal large models in real "understanding," transforming "seeing ≠ understanding" from a philosophical proposition into a quantifiable empirical paradigm, providing a scientific basis for model diagnosis and improvement.

[0018] 2. By capturing attentional differences through a two-stage probe mechanism, interpretable separation of perception and cognition is achieved: This invention pre-defines a trigger layer in the Transformer decoding layer, extracting attention twice from the same image using both cognitive and perceptual probes. The visual token that truly carries the cognitive intent is then separated in an interpretable manner at the trigger layer, avoiding the linguistic illusions and overthinking problems caused by explicit thought chains (CoT). This method lays a reliable foundation for subsequent fine-tuning.

[0019] 3. By calculating the difference between the cognitive attention map and the perceptual attention map, and applying ReLU activation and absolute threshold scaling, a soft mask located in the [0,1] interval is generated. Compared with the traditional Min-Max normalization, this mechanism preserves the absolute magnitude information of cognitive bias and has significant robustness to outliers, thus providing a stable and interpretable assessment of cognitive confidence in image tokens.

[0020] 4. By calculating the maximum cosine similarity between image tokens and text tokens, a cross-modal redundancy score is generated, and a penalty coefficient inversely proportional to the redundancy is determined accordingly. This mechanism concentrates the limited amplification budget on non-redundant visual tokens that carry the true cognitive intent, achieving fine-grained semantic focus and effectively suppressing the interference of redundant visual information on high-level reasoning.

[0021] 5. A pure inference-stage intervention using joint residual modulation significantly improves cognitive reasoning ability without parameter fine-tuning: This invention uses the element-wise product of a soft mask and a redundancy penalty coefficient as the joint modulation intensity, updating the hidden state of the image token through residuals, and preserving the original representation as an unshakeable baseline. This method is a pure inference-stage intervention, requiring no modification to any model parameters and no introduction of additional training data. In a preferred embodiment, the absolute accuracy of Qwen3VL-8B on high-level cognitive tasks is improved by 7.8% (from 54.3% to 62.1%), significantly outperforming baseline solutions such as explicit thought chains, demonstrating high practical value and ease of deployment. Attached Figure Description

[0022] Figure 1 This is an overall flowchart of the visual understanding enhancement method for multimodal large language models based on perception-cognition alignment, which is an embodiment of the present invention.

[0023] Figure 2 This is a block diagram of the visual understanding enhancement system for a multimodal large language model based on perception-cognition alignment, according to Embodiment 2 of the present invention.

[0024] Figure 3 The diagram illustrates the Visual Room concept, demonstrating how a multimodal large language model can map visual symbols to language labels using only a statistical rule manual in the absence of semantic anchoring. Figure 4 A schematic diagram of the three-stage inference pipeline and trigger layer modulation for perception-cognition alignment (PCA); Figure 5 The results are experimental results comparing the performance of this invention with several baselines at different cognitive difficulty levels on the Visual Room 2.0 benchmark. Detailed Implementation

[0025] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0026] like Figure 1As shown, this invention provides a method for enhancing visual understanding of a multimodal large language model based on perception-cognition alignment. The process includes the following specific steps: Step 1: Construct a hierarchical visual understanding evaluation benchmark, selecting the multimodal large language model to be enhanced as the backbone network; the hierarchical visual understanding evaluation benchmark divides the perceptual task and cognitive task into three levels: low, medium, and high. Each multimodal sample corresponds to six progressive questions, covering the entire process from low-level visual signal recognition to high-level causal and social relationship reasoning; the specific construction method is as follows: Step 1.1: High-quality multimodal sample collection, i.e., targeted crawling of image and text samples from multiple internet communities rich in cognitive signals (such as r / sad, r / happy, r / humor, r / sarcasm). The screening strategy requires that each sample contains both images and text, and that the author has clearly labeled its emotional attributes, while also meeting the minimum social activity threshold (≥3 comments or ≥30 likes). In a preferred embodiment of the present invention, 1083 high-quality multimodal samples were ultimately collected.

[0027] Step 1.2: Perform granular layering of perceptual tasks, dividing perceptual tasks into three layers from low to high semantic granularity: the bottom layer covers low-level visual processing tasks such as attribute recognition and subgraph detection; the middle layer covers mid-level feature extraction tasks such as object detection, OCR, and scene classification; and the top layer covers high-level semantic capture tasks such as image description and scene understanding. This layered design enables a gradual transition of model capabilities from pixel-level raw signal processing to global scene understanding.

[0028] Step 1.3: Implement a causal ladder layering for cognitive tasks, dividing them into three layers along Pearl's causal ladder from low to high: the bottom layer covers explicit symbol alignment tasks such as textual entailment, text matching, and action recognition; the middle layer covers implicit semantic inference tasks such as emotion recognition, irony detection, and humor understanding; and the top layer covers deep logical deduction tasks such as common sense reasoning, causal reasoning, intent recognition, and social relationship reasoning. This layered design reflects the leap in model reasoning from surface symbol matching to deep implicit human-like reasoning.

[0029] Step 1.4: Construct a progressive assessment question bank, which involves designing six progressive questions for each sample, all of which are precisely annotated manually. The question types are as follows: five multiple-choice questions focusing on objective cognitive judgment; and one open-ended scenario description question, requiring a word count of no more than 30 words, and requiring a third-person perspective and emotional neutrality. Through this design, a complete assessment dimension is constructed, ranging from low-level perceptual response to high-level cognitive reasoning. Based on the total number of samples collected above, in the preferred embodiment, a final assessment question bank of 1083 × 6 = 6498 questions is constructed.

[0030] Step 2: In the Transformer decoding layer of the backbone network, a number of trigger layers are preset at fixed intervals to dynamically intervene in the hidden state of the image token during the inference phase. Specifically, a multimodal large language model to be enhanced is selected, and its total number of Transformer decoding layers is denoted as L. In the decoding layer sequence, at fixed intervals... In a preferred embodiment of the present invention, a plurality of trigger layers are uniformly preset as intervention nodes. Meanwhile, an absolute threshold τ and a maximum magnification scale λ are set as hyperparameters to regulate the intensity of intervention, so as to implement precise dynamic intervention on the hidden state of image tokens during the inference stage.

[0031] Step 3: Enter the cognitive probe stage. A cognitive guidance prefix is ​​appended before the user's original question to guide the multimodal large language model to be enhanced to associate potential causal and logical cues during visual encoding. A cognitive attention map is then captured at the trigger layer. ; Specifically, for each set of input samples (image, question), a specific cognitive guidance prefix is ​​appended before the user's original question, such as: "Please deeply analyze the logical clues, causal relationships, and implicit context in this image." This guides the multimodal large language model to be enhanced to actively associate potential causal and logical clues during the visual encoding process. Subsequently, the appended enhanced input is fed into the model for a forward propagation, and at each preset trigger layer, the high-dimensional attention distribution of the model to the image tokens is captured and recorded, which is recorded as a cognitive attention map. .

[0032] Step 4: Enter the perception probe stage, replace the cognitive guidance prefix with the perception guidance prefix, and reconstrain the attention of the multimodal large language model to be enhanced to pixel-level and object-level surface features, capturing the basic perception attention map at the same trigger layer. ; Specifically, the cognitive-guided prefix is ​​replaced with a basic perception-guided prefix (e.g., "Please objectively describe the visible physical features in the image: color, shape, objects, and layout") to reconstrain the model's attention to pixel-level and object-level surface features. The concatenated input is then fed into the multimodal large language model again to complete one forward propagation, and the basic perception attention map is captured at the exact same trigger layer position as in step 3. .

[0033] Step 5: Enter the enhanced inference phase. Input the original user question without prefixes into the model. When the forward propagation of the network reaches any of the aforementioned triggering layers, activate the perception-cognition alignment mechanism based on the cognitive attention map. With the aforementioned basic perceptual attention map The difference is used to generate a soft mask m for adjusting the attention distribution, and a cross-modal redundancy score is generated based on the cross-modal redundancy between image tokens and text tokens. Then, based on the redundancy score, the modulation strength used to suppress redundant information is determined, and finally the hidden state of the image token is updated by joint residual modulation to obtain the enhanced image token representation. Specifically, the original user question without any prefix is ​​input into the multimodal large language model. When the forward propagation reaches any triggering layer, the perception-cognition alignment mechanism is immediately activated: based on the information captured in steps 3 and 4... and Calculate the attention difference Δ; further generate a soft mask m and a redundancy penalty coefficient γ, and jointly modulate the hidden state of the image token in the current layer accordingly to obtain the enhanced image token representation.

[0034] The specific processing method for this step is as follows: Step 5.1: Construct a soft mask. For the i-th image token, calculate the difference between the cognitive attention map and the basic perceptual attention map:

[0035] in, For the attention difference map of the i-th image token, Let i be the cognitive attention graph of the i-th image token. Let i be the base perceptual attention map for the i-th image token.

[0036] right ReLU activation is applied to strictly filter negative offsets, and the ReLU results are linearly scaled by an absolute threshold τ. Finally, min(1, ReLU( ) / τ) as a soft mask Its value increases linearly with the intensity of cognitive shift in the interval [0,1], and saturates at τ: = min( 1 , ReLU( ) / τ ) in, Let be the soft mask value of the i-th image token, located in the interval [0,1], used to characterize the cognitive importance weight of the corresponding token. The closer the value is to 1, the more the token reflects deeper cognitive intent rather than surface perceptual features. ReLU( ) represents the modified linear unit activation function. Its role here is to non-linearly truncate the difference values, strictly filtering out negative values ​​(i.e., cases where the perceptual weight is higher than the cognitive weight), retaining only the positive cognitive bias signal. τ is the absolute threshold hyperparameter.

[0037] Compared to traditional Min-Max normalization, the absolute threshold scaling preserves the absolute magnitude information of cognitive bias and is significantly robust to outliers, thus providing a stable and interpretable assessment of cognitive confidence in image tokens.

[0038] Step 5.2: Construct cross-modal redundancy penalty. Let the hidden state vector of the image token be denoted as . The accompanying text token sequence is { | j=1,…, }. Calculate the maximum cosine similarity between the i-th image token and the text token sequence, and linearly map it to the interval [0,1] to obtain the redundancy score. :

[0039] in, Let be the redundancy score of the i-th image token, with a value range of [0,1]. This score characterizes whether the semantics of the image token have been explicitly covered by the text. The closer it is to 1, the higher the redundancy. The closer to 0, the greater the semantic gap. Let be the hidden state vector (i.e., image feature representation) of the i-th image token. Let be the hidden state vector (i.e., text feature representation) of the j-th token in the text sequence. The cosine similarity function is used to calculate the degree of similarity between two vectors (image token and text token) in semantic space. To perform the maximum value operation, iterate through all text tokens (j=1,…,L). t Find the value with the highest similarity to the current image token, and use this to measure the maximum overlap between the image token and the overall semantics of the text.

[0040] Then, the redundancy penalty coefficient is defined. , where λ is the preset maximum magnification scale.

[0041] Step 5.3, Joint residual modulation. Using a soft mask. With redundancy penalty coefficient The element-wise product of the values ​​is used as the final modulation intensity, and the hidden state of the image token is updated as follows:

[0042] Where ⊙ represents element-wise multiplication, the residual form preserves the original image token representation as an unshakeable baseline: for background noise ( →0) or highly redundant text ( For tokens with a negative error (→0), the residual increment naturally tends to zero; while for tokens with both strong cognitive bias and significant cross-modal semantic differences, they will be adaptively amplified, thus seamlessly integrating into the computation of subsequent Transformer layers.

[0043] Step 6: Input the image token representation enhanced by perception-cognition alignment into the remaining network layers after the trigger layer to participate in the subsequent reasoning and decoding process, and finally output the final cognitive reasoning result.

[0044] Specifically, the enhanced image token hidden states are fed into the remaining network layers after the trigger layer to participate in subsequent reasoning and decoding processes, outputting the final cognitive reasoning result. In a preferred evaluation embodiment, differentiated evaluation metrics are used for different question types: for multiple-choice questions, accuracy is used as the evaluation standard; for open-ended scenario description questions, a hybrid similarity metric (sim) is used. , S) = · + · A quantitative evaluation was conducted, and 0.75 was used as the threshold boundary for judging the accuracy of the prediction.

[0045] To further verify the alignment effect of this invention, a perceptual-cognitive gap (Gap) metric is introduced. Let N be the total number of perceptual questions in a single sample, and N be the number of correct answers. + The predicted cognitive answer is The true perceived answer is y. Based on this, the perceptual accuracy rate is defined by the following formulas: Accuracy of Conditioned Recognition : = 1{ N + / N ≥ δ} = P( = y | = 1 ) Where 1{.} is the indicator function, and δ is the threshold for determining perception accuracy. Therefore, the perception-cognition gap metric can be obtained: =1 This metric explicitly quantifies "the probability that cognitive reasoning still fails under conditions of perfect accuracy at the perceptual level," and is a core indicator for evaluating the actual gain of this invention.

[0046] Furthermore, by repeating steps 3 to 6, training-free perception-cognition aligned enhanced reasoning can be achieved for any sample to be evaluated without modifying any inherent parameters of the model.

[0047] Example 2: Figure 2 As shown, the multimodal large language model visual understanding enhancement system based on perception-cognition alignment according to the present invention is used to implement the method described in any of the above. The system includes a baseline construction module 100, a trigger layer preset module 200, a cognitive probe module 300, and: connected in sequence. The benchmark construction module 100 is used to construct a hierarchical visual understanding evaluation benchmark and selects the multimodal large language model to be enhanced as the backbone network. The hierarchical visual understanding evaluation benchmark divides the perceptual task and cognitive task into three levels: low, medium and high. Each multimodal sample corresponds to six progressive questions, covering the entire process from low-level visual signal recognition to high-level causal and social relationship reasoning. The trigger layer preset module 200 is used to preset a number of trigger layers at fixed intervals in the Transformer decoding layer of the backbone network, and to dynamically intervene in the hidden state of the image token during the inference phase. The cognitive probe module 300 is used to enter the cognitive probe stage. In this stage, a cognitive guidance prefix is ​​appended before the user's original question to guide the multimodal large language model to be enhanced to associate potential causal and logical cues during visual encoding. A cognitive attention map is then captured at the trigger layer. ; The perception probe module 400 is used to enter the perception probe stage, replace the cognitive guidance prefix with the perception guidance prefix, reconstrain the attention of the multimodal large language model to be enhanced to pixel-level and object-level surface features, and capture the basic perception attention map at the same trigger layer. ; The enhanced inference module 500 is used to enter the enhanced inference phase. It inputs the original user question without prefixes into the model. When the forward propagation of the network reaches any of the trigger layers, it initiates a perception-cognition alignment mechanism based on the cognitive attention map. With the aforementioned basic perceptual attention map The difference is used to generate a soft mask m for adjusting the attention distribution, and a cross-modal redundancy score is generated based on the cross-modal redundancy between image tokens and text tokens. Then, based on the redundancy score, the modulation strength used to suppress redundant information is determined, and finally the hidden state of the image token is updated by joint residual modulation to obtain the enhanced image token representation. The output module is used to input the image token representation enhanced by perception-cognition alignment into the remaining network layers after the trigger layer, participate in the subsequent reasoning and decoding process, and output the final cognitive reasoning result.

[0048] like Figure 3 The diagram shown illustrates the visual room argument.

[0049] like Figure 4 The diagram illustrates the three-stage inference pipeline and trigger layer modulation of Perception-Cognition Alignment (PCA), providing a detailed flow of the PCA three-stage inference pipeline. First, cognitive and perceptual attention matrices are obtained from the trigger layer using cognitive and perceptual probes, and their differences are calculated. Simultaneously, cross-modal similarity is calculated using textual information to determine weighting coefficients. Finally, a modulation signal is constructed using this difference and weighting coefficients to adaptively enhance the feature representation of the trigger layer.

[0050] In a preferred embodiment, for the Qwen 3 VL-8B backbone, under zero-sample baseline conditions, the high-level cognitive accuracy was 54.3%; after applying Explicit Thinking Chain (CoT) intervention, the accuracy only improved to 57.2%; however, after applying the perception-cognition alignment method proposed in this invention, the high-level cognitive accuracy improved to 62.1%, an absolute improvement of 7.8% relative to the zero-sample baseline and an absolute improvement of 4.9% relative to the explicit thinking chain; simultaneously, the mid-level cognitive accuracy improved from 60.6% to 65.7% (an improvement of 5.1%), and the low-level cognitive accuracy improved from 82.1% to 83.5% (an improvement of 1.4%). For the Qwen 3 VL-4B backbone, after applying the method of this invention, the high-level cognitive accuracy improved from 45.8% to 52.3% (an absolute improvement of 6.5%). The above results show that this invention can significantly enhance the high-level cognitive reasoning ability of multimodal large language models without any parameter fine-tuning.

[0051] Further ablation experiments showed that after removing the soft mask mechanism (i.e., retaining only the cross-modal redundancy penalty), the high-level cognitive accuracy dropped to 55.6%, a decrease of 6.5% compared to the complete method; after removing the cross-modal redundancy penalty (i.e., retaining only the soft mask), the high-level cognitive accuracy dropped to 57.9%, a decrease of 4.2% compared to the complete method. These results demonstrate that the soft mask and the cross-modal redundancy penalty have a significant synergistic effect and are both indispensable. Sensitivity analysis of the trigger layer spacing showed that when... When ∈[8,10], the present invention achieves optimal performance; when Too small (e.g.) When =2), it will excessively perturb the pre-trained structure, leading to a decrease in basic perception performance; when Too large (e.g.) If the threshold is ≥16, it is insufficient to form an effective cognitive intervention.

[0052] like Figure 5 As shown, the experimental results compare the performance of the present invention with several baselines at different cognitive difficulty levels on the Visual Room 2.0 benchmark. (a) Performance of the present invention (PLA) on the Visual Room 2.0 benchmark, based on the Qwen 3 VL-8B backbone network, compared with the zero-shot baseline and the CoT method. (b) Performance of the present invention (PLA) on the Visual Room 2.0 benchmark, based on the Qwen 3 VL-4B backbone network, compared with the zero-shot baseline and the CoT method. The horizontal axis represents the cognitive difficulty level (low, medium, high), and the vertical axis represents the performance score. The experimental results show that the performance of each method decreases as the cognitive difficulty increases; however, under all difficulty settings, the performance of the present invention (red curve) is significantly better than other baseline methods, especially showing stronger robustness and generalization ability at the Cognition (High) difficulty level.

[0053] Thirdly, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements any of the methods for enhancing visual understanding of multimodal large language models based on perception-cognition alignment.

[0054] Fourthly, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the visual understanding enhancement method for multimodal large language models based on perception-cognition alignment as described in any one of the claims.

[0055] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.

Claims

1. A method for enhancing visual understanding of a multimodal large language model based on perception-cognition alignment, characterized in that, Includes the following steps: Step 1: Construct a hierarchical visual understanding evaluation benchmark and select the multimodal large language model to be enhanced as the backbone network. The hierarchical visual understanding evaluation benchmark divides the perceptual task and cognitive task into three levels: low, medium and high. Each multimodal sample corresponds to six progressive questions, covering the entire process from low-level visual signal recognition to high-level causal and social relationship reasoning. Step 2: In the Transformer decoding layer of the backbone network, a number of trigger layers are preset at fixed intervals to dynamically intervene in the hidden state of the image token during the inference phase. Step 3: Enter the cognitive probe stage. A cognitive guidance prefix is ​​appended before the user's original question to guide the multimodal large language model to be enhanced to associate potential causal and logical cues during visual encoding. A cognitive attention map is then captured at the trigger layer. ; Step 4: Enter the perception probe stage, replace the cognitive guidance prefix with the perception guidance prefix, and reconstrain the attention of the multimodal large language model to be enhanced to pixel-level and object-level surface features, capturing the basic perception attention map at the same trigger layer. ; Step 5: Enter the enhanced inference phase. Input the original user question without prefixes into the model. When the forward propagation of the network reaches any of the aforementioned triggering layers, activate the perception-cognition alignment mechanism based on the cognitive attention map. With the aforementioned basic perceptual attention map The difference is used to generate a soft mask m for adjusting the attention distribution, and a cross-modal redundancy score is generated based on the cross-modal redundancy between image tokens and text tokens. Then, based on the redundancy score, the modulation strength used to suppress redundant information is determined, and finally the hidden state of the image token is updated by joint residual modulation to obtain the enhanced image token representation. Step 6: Input the image token representation enhanced by perception-cognition alignment into the remaining network layers after the triggering layer to participate in the subsequent reasoning and decoding process, and output the final cognitive reasoning result.

2. The method for enhancing visual understanding of a multimodal large language model based on perception-cognition alignment according to claim 1, characterized in that, The process of constructing the hierarchical visual understanding assessment benchmark in step 1 includes: Image and text samples are crawled from multiple internet communities, and the samples meet a preset social activity threshold; The perception task is divided into three levels: low, medium, and high, along the semantic granularity axis. Cognitive tasks are divided into three levels—low, medium, and high—along a causal ladder. For each sample, a set of progressive questions is designed, covering the entire process from low-level visual signal recognition to high-level causal and social relationship reasoning; The trigger layer interval in step 2 is an integer greater than or equal to 6 and less than or equal to 12.

3. The method for enhancing visual understanding of multimodal large language models based on perception-cognition alignment according to claim 1, characterized in that, The process of generating the soft mask m in step 5 includes: Calculate the cognitive attention map With the aforementioned basic perceptual attention map The difference is used to obtain the attention difference; Apply the ReLU activation function to the attentional differences to filter out negative cognitive biases; The result after ReLU activation is divided by a preset absolute threshold, and the result is restricted to between 0 and 1 to obtain the soft mask m.

4. The method for enhancing visual understanding of multimodal large language models based on perception-cognition alignment according to claim 1, characterized in that, The process of generating the cross-modal redundancy score ρ in step 5 includes: Calculate the maximum cosine similarity between the hidden state of each image token and the hidden states of all text tokens; The maximum cosine similarity is linearly mapped to the interval between 0 and 1 to obtain the cross-modal redundancy score. .

5. The method for enhancing visual understanding of a multimodal large language model based on perception-cognition alignment according to claim 1, characterized in that, The process of calculating the redundancy penalty coefficient γ in step 5 includes: The redundancy penalty coefficient γ is calculated based on the cross-modal redundancy score ρ, wherein the redundancy penalty coefficient γ is inversely proportional to the cross-modal redundancy score ρ.

6. The method for enhancing visual understanding of multimodal large language models based on perception-cognition alignment according to claim 1, characterized in that, The joint residual modulation method in step 5 is as follows: The element-wise product of the soft mask m and the redundancy penalty coefficient γ is used as the modulation intensity; The modulation intensity is multiplied by the hidden state of the original image token to obtain the residual term; The residual term is added back to the hidden state of the original image token to obtain the enhanced image token representation.

7. The method for enhancing visual understanding of multimodal large language models based on perception-cognition alignment according to claim 1, characterized in that, For the redundancy score mentioned in step 5, a redundancy penalty coefficient is further set. , where λ is the preset maximum magnification scale.

8. A multimodal large language model visual understanding enhancement system based on perception-cognition alignment, used to implement the method according to any one of claims 1 to 6, characterized in that, include: The benchmark construction module is used to construct a hierarchical visual understanding evaluation benchmark, and selects the multimodal large language model to be enhanced as the backbone network. The hierarchical visual understanding evaluation benchmark divides the perceptual task and the cognitive task into three levels: low, medium and high. Each multimodal sample corresponds to six progressive questions, covering the entire process from low-level visual signal recognition to high-level causal and social relationship reasoning. The trigger layer preset module is used to preset a number of trigger layers at fixed intervals in the Transformer decoding layer of the backbone network, and to dynamically intervene in the hidden state of the image token during the inference phase. The cognitive probe module is used to enter the cognitive probe stage. In this stage, a cognitive guidance prefix is ​​appended before the user's original question to guide the multimodal large language model to be enhanced to associate potential causal and logical cues during visual encoding. The cognitive attention map is then captured at the trigger layer. ; The perception probe module is used to enter the perception probe stage, replace the cognitive guidance prefix with the perception guidance prefix, reconstrain the attention of the multimodal large language model to be enhanced to pixel-level and object-level surface features, and capture the basic perception attention map at the same trigger layer. ; The enhanced inference module is used to enter the enhanced inference phase. It inputs the original user question without prefixes into the model. When the network's forward propagation reaches any of the trigger layers, it initiates the perception-cognition alignment mechanism based on the cognitive attention graph. With the aforementioned basic perceptual attention map The difference is used to generate a soft mask m for adjusting the attention distribution, and a cross-modal redundancy score is generated based on the cross-modal redundancy between image tokens and text tokens. Then, based on the redundancy score, the modulation strength used to suppress redundant information is determined, and finally the hidden state of the image token is updated by joint residual modulation to obtain the enhanced image token representation. The output module is used to input the image token representation enhanced by perception-cognition alignment into the remaining network layers after the trigger layer, participate in the subsequent reasoning and decoding process, and output the final cognitive reasoning result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the visual understanding enhancement method for multimodal large language models based on perception-cognition alignment as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the visual understanding enhancement method for multimodal large language models based on perception-cognition alignment as described in any one of claims 1 to 8.

Images