Multi-modal large model adaptive reasoning and pruning method based on evidence consistency

Abstract

The invention provides a multi-modal large model adaptive reasoning and pruning method based on evidence consistency, which is used for image-text questions and answers, reducing FLOPs and video memories and finely controlling calculation budget under the condition of not changing backbone parameters. The method comprises the following steps: firstly, extracting image and problem features by using a multi-modal basic model, and generating a structure sensing Top-k mask through QCEG-SA; evidence perturbation is carried out under the three views, evidence gain, consistency and unified uncertainty characteristics are obtained, and lightweight alignment is carried out with UAEL-mm; and in combination with short chain self-consistency (SCSC) and lightweight verification (CAV), off-line evaluation is performed on different pruning rates and reasoning actions, and an AARC adaptive resource scheduler is trained. In the online reasoning process, the AARC automatically selects the pruning rate, the SCSC chain number and the CAV configuration for each sample, structural perception pruning and multi-modal reasoning are executed, and efficient deployment of resource limited scenes such as a single-card GPU is achieved while the precision is basically not reduced or even slightly increased.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, computer vision, and natural language processing, and in particular to an adaptive reasoning and pruning method for multimodal large model (MLLM) based on evidence consistency. Background Technology

[0002] With the rapid development of multimodal large models (such as 7B to 8B scale models containing visual encoders and language decoders), the performance of visual question answering (VQA) based on image and natural language problems is constantly improving. However, the computational and memory overhead of its inference stage is still huge, especially in resource-constrained scenarios such as single-GPU, edge devices, or online services. How to reduce computational costs with "almost no sacrifice in accuracy" has become a key challenge.

[0003] Existing technologies mainly attempt to improve inference efficiency from the following directions:

[0004] (1) Static or segmented static pruning / sparseness: Many methods employ fixed-ratio token pruning, channel pruning, or structured sparsity on the visual encoder or multimodal interaction layer to reduce the number of features involved in computation. These methods often use a single saliency metric (such as attention weight) for pruning, and their pruning ratio is usually fixed or segmented. It is difficult to perform fine-grained, continuous resource scheduling for specific problems and image instances, and it is also difficult to ensure that "evidence is truly utilized" in complex scenarios.

[0005] (2) Pruning based solely on saliency or visual attention: Another type of method uses visual saliency maps, gradient saliency, or self-attention weights to select and retain several “salient” image regions. While this type of method can focus on key regions to some extent, it usually lacks explicit modeling of the “problem semantics” and does not fully utilize the correlation between text and image. As a result, the retained regions may not be highly matched with the semantics of the current problem, thus limiting the compression space under strict accuracy constraints.

[0006] (3) Adaptive computation based on language-side self-consistency and uncertainty: In pure text tasks, existing works have used mechanisms such as multi-link self-consistency, Monte Carlo Dropout, temperature scaling, and early exit to allocate different computational amounts to different samples based on the uncertainty of the model, thereby achieving adaptive inference of "more computation for difficult problems and less computation for easy problems". However, most of these methods only apply to the language model itself and do not explicitly consider whether visual evidence is actually used, and they also lack a systematic measurement and scheduling mechanism for the "joint vision-language computation path".

[0007] (4) Lack of learnable evidence consistency metrics and scheduling loop: Existing multimodal acceleration solutions often only focus on the two endpoints of "computational cost" and "accuracy", rarely explicitly answering these questions. "When we retain visual regions relevant to the problem, does the model's confidence level improve significantly?", "When we destroy these key regions, does the model really become more uncertain, and does the prediction deteriorate significantly?", "How can we transform 'evidence usage' into a stable, learnable signal to further drive an interpretable scheduler to finely allocate the visual pruning ratio, the number of self-consistent links, and the number of additional validations for each sample?"

[0008] Due to the lack of system coupling among the aforementioned "evidence-uncertainty-resources," pruning based solely on a single saliency signal is not robust enough and is prone to mispruning key decision-making regions. Furthermore, using a fixed or coarse-grained pruning ratio makes it impossible to perform continuous and adaptive dynamic scheduling based on the difficulty of different samples and the distribution of evidence. Therefore, there is an urgent need for a new technical solution: introducing structure-aware pruning based on problem conditions on the visual side, constructing measurable evidence consistency and uncertainty signals under multi-view perturbation, and using these signals to drive a continuously adaptive scheduler to finely control the pruning ratio and additional computation on the language side at the sample level. This would enable multimodal question-answering reasoning with "lower computational cost at the same precision" in a single-GPU environment. Summary of the Invention

[0009] This invention aims to provide a multimodal question-answering reasoning method based on evidence consistency and continuous adaptive pruning. Without modifying the parameters of the online backbone model, it achieves finely controllable optimization of the computational budget: significantly reducing relative FLOPs and peak memory usage at the decoding end while maintaining near-constant or slightly improving accuracy. The specific steps are as follows:

[0010] (1) Construction and preprocessing of multimodal question answering samples

[0011] (1.1) Sample Construction. Collect multimodal question-answering samples from public datasets or business data. Each sample contains at least one image III, one natural language question q, and the corresponding standard answer y. Depending on the task requirements, it may contain multiple question types, such as open-ended questions, multiple-choice questions, and yes / no judgment questions.

[0012] (1.2) Image preprocessing. The input image is normalized in size, and the longest side is scaled to a preset upper limit while maintaining the aspect ratio, and pixel normalization is performed. Then, the image is divided into several patches using a regular grid or the patch partitioning method inside the visual encoder. Each patch is mapped to a visual feature in vector form, providing a basis for subsequent text-image correlation calculation and pruning scoring.

[0013] (1.3) Question text preprocessing. The question q is segmented into words or sub-words and input into the text encoding module inside the pre-trained language encoder or multimodal model to obtain the hidden vector sequence of the question. The global representation vector of the question is obtained by pooling or reading specific positions ([CLS]) and used to establish correlation with image patch features.

[0014] (2) Multimodal basic model and lightweight feature extraction on the problem side.

[0015] (2.1) Construction of the Multimodal Base Model. Select any large multimodal model with image-text question-answering capabilities as the base model. This model typically includes a visual encoder, a visual-to-language projection module, and a language decoder. The visual encoder maps the preprocessed image into a sequence of patch features, and the language decoder outputs the answer sequence after receiving the question text and visual features.

[0016] (2.2) Lightweight feature extraction on the question side. Without significantly increasing additional computational costs, a set of lightweight features is extracted from the question text for use as input to the subsequent AARC scheduler. These lightweight features may include: question length (number of words or tokens), question type label (e.g., whether it is a yes / no question), dataset label, and whether it involves coarse-grained semantic categories such as count, color, and location, so as to quickly distinguish between "simple questions" and "complex questions" during the inference stage.

[0017] (3) Construction of QCEG-SA structure-aware pruning score map based on text-image correlation. Based on the above, this invention constructs a structure-aware score map for pruning through the QCEG-SA (Question-Conditioned, Evidence-Guided, Structure-Aware) module, focusing on introducing the correlation between text and image, so as to more accurately select visual evidence related to the question.

[0018] (3.1) Alignment of visual patch features with problem vectors. For each image patch, the visual feature vector v is obtained. x By mapping through a linear layer to a space of the same dimension as the problem vector, we obtain z. x Simultaneously, the global problem vector h q Mapped to z via a linear layer or MLP q Thus, z x With z q Being in the same semantic space facilitates subsequent relevance calculations.

[0019] (3.2) Construct a saliency map HM(x) based on text-image correlation. This invention no longer uses a saliency index that relies solely on visual characteristics, but instead defines saliency by calculating the correlation between patch features and the question vector: z x With z q A normalized dot product or other similarity metric is used to obtain a relevance score. This score is then scaled and translated before being input into a non-linear function (Sigmoid) to obtain an HM(x) value between 0 and 1. In this way, HM(x) directly represents "the importance of the patch in the current problem context," ensuring that the pruning process always revolves around the semantics of the problem.

[0020] (3.3) Constructing the cross-modal correlation map Rel(q,x). Based on the calculation of HM(x), this invention further normalizes the relative importance of patches across the entire image. By performing a softmax operation on the correlation scores of all patches, Rel(q,x) is obtained, whose value reflects "the relative attention weight of position x to question q in all patches". Unlike the point-by-point scoring of HM(x), Rel(q,x) emphasizes the relative ranking at a global scale.

[0021] (3.4) Fusion and Local Smoothing to Obtain the Structure-Aware Scoring Map. HM(x) and Rel(q,x) are linearly fused according to their weights, and a local smoothing operation with small 3×3 convolutional kernels is applied to the fused result. This ensures continuity in scores between adjacent patches, forming high-scoring regions with consistent structure. The fused scoring map can be denoted as... Numerically, it takes into account both local correlation and global ranking, and through smoothing, it ensures that the pruning results present connected regions rather than isolated points, which is beneficial for preserving complete objects or key regions in subsequent reasoning.

[0022] (3.5) Construct a retention mask based on the scoring map. According to... Sort all patches and select the top-k patches with the highest scores as candidate retention regions to obtain the retention mask. Here, top-k can be represented by retention percentages (e.g., 30%, 40%, 50%) or by the absolute number of patches. Subsequent different values ​​of k will be used by AARC as part of the action to achieve continuously variable pruning intensity.

[0023] (4) Construction of three-view evidence perturbation and calculation of evidence consistency and uncertainty index. After obtaining the retention mask based on text-image correlation, this invention constructs three views and conducts evidence perturbation experiments to convert the "impact of retaining and destroying evidence on model behavior" into a measurable index.

[0024] (4.1) Constructing the original view, the retained view, and the destroyed view. On the same input image, three views are generated by retaining a mask: the original view (orig): keeping the original image unchanged; the retained view (keep): retaining only the top-k high-scoring regions and setting other patches to zero or filling them with the mean; and the destroyed view (destroy): retaining only the low-scoring regions and setting the top-k high-scoring regions to zero or blurring them. In this way, the orig, keep, and destroyed views correspond to the three states of "original evidence," "emphasizing key evidence," and "deliberately destroying key evidence," respectively.

[0025] (4.2) Multi-view forward inference and output statistics. The three views are input into the multimodal base model along with the same problem to obtain the corresponding output logits and probability distributions. The marginal loss, probability entropy, and class probability variance for the true class are calculated under multiple lightweight sampling conditions (e.g., MC=3). These statistics are used to measure the confidence and stability of the model under different evidence configurations.

[0026] (4.3) Construction of Evidence Consistency and Uncertainty Indicators. This invention defines multiple indicators based on the aforementioned statistics, including but not limited to: Evidence Gain (EAG): measures the marginal improvement from the destroy view to the keep view, reflecting whether emphasizing key areas significantly increases the confidence of the correct answer; Consistency Probability (CEP): measures whether the ideal order of "keep outperforms orig, and orig outperforms destroy" is generally satisfied in multiple inferences; Uncertainty Gain (URG-mm): measures the change in entropy and variance from the keep to the destroy view, reflecting whether destroying key evidence significantly increases uncertainty; Unified Uncertainty Feature U: synthesizes the entropy and variance under the orig view into a single feature to describe the overall uncertainty level of the problem under unpruned conditions. These indicators transform "whether the model truly uses the evidence we selected" into quantifiable signals, serving as important inputs for subsequent training of the AARC scheduler.

[0027] (5) Lightweight representation alignment training based on UAEL-mm. After completing the construction of the three-view diagram and the calculation of the index, the present invention further performs lightweight representation alignment training on the basic model in the offline stage to improve the model's sensitivity to the behavioral pattern of "keeping the view increases confidence and destroying the view increases uncertainty".

[0028] (5.1) Construct multi-view task loss. Calculate task cross-entropy loss on the original and keep views respectively, to encourage the model to give the correct answer on both the original and keep views, and appropriately increase the importance of the keep view, so that the model learns to rely more on regions that are more relevant to the problem.

[0029] (5.2) Constructing the view Figure 1 Consistency and contrast loss. By adding a distribution consistency constraint between the keep and orig views, the output probability distributions of the two are made as close as possible while maintaining correct predictions. At the same time, the marginal difference constraint between orig and destroy is used to strengthen the contrast effect that "destroying key evidence will significantly reduce confidence".

[0030] (5.3) Construct uncertainty regularization. When the model already has sufficient confidence in the orig view, negative entropy or variance regularization is used to encourage it to reduce excessive uncertainty, thereby stabilizing subsequent scheduling based on the uncertainty feature U.

[0031] (5.4) Construct the UAEL-mm loss and train it offline. Combine the task loss, consistency loss, and uncertainty regularization term above into the total UAEL-mm loss, and perform light fine-tuning on the base model or targeted training on some modules (such as the projection layer). After training, the updated parameters can be used as the runtime model, or only as an evaluator during the AARC training phase to guide AARC to more accurately perceive behavioral differences under different views.

[0032] (6) Short Chain Self-Consistency (SCSC) and Lightweight Consistency Verification (CAV) Inference Action Design. After completing the lightweight alignment of the model, this invention further defines two types of "inference actions" that can be used for scheduling, which are used to enhance language-side consistency and factual consistency, respectively. These two types of actions will form a complete action combination together with the pruning ratio during AARC training.

[0033] (6.1) Short-Chain Self-Consistent (SCSC) Action. Given the pruning ratio and decoding parameters, multiple shorter generating chains are executed for the same problem. Each chain employs a random sampling or small-perturbation decoding strategy, resulting in multiple candidate answers. These answers are aggregated through majority voting or distribution averaging, which can significantly reduce random errors caused by single-sample noise. The SCSC switch and the number of chains S are included as part of the action, and AARC determines "when to calculate more times and when one is enough".

[0034] (6.2) Lightweight Consistency Verification (CAV) Action. For yes / no questions, after the model provides an initial answer, this invention automatically constructs several verification or decision questions related to that answer, and performs up to M additional, highly simplified verification inferences under conditions of high pruning intensity or high uncertainty. By statistically analyzing the consistency ratio during the verification process and comparing it with a threshold τ, some answers that are clearly inconsistent with the image evidence can be filtered out at a relatively low cost. The CAV on / off state, the number of verifications (MMM), and the threshold τ are also included as part of the action for AARC scheduling.

[0035] (7) Combine SCSC and CAV to generate slate experience and train the AARC continuous adaptive controller. Based on the above preparations, this invention allows the model to run in real-world scenarios under different “pruning + SCSC + CAV” configurations, collects the accuracy and cost of each configuration, and constructs an experience slate for training the AARC continuous adaptive resource controller.

[0036] (7.1) Action Space Design and Empirical Data Acquisition. For each sample, a series of candidate pruning ratios k (e.g., 0.3, 0.4, 0.5), candidate SCSC chain numbers (e.g., 1, 2, 3 chains), candidate CAV thresholds, and combinations of maximum validation counts (e.g., no CAV, light validation, heavy validation, etc.) are pre-defined. For each action combination (k, S, τ, M), the model is allowed to perform one or more complete question-answering inference operations in an offline environment.

[0037] If the model answer under this combination is correct, then the precision index is recorded as 1; otherwise, it is recorded as 0.

[0038] Simultaneously, the visual computational cost, language generation steps, SCSC chain count, and CAV count after pruning are statistically analyzed. Combined with a pre-set unit cost coefficient, the comprehensive cost of this action combination is obtained.

[0039] In the above way, a set of "candidate action - accuracy - cost" records can be obtained for each sample, and this set of records constitutes the slate experience of that sample.

[0040] (7.2) Construct reward and target distributions based on accuracy and cost. For each action combination, the accuracy and cost are combined into a reward value according to a preset formula (e.g., reward = accuracy - cost coefficient × cost). Then, the reward is normalized into a target probability distribution through Boltzmann-soft and other methods to guide AARC to tend to select actions that have "lower cost" while "ensuring accuracy".

[0041] (7.3) AARC Input Feature Construction and Network Training. For each sample, the input vector of AARC is formed by combining the lightweight feature t(q) on the problem side, the evidence consistency index (such as EAG, CEP, URG-mm) obtained in the three-view experiment, the overall uncertainty index U, and the margins of the original view. AARC encodes the input through a multilayer perceptron and outputs the probability distribution of action combinations and regression estimates of accuracy and cost. During training, the action distribution output by AARC is made to approximate the target distribution constructed by the reward, while the ability to predict accuracy and cost is improved through regression loss. After sufficient training, AARC can predict the "most suitable pruning ratio, number of SCSC chains, and CAV configuration" for each new sample based solely on the input features, achieving true continuous adaptive scheduling.

[0042] (8) Joint scheduling and question-answering output during the online inference phase. In the actual deployment phase, this invention uses AARC as the core scheduler to connect the above steps into a complete inference process.

[0043] (8.1) Real-time feature computation and scheduling decision. For new input images and questions, preprocessing and feature extraction are performed first. Then, the text-image correlation score map is calculated using QCEG-SA to obtain a preliminary retention mask. At the same time, lightweight features on the question side and necessary evidence consistency and uncertainty features are calculated or approximately estimated. These features are input into AARC to obtain the action decision corresponding to the current sample.

[0044] (8.2) Perform structure-aware pruning and multimodal reasoning. Based on the pruning ratio k output by AARC, update the preserved mask and perform structure-aware patch pruning on the visual features. Then, send the pruned visual features and the question text together into the language decoder to perform main chain decoding. If the action contains SCSC, perform multiple short chain decodings according to the specified chain number and aggregate the answer. If the action contains CAV, perform the corresponding number of lightweight verifications for yes / no questions, correct inconsistent results, or trigger re-decoding.

[0045] (8.3) Output the answer while satisfying budget constraints. After the above steps are completed, output the final answer and optional uncertainty estimates. Throughout the process, the pruning ratio, SCSC chain count, and CAV count are automatically selected by AARC, thereby achieving differentiated resource allocation for different samples under given budget constraints (such as FLOPs, memory, etc.). Stronger pruning and minimal additional computation are used for "simple, well-evidenced" samples, while computation is appropriately increased for "complex or poorly evidenced" samples, thus achieving the overall effect of "approaching the accuracy of the original model and significantly reducing the average computational cost".

[0046] Compared with the prior art, the advantages of the present invention are as follows:

[0047] (1) By explicitly introducing the correlation between text and image at the Patch level through the QCEG-SA module, HM(x) and Rel(q,x) are used to jointly characterize the "visual saliency under problem conditions" and "global relative importance", which can more accurately preserve key evidence areas while ensuring structural continuity and significantly reduce the risk of miscutting.

[0048] (2) By designing three-view evidence perturbations using orig / keep / destroy, evidence consistency and uncertainty indices such as EAG, CEP, and URG-mm are constructed. Combined with lightweight representation alignment training using UAEL-mm, this enables the base model to...

[0049] The behavioral differences between the two extreme cases of "preserving key evidence" and "destroying key evidence" are more controllable, providing the scheduler with stable and reliable monitoring signals.

[0050] (3) Short chain self-consistency (SCSC) and lightweight consistency verification (CAV) are explicitly modeled as schedulable inference actions, and together with different pruning ratios, they form an action space in the offline stage. By obtaining slate experience containing accuracy and cost information through real running models, the continuous adaptive resource controller AARC is trained to automatically select the optimal action combination at the sample level, so as to achieve true "on-demand allocation of computing budget".

[0051] (4) The various sub-steps of this invention, from data preprocessing, relevance scoring map construction, evidence perturbation and alignment training, to action definition, slate experience construction and AARC scheduling training, and then to joint scheduling in the online inference stage, constitute a complete process flow that is closely connected. It has good interpretability, scalability and engineering implementation value, and is applicable to a variety of mainstream multimodal large models and a variety of question answering task scenarios. Attached Figure Description

[0052] Figure 1 : A schematic diagram of the multimodal question-answering reasoning framework of the present invention

[0053] Figure 2 Three-view diagrams (orig / keep / destroy)

[0054] Figure 3 A schematic diagram of the AARC scheduler structure provided in this embodiment of the invention. Detailed Implementation

[0055] 1. System Environment and Basic Model Configuration

[0056] In this embodiment, inference and training can be performed in a single-GPU or multi-GPU environment. A typical hardware environment may include: at least one GPU with 16GB–24GB of video memory (such as an NVIDIA RTX 3090), a multi-core CPU, at least 16GB of RAM, and a standard SATA or NVMe hard drive. The software environment may include a Linux operating system, a deep learning framework (such as PyTorch), a multimodal model loading library, and a conventional image processing library. In this environment, any multimodal base model with text-image question answering capabilities is selected as the backbone model. This backbone model typically includes: a visual encoder for dividing the input image into patches and extracting visual features; a projection layer or cross-modal alignment module for mapping visual features to a language space; and a language decoder for jointly modeling the question text and visual features and generating the answer. The base model can be an existing publicly available large multimodal model or a self-developed model within an enterprise. This invention does not limit the specific model name, only requiring that it possess text-image joint inference capabilities.

[0057] 2. Multimodal question-answering data preparation and preprocessing

[0058] In this embodiment, a multimodal question-answering dataset containing images, natural language questions, and standard answers needs to be prepared for offline training and online evaluation.

[0059] First, collect image-text question-answering samples from multiple sources and scenarios, including common objective questions, open-ended questions, and yes / no questions. For each sample, denote the image as I, the question text as q, and the corresponding standard answer as y. Different dataset labels can be set according to business needs, such as "General VQA," "Document Question Answering," and "Knowledge Question Answering."

[0060] In the image preprocessing stage, the input image is uniformly scaled to a preset maximum side length, for example, limiting the longest side to no more than 448 pixels or 576 pixels, while scaling the shorter side proportionally while maintaining the aspect ratio. Subsequently, the pixel values ​​are normalized, mapped to [0,1] or standardized according to the statistics used during model pre-training.

[0061] In the text preprocessing stage, the question q is segmented into words or sub-words, and encoded into a token sequence based on the vocabulary of the multimodal model. The hidden state sequence of the question is then obtained through an embedding layer and several Transformer layers. By taking the [CLS] position vector or performing average pooling on all positions, the global representation vector of the question is obtained, which is used for subsequent calculation of its relevance to the image patch.

[0062] 3. QCEG-SA Text-Image Correlation Scoring and Preservation Mask Generation

[0063] After completing image and text preprocessing, this embodiment uses the QCEG-SA module to calculate the importance score of each image patch in the current problem context, and generates a pruning-preserving mask accordingly.

[0064] First, the feature map output by the visual encoder is expanded according to the patch position to obtain the feature vector v of each patch. x To facilitate correlation calculation with the question vector, a linear transformation is used to map the patch features to a unified embedding space, resulting in z. x =W v v x +b v At the same time, the global vector of the problem h q Mapping to the same space, we get z q =W q h q +b q z x and z q A joint representation space in the same dimension can reflect the relationship between image regions and the semantics of the problem. b v ,b q This is the bias vector.

[0065] Then, based on z x With z q Calculate the text-image correlation significance map HM(x). For example, you can first calculate the cosine similarity between the two:

[0066]

[0067] Then, normalization is performed using affine transformation and the Sigmoid function:

[0068] HM(x)=σ(γ H ·s H (x)+β H )

[0069] Where γ H and β H HM(x) represents learnable or preset scaling and translation parameters, and σ is the sigmoid function. The larger the value of HM(x), the more relevant the patch is to the problem semantics, and the more it belongs to the region with "problem condition saliency".

[0070] Subsequently, a cross-modal correlation graph Rel(q,x) is constructed. This allows z to be... x With z q The inner product e(x) is used as the attention score, and softmax normalization is performed at all patch locations:

[0071]

[0072] Rel(q,x) emphasizes the relative importance of Patch x to other Patches within the entire image. HM(x) and Rel(q,x) characterize local correlation and global ranking, respectively.

[0073] Next, HM(x) and Rel(q,x) are linearly fused according to their weight coefficients, and local smoothing is performed using a 3×3 or similarly sized convolutional kernel to obtain the structure-aware scoring map. Local smoothing can make adjacent patches more consistent in their scores, helping to preserve connected object regions rather than scattered isolated points.

[0074] Finally, according to The scores of all patches are sorted, and the top-k patches with the highest scores are selected to form the retention mask. k can be a retention percentage (e.g., 30%, 40%, 50%) or an absolute number of patches. Different k values ​​correspond to different pruning intensities and will be part of the optional actions of the subsequent AARC scheduler. Patches with lower scores can be excluded from calculation in subsequent visual encoding stages, thus achieving pruning and acceleration.

[0075] 4. Construction of three-view diagrams for orig / keep / destroy and forward reasoning of the model

[0076] To assess the impact of "preserving or destroying key evidence" on model behavior, this embodiment constructs three image views by preserving masks during the offline phase and performs forward reasoning on each of the three views.

[0077] First, construct the original view orig by directly using the unpruned original image I as input and providing the model output with question q to obtain the logits and probability distribution of the orig view.

[0078] Then, based on the retain mask generated in the previous step, a patch-by-patch multiplication operation is performed on the image to construct the retain view (keep). For patches with a mask of 1, the original pixels or features are retained; for patches with a mask of 0, the pixels are replaced with zero values, average values, or slightly blurred values. The resulting keep view retains areas highly relevant to the problem while weakening irrelevant parts such as the background.

[0079] Next, a destruction view is constructed. An inverse mask that preserves the mask can be used to zero out, blur, or replace noise in the top-k high-scoring patches, retaining only the original content of the low-scoring patches. The destruction view intentionally destroys key evidence areas while preserving the background or irrelevant areas.

[0080] For the three views of orig, keep, and destroy, the same multimodal base model is used to perform forward inference with the same question q. The logits, probability distribution, entropy, and probability variance of the true answer y under multiple random decoding or Monte Carlo sampling conditions are recorded for each view, in preparation for subsequent calculation of evidence consistency and uncertainty indicators.

[0081] 5. Calculation of Evidence Consistency and Uncertainty Indicators and Training of UAEL-mm

[0082] After obtaining the prediction results of the three-view diagram, this embodiment calculates a variety of evidence consistency and uncertainty indices, and uses these indices to perform lightweight representation alignment training (UAEL-mm) to enhance the model's sensitivity to "evidence is preserved" and "evidence is destroyed".

[0083] First, for each view, calculate the margin of the true answer category, which is the difference between the logits of the true answer category and the logits of other categories. The larger the margin, the more confident the model is in its predictions for that view.

[0084] Then, the marginal difference between the keep view and the destroy view is statistically analyzed to obtain the evidence gain index, which is used to measure "how much the model confidence has improved after emphasizing the key area compared to deliberately destroying the key area".

[0085]

[0086] At the same time, statistics show whether the ideal order of "keep performs better than orig, and orig performs better than destroy" is widely satisfied in multiple inferences based on three views, forming a consistency probability index to characterize whether the model truly uses high-scoring patches as reliable evidence.

[0087]

[0088] In addition, the entropy of the probability distribution and the variance of the class probability under multiple sampling conditions are calculated on each view, and the entropy difference and variance difference between keep and destroy are combined to form an uncertainty gain index, which describes whether "destroying key evidence significantly increases uncertainty".

[0089] URG mm =(H(p) destroy,k )-H(p keep,k ))+Var mm (p keep,k ,p destroy,k ),

[0090] For the orig view, entropy and variance are combined by weight into a unified uncertainty feature U, which reflects the basic difficulty of the sample without pruning.

[0091] U=γH(p orig )+δVar mm (p orig ),

[0092] After obtaining the above metrics, this embodiment constructs a system that includes task loss and view... Figure 1 The UAEL-mm loss function for consistency loss and uncertainty regularization term.

[0093] The task loss is used to ensure that the model's predictions are correct on the orig and keep views:

[0094] L task =CE(y,p orig )+α·CE(y,p keep,k ),

[0095] Where α is the equilibrium coefficient

[0096] See Figure 1 The consistency loss constraint keeps keep and orig close in output distribution, while strengthening the negative contrast between orig and destroy at the margin:

[0097] L cons =γKL(p keep,k ||p orig )+λ(m(orig)-m(destroy k )),

[0098] Where KL(·‖·) is the Kullback-Leibler divergence.

[0099] Uncertainty regularization encourages models to reduce unnecessary uncertainty when there is sufficient evidence:

[0100] L unc =-βVar mm (p orig ).

[0101] The total loss of UAEL-mm is a combination of the losses from the above three parts:

[0102] L = L task +L cons +L unc

[0103] By using UAEL-mm loss to perform light fine-tuning of the backbone model or training of some modules, the model can be made more stable when retaining key evidence and exhibit more obvious uncertainty changes when destroying key evidence, thereby providing a more reliable training signal for the subsequent AARC scheduler.

[0104] 6. Implementation of Short-Chain Self-Consistent (SCSC) Inference Strategy

[0105] In this embodiment, Short Chain Self-Consistency (SCSC) is an adjustable inference action used to improve answer stability during the language decoding stage by aggregating the results of multiple short chains. In implementation, a short chain decoding strategy is first configured for each question. This strategy can limit parameters such as the maximum number of tokens, sampling temperature, and Top-p range for each generating chain, keeping the time and memory overhead of each chain low. Furthermore, different random seeds or sampling orders are set for each chain to generate some diversity while maintaining the overall decoding style. When SCSC is enabled, under given pruning ratios and visual features, several short chains are decoded for the same question, resulting in a set of candidate answers. For yes / no questions, a selection strategy based on majority voting, probability averaging, or logarithmic probability summation can be used to determine the final answer from these candidate answers; simultaneously, the consistency of answer distributions between chains can be recorded as an auxiliary signal for assessing the difficulty or uncertainty of the sample. The number of short chains, S, is part of the AARC action. During the slate experience collection phase, the possible values ​​of S (e.g., 1 chain, 2 chains, 3 chains, etc.) can be enumerated, and the accuracy and computational cost under different chain numbers can be recorded so that AARC can be trained to adaptively select S under the principle of "less computation for simple problems and more computation for complex problems".

[0106] 7. Implementation of Lightweight Consistency Verification (CAV) Inference Strategy

[0107] Lightweight Consistency Validation (CAV) primarily targets yes / no questions, performing factual consistency checks on the initial answer provided by the model with minimal additional computation. In implementation, after the model completes main chain decoding on the pruned view and provides an initial answer, this embodiment automatically constructs one or more validation questions based on the original question q and candidate answer a. For example, the original question can be rewritten as a statement asking "Is this statement consistent with the image content?", or it can directly ask, "Based on the image content, is your previous answer 'a' correct? Please answer yes or no," etc. For each validation question, this embodiment allows multiple executions of simplified inference within a limited maximum number of validations M. Simplified inference can employ a shorter maximum generation length and lower sampling diversity to control additional costs. The proportion of "correct answer," "consistent answer," or "Yes" given by the model in each validation inference is recorded and compared with a preset threshold τ. When the proportion is below the threshold τ, the current initial answer is considered insufficiently credible, triggering re-decoding, changing the pruning configuration, or outputting a conservative "uncertain" answer. Specific strategies can be set according to business needs. During the slate experience phase, this embodiment enumerates different combinations of threshold τ and maximum number of verifications M, and records the improvement of overall accuracy and increase in computational cost of CAV under each combination, so that AARC can learn when to enable CAV and what strength of verification is most appropriate during training.

[0108] 8. Combining SCSC and CAV slate experience generation with AARC controller training

[0109] In this embodiment, AARC is a continuous adaptive resource scheduler whose training depends on the actual running results of the model under different pruning intensities, SCSC and CAV configurations, i.e., slate experience.

[0110] Specifically, for each training sample, multiple action combinations are enumerated in an offline environment, including different pruning ratios k, different short chain numbers S, different CAV thresholds τ, and different maximum number of validations M. For each action combination, this embodiment runs the multimodal base model with QCEG-SA pruning enabled, and enables or disables SCSC and CAV according to the action combination, thereby completing a complete question-answering inference process.

[0111] After each inference, record whether the model provides the correct answer under that action combination, and the relative computational cost consumed in visual encoding, language decoding, SCSC linking, and CAV verification. The number of patches, generation steps, and verification attempts can be normalized into a unified cost metric, defining a comprehensive cost, based on actual needs.

[0112] Cost = k + u S S+(b C +mC M),

[0113] Where u S ,b C ,m C Unit cost coefficient for each action

[0114] Then, based on the correctness and cost of each action combination, a reward value r = c - λCost is constructed. For example, the reward can be set as "whether the prediction is correct minus the cost coefficient multiplied by the cost". By normalizing the rewards of all action combinations, the target action distribution of the sample is obtained, with action combinations that have higher rewards having higher weights.

[0115]

[0116] Meanwhile, an input feature vector for AARC is constructed for each sample. The input features include lightweight features from the problem side (such as problem length, problem type, dataset label, etc.), indicators such as evidence gain, consistency probability, and uncertainty gain obtained from the three-view evidence perturbation stage, as well as the unified uncertainty features U and margins under the orig view. These features collectively characterize the sample difficulty, evidence utilization, and model uncertainty.

[0117] Finally, the AARC controller is trained using the aforementioned input features and target action distribution. AARC consists of a six-layer MLP backbone network f ω (x) is constructed, and Actor branches are connected to its output: a three-layer MLP + LayerNorm + GELU, with the output decomposition strategy distribution π. k ,π S ,π C ; Critic branch: Shares a main branch followed by two regression heads, which predict accuracy respectively. With cost During training, the AARC's action distribution is made to approximate the target action distribution, and its ability to estimate accuracy and cost is optimized. After training, AARC has the ability to automatically select appropriate action combinations given sample features.

[0118]

[0119] 9. Joint scheduling and question-answering output during the online inference phase

[0120] During the online deployment phase, the method of this invention uses an AARC controller to uniformly schedule pruning intensity, SCSC, and CAV configurations, enabling multimodal question answering inference to achieve significant computational budget savings while meeting accuracy requirements.

[0121] When the system receives a new image-text question-answering request, it first preprocesses and encodes the image and question. The QCEG-SA module is then used to calculate the text-image relevance significance HM(x) and Rel(q,x), generating a structure-aware scoring map and obtaining a set of candidate retention masks. These lightweight text features, along with necessary evidence consistency and uncertainty estimations, form the input features for the current sample, which are then fed into the pre-trained AARC controller.

[0122] AARC outputs the action decision corresponding to the sample, including the pruning ratio k, the number of short chains S, the validation threshold τ, and the maximum number of validations M. The retained mask is updated according to the pruning ratio k, and structure-aware patch pruning is performed on the visual features, retaining only the top-k patches for subsequent vision-to-language projection and interaction layers. During the language decoding stage, self-consistent short chain generation is performed according to S, and CAV validation is enabled for yes / no questions according to τ and M when necessary. The FLOPs and peak memory usage of the entire inference process are significantly lower than the baseline without pruning and adaptive scheduling.

[0123] After inference is complete, the system outputs the final answer and optional confidence information. Through AARC's adaptive scheduling, simple samples are typically allocated a larger pruning ratio and less additional computation, while complex samples receive a higher computational budget. This significantly reduces average computational overhead and GPU memory usage while achieving "almost no loss of accuracy or even a slight improvement" overall.

Claims

1. A multimodal large-scale adaptive reasoning and pruning method based on evidence consistency, characterized in that, Includes the following steps: Step S01: Multimodal question answering sample construction and preprocessing. Obtain a sample containing image I, question text q, and standard answer y. Normalize the size of the image and divide it into patches. Perform word segmentation and encoding on the question text to obtain the global representation vector of the question. Step S02: Construct a multimodal base model, which includes a visual encoder, a vision-to-language projection module, and a language decoder, for jointly modeling the preprocessed image and the question and outputting the answer; Step S03: Construct a QCEG-SA structure-aware pruning scoring map based on the correlation between text and images. Align the patch features of each image with the question vector, and calculate the saliency map HM(x) and the cross-modal correlation map. Rel(q,x) is used to fuse and smooth the structure-aware scoring map, and different Top-k retention masks are generated based on the structure-aware scoring map. Step S04: Using the retention mask, construct three views on the same image: original view (orig), retain view (keep), and destroy view (destroy). Input the three views and the question text into the multimodal basic model respectively to obtain the corresponding prediction results. Calculate evidence consistency and uncertainty indices such as evidence gain, consistency probability, uncertainty gain, and unified uncertainty feature. Step S05: Based on the three-view prediction results and the evidence consistency and uncertainty index, construct the UAEL-mm loss, and perform lightweight representation alignment training on the multimodal basic model to make the difference in the model’s behavior under the two states of retaining key evidence and destroying key evidence more controllable. Step S06: Design short chain self-consistent SCSC inference actions and lightweight consistency verification CAV inference actions. Use the pruning ratio k, the number of SCSC chains S, the CAV verification threshold τ, and the maximum number of verifications M as adjustable action parameters. Enumerate different action combinations under the condition of enabling QCEG-SA pruning, and run the multimodal basic model to obtain slate experience including accuracy and computational cost. Step S07: Using the problem lightweight feature t(q), evidence gain, consistency probability, uncertainty gain, unified uncertainty feature, and original view margin as input vectors x, construct an AARC continuous adaptive resource scheduler consisting of a six-layer MLP backbone network fω(x), an Actor branch, and a Critic branch. Use the slate experience to construct a reward and target action distribution based on accuracy and cost, and train the AARC. Step S08, the online inference stage, preprocesses the new input image and question, calculates the structure-aware scoring map and lightweight features of the question, and feeds the input vector x into the trained AARC to obtain the result containing the pruning ratio k and SCSC. The action decision is configured based on the number of chains S and CAV. The structure-aware patch pruning, multi-chain self-consistency generation and consistency verification are performed according to the action decision, and the final question answering result is output.

2. The multimodal large model adaptive inference and pruning method as described in claim 1, characterized in that, Step S3 The construction of the saliency plot HM(x) includes: Step S03-1: Obtain the first embedding vector by linearly mapping each patch feature, and obtain the second embedding vector by linearly mapping the global representation of the problem. Step S03-2: Calculate the similarity between the first embedding vector and the second embedding vector as a relevance score, and after scaling and translation, input it into the Sigmoid function for normalization to obtain HM(x); Step S03-3, the cross-modal correlation map Rel(q,x) is obtained by performing Softmax normalization on the correlation scores of all patches.

3. The multimodal large model adaptive inference and pruning method as described in claim 1, characterized in that, In step S3, HM(x) and Rel(q,x) are linearly fused according to preset weights, and a 3×3 convolution kernel is used to locally smooth the fusion result to obtain a structure-aware scoring map. The structure-aware scoring map is used to form connected high-scoring regions at the patch level.

4. The multimodal large model adaptive inference and pruning method as described in claim 1, characterized in that, Step S4 The construction method of the three views in the middle is as follows: The original view (orig) is the unmasked, raw image; The view keep is obtained by preserving the original pixels or features of the Top-k high-scoring Patch and replacing the non-Top-k Patch with zero, mean, or blurred values. The destroy view is obtained by replacing the top-k high-scoring patches with zero values, noise, or blurry values, while preserving the original pixels or features of the remaining patches.

5. The multimodal large model adaptive inference and pruning method as described in claim 1, characterized in that, The evidence consistency and uncertainty indicators in step S4 include: Evidence Gain (EAG) measures the difference between destroyed and retained views at the true category margin. Consistency probability (CEP) is used to measure whether the order of preserving the view is better than the original view and the original view is better than the destroyed view is satisfied in multiple inferences. Uncertainty gain URG-mm is used to measure the increase in entropy and multiview variance of destroyed views relative to retained views. The unified uncertainty feature U is obtained by weighting the entropy of the original view and the probability variance of the multiple views.

6. The multimodal large model adaptive inference and pruning method as described in claim 1, characterized in that, The UAEL-mm loss in step S5 includes: The task cross-entropy loss for the original view and the retained view is used to ensure that the predictions for both views are correct; Distribution consistency loss between the retained view and the original view, and marginal contrast loss between the original view and the destroyed view; A regularization term for the uncertainty of the original view is used to suppress excessive uncertainty when there is sufficient evidence. The total loss of UAEL-mm is the weighted sum of the losses mentioned above.

7. The multimodal large model adaptive inference and pruning method as described in claim 1, characterized in that, In step S6, the comprehensive cost Cost of each action combination is calculated, and the accuracy index c is obtained according to whether the prediction is correct. The reward r is constructed according to r = c - λ·Cost, where λ is the cost coefficient. Then, the reward is transformed into the target action distribution using the Boltzmann-soft method.

8. The multimodal large model adaptive inference and pruning method as described in claim 1, characterized in that, The lightweight feature t(q) of the problem includes at least one or more of the following: problem length, whether it is a yes / no problem, the dataset to which it belongs, and the problem type.

9. The multimodal large model adaptive inference and pruning method as described in claim 1, characterized in that, The AARC continuous adaptive resource scheduler includes: Six-layer MLP backbone network f ω (x), used to encode the input vector x to obtain an intermediate representation; The Actor branch, with the intermediate representation as input, consists of three layers of MLP and subsequent LayerNorm and GELU, and is used to output the pruning ratio k, the number of SCSC chains S, and the action distribution of the CAV configuration. The Critic branch shares the intermediate representation with the backbone network and is connected to two regression heads, which respectively predict the accuracy estimate. With cost estimate 10. The multimodal large model adaptive inference and pruning method as described in claim 9, characterized in that, The action distribution of the Actor branch output is presented in a decomposition form: First, output the pruning ratio distribution π. k (k∣f ω (x)); Output the SCSC chain number distribution π under a given pruning ratio. s (S∣f ω (x),k); Given the pruning ratio and the number of SCSC chains, output the joint distribution of the CAV threshold and the maximum number of validations. p C (t,M∣f ω (x),k,S).

11. The multimodal large model adaptive inference and pruning method as described in claim 1, characterized in that, The short-chain self-consistent SCSC inference action obtains multiple candidate answers by performing multiple short-chain decodings on the same question under given pruning ratio and decoding parameters, and aggregates them using majority voting, probability averaging, or logarithmic probability summation to obtain the final answer.

12. The multimodal large model adaptive inference and pruning method as described in claim 1, characterized in that, The lightweight consistency verification CAV inference action is only for yes / no questions. After the model gives the initial answer, the verification question is automatically constructed based on the original question and candidate answers. The maximum number of verifications M is limited. In each verification, simplified inference is performed and the proportion of consistent answers is counted. When the proportion is lower than the threshold τ, the initial answer is rejected, re-decoded, or marked as uncertain.

Images