Visual language model reasoning ability enhancement method and device, equipment and medium

Abstract

The invention discloses a visual language model reasoning ability enhancing method, device and equipment and a medium, and is applied to scenes in the financial and medical field. The visual language model reasoning ability enhancing method comprises the following steps: selecting a language component and a source reasoning language model of a target visual language model based on the same pre-training base; and extracting a first capability increment and a second capability increment of the two relative to the base model by using parameter difference, carrying out weighted fusion on the two capability increments to generate a target fusion language model, replacing the original language component with the target fusion language model, and constructing an enhanced visual language model. According to the method, the performance of the visual language model in complex logic and mathematical reasoning tasks is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology in financial and medical scenarios, and in particular to a method, apparatus, device and medium for enhancing the reasoning ability of a visual language model. Background Technology

[0002] With the development of artificial intelligence, visual language models (VLMs) have made significant progress in tasks such as image captioning, visual question answering, and image-text retrieval in financial and medical scenarios. A typical VLM usually consists of three parts: a visual encoder, a projector, and a language model. These models generally acquire multimodal understanding capabilities through fine-tuning on large-scale image-text datasets; representative works include LLaVA and Qwen-VL.

[0003] However, in advanced cognitive tasks involving complex logical reasoning, mathematical calculations, or multi-step derivations, existing visual language models still lag far behind large text-based language models (LLMs) of similar size. The root cause of this problem lies in the fact that the current training paradigm of VLMs relies excessively on the fine-tuning process of multimodal supervision signals, which has several key limitations.

[0004] First, high-quality multimodal inference data is extremely scarce and labeling is costly, making it difficult for models to fully master deep inference patterns through conventional training. Second, the inference capabilities introduced during fine-tuning often conflict with the original visual perception capabilities, easily leading to the "catastrophic forgetting" phenomenon, where the model improves inference performance but impairs its accurate understanding of image content and its ability to capture details. Third, existing model architectures often directly inject visual information into the input layer or the first few layers of the language model, with inference capabilities mainly relying on the language model's own structure. However, due to the lack of targeted training mechanisms, the later layers have failed to effectively develop stable chain-of-thought (CoT) generation capabilities.

[0005] While pure text-based large language models have demonstrated powerful zero-shot or multi-step reasoning capabilities in mathematical and logical tasks, these capabilities cannot be directly transferred to visual language models due to modal heterogeneity and differences in training objectives, forming "capability silos." Current technologies have not yet provided an efficient, low-cost, and training-free method to effectively inject mature text reasoning capabilities into the language components of VLMs, thereby compensating for their structural shortcomings in complex reasoning tasks.

[0006] Therefore, how to break through the limitations of the traditional fine-tuning paradigm and effectively enhance the complex logic and mathematical reasoning capabilities of visual language models without damaging the original visual perception capabilities, so as to meet users' performance requirements for the models, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] The embodiments of the present invention provide a method, apparatus, device and medium for enhancing the reasoning ability of a visual language model, aiming to solve the technical problem that the existing multimodal models are insufficient in complex logic and mathematical reasoning tasks.

[0008] In a first aspect, embodiments of the present invention provide a method for enhancing the reasoning ability of a visual language model. The method includes: acquiring a language model component and a source reasoning language model of a target visual language model, wherein the language model component and the source reasoning language model are based on the same pre-trained base model; extracting a first capability increment between the language model component and the pre-trained base model through parameter difference, and extracting a second capability increment between the source reasoning language model and the pre-trained base model; performing weighted fusion based on the first capability increment and the second capability increment to generate a target fused language model; and replacing the language model component with the target fused language model to form an enhanced visual language model.

[0009] Secondly, embodiments of the present invention also provide a visual language model reasoning ability enhancement device for performing the visual language model reasoning ability enhancement method as described above.

[0010] Thirdly, embodiments of the present invention also provide a computer device, the computer device including a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program stored in the memory to perform the steps of the above-described method for enhancing the reasoning ability of visual language models.

[0011] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, can implement the steps of the above-described method for enhancing the reasoning ability of a visual language model.

[0012] Compared with the prior art, the beneficial effects of the present invention are: In the technical solution of this invention, the method for enhancing the reasoning ability of a visual language model selects language components of a target visual language model and a source reasoning language model based on the same pre-training base. It then uses parameter difference to extract the first and second capability increments of both relative to the base model, weightedly fuses the two capability increments to generate a target fused language model, and uses this model to replace the original language components, thus constructing an enhanced visual language model. This method significantly improves the performance of visual language models in complex logic and mathematical reasoning tasks. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart of the method for enhancing the reasoning ability of a visual language model provided by the present invention; Figure 2 The first sub-flowchart of the visual language model reasoning ability enhancement method provided by the present invention; Figure 3 The second sub-flowchart of the visual language model reasoning ability enhancement method provided by the present invention; Figure 4 The third sub-flowchart of the visual language model reasoning ability enhancement method provided by the present invention; Figure 5 The fourth sub-flowchart of the visual language model reasoning ability enhancement method provided by the present invention; Figure 6 The fifth sub-flowchart of the visual language model reasoning ability enhancement method provided by the present invention; Figure 7 The sixth sub-flowchart of the visual language model reasoning ability enhancement method provided by the present invention; Figure 8 A schematic block diagram of the unit of the visual language model reasoning ability enhancement device provided by the present invention; Figure 9 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0017] It should also be understood that the terminology used in this specification is for the purpose of describing embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0018] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0019] In order to address the technical problem that existing multimodal models are insufficient in complex logic and mathematical reasoning tasks, this invention discloses a method for enhancing the reasoning ability of visual language models.

[0020] Reference Figures 1 to 7 The method for enhancing the reasoning ability of the visual language model includes the following steps: S110. Obtain the language model component and source inference language model of the target visual language model, wherein the language model component and source inference language model are based on the same pre-trained base model. S120. Extract the first capability increment between the language model component and the pre-trained base model through parameter difference, and extract the second capability increment between the source inference language model and the pre-trained base model. S130. Based on the first capability increment and the second capability increment, a weighted fusion is performed to generate a target fusion language model; S140. Replace the language model component with the target fusion language model to form an enhanced visual language model.

[0021] Taking LLaVA-8B as an example, the target visual language model consists of CLIP-ViT as the visual encoder, a linear projector, and a large language model component based on LLaMA-3-8B, which is the main part to be replaced. The source inference language model is Dart-Math-Prop2diff, a pure text language model fine-tuned on a large-scale mathematical problem dataset. It also uses LLaMA-3-8B as the pre-trained base model, thus having a completely consistent parameter space structure with the language model component of the target visual language model, satisfying the prerequisite for cross-model parameter operations.

[0022] After acquiring the two models mentioned above, the next step is to extract the capability increment. The parameters of the original LLaMA-3-8B model are loaded as the parameters of the pre-trained base model. Through layer-by-layer, parameter-by-parameter subtraction, the difference between the language model component parameters of the target visual language model and the parameters of the pre-trained base model is calculated to obtain the first capability increment. This increment reflects the knowledge shift required to evolve from pure text understanding capabilities to support image-text alignment and multimodal semantic understanding. Similarly, the difference between the parameters of the source inference language model and the parameters of the pre-trained base model is calculated to obtain the second capability increment. This increment encodes specialized capabilities for complex mathematical reasoning, symbolic computation, and thought chain generation.

[0023] Subsequently, the first and second capability increments are weighted and fused, specifically using a linear combination method. For example, the weighting coefficient λ is set to 0.9, meaning that when fusing the task vector, the first capability increment contributes 90% and the second capability increment contributes 10% to generate the fused task vector. This fused task vector is then added to the pre-trained base model parameters to reconstruct the target fused language model.

[0024] Finally, the target fusion language model is loaded into the original LLaVA-8B architecture to replace its original language model components, while keeping the parameters of the visual encoder and projector frozen, thus forming a complete enhanced visual language model.

[0025] In practical deployments, this enhanced visual language model can receive image inputs including charts, geometric figures, and mathematical expressions, and combine them with text questions to generate answers that include multi-step derivation processes, significantly improving performance on multimodal mathematical reasoning benchmarks such as MathVista and MathVerse.

[0026] Furthermore, differentiated fusion weights can be set at different network levels. For example, higher weighting coefficients can be used in the shallow network of the language model to preserve the original visual semantic alignment ability; while the weighting coefficients can be reduced in the medium and deep network to enhance the injection strength of mathematical reasoning ability, thereby achieving fine decoupling and collaborative optimization of perception and reasoning abilities.

[0027] In one embodiment, step S110 includes: S111. Select a visual language model whose reasoning ability needs to be enhanced as the target visual language model. S112. Select a plain text language model that has been fine-tuned for logical or mathematical reasoning tasks as the source reasoning language model. S113. Confirm that the pre-training basis of the source inference language model is the same as the pre-training basis of the language model component of the target visual language model.

[0028] Target visual language models are multimodal models that require enhanced reasoning capabilities. Mainstream architectures such as LLaVA-8B, Shikra, or Qwen-VL typically employ a complete three-part structure: a visual encoder, a projector, and a language model component. This allows them to receive joint image and text input and generate natural language responses. The language model component of such models is usually adapted and fine-tuned from a large open-source language model.

[0029] Next, a pure text language model that has been specifically fine-tuned on logical or mathematical reasoning tasks is selected as the source reasoning language model, such as the Dart-Math series of models, which has been supervised and fine-tuned on a large number of mathematical question-answering data, such as MATH, AMC, and AIME, and has excellent thought chain generation and symbolic reasoning capabilities.

[0030] Once selected, it is confirmed that the pre-training basis of the source inference language model is consistent with the pre-training basis upon which the language model components of the target visual language model rely. For example, both may use LLaMA-3-8B as their initial parameter source. This step can be verified by comparing model release information, configuration files, or parameter hash values. Only when it is confirmed that both have the same pre-training basis can subsequent capability increment extraction and fusion operations proceed to ensure parameter space alignment and avoid fusion failure or semantic confusion due to differences in architecture or vocabulary. This step constitutes the fundamental constraint of the entire method, ensuring the technical feasibility of cross-model knowledge transfer.

[0031] In practical engineering deployments, to improve system flexibility, a base model compatibility registry can be constructed to pre-register all supported source inference models and their corresponding pre-trained bases. For example, multiple combinations such as "dart-math-llama3-8b-uniform" (base: LLaMA-3-8B) and "logic-llm-qwen-7b" (base: Qwen-7B) can be registered simultaneously. When a user selects a target visual language model, the system automatically matches the list of source inference models with the same base and prompts for available options. This mechanism significantly improves the scalability and automation of the method in real-world scenarios, making it suitable for building cloud service systems that support multi-domain inference enhancements. Furthermore, all matching logic is based on publicly available model metadata and does not involve any new training or parameter modifications.

[0032] In one embodiment, step S120 includes: S121. Load the pre-trained base model parameters that are of the same origin as the language model component of the target visual language model; S122. By calculating the difference between the parameters of the language model component of the target visual language model and the parameters of the pre-trained base model, a first capability increment representing the multimodal perception and language alignment capability is obtained. S123. By calculating the difference between the parameters of the source reasoning language model and the parameters of the pre-trained base model, a second capability increment representing the ability to perform complex logic and mathematical reasoning is obtained.

[0033] First, load the parameters of a pre-trained base model that is derived from the language model component of the target visual language model. These pre-trained base model parameters are derived from the original large language model that has not undergone any downstream task fine-tuning, such as the official release of LLaMA-3-8B, which serves as a common reference benchmark for parameter differencing operations. This base model must share the same network architecture, number of layers, hidden dimensions, and vocabulary structure with the language model component of the target visual language model and the source inference language model to ensure that the parameters can be aligned one-to-one.

[0034] Subsequently, the first capability increment is extracted. The parameters of the language model components in the target visual language model are subtracted layer by layer and tensor by tensor from the parameters of the pre-trained base model. Specifically, a deep learning framework is used to read the complete parameter dictionaries of the language components and base models of the target visual language model, align the corresponding parameter tensors by layer name, and calculate the difference. The set of these difference tensors is the first capability increment: in, This is the parameter set for the language model components of the target visual language model. This is the original parameter set of the pre-trained base model. This increment represents the language model component's capabilities regarding image-text alignment, visual-semantic mapping, and cross-modal contextual response, achieved through fine-tuning via multimodal instructions, building upon its original text understanding capabilities.

[0035] Next, the second capability increment is extracted. The term-by-term differences between all parameters of the source inference language model and the parameters of the pre-trained base model are calculated in the same manner to obtain the second capability increment. For example, taking a source inference language model, its complete parameter dictionary is loaded and compared with... By performing a step-by-step subtraction operation, the second capability increment is obtained: in, This represents the parameter set of the source reasoning language model. This increment reflects the capability shift of the source model in mathematical reasoning, logical deduction, and step-by-step reasoning problem-solving. Both the first and second capability increments are sets of difference tensors with the same parameter structure as the original model, providing an operable capability representation carrier for subsequent weighted fusion.

[0036] To improve the robustness and interpretability of the difference process, the parameters can be differencingd after the difference process. and Layer normalization or gradient magnitude standardization can be performed to eliminate fusion bias caused by differences in dimensions between layers with different parameters. In addition, the difference vector can be visualized and analyzed, for example, using attention heatmaps or parameter gradient activation maps, to observe in which layers and attention heads the difference in the inference model is significantly higher than that in the perception model, thus providing empirical basis for subsequent layer weight allocation.

[0037] In one embodiment, step S130 includes: S131. Set a fusion weight coefficient, which is used to adjust the relative contributions of the first capability increment and the second capability increment in the fusion process; S132. The first capability increment and the second capability increment are linearly weighted according to the fusion weight coefficient to obtain the fusion task vector; S133. Add the fusion task vector to the pre-trained base model parameters to generate the parameters of the target fusion language model.

[0038] First, a fusion weight coefficient is set. This coefficient is used to precisely adjust the relative contributions of the first and second capability increments in the fusion process, thereby preserving the original perceptual capabilities of the visual language model while appropriately injecting the logical reasoning capabilities of the source inference language model. Experimental verification shows that, to avoid compromising the model's ability to recognize and describe image content, the first capability increment should be given a higher weight, and the second capability increment a lower weight. For example, the fusion weight coefficient for the first capability increment can be set as follows: The fusion weight coefficient for the second capability increment is 0.9, and the fusion weight coefficient for the second capability increment is 0.1.

[0039] Subsequently, the first capability increment and the second capability increment are linearly weighted according to the set fusion weight coefficients to obtain the fused task vector. Specifically, the first capability increment is weighted at each layer of the model. Multiply by 0.9 to increase the second capability. Multiply by 0.1, then add the two together to obtain the weighted difference vector for this layer, which is the fusion task vector: Fusion task vectors Essentially, it is a "hybrid of capabilities," whose numerical distribution retains the visual language model's tendency to understand image semantics while incorporating the pure text reasoning model's sensitivity to logical structure.

[0040] Finally, the fused task vectors are added to the parameters of the pre-trained base model to generate the parameters of the target fused language model. Specifically, the fused task vectors of each layer are... Element-wise superposition of parameters onto the pre-trained base model The final parameters for the fusion language component are obtained from the above. The final generated The parameter set, which is the target fusion language model, has the same structure as the original language model components but possesses a stronger ability to generate thought chains. This parameter set has the same network structure and dimensional configuration as the original language model components and can be directly used for model replacement.

[0041] Furthermore, the configuration method for setting the fusion weight coefficient in S131 is as follows: S1311. In the shallow network of the target fusion language model, the first fusion weight coefficient is adopted, and the capabilities related to image recognition and visual perception are retained first. S1312. In the middle and deep networks of the target fusion language model, a second fusion weight coefficient is adopted to enhance the ability related to logical reasoning and thought chain generation, wherein the first fusion weight coefficient is higher than the second weight fusion coefficient.

[0042] Based on the interpretability analysis of the activation patterns and attention weights of the internal parameters in visual language models and pure text reasoning models, it was found that the semantic encoding capabilities related to image recognition and visual perception in the language model components are mainly distributed in the shallow networks of the Transformer architecture. These parameters are most sensitive to tasks such as image region description, object category recognition, and spatial relationship understanding. Meanwhile, capabilities related to logical reasoning, mathematical calculation, and thought chain generation are mainly concentrated in the mid-to-deep networks, where these layers show significant activation enhancement for symbolic manipulation, causal reasoning, multi-step derivation, and conditional judgment.

[0043] Therefore, when generating the target fusion language model, the fusion weight coefficients are not globally uniform, but are configured hierarchically according to the network layers. In the shallow network of the target fusion language model, the first fusion weight coefficient is used, and its value is set relatively high to prioritize preserving the visual-language alignment ability learned by the language components of the target visual language model in multimodal instruction fine-tuning, and to prevent interference with the stable expression of image semantics due to the introduction of external inference increments. This setting ensures that the model can still accurately identify object attributes and contextual relationships when processing basic visual question answering tasks.

[0044] In the middle and deep networks of the target fusion language model, a second fusion weight coefficient is adopted, with a value lower than that of the first fusion weight coefficient, to significantly enhance the logical reasoning and thought chain generation capabilities carried by the source reasoning language model. This configuration allows for the introduction of more parameter increments contributed by the reasoning model into the deep attention head and feedforward network of the model, enabling the model to spontaneously generate structured reasoning steps when facing complex reasoning problems.

[0045] To achieve more precise control, the boundaries between the perception layer and the inference layer can be predefined according to the structure of the target visual language model in the actual deployment system. For example, the optimal boundary point can be automatically determined by performing cluster analysis on the gradient activation intensity of each layer after model training.

[0046] In one embodiment, step S140 includes: S141. Load the target fusion language model into the target visual language model to replace its language model component, thereby forming the enhanced visual language model; S142. During the deployment and inference phase of the enhanced visual language model, the parameters of the visual encoder and projector of the enhanced visual language model are kept frozen. S143. After the enhanced visual language model receives the graphic input, it generates a response with a multi-step logical derivation process.

[0047] The target fusion language model generated through the aforementioned fusion steps is fully embedded into the overall architecture of the target visual language model by parameter loading, replacing its original language model components, thereby constructing a brand-new multimodal model with enhanced reasoning capabilities, namely, the enhanced visual language model.

[0048] In practice, this process relies on the model state dictionary management mechanism provided by the deep learning framework. Specifically, the complete structure of the target visual language model is first loaded, including its visual encoder, projector, and the original language model components. Then, all parameters of the generated target fusion language model are precisely matched and written into the corresponding language decoder part of the structure, ensuring that the weight tensors of each layer are completely aligned with the original language model components in terms of name, shape, and position. Since the parameter structure of the target fusion language model is consistent with the original language model components, this replacement process does not require modification of the network topology or the introduction of additional adaptation modules, making it simple to operate and highly compatible.

[0049] After model replacement, the deployment and inference phase of the enhanced visual language model begins. During this phase, to ensure the model's stability and robustness in multimodal understanding, it is crucial to keep the parameters of the visual encoder and projector frozen. This means that the weights of these two components are not updated during any input processing, and gradient calculations are not introduced. This freezing mechanism ensures that the image feature extraction process still follows the original training path, accurately capturing visual semantic information such as objects, text, layout, and spatial relationships in the input image. Simultaneously, the projector stably maps these features to a joint representation space understandable by the language model, preventing interference from parameter updates in the language component on the visual pathway. This effectively prevents perceptual degradation or modal misalignment.

[0050] When this enhanced visual language model receives actual input, the input is typically in the form of an image-text pair. The model first encodes the image using a visual encoder to extract high-dimensional visual features. Then, a projector converts these features into a series of discrete visual tokens, which are concatenated with the text tokens corresponding to the question text to form a unified input sequence that is fed into the replaced language model component. Thanks to the strong reasoning capabilities injected into the target fusion language model, this language model component no longer relies solely on simple pattern matching or shallow associations during the decoding process; instead, it can proactively organize and generate responses that include multi-step logical deduction processes.

[0051] In one embodiment, the visual language model reasoning ability enhancement method of the present invention further includes: S150. The enhanced visual language model is evaluated using a multimodal reasoning task, and the fusion weight coefficients of each network layer are adjusted based on the evaluation results.

[0052] After constructing the enhanced visual language model, it is evaluated on multimodal reasoning tasks. This evaluation uses representative publicly available benchmark sets such as MathVista, MathVerse, POPE, or ScienceQA. These sets contain a rich variety of text-based mathematical problems, logical reasoning problems, and scientific knowledge questions, comprehensively assessing the model's performance on complex tasks. Evaluation metrics include, but are not limited to, overall accuracy, the rationality of thought chain generation, resistance to confusion from distracting options, and the accuracy of understanding visual details. By running the enhanced visual language model on the aforementioned benchmark sets and collecting the output results, the balance between improved reasoning ability and preserved visual perception can be quantified.

[0053] Based on the evaluation results, the performance differences of the model across different task types and network layers were analyzed. For example, if the model still showed missing steps or incorrect reasoning in mathematical calculation tasks, it indicated insufficient injection of the second capability increment—the capability of the source inference language model—in the mid-to-deep networks. If errors occurred in recognizing text or graphics in images, it suggested that the proportion of the first capability increment in the shallow networks was too low, resulting in impaired perceptual capabilities. Based on this analysis, the fusion weight coefficients of each network layer were adjusted in reverse during the fusion process. For task-related layers that performed poorly, the weights of the corresponding capability increments were appropriately increased.

[0054] This adjustment process can be performed without retraining the model. Specifically, the parameters of the target fusion language model are recalculated, the updated fusion weight coefficients are applied within the specified network layer range, the parameter difference and weighted fusion operations are re-executed, a new target fusion language model is generated, and the original language model components are replaced again.

[0055] Figure 8 This is a schematic block diagram of a visual language model reasoning ability enhancement device 600 provided in an embodiment of the present invention. Figure 8 As shown, corresponding to the above-described method for enhancing visual language model reasoning ability, the present invention also provides a visual language model reasoning ability enhancement device 600. This device 600 includes a unit for executing the above-described method for enhancing visual language model reasoning ability, and can be configured in a desktop computer, tablet computer, smartphone, or other terminal.

[0056] Specifically, please refer to Figure 8 The visual language model reasoning ability enhancement device 600 includes: The model acquisition unit 610 is used to acquire the language model components and the source inference language model of the target visual language model, wherein the language model components and the source inference language model are based on the same pre-trained base model. The incremental extraction unit 620 is used to extract a first capability increment between the language model component and the pre-trained base model through parameter difference, and to extract a second capability increment between the source inference language model and the pre-trained base model. The weighted fusion unit 630 is used to perform weighted fusion based on the first capability increment and the second capability increment to generate a target fusion language model. The enhanced model forming unit 640 is used to replace the language model component with the target fusion language model to form an enhanced visual language model.

[0057] In one embodiment, the model acquisition unit 610 includes: The target visual language model unit is used to select a visual language model whose reasoning ability needs to be enhanced as the target visual language model. The source reasoning language model unit is used to select a plain text language model that has been fine-tuned for logical or mathematical reasoning tasks as the source reasoning language model. The pre-training basis determination unit is used to confirm that the pre-training basis of the source inference language model is the same as the pre-training basis of the language model component of the target visual language model.

[0058] In one embodiment, the incremental extraction unit 620 includes: The parameter loading unit is used to load pre-trained base model parameters that are homologous to the language model components of the target visual language model. The first capability increment acquisition unit is used to obtain a first capability increment representing multimodal perception and language alignment capability by calculating the difference between the parameters of the language model component of the target visual language model and the parameters of the pre-trained base model. The second capability increment acquisition unit is used to obtain a second capability increment representing complex logic and mathematical reasoning ability by calculating the difference between the parameters of the source reasoning language model and the parameters of the pre-trained base model.

[0059] In one embodiment, the weighted fusion unit 630 includes: A fusion weight coefficient setting unit is used to set a fusion weight coefficient, which is used to adjust the relative contributions of the first capability increment and the second capability increment in the fusion process. The task vector fusion acquisition unit is used to linearly weight the first capability increment and the second capability increment according to the fusion weight coefficient to obtain the fusion task vector. The target fusion language model parameter unit is used to add the fusion task vector to the parameters of the pre-trained base model to generate the parameters of the target fusion language model.

[0060] Furthermore, the fusion weight coefficient setting unit includes: A shallow network processing unit is used in the shallow network of the target fusion language model to adopt the first fusion weight coefficient and preferentially retain the capabilities related to image recognition and visual perception. The mid-to-deep network processing unit is used to employ a second fusion weight coefficient in the mid-to-deep networks of the target fusion language model and enhance the ability related to logical reasoning and thought chain generation, wherein the first fusion weight coefficient is higher than the second weight fusion coefficient.

[0061] In one embodiment, the enhanced model forming unit 640 includes: A model replacement unit is used to load the target fusion language model into the target visual language model, replacing its language model components to form the enhanced visual language model; The parameter freezing unit is used to freeze the parameters of the visual encoder and projector of the enhanced visual language model during the deployment and inference phase of the enhanced visual language model. The response generation unit is used to generate a response with a multi-step logical derivation process after the enhanced visual language model receives graphic input.

[0062] In one embodiment, the visual language model reasoning ability enhancement device 600 further includes: The model adjustment unit is used to evaluate the enhanced visual language model for multimodal reasoning tasks and adjust the fusion weight coefficients of each network layer during the fusion process based on the evaluation results.

[0063] The aforementioned visual language model reasoning ability enhancement device 600 can be implemented as a computer program, which can, for example... Figure 9 It runs on the computer device shown.

[0064] Please see Figure 9 , Figure 9 This is a schematic block diagram of a computer device 500 provided in an embodiment of this application. The computer device 500 can be a terminal or a server. The terminal can be an electronic device with communication functions, such as a desktop computer, tablet computer, or smartphone. The server can be a standalone server or a server cluster composed of multiple servers.

[0065] See Figure 9 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0066] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a method for enhancing visual language model reasoning capabilities.

[0067] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0068] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a method for enhancing the reasoning ability of a visual language model.

[0069] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0070] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of the above method.

[0071] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0072] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0073] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the steps of the above-described method.

[0074] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0075] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0076] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0077] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0078] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0079] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for enhancing the reasoning ability of a visual language model, characterized in that, The method includes: Obtain the language model components and source inference language model of the target visual language model, wherein the language model components and source inference language model are based on the same pre-trained base model; The first capability increment between the language model component and the pre-trained base model is extracted by parameter difference, and the second capability increment between the source inference language model and the pre-trained base model is extracted. A target fusion language model is generated by weighted fusion of the first and second capability increments. An enhanced visual language model is formed by replacing the language model components with the target fusion language model.

2. The method for enhancing the reasoning ability of a visual language model according to claim 1, characterized in that, The step of obtaining the language model components and source inference language model of the target visual language model, wherein the language model components and source inference language model are based on the same pre-trained base model, includes: Select a visual language model whose reasoning ability needs to be enhanced as the target visual language model; Select a plain text language model that has been fine-tuned for logical or mathematical reasoning tasks as the source reasoning language model; It is confirmed that the pre-training basis of the source inference language model is the same as the pre-training basis of the language model component of the target visual language model.

3. The method for enhancing the reasoning ability of a visual language model according to claim 1, characterized in that, The steps of extracting the first capability increment between the language model component and the pre-trained base model through parameter difference, and extracting the second capability increment between the source inference language model and the pre-trained base model, include: Load the parameters of a pre-trained base model that are homologous to the language model component of the target visual language model; By calculating the difference between the parameters of the language model component of the target visual language model and the parameters of the pre-trained base model, a first capability increment representing multimodal perception and language alignment capability is obtained. By calculating the difference between the parameters of the source reasoning language model and the parameters of the pre-trained base model, a second capability increment representing complex logic and mathematical reasoning ability is obtained.

4. The method for enhancing the reasoning ability of a visual language model according to claim 1, characterized in that, The step of generating a target fusion language model by weighted fusion based on the first capability increment and the second capability increment includes: A fusion weighting coefficient is set, which is used to adjust the relative contributions of the first capability increment and the second capability increment in the fusion process; The first capability increment and the second capability increment are linearly weighted according to the fusion weight coefficient to obtain the fusion task vector; The parameters of the target fusion language model are generated by adding the fusion task vector to the parameters of the pre-trained base model.

5. The method for enhancing the reasoning ability of a visual language model according to claim 4, characterized in that, The fusion weight coefficient is configured as follows: In the shallow network of the target fusion language model, a first fusion weight coefficient is adopted, and the capabilities related to image recognition and visual perception are preferentially retained. In the middle and deep networks of the target fusion language model, a second fusion weight coefficient is adopted to enhance the ability related to logical reasoning and thought chain generation. The first fusion weight coefficient is higher than the second weight fusion coefficient.

6. The method for enhancing the reasoning ability of a visual language model according to claim 2, characterized in that, The step of replacing the language model component with the target fusion language model to form an enhanced visual language model includes: The target fusion language model is loaded into the target visual language model, replacing its language model components, to form the enhanced visual language model; During the deployment and inference phase of the enhanced visual language model, the parameters of the visual encoder and projector of the enhanced visual language model are kept frozen; After receiving text and image input, the enhanced visual language model generates a response with a multi-step logical derivation process.

7. The method for enhancing the reasoning ability of a visual language model according to claim 1, characterized in that, The method further includes: The enhanced visual language model is evaluated using a multimodal reasoning task, and the fusion weight coefficients of each network layer are adjusted based on the evaluation results.

8. A device for enhancing the reasoning ability of a visual language model, characterized in that, Used to perform the visual language model reasoning ability enhancement method as described in any one of claims 1 to 7.

9. A computer device, characterized in that, The computer device includes a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, can implement the steps of the method as described in any one of claims 1 to 7.

Images