A large model-oriented complex capability decoupling migration method and system

Abstract

The application relates to the technical field of multimodal large language models, and discloses a complex capability decoupling transfer method and system for a large model. The method comprises the following steps: constructing a positive and negative sample pair for each sub-capability; inputting the sample pair into a target large model, extracting a hidden state vector of a specified Transformer layer, calculating a positive and negative sample difference mean value, and obtaining a positioning guide vector and an inference guide vector; in a word-by-word element generation process of the model, identifying a current generation stage according to a generated word element sequence; when being in a positioning stage, injecting the positioning guide vector into a preset shallow layer to guide the generation of image region coordinates or tool calling content; when being in an inference stage, injecting the inference guide vector into a preset deep layer to guide the generation of an analysis and inference track based on image evidence; and not updating model parameters throughout the process. According to the application, low-cost and controllable transfer of complex capabilities of a large model can be realized without retraining or fine-tuning.

Description

Technical Field

[0001] This invention relates to the field of multimodal large language model technology, specifically to a method and system for decoupling and transferring complex capabilities for large models. Background Technology

[0002] In recent years, Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have demonstrated strong general capabilities in natural language understanding, text generation, visual question answering, graph-text reasoning, tool invocation, and agent tasks. As the model size and training data scale continuously expand, models can produce near-human reasoning, planning, and interpretive outputs in more tasks. However, whether a model possesses a particular complex capability typically depends heavily on pre-training corpora, instruction fine-tuning data, preference alignment data, and the computational resources invested during training. When it is necessary to acquire new complex capabilities for an existing model, a common approach is to collect a large amount of specialized data and perform full-scale fine-tuning, efficient parameter fine-tuning, or distillation training. While these methods are effective, they suffer from high data construction costs, high computational resource consumption during training, long deployment cycles, high requirements for the openness of model parameters, and the potential to damage existing capabilities.

[0003] In multimodal scenarios, these problems are even more pronounced. For example, Visual Chain-of-Thought (Visual CoT) capability requires models not only to identify targets, regions, attributes, and spatial relationships in images, but also to perform multi-step reasoning around visual evidence and, when necessary, output structured trajectories containing localization, local observation, tool invocation, or intermediate thinking processes. To acquire this capability, it is typically necessary to construct a large amount of labeled data containing images, visual localization processes, reasoning processes, and final answers, and then train or distill it using a large multimodal model. Due to the complexity of visual chain-of-thought data construction, high annotation costs, and diverse forms of reasoning trajectories, directly training a model with stable visual chain-of-thought capabilities often requires a significant investment of data and computing power.

[0004] Training-free capability transfer is therefore of great significance. Training-free capability transfer refers to enabling a target large model to exhibit previously weak or missing capabilities in a specific task by adjusting external control signals or internal representations during the inference phase, without updating the target model's parameters. This type of method can reduce the cost of deploying new capabilities while keeping the main model parameters unchanged, and also reduces the resource waste and risk of capability forgetting caused by repeated training.

[0005] Vector steering is a typical method for regulating model behavior during the inference stage. Its basic idea is that the model possesses directions in its intermediate hidden state space related to a certain type of behavior, style, knowledge, or ability. By constructing positive and negative sample pairs and calculating the difference between the hidden states, a direction vector representing this ability or behavior can be obtained. During the model's inference generation process, this direction vector is injected with a certain strength into the hidden states of a specified layer, thus guiding the model's output towards the target behavior. Existing research and practice have shown that vector steering can be used in scenarios such as language style transfer, memory or knowledge regulation, inference behavior enhancement, and task preference adjustment, and may produce transfer effects across tasks, modalities, or models.

[0006] However, existing vector guidance methods still have shortcomings in the transfer of complex abilities. First, complex abilities are usually not a single behavioral direction, but rather composed of multiple sub-abilities. For example, visual chain thinking includes at least visual localization or grounding ability, fine-grained observation ability, logical reasoning ability, and answer organization ability. If a single ability vector is calculated using only a set of overall comparisons, the vector will mix multiple ability components, making it difficult to control precisely during reasoning. Second, vector guidance is sensitive to the model layer position; different tasks, different datasets, and different ability components may correspond to different effective guidance layers. Visual perception or localization ability may be more suitable for guidance at shallower or mid-levels, while abstract reasoning ability may be more suitable for guidance at deeper or mid-levels. Third, complex abilities are staged in a single generation process. For example, the model outputs... The field primarily performs inference organization and output. When locating fields, visual grounding or tool calls are primarily performed. If multiple capability vectors are injected simultaneously in all generation steps and all relevant layers, it may cause mutual interference between vectors, or even prevent the generation of superposition gain.

[0007] Therefore, there is an urgent need for a general method for transferring complex capabilities to large models and multimodal large models. This method can decompose complex capabilities into several sub-capabilities under training-free conditions, construct comparison pairs, extract sub-capability direction vectors, and selectively guide them according to the output stage and layer position during the model inference stage, thereby achieving low-cost, controllable, and scalable transfer of complex capabilities. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides a method and system for decoupling and transferring complex capabilities in large models. Without requiring retraining or fine-tuning of the target large model's parameters, it achieves complex capability transfer by constructing capability sample pairs, extracting capability direction vectors, and providing phased vector guidance during the inference stage. This invention is applicable to general complex capability transfer in large models and multimodal large models, and is particularly suitable for scenarios such as complex visual question answering, visual reasoning, image region localization, local visual evidence retrieval, and visual chain inference trajectory generation.

[0009] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for decoupling and transferring complex capabilities for large models, comprising: Decouple visual reasoning ability into visual evidence localization sub-ability and visual evidence reasoning sub-ability; For each sub-capability, construct positive and negative sample pairs. The positive and negative sample pairs contain the same image and text questions. The positive sample contains a prompt that induces the target large model to generate the corresponding sub-capability behavior output, while the negative sample weakens the behavior output. Input positive and negative sample pairs into the target large model, extract the hidden state vector of the specified Transformer layer, calculate the difference between the hidden states of positive and negative samples and calculate the mean, and obtain the localization guidance vector and the inference guidance vector. In the reasoning process of generating output word by word in the target large model, the current generation stage is identified based on the generated word sequence; When in the visual evidence localization stage, shallow localization guidance vectors are injected into the preset to guide the target large model to generate regional localization, local observation or tool call content. When in the visual evidence reasoning stage, deep reasoning guidance vectors are injected into the preset to guide the target large model to generate visual analysis, reasoning trajectory or evidence interpretation content. The parameters of the target large model are not updated throughout the entire process.

[0010] In one embodiment, the visual evidence localization sub-capability is used to guide the target large model to locate target regions, object locations, local details, or spatial relationships related to the text question in the image before answering, and to generate region localization, local observation, or tool call content; the visual evidence reasoning sub-capability is used to guide the target large model to generate visual analysis, reasoning trajectory, or evidence interpretation content based on the located visual evidence.

[0011] In one embodiment, constructing positive and negative sample pairs for each sub-capability specifically includes: Constructing a set of localization capability sample pairs for visual evidence localization sub-capabilities ;in, express The number of localization capability sample pairs in the data, each containing the same image and text question, and positive localization capability samples. The large model outputs content related to the target's location, local observation, or tool invocation, and its localization capability is reflected in negative samples. Enables the target large model to directly output the answer; Constructing a set of reasoning ability sample pairs for visual evidence reasoning sub-ability ;in, express The number of reasoning ability sample pairs is medium, each reasoning ability sample pair contains the same image and text questions, and the number of positive reasoning ability samples is medium. The target model outputs visual analysis, reasoning trajectory, or evidence interpretation, while negative samples demonstrate reasoning ability. Weaken or eliminate explicit reasoning processes.

[0012] In one embodiment, the calculation process for the positioning guide vector includes: The first Positive positioning capability samples in a pair of positioning capability samples and localization capability negative samples Input the target large model respectively, and at the first Layer extraction of the target large model before generating region localization, local observation, or tool invocation of content: ; ; in, , They are respectively , The corresponding location capability is hidden. The target large model is represented by the first The layer's output function; This indicates a positive prompt or trajectory prefix for the location content, local observation content, or tool call content generated by the large model of the induced target; This indicates a negative prompt or trajectory prefix that weakens the content of the area location, the content of the local observation, or the content called by the tool; This indicates the position of keyword meta tags before the content generated by the target large model, local observation, or tool invocation. Indicates the first Image and text issues in a pair of localization capability samples; No. The positioning capability sample pairs in the target large model Layer positioning capability hides state difference for: ; For all The mean of the hidden state differences of the localization capabilities of each pair of localization capability samples is used to obtain the target large model. Layer positioning guide vector : .

[0013] In one embodiment, the calculation process of the inference guidance vector includes: The first Positive inference ability samples from each inference ability sample pair Negative samples of reasoning ability Input the target large model respectively, and at the first The hidden states extracted from the target large model before generating visual analysis, reasoning trajectories, or evidence interpretations are denoted as follows: ; ; in, , They are respectively , The corresponding reasoning ability is hidden. The target large model is represented by the first The layer's output function, This indicates a positive cue or trajectory prefix that guides the visual reasoning process generated by the large model of the target; Negative cues or trajectory prefixes that indicate a weakening of the visual reasoning process; This indicates the keyword meta-position before the visual analysis content, reasoning trajectory content, or evidence interpretation content generated by the target large model. Indicates the first Image and text issues in a pair of localization capability samples; No. The reasoning ability sample pairs in the target large model Layer reasoning ability hidden state difference for: ; For all The mean of the hidden state differences of reasoning ability for each pair of reasoning ability sample pairs is used to obtain the target large model. Layered inference guidance vector : .

[0014] In one embodiment, the step of identifying the current generation stage based on the generated word sequence during the inference process of generating output per word in the target large model specifically includes: The generation phase includes the visual evidence localization phase, the visual evidence reasoning phase, and the answer generation phase. Based on the generated word sequence The current generation stage is determined by parsing special tags, regular rules, or finite state machines. Specifically, when the generated word sequence is located in the tool call field or the area location field, it is determined to be in the visual evidence location stage; when the generated word sequence is located in the visual reasoning field or the visual analysis field, it is determined to be in the visual evidence reasoning stage; when the generated content is located in the answer field, the current generation stage is determined to be in the answer generation stage.

[0015] In one embodiment, the shallow injection of the positioning guidance vector in the preset layer specifically includes: During the current generation phase Belongs to the visual evidence localization stage and the target is a large model. The layer belongs to the preset set of shallow to medium-level guiding layers. At that time, for the first The original positioning capabilities of the layer are hidden in the execution of the overlay: ; in, The target large model is represented by the first The layer in the first The hidden state of the original localization capability of each generation step. The hidden state for localization capability after injecting the guiding vector. For the target large model Layer positioning guide vector, To determine the intensity of visual evidence.

[0016] In one embodiment, the step of injecting the inference guidance vector deep into the preset layer specifically includes: During the current generation phase Belongs to the visual evidence reasoning stage and the target is a large model. The layer belongs to the pre-defined set of medium-deep guide layers. At that time, for the first The original reasoning ability of the layer is hidden in the execution of the superposition: ; in, The target large model is represented by the first The layer in the first The hidden state of the original reasoning ability of each generation step. The hidden state for inference capabilities after the injection of the guiding vector. For the target large model Layered reasoning guidance vectors, Intensity of reasoning based on visual evidence.

[0017] In one embodiment, the process further includes norm calibration of the hidden state after the injection of the guide vector, specifically including: calculating the ratio of the L2 norm of the hidden state after the injection of the guide vector to the L2 norm of the original hidden state, scaling the hidden state after the injection of the guide vector with the ratio to obtain the calibrated hidden state; and continuing to input the calibrated hidden state into the subsequent network layers of the target large model.

[0018] Secondly, the present invention provides a complex capability decoupling and transfer system for large models, used to implement the method of any embodiment of the first aspect, comprising: The capability decoupling module is used to decouple the visual reasoning capability into the visual evidence localization sub-capability and the visual evidence reasoning sub-capability. The contrast sample construction module is used to construct positive and negative sample pairs for each sub-capability. The positive and negative sample pairs contain the same image and text questions. The positive sample contains a prompt that induces the target large model to generate the corresponding sub-capability behavior output, while the negative sample weakens the behavior output. The guidance vector extraction module is used to extract the hidden state vector of a specified Transformer layer from the input target large model with positive and negative samples, calculate the difference between the hidden states of positive and negative samples and calculate the mean, and obtain the localization guidance vector and the inference guidance vector. The stage identification module is used to identify the current generation stage based on the generated word sequence during the inference process of generating output word by word in the target large model; The vector injection module is used to inject shallow positioning guidance vectors in the preset to guide the target large model to generate regional positioning, local observation or tool call content when it is in the visual evidence localization stage; and to inject deep reasoning guidance vectors in the preset to guide the target large model to generate visual analysis, reasoning trajectory or evidence interpretation content when it is in the visual evidence reasoning stage. The system does not update the parameters of the target large model throughout the entire process.

[0019] The system and method in this invention correspond to each other; the specific technical solutions applicable to the method are also applicable to the system.

[0020] Compared with the prior art, the beneficial technical effects of the present invention are: 1. Reduce the cost of transferring complex capabilities. This invention does not require retraining or fine-tuning the target large model. Capability transfer is achieved solely through the injection of hidden state vectors during the inference stage, which can significantly reduce the costs of data annotation, computing power consumption, and model deployment.

[0021] 2. Improve the controllability of complex capability transfer. By decoupling complex capabilities into multiple sub-capabilities and calculating sub-capability vectors separately, the capabilities exhibited by the model at different stages can be controlled more precisely, avoiding the mixing of multiple behavioral directions by a single overall vector.

[0022] 3. Reduce interference from multiple capability vectors. Through stage recognition and phased guidance, a positioning guidance vector is injected when perceptual positioning capability is needed, and a reasoning guidance vector is injected when reasoning capability is needed, avoiding the indiscriminate superposition of multiple vectors in all words and all layers, which leads to unstable output.

[0023] 4. Improved layer location adaptability. This invention allows for the selection of different layers or sets of layers for different sub-capabilities, adapting to the differences in the representation hierarchy of capabilities such as visual perception, grounding, and semantic reasoning within the model.

[0024] 5. Keep the main parameters of the target model unchanged. Since the model weights are not modified, this invention can enable, disable, or switch different capability vectors on the same model as needed, facilitating rapid deployment and rollback, and reducing damage to the original model capabilities.

[0025] 6. It has good general scalability. The "sub-capability sample pair construction - guided vector calculation - staged vector injection" process of this invention does not depend on a specific model or a specific task, and can be extended to large text models, multimodal large models and other generative Transformer models.

[0026] 7. It has a direct effect on visual chain thinking scenarios. By injecting positioning guidance vectors and inference guidance vectors in a layered and phased manner, the target multimodal model can more stably generate output trajectories containing visual grounding, tool invocation, and multi-step inference without the need for special training on visual chain thinking data, thereby enhancing the interpretability and problem-solving ability in complex visual scenarios. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the overall process of transferring decoupled visual chain thinking ability according to the present invention. Figure 2 This is a schematic diagram illustrating the construction of sub-capability sample pairs and the calculation of guiding vectors in this invention. Figure 3 This is a schematic diagram illustrating the hierarchical and phased guidance of visual chain thinking ability in this invention. Figure 4 This is a flowchart of the method of the present invention. Detailed Implementation

[0028] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] like Figure 4As shown, a method for decoupling and transferring complex capabilities for large models in this invention includes the following steps: S1 decouples visual reasoning ability into visual evidence location sub-ability and visual evidence reasoning sub-ability; S2, construct positive and negative sample pairs for each sub-capability. The positive and negative sample pairs contain the same image and text questions. The positive sample contains a prompt that induces the target large model to generate the corresponding sub-capability behavior output, and the negative sample weakens the behavior output. S3, input positive and negative samples into the target large model, extract the hidden state vector of the specified Transformer layer, calculate the difference between the hidden states of positive and negative samples and calculate the mean, and obtain the localization guidance vector and the inference guidance vector. S4, In the reasoning process of generating output word by word in the target large model, the current generation stage is identified based on the generated word sequence; S5, when in the visual evidence localization stage, shallow localization guidance vectors are injected into the preset to guide the target large model to generate regional localization, local observation or tool call content; when in the visual evidence reasoning stage, deep reasoning guidance vectors are injected into the preset to guide the target large model to generate visual analysis, reasoning trajectory or evidence interpretation content; the parameters of the target large model are not updated throughout the process.

[0030] The following description uses the transfer of visual reasoning ability in a complex visual question answering task as a specific example, but the present invention is not limited to this example.

[0031] In this embodiment, the target large model is a multimodal large model, and the input includes the image to be analyzed. and the natural language problem for this image The output includes the final answer corresponding to the image content. In complex visual question answering tasks, the target large model not only needs to generate answers but also needs to identify key targets, local regions, spatial relationships, or fine-grained visual attributes related to the question in the image and complete reasoning based on this visual evidence. Therefore, this invention constructs sub-capability sample pairs in the visual question answering scenario, extracts visual evidence localization vectors and visual evidence inference vectors, and provides staged vector guidance when the target large model generates different output stages, enabling the target large model to acquire visual reasoning capabilities without retraining or fine-tuning parameters.

[0032] 1. Identify the complex capabilities to be migrated and decouple the sub-capabilities.

[0033] (1) Identify the complex capabilities to be transferred in the target task: In this embodiment, the complex capability to be transferred is the visual reasoning capability in a complex visual question answering task. This capability is used to enable large target models to be displayed in the received image. and the problem Subsequently, it can generate region localization, local observation, tool invocation, visual analysis, reasoning trajectory, evidence interpretation, and final answer around key visual evidence in the image.

[0034] (2) Based on the generation process of complex visual question answering tasks, visual reasoning ability is incorporated. Decoupling into at least two sub-capabilities: ; in, This indicates the ability to locate visual evidence. This indicates the ability to reason based on visual evidence.

[0035] Visual evidence location capability Used to guide the target large model to locate images before answering questions. The problem The relevant target area, object location, local details, or spatial relationships are identified, and content for area positioning, local observation, or tool invocation is generated.

[0036] Visual evidence reasoning sub-ability This is used to guide the target large model to generate intermediate visual analysis, reasoning trajectories, or evidence interpretations based on located or observed visual evidence, and to establish a connection between visual evidence and the final answer.

[0037] (3) Set the corresponding guidance layer set and guidance strength for each sub-capability: For the visual evidence localization sub-capability, the guiding layer set is defined as follows: The guiding strength is .

[0038] For the visual evidence reasoning sub-capability, the guiding layer set is defined as follows: The guiding strength is .

[0039] in, This typically corresponds to the shallow or visual fusion layers in the target large model that are related to visual perception, region localization, or local information extraction. This typically corresponds to the mid-to-deep layers in the target large model that are related to semantic integration, logical reasoning, or answer organization. The specific layer number can be determined based on the structure of the target large model and a small number of validation samples.

[0040] 2. Construct sub-capability comparison samples in visual question answering scenarios.

[0041] (1) Constructing a sample pair set of visual evidence localization capabilities: Targeting visual evidence localization capabilities Construct a set of localization capability samples : ; in, This indicates the number of localization capability sample pairs. Each sample pair contains the same image. and the problem However, positive and negative samples differ in whether they induce the target large model to generate visual evidence to locate content.

[0042] Positive samples Includes images ,question This includes prompts or trajectories that can induce the target large model to output "region localization, local observation, or tool invocation." For example, positive samples require the target large model to generate question-related image regions, local observation instructions, or tool invocation parameters before answering.

[0043] negative samples Contains the same image and the problem However, it does not require the target large model to generate region localization, local observation or tool call content, but directly generates the answer or only generates ordinary text explanation.

[0044] (2) Construct a sample pair set of visual evidence reasoning sub-ability: Sub-ability for reasoning based on visual evidence Construct a set of reasoning ability sample pairs : ; in, The number of sample pairs representing reasoning ability. Each sample pair also contains an image. and the problem .

[0045] Positive samples This is used to induce large target models to generate intermediate analysis processes based on visual evidence, such as describing relationships between targets in an image, explaining the connection between local visual evidence and the answer, or forming structured visual reasoning trajectories.

[0046] negative samples Reduce or eliminate explicit visual reasoning processes, for example, by only retaining tool calls and the final answer, or by directly outputting the final answer.

[0047] (3) When constructing the above sample pairs, the positive and negative samples should keep the image content, question semantics and final answer consistent as much as possible, so that the sample differences are mainly reflected in whether they have the corresponding visual sub-capabilities. In this way, the vectors calculated later correspond to the specific technical behaviors in complex visual question answering tasks, namely visual evidence localization behavior and visual evidence reasoning behavior, rather than just reflecting the differences in abstract language style.

[0048] 3. Extract the hidden states and compute the visual sub-capability vector.

[0049] (1) Calculate the visual evidence localization guidance vector: Positioning visual evidence as positive samples and negative samples Input the target large model separately. For the first... The nth sample pair, in the nth... The hidden state of the target large model before the generation of region localization fields, local observation fields, or tool call fields is extracted is denoted as follows: ; ; in, The target large model is represented by the first The layer's output function; This indicates a positive prompt or trajectory prefix for the visual evidence generated by the large model to guide the target to locate the content. This indicates a negative suggestion or trajectory prefix that weakens the visual evidence used to locate the content; This indicates the position of keyword meta tags before the content generated by the target large model, local observation, or tool call.

[0050] No. The sample pair at the th ... The difference in the hidden state of the layer's positioning capability is: ; The mean of the differences among all positioning capability samples is used to obtain the first... Visual evidence localization guide vectors for layers: ; in, This is used to enhance the tendency of large target models to generate region localization, local observation, or tool-calling content in complex visual question answering tasks.

[0051] (2) Calculate the visual evidence reasoning guidance vector: Positive samples of visual evidence reasoning and negative samples Input the target large model separately. For the first... The sample, in the ... The hidden states of the target large model before the generation of visual inference fields or intermediate analysis fields are extracted are denoted as follows: ; ; in, This indicates a positive hint or trajectory prefix that represents the intermediate visual analysis process, reasoning trajectory, or evidence interpretation content generated by the large model of the target; Negative cues or trajectory prefixes that indicate a weakening of the visual reasoning process; This indicates the keyword meta-position before the content of visual analysis, reasoning trajectory, or evidence interpretation generated by the target large model.

[0052] No. The sample pair at the th ... The difference in hidden state of the reasoning ability of the layer is: ; The mean of the differences between all pairs of reasoning ability samples is taken to obtain the first... Visual evidence reasoning guidance vector for the layer: ; in, This is intended to enhance the tendency of large target models to generate image-based visual analysis and reasoning trajectories in complex visual question answering tasks.

[0053] Optionally, for and L2 normalization, centering, scaling, principal component filtering, or truncation are performed to improve vector stability.

[0054] If you need to select the guiding position from multiple layers in a large target model, you can do so in the candidate layer set. Calculate separately and The set of guiding layers was determined by using a small number of complex visual question-answering validation samples. and For example, visual evidence localization guidance vectors can be preferentially placed in the visual fusion layer, the shallow to medium layer, or the layer with strong local visual information; visual evidence reasoning guidance vectors can be preferentially placed in the semantic integration layer, the deep to medium layer, or the layer with strong reasoning effect.

[0055] 4. Construct a recognition mechanism for the complex visual question-answering output stage.

[0056] (1) In the process of generating the target large model inference, the generation stage is divided according to the semantic or structural labels of the output content. For complex visual question answering tasks, it can be divided into at least the following stages: Visual evidence localization stage : Target large model generation area localization, local observation, tool calls or other visual evidence acquisition content.

[0057] Visual evidence reasoning stage The target large model generates intermediate visual analysis, reasoning trajectories, or evidence interpretation content.

[0058] Answer generation stage The target large model generates the final answer based on image evidence and reasoning results.

[0059] (2) Based on the generated word sequence Determine the current generation stage Stage determination can be implemented based on special tags, regular expressions, finite state machines, output templates, decoding prefixes, tool call formats, or external controllers.

[0060] For example, when the generated content is located in a tool call (<tool call> ), regional positioning ( <region>), bounding box ( <bbox>When the field contains information such as ) or , it is determined that the current stage belongs to the visual evidence localization stage. When the generated content is in the thinking ( <think>Visual reasoning<visual reasoning> If the current stage is within the visual evidence reasoning stage, or within the visual analysis field, then the current stage is determined to be within the visual evidence reasoning stage. When the generated content is located in the answer ( <answer>When the field is within the specified range, it is determined that the current stage belongs to the answer generation stage. .

[0061] (3) Establish the mapping relationship between the generation stage and the visual sub-capability vector: , ; ,like and ; ,like Or the current layer does not require guidance in, This represents the guiding vector that needs to be activated in the current generation phase.

[0062] 5. Perform phased vector guidance during the inference phase.

[0063] (1) During the current generation stage This belongs to the visual evidence localization stage. And the current layer Belongs to the visual evidence localization guidance layer set At that time, for the first The original hidden states of the layer are stacked as follows: ; The target large model is represented by the first The layer in the first The hidden state of the original localization capability of each generation step. The hidden state for localization capability after injecting the guiding vector. For the target large model Layer positioning guide vector, The strength of the guidance for visual evidence location is used to control the tendency of the target large model to generate image region location, local observation or tool call content.

[0064] (2) In order to maintain the stability of the hidden state norm and prevent abnormal amplitude of the hidden state after injection, the norm of the localization capability hidden state after the injection of the guiding vector is calibrated: ; in, Represents the L2 norm. This indicates the hidden state of the calibrated localization capability. The calibrated hidden state of the localization capability is then input into subsequent layers of the target large model or into the decoding head.

[0065] (3) During the current generation stage Belongs to the visual evidence reasoning stage And the current layer Belongs to the visual evidence reasoning guidance layer set At that time, for the first The original hidden states of the layer are stacked as follows: ; in, The target large model is represented by the first The layer in the first The hidden state of the original reasoning ability of each generation step. The hidden state for inference capabilities after the injection of the guiding vector. For the target large model Layered reasoning guidance vectors, The strength of visual evidence reasoning guidance is used to control the tendency of the target large model to generate intermediate visual analysis processes, reasoning trajectories, or evidence interpretation content.

[0066] Perform norm calibration: ; This represents the calibrated hidden state of inference capability. The calibrated hidden state of inference capability continues to be input into subsequent layers of the target large model or the decoding head.

[0067] (4) During the current generation stage This belongs to the answer generation stage. In such cases, vector injection can be turned off, or visual reasoning guidance vectors can be retained with less intensity to avoid the final answer stage being affected by an overly strong intermediate reasoning format.

[0068] (5) If the current stage does not belong to any visual sub-capability corresponding stage, or the current layer does not belong to any visual sub-capability guidance layer set, then the original hidden state remains unchanged.

[0069] (6) Repeat Part 4 and Part 5 until the target large model completes all word generation and obtains a complex visual question answering output containing visual evidence location content, visual reasoning process and final answer.

[0070] 6. Taking the visual chain thinking output format as an example, the target large model output format includes visual reasoning fields, tool call fields, and answer fields, for example: <think> ...< / think> <tool call> ...< / tool call> <answer> ...< / answer> in, <think>The field corresponds to the visual evidence reasoning stage.<tool call> The field corresponds to the visual evidence localization stage. <answer>The field corresponds to the final answer generation stage.

[0071] like Figure 3 As shown, in generating <think>In the field, only in the visual evidence reasoning guidance layer set Injection And perform norm calibration.

[0072] In generation<tool call> When using fields, region location fields, or local observation fields, only in the visual evidence location guidance layer set. Injection And perform norm calibration.

[0073] In generation <answer>When processing fields, the target large model generates the final answer based on the visual evidence localization results and visual reasoning results. This stage may not require vector injection, or may only retain weak visual reasoning guidance, to ensure the answer expression is concise and stable.

[0074] Through the above methods, the target large model is affected when it is necessary to locate image evidence. Guided, when a visual reasoning process needs to be generated. This guides the process, preventing multiple sub-capability vectors from being indiscriminately superimposed in the same generation step and reducing mutual interference between sub-capabilities.

[0075] This invention proposes a training-free complex capability transfer framework. This framework does not require retraining or fine-tuning of the target large model; capability transfer is achieved only during the inference stage based on hidden state vector injection, reducing data construction costs, training costs, and deployment costs.

[0076] This invention proposes a capability modeling approach that decouples complex capabilities. For a target complex capability, it is first broken down into multiple operable sub-capabilities, and then a comparison pair of each sub-capability is constructed, thereby avoiding the mixing of multiple behavioral directions by a single capability vector.

[0077] This invention proposes a phased and layered vector guidance strategy. Based on the stage of the generated content, a corresponding sub-capability vector is selected and guided within the corresponding layer or set of layers, avoiding interference caused by the indiscriminate superposition of multiple sub-capability vectors in the same generation stage.

[0078] The method of this invention is universal and scalable. In addition to visual chain thinking, this invention can also be extended to complex capability transfer scenarios such as tool invocation, mathematical reasoning, planning execution, cross-modal alignment, domain knowledge representation, and specific format adherence; the target large model can be a multimodal large model or other generative models based on the Transformer structure.

[0079] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms "comprising," "including," etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0080] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0081] This invention also provides a system. The system can be a system that uses software (applications), modules, components, servers, clients, etc., employing the methods described in the embodiments of this specification, combined with necessary hardware implementation. Since the implementation schemes and methods for solving the problem are similar, the specific system implementations in the embodiments of this specification can be found in the implementations of the foregoing methods, and repeated details will not be elaborated upon. Although the system is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0082] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0083] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0084] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.< / answer> < / think> < / answer> < / think> < / answer> < / think> < / bbox> < / region>

Claims

1. A method for decoupling and transferring complex capabilities in large models, characterized in that, include: Decouple visual reasoning ability into visual evidence localization sub-ability and visual evidence reasoning sub-ability; For each sub-capability, construct positive and negative sample pairs. The positive and negative sample pairs contain the same image and text questions. The positive sample contains a prompt that induces the target large model to generate the corresponding sub-capability behavior output, while the negative sample weakens the behavior output. Input positive and negative sample pairs into the target large model, extract the hidden state vector of the specified Transformer layer, calculate the difference between the hidden states of positive and negative samples and calculate the mean, and obtain the localization guidance vector and the inference guidance vector. In the reasoning process of generating output word by word in the target large model, the current generation stage is identified based on the generated word sequence; When in the visual evidence localization stage, shallow localization guidance vectors are injected into the preset to guide the target large model to generate regional localization, local observation or tool call content. When in the visual evidence reasoning stage, deep reasoning guidance vectors are injected into the preset to guide the target large model to generate visual analysis, reasoning trajectory or evidence interpretation content. The parameters of the target large model are not updated throughout the entire process.

2. The method for decoupling and transferring complex capabilities for large models according to claim 1, characterized in that, The visual evidence localization sub-capability is used to guide the target large model to locate the target region, object location, local details or spatial relationships related to the text question in the image before answering, and generate region localization, local observation or tool call content; the visual evidence reasoning sub-capability is used to guide the target large model to generate visual analysis, reasoning trajectory or evidence interpretation content based on the located visual evidence.

3. The method for decoupling and transferring complex capabilities for large models according to claim 1, characterized in that, The construction of positive and negative sample pairs for each sub-capability specifically includes: Constructing a set of localization capability sample pairs for visual evidence localization sub-capabilities ;in, express The number of localization capability sample pairs in the data, each containing the same image and text question, and positive localization capability samples. The large model outputs content related to the target's location, local observation, or tool invocation, and its localization capability is reflected in negative samples. Enables the target large model to directly output the answer; Constructing a set of reasoning ability sample pairs for visual evidence reasoning sub-ability ;in, express The number of reasoning ability sample pairs is medium, each reasoning ability sample pair contains the same image and text questions, and the number of positive reasoning ability samples is medium. The target model outputs visual analysis, reasoning trajectory, or evidence interpretation, while negative samples demonstrate reasoning ability. Weaken or eliminate explicit reasoning processes.

4. The method for decoupling and transferring complex capabilities for large models according to claim 1, characterized in that, The calculation process for the positioning guide vector includes: The first Positive positioning capability samples in a pair of positioning capability samples and localization capability negative samples Input the target large model respectively, and at the first Layer extraction of the target large model before generating region localization, local observation, or tool invocation of content: ； ； in, , They are respectively , The corresponding location capability is hidden. The target large model is represented by the first The layer's output function; This indicates a positive prompt or trajectory prefix for the location content, local observation content, or tool call content generated by the large model of the induced target; This indicates a negative prompt or trajectory prefix that weakens the content of the area location, the content of the local observation, or the content called by the tool; This indicates the position of keyword meta tags before the content generated by the target large model, local observation, or tool invocation. Indicates the first Image and text issues in a pair of localization capability samples; No. A pair of positioning capability samples in the target large model Layer positioning capability hides state difference for: ； For all The mean of the hidden state differences of the localization capabilities of each pair of localization capability samples is used to obtain the target large model. Layer positioning guide vector : 。 5. The method for decoupling and transferring complex capabilities for large models according to claim 1, characterized in that, The calculation process of the inference guidance vector includes: The first Positive inference ability samples from each inference ability sample pair Negative samples of reasoning ability Input the target large model respectively, and at the first The hidden states extracted from the target large model before generating visual analysis, reasoning trajectories, or evidence interpretations are denoted as follows: ； ； in, , They are respectively , The corresponding reasoning ability is hidden. The target large model is represented by the first The layer's output function, This indicates a positive cue or trajectory prefix that guides the visual reasoning process generated by the large model of the target; Negative cues or trajectory prefixes that indicate a weakening of the visual reasoning process; This indicates the keyword meta-position before the visual analysis content, reasoning trajectory content, or evidence interpretation content generated by the target large model. Indicates the first Image and text issues in a pair of localization capability samples; No. The reasoning ability sample pairs in the target large model Layer reasoning ability hidden state difference for: ； For all The mean of the hidden state differences of reasoning ability for each pair of reasoning ability sample pairs is used to obtain the target large model. Layered inference guidance vector : 。 6. The method for decoupling and transferring complex capabilities for large models according to claim 1, characterized in that, In the reasoning process of generating output word by word in the target large model, identifying the current generation stage based on the generated word sequence specifically includes: The generation phase includes the visual evidence localization phase, the visual evidence reasoning phase, and the answer generation phase. Based on the generated word sequence The current generation stage is determined by parsing special tags, regular rules, or finite state machines. Specifically, when the generated word sequence is located in the tool call field or the area location field, it is determined to be in the visual evidence location stage; when the generated word sequence is located in the visual reasoning field or the visual analysis field, it is determined to be in the visual evidence reasoning stage; when the generated content is located in the answer field, the current generation stage is determined to be in the answer generation stage.

7. The method for decoupling and transferring complex capabilities for large models according to claim 1, characterized in that, The shallow injection of the positioning guidance vector in the preset layer specifically includes: During the current generation phase Belongs to the visual evidence localization stage and the target is a large model. The layer belongs to the preset set of shallow to medium-level guiding layers. At that time, for the first The original positioning capabilities of the layer are hidden in the execution of the overlay: ； in, The target large model is represented by the first The layer in the first The hidden state of the original localization capability of each generation step. The hidden state for localization capability after injecting the guiding vector. For the target large model Layer positioning guide vector, To determine the intensity of visual evidence.

8. The method for decoupling and transferring complex capabilities for large models according to claim 1, characterized in that, The deep injection of the inference guidance vector in the preset context specifically includes: During the current generation phase Belongs to the visual evidence reasoning stage and the target is a large model. The layer belongs to the pre-defined set of medium-deep guide layers. At that time, for the first The original reasoning ability of the layer is hidden in the execution of the superposition: ； in, The target large model is represented by the first The layer in the first The hidden state of the original reasoning ability of each generation step. The hidden state for inference capabilities after the injection of the guiding vector. For the target large model Layered reasoning guidance vectors, Intensity of reasoning based on visual evidence.

9. The method for decoupling and transferring complex capabilities for large models according to claim 7 or 8, characterized in that, It also includes a process of norm calibration of the hidden state after the injection of the guide vector, specifically including: calculating the ratio of the L2 norm of the hidden state after the injection of the guide vector to the L2 norm of the original hidden state, scaling the hidden state after the injection of the guide vector with the ratio to obtain the calibrated hidden state; and continuing to input the calibrated hidden state into the subsequent network layers of the target large model.

10. A complex capability decoupling transfer system for large models, used to implement the method as described in any one of claims 1 to 9, characterized in that, include: The capability decoupling module is used to decouple the visual reasoning capability into the visual evidence localization sub-capability and the visual evidence reasoning sub-capability. The contrast sample construction module is used to construct positive and negative sample pairs for each sub-capability. The positive and negative sample pairs contain the same image and text questions. The positive sample contains a prompt that induces the target large model to generate the corresponding sub-capability behavior output, while the negative sample weakens the behavior output. The guidance vector extraction module is used to extract the hidden state vector of a specified Transformer layer from the input target large model with positive and negative samples, calculate the difference between the hidden states of positive and negative samples and calculate the mean, and obtain the localization guidance vector and the inference guidance vector. The stage identification module is used to identify the current generation stage based on the generated word sequence during the inference process of generating output word by word in the target large model; The vector injection module is used to inject shallow positioning guidance vectors in the preset to guide the target large model to generate regional positioning, local observation or tool call content when it is in the visual evidence localization stage; and to inject deep reasoning guidance vectors in the preset to guide the target large model to generate visual analysis, reasoning trajectory or evidence interpretation content when it is in the visual evidence reasoning stage. The system does not update the parameters of the target large model throughout the entire process.

Images