Model training method and apparatus, device, medium, and product

Abstract

The application discloses a model training method and device, equipment, medium and product, including: obtaining a sample image, a sample prompt word and a reference text; using an initial multi-modal large language model to predict the sample image and the sample prompt word, obtaining a predicted text, the predicted text including multiple tokens, the initial multi-modal large language model including a modal encoder and an LLM; for the first layer of the modal encoder, obtaining a token set corresponding to the first layer; according to the back propagation gradient of each token in the token set, determining a plurality of target propagation gradients corresponding to the first layer; projecting the plurality of target propagation gradients corresponding to the first layer to the first layer of the modal encoder, obtaining a gradient injection item of the first layer; according to the gradient injection item of the first layer and the original gradient of the first layer, obtaining a synthesized gradient of the first layer; updating the model parameters of the first layer of the modal encoder according to the synthesized gradient of the first layer. Improve the image semantic understanding ability of the modal encoder.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a model training method, apparatus, device, medium and product. Background Technology

[0002] In recent years, Multimodal Large Language Models (MLLMs) have made significant progress in the field of vision and language fusion. A typical architecture consists of a modal encoder, a cross-modal connector, and a Large Language Model (LLM). Mainstream training strategies often employ freezing the LLM and fine-tuning the modal encoder and cross-modal connector to fully utilize the semantic capabilities of the language model and achieve efficient multimodal knowledge transfer. The three components of MLLMs work in a pipeline-like collaborative manner, with each component responsible for a different task, ultimately achieving a complete process of "inputting an image and text prompts, outputting a text response." However, research has found that as the scale of LLM increases, the guiding role of the inverse signal (i.e., gradient signal) generated during the joint training phase significantly enhances, improving the discriminability and semantic consistency of visual features. But in small- to medium-scale models, this gradient signal effect weakens significantly, forming a "signal bottleneck" that restricts model performance. Summary of the Invention

[0003] This application provides a model training method, apparatus, device, medium, and product that can solve the problem of insufficient information density of the inverse signal (i.e., gradient signal) received by the modal encoder in multimodal large language model training.

[0004] In a first aspect, embodiments of this application provide a model training method, the method comprising: Obtain training samples, which include sample images, sample prompts, and reference text; The sample image and the sample prompt words are predicted using an initial multimodal large language model to obtain predicted text, which includes multiple tokens. The initial multimodal large language model includes a modal encoder and a large language model (LLM). For the modal encoder, the first Layer, obtain the first The set of tokens corresponding to the layer, the first The token set corresponding to the layer includes multiple tokens that correspond to the first token of the modal encoder. Tokens for layer receptive field alignment; and, The backpropagation gradient from LLM to modal encoder for each token in the token set is determined based on the token's backpropagation gradient. The layer corresponds to multiple target propagation gradients; and, The first The multiple target propagation gradients corresponding to the layer are projected onto the modal encoder's first... Layer, to obtain the first The gradient injection term of the layer; and, According to the first The gradient injection term of the layer and the first The original gradient of the layer is obtained from the first layer. The synthetic gradient of the layer; and, According to the first The synthesized gradient of the layer is related to the first layer of the modal encoder. The model parameters of the layer are updated.

[0005] Secondly, embodiments of this application provide a model training apparatus, the apparatus being applied in the cloud, the apparatus comprising: The first acquisition module is used to acquire training samples, which include sample images, sample prompts, and reference text. The prediction module is used to predict the sample image and the sample prompt words using an initial multimodal large language model to obtain predicted text. The predicted text includes multiple tokens. The initial multimodal large language model includes a modal encoder and a large language model (LLM). The second acquisition module is used for the first acquisition module of the modal encoder. Layer, obtain the first The set of tokens corresponding to the layer, the first The token set corresponding to the layer includes multiple tokens that correspond to the first token of the modal encoder. Tokens for layer receptive field alignment; The first determining module is configured to determine the first [total] token based on the backpropagation gradient from the LLM to the modal encoder for each token in the token set. Propagate gradients to multiple targets corresponding to the layer; The second determining module is used to determine the first The multiple target propagation gradients corresponding to the layer are projected onto the modal encoder's first... Layer, to obtain the first Gradient injection terms of the layer; The third determining module is used to determine the first... The gradient injection term of the layer and the first The original gradient of the layer is obtained from the first layer. Synthetic gradient of the layer; The update module is used to update the module according to the first... The synthesized gradient of the layer is related to the first layer of the modal encoder. The model parameters of the layer are updated.

[0006] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the model training method as described in the first aspect.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the model training method as described in the first aspect.

[0008] Fifthly, embodiments of this application provide a computer program product, wherein instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform the model training method as described in the first aspect.

[0009] In this embodiment of the application, by calculating the first... The gradient injection term of the layer, and according to the first... The gradient injection term of the layer and the first The original gradient of the layer is obtained from the first layer. The synthesized gradient of the layer; according to the first The synthesized gradient of the layer is related to the first layer of the modal encoder. The model parameters of the layer are updated. This method of directly injecting the original gradient optimizes the backpropagation path of the gradient and effectively avoids the physical decay of the original gradient in the layer-by-layer mapping in the traditional method (the original gradient will gradually decay in multi-layer mapping structures such as cross-modal connectors and long chain propagation paths, resulting in poor signal quality and low information density), making the model parameters of the layer updated. The synthetic gradient of the layer (with high information density) can overcome spatial limitations, guide and enhance the modal encoder to learn key visual features, and ensure that the inverse signal (i.e., the synthetic gradient) reaching the modal encoder has sufficient information density, thereby improving the image semantic understanding ability of the modal encoder. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating the model training method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application; Figure 3This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0012] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0013] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0014] In all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. Additionally, when embodiments of this application require access to sensitive personal information, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments obtained.

[0015] Studies have found that as the scale of an LLM increases, the inverse signal generated during the joint training phase significantly enhances its guiding role for the modal encoder, improving the discriminability and semantic consistency of visual features. However, in small- to medium-sized models, this signal effect weakens significantly, forming a "signal bottleneck" that restricts model performance.

[0016] The signal bottleneck is mainly manifested in: (1) Information dilution: The inverse gradient gradually decays in multi-layer mapping and long chain propagation, and the noise ratio increases, resulting in a decrease in the information density of the visual signal. (2) Gradient mismatch: The language generation goal of LLM and the representation learning goal of modal encoder are different, resulting in inconsistent gradient directions, which affects feature discriminability; (3) Cost limitations: Although large models have strong signal guidance capabilities, their training and inference costs are high, making them unsuitable for resource-constrained scenarios.

[0017] Signal bottlenecks cause small and medium-sized MLLMs to exhibit insufficient semantic alignment and weak visual detail understanding in visual question answering, command dialogue and robot vision tasks, which limits their application in high-precision fields such as autonomous driving, medical imaging and intelligent agent interaction.

[0018] Improving the quality of the inverse signal from LLM to modal encoder and enhancing cross-modal semantic alignment without requiring ultra-large-scale LLMs has become a key challenge in the current development of MLLMs.

[0019] To address the aforementioned issues, embodiments of this application provide a model training method, apparatus, device, medium, and product that can optimize the backpropagation path of gradients. This effectively avoids the physical attenuation of the original gradient in layer-by-layer mapping in traditional methods (the original gradient gradually attenuates in multi-layer mapping structures such as cross-modal connectors and long chain propagation paths, resulting in poor signal quality and low information density), ensuring that the backpropagation signal reaching the modal encoder has sufficient information density.

[0020] The model training method provided in the embodiments of this application will be introduced first below.

[0021] Figure 1 A schematic flowchart of a model training method provided in one embodiment of this application is shown. Figure 1 As shown, the model training method provided in this application embodiment is applied to an electronic device and includes the following steps 101-107, wherein: Step 101: Obtain training samples, which include sample images, sample prompts, and reference text.

[0022] There can be multiple training samples, which is not limited here. Each training sample includes a sample image, a sample prompt word, and reference text, which can be regarded as the label of the training sample.

[0023] For example, the training sample image is a picture of a cat, the sample prompt is "explain this image", and the reference text is "there is a white cat in the picture, sitting on the sofa".

[0024] Step 102: Use an initial multimodal large language model to predict the sample image and the sample prompt words to obtain predicted text. The predicted text includes multiple tokens. The initial multimodal large language model includes a modal encoder and a large language model (LLM).

[0025] Multimodal Large Language Models (MLLMs) can process both images and text simultaneously, such as GPT-4V and Gemini. In forward propagation, the MLLM's computation process from input (including images and text prompts) to output (i.e., predicted text) is the forward flow of the model "thinking and generating answers." In backpropagation, the MLLM calculates gradients and updates parameters based on the differences between the predicted text and the true labels (i.e., reference text in the training samples), which is the reverse flow of the model "learning and correcting errors."

[0026] In this embodiment, for ease of description, the untrained MLLM is referred to as the initial multimodal large language model. The initial multimodal large language model is trained using training samples, that is, sample images and sample prompt words are input into the initial multimodal large language model. The initial multimodal large language model generates output text through forward propagation, which is referred to as the predicted text for ease of description.

[0027] The initial multimodal large language model includes a modal encoder and a large language model (LLM). The modal encoder can be a visual encoder (VE) (for image processing) or other encoders. The modal encoder can process at least one of the images and audio. In addition, the initial multimodal large language model includes a cross-modal connector. The modal encoder is responsible for processing at least one of the images and audio, while the LLM is responsible for processing text. The cross-modal connector solves the "language incompatibility" problem between visual features and text features, allowing them to be fused in the same semantic space. Specifically, the cross-modal connector receives the visual features output by the modal encoder, maps the visual features to the same semantic space as the text features of the LLM (this process is called "alignment"), and concatenates the aligned visual vectors with the text features of the prompt words to form a unified multimodal input. The LLM receives the multimodal input (including visual and text features) output by the cross-modal connector and performs self-attention calculation on the multimodal input as if processing plain text input to generate a natural language response. For example, for the training sample in the above example, the generated predicted text is: "There is a white cat sitting on the sofa."

[0028] In the embodiments of the present application, the training strategy adopts the approach of freezing the LLM and fine-tuning the modal encoder, so as to fully utilize the semantic understanding capability of the LLM and achieve efficient multimodal knowledge transfer. The core logic for training the initial multimodal large language model is: using the "language understanding capability" of the LLM to guide the "visual understanding capability" of the modal encoder, so that the modal encoder can convert images into visual features understandable by the LLM.

[0029] In the field of large language models, a Token is the basic processing unit used by the model for text encoding, calculation and generation, and is a symbolic unit obtained by splitting the original text according to specific rules via a Tokenizer. A Token can be understood as the minimum semantic unit for the model to process text, such as a word, a phrase, a Chinese character, etc.

[0030] The predicted text includes a plurality of tokens, which may be obtained by tokenizing the predicted text, or obtained by screening the tokens resulting from tokenizing the predicted text. If the number of tokens included in the predicted text is relatively large, processing each token will result in an extremely large computational load. Considering that not every token is equally important, for example, in the predicted text "there is a white cat sitting on the sofa", the conjunctions "de", "shi", etc. contribute far less to model training than core content words such as "cat" and "sofa". To balance efficiency and accuracy, one token can be acquired every K (for example, K=4 or 8) tokens and added to the plurality of tokens, which can not only ensure that model training captures key information in the text, but also greatly reduce the computational load, making training feasible.

[0031] Step 103, for the th layer of the modal encoder, acquire the token set corresponding to the th layer, and the token set corresponding to the th layer includes a plurality of tokens among the tokens that are aligned with the receptive field of the th layer of the modal encoder.

[0032] The modal encoder has multiple layers, (also written as " " in the text) is the layer index of the modal encoder. The th layer usually corresponds to features of a specific scale of the image, for example, the low layer ( is small): corresponding to local details of the image (such as the cat's ears and eyes); the middle layer ( is medium): corresponding to object-level features of the image (such as the overall shape of the cat); the high layer ( is large): corresponding to the global semantics of the image (such as the scene where the cat is located). The more layers the modal encoder has, the larger the receptive field of the high layer, and the richer the global semantic features that can be learned. The Layer receptive field refers to the first layer of the modal encoder. The image area that a layer can see varies. For example, the receptive field of the lower layer of a modal encoder is small, and it can only see local details; the receptive field of the higher layers is large, and it can see the global scene. No. The token set corresponding to the layer, according to different The token set will also correspond to semantics at different scales. For example, the bottom layer may contain tokens with details such as "ears" and "eyes"; the middle layer may contain tokens with objects such as "cats" and "animals"; and the top layer may contain tokens with scenes such as "living room" and "sofa".

[0033] For example, the modal encoder's first... The visual features of the layer are used to find tokens among multiple tokens whose semantic similarity to the visual feature is greater than a similarity threshold. These found tokens constitute the first layer. The set of tokens corresponding to each layer. Alternatively, when LLM processes multimodal input, record the token pair for each modal encoder. The attention weights of the visual features output by the layer are set to a threshold. Tokens with attention weights higher than the threshold are added to the next layer. The set of tokens corresponding to the layer.

[0034] The value range of can be the mid-to-high levels of the modal encoder, and is not limited here. During model training, in each training round... Multiple values ​​can be taken, in which case the model parameters of the modal encoder can be updated in each training round.

[0035] Step 104: Determine the first [total] token based on the backpropagation gradient from LLM to modal encoder for each token in the token set. The gradients are propagated to multiple targets corresponding to the layer.

[0036] For example, based on the predicted text and the reference text, the backpropagation gradient of each token in the token set from the LLM to the modal encoder is determined; the backpropagation gradient of each token is filtered, pruned, and recalibrated to obtain the first... The gradients are propagated to multiple targets corresponding to the layer.

[0037] Step 105, the first The multiple target propagation gradients corresponding to the layer are projected onto the modal encoder's first... Layer, to obtain the first Gradient injection term for the layer.

[0038] For example, the first... The multiple target propagation gradients corresponding to the layer are projected onto the modal encoder's first... Layer, to obtain the first Gradient injection term for the layer.

[0039] Step 106, according to the first The gradient injection term of the layer and the first The original gradient of the layer is obtained from the first layer. Synthetic gradient of the layer.

[0040] For example, the first The gradient injection term of the layer and the first The original gradients of the layers are added together to obtain the first layer. The synthesized gradient of the layer. Or, the first layer... The gradient injection term of the layer and the first Multiply the gradient scaling factor of the layer to obtain the fourth value; then multiply the fourth value by the gradient scaling factor of the first layer. The sum of the original gradients of the layer is used as the first... The synthesized gradient of the layer. The gradient scaling factor of the layer is used to control the strength of the injected term; in this step, the gradient scaling factor of the first layer is directly applied to the second layer. Gradient injection of the original gradient of the layer ensures that the injected gradient is targeted.

[0041] No. The original gradient of a layer refers to the gradient obtained solely through training samples, without injecting LLM gradients into the modal encoder. For details on how to determine the original gradient of a layer, please refer to the relevant technical documentation, which will not be elaborated here.

[0042] Step 107, according to the first The synthesized gradient of the layer is related to the first layer of the modal encoder. The model parameters of each layer are updated. Through steps 101-107 above, the modal encoder in the initial multimodal large language model can be trained. After training, the target multimodal large language model can be obtained. The entire training process only requires a small to medium scale LLM to achieve high information density gradient transfer, thereby improving the training effect of the modal encoder.

[0043] The gradients of LLM can be injected into multiple layers of the modal encoder (e.g., mid-to-high layers) in the manner described above, allowing each layer of the modal encoder to learn text-related semantic features (i.e., the text understanding capability of LLM), thereby improving the multimodal understanding capability of the target multimodal large language model. This method of injecting gradients into different layers of the modal encoder can improve training performance.

[0044] In this embodiment of the application, by calculating the first... The gradient injection term of the layer, and according to the first... The gradient injection term of the layer and the first The original gradient of the layer is obtained from the first layer. The synthesized gradient of the layer; according to the first The synthesized gradient of the layer for the first The model parameters of the layer are updated. This method of directly injecting the original gradient optimizes the backpropagation path of the gradient and effectively avoids the physical decay of the original gradient in the layer-by-layer mapping in the traditional method (the original gradient will gradually decay in multi-layer mapping structures such as cross-modal connectors and long chain propagation paths, resulting in poor signal quality and low information density), making the model parameters of the layer updated. The synthetic gradient of the layer (with high information density) can overcome spatial limitations, guide and enhance the modal encoder to learn key visual features, and ensure that the inverse signal (i.e., the synthetic gradient) reaching the modal encoder has sufficient information density, thereby improving the image semantic understanding ability of the modal encoder.

[0045] The model training method provided in this application can solve the problems of low quality and insufficient information density of the inverse signal received by the modal encoder in MLLMs training.

[0046] In one embodiment of this application, the backpropagation gradient of each token in the token set from the LLM to the modal encoder is determined. The multiple target propagation gradients corresponding to the layer include: The information strength of each token in the token set is evaluated to obtain the information strength of each token. The information strength of the token is used to characterize the effective contribution of the token to the model parameter update of the modal encoder. Based on the information strength of each token in the token set and the backpropagation gradient of each token from the LLM to the modal encoder, the first... The gradients are propagated to multiple targets corresponding to the layer.

[0047] In this embodiment, by quantifying the information strength of each token, key generation signals with high information density, low noise, and strong correlation with visual evidence can be dynamically identified and captured (the generation signal can be understood as the backpropagation gradient corresponding to the token). The backpropagation gradient of the token is then adjusted based on its information strength to obtain the first... The propagation gradients of multiple targets corresponding to the layer effectively ensure that the signals arriving at the modal encoder have sufficient information density.

[0048] The information strength of a token can be calculated based on the token's contribution, the token's gradient entropy, and the token's evidentiary consistency.

[0049] For example, the information strength of each token can be calculated as follows: For each of the tokens in the token set, the contribution of the token is obtained, which is the token-level information contribution based on the token-level Fisher diagonal approximation of the token. The contribution of the token is normalized to obtain a first value; The second value is obtained by calculating the gradient entropy of the token, which is used to characterize the gradient diffusion and noise level of the token. The information strength of the token is obtained by weighted summation of the first value, the second value, and the evidence consistency of the token. The evidence consistency is used to measure how closely the token is aligned with the visual features output by the modal encoder.

[0050] Specifically, the information strength of token t is calculated according to equation (1): (1); in, For normalization function, The weights can be preset or dynamically set through a pre-configured Dynamic Amplification Policy (DAP), and there is no limitation on this.

[0051] It is the token-level information contribution based on the token-level Fisher diagonal approximation (also known as the diagonal approximation of the empirical Fisher information matrix) of token t. It is the gradient entropy, used to reflect gradient diffusion and noise level. It is the evidence consistency of token t.

[0052] In the above, the contribution of token t is calculated according to equation (2): (2); EMA stands for Exponential Moving Average, which is used to significantly reduce computational fluctuations and overhead while maintaining signal stability.

[0053] : Gradient calculation for model parameter θ. θ is limited to a subset of parameters related to visual alignment (such as the projector and cross-attn key / value matrix); : represent the given input sample image I and the sample prompt word q, respectively; : The t-th token currently predicted and generated; The sequence of historical tokens generated before the t-th token.

[0054] The token contribution quantifies the information contribution of the token to the model parameters.

[0055] p represents probability. This refers to the probability distribution function given by the model when the model parameter is θ.

[0056] normalized gradient (backward path gradient) The amplitude distribution obtained after normalization Define the gradient entropy of token t. : (3); To calculate the gradient entropy of token t We need to look at the normalized gradient of token t. amplitude distribution Specifically, to reduce computational overhead, the magnitude of the normalized gradient is divided into multiple consecutive intervals (bins of the histogram). In equation (3), i represents the i-th histogram interval, and N is the total number of histogram intervals. Represents the normalized gradient of the t-th token. The probability that the amplitude falls into the i-th interval.

[0057] Low This means the gradients are more concentrated and have stronger directional consistency, implying that the gradient signal has higher determinism and learning value. Gradient entropy is the Shannon entropy or sparsity index of the gradient magnitude distribution. Low gradient entropy usually indicates a more concentrated and directional optimization direction; high gradient entropy indicates high noise or diffusion, which should be suppressed.

[0058] When visual feature noise is high, the gradient direction is discrete, resulting in the gradient entropy. It will rise, which in turn leads to reduce.

[0059] Because the language generation goal of LLM and the representation learning goal of modal encoder are inherently different, the gradient direction is inconsistent and the noise ratio increases significantly during backpropagation, resulting in poor quality of the backpropagation signal (i.e., the backpropagation gradient). In this embodiment, the information contribution of the gradient is quantified using a token-level Fisher information matrix, and the gradient entropy is combined to evaluate the consistency of the gradient direction and the noise distribution level, thereby quantifying the information strength of the token t.

[0060] The information strength of each token can be quantified in the above way. By using the information strength of each token, key generation signals with high information density, low noise and strong correlation with visual evidence can be dynamically identified and captured (the generation signal can be understood as the backpropagation gradient corresponding to the token).

[0061] In the above, the information strength of each token in the token set and the backpropagation gradient of each token from the LLM to the modal encoder are used to obtain the first... The multiple target propagation gradients corresponding to the layer include: Based on the predicted text and the reference text, determine the backpropagation gradient of each token in the token set from the LLM to the modal encoder; Based on the information strength of each token in the token set, the backpropagation gradient of each token is filtered, pruned, and recalibrated to obtain the first... The gradients are propagated to multiple targets corresponding to the layer.

[0062] Specifically, the backpropagation gradient of each token in the token set from LLM to modal encoder is determined based on the predicted text and the reference text. For details, please refer to the relevant art records, which will not be repeated here.

[0063] After obtaining the information strength of each token in the token set, the backpropagation gradients of the tokens are filtered, pruned, and recalibrated using the information strength of the tokens. Filtering can employ soft gating to suppress low-information, high-noise gradients, thereby reducing invalid or interfering backpropagation gradients. Pruning limits the gradients to a reasonable range, avoiding gradient explosion. Recalibration converts the information strength of each token in the token set into a probability distribution, enabling adaptive adjustment of gradient weights. This makes the modal encoder more sensitive to the gradients of important tokens and less sensitive to the gradients of unimportant tokens, enhancing the influence of key gradients in cross-modal fusion.

[0064] In one embodiment of this application, the backpropagation gradient of each token is filtered, pruned, and recalibrated based on the information strength of each token in the token set to obtain the first... The multiple target propagation gradients corresponding to the layer include: For each token in the token set, the backpropagation gradient of the token is soft-gated according to the information strength of the token and a preset gating threshold to obtain the first gradient of the token; and, The first gradient of the token is pruned to obtain the pruned gradient of the token, wherein the pruning process is used to limit the norm of the first gradient of the token within a preset threshold; and, Recalibrate the tokens according to the clipping gradient of each token in the token set to obtain the first token. The gradients are propagated to multiple targets corresponding to the layer.

[0065] For example, the difference between the information strength of the token and the gating threshold is multiplied by a preset steepness to obtain a third value; the third value is normalized to obtain the gating coefficient corresponding to the token; the gating coefficient corresponding to the token is multiplied by the backpropagation gradient of the token to obtain the first gradient of the token. Specifically, the first gradient of the token t can be obtained according to equation (4). : (4); In the above formula, where It is the Sigmoid function. It is the gating threshold, and k is the kurtosis. This is the backpropagation gradient of token t. The output value (0,1) of the Sigmoid function can be seen as the "weight" for the backpropagation gradient, where 0 represents completely blocking the backpropagation gradient, and 1 represents fully allowing the backpropagation gradient to pass. When the information intensity of token t is lower than the gating threshold τ, its corresponding backpropagation gradient will be partially blocked, thereby reducing noise propagation; while when the information intensity of token t is higher than the gating threshold τ, its corresponding backpropagation gradient can be fully propagated.

[0066] For example, when the visual feature noise is large, the gradient direction of token t is discrete, resulting in the gradient entropy of token t. It will increase, which in turn will cause the token t to... According to equation (4), the reduction is as follows: Below the gate threshold τ, such that ( -τ) is a negative value, and after being mapped by the Sigmoid function, it outputs a scalar that infinitely approaches 0, which is related to the backpropagation gradient of the token t. The value after multiplication also approaches 0 infinitely, thus suppressing visual feature noise.

[0067] In the above, the backpropagation gradient of the token is soft-gated by equation (4). Based on the information strength of the token t, the backpropagation gradient of the token t is selectively transmitted. The backpropagation gradient of the token with high information strength (in this case, the noise of the backpropagation gradient of the token t is small) will be transmitted more, while the backpropagation gradient of the token with low information strength (in this case, the noise of the backpropagation gradient of the token t is large) will be suppressed.

[0068] To avoid token gradient explosion (e.g., excessively large first gradient values ​​causing drastic fluctuations in modal encoder model parameters), the first gradient of the token is pruned to obtain a pruned gradient. This pruning process limits the norm of the token's first gradient to a preset threshold. The purpose of the pruning process is to restrict the norm of the first gradient within a threshold; when the first gradient norm exceeds the threshold, the first gradient is proportionally reduced until its norm equals the threshold. The gradient pruning method can be found in relevant technical documents and will not be elaborated upon here.

[0069] Recalibration based on the token's clipping gradient can be performed as follows: The information intensity of each token in the token set is recalibrated using a normalization function to obtain the gradient weight of each token in the token set. The normalization function is used to transform the clipping gradient of each token into a probability distribution. For each token in the token set, the product of the token's gradient weight and the token's clipping gradient is used as the target propagation gradient of the token. Specifically, the gradient weight of token t can be obtained according to equation (5). The target propagation gradient of token t is obtained according to equation (6). : (5); (6); Among them, η is the clipping gradient of token t, η is the total scaling factor, which controls the total scaling ratio of the gradient weights, and T is the temperature parameter (pre-set or dynamically adjusted by DAP). Equation (5) allows tokens with high information intensity to obtain larger gradient weights, thereby making the target propagation gradient of the token larger, thus accelerating visual feature learning.

[0070] Recalibration refers to converting the information intensity of tokens into a probability distribution using a temperature-based softmax function, thereby achieving adaptive adjustment of gradient weights. The sharpness of the probability distribution is controlled by the temperature parameter T. The lower the temperature parameter T, the sharper the probability distribution, and tokens with high information intensity will receive greater weights (i.e., higher probabilities). The higher the temperature parameter T, the smoother the probability distribution, and all tokens will receive roughly the same weights.

[0071] Tokens strongly correlated with sample images The value will be large; recalibration will be used to handle this. It will output a larger value Thus, during backpropagation, the clipping gradient of tokens with larger values ​​can be explicitly assigned greater weight.

[0072] In the above, tokens strongly correlated with sample images With a larger value, the information intensity of the token can be amplified with a large weight through equation (5), resulting in a high degree of amplification of the token's clipping gradient. This allows the token's clipping gradient to have a greater weight in backpropagation, thus enabling the priority transmission of high-information signals; tokens strongly correlated with the sample image... With a smaller value, the information strength of the token can be amplified with a small weight using equation (5), resulting in a low amplification of the token's clipping gradient. This allows the token's clipping gradient to have a smaller weight in backpropagation, thus achieving a dynamic optimization process for adaptive weakening of low-quality signals. The above method effectively realizes the hierarchical screening and nonlinear amplification of gradients, making backpropagation more focused on effective signal paths that can promote modal encoder learning.

[0073] After obtaining the target propagation gradient of the token, the first... The multiple target propagation gradients corresponding to the layer are projected onto the modal encoder's first... Layer, to obtain the first The gradient injection term of the layer, specifically, can be calculated according to equation (7) for the first layer. Gradient injection term of layer : (7); in, It is the first The learnable linear / low-rank (LoRA) mapping of the layer is used to convert the gradient of the LLM into a gradient that the modal encoder can understand, that is, to map the gradient space on the LLM side to the layer space on the modal encoder side. It is the first The token set of the layer; It is the target propagation gradient of token t.

[0074] The first number is calculated according to formula (8). Synthetic gradient of layer : (8); in, It is the hierarchical coefficient (i.e., the first) The gradient scaling factor of the layer is used to control the injection intensity, which is preferentially allocated to the middle and high layers of the modal encoder to affect more abstract visual features; It is the first The original gradient of the layer.

[0075] As the modal encoder model parameters are updated, the characteristics of the backpropagation gradients change. The Dynamic Amplification Policy (DAP) calculates and adjusts the hyperparameters (in the IBS estimator) needed for the next round by reading and analyzing the current model's state feedback (such as the Gini coefficients in the current gradient set, convergence speed, etc.). T This leads to adjustments in training strategies, shifting the emphasis from "low noise" in the early stages of training to "high information" in the later stages.

[0076] The model training method provided in this application proposes a novel Informative Backprop Signal (IBS) metric system to quantitatively evaluate the effective contribution of each generated token t to the modal encoder parameter update. This enables the system to dynamically identify and capture key generated signals with high information density, low noise, and strong correlation with visual evidence. It retains proprietary metrics such as token-level Fisher information and gradient entropy to quantitatively characterize which signals have high value in modal encoder weight updates, thereby improving training efficiency and the model's sensitivity to visual features.

[0077] The four-stage signal amplification process is proposed: measurement, filtering, amplification, and injection. This transforms the experience of "large LLM signals" into a pluggable training middleware. This systematic methodology differs from traditional gradient clipping or loss reweighting and is an innovative practice of metric-driven gradient engineering.

[0078] Cross-Layer Injection (CLI) Mechanism: The CLI mechanism aims to achieve multi-level gradient mapping and fusion between the LLM and the modal encoder. Gradient signals filtered by IBS. The data augmentation process involves projecting the IBS (Information Base Layer) onto various levels of the modal encoder and injecting it as a residual, forming a "third path" for data augmentation. This reproduces the high-information gradient effect of large LLMs while maintaining the same model size. The selected IBS is projected onto multiple levels of the modal encoder (rather than just the input layer) and superimposed as residuals. This ensures that the IBS can more directly and effectively influence the key representation layers of the modal encoder, significantly improving the learning efficiency and feature quality of the modal encoder.

[0079] Dynamic amplification and overhead control: Mechanisms such as DAP and intermittent estimation are designed to ensure that while improving signal quality, the additional training overhead is kept within an acceptable range (target additional training time ≤10%-15%), which has high engineering practical value. Specifically, intermittent estimation is used, calculated once every K steps (e.g., K=8). During this process, gating and amplification coefficients are reused, significantly reducing computation frequency. The token Fisher information matrix is ​​approximated using a diagonal or K-FAC low-rank approximation; gradient entropy is calculated based on subsample histograms, balancing accuracy and efficiency. Only cross-modal key sub-modules (such as the projector layer and cross-attention layer) participate in IBS estimation to reduce parameter redundancy.

[0080] High cost-effectiveness and practical value: This technology has high cost-effectiveness and practical value in visual tasks such as instrument reading, complex chart understanding, counting / positioning, and visual command execution in human-machine collaboration, achieving the "small but big" effect of replacing model expansion with signal engineering.

[0081] Figure 2 A structural diagram of the model training apparatus provided in an embodiment of this application is shown. Figure 2 As shown, the model training device 200 includes: The first acquisition module 201 is used to acquire training samples, the training samples including sample images, sample prompt words and reference text; Prediction module 202 is used to predict the sample image and the sample prompt words using an initial multimodal large language model to obtain predicted text, the predicted text including multiple tokens, and the initial multimodal large language model including a modal encoder and a large language model LLM; The second acquisition module 203 is used for the first acquisition module of the modal encoder. Layer, obtain the first The set of tokens corresponding to the layer, the first The token set corresponding to the layer includes multiple tokens that correspond to the first token of the modal encoder. Tokens for layer receptive field alignment; The first determining module 204 is configured to determine the first token based on the backpropagation gradient from the LLM to the modal encoder for each token in the token set. The backpropagation gradients of multiple targets corresponding to the layer are determined based on the reference text and the predicted text for each token in the token set from the LLM to the modal encoder. The second determining module 205 is used to determine the first The multiple target propagation gradients corresponding to the layer are projected onto the modal encoder's first... Layer, to obtain the first Gradient injection terms of the layer; The third determining module 206 is used to determine, according to the first The gradient injection term of the layer and the first The original gradient of the layer is obtained from the first layer. Synthetic gradient of the layer; Update module 207, used to update according to the first The synthesized gradient of the layer is related to the first layer of the modal encoder. The model parameters of the layer are updated.

[0082] The model training device 200 provided in this application embodiment can implement all the processes implemented in the aforementioned cloud-executed model training method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0083] Figure 3 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0084] The electronic device may include a processor 601 and a memory 602 storing computer program instructions.

[0085] Specifically, the processor 601 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0086] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 602 is non-volatile solid-state memory.

[0087] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to the first or second aspect of this disclosure.

[0088] The processor 601 reads and executes computer program instructions stored in the memory 602 to implement any of the model training methods in the above embodiments.

[0089] In one example, the electronic device may also include a communication interface 603 and a bus 610. For example, Figure 3 As shown, the processor 601, memory 602, and communication interface 603 are connected through bus 610 and complete communication with each other.

[0090] The communication interface 603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0091] Bus 610 includes hardware, software, or both. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 610 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0092] Furthermore, in conjunction with the model training methods described in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the model training methods described in the above embodiments.

[0093] This application provides a computer program product in which the instructions are executed by the processor of an electronic device, causing the electronic device to perform any of the model training methods described in the above embodiments.

[0094] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described as examples. However, the method process of this application is not limited to the specific steps described. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0095] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0096] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0097] The foregoing flowcharts and / or block diagrams of methods, apparatus (systems) according to embodiments of the present disclosure have described various aspects of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to create a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowcharts and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0098] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A model training method, characterized in that, The method includes: Obtain training samples, which include sample images, sample prompts, and reference text; The sample image and the sample prompt words are predicted using an initial multimodal large language model to obtain predicted text. The predicted text includes multiple tokens. The initial multimodal large language model includes a modal encoder, a cross-modal connector, and a large language model LLM. The modal encoder is responsible for processing the image, the LLM is responsible for processing the text, and the cross-modal connector receives the visual features output by the modal encoder after processing the image and maps the visual features to the same semantic space as the text features of the LLM. For the modal encoder, the first Layer, obtain the first The set of tokens corresponding to the layer, the first The token set corresponding to the layer includes multiple tokens that correspond to the first token of the modal encoder. Tokens for layer receptive field alignment; and, Based on the information strength of each token in the token set and the backpropagation gradient of each token in the token set from the LLM to the modal encoder, the first... The layer corresponds to multiple target propagation gradients. The backpropagation gradient of each token in the token set from the LLM to the modal encoder is determined based on the reference text and the predicted text. The information strength of the token is used to characterize the effective contribution of the token to the model parameter update of the modal encoder; and, The first The multiple target propagation gradients corresponding to the layer are projected onto the modal encoder's first... Layer, to obtain the first The gradient injection term of the layer; and, According to the first The gradient injection term of the layer and the first The original gradient of the layer is obtained from the first layer. The synthesized gradient of the layer, the first The original gradient of a layer refers to the gradient obtained by the modality encoder through training samples without gradient injection; and, According to the first The synthesized gradient of the layer is related to the first layer of the modal encoder. The model parameters of the layer are updated.

2. The model training method according to claim 1, characterized in that, The method further includes: The information strength of each token in the token set is evaluated to obtain the information strength of each token.

3. The model training method according to claim 2, characterized in that, The information strength of each token in the token set is evaluated to obtain the information strength of each token, including: For each token in the token set, obtain the contribution of that token; The contribution of the token is normalized to obtain a first value; The second value is obtained by calculating the gradient entropy of the token, which is used to characterize the gradient diffusion and noise level of the token. The information strength of the token is obtained by weighted summation of the first value, the second value, and the evidence consistency of the token. The evidence consistency is used to measure how closely the token is aligned with the visual features output by the modal encoder.

4. The model training method according to claim 2, characterized in that, Based on the information strength of each token in the token set and the backpropagation gradient of each token from the LLM to the modal encoder, the first... The multiple target propagation gradients corresponding to the layer include: Based on the predicted text and the reference text, determine the backpropagation gradient of each token in the token set from the LLM to the modal encoder; Based on the information strength of each token in the token set, each backpropagation gradient is filtered, pruned, and recalibrated to obtain the first... The gradients are propagated to multiple targets corresponding to the layer.

5. The model training method according to claim 4, characterized in that, Based on the information strength of each token in the token set, each backpropagation gradient is filtered, pruned, and recalibrated to obtain the first... The multiple target propagation gradients corresponding to the layer include: For each token in the token set, the backpropagation gradient of the token is soft-gated according to the information strength of the token and a preset gating threshold to obtain the first gradient of the token; and, The first gradient of the token is pruned to obtain the pruned gradient of the token, wherein the pruning process is used to limit the norm of the first gradient of the token within a preset threshold; and, Recalibrate the tokens according to the clipping gradient of each token in the token set to obtain the first token. The gradients are propagated to multiple targets corresponding to the layer.

6. The model training method according to claim 5, characterized in that, Based on the information strength of the token and a preset gating threshold, the backpropagation gradient of the token is soft-gated to obtain the first gradient of the token, including: The difference between the information strength of the token and the gating threshold is multiplied by a preset steepness to obtain a third value; The third value is normalized to obtain the gating coefficient corresponding to the token. The first gradient of the token is obtained by multiplying the gating coefficient corresponding to the token with the backpropagation gradient of the token.

7. The model training method according to claim 5, characterized in that, Recalibration is performed based on the pruning gradient of each token in the token set to obtain the target propagation gradient of each token in the token set, including: The information strength of each token in the token set is recalibrated by a normalization function to obtain the gradient weight of each token in the token set. The normalization function is used to transform the clipping gradient of each token into a probability distribution. For each token in the token set, the product of the token's gradient weight and the token's clipping gradient is used as the target propagation gradient of the token.

8. The model training method according to claim 1, characterized in that, According to the first The gradient injection term of the layer and the first The original gradient of the layer is obtained from the first layer. The synthesized gradient of the layer includes: The first The gradient injection term of the layer and the first The gradient scaling factors of the layers are multiplied together to obtain the fourth value; Combine the fourth value with the first The sum of the original gradients of the layer is used as the first... Synthetic gradient of the layer.

9. A model training device, characterized in that, The device includes: The first acquisition module is used to acquire training samples, which include sample images, sample prompts, and reference text. The prediction module is used to predict the sample image and the sample prompt words using an initial multimodal large language model to obtain predicted text. The predicted text includes multiple tokens. The initial multimodal large language model includes a modal encoder, a cross-modal connector, and a large language model LLM. The modal encoder is responsible for processing the image, the LLM is responsible for processing the text, and the cross-modal connector receives the visual features output by the modal encoder after processing the image and maps the visual features to the same semantic space as the text features of the LLM. The second acquisition module is used for the first acquisition module of the modal encoder. Layer, obtain the first The set of tokens corresponding to the layer, the first The token set corresponding to the layer includes multiple tokens that correspond to the first token of the modal encoder. Tokens for layer receptive field alignment; The first determining module is configured to determine the first [token] based on the information strength of each token in the token set and the backpropagation gradient of each token in the token set from the LLM to the modal encoder. The layer corresponds to multiple target propagation gradients; the backpropagation gradient of each token in the token set from LLM to modal encoder is determined according to the reference text and the predicted text, and the information strength of the token is used to characterize the effective contribution ability of the token to the model parameter update of the modal encoder; The second determining module is used to determine the first The multiple target propagation gradients corresponding to the layer are projected onto the modal encoder's first... Layer, to obtain the first Gradient injection terms of the layer; The third determining module is used to determine the first... The gradient injection term of the layer and the first The original gradient of the layer is obtained from the first layer. The synthesized gradient of the layer, the first The original gradient of a layer refers to the gradient obtained by the modal encoder through training samples without gradient injection. The update module is used to update the module according to the first... The synthesized gradient of the layer is related to the first layer of the modal encoder. The model parameters of the layer are updated.

10. An electronic device, characterized in that, include: Processor and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the model training method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the model training method as described in any one of claims 1-8.

12. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the model training method as described in any one of claims 1-8.

Images