Vision language model dynamic multi-teacher hierarchical distillation training method and system

Abstract

The application belongs to the technical field of artificial intelligence, and relates to a visual language model dynamic multi-teacher hierarchical distillation training method and system. The method comprises the following steps: S1: training sample construction and model initialization; S2: hidden state sequence and output distribution acquisition; S3: sample level dynamic distillation control coefficient construction; S4: token level dynamic distillation control coefficient construction; S5: language distillation signal generation; S6: visual distillation signal generation; S7: output layer multi-teacher distillation loss calculation; S8: intermediate representation distillation loss calculation; S9: total loss function construction; and S10: distillation training. The method can balance language ability recovery and visual positioning ability maintenance, thereby improving the comprehensive performance and training stability of the visual language model.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to a dynamic multi-teacher hierarchical distillation training method and system for visual language models, particularly a dynamic multi-teacher hierarchical distillation training method and system for visual language models that balances language ability and visual localization ability. Background Technology

[0002] With the rapid development of artificial intelligence technology, visual language models have been widely applied in fields such as multimodal question answering, document understanding, invoice recognition, form extraction, image-text reasoning, scene text understanding, and complex visual question answering. Visual language models typically combine a pre-trained language model with a visual encoding module, enabling the model to simultaneously receive image and text input and output task-relevant text results. In practical applications, to achieve better performance for specific industry scenarios or tasks, further optimization of the model is usually required through multimodal fine-tuning, instruction fine-tuning, or distillation training.

[0003] In an ideal visual language model training scenario, the model should acquire image and text understanding capabilities while preserving, as far as possible, the original language model's existing language understanding, knowledge retention, reasoning generation, and text organization capabilities. It should also maintain strong visual localization and accurate reading capabilities in document, invoice, and OCR tasks. However, in current visual language model training practices, the optimization process for model capabilities remains relatively crude, lacking fine-grained modeling of the interrelationships between different capabilities. This leads to problems such as language capability degradation, impaired visual localization capabilities, or difficulty in simultaneously developing both types of capabilities after multimodal training.

[0004] Especially with the increasing scale and complexity of visual language models, these models not only need to understand image semantics but also accurately read text, locate key fields, align layout structures, and perform reasoning based on combined text and image information. For such tasks, language ability and visual localization ability have different emphases during model training and are prone to interference: if the training process overemphasizes language recovery, the model may lean towards semantic rewriting rather than precise reading in OCR, invoice field extraction, and document question answering tasks; conversely, if the training process overemphasizes visual perception and multimodal alignment, the model's original language reasoning and knowledge understanding abilities may decline. Therefore, balancing language ability and visual localization ability during training has become a key technical problem in visual language model optimization.

[0005] In recent years, research and engineering practices related to the problems of capability preservation, capability recovery, and distillation optimization in visual language model training can be mainly summarized into the following technical paths.

[0006] 1. A multimodal alignment scheme based on a frozen language backbone with added adaptation modules.

[0007] This type of approach typically keeps the main parameters of the pre-trained language model unchanged, adding bridging modules, adapter modules, Q-Formers, visual expert branches, or cross-modal alignment modules on the visual input side or in the middle layers of the model. This allows image features to be mapped to a representation space that the language model can process, thereby reducing the damage to the original language model's language capabilities caused by multimodal training. A representative idea behind this approach is to "freeze the language model + introduce additional modules" to retain the original capabilities of the language model to a certain extent while simultaneously endowing the model with image understanding capabilities.

[0008] While such approaches can mitigate language ability degradation, they inherently rely on additional modules for multimodal fusion, typically increasing model complexity and parameter size, and potentially incurring additional inference overhead. Furthermore, these methods focus on "preventing the destruction of the language backbone" rather than "restoring language abilities after ability drift has occurred," resulting in limited ability compensation for visual language models that have already undergone multimodal alignment.

[0009] 2. A capability recovery scheme based on conventional multimodal fine-tuning or language data supplementary training.

[0010] Some existing solutions, after completing the basic training of the visual language model, further fine-tune the model by introducing language data, knowledge-based question-answering data, or language-intensive task data, attempting to restore the model's performance in language understanding, commonsense reasoning, and text generation. The core idea of ​​this type of approach is to allow the model to re-expose itself to more language information through retraining, thereby mitigating the loss of language ability caused during the multimodal adaptation stage.

[0011] However, such approaches typically still rely on standard supervision objectives or uniform training goals, lacking targeted modeling of the root causes of ability degradation. Since visual language models undergo representation space shifts after multimodal training, relying solely on supplementary language data for training often fails to fully recover the lost language priors and knowledge distributions in the original language model. In practice, these methods often only yield limited recovery results and cannot fundamentally solve the language ability drift problem caused by multimodal training.

[0012] 3. A knowledge distillation scheme for the output layer based on a single teacher model.

[0013] To further restore the language capabilities of visual language models, existing solutions have begun to explore introducing teacher models to distill and train student visual language models. A concept closely related to this invention is as follows: the original language model before multimodal adaptation is used as the teacher model, and the multimodal trained visual language model is used as the student model. During the training phase, the language priors of the original language model are distilled back into the student model by aligning the output probability distribution. To enable the language teacher to participate in image-text scene training, related solutions also design a mechanism based on shared key-value caching (KV-cachesharing), allowing the frozen language teacher to provide visually conditional language supervision to the student output using the multimodal contextual representations generated by the student model.

[0014] Compared to simply continuing fine-tuning, this type of approach has a clearer language ability recovery goal and does not require retaining the teacher model after training, thus having the advantage of not increasing inference parameters. From a technical perspective, this method is quite close to the problem addressed in this invention: recovering specific abilities with the help of teacher supervision during visual language model training while minimizing inference costs.

[0015] However, such schemes typically have two obvious limitations: (1) The teacher model is usually only a single-language teacher, and its distillation signal mainly emphasizes language distribution and text generation ability, lacking targeted support for tasks that rely on visually precise positioning, such as OCR, form field extraction, and document layout understanding; (2) Distillation methods are mostly concentrated in the output layer, mainly constraining the student model through soft label distribution alignment, and rarely involving the joint guidance of internal information such as intermediate representations, attention distribution, and regional features. Therefore, although such schemes help to restore language ability, in scenarios such as tickets, receipts, forms, and documents, they are prone to causing the model to favor "semantically reasonable" rather than "positionally accurate" outputs, such as rewriting the strings in the image into content that is more in line with natural language habits, or extracting seemingly reasonable but actually incorrect answers from neighboring fields, thereby reducing visual positioning ability.

[0016] 4. Data category differentiation scheme based on static selective distillation.

[0017] To mitigate the capability conflict caused by uniform distillation by a single teacher, a more advanced approach has been proposed: selective distillation. This involves applying different distillation strategies to different types of samples based on the source or category of the training data. For example, a higher weight is given to language teacher distillation for language-intensive and knowledge-reasoning data; while for tasks such as OCR and document comprehension, language distillation is reduced or disabled, retaining only the student model's own supervision to minimize interference from the language teacher in visual grounding. This approach can, to some extent, achieve "more distillation where the teacher is reliable and less distillation where the teacher is unreliable," thus achieving a better balance between language capability recovery and visual task preservation compared to uniform distillation.

[0018] This type of approach is also the closest existing implementation path to the present invention. Its closest implementation can be summarized as follows: the original text language model is used as a single teacher, and the visual language model is used as a student; during the training phase, by sharing the student's multimodal representation, the language teacher can generate distillation signals based on the text-image context; then, according to the predefined data categories, the distillation loss and hard label loss are statically weighted, thereby applying language distillation to some samples and retaining the original supervision on other samples.

[0019] Although this type of scheme is superior to the unified distillation scheme, it is still essentially a single-teacher distillation method that uses coarse-grained, static division based on dataset category. It still has the following shortcomings: (1) The distillation control granularity is relatively coarse, and it can only decide whether to distill based on the dataset category to which the sample belongs, making it difficult to adapt to the complex ability requirements within the sample; (2) In the same sample, different output tokens have different degrees of dependence on language ability and visual positioning ability, and this type of static method cannot carry out differentiated distillation for different tokens; (3) The source of teachers is still singular, and there is a lack of specialized teachers for visual positioning ability, making it difficult to actively strengthen OCR, field positioning and layout understanding ability while restoring language; (4) The distillation level is mainly concentrated in the output layer, and there is a lack of joint constraints on the intermediate layer of hidden state, attention distribution and region representation, resulting in insufficient ability restoration and ability maintenance process.

[0020] 5. Existing solutions lack common technologies.

[0021] In summary, while existing visual language model training and distillation optimization schemes have alleviated the problem of ability degradation after multimodal training to some extent, they still generally suffer from the following key technical deficiencies: (1) lack of a unified training mechanism that can simultaneously take into account language ability recovery and visual localization ability maintenance; (2) difficulty in providing visual localization teacher signals that complement language teachers for tasks such as OCR, ticket extraction, and document question answering; (3) lack of fine-grained dynamic perception of differences in training samples or generated token ability requirements; (4) difficulty in adaptively adjusting the distillation weights of different teachers according to specific samples or specific output positions; (5) the distillation process is mainly concentrated in the output layer, lacking hierarchical joint constraints on intermediate representations, attention patterns, and regional features; and (6) difficulty in providing refined training support for language ability recovery, visual localization ability maintenance, and multimodal collaborative optimization of visual language models. Based on the above analysis, there is an urgent need for a technical solution that can simultaneously optimize language ability recovery and visual localization ability during the training process of visual language models, thereby providing a reliable training foundation for the stable optimization and comprehensive performance improvement of visual language models. Summary of the Invention

[0022] To address the problems existing in the prior art, this invention proposes a dynamic multi-teacher hierarchical distillation training method and system for visual language models. During the visual language model training process, a dynamic multi-teacher collaborative distillation mechanism is introduced. Based on the capability requirements of the training samples or output tokens, the distillation signals of the language teacher model and the visual localization teacher model are dynamically selected, and hierarchical distillation is jointly implemented in the output layer and the intermediate representation layer. This constructs a training mechanism that balances language capability recovery and visual localization capability preservation, which is used for subsequent optimization of the student visual language model to improve the overall performance and training stability of the visual language model.

[0023] To achieve the above objectives, the present invention provides the following technical solution: A dynamic multi-teacher hierarchical distillation training method for visual language models, characterized by the following steps: S1: Construct training samples and initialize the student visual language model, language teacher model, and visual positioning teacher model. The training samples include image input, text input, and target output sequence. S2: Input the training samples into the student visual language model to obtain the hidden state sequence and output distribution of the student visual language model; S3: Construct sample-level dynamic distillation control coefficients based on the sample features of the training samples; S4: Construct token-level dynamic distillation control coefficients based on the sample-level dynamic distillation control coefficients and the hidden state sequence of the student visual language model; S5: Based on the training samples and the hidden state sequence of the student visual language model, the language teacher model generates a language distillation signal; S6: Based on the training samples and the hidden state sequence of the student visual language model, a visual distillation signal is generated by the visual localization teacher model; S7: Calculate the distillation loss of the language teacher model, the distillation loss of the visual positioning teacher model, and the hard label supervision loss based on the output distribution of the student visual language model, the language distillation signal, the visual distillation signal, the token-level dynamic distillation control coefficient, and the target output sequence. S8: Calculate the intermediate layer distillation loss based on the hidden state sequence of the student visual language model, the hidden state sequence of the language teacher model, and the hidden state sequence of the visual localization teacher model; calculate the attention distillation loss based on the attention matrix of the student visual language model, the attention matrix of the language teacher model, and the attention matrix of the visual localization teacher model. S9: Construct a total loss function based on the distillation loss of the language teacher model, the distillation loss of the visual localization teacher model, the hard label supervision loss, the intermediate layer distillation loss, and the attention distillation loss; S10: Distill the student visual language model based on the total loss function to update its model parameters.

[0024] Preferably, step S2 specifically includes: S21: Encode the image input using the image encoder in the student visual language model to obtain a visual token sequence representation:

[0025] in, Indicates the first A visual token sequence representation of each training sample; Indicates an image encoder; Indicates the first Input images of training samples; Indicates the first Input images of training samples The Feature vector of a visual token; Indicates the first Input images of training samples The total number of visual tokens; S22: Encode the text input using a text encoder or text embedding layer in the student visual language model to obtain a text token sequence representation:

[0026] in, Indicates the first The text token sequence representation of each training sample; Indicates the first Text input for each training sample; This indicates a text encoder or text embedding layer; Indicates the first Text input of training samples The Feature vectors of a text token; Indicates the first Text input of training samples The total number of text tokens; S23: Input the visual token sequence representation and the text token sequence representation into the multimodal fusion module or decoding module of the student visual language model to obtain the hidden state sequence of each layer of the student visual language model:

[0027] in, Indicates based on the first The first training sample yielded the student visual language model. The hidden state sequence of the layer; The first character representing the student's visual language model The multimodal representation transformation process of the layer; Indicates based on the first The first training sample yielded the student visual language model. The layer in the first Hidden state vectors at each output position; This represents the total number of layers in the student's visual language model; The first character representing the student's visual language model The total number of output positions of the layer, that is, the total number of target outputs of the target output sequence; S24: Use the hidden state sequence of the last layer of the student visual language model as the input to the output layer of the student visual language model, and map the hidden state vector of each output position to the corresponding prediction score in the vocabulary space through the output layer:

[0028] in, Indicates based on the first The last layer of the student visual language model obtained from the training samples is in the... Hidden state vectors at each output position; This represents the output projection matrix of the output layer of the student visual language model; This represents the bias term of the output layer of the student visual language model; Indicates based on the first The output layer of the student visual language model obtained from the training samples is at the . The predicted score for each candidate token in the vocabulary at each output position; S25: Based on the predicted score at each output position, obtain the overall predicted score sequence for each training sample:

[0029] in, Indicates based on the first The overall predicted score sequence of the student visual language model obtained from training samples; S26: Normalize the overall prediction score to obtain the output distribution of the student visual language model:

[0030] in, Indicates based on the first The output distribution of the student visual language model obtained from training samples; Indicates based on the first The student visual language model obtained from the training samples is in the _th ... Output at each output position; This represents the student's temperature coefficient.

[0031] Preferably, step S3 specifically comprises:

[0032]

[0033]

[0034] in, Indicates based on the first Sample-level dynamic distillation control coefficients obtained from training samples; Indicates based on the first Language teacher distillation weights obtained from training samples; Indicates based on the first The visual positioning teacher distillation weights obtained from each training sample; Indicates based on the first Hard-labeled supervision weights obtained from each training sample; Indicates a gating network; Indicates the first Sample features of each training sample.

[0035] Preferably, step S4 specifically comprises:

[0036]

[0037]

[0038] in, Indicates based on the first The training samples obtained at the th th Token-level dynamic distillation control coefficients at each output position; Indicates based on the first The training samples obtained at the th th Token-level language teacher distillation weights at each output position; Indicates based on the first The training samples obtained at the th th Token-level visual positioning teacher distillation weights at each output position; Indicates based on the first The training samples obtained at the th th Token-level hard-label supervision weights at each output position; Indicates based on the first The training samples obtained at the th th Token-level refined language teacher distillation weights at each output position; Indicates based on the first The training samples obtained at the th th Token-level refined visual positioning teacher distillation weights at each output position; Indicates based on the first The training samples obtained at the th th Token-level refined hard-label supervision weights at each output position; These are trainable parameters; Indicates based on the first The last layer of the student visual language model obtained from the training samples is in the... The hidden state vector at each output position.

[0039] Preferably, step S5 specifically comprises: S51: Obtain the multimodal conditional representation from the hidden state sequence of the student's visual language model:

[0040] In the formula, Indicates based on the first The first training sample yielded the student visual language model. Multimodal conditional representation of layers; Indicates based on the first The first training sample yielded the student visual language model. The hidden state sequence of the layer; Represents a mapping function; S52: Input the text of the training samples and the multimodal conditional representation into the language teacher model to generate the hidden state sequence of the language teacher model:

[0041] in, Indicates based on the first The language teacher model obtained from the training samples. The hidden state sequence of the layer; Indicates based on the first The language teacher model obtained from the training samples. The layer in the first Hidden state vectors at each output position; The first language teacher model The total number of output positions of the layer, that is, the total number of target outputs of the target output sequence; It is the first Text input for each training sample; Represents a language teacher model; S53: Use the hidden state sequence of the last layer of the language teacher model as the input to the output layer of the language teacher model, and map the hidden state vector of each output position to the corresponding predicted score in the vocabulary space through the output layer:

[0042] in, Indicates based on the first The last layer of the language teacher model obtained from the training samples is in the... Hidden state vectors at each output position; This represents the output projection matrix of the output layer of the language teacher model; This represents the bias term of the output layer of the language teacher model; Indicates based on the first The output layer of the language teacher model obtained from the training samples is in the th training sample. The predicted score for each candidate token in the vocabulary at each output position; S54: Based on the predicted score at each output position, obtain the overall predicted score sequence for each training sample:

[0043] in, Indicates based on the first The overall predicted score sequence of the language teacher model obtained from training samples; S55: Normalize the overall predicted score sequence to obtain the output distribution of the language teacher model:

[0044] in, Indicates based on the first The output distribution of the language teacher model obtained from the training samples, i.e., the language distillation signal; Indicates based on the first The language teacher model obtained from the training samples is in the . Output at each output position; This represents the temperature coefficient of language teachers.

[0045] Preferably, step S7 specifically includes: S71: Distillation Loss in Computational Language Teacher Model :

[0046] in, This represents the total number of training samples; This represents the total number of target outputs in the target output sequence; Indicates based on the first The training sample obtained from the first training sample Token-level language teacher distillation weights at each output position; Indicates the temperature coefficient of language teachers; Indicates based on the first The language teacher model obtained from the training samples is in the . Output at each output position; Indicates based on the first The student visual language model obtained from the training samples is in the _th ... Output at each output position; S72: Distillation Loss of Computational Visual Localization Teacher Model :

[0047] in, Indicates based on the first The training sample obtained from the first training sample Token-level visual positioning teacher distillation weights at each output position; Indicates the visual positioning teacher's temperature coefficient; Indicates based on the first The visual positioning teacher model obtained from the training samples is in the . Output at each output position; S73: Calculate the hard-label supervision loss ;

[0048] in, Indicates the first The target output sequence of the training samples is the first one. One target output; Indicates based on the first The training sample obtained from the first training sample Token-level hard-label supervision weights at each output position; Indicates based on the first The output layer of the student visual language model obtained from the training samples is at the . The predicted score for each candidate token in the vocabulary at each output position.

[0049] Preferably, step S8 specifically includes: S81: Perform dimension alignment on the hidden state sequences of the student language model, the language teacher model, and the visual positioning teacher model to obtain the dimension-aligned hidden state sequences of the language teacher model and the visual positioning teacher model. S82: Calculate the intermediate layer distillation loss :

[0050] in, This represents the total number of training samples; This represents the total number of layers in the student's visual language model; Indicates based on the first The first training sample yielded the student visual language model. The hidden state sequence of the layer; Represents the dimension-aligned representation based on the first dimension. The language teacher model obtained from the training samples. The hidden state sequence of the layer; Represents the dimension-aligned representation based on the first dimension. The visual positioning teacher model obtained from the training samples. The hidden state sequence of the layer; Indicates based on the first The language teacher model obtained from the training samples. Layer language distillation weights; Indicates based on the first The visual positioning teacher model obtained from the training samples. Layer language distillation weights; S83: Calculate the attention distillation loss :

[0051] in, , and They are based on the first The student visual language model, language teacher model, and visual positioning teacher model obtained from the training samples are the first... Attention matrix of the layer; Indicates based on the first The language teacher model obtained from the training samples. Attention distillation weights of the layer; Indicates based on the first The visual positioning teacher model obtained from the training samples. Attention distillation weights for layers.

[0052] Furthermore, the present invention also provides a dynamic multi-teacher hierarchical distillation training system for visual language models, characterized in that it includes: The training sample construction and model initialization module is used to construct training samples and initialize the student visual language model, language teacher model and visual positioning teacher model. The training samples include image input, text input and target output sequence. The hidden state sequence and output distribution acquisition module is used to input the training samples into the student visual language model to obtain the hidden state sequence and output distribution of the student visual language model. A sample-level dynamic distillation control coefficient construction module is used to construct sample-level dynamic distillation control coefficients based on the sample features of the training samples. The token-level dynamic distillation control coefficient construction module is used to construct token-level dynamic distillation control coefficients based on the sample-level dynamic distillation control coefficients and the hidden state sequence of the student visual language model. A language distillation signal generation module is used to generate a language distillation signal from a language teacher model based on the training samples and the hidden state sequence of the student visual language model. A visual distillation signal generation module is used to generate a visual distillation signal from a visual localization teacher model based on the training samples and the hidden state sequence of the student visual language model. The output layer multi-teacher distillation loss calculation module is used to calculate the distillation loss of the language teacher model, the distillation loss of the visual positioning teacher model, and the hard label supervision loss based on the output distribution of the student visual language model, the language distillation signal, the visual distillation signal, the token-level dynamic distillation control coefficient, and the target output sequence. The intermediate representation distillation loss calculation module is used to calculate the intermediate layer distillation loss based on the hidden state sequence of the student visual language model, the hidden state sequence of the language teacher model, and the hidden state sequence of the visual positioning teacher model, and to calculate the attention distillation loss based on the attention matrix of the student visual language model, the attention matrix of the language teacher model, and the attention matrix of the visual positioning teacher model. The total loss function construction module is used to construct the total loss function based on the language teacher model distillation loss, the visual localization teacher model distillation loss, the hard label supervision loss, the intermediate layer distillation loss, and the attention distillation loss. A distillation training module is used to perform distillation training on the student visual language model based on the total loss function to update its model parameters.

[0053] Furthermore, the present invention also provides a dynamic multi-teacher hierarchical distillation training device for visual language models, characterized in that it includes: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the visual language model dynamic multi-teacher hierarchical distillation training method as described above. Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the program is executed by a processor, it implements the steps of the dynamic multi-teacher hierarchical distillation training method for visual language models as described above.

[0054] Compared with existing visual language model optimization methods that mainly rely on single-teacher distillation, static data classification, uniform application of distillation signals, or distillation training based solely on the output layer, the dynamic multi-teacher hierarchical distillation training method and system for visual language models that balances language ability and visual localization ability proposed in this invention has the following significant advantages in terms of language ability recovery effect, visual localization ability preservation effect, training stability, and distillation control precision.

[0055] (1) Based on the training mechanism of dynamic multi-teacher collaborative distillation, the synergy between language ability recovery and visual positioning ability maintenance is fundamentally improved.

[0056] Existing visual language model distillation methods typically employ a single-teacher model, especially a single-language teacher, to distill and train student models. While this can restore the model's language comprehension, knowledge retention, and text generation capabilities to some extent, the lack of targeted guidance for visual positioning-related capabilities such as OCR, field location, and document layout understanding can easily lead to a decline in visual positioning capabilities while language proficiency improves.

[0057] This invention introduces both a language teacher model and a visual positioning teacher model simultaneously, and dynamically coordinates different teacher signals according to the ability requirements of training samples or output positions. This enables the student visual language model to absorb both the language priors and reasoning knowledge provided by the language teacher and the region alignment, field recognition, and layout positioning information provided by the visual positioning teacher during the training process. This achieves synergistic optimization of language ability recovery and visual positioning ability maintenance from the training mechanism, significantly improving the overall capabilities of the model.

[0058] (2) The distillation method based on sample-level and token-level dynamic control significantly improves the targeting and precision of the distillation process.

[0059] In existing technologies, distillation signals are usually applied uniformly to all training samples, or are only coarsely divided based on dataset categories, which makes it difficult to reflect the real differences in the dependence of different samples and different output tokens on language ability and visual localization ability.

[0060] This invention introduces sample-level and token-level dynamic control mechanisms during the distillation process. It can determine whether each training sample, and even each generation location, leans more towards language understanding and knowledge reasoning, or relies more on OCR, field extraction, and visual region localization. Based on this, the weights of the language teacher, visual localization teacher, and hard-label supervision are dynamically adjusted. This approach transforms distillation training from a "uniform application" to an "on-demand allocation," significantly improving the precision of the distillation process and the effective utilization of training signals.

[0061] (3) Enhance the integrity of the internal capability transfer of the student model by distilling the output layer and the intermediate representation layer together.

[0062] Most existing distillation schemes focus on the alignment of soft label distribution in the output layer, which mainly constrains the consistency between the final prediction results of the student visual language model and the output of the teacher model. They are difficult to provide sufficient guidance for the hidden states, attention distribution, region representation and cross-modal alignment process within the student model.

[0063] This invention not only performs multi-teacher distillation on the student model at the output layer, but also further implements joint constraints on hidden states, attention patterns, and visual region features at the intermediate representation level. This ensures that the student model not only "approaches the teacher's output," but also that "its internal representation process can learn the teacher's ability structure." In this way, the original language ability can be recovered more completely, while visual localization-related representations are preserved more effectively, avoiding the problem of insufficient ability transfer caused by relying solely on output layer distillation.

[0064] (4) Effectively avoids interference from single language distillation on OCR and document-type tasks, and significantly improves the stability and capability balance of the training process.

[0065] In existing technologies, while uniformly employing language teachers to distill all samples can improve performance on some language and knowledge-based tasks, it can negatively impact tasks such as DocVQA, OCRbench, document field extraction, and form-based question answering. Specifically, it may correct non-standard characters in images to "more natural language expressions," read incorrect answers from adjacent fields, or cause rewriting and offsetting when extracting precise numerical values, dates, names, and other information. Uniform distillation leads to a more significant performance degradation on visually detailed tasks like OCRbench and DocVQA, while selective distillation can mitigate this problem to some extent.

[0066] This invention introduces a visual grounding teacher and implements a dynamic allocation strategy, enabling the model to prioritize receiving supervisory signals related to position, field, and layout structure in visual precision reading tasks. This effectively reduces the misleading effect of language teachers on visual grounding, making the model training process more stable and maintaining a more balanced ability.

[0067] In summary, this invention, by introducing a dynamic multi-teacher collaborative distillation mechanism, a sample-level and token-level dynamic control mechanism, and a joint distillation mechanism between the output layer and the intermediate representation layer, achieves refined control over the relationship between the recovery of language ability and the preservation of visual localization ability in a visual language model without significantly increasing the model complexity during the inference stage. This effectively improves the stability, controllability, and overall multimodal performance of the model training process. Attached Figure Description

[0068] Figure 1 This is a flowchart of the visual language model dynamic multi-teacher hierarchical distillation training method of the present invention.

[0069] Figure 2 This is a schematic diagram of the structure of the visual language model dynamic multi-teacher hierarchical distillation training system of the present invention.

[0070] Figure 3 This is a structural block diagram of the visual language model dynamic multi-teacher hierarchical distillation training device of the present invention. Detailed Implementation

[0071] Before detailing any embodiment of the invention, it should be understood that the invention, in its application, is not limited to the details of the construction and arrangement of the components set forth in the following description or illustrated in the following figures. The invention can have other embodiments and can be practiced or carried out in various ways. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered limiting. The use of “comprising” or “having” and variations thereof in this invention is intended to cover the items set forth below and their equivalents, as well as any additional items. Unless otherwise specified or limited, the terms “installation,” “connection,” “support,” and “linkage,” and variations thereof are used broadly and cover both direct and indirect installation, connection, support, and linking. Moreover, “connection” and “linkage” are not limited to physical or mechanical connections or links. Furthermore, firstly, in the disclosure of this invention, the terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the above terms should not be construed as limiting this invention. Secondly, the term "a" should be understood as "at least one" or "one or more," that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple. The term "a" should not be construed as a limitation on the quantity.

[0072] To address the problems existing in the training of visual language models, this invention proposes a dynamic multi-teacher hierarchical distillation training method and system for visual language models that balances language ability and visual localization ability. During the training process, a dynamic multi-teacher collaborative distillation mechanism is introduced. Based on the ability requirements of the training samples or output tokens, the distillation signals of the language teacher model and the visual localization teacher model are dynamically selected, and hierarchical distillation is jointly implemented in the output layer and intermediate representation layer. This constructs a training mechanism that balances language ability recovery and visual localization ability preservation, which is used for subsequent student model optimization to improve the overall performance and training stability of the visual language model.

[0073] Specifically, the present invention improves the distillation training mechanism of visual language model in the following ways: (1) When the training samples or generation positions are highly dependent on language understanding, knowledge reasoning and text generation capabilities, the distillation effect of the language teacher model on the student visual language model is enhanced to promote the recovery of the model's language capabilities; (2) When the training samples or generation positions are highly dependent on OCR, field extraction, layout understanding and region positioning capabilities, the distillation effect of the visual positioning teacher model on the student visual language model is enhanced to maintain the model's visual positioning capabilities; (3) When the training samples contain both language expression needs and visual precision reading needs, different teacher signals are dynamically weighted and fused, and combined with the distillation of the output layer and intermediate representation layer, the joint learning of the student visual language model on language capabilities and visual positioning capabilities is coordinated.

[0074] The main technological breakthroughs achieved by this invention are reflected in the following three aspects: (1) Introduce a dynamic multi-teacher distillation mechanism for balancing the capabilities of visual language models.

[0075] During the training of the visual language model, the distillation weights of the language teacher model and the visual localization teacher model are dynamically adjusted according to the differences in the ability requirements of the training samples or output tokens, so as to achieve targeted guidance in different ability directions and avoid the ability bias problem caused by single teacher distillation.

[0076] (2) Establish a stratified distillation training mechanism that takes into account both the recovery of language ability and the maintenance of visual positioning ability.

[0077] By combining output layer distillation and intermediate representation layer distillation, not only is the final prediction distribution of the student's visual language model constrained, but the internal representations such as hidden states, attention distribution, and regional features are also guided, thereby improving the language ability recovery effect and reducing the risk of visual localization ability decline.

[0078] (3) A distillation training method with sample-level and token-level dynamic perception capabilities is realized.

[0079] By explicitly introducing sample-level or token-level capability requirement analysis during the distillation control phase, the distillation training process is made capable of fine-grained dynamic adjustment, providing a reliable training mechanism foundation for the subsequent stable training, comprehensive capability improvement, and multimodal collaborative optimization of the visual language model.

[0080] Figure 1 A flowchart of the visual language model dynamic multi-teacher hierarchical distillation training method of the present invention is shown. Figure 1 As shown, the visual language model dynamic multi-teacher hierarchical distillation training method of the present invention includes the following steps: S1: Training sample construction and model initialization.

[0081] First, prepare the training sample set. :

[0082] in: Indicates the first Input images of training samples; Indicates the first The text input for each training sample, such as questions, instructions, and prompts; This represents the target output sequence, i.e., the output result at each output position; Indicates the first in the target output sequence The target output, that is, the first target output. The output results at each output position; This represents the total number of target outputs in the target output sequence; The training sample set The total number of training samples in the dataset. Then, three models are initialized: student visual language model. Language teacher model Visual positioning teacher model . Among them, the model parameters of the student visual language model Trainable language teacher model Model parameters of the visual positioning teacher model Freezing, that is, .

[0083] In this invention, the student visual language model refers to a visual language model to be trained or optimized, used to receive image input and text input, and output corresponding text results. The student visual language model is the target model in this invention that receives distillation supervision and undergoes parameter updates. The language teacher model refers to a teacher model that provides language ability-related supervision signals to the student visual language model during distillation training, mainly used to convey abilities in language comprehension, knowledge memorization, text generation, and language reasoning. The language teacher model is preferably the original language model before multimodal adaptation of the student visual language model, or other text models with strong language capabilities. The visual positioning teacher model refers to a teacher model that provides visual positioning-related supervision signals to the student visual language model during distillation training, mainly used to convey abilities in OCR recognition, field positioning, region alignment, layout understanding, and accurate visual reading. The visual positioning teacher model can be a document understanding model, an OCR expert model, a region positioning model, or other models with accurate visual positioning capabilities.

[0084] S2: Obtaining the hidden state sequence and output distribution.

[0085] The training samples are input into the student visual language model to obtain the hidden state sequence and output distribution of the student visual language model, which specifically includes: 1. Encode the image input using the image encoder in the student visual language model to obtain a visual token sequence representation:

[0086] in, Indicates the first A visual token sequence representation of each training sample; Indicates an image encoder; Indicates the first Input images of training samples; Indicates the first Input images of training samples The Feature vector of a visual token; Indicates the first Input images of training samples The total number of visual tokens.

[0087] 2. Encode the text input using a text encoder or text embedding layer in the student visual language model to obtain a text token sequence representation:

[0088] in, Indicates the first The text token sequence representation of each training sample; Indicates the first Text input for each training sample; This indicates a text encoder or text embedding layer; Indicates the first Text input of training samples The Feature vectors of a text token; Indicates the first Text input of training samples The total number of text tokens.

[0089] In some implementations, visual token sequences are represented Represented by text token sequence They can be represented in matrix form respectively:

[0090] in, The dimension of the feature vector representing the visual token; The dimension of the feature vector of the text token.

[0091] To achieve cross-modal fusion, the visual token sequence can be further represented. and text token sequence representation Mapping to a unified feature space, we get:

[0092] in, and Represents the feature projection function; and This maps the visual token sequence representation and text token sequence representation to a unified dimension. Subsequent uses of both visual and text token sequence representations can adopt the unified dimension sequence representation.

[0093] 3. Input the visual token sequence representation and the text token sequence representation into the multimodal fusion module or decoding module of the student visual language model to obtain the hidden state sequence of each layer of the student visual language model:

[0094]

[0095] in, Indicates based on the first The first training sample yielded the student visual language model. The hidden state sequence of the layer; The first character representing the student's visual language model The multimodal representation transformation process of the layer; Indicates based on the first The first training sample yielded the student visual language model. The layer in the first Hidden state vectors at each output position; This represents the total number of layers in the student's visual language model; The first character representing the student's visual language model The total number of output positions of the layer, that is, the total number of target outputs of the target output sequence.

[0096] 4. Use the hidden state sequence of the last layer of the student visual language model as the input to the output layer of the student visual language model, and map the hidden state vector at each output position to the corresponding prediction score in the vocabulary space through the output layer:

[0097] in, Indicates based on the first The last layer (the first) of the student visual language model obtained from the training samples. (layer) in the first Hidden state vectors at each output position; This represents the output projection matrix of the output layer of the student visual language model; This represents the bias term of the output layer of the student visual language model; Indicates based on the first The output layer of the student visual language model obtained from the training samples is at the . The predicted score for each candidate token in the vocabulary at each output position.

[0098] 5. Based on the predicted score at each output position, obtain the overall predicted score sequence for each training sample:

[0099] in, Indicates based on the first The overall predicted score sequence of the student visual language model is obtained from the training samples.

[0100] 6. Normalize the overall predicted scores to obtain the output distribution of the student visual language model:

[0101] in, Indicates based on the first The output distribution of the student visual language model obtained from training samples; Indicates based on the first The student visual language model obtained from the training samples is in the _th ... Output at each output position; This represents the student's temperature coefficient.

[0102] S3: Construction of sample-level dynamic distillation control coefficients.

[0103] To avoid the coarse-grained division based solely on dataset categories, as is the case with existing technologies, this invention first calculates a sample-level control vector for each training sample. That is, it constructs sample-level dynamic distillation control coefficients based on the sample features of the training sample. These sample-level dynamic control coefficients refer to the distillation weight parameters generated for a single training sample, representing the participation ratio of the language teacher model, the visual localization teacher model, and the hard-labeled supervision signal on that training sample, thereby achieving differentiated distillation control for samples with different ability requirements.

[0104] Specifically, the constructed sample-level dynamic distillation control coefficients are:

[0105]

[0106]

[0107] in, Indicates based on the first Sample-level dynamic distillation control coefficients obtained from training samples; Indicates based on the first Language teacher distillation weights obtained from training samples; Indicates based on the first The visual positioning teacher distillation weights obtained from each training sample; Indicates based on the first Hard-labeled supervision weights obtained from each training sample; Indicates a gating network; Indicates the first Sample features of each training sample.

[0108] In this invention, the sample features This may include: question type characteristics; OCR token ratio; image layout complexity; student visual language model attention distribution; text length, number of regions, table density, etc. S4: Construction of token-level dynamic distillation control coefficients.

[0109] Based on the sample-level dynamic distillation control coefficients, further refined control is applied to each token position to construct token-level dynamic distillation control coefficients. These token-level dynamic control coefficients refer to the distillation weight parameters generated at a specific output token position during the student visual-language model's generation process. They represent the participation ratio of the language teacher model, the visual localization teacher model, and the hard-labeled supervision signal at that generation position, enabling fine-grained adjustment of the capability requirements for different generation positions.

[0110] For the The first sample Each token defines a token-level dynamic distillation control coefficient.

[0111]

[0112]

[0113] in, Indicates based on the first The training samples obtained at the th th Token-level dynamic distillation control coefficients at each output position; Indicates based on the first The training samples obtained at the th th Token-level language teacher distillation weights at each output position; Indicates based on the first The training samples obtained at the th th Token-level visual positioning teacher distillation weights at each output position; Indicates based on the first The training samples obtained at the th th Token-level hard-label supervision weights at each output position; Indicates based on the first The training samples obtained at the th th Token-level refined language teacher distillation weights at each output position; Indicates based on the first The training samples obtained at the th th Token-level refined visual positioning teacher distillation weights at each output position; Indicates based on the first The training samples obtained at the th th Token-level refined hard-label supervision weights at each output position; These are trainable parameters; Indicates based on the first The last layer of the student visual language model obtained from the training samples is in the... Hidden state vectors at each output position; It is a vector normalization function; This indicates element-wise multiplication.

[0114] S5: Language distillation signal generation.

[0115] Based on the training samples and the hidden state sequence of the student visual language model, the language teacher model generates a language distillation signal, which specifically includes: 1. Obtain the multimodal conditional representation from the hidden state sequence of the student's visual language model:

[0116] In the formula, Indicates based on the first The first training sample yielded the student visual language model. Multimodal conditional representation of layers; Indicates based on the first The first training sample yielded the student visual language model. The hidden state sequence of the layer; This represents a mapping function.

[0117] 2. Input the text of the training samples and the multimodal conditional representation into the language teacher model to generate the hidden state sequence of the language teacher model:

[0118] in, Indicates based on the first The language teacher model obtained from the training samples. The hidden state sequence of the layer; Indicates based on the first The language teacher model obtained from the training samples. The layer in the first Hidden state vectors at each output position; The first language teacher model The total number of output positions of the layer, that is, the total number of target outputs of the target output sequence; Indicates the first Text input for each training sample.

[0119] 3. The hidden state sequence of the last layer of the language teacher model is used as the input to the output layer of the language teacher model, and the hidden state vector at each output position is mapped to the corresponding predicted score in the vocabulary space through the output layer:

[0120] in, Indicates based on the first The last layer (the first) of the language teacher model obtained from the training samples. (layer) in the first Hidden state vectors at each output position; This represents the output projection matrix of the output layer of the language teacher model; This represents the bias term of the output layer of the language teacher model; Indicates based on the first The output layer of the language teacher model obtained from the training samples is in the th training sample. The predicted score for each candidate token in the vocabulary at each output position.

[0121] 4. Based on the predicted score at each output position, obtain the overall predicted score sequence for each training sample:

[0122] in, Indicates based on the first The overall predicted score sequence of the language teacher model is obtained from the training samples.

[0123] 5. Normalize the overall predicted score sequence to obtain the output distribution of the language teacher model:

[0124] in, Indicates based on the first The output distribution of the language teacher model obtained from the training samples, i.e., the language distillation signal; Indicates based on the first The language teacher model obtained from the training samples is in the . Output at each output position; This represents the temperature coefficient of language teachers.

[0125] S6: Visual distillation signal generation.

[0126] Based on the training samples and the hidden state sequence of the student visual language model, a visual distillation signal is generated by the visual localization teacher model, which specifically includes: 1. Obtain the multimodal conditional representation from the hidden state sequence of the student's visual language model:

[0127] In the formula, Indicates based on the first The first training sample yielded the student visual language model. Multimodal conditional representation of layers; Indicates based on the first The first training sample yielded the student visual language model. The hidden state sequence of the layer; This represents a mapping function.

[0128] 2. Input the images of the training samples, the OCR region set, the layout structure information, and the multimodal conditional representation into the visual localization teacher model to generate the hidden state sequence of the visual localization teacher model:

[0129] in, Indicates based on the first The visual positioning teacher model obtained from the training samples. The hidden state sequence of the layer; Indicates based on the first The visual positioning teacher model obtained from the training samples. The layer in the first Hidden state vectors at each output position; The first step in representing the visual positioning teacher model The total number of output positions of the layer, that is, the total number of target outputs of the target output sequence; Indicates the first Input images of training samples; Indicates the first Input images of training samples The set of OCR regions can be based on the first Input images of training samples get; Indicates the first Input images of training samples The layout structure information can be based on the first Input images of training samples get.

[0130] 3. The hidden state sequence of the last layer of the visual localization teacher model is used as the input to the output layer of the visual localization teacher model, and the hidden state vector at each output position is mapped to the corresponding prediction score in the vocabulary space through the output layer:

[0131] in, Indicates based on the first The last layer (the first) of the visual positioning teacher model obtained from the training samples. (layer) in the first Hidden state vectors at each output position; This represents the output projection matrix of the output layer of the visual positioning teacher model; This represents the bias term of the output layer of the visual positioning teacher model; Indicates based on the first The output layer of the visual positioning teacher model obtained from the training samples is at the ... The predicted score for each candidate token in the vocabulary at each output position.

[0132] 4. Based on the predicted score at each output position, obtain the overall predicted score sequence for each training sample:

[0133] in, Indicates based on the first The overall prediction score sequence of the visual positioning teacher model obtained from the training samples.

[0134] 5. Normalize the overall predicted score sequence to obtain the output distribution of the visual localization teacher model:

[0135] in, Indicates based on the first The output distribution of the visual localization teacher model obtained from the training samples, i.e., the visual distillation signal; Indicates based on the first The visual positioning teacher model obtained from the training samples is in the . Output at each output position; This represents the temperature coefficient of the visual positioning teacher model.

[0136] In this invention, if the visual localization teacher model outputs not only token-level output distribution but also region localization or layout attention information, then visual region features can also be defined. Specifically, based on the OCR detection results, field detection results, layout analysis results, or candidate region proposal results of the input images of the training samples, a set of visual regions is constructed:

[0137] in, Indicates based on the first Input images of training samples The resulting set of visual regions; Indicates based on the first Input images of training samples The obtained number One visual area; For the first Input images of training samples The total number of visual regions obtained.

[0138] Then, the visual positioning teacher model performs region-level feature extraction on the set of visual regions to obtain the teacher region feature sequence.

[0139] That is, regarding the first Input images of training samples The visual area It can be based on the input image The visual area and their corresponding position codes or page layout codes (From the input image) (Obtained) Extract teacher region feature sequence:

[0140]

[0141] in, The visual positioning teacher model is based on the first Input images of training samples Extracted teacher region feature sequences; This represents the region feature extraction function of the visual positioning teacher model; The visual positioning teacher model is based on the first Input images of training samples The Teacher region features extracted from visual regions.

[0142] Similarly, for the student visual language model, targeting the first Input images of training samples The visual area It can be based on the input image The visual area and their corresponding position codes or page layout codes (From the input image) (Obtained) Extract student region feature sequence:

[0143]

[0144] in, The student visual language model is based on the first Input images of training samples Extracted regional feature sequences; A function representing the region feature extraction function of a student's visual language model; The student visual language model is based on the first Input images of training samples The Student region features extracted from visual regions.

[0145] S7: Calculation of distillation loss for multiple teachers in the output layer.

[0146] In this invention, the distillation loss of the language teacher model, the distillation loss of the visual localization teacher model, and the hard-label supervision loss are calculated based on the output distribution of the student visual language model, the language distillation signal, the visual distillation signal, the token-level dynamic distillation control coefficients, and the target output sequence. Specifically, this includes: 1. Distillation Loss in Computational Language Teacher Model :

[0147] in, This represents the total number of training samples; This represents the total number of target outputs in the target output sequence; Indicates based on the first The training sample obtained from the first training sample Token-level language teacher distillation weights at each output position; Indicates the temperature coefficient of language teachers; Indicates based on the first The language teacher model obtained from the training samples is in the . Output at each output position; Indicates based on the first The student visual language model obtained from the training samples is in the _th ... Output at each output position.

[0148] 2. Distillation Loss of the Computational Visual Localization Teacher Model :

[0149] in, Indicates based on the first The training sample obtained from the first training sample Token-level visual positioning teacher distillation weights at each output position; Indicates the visual positioning teacher's temperature coefficient; Indicates based on the first The visual positioning teacher model obtained from the training samples is in the . Output at each output position.

[0150] 3. Calculate the hard-label supervision loss ;

[0151] in, Indicates the first The target output sequence of the training samples is the first one. One target output; Indicates based on the first The training sample obtained from the first training sample Token-level hard-label supervision weights at each output position; Indicates based on the first The output layer of the student visual language model obtained from the training samples is at the . The predicted score for each candidate token in the vocabulary at each output position; This represents the cross-entropy loss.

[0152] S8: Calculation of distillation loss for intermediate characterization.

[0153] To enable the student visual language model to not only approximate the language teacher model and the visual positioning teacher model at the output level, but also to learn the internal capability structure of the language teacher model and the visual positioning teacher model at the intermediate semantic representation and intermediate visual positioning representation levels, this invention applies distillation constraints to the corresponding layer hidden states of the student visual language model, the language teacher model, and the visual positioning teacher model. Specifically, it calculates the intermediate layer distillation loss based on the hidden state sequences of the student visual language model, the language teacher model, and the visual positioning teacher model, and calculates the attention distillation loss based on the attention matrices of the student visual language model, the language teacher model, and the visual positioning teacher model. This specifically includes: 1. Calculate the intermediate layer distillation loss .

[0154] Since the network structures, hidden dimensions, or representation spaces of the student visual language model, language teacher model, and visual localization teacher model may differ, a mapping layer is first used to align the hidden state sequences of the language teacher model and the visual localization teacher model, resulting in:

[0155]

[0156] in, and These represent the effects applied to the language teacher model. Hidden state sequence of layer The first visual positioning teacher model Hidden state sequence of layer The mapping function is used to connect the language teacher model and the visual positioning teacher model. The hidden state sequence of the layer is mapped to the first layer of the student visual language model. The hidden state sequences of the layers are consistent or comparable feature spaces; Represents the dimension-aligned representation based on the first dimension. The language teacher model obtained from the training samples. The hidden state sequence of the layer; Represents the dimension-aligned representation based on the first dimension. The visual positioning teacher model obtained from the training samples. The hidden state sequence of the layer.

[0157] Then, define the intermediate layer distillation loss. :

[0158] in, This represents the total number of training samples; This represents the total number of layers in the student's visual language model; Indicates based on the first The first training sample yielded the student visual language model. The hidden state sequence of the layer; Indicates based on the first The language teacher model obtained from the training samples. Layer language distillation weights; Indicates based on the first The visual positioning teacher model obtained from the training samples. Layer language distillation weights.

[0159] In this invention, it is possible to make in, The first language teacher model The monotonic mapping function of the layer (which can be linear or nonlinear) is used to... Mapped to ; The first step in representing the visual positioning teacher model The monotonic mapping function of the layer (which can be linear or nonlinear) is used to... Mapped to .

[0160] 2. Calculate the attention distillation loss. :

[0161] in, , and They are based on the first The student visual language model, language teacher model, and visual positioning teacher model obtained from the training samples are the first... Attention matrix of the layer; Indicates based on the first The language teacher model obtained from the training samples. Attention distillation weights of the layer; Indicates based on the first The visual positioning teacher model obtained from the training samples. Attention distillation weights for layers.

[0162] In this invention, it is possible to make in, The first language teacher model Another monotonic mapping function (which can be linear or nonlinear) of the layer is used to... Mapped to ; The first step in representing the visual positioning teacher model Another monotonic mapping function (which can be linear or nonlinear) of the layer is used to... Mapped to .

[0163] 3. Calculate the distillation loss by aligning the regions.

[0164] For scenarios where OCR / documents exist in the training samples, region alignment distillation loss can also be added. :

[0165] in, The visual positioning teacher model is based on the first Input images of training samples The Teacher region features extracted from visual regions; The student visual language model is based on the first Input images of training samples The Student region features extracted from visual regions.

[0166] This allows the student's visual language model to not only approximate the visual positioning teacher model in terms of answer, but also in terms of field positioning and page area selection.

[0167] S9: Construct the total loss function.

[0168] Based on the aforementioned language teacher model distillation loss, visual positioning teacher model distillation loss, hard-label supervision loss, intermediate layer distillation loss, attention distillation loss, and region alignment distillation loss. Construct the total loss function (if it exists).

[0169] In this invention, the total loss function for:

[0170] in: These are global hyperparameters. S10: Distillation training.

[0171] The student visual language model is distilled and trained based on the total loss function to update its model parameters.

[0172] In each training step, the total loss function is minimized:

[0173] in, This is the learning rate.

[0174] Furthermore, the model parameters of the language teacher model and the visual positioning teacher model remain unchanged: .

[0175] After distillation training is completed, only the student's visual language model is retained. During inference, the language teacher model, the visual positioning teacher model, and the dynamic distillation calculation-related content can all be removed. Therefore, this invention does not increase the number of inference parameters or inference overhead during final deployment.

[0176] Figure 2 A schematic diagram of the structure of the visual language model dynamic multi-teacher hierarchical distillation training system of the present invention is shown. Figure 2 As shown, the visual language model dynamic multi-teacher hierarchical distillation training system of the present invention includes: 1. Training sample construction and model initialization module.

[0177] The training sample construction and model initialization module is used to construct training samples and initialize the student visual language model, language teacher model, and visual positioning teacher model. The training samples include image input, text input, and target output sequence.

[0178] 2. Hidden state sequence and output distribution acquisition module.

[0179] The hidden state sequence and output distribution acquisition module is used to input the training samples into the student visual language model to obtain the hidden state sequence and output distribution of the student visual language model.

[0180] 3. Sample-level dynamic distillation control coefficient construction module.

[0181] The sample-level dynamic distillation control coefficient construction module is used to construct sample-level dynamic distillation control coefficients based on the sample features of the training samples.

[0182] 4. Token-level dynamic distillation control coefficient construction module.

[0183] The token-level dynamic distillation control coefficient construction module is used to construct token-level dynamic distillation control coefficients based on the sample-level dynamic distillation control coefficients and the hidden state sequence of the student visual language model.

[0184] 5. Language distillation signal generation module.

[0185] The language distillation signal generation module is used to generate language distillation signals from the language teacher model based on the training samples and the hidden state sequence of the student visual language model.

[0186] 6. Visual distillation signal generation module.

[0187] The visual distillation signal generation module is used to generate a visual distillation signal from the visual localization teacher model based on the training samples and the hidden state sequence of the student visual language model.

[0188] 7. Output layer multi-teacher distillation loss calculation module.

[0189] The output layer multi-teacher distillation loss calculation module is used to calculate the language teacher model distillation loss, visual positioning teacher model distillation loss, and hard-label supervision loss based on the output distribution of the student visual language model, the language distillation signal, the visual distillation signal, the token-level dynamic distillation control coefficient, and the target output sequence.

[0190] 8. Intermediate characterization distillation loss calculation module.

[0191] The intermediate representation distillation loss calculation module is used to calculate the intermediate layer distillation loss based on the hidden state sequence of the student visual language model, the hidden state sequence of the language teacher model, and the hidden state sequence of the visual localization teacher model, and to calculate the attention distillation loss based on the attention matrix of the student visual language model, the attention matrix of the language teacher model, and the attention matrix of the visual localization teacher model.

[0192] 9. Total loss function construction module.

[0193] The total loss function construction module is used to construct the total loss function based on the language teacher model distillation loss, the visual localization teacher model distillation loss, the hard label supervision loss, the intermediate layer distillation loss, and the attention distillation loss.

[0194] 10. Distillation Training Module.

[0195] The distillation training module is used to perform distillation training on the student visual language model based on the total loss function in order to update its model parameters.

[0196] Furthermore, this invention also provides a dynamic multi-teacher hierarchical distillation training device for visual language models. For example... Figure 3 As shown, the visual language model dynamic multi-teacher hierarchical distillation training device of the present invention includes: a memory 11 for storing one or more programs; one or more processors 12; when the one or more programs are executed by the one or more processors 12, the one or more processors 12 implement the visual language model dynamic multi-teacher hierarchical distillation training method of the present invention. Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the visual language model dynamic multi-teacher hierarchical distillation training method of the present invention.

[0197] The computer-readable storage medium includes both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined in this invention, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0198] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0199] The steps of the methods or algorithms described in conjunction with the embodiments disclosed in this invention can be implemented in hardware, software modules executed by a processor, or a combination of both. The software modules can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention based on the concept of the present invention, without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A method for dynamic multi-teacher hierarchical distillation training of a visual language model, characterized in that, Includes the following steps: S1: Construct training samples and initialize the student visual language model, language teacher model, and visual positioning teacher model. The training samples include image input, text input, and target output sequence. S2: Input the training samples into the student visual language model to obtain the hidden state sequence and output distribution of the student visual language model; S3: Construct sample-level dynamic distillation control coefficients based on the sample features of the training samples; S4: Construct token-level dynamic distillation control coefficients based on the sample-level dynamic distillation control coefficients and the hidden state sequence of the student visual language model; S5: Based on the training samples and the hidden state sequence of the student visual language model, the language teacher model generates a language distillation signal; S6: Based on the training samples and the hidden state sequence of the student visual language model, a visual distillation signal is generated by the visual localization teacher model; S7: Calculate the distillation loss of the language teacher model, the distillation loss of the visual positioning teacher model, and the hard label supervision loss based on the output distribution of the student visual language model, the language distillation signal, the visual distillation signal, the token-level dynamic distillation control coefficient, and the target output sequence. S8: Calculate the intermediate layer distillation loss based on the hidden state sequence of the student visual language model, the hidden state sequence of the language teacher model, and the hidden state sequence of the visual localization teacher model; calculate the attention distillation loss based on the attention matrix of the student visual language model, the attention matrix of the language teacher model, and the attention matrix of the visual localization teacher model. S9: Construct a total loss function based on the distillation loss of the language teacher model, the distillation loss of the visual localization teacher model, the hard label supervision loss, the intermediate layer distillation loss, and the attention distillation loss; S10: Distill the student visual language model based on the total loss function to update its model parameters.

2. The method for dynamic multi-teacher hierarchical distillation training of visual language models according to claim 1, characterized in that, Step S2 specifically includes: S21: Encode the image input using the image encoder in the student visual language model to obtain a visual token sequence representation: in, Indicates the first A visual token sequence representation of each training sample; Indicates an image encoder; Indicates the first Input images of training samples; Indicates the first Input images of training samples The Feature vector of a visual token; Indicates the first Input images of training samples The total number of visual tokens; S22: Encode the text input using a text encoder or text embedding layer in the student visual language model to obtain a text token sequence representation: in, Indicates the first The text token sequence representation of each training sample; Indicates the first Text input for each training sample; This indicates a text encoder or text embedding layer; Indicates the first Text input of training samples The Feature vectors of a text token; Indicates the first Text input of training samples The total number of text tokens; S23: Input the visual token sequence representation and the text token sequence representation into the multimodal fusion module or decoding module of the student visual language model to obtain the hidden state sequence of each layer of the student visual language model: in, Indicates based on the first The first training sample yielded the student visual language model. The hidden state sequence of the layer; The first character representing the student's visual language model The multimodal representation transformation process of the layer; Indicates based on the first The first training sample yielded the student visual language model. The layer in the first Hidden state vectors at each output position; This represents the total number of layers in the student's visual language model; The first character representing the student's visual language model The total number of output positions of the layer, that is, the total number of target outputs of the target output sequence; S24: Use the hidden state sequence of the last layer of the student visual language model as the input to the output layer of the student visual language model, and map the hidden state vector of each output position to the corresponding prediction score in the vocabulary space through the output layer: in, Indicates based on the first The last layer of the student visual language model obtained from the training samples is in the... Hidden state vectors at each output position; This represents the output projection matrix of the output layer of the student visual language model; This represents the bias term of the output layer of the student visual language model; Indicates based on the first The output layer of the student visual language model obtained from the training samples is at the . The predicted score for each candidate token in the vocabulary at each output position; S25: Based on the predicted score at each output position, obtain the overall predicted score sequence for each training sample: in, Indicates based on the first The overall predicted score sequence of the student visual language model obtained from training samples; S26: Normalize the overall prediction score to obtain the output distribution of the student visual language model: in, Indicates based on the first The output distribution of the student visual language model obtained from training samples; Indicates based on the first The student visual language model obtained from the training samples is in the _th ... Output at each output position; This represents the student's temperature coefficient.

3. The method for dynamic multi-teacher hierarchical distillation training of visual language models according to claim 1, characterized in that, Step S3 specifically involves: in, Indicates based on the first Sample-level dynamic distillation control coefficients obtained from training samples; Indicates based on the first Language teacher distillation weights obtained from training samples; Indicates based on the first The visual positioning teacher distillation weights obtained from each training sample; Indicates based on the first Hard-labeled supervision weights obtained from each training sample; Indicates a gating network; Indicates the first Sample features of each training sample.

4. The method for dynamic multi-teacher hierarchical distillation training of visual language models according to claim 3, characterized in that, Step S4 specifically involves: in, Indicates based on the first The training samples obtained at the th th Token-level dynamic distillation control coefficients at each output position; Indicates based on the first The training samples obtained at the th th Token-level language teacher distillation weights at each output position; Indicates based on the first The training samples obtained at the th th Token-level visual positioning teacher distillation weights at each output position; Indicates based on the first The training samples obtained at the th th Token-level hard-label supervision weights at each output position; Indicates based on the first The training samples obtained at the th th Token-level refined language teacher distillation weights at each output position; Indicates based on the first The training samples obtained at the th th Token-level refined visual positioning teacher distillation weights at each output position; Indicates based on the first The training samples obtained at the th th Token-level refined hard-label supervision weights at each output position; These are trainable parameters; Indicates based on the first The last layer of the student visual language model obtained from the training samples is in the... The hidden state vector at each output position.

5. The method for dynamic multi-teacher hierarchical distillation training of visual language models according to claim 1, characterized in that, Step S5 specifically involves: S51: Obtain the multimodal conditional representation from the hidden state sequence of the student's visual language model: In the formula, Indicates based on the first The first training sample yielded the student visual language model. Multimodal conditional representation of layers; Indicates based on the first The first training sample yielded the student visual language model. The hidden state sequence of the layer; Represents a mapping function; S52: Input the text of the training samples and the multimodal conditional representation into the language teacher model to generate the hidden state sequence of the language teacher model: in, Indicates based on the first The language teacher model obtained from the training samples. The hidden state sequence of the layer; Indicates based on the first The language teacher model obtained from the training samples. The layer in the first Hidden state vectors at each output position; The first language teacher model The total number of output positions of the layer, that is, the total number of target outputs of the target output sequence; Indicates the first Text input for each training sample; Represents a language teacher model; S53: Use the hidden state sequence of the last layer of the language teacher model as the input to the output layer of the language teacher model, and map the hidden state vector of each output position to the corresponding predicted score in the vocabulary space through the output layer: in, Indicates based on the first The last layer of the language teacher model obtained from the training samples is in the... Hidden state vectors at each output position; This represents the output projection matrix of the output layer of the language teacher model; This represents the bias term of the output layer of the language teacher model; Indicates based on the first The output layer of the language teacher model obtained from the training samples is in the th training sample. The predicted score for each candidate token in the vocabulary at each output position; S54: Based on the predicted score at each output position, obtain the overall predicted score sequence for each training sample: in, Indicates based on the first The overall predicted score sequence of the language teacher model obtained from training samples; S55: Normalize the overall predicted score sequence to obtain the output distribution of the language teacher model: in, Indicates based on the first The output distribution of the language teacher model obtained from the training samples, i.e., the language distillation signal; Indicates based on the first The language teacher model obtained from the training samples is in the . Output at each output position; This represents the temperature coefficient of language teachers.

6. The method for dynamic multi-teacher hierarchical distillation training of visual language models according to claim 1, characterized in that, Step S7 specifically includes: S71: Distillation Loss in Computational Language Teacher Model : in, This represents the total number of training samples; This represents the total number of target outputs in the target output sequence; Indicates based on the first The training sample obtained from the first training sample Token-level language teacher distillation weights at each output position; Indicates the temperature coefficient of language teachers; Indicates based on the first The language teacher model obtained from the training samples is in the . Output at each output position; Indicates based on the first The student visual language model obtained from the training samples is in the _th ... Output at each output position; S72: Distillation Loss of Computational Visual Localization Teacher Model : in, Indicates based on the first The training sample obtained from the first training sample Token-level visual positioning teacher distillation weights at each output position; Indicates the visual positioning teacher's temperature coefficient; Indicates based on the first The visual positioning teacher model obtained from the training samples is in the . Output at each output position; S73: Calculate the hard-label supervision loss ; in, Indicates the first The target output sequence of the training samples is the first one. One target output; Indicates based on the first The training sample obtained from the first training sample Token-level hard-label supervision weights at each output position; Indicates based on the first The output layer of the student visual language model obtained from the training samples is at the . The predicted score for each candidate token in the vocabulary at each output position.

7. The method for dynamic multi-teacher hierarchical distillation training of visual language models according to claim 1, characterized in that, Step S8 specifically includes: S81: Perform dimension alignment on the hidden state sequences of the student language model, the language teacher model, and the visual positioning teacher model to obtain the dimension-aligned hidden state sequences of the language teacher model and the visual positioning teacher model. S82: Calculate the intermediate layer distillation loss : in, This represents the total number of training samples; This represents the total number of layers in the student's visual language model; Indicates based on the first The first training sample yielded the student visual language model. The hidden state sequence of the layer; Represents the dimension-aligned representation based on the first dimension. The language teacher model obtained from the training samples. The hidden state sequence of the layer; Represents the dimension-aligned representation based on the first dimension. The visual positioning teacher model obtained from the training samples. The hidden state sequence of the layer; Indicates based on the first The language teacher model obtained from the training samples. Layer language distillation weights; Indicates based on the first The visual positioning teacher model obtained from the training samples. Layer language distillation weights; S83: Calculate the attention distillation loss : in, , and They are based on the first The student visual language model, language teacher model, and visual positioning teacher model obtained from the training samples are the first... Attention matrix of the layer; Indicates based on the first The language teacher model obtained from the training samples. Attention distillation weights of the layer; Indicates based on the first The visual positioning teacher model obtained from the training samples. Attention distillation weights for layers.

8. A dynamic multi-teacher hierarchical distillation training system for visual language models, characterized in that, include: The training sample construction and model initialization module is used to construct training samples and initialize the student visual language model, language teacher model and visual positioning teacher model. The training samples include image input, text input and target output sequence. The hidden state sequence and output distribution acquisition module is used to input the training samples into the student visual language model to obtain the hidden state sequence and output distribution of the student visual language model. A sample-level dynamic distillation control coefficient construction module is used to construct sample-level dynamic distillation control coefficients based on the sample features of the training samples. The token-level dynamic distillation control coefficient construction module is used to construct token-level dynamic distillation control coefficients based on the sample-level dynamic distillation control coefficients and the hidden state sequence of the student visual language model. A language distillation signal generation module is used to generate a language distillation signal from a language teacher model based on the training samples and the hidden state sequence of the student visual language model. A visual distillation signal generation module is used to generate a visual distillation signal from a visual localization teacher model based on the training samples and the hidden state sequence of the student visual language model. The output layer multi-teacher distillation loss calculation module is used to calculate the distillation loss of the language teacher model, the distillation loss of the visual positioning teacher model, and the hard label supervision loss based on the output distribution of the student visual language model, the language distillation signal, the visual distillation signal, the token-level dynamic distillation control coefficient, and the target output sequence. The intermediate representation distillation loss calculation module is used to calculate the intermediate layer distillation loss based on the hidden state sequence of the student visual language model, the hidden state sequence of the language teacher model, and the hidden state sequence of the visual positioning teacher model, and to calculate the attention distillation loss based on the attention matrix of the student visual language model, the attention matrix of the language teacher model, and the attention matrix of the visual positioning teacher model. The total loss function construction module is used to construct the total loss function based on the language teacher model distillation loss, the visual localization teacher model distillation loss, the hard label supervision loss, the intermediate layer distillation loss, and the attention distillation loss. A distillation training module is used to perform distillation training on the student visual language model based on the total loss function to update its model parameters.

9. A dynamic multi-teacher hierarchical distillation training device for visual language models, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the visual language model dynamic multi-teacher hierarchical distillation training method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the dynamic multi-teacher hierarchical distillation training method for visual language models as described in any one of claims 1-7.

Images