Visual language model conversion method, apparatus and device

Abstract

The application discloses a visual language model conversion method, device and equipment, relates to the technical field of artificial intelligence and computer vision, and comprises the following steps: acquiring a target total parameter budget and a target activation parameter budget, and one or more candidate pre-training dense visual language models; performing a budget-aware structure search on the candidate pre-training dense visual language models under the constraint of the target total parameter budget and the target activation parameter budget, and outputting a target hybrid expert configuration meeting the constraint; converting a language model feedforward network in the candidate pre-training dense visual language model into a hybrid expert feedforward network according to the target hybrid expert configuration, performing scale calibration on the aggregated output of the converted hybrid expert feedforward layer, and generating a calibrated hybrid expert visual language model; performing stage-by-stage training on the calibrated hybrid expert visual language model, and outputting a visual language hybrid expert model. The application realizes efficient deployment of the model on a resource-limited platform.

Description

Technical Field

[0001] This application relates to the technical fields of artificial intelligence and computer vision, and in particular to a visual language model conversion method, apparatus and device. Background Technology

[0002] With the widespread application of visual language models in real-world scenarios such as image understanding, document analysis, and multimodal question answering, the deployment of these models on hardware platforms such as mobile devices, edge computing terminals, and memory-constrained servers imposes stringent resource constraints on the total parameter size, the number of activation parameters per inference iteration, memory usage, and inference efficiency. Hybrid expert models, through sparse activation mechanisms, enable models to maintain a large total capacity while controlling the computational load per forward iteration. Therefore, they have become an important technical approach for building various budget-level versions of visual language models on these resource-constrained platforms.

[0003] The common approach is to use a pre-trained dense visual language model as a foundation, transform the language-side feedforward network into a hybrid expert structure, and then restore the model's performance through subsequent training. However, existing technologies typically employ fixed activation ratios or empirical configurations. When faced with given hardware storage and computing power budgets in real-world deployment scenarios, they lack automated decision-making mechanisms oriented towards a predetermined budget. This makes it difficult to stably obtain a target structure that balances budget constraints and recovery quality in a discrete configuration space, resulting in the transformed model being unable to adapt to the storage or computing power limitations of the target device.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a visual language model conversion method, apparatus, and device, which aims to solve the technical problem of how to achieve efficient deployment of visual language models on resource-constrained platforms under given hardware constraints.

[0006] To achieve the above objectives, this application proposes a visual language model conversion method, the method comprising: Obtain the target total parameter budget and target activation parameter budget, as well as one or more candidate pre-trained dense visual language models; Using the target total parameter budget and target activation parameter budget as constraints, a budget-aware structure search is performed on the candidate pre-trained dense visual language model to output a target hybrid expert configuration that satisfies the constraints; According to the target hybrid expert configuration, the language model feedforward network in the candidate pre-trained dense visual language model is converted into a hybrid expert feedforward network, and scale calibration is performed on the aggregated output of the converted hybrid expert feedforward layer to generate a calibrated hybrid expert visual language model. The calibrated hybrid expert visual language model is trained in stages, and the final visual language hybrid expert model is output.

[0007] In one embodiment, the step of performing a budget-aware structure search on the candidate pre-trained dense visual language model and outputting a target hybrid expert configuration that satisfies the constraints includes: Extract key structural parameters from the candidate pre-trained dense visual language model. The key structural parameters include the hidden dimension of the language model, the dimension of the intermediate layer of the dense feedforward network, the number of layers of the language model, and the number of always-activated parameters. Construct a discrete configuration space consisting of the number of shards, the total number of experts, and the number of activated experts, and generate a set of candidate discrete configurations.

[0008] In one embodiment, the step of performing a budget-aware structure search on the candidate pre-trained dense visual language model and outputting a target hybrid expert configuration that satisfies the constraints further includes: For each configuration in the candidate discrete configuration set, a legality screening is performed, and configurations that do not satisfy the partitioning divisibility relationship or the constraint relationship between the number of activated experts and the total number of experts are eliminated, thus obtaining a legal configuration set; For each configuration in the set of legal configurations, budget estimation is performed to filter the configurations, the activation feedforward dimension and the total feedforward dimension are calculated, and the corresponding total parameter quantity and activation parameter quantity are calculated based on the always activated parameter quantity, language model hidden dimension, language model layer number, activation feedforward dimension and total feedforward dimension. Configurations that exceed the target total parameter budget or target activation parameter budget are removed to obtain the budget-satisfied configuration set. For each configuration in the budget-satisfying configuration set, perform structural matching screening, calculate the key structural ratio, and remove configurations whose key structural ratios do not fall within a preset reasonable range to obtain a structural matching configuration set. In the set of structure matching configurations, the configuration with the best score is selected as the target hybrid expert configuration based on a comprehensive ranking of budget deviation and training stability preset indicators.

[0009] In one embodiment, converting the language model feedforward network in the candidate pre-trained dense visual language model into a hybrid expert feedforward network according to the target hybrid expert configuration includes performing a structural transformation step: Based on the number of shards and the total number of experts in the target hybrid expert configuration, the intermediate layer of the dense feedforward network to be transformed is divided into equal-width shards according to the number of shards, so that each shard corresponds to a basic expert unit. Based on the total number of experts, the basic expert units are retained, copied, or combined to form an expert group with the total number of experts. The output consists of an initial hybrid expert feedforward network composed of experts equal to the total number of experts.

[0010] In one embodiment, the step of converting the language model feedforward network in the candidate pre-trained dense visual language model into a hybrid expert feedforward network according to the target hybrid expert configuration further includes performing a route aggregation step: Based on the initial hybrid expert feedforward network and the hidden representation of the output token, the routing score of the corresponding expert is output through the router in the transformed feedforward layer; Based on the routing scores, the Top K experts are selected as active experts, and the aggregation weights are calculated. The output of the activated expert is weighted and aggregated according to the aggregation weight to obtain the hybrid expert feedforward output of the current layer.

[0011] In one embodiment, the step of performing scale calibration on the aggregated output of the transformed hybrid expert feedforward layer to generate a calibrated hybrid expert visual language model includes: Construct mean matching coefficients, variance matching coefficients, or second moment matching coefficients respectively, and then generate scaling factors based on preset combinations; A dynamically determined scaling factor is introduced at the aggregation output of the hybrid expert feedforward layer; A scaling operation is performed on the hybrid expert feedforward output based on the scaling factor to obtain the calibrated hybrid expert feedforward output. The original dense feedforward output is replaced with the calibrated hybrid expert feedforward output to generate the calibrated hybrid expert visual language model.

[0012] In one embodiment, performing phased training on the calibrated hybrid expert visual language model and outputting the final visual language hybrid expert model includes performing a first-stage recovery training step: Based on the calibrated hybrid expert visual language model, the visual tower parameters and visual side alignment module parameters are frozen, and the language model and language side alignment related modules are trained. The training process is monitored until the preset stability conditions are reached, and the first-stage intermediate model with stable sparse structure on the language side is output.

[0013] In one embodiment, the step of performing phased training on the calibrated hybrid expert visual language model and outputting the final visual language hybrid expert model further includes performing a second-stage recovery training step: Based on the intermediate model of the first stage, the model parameters are unfrozen and end-to-end joint training is performed. In joint training, the learning rate or update pace of the visual parameters is lower than that of the language parameters, and a load balancing constraint is introduced in the joint training. When the joint training reaches the preset termination condition, the final visual-language hybrid expert model is output.

[0014] Furthermore, to achieve the above objectives, this application also proposes a visual language model conversion device, which includes: The data acquisition module is used to acquire the target total parameter budget and the target activation parameter budget, as well as one or more candidate pre-trained dense visual language models; The target configuration module is used to perform budget-aware structure search on the candidate pre-trained dense visual language model with the target total parameter budget and the target activation parameter budget as constraints, and output the target hybrid expert configuration that satisfies the constraints; The conversion calibration module is used to convert the language model feedforward network in the candidate pre-trained dense visual language model into a hybrid expert feedforward network according to the target hybrid expert configuration, and to perform scale calibration on the aggregated output of the converted hybrid expert feedforward layer to generate a calibrated hybrid expert visual language model. The model recovery module is used to perform phased training on the calibrated hybrid expert visual language model and output the final visual language hybrid expert model.

[0015] In addition, to achieve the above objectives, this application also proposes a visual language model conversion device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the visual language model conversion method as described above.

[0016] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the visual language model conversion method described above.

[0017] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the visual language model conversion method described above.

[0018] This application proposes a visual language model conversion method, apparatus, and device. The method includes: obtaining a target total parameter budget and a target activation parameter budget, as well as one or more candidate pre-trained dense visual language models; performing budget-aware structure search on the candidate pre-trained dense visual language models with the target total parameter budget and target activation parameter budget as constraints, and outputting a target hybrid expert configuration that satisfies the constraints; converting the language model feedforward network in the candidate pre-trained dense visual language models into a hybrid expert feedforward network according to the target hybrid expert configuration, and performing scale calibration on the aggregated output of the converted hybrid expert feedforward layer to generate a calibrated hybrid expert visual language model; performing phased training on the calibrated hybrid expert visual language model, and outputting the final visual language hybrid expert model. This application employs a series of techniques, including budget-aware structure search constrained by a target budget, scale calibration of the transformed aggregated output, and phased training of the model. These techniques address the shortcomings of existing technologies, such as the lack of automated decision-making mechanisms for predetermined budgets and the difficulty in stably obtaining target structures that balance budget constraints and recovery quality in discrete configuration spaces. Compared with existing technologies, this approach enables the automatic determination of suitable hybrid expert structures under given hardware budget constraints, eliminates output distribution shifts caused by structure transformations to improve training stability, and suppresses cascading errors in multimodal recovery to ensure final recovery quality. This allows for the efficient deployment of visual language models on resource-constrained platforms. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating an embodiment of the visual language model conversion method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the visual language model conversion method of this application. Figure 3 This is a flowchart illustrating Embodiment 3 of the visual language model conversion method of this application; Figure 4 This is a flowchart illustrating Embodiment 4 of the visual language model conversion method of this application; Figure 5This is a schematic diagram of the module structure of the visual language model conversion device according to an embodiment of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the visual language model conversion method in the embodiments of this application.

[0022] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0024] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0025] This application provides a solution that employs techniques such as budget-aware structure search constrained by a target budget, scale calibration of the transformed aggregated output, and phased training of the model. This addresses the problems of existing technologies lacking automated decision-making mechanisms for given budgets and struggling to stably obtain target structures that balance budget constraints and recovery quality in discrete configuration spaces. Compared with existing technologies, this solution achieves automatic determination of suitable hybrid expert structures under given hardware budget constraints, eliminates output distribution shifts caused by structure transformation to improve training stability, and suppresses cascading errors in multimodal recovery to ensure final recovery quality, thereby enabling efficient deployment of visual language models on resource-constrained platforms.

[0026] Based on this, embodiments of this application provide a visual language model conversion method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the visual language model conversion method of this application.

[0027] In this embodiment, the visual language model conversion method includes steps S10 to S40: Step S10: Obtain the target total parameter budget and the target activation parameter budget, as well as one or more candidate pre-trained dense visual language models.

[0028] It should be noted that, in this embodiment, the target total parameter budget refers to the upper limit of the overall parameter size of the model allowed in the deployment scenario. This upper limit is determined by the storage capacity of the target deployment device (such as a mobile device or edge computing terminal), which directly constrains the memory usage of the model on the device. The target activation parameter budget refers to the upper limit of the number of parameters actually involved in the calculation of the model in a single forward inference process. This upper limit is determined by the computing power and power consumption budget of the target deployment device, which affects the computing power consumption and response latency of the device in a single inference. The candidate pre-trained dense visual language model refers to a multimodal basic model with complete visual and language processing capabilities that has not been sparsed.

[0029] In one possible implementation, the target total parameter budget and the target activation parameter budget can be manually input by the user based on the target device's video memory capacity and computing power limit, or they can be automatically calculated and generated by the system based on the target device's hardware configuration; the candidate pre-trained dense visual language model can be multiple models of different sizes selected from the same model family, or it can be a heterogeneous model from different model families.

[0030] In a specific implementation scenario, a visual language model for image description generation is deployed on a mobile smartphone equipped with 8GB of RAM, of which a maximum of 3GB of GPU memory is allocated to model inference, and the peak computing power of the neural network accelerator is 15 TOPS. Based on this hardware configuration, the system automatically calculates and sets the target total parameter budget to 2 billion parameters (corresponding to the model weight file occupying approximately 3GB of GPU memory), and the target activation parameter budget to 600 million parameters (corresponding to a single forward inference that can be completed within 200ms with 15 TOPS computing power, meeting the user's requirement for real-time image description generation after taking a photo).

[0031] Step S20: Using the target total parameter budget and target activation parameter budget as constraints, perform budget-aware structure search on the candidate pre-trained dense visual language model and output the target hybrid expert configuration that satisfies the constraints.

[0032] It should be noted that, in this embodiment, budget-aware structure search refers to the process of automatically selecting the optimal model structure configuration within a discrete configuration space, using the target total parameter budget and the target activation parameter budget as hard constraints, combined with the structure matching criterion; target hybrid expert configuration refers to a combination of discrete parameters, including the number of fragments NS, the total number of experts NE, and the number of activated experts TopK, that satisfies the budget constraints and has the best structure matching degree.

[0033] This embodiment aims to overcome the structural mismatch and budget overrun problems caused by existing technologies that rely on fixed ratios or empirical configurations. By using a backward-engineering mechanism, it seeks feasible solutions that balance budget constraints and model recovery quality within the discrete operating space composed of sharding, expansion, and their combinations. This ensures that the total parameter size of the converted model does not exceed the storage capacity of the target device to reduce memory usage, and that the number of activated parameters does not exceed the computing power limit of the target device to reduce inference power consumption and ensure real-time task processing. At the same time, it ensures that the converted model has optimal structural stability under given resources.

[0034] Additionally, it should be noted that when performing budget-aware structure search, the system needs to extract parameters such as the hidden dimension and the intermediate layer dimension of the dense feedforward network of the candidate pre-trained dense visual language model, establish a candidate discrete configuration set, and sequentially perform legality screening, budget estimation screening, and structure matching screening. The structure matching screening depends on calculating the key structure ratio, that is, the ratio of the activation feedforward dimension to the hidden dimension.

[0035] In one possible implementation, the preset reasonable range of the key structure proportion in the structure matching screening can be determined in advance based on the offline statistical results of the structure proportion distribution of different model families. The system retains the candidate configurations whose key structure proportions fall within the preset reasonable range, and selects the one with the best score from the remaining configurations by comprehensively considering budget deviation and training stability.

[0036] Step S30: According to the target hybrid expert configuration, the language model feedforward network in the candidate pre-trained dense visual language model is converted into a hybrid expert feedforward network, and scale calibration is performed on the aggregated output of the converted hybrid expert feedforward layer to generate a calibrated hybrid expert visual language model.

[0037] It should be noted that, in this embodiment, the language model feedforward network refers to the dense fully connected layer structure in the visual language model that is responsible for feature transformation on the language side; the hybrid expert feedforward network refers to the sparse layer structure in which the dense feedforward layer is split and reorganized into multiple expert networks working in parallel and scheduled by a router; the aggregated output refers to the result of the activated expert outputs being weighted and summed according to the routing weights; the scale calibration refers to the operation of correcting the numerical distribution of the aggregated output by introducing a dynamic scaling factor to eliminate the distribution offset; the calibrated hybrid expert visual language model refers to the intermediate model that has completed the physical structure transformation and whose output distribution is aligned with the original dense model.

[0038] This embodiment, while completing the physical transformation of the model to achieve sparsity, overcomes the shift in output statistical mean and variance caused by the change in aggregation method, thereby reducing the initial loss and accelerating the convergence speed of subsequent fine-tuning. This shortens the deployment and debugging cycle of the model on target resource-constrained devices, enabling the model to reach usable accuracy more quickly in specific tasks such as image description generation and visual problems.

[0039] Additionally, it should be noted that during the structural transformation process, the system only performs transformation operations on the language model feedforward network, maintaining the visual tower and the always-activated part as the original dense structure. During the scale calibration process, the system can determine the scaling factor of the aggregated output by jointly considering expectation matching and fluctuation matching, so that the statistical characteristics of the hybrid expert feedforward output are as close as possible to the original dense state, thereby maintaining the output accuracy of the model in specific visual language tasks such as image understanding, document analysis, and multimodal question answering.

[0040] In one possible implementation, the scaling factor in the scaling operation is generated by constructing a mean matching coefficient to reflect the expected consistency and a second-moment matching coefficient to reflect the consistency of output fluctuations, respectively, and then generating the coefficient according to a preset combination rule of geometric mean.

[0041] Step S40: Perform phased training on the calibrated hybrid expert visual language model and output the final visual language hybrid expert model.

[0042] It should be noted that, in this embodiment, phased training refers to dividing the model recovery process into multiple independent training phases with different order and parameter update strategies; the final visual-language hybrid expert model refers to a usable model that has completed the entire training recovery process, has stable multimodal capabilities, and meets the target budget requirements.

[0043] Additionally, it should be noted that the phased training includes at least a first phase of recovery training and a second phase of joint fine-tuning. In the first phase of recovery training, the system freezes the parameters of the visual tower and visual-side alignment module, training only the language model and language-side alignment-related modules, prioritizing the recovery of the sparse language-side structure until stability is achieved. In the second phase of joint fine-tuning, the system unfreezes all or most parameters for end-to-end joint training, employing a conservative parameter update rhythm to reduce the risk of oscillations. Through this phased training strategy, the stability of the sparse language-side structure is restored first, followed by joint optimization of the visual and language sides, effectively suppressing the generation of cascading errors in the multimodal architecture. This ensures the accuracy recovery quality of the final model in multimodal tasks such as image description generation, visual question answering, and document information extraction, enabling it to provide service performance that meets business needs even on resource-constrained platforms.

[0044] This application searches for a target hybrid expert configuration with a target budget as a constraint, transforms the language model feedforward network into a hybrid expert structure, performs scale calibration on the aggregated output, and then outputs the final visual-language hybrid expert model through staged training. This scheme enables automatic decision-making and matching of hybrid expert structures on resource-constrained hardware platforms such as mobile devices and edge computing terminals to avoid exceeding the device's storage capacity or computing power limits. By eliminating the output distribution shift caused by structure transformation, it ensures the model's accuracy recovery in specific tasks such as image understanding and document analysis, while effectively reducing the device's power consumption and memory usage per inference, guaranteeing the real-time performance of visual-language task processing.

[0045] Further, the step of performing a budget-aware structure search on the candidate pre-trained dense visual language model and outputting a target hybrid expert configuration that satisfies the constraints includes A201~A202: Step A201: Extract the key structural parameters from the candidate pre-trained dense visual language model. The key structural parameters include the hidden dimension of the language model, the dimension of the intermediate layer of the dense feedforward network, the number of layers of the language model, and the number of always activated parameters. Step A202: Construct a discrete configuration space consisting of the number of shards, the total number of experts, and the number of activated experts, and generate a set of candidate discrete configurations.

[0046] It should be noted that, in this embodiment, the candidate pre-trained dense visual language model refers to a multimodal basic model with complete visual and language processing capabilities that has not undergone sparsification modification; key structural parameters include the language model hidden dimension, the dense feedforward network intermediate layer dimension, the number of language model layers, and the number of always-activated parameters; the language model hidden dimension refers to the width DH of the hidden layer of the language model; the dense feedforward network intermediate layer dimension refers to the width Ddense of the intermediate layer of the feedforward network; the number of language model layers refers to the number of language model layers L; the number of always-activated parameters refers to the number of parameters Pfixed that are always activated during the forward pass, including attention, embedding, normalization, and other parameters that do not participate in hybrid expertization.

[0047] Additionally, it should be noted that the number of fragments refers to the number of equal-width fragments (NS) into which the dense feedforward network's intermediate layer is evenly divided; the total number of experts (NE) refers to the total number of experts formed by retaining, copying, or combining them; the number of activated experts (TopK) refers to the number of experts activated for each input token during the forward pass; and a set of candidate discrete configurations is generated by traversing or sampling these discrete parameters.

[0048] This embodiment aims to quantify the fuzzy structural modification target into a specific discrete parameter combination, so that the search process can fully consider the structural differences between the feedforward layer width and hidden dimension of different base models, avoid relying on a single empirical parameter for configuration selection, thereby effectively preventing structural mismatch problems caused by different model families or scales, ensuring that the model corresponding to the selected configuration can adapt to the storage capacity and computing power constraints of the target resource-constrained device, and avoiding the model's inability to run normally or low inference efficiency on the target device due to structural mismatch.

[0049] In one possible implementation, the set of candidate discrete configurations in the discrete configuration space can be generated by performing a full permutation of discrete parameters through a grid search, or it can be generated by sampling based on prior knowledge using heuristic rules, in order to achieve a balance between the size of the configuration space and search efficiency.

[0050] In one specific implementation, for the image description generation scenario on a mobile smartphone, the system acquires a candidate pre-trained dense visual language model with 7 billion parameters. Through the structure calculation module, its key structural parameters are extracted, resulting in a language model hidden dimension DH of 4096, a dense feedforward network intermediate layer dimension Ddense of 11008, a language model layer number L of 32, and a total number of always-activated parameters Pfixed of 2 billion. Subsequently, the system sets the number of slices NS to a range of {2, 4, 8}, the total number of experts NE to a range of {4, 8, 16, 32}, and the number of activated experts TopK to a range of {1, 2, 4}, thus constructing a discrete configuration space. All permutations and combinations of these discrete parameters generate a candidate discrete configuration set containing 36 candidate configurations, used for structure search and filtering based on the target total parameter budget and the target activation parameter budget.

[0051] Furthermore, referring to Figure 2 The second embodiment of the visual language model conversion method of this application provides a flowchart, based on the above. Figure 2 The embodiment shown further refines the step S20, "perform budget-aware structural search on the candidate pre-trained dense visual language model and output a target hybrid expert configuration that satisfies the constraints," including steps A301-A304: Step A301: Perform a legality screening on each configuration in the candidate discrete configuration set, and remove those that do not satisfy the partitioning divisibility relationship or the constraint relationship between the number of activated experts and the total number of experts, to obtain a legal configuration set.

[0052] It should be noted that, in this embodiment, the legality screening refers to the process of verifying whether the configuration parameters can be implemented mathematically and logically; the partition divisibility relationship refers to the constraint that the intermediate layer dimension of the dense feedforward network must be divisible by the number of partitions; the constraint relationship between the number of activated experts and the total number of experts refers to the logical condition that the number of activated experts must be less than or equal to the total number of experts and greater than zero; the legal configuration set refers to the subset of configurations retained after passing the legality screening.

[0053] This embodiment performs a validity screening on the candidate discrete configuration set, eliminating configurations that are physically impossible to implement or logically conflicting, thereby avoiding meaningless calculations on invalid configurations and improving the efficiency and reliability of structure search.

[0054] Step A302: For each configuration in the set of legal configurations, budget estimation is performed to filter the configurations, the activation feedforward dimension and the total feedforward dimension are calculated, and the corresponding total parameter quantity and activation parameter quantity are calculated based on the always active parameter quantity, the language model hidden dimension, the number of language model layers, the activation feedforward dimension and the total feedforward dimension. Configurations that exceed the target total parameter budget or the target activation parameter budget are removed to obtain the budget-satisfied configuration set.

[0055] It should be noted that, in this embodiment, budget estimation screening refers to the process of eliminating configurations that exceed the budget limit based on hard budget constraints; activation feedforward dimension refers to the actual activated intermediate layer dimension of the feedforward network during the forward process for each input token; total feedforward dimension refers to the sum of the intermediate layer dimensions of the feedforward networks of all experts in the hybrid expert structure; total parameter quantity refers to the overall parameter scale of the model; activation parameter quantity refers to the number of parameters actually involved in the calculation during a single forward inference process; budget-satisfied configuration set refers to the subset of configurations that meet the dual hard constraints of the target total parameter budget and the target activation parameter budget.

[0056] This embodiment performs budget estimation screening based on a set of legal configurations. Quantitative calculations ensure that the selected configurations strictly match the computing power and memory limitations of the actual deployed hardware, thus addressing the problem of existing technologies lacking automated decision-making mechanisms oriented towards target budgets and easily leading to model size expansion and budget overruns. Through budget estimation screening, it ensures that the total number of model parameters corresponding to the selected configuration does not exceed the target device's storage capacity to reduce memory usage, and that the number of activated parameters does not exceed the target device's computing power limit to reduce power consumption per inference, thereby ensuring that the model can be deployed and run normally on resource-constrained target devices.

[0057] In one possible implementation, the structural quantity calculation module uses the formula DI = (TopK / NS) × Ddense when calculating the activation feedforward dimension, and the formula Dtotal = (NE / NS) × Ddense when calculating the total feedforward dimension. Then, it combines the always-activated parameter quantity, the language model hidden dimension, and the number of language model layers to accurately estimate the total parameter quantity and the activation parameter quantity.

[0058] Step A303: Perform structural matching screening on each configuration in the budget-satisfied configuration set, calculate the key structural ratio, and remove configurations whose key structural ratios do not fall within a preset reasonable range to obtain a structural matching configuration set.

[0059] It should be noted that, in this embodiment, structural matching screening refers to the process of eliminating configurations that are prone to structural mismatch based on model structural adaptability criteria; the key structural ratio refers to the ratio of the activation feedforward dimension to the language model hidden dimension; the preset reasonable range is a range of proportions that can be pre-determined based on offline statistical results and can ensure a good degree of matching between the candidate structure and the hidden structure of the base model; the structural matching configuration set refers to the subset of configurations whose key structural ratio falls within the preset reasonable range and meets the budget constraints.

[0060] This embodiment performs structure matching screening on the budget-compliant configuration set. The aim is to introduce a key structure ratio that reflects the degree of matching between the activation feedforward structure and the hidden structure of the base model. This avoids the structure mismatch problem caused by relying solely on a fixed activation ratio, ensuring a more consistent basis for structure judgment across different model families and model scales, and guaranteeing consistent stability after modification. Through structure matching screening, it ensures that the selected configuration can maintain the model's output accuracy in specific visual language tasks such as image understanding, document analysis, and multimodal question answering, avoiding severe performance degradation after deployment on the target device due to structure mismatch.

[0061] In one possible implementation, when performing structural matching screening, the preset reasonable range of the key structural proportion can be dynamically adjusted based on the offline statistical results of different base models.

[0062] In another possible implementation, this embodiment calculates the critical structure ratio R = DI / DH for the budget-satisfying configuration set. If the R value of a certain configuration is 1.5, which does not fall within the preset reasonable range [2, 8], it is removed, and configurations with R values ​​between 2 and 8 are retained to form a structure-matching configuration set.

[0063] In one specific embodiment, for the image description generation scenario of a mobile smartphone, a structure search is performed based on a 3B candidate model. It is assumed that after legality screening and budget estimation screening, five configurations satisfy the dual constraints of a target total parameter budget of 2 billion and a target activation parameter budget of 600 million. The system further calculates the critical structure ratio R value for each configuration. Configuration A (NS=4, NE=16, TopK=2) has an R value of 2.5, and configuration B (NS=2, NE=8, TopK=1) has an R value of 3.4, both falling within the preset reasonable range [2, 8]. Configuration C (NS=8, NE=32, TopK=4) has an R value of 1.8, which does not fall within the reasonable range and is therefore eliminated. The system ultimately ranked configurations A and B. Considering the strict requirements of mobile devices for inference latency, configuration B has fewer activated experts (TopK=1), resulting in lower computational cost per inference and smaller budget deviation. Configuration B was ultimately selected as the target hybrid expert configuration. This configuration corresponds to approximately 520 million model activation parameters and can achieve an inference latency of approximately 150ms on a mobile device with 15 TOPS of computing power, meeting the user's interactive experience requirement of obtaining real-time image descriptions after taking a photo.

[0064] Step A304: In the set of structure matching configurations, the configuration with the best score is selected as the target hybrid expert configuration based on a comprehensive ranking of budget deviation and training stability preset indicators.

[0065] It should be noted that, in this embodiment, budget deviation refers to the difference between the estimated number of parameters configured and the target budget; training stability preset index refers to the index used to evaluate the convergence stability of the model during fine-tuning; comprehensive ranking refers to the operation of scoring the configurations according to multiple preset indexes and ranking their superiority; the configuration with the best score is the configuration with the highest ranking in the comprehensive evaluation system; target hybrid expert configuration refers to the combination of discrete parameters including the number of slices, the total number of experts, and the number of activated experts that meets the budget constraints and has the best structural matching degree.

[0066] This embodiment optimizes all feasible configurations, balancing precise budget matching with stable recovery training, thereby consistently obtaining a target structure that balances budget constraints and recovery quality. A smaller budget deviation means that the model corresponding to the selected configuration can fully utilize the target device's storage and computing resources, maximizing model capacity without exceeding device limitations. This results in reduced memory usage and inference power consumption while maintaining better task accuracy.

[0067] In one possible implementation, this embodiment can use a weighted summation method to calculate the total score when performing comprehensive ranking. The smaller the budget deviation, the higher the score; the closer the sparsity representation is to the preset empirical value, the higher the training stability score. Finally, the configuration with the highest total score is selected as the target hybrid expert configuration. The sparsity representation S = total feedforward dimension Dtotal / activation feedforward dimension DI = NE / TopK, which can be used to reflect the relationship between the total capacity of the model and the activation intensity under a given activation dimension.

[0068] This application sequentially filters candidate configurations based on legality, budget estimation, and structure matching, and then combines budget bias and training stability to select the optimal target hybrid expert configuration. This scheme achieves automated structure decision-making for resource-constrained devices, ensuring that the selected configuration effectively avoids task accuracy degradation caused by structure mismatch while meeting device storage capacity and computing power constraints. Simultaneously, it considers accurate budget matching and training recovery stability, enabling the model to achieve efficient visual language task processing on resource-constrained platforms.

[0069] Furthermore, referring to Figure 3 The third embodiment of the visual language model conversion method of this application provides a flowchart, based on the above. Figure 3 The embodiment shown further refines the execution structure conversion step in step S30, which involves "converting the language model feedforward network in the candidate pre-trained dense visual language model into a hybrid expert feedforward network according to the target hybrid expert configuration," and includes steps A401 to A403: Step A401: Based on the number of shards and the total number of experts in the target hybrid expert configuration, the intermediate layer of the dense feedforward network to be converted is divided into equal-width shards according to the number of shards, so that each shard corresponds to a basic expert unit.

[0070] It should be noted that in this embodiment, the number of slices refers to the number of equal-width slices into which the dense feedforward network intermediate layer is evenly divided; the total number of experts refers to the total number of experts formed by retention, duplication, or combination; the dense feedforward network intermediate layer to be transformed refers to the higher-dimensional hidden layer in the language-side feedforward network of the candidate pre-trained dense visual language model; equal-width slices refer to the operation of dividing the weight matrix into multiple sub-matrices of the same size along the feature dimension; and the basic expert unit refers to the smallest expert network unit with independent computing power obtained after equal-width slices.

[0071] This embodiment achieves a mapping from dense to sparse physical structures by transforming a wide, dense feedforward network into reconfigurable, fine-grained basic units. This mapping allows only the activated expert subnetworks to be loaded and computed during a single inference iteration, rather than all dense feedforward parameters, thereby significantly reducing the memory usage and computational power consumption of the target device during inference.

[0072] In one possible implementation, when performing equal-width sharding on the intermediate layer, this embodiment can equally cut the weight matrix and bias terms along the feature dimension to ensure that each basic expert unit has the same computing capacity and input / output dimensions, thereby guaranteeing alignment consistency during subsequent route aggregation.

[0073] Step A402: Based on the total number of experts, perform operations such as retaining, copying, or combining the basic expert units to form an expert group with the total number of experts.

[0074] It should be noted that in this embodiment, the retention operation refers to the method of directly using the original basic expert unit as an independent expert without changing its parameters; the copy operation refers to the method of completely copying one or more copies of the parameters of the original basic expert unit to form a new expert; the combination operation refers to the method of splicing or recombining multiple different basic expert units to form a new expert; and the expert refers to the network module in the hybrid expert structure that independently undertakes part of the forward computing task.

[0075] This embodiment expands the model capacity or constructs a multi-expert structure by reorganizing existing pre-trained weights without re-initializing the model parameters. This fully utilizes the knowledge of the original dense model, reduces losses during the initialization phase, and ensures training stability. Because it retains the effective knowledge representation of the pre-trained weights, the transformed model can maintain basic processing capabilities for tasks such as image understanding and document analysis during the initialization phase. This reduces the computational resources and time costs required for training from scratch, facilitating rapid model deployment on resource-constrained devices.

[0076] In one possible implementation, when the number of shards in the target hybrid expert configuration equals the total number of experts, and experts are directly formed from shards, a sharding path is used, and the basic expert units are directly used as experts. When the number of shards in the target hybrid expert configuration is greater than the total number of experts, and they are formed through replication and expansion, an expansion path is used, and the basic expert units are replicated to expand to a number equal to the total number of experts. When the number of shards in the target hybrid expert configuration is not equal to the total number of experts, and sharding and replication are used simultaneously, a combination path is used, where some basic expert units are retained, and the remaining basic expert units are replicated or combined to form a number equal to the total number of experts.

[0077] Step A403: Output an initial hybrid expert feedforward network consisting of experts whose number is equal to the total number of experts.

[0078] It should be noted that, in this embodiment, the initial hybrid expert feedforward network refers to a feedforward network structure consisting of experts equal to the total number of experts, without the introduction of routers and scale calibration mechanisms.

[0079] In one possible implementation, when outputting the initial hybrid expert feedforward network, this embodiment maintains the original dense structure in the always activated parts such as the visual tower, attention layer, embedding layer, and normalization layer, and only performs the transformation operation on the language model feedforward network, thereby minimizing the instability caused by structural changes. In addition, to improve overall stability, this embodiment may perform the transformation operation only on some feedforward layers and allow specific layers at the beginning or end to be retained as dense structures.

[0080] This application divides the intermediate layers of the dense feedforward network into equal-width slices based on the number of slices configured for the target hybrid expert network, and performs operations such as retention, duplication, or combination of the basic expert units according to the total number of experts to output the initial hybrid expert feedforward network. It realizes the reshaping of the dense to sparse physical structure based on pre-trained weights, so that only a portion of the expert sub-networks are activated in a single inference, thereby significantly reducing memory usage and inference power consumption on target resource-constrained devices. At the same time, it expands the model capacity without re-initializing the parameters, effectively maintaining the initial accuracy of the model on visual language tasks and ensuring training stability.

[0081] In one specific embodiment, for document analysis scenarios on edge computing terminals, the target hybrid expert configuration is NS=4, NE=16, and TopK=4. The system divides the intermediate layer (dimension Ddense=11008) of each layer of the language model feedforward network in the 7B candidate model into equal-width segments according to the number of segments NS=4, with each segment having a width of 2752, resulting in 4 basic expert units. Since the total number of experts NE=16 is greater than the number of segments 4, the system adopts a combination path, copying each of the 4 basic expert units 3 times and merging them with the original 4 basic expert units to form 16 experts. The feedforward intermediate layer dimension of each expert is 2752, and a single inference activates TopK=4 experts, activating an intermediate layer dimension of 4×2752=11008, consistent with the original dense feedforward network intermediate layer dimension. This design allows the model to maintain a feature representation capacity comparable to the original dense model in document analysis scenarios, which is beneficial for accurately identifying details such as table structures, stamped text, and handwritten annotations in documents, while keeping the total number of model parameters within the storage budget of the target device.

[0082] It achieves dense-to-sparse physical structure reshaping based on pre-trained weights, expands model capacity without re-initializing parameters, effectively reduces initial loss and ensures training stability.

[0083] Furthermore, the step of converting the language model feedforward network in the candidate pre-trained dense visual language model into a hybrid expert feedforward network according to the target hybrid expert configuration further includes performing route aggregation steps A501~A503: Step A501: Based on the initial hybrid expert feedforward network and the hidden representation of the output token, the routing score of the corresponding expert is output through the router in the transformed feedforward layer.

[0084] It should be noted that, in this embodiment, the hidden representation of the output token refers to the feature vector input to the feedforward layer; the router refers to the network module used to calculate the matching degree between the input and each expert and determine the activation state of the expert; the routing score refers to the numerical value output by the router that reflects the corresponding expert's processing ability for the current input.

[0085] This embodiment dynamically evaluates the importance of each expert to the current input through a gating mechanism, enabling the model to adaptively select the most relevant experts for computation based on input features. This achieves precise allocation of computing resources and lays the foundation for sparse activation. By activating only a small number of the most relevant experts in forward computation through a routing mechanism, the computational load and power consumption of a single inference are significantly reduced, thereby ensuring that the model can achieve low-power, low-latency inference operation on resource-constrained devices.

[0086] In one possible implementation, the router may employ a linear layer to map the hidden representation of the output token to logits of the total number of experts, thereby serving as the routing score for the corresponding expert.

[0087] Step A502: Select the Top K experts as active experts based on the routing scores, and calculate the aggregate weight.

[0088] It should be noted that in this embodiment, TopK experts refer to the top K experts in terms of routing score; activated experts refer to the experts selected to participate in the calculation in the current forward computation; and aggregate weight refers to the weight coefficient used to merge the output results of each activated expert.

[0089] This embodiment performs a routing operation with a fixed number of activated experts. While maintaining a large total model capacity, it controls the computational load per forward pass, thereby meeting the constraints of the target activation parameter budget. By fixing the number of activated experts, the computational load of a single inference pass is strictly controlled within the activation parameter budget, ensuring that the device's inference latency and power consumption do not exceed the hardware capability limit, thus guaranteeing the real-time performance of visual language task processing.

[0090] In one possible implementation, the system can use the softmax function to normalize the routing scores of the selected TopK experts, and use the normalized result as the aggregation weight to ensure that the sum of the weights of each activated expert is 1, thus maintaining the stability of the output distribution.

[0091] Step A503: Perform a weighted aggregation operation on the output of the activated expert according to the aggregation weight to obtain the hybrid expert feedforward output of the current layer.

[0092] It should be noted that in this embodiment, the output of the activated expert refers to the result of the activated expert network performing a nonlinear transformation on the hidden representation of the input token; the output of the hybrid expert feedforward refers to the final feature representation obtained after the current feedforward layer undergoes a weighted aggregation operation.

[0093] This embodiment effectively integrates the computation results of multiple sparse activation experts, combines the local features of each expert to form the globally optimal feature representation, thereby completing the forward computation transformation of the language model feedforward network from dense to hybrid experts, and realizing feature extraction under the sparse activation mechanism.

[0094] In one specific embodiment, for the image description generation scenario on a mobile smartphone, the converted hybrid expert feedforward network is configured with NE=8 and TopK=1. When a user takes a photo containing a city street scene with their mobile phone, the visual tower first encodes the image into a visual feature sequence, which is then mapped to the language space by the alignment module and fed into the language model along with the user-input description command text token. In each layer of the hybrid expert feedforward network, the router receives the hidden representation of the current lexical token, outputs the routing scores of 8 experts through linear mapping, selects the expert with the highest score as the activated expert, and the other 7 experts do not participate in the calculation. This sparse activation mechanism makes the computational cost of each layer of the feedforward network only about 1 / 8 of the original dense model when processing the image description request (single expert dimension 4096 / original dimension 8192 = 1 / 2, and only 1 / 8 of the experts are activated). Fast inference can be achieved with 15 TOPS computing power on a mobile device, and the text description result of the image can be obtained in about 150ms after the user takes the photo.

[0095] Furthermore, referring to Figure 4 The fourth embodiment of the visual language model conversion method of this application provides a flowchart, based on the above. Figure 4 The embodiment shown further refines the step S30, "perform scale calibration on the aggregated output of the transformed hybrid expert feedforward layer to generate a calibrated hybrid expert visual language model," including steps A601 to A604: Step A601: Construct the mean matching coefficient, variance matching coefficient, or second moment matching coefficient respectively, and then generate the scaling factor based on the preset combination.

[0096] It should be noted that, in this embodiment, the mean matching coefficient is a value used to reflect the degree of consistency between the expected output of the hybrid expert and the expected output of the original dense feedforward; the variance matching coefficient is a value used to reflect the degree of consistency between the fluctuation of the hybrid expert output and the fluctuation of the original dense feedforward output; the second moment matching coefficient is a value used to reflect the degree of consistency between the second moment of the hybrid expert output and the second moment of the original dense feedforward output; the preset combination is a preset rule for integrating multiple matching coefficients to generate the final parameters; and the scaling factor refers to a dynamic parameter used to scale the aggregated output to correct the distribution offset.

[0097] This embodiment employs moment matching technology to simultaneously ensure unbiased expectation and suppress output fluctuations, maximizing the statistical characteristics of the hybrid expert network's forward output to closely approximate the original dense state. This overcomes the output scale and distribution shift issues caused by differences in output aggregation methods after fragmentation and replication in dense feedforward networks, reducing initial loss and accelerating subsequent fine-tuning convergence. This scale calibration technique allows the model to maintain near-original dense model accuracy in specific tasks such as image understanding and document analysis without extensive additional training after structural transformation, significantly reducing the computational and time costs of fine-tuning required for deployment on resource-constrained devices.

[0098] In one possible implementation, when generating a scaling factor based on a preset combination, this embodiment can combine the mean matching coefficient and the variance matching coefficient according to the geometric mean rule to generate a final scaling factor that takes into account both expectation and volatility constraints.

[0099] Step A602: Introduce a dynamically determined scaling factor at the aggregated output of the hybrid expert feedforward layer.

[0100] It should be noted that, in this embodiment, the aggregated output of the hybrid expert feedforward layer refers to the output position of the result after weighted summation of the activated expert outputs in the hybrid expert structure; the dynamically determined scaling factor refers to the scaling parameter that is calculated and generated in real time based on the statistical characteristics of the current output of the model, rather than being fixedly assigned.

[0101] This embodiment applies a distribution correction mechanism directly to the key output nodes of the network structure, enabling the aggregated output distribution to more closely approximate the original dense feedforward output on a numerical scale. This eliminates the distribution shift caused by sparse selection replacing the dense sum output during the initialization phase, significantly improving the stability of training recovery. By eliminating the distribution shift during initialization, the model can quickly reach a usable state on the target device after transformation, reducing the large-scale computational overhead required for accuracy recovery due to distribution shift. This facilitates efficient model deployment on edge computing terminals with limited computing power.

[0102] Step A603: Perform a scaling operation on the hybrid expert feedforward output based on the scaling factor to obtain the calibrated hybrid expert feedforward output.

[0103] It should be noted that, in this embodiment, the scale calibration operation refers to the process of correcting the output numerical distribution by introducing a scaling factor to eliminate the offset; the calibrated hybrid expert feedforward output refers to the hybrid expert layer result whose statistical mean and variance are aligned with the original dense feedforward output.

[0104] This embodiment transforms the forward calculation formula from a simple weighted summation to a calibrated summation with a scaling factor, fundamentally achieving unbiased expectation and suppression of output fluctuations. This allows the statistical characteristics of the forward output of the hybrid expert network to approach the original dense state as closely as possible, thereby effectively reducing initial loss, accelerating the fine-tuning convergence of sparse networks, and ensuring the model's accuracy recovery efficiency in visual language tasks.

[0105] In one possible implementation, the formula for performing the scale calibration operation in this embodiment is y_moe(x) = C× Σ p_i(x)E_i(x), where y_moe(x) is the calibrated hybrid expert feedforward output, C is the scaling factor, p_i(x) is the aggregation weight, and E_i(x) is the output of the activated expert.

[0106] Step A604: Replace the original dense feedforward output with the calibrated hybrid expert feedforward output to generate the calibrated hybrid expert visual language model.

[0107] It should be noted that in this embodiment, the original dense feedforward output refers to the feature representation directly output by the feedforward layer in the dense visual language model before conversion; the calibrated hybrid expert visual language model refers to the intermediate model that has completed the physical structure conversion and whose output distribution has been calibrated and aligned with the original model.

[0108] This embodiment fully embeds the distributed-corrected sparse feedforward structure into the original visual language model, so that while maintaining the advantages of hybrid expert sparse activation, the model eliminates the output scale and distribution offset caused by the initialization stage. Finally, an intermediate model with a valid structure, stable output, and can be directly entered into subsequent phased recovery training is obtained. This intermediate model has the basic ability to perform inference on target resource-constrained devices, and its output accuracy in tasks such as image understanding and document analysis has been initially guaranteed through scale calibration.

[0109] In one specific embodiment, targeting the image captioning generation scenario on mobile smartphones, after the model completes the structural transformation, since only TopK=1 experts are activated in a single inference, the output of the activated experts is directly used as the output of the current feedforward layer. Its numerical distribution differs significantly from the output distribution of the original dense feedforward network, where all intermediate neurons participate in the calculation. The system constructs a scaling factor C through a scale calibration operation. First, based on calibration sample data (e.g., sampling 1000 image-text pairs from the image captioning training set), the mean matching coefficient and the second-order moment matching coefficient are calculated respectively. Then, a scaling factor C=0.87 is generated according to the geometric mean rule. The system introduces this scaling factor into the aggregated output of the hybrid expert feedforward layer to perform scale calibration on the output. After calibration, the model's initial BLEU score on the image captioning generation task increases from 12.3 before calibration to 28.7 (compared to 31.2 in the original dense model). This indicates that scale calibration effectively eliminates the impact of distribution offset on task accuracy, enabling the model to provide usable image captioning quality without large-scale additional training when deployed on mobile devices.

[0110] Further, the step of performing phased training on the calibrated hybrid expert visual language model and outputting the final visual language hybrid expert model includes performing the first phase of recovery training steps A701~A702: Step A701: Based on the calibrated hybrid expert visual language model, freeze the visual tower parameters and visual side alignment module parameters, and train the language model and language side alignment related modules.

[0111] It should be noted that, in this embodiment, the visual tower parameters refer to the weights of the visual encoder network responsible for extracting image features in the visual language model; the visual side alignment module parameters refer to the weights of the connection module used to map visual features to the feature space of the language model; the language model refers to the network module that processes text input and generates output; and the language side alignment related modules refer to the network modules related to language feature docking and projection.

[0112] This embodiment employs a gradient isolation mechanism to freeze the visual-side state, ensuring the purity of the feature input. It prioritizes restoring stable representation capabilities to the modified language model, preventing initial oscillations caused by sparse modification from propagating through the alignment module and contaminating the originally intact visual tower features. This effectively avoids catastrophic cascading errors in multimodal recovery scenarios. This gradient isolation strategy avoids repeated training and parameter rollback caused by cascading errors, thereby reducing the total computational cost and number of iterations required for recovery training. This facilitates accurate model recovery on resource-constrained devices with lower power consumption and shorter time.

[0113] Step A702: Monitor the training process until the preset stability condition is reached, and output the first-stage intermediate model of language-side sparse structure recovery stability.

[0114] It should be noted that in this embodiment, the preset stability condition refers to the evaluation index threshold or trend judgment rule used to determine whether the language-side sparse structure has converged to the ideal state; the first-stage intermediate model refers to the transitional model in which the language-side sparse structure has recovered to stability but the visual side remains frozen.

[0115] In one possible implementation, the preset stability condition can be set as the training loss decreasing by less than a set threshold for multiple consecutive iterations, or the language modeling perplexity on the validation set reaching a specific target value. When the system detects the above convergence trend, it determines that the language-side sparse structure has recovered and stabilizes and outputs the corresponding model.

[0116] Furthermore, the step of performing phased training on the calibrated hybrid expert visual language model and outputting the final visual language hybrid expert model also includes performing a second-stage recovery training step A801~A803: Step A801: Based on the intermediate model of the first stage, unfreeze the model parameters and perform end-to-end joint training.

[0117] It should be noted that, in this embodiment, unfreezing the model parameters refers to opening up the weights of the previously frozen visual tower and visual-side alignment module so that they can participate in gradient backpropagation and update; end-to-end joint training refers to the overall optimization process of simultaneously updating the parameters of all modules on the visual and language sides.

[0118] This embodiment breaks the feature isolation state by fully optimizing all parameters after the sparse structure on the language side has been restored to a stable representation. This allows visual and linguistic information to be fully aligned and interact in the joint space, thereby further improving the overall multimodal performance ceiling and ensuring that the output accuracy of the final model in multimodal tasks such as image description generation, visual question answering, and document information extraction can meet the actual business needs.

[0119] In one possible implementation, the system can adopt a progressive unfreezing strategy when unfreezing model parameters, prioritizing the unfreezing of high-level network parameters on the visual side that are close to the alignment module, and then gradually unfreezing the bottom feature extraction layer, so as to reduce the training oscillation caused by releasing a large number of parameters at the same time in a smooth transition.

[0120] Step A802: In joint training, the learning rate or update pace of the visual parameters is lower than that of the language parameters, and a load balancing constraint is introduced in the joint training.

[0121] It should be noted that in this embodiment, visual-side parameters refer to the weights in the visual encoder and visual-side alignment module; learning rate refers to the hyperparameter that controls the step size of each parameter update; smooth update rhythm refers to the training strategy of slowing down the parameter update speed by accumulating gradients or synchronizing parameters at a lower frequency; load balancing constraint refers to the auxiliary penalty term added to the loss function to encourage experts to be selected evenly to prevent some experts from being overloaded.

[0122] This embodiment reduces the risk of oscillations caused by disturbances to visual features by using a conservative differential learning rate or update rhythm, avoiding damage to the original excellent feature representation of the visual tower. Simultaneously, load balancing constraints force routers to distribute computational load more evenly, reducing expert overload or routing collapse and ensuring the effective utilization of the hybrid expert structure. Load balancing constraints ensure even utilization of all experts, preventing some experts from becoming inference bottlenecks due to overload, thereby guaranteeing the stability of inference latency and power consumption uniformity on resource-constrained devices and maintaining the real-time performance of task processing.

[0123] Step A803: When the joint training reaches the preset termination condition, the final visual-language hybrid expert model is output.

[0124] It should be noted that, in this embodiment, the preset termination condition refers to the evaluation criteria used to determine whether the joint training has reached convergence and whether the model performance meets the requirements; the final visual-language hybrid expert model refers to a usable model that has completed all structural transformations and phased training processes, has stable multimodal capabilities, and meets the target budget constraints.

[0125] This embodiment achieves full interaction between the visual and language sides and, once the expert routing reaches a balanced and stable state, promptly terminates training to prevent overfitting. This results in the output of a target model that, under given budget constraints, balances structural rationality, training stability, and multimodal capability retention. This model can be efficiently deployed on resource-constrained devices, providing real-time inference services that meet business accuracy requirements in specific visual-language tasks such as image description generation, visual question answering, and document analysis, with low memory usage and inference power consumption.

[0126] In one possible implementation, the preset termination condition can be set as follows: the comprehensive evaluation index on the multimodal validation set no longer increases for several consecutive rounds, or the training loss decreases to a set threshold and the expert load distribution remains stable. When the above conditions are detected, the current weights are saved and the final model is output.

[0127] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the visual language model conversion method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0128] This application also provides a visual language model conversion device, please refer to... Figure 5 The visual language model conversion device includes: The data acquisition module 10 is used to acquire the target total parameter budget and the target activation parameter budget, as well as one or more candidate pre-trained dense visual language models; The target configuration module 20 is used to perform budget-aware structure search on the candidate pre-trained dense visual language model with the target total parameter budget and the target activation parameter budget as constraints, and output the target hybrid expert configuration that satisfies the constraints; The conversion calibration module 30 is used to convert the language model feedforward network in the candidate pre-trained dense visual language model into a hybrid expert feedforward network according to the target hybrid expert configuration, and to perform scale calibration on the aggregated output of the converted hybrid expert feedforward layer to generate a calibrated hybrid expert visual language model. The model recovery module 40 is used to perform phased training on the calibrated hybrid expert visual language model and output the final visual language hybrid expert model.

[0129] The visual language model conversion device provided in this application, employing the visual language model conversion method in the above embodiments, can solve the technical problem of how to achieve efficient deployment of visual language models on resource-constrained platforms under given hardware constraints. Compared with the prior art, the beneficial effects of the visual language model conversion device provided in this application are the same as those of the visual language model conversion method provided in the above embodiments, and other technical features in the visual language model conversion device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0130] This application provides a visual language model conversion device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the visual language model conversion method in the above embodiment 1.

[0131] The following is for reference. Figure 6This document illustrates a structural schematic diagram of a visual language model conversion device suitable for implementing embodiments of this application. The visual language model conversion device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The visual language model conversion device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0132] like Figure 6 As shown, the visual language model conversion device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the visual language model conversion device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the visual language model conversion device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show visual language model conversion devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0133] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0134] The visual language model conversion device provided in this application, employing the visual language model conversion method in the above embodiments, can solve the technical problem of how to achieve efficient deployment of visual language models on resource-constrained platforms under given hardware constraints. Compared with the prior art, the beneficial effects of the visual language model conversion device provided in this application are the same as those of the visual language model conversion method provided in the above embodiments, and other technical features in this visual language model conversion device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0135] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0136] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A visual language model conversion method, characterized in that, The visual language model conversion method includes: Obtain the target total parameter budget and target activation parameter budget, as well as one or more candidate pre-trained dense visual language models; Using the target total parameter budget and target activation parameter budget as constraints, a budget-aware structure search is performed on the candidate pre-trained dense visual language model to output a target hybrid expert configuration that satisfies the constraints; According to the target hybrid expert configuration, the language model feedforward network in the candidate pre-trained dense visual language model is converted into a hybrid expert feedforward network, and scale calibration is performed on the aggregated output of the converted hybrid expert feedforward layer to generate a calibrated hybrid expert visual language model. The calibrated hybrid expert visual language model is trained in stages, and the final visual language hybrid expert model is output.

2. The visual language model conversion method as described in claim 1, characterized in that, The step of performing budget-aware structure search on the candidate pre-trained dense visual language model and outputting a target hybrid expert configuration that satisfies the constraints includes: Extract key structural parameters from the candidate pre-trained dense visual language model. The key structural parameters include the hidden dimension of the language model, the dimension of the intermediate layer of the dense feedforward network, the number of layers of the language model, and the number of always-activated parameters. Construct a discrete configuration space consisting of the number of shards, the total number of experts, and the number of activated experts, and generate a set of candidate discrete configurations.

3. The visual language model conversion method as described in claim 2, characterized in that, The step of performing budget-aware structural search on the candidate pre-trained dense visual language model and outputting a target hybrid expert configuration that satisfies the constraints further includes: For each configuration in the candidate discrete configuration set, a legality screening is performed, and configurations that do not satisfy the partitioning divisibility relationship or the constraint relationship between the number of activated experts and the total number of experts are eliminated, thus obtaining a legal configuration set; For each configuration in the set of legal configurations, budget estimation is performed to filter the configurations, the activation feedforward dimension and the total feedforward dimension are calculated, and the corresponding total parameter quantity and activation parameter quantity are calculated based on the always activated parameter quantity, language model hidden dimension, language model layer number, activation feedforward dimension and total feedforward dimension. Configurations that exceed the target total parameter budget or target activation parameter budget are removed to obtain the budget-satisfied configuration set. For each configuration in the budget-satisfying configuration set, perform structural matching screening, calculate the key structural ratio, and remove configurations whose key structural ratios do not fall within a preset reasonable range to obtain a structural matching configuration set. In the set of structure matching configurations, the configuration with the best score is selected as the target hybrid expert configuration based on a comprehensive ranking of budget deviation and training stability preset indicators.

4. The visual language model conversion method as described in claim 3, characterized in that, The step of converting the language model feedforward network in the candidate pre-trained dense visual language model into a hybrid expert feedforward network according to the target hybrid expert configuration includes performing a structural transformation step: Based on the number of shards and the total number of experts in the target hybrid expert configuration, the intermediate layer of the dense feedforward network to be transformed is divided into equal-width shards according to the number of shards, so that each shard corresponds to a basic expert unit. Based on the total number of experts, the basic expert units are retained, copied, or combined to form an expert group with the total number of experts. The output consists of an initial hybrid expert feedforward network composed of experts equal to the total number of experts.

5. The visual language model conversion method as described in claim 4, characterized in that, The step of converting the language model feedforward network in the candidate pre-trained dense visual language model into a hybrid expert feedforward network according to the target hybrid expert configuration further includes performing a route aggregation step: Based on the initial hybrid expert feedforward network and the hidden representation of the output token, the routing score of the corresponding expert is output through the router in the transformed feedforward layer; Based on the routing scores, the Top K experts are selected as active experts, and the aggregation weights are calculated. The output of the activated expert is weighted and aggregated according to the aggregation weight to obtain the hybrid expert feedforward output of the current layer.

6. The visual language model conversion method as described in claim 5, characterized in that, The step of performing scale calibration on the aggregated output of the transformed hybrid expert feedforward layer to generate a calibrated hybrid expert visual language model includes: Construct mean matching coefficients, variance matching coefficients, or second moment matching coefficients respectively, and then generate scaling factors based on preset combinations; A dynamically determined scaling factor is introduced at the aggregation output of the hybrid expert feedforward layer; A scaling operation is performed on the hybrid expert feedforward output based on the scaling factor to obtain the calibrated hybrid expert feedforward output. The original dense feedforward output is replaced with the calibrated hybrid expert feedforward output to generate the calibrated hybrid expert visual language model.

7. The visual language model conversion method as described in claim 6, characterized in that, The step of performing phased training on the calibrated hybrid expert visual language model and outputting the final visual language hybrid expert model includes performing a first-stage recovery training step: Based on the calibrated hybrid expert visual language model, the visual tower parameters and visual side alignment module parameters are frozen, and the language model and language side alignment related modules are trained. The training process is monitored until the preset stability conditions are reached, and the first-stage intermediate model with stable sparse structure on the language side is output.

8. The visual language model conversion method as described in claim 7, characterized in that, The step of performing phased training on the calibrated hybrid expert visual language model and outputting the final visual language hybrid expert model also includes performing a second-stage recovery training step: Based on the intermediate model of the first stage, the model parameters are unfrozen and end-to-end joint training is performed. In joint training, the learning rate or update pace of the visual parameters is lower than that of the language parameters, and a load balancing constraint is introduced in the joint training. When the joint training reaches the preset termination condition, the final visual-language hybrid expert model is output.

9. A visual language model conversion device, characterized in that, The visual language model conversion device includes: The data acquisition module is used to acquire the target total parameter budget and the target activation parameter budget, as well as one or more candidate pre-trained dense visual language models; The target configuration module is used to perform budget-aware structure search on the candidate pre-trained dense visual language model with the target total parameter budget and the target activation parameter budget as constraints, and output the target hybrid expert configuration that satisfies the constraints; The conversion calibration module is used to convert the language model feedforward network in the candidate pre-trained dense visual language model into a hybrid expert feedforward network according to the target hybrid expert configuration, and to perform scale calibration on the aggregated output of the converted hybrid expert feedforward layer to generate a calibrated hybrid expert visual language model. The model recovery module is used to perform phased training on the calibrated hybrid expert visual language model and output the final visual language hybrid expert model.

10. A visual language model conversion device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the visual language model conversion method as described in any one of claims 1 to 8.

Images