Multi-modal large model deployment method for neural network processor NPU

Abstract

The application discloses a kind of multi-modal large model deployment methods for neural network processor NPU, comprising: based on the visual branch of the original multi-modal large model, text branch and cross-modal fusion branch of the NPU hardware constraint parameter obtained are compressed respectively, and the simplified multi-modal large model is obtained.Processing the original multi-modal large model as teacher model, according to NPU hardware constraint parameter, the lightweight training of simplified multi-modal large model is carried out, and the lightweight multi-modal large model suitable for NPU hardware is obtained, and lightweight model parameter and compression configuration information, and based on this, the compilation optimization processing of lightweight multi-modal large model is carried out, and the multi-flow parallel instruction sequence and deployment package suitable for NPU hardware are obtained, and then the lightweight multi-modal large model is deployed to NPU hardware.This technical scheme can effectively adapt to NPU hardware resource constraint, guarantee the stability of inference, while taking into account cross-modal semantic inference accuracy and real-time inference performance.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method for deploying multimodal large models for neural network processors (NPUs). Background Technology

[0002] As artificial intelligence technology rapidly iterates towards multimodal fusion and edge intelligence scenarios, large-scale multimodal models with image-text collaborative understanding capabilities have become a core technological support for embedded scenarios such as edge perception, intelligent interaction, and industrial inspection. Compared to traditional single-modal models, large-scale multimodal models, with their massive parameters and cross-modal fusion architecture, possess stronger environmental perception and semantic understanding capabilities. However, they also suffer from complex computational logic, a large number of parameters, and high inference overhead. Embedded intelligent hardware generally suffers from limited computing power, scarce memory resources, and strict power consumption constraints, making it difficult to directly support the complete inference operations of original large-scale multimodal models. Therefore, how to achieve lightweight adaptation, hardware compatibility, and efficient inference deployment of large-scale multimodal models has become a core research hotspot and an essential industrial requirement in the current field of edge embedded intelligence.

[0003] Currently, the industry has developed a variety of optimization solutions for embedded deployment of multimodal large models. The mainstream solutions include using traditional model compression techniques such as pruning, quantization, and knowledge distillation to simplify and optimize the model modal branches, using general conversion tools to migrate and adapt the training framework model to the embedded hardware adaptation format, and relying on dedicated hardware toolchains to optimize the deployment of single-modal models. There are also local optimization methods for single links in the deployment chain, as well as adaptation methods that reduce computational overhead by simplifying the model structure.

[0004] Existing adaptation and optimization solutions are all local optimization methods and have not formed a full-link, systematic deployment and adaptation system. They have significant technical limitations, cannot take into account model accuracy, compatibility and inference performance, and are difficult to meet the core requirements of efficient embedded deployment of multimodal large models. Among these issues, traditional single-modal compression methods ignore the semantic association characteristics of multimodal systems, easily leading to cross-modal feature misalignment and a significant decrease in understanding accuracy. Furthermore, they are not optimized in conjunction with hardware architecture characteristics, resulting in low model inference efficiency. General model conversion schemes do not deeply adapt to multimodal special operators and computation graph structures, easily causing problems such as operator compatibility anomalies and computation graph breaks, preventing the full utilization of hardware computing power. Dedicated hardware adaptation schemes only adapt to single-modal inference scenarios and cannot be compatible with multimodal fusion computation logic, resulting in inefficient memory access and low instruction matching. Single-stage optimization schemes lack a full-process collaborative design, with each optimization stage being isolated from the others, leading to prominent fragmentation issues in deployment and adaptation, long cycles, and high barriers to entry. Model structure simplification schemes sacrifice cross-modal understanding capabilities, resulting in severe accuracy loss, and cannot solve the underlying adaptation barriers between the model and the hardware toolchain, making it difficult to achieve large-scale, high-quality deployment of multimodal large models on embedded hardware. Summary of the Invention

[0005] This invention provides a method for deploying large multimodal models to a neural network processor (NPU) to address the problems of model size exceeding hardware storage and computing power limits, difficulty in adapting the computation graph of the native training model to dedicated compilation tools, and low hardware execution efficiency of multimodal fusion operations when deploying large multimodal models to the NPU.

[0006] According to one aspect of the present invention, a method for deploying multimodal large models for a neural network processor (NPU) is provided, comprising: Obtain the original multimodal large model and NPU hardware constraint parameters. Based on the NPU hardware constraint parameters, compress the visual branch, text branch and cross-modal fusion branch of the original multimodal large model to obtain a simplified multimodal large model. Using the original multimodal large model as the teacher model, a cross-modal knowledge distillation method is adopted to perform lightweight training on the simplified multimodal large model according to the NPU hardware constraint parameters, so as to obtain a lightweight multimodal large model adapted to the NPU hardware, and obtain the matching lightweight model parameters and compression configuration information. Based on the lightweight model parameters and compression configuration information, the lightweight multimodal large model is compiled and optimized to obtain a multi-stream parallel instruction sequence and deployment package adapted to NPU hardware. Based on the multi-stream parallel instruction sequence and deployment package adapted to NPU hardware, the lightweight multimodal large model is deployed to the NPU hardware, completing the environment configuration for loading and running the lightweight multimodal large model and the initialization of multi-stream computing channels.

[0007] According to another aspect of the present invention, a multimodal large model deployment apparatus for a neural network processor (NPU) is provided, comprising: The model compression module is used to obtain the original multimodal large model and NPU hardware constraint parameters. Based on the NPU hardware constraint parameters, the visual branch, text branch and cross-modal fusion branch of the original multimodal large model are compressed to obtain a simplified multimodal large model. The lightweight model training module is used to take the original multimodal large model as the teacher model, and use the cross-modal knowledge distillation method to perform lightweight training on the simplified multimodal large model according to the NPU hardware constraint parameters to obtain a lightweight multimodal large model adapted to the NPU hardware, and obtain the matching lightweight model parameters and compression configuration information. The model compilation optimization module is used to perform compilation optimization on the lightweight multimodal large model based on the lightweight model parameters and compression configuration information, so as to obtain a multi-stream parallel instruction sequence and deployment package adapted to NPU hardware. The model deployment module is used to deploy lightweight multimodal large models to NPU hardware according to the multi-stream parallel instruction sequence and deployment package adapted to NPU hardware, and to complete the environment configuration for loading and running lightweight multimodal large models and the initialization of multi-stream computing channels.

[0008] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to execute the multimodal large model deployment method for neural network processors (NPUs) according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the multimodal large model deployment method for a neural network processor (NPU) according to any embodiment of the present invention.

[0010] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the method as described in any embodiment of the present invention.

[0011] The technical solution of this invention obtains the original multimodal large model and NPU hardware constraint parameters, and performs targeted compression processing on the visual branch, text branch, and cross-modal fusion branch of the original multimodal large model according to the NPU hardware constraint parameters to obtain a simplified multimodal large model with a streamlined structure. Then, using the original multimodal large model as the teacher model, a cross-modal knowledge distillation method is employed, combined with the NPU hardware constraint parameters, to perform lightweight training on the simplified multimodal large model, further obtaining a lightweight multimodal large model adapted to the NPU hardware, and simultaneously acquiring the matching lightweight model parameters and compression configuration information. Subsequently, based on the obtained lightweight model parameters and compression configuration information, the lightweight multimodal large model undergoes overall compilation optimization processing to generate a multi-stream parallel instruction sequence and corresponding deployment package adapted to the NPU hardware's operating characteristics. Finally, relying on the multi-stream parallel instruction sequence and deployment package adapted to NPU hardware, the lightweight multimodal large model is deployed to the NPU hardware. This completes the model loading, runtime environment configuration, and multi-stream computation channel initialization of the lightweight multimodal large model, achieving end-to-end lightweight adaptation and deployment of the multimodal large model for NPU hardware. This technical solution solves the problems of insufficient memory and computing power, difficulty in adapting framework operators, inefficient multimodal inference, and impaired semantic alignment when deploying multimodal large models on NPU hardware through hardware-aware cross-modal lightweighting and compilation-based collaborative optimization. It effectively adapts to NPU hardware resource constraints, ensuring inference stability while balancing cross-modal semantic inference accuracy and real-time inference performance.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a multimodal large model deployment method for a neural network processor (NPU) according to Embodiment 1 of the present invention; Figure 2 This is a flowchart of a multimodal large model deployment method for a neural network processor (NPU) according to Embodiment 2 of the present invention; Figure 3This is a schematic diagram of a multimodal large model deployment device for a neural network processor (NPU) according to Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device that implements the multimodal large model deployment method for neural network processors (NPUs) according to embodiments of the present invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] Example 1 Figure 1 This is a flowchart of a multimodal large model deployment method for a neural network processor (NPU) provided in Embodiment 1 of the present invention. This embodiment is applicable to the deployment of multimodal large models on edge embedded hardware. While ensuring cross-modal semantic alignment accuracy, it addresses issues such as model incompatibility with embedded hardware, poor inference performance, and fragmented deployment links. This method can be executed by a multimodal large model deployment device for a neural network processor (NPU). This device can be implemented in hardware and / or software and is generally configured in an electronic device that executes the multimodal large model deployment method for a neural network processor (NPU). Figure 1 As shown, the method includes: S110. Obtain the original multimodal large model and NPU hardware constraint parameters. Based on the NPU hardware constraint parameters, compress the visual branch, text branch and cross-modal fusion branch of the original multimodal large model respectively to obtain a simplified multimodal large model.

[0018] The original multimodal large model refers to a natively pre-trained multimodal basic model with complete visual feature encoding, text semantic parsing, and cross-modal feature fusion capabilities. This model has a complete parameter structure and has not undergone lightweight pruning or accuracy optimization, serving as the foundational model for model compression and deployment adaptation optimization. NPU hardware constraint parameters refer to the inherent hardware resources and computational constraints of the neural network processor, including hardware computing unit architecture specifications, multi-level memory capacity, bandwidth characteristics, and operator compatibility adaptation rules—underlying hardware boundary conditions that limit model inference performance. The visual branch refers to an independent visual feature processing branch within the multimodal large model, used for feature extraction, encoding, and dimensional representation of image materials, forming the core module for multimodal visual information perception. The text branch refers to an independent text semantic processing branch within the multimodal large model, used for semantic extraction, feature encoding, and semantic representation of natural language text, supporting text understanding and semantic matching functions in multimodal tasks. A cross-modal fusion branch can refer to the core fusion computation module of a large multimodal model. It can be used to complete the dimensional alignment, feature interaction, and unified representation fusion of visual and textual features, ensuring semantic consistency of multimodal features and inference accuracy for cross-modal tasks. A streamlined large multimodal model can refer to a lightweight multimodal model that, after undergoing multi-branch lightweight optimization based on hardware constraints, retains complete multimodal business capabilities and core inference accuracy while optimizing parameter scale, computational load, and memory usage, thus adapting to the characteristics of NPU hardware inference.

[0019] Understandably, after obtaining the original multimodal large model to be deployed and collecting the hardware constraint parameters corresponding to the NPU of the multimodal large model to be deployed, hardware-aware differentiated compression optimization can be implemented on the visual branch, text branch, and cross-modal fusion branch of the original multimodal large model based on the constraints of the NPU hardware's underlying computing power, memory resources, and operation scheduling characteristics. Under the premise of constraining cross-modal semantic offset and ensuring model inference accuracy, redundant parameters and invalid computing links of each operation branch are eliminated, reducing the model inference computing overhead and memory occupation, and a simplified multimodal large model adapted to the NPU hardware deployment scenario can be obtained.

[0020] S120. Using the original multimodal large model as the teacher model, a cross-modal knowledge distillation method is adopted to perform lightweight training on the simplified multimodal large model according to the NPU hardware constraint parameters, so as to obtain a lightweight multimodal large model adapted to the NPU hardware, and obtain the matching lightweight model parameters and compression configuration information.

[0021] The teacher model can refer to a large, original multimodal model with a complete network structure, possessing comprehensive visual encoding, text encoding, and cross-modal fusion representation capabilities. It can output high-precision, robust multimodal features and semantic distributions, providing a knowledge benchmark for supervised learning in the lightweight student model. The lightweight model parameters refer to the core set of model parameters—network weights, bias parameters, feature dimensions, and branch structure parameters—that can be directly used for model inference and deployment after hardware-aware compression and cross-modal distillation training iterations and optimization. The compressed configuration information refers to the policy configuration parameters corresponding to the entire lightweight optimization process, which can include structured configuration data such as pruning granularity, quantization accuracy, feature alignment constraints, knowledge distillation hyperparameters, and hardware adaptation constraints for the visual, text, and cross-modal fusion branches.

[0022] Understandably, using the original multimodal large model as the teacher model, and leveraging cross-modal knowledge distillation combined with NPU hardware constraints, a lightweight iterative training process is performed on the structurally simplified multimodal large model. The teacher model can facilitate transfer learning of multimodal features and semantic knowledge, compensating for the performance loss caused by model compression while adapting to NPU hardware resources and computational characteristics, thus improving the cross-modal representation capability and hardware adaptability of the simplified multimodal large model. This training yields a lightweight multimodal large model adapted for NPU hardware deployment and inference, and can simultaneously output corresponding lightweight model parameters and compression configuration information, providing a model foundation and configuration support for subsequent model compilation, hardware adaptation deployment, and inference execution.

[0023] S130. Based on the lightweight model parameters and compression configuration information, the lightweight multimodal large model is compiled and optimized to obtain a multi-stream parallel instruction sequence and deployment package adapted to NPU hardware.

[0024] Here, a multi-stream parallel instruction sequence can refer to a standardized set of hardware computation instructions that, after compilation optimization and cross-modal computation graph co-adaptation, can be scheduled and executed by the NPU hardware and support parallel scheduling of various computational branches in multiple modalities. A deployment package can refer to an integrated program data package that contains compiled and optimized model files, hardware adaptation configuration information, and scheduling execution parameters, and can be directly loaded, parsed, and executed for inference in the NPU hardware device.

[0025] Understandably, based on the obtained lightweight model parameters and compressed configuration information, a systematic compilation and optimization process can be carried out on the lightweight multimodal large model. This process can be based on the collaborative optimization of the cross-modal computation graph between the compilation optimization layer and the hardware abstraction layer. Combining the computational characteristics of the visual branch, text branch, and cross-modal fusion branch of the multimodal model, and matching the execution rules and resource constraints of the NPU hardware architecture, this process achieves unified optimization of the multimodal computation graph, operator adaptation and reconstruction, and hardware instruction adaptation and conversion. This results in a deep match between the computational logic of the lightweight multimodal large model and the underlying hardware execution unit. Through global collaborative optimization at the compilation level, the model computation topology and data memory layout are reconstructed, generating multi-stream parallel instruction sequences that support concurrent scheduling of multimodal tasks. Furthermore, the model configuration, scheduling strategy, and hardware adaptation parameters are integrated to generate corresponding model deployment packages, providing a standardized deployment carrier for subsequent target NPU hardware devices.

[0026] S140. Based on the multi-stream parallel instruction sequence and deployment package adapted to the NPU hardware, deploy the lightweight multimodal large model to the NPU hardware, and complete the environment configuration for loading and running the lightweight multimodal large model and the initialization of the multi-stream computing channels.

[0027] Understandably, after obtaining the multi-stream parallel instruction sequence and deployment package obtained after compiling and optimizing the lightweight multimodal large model, the deployment package can be parsed and verified. During the process of porting and deploying the lightweight multimodal large model to NPU hardware, the software and hardware adaptation environment required for model operation is configured, a basic operating environment adapted to multimodal parallel computing is built, and the multi-stream computing channels used for parallel inference of vision, text and cross-modal fusion tasks are initialized and configured. Standardized scheduling and data interaction between the hardware layer and the model computing logic are established, thereby completing the deployment of the lightweight multimodal large model on NPU hardware.

[0028] The technical solution of this invention obtains the original multimodal large model and NPU hardware constraint parameters, and performs targeted compression processing on the visual branch, text branch, and cross-modal fusion branch of the original multimodal large model according to the NPU hardware constraint parameters to obtain a simplified multimodal large model with a streamlined structure. Then, using the original multimodal large model as the teacher model, a cross-modal knowledge distillation method is employed, combined with the NPU hardware constraint parameters, to perform lightweight training on the simplified multimodal large model, further obtaining a lightweight multimodal large model adapted to the NPU hardware, and simultaneously acquiring the matching lightweight model parameters and compression configuration information. Subsequently, based on the obtained lightweight model parameters and compression configuration information, the lightweight multimodal large model undergoes overall compilation optimization processing to generate a multi-stream parallel instruction sequence and corresponding deployment package adapted to the NPU hardware's operating characteristics. Finally, relying on the multi-stream parallel instruction sequence and deployment package adapted to NPU hardware, the lightweight multimodal large model is deployed to the NPU hardware. This completes the model loading, runtime environment configuration, and multi-stream computation channel initialization of the lightweight multimodal large model, achieving end-to-end lightweight adaptation and deployment of the multimodal large model for NPU hardware. This technical solution solves the problems of insufficient memory and computing power, difficulty in adapting framework operators, inefficient multimodal inference, and impaired semantic alignment when deploying multimodal large models on NPU hardware through hardware-aware cross-modal lightweighting and compilation-based collaborative optimization. It effectively adapts to NPU hardware resource constraints, ensuring inference stability while balancing cross-modal semantic inference accuracy and real-time inference performance.

[0029] Example 2 Figure 2 This is a flowchart of a multimodal large model deployment method for a neural network processor (NPU) provided in Embodiment 2 of the present invention. This embodiment is a specific embodiment based on the above embodiments, including: specific methods for multimodal large model compression processing, specific methods for simplifying multimodal large model lightweight training, and specific methods for multimodal large model compilation optimization processing. Figure 2 As shown, the method includes: S210. Obtain NPU hardware constraint parameters, including Cube computing unit specification parameters, memory level characteristic parameters, and mixed precision computing characteristic parameters.

[0030] Wherein, the Cube computing unit specification parameters may refer to the hardware size of the Cube matrix acceleration core, which is the matrix acceleration computing unit included in the NPU hardware, and the inherent hardware parameters related to parallel computing granularity, which can be used to constrain the pruning dimension of each model branch and the design of operator computing size. The memory hierarchy characteristic parameters may refer to the hardware parameters related to the capacity, bandwidth and data scheduling rules corresponding to the multi-level storage medium of the NPU hardware, which can be used to limit the upper limit of memory peak and the throughput index of data interaction between memories at all levels during the inference process of the multi-modal large model. The mixed-precision computing characteristic parameters may refer to the hardware capability parameters of the NPU hardware that supports multi-precision numerical parallel computing, which are used to guide the formulation of model quantization and cross-modal feature precision allocation strategies.

[0031] It can be understood that obtaining the NPU hardware constraint parameters of the NPU hardware platform for the multi-modal large model to be deployed, which can cover three types of underlying hardware boundaries including hardware computing, storage and numerical operation, and may include Cube computing unit specification parameters, memory hierarchy characteristic parameters and mixed-precision computing characteristic parameters, can be used as the benchmark parameters for compression processing of the multi-modal large model, and is used to ensure that the model structure, quantization strategy and calculation graph layout of the multi-modal large model after compression processing are adapted to the native operation characteristics of the underlying NPU hardware.

[0032] S220, aligning the pruning granularity with the dimension of the Cube computing unit according to the specification parameters of the Cube computing unit, and pruning the visual branch of the original multi-modal large model according to the pruning granularity, to obtain a visual branch pruning structure aligned with the NPU hardware topology.

[0033] It can be understood that, for performing the NPU hardware-adaptive structured pruning operation on the visual branch of the original multi-modal large model, it is first necessary to carry out channel importance evaluation on the visual branch of the original multi-modal large model, quantify the contribution of each convolution kernel through gradient sensitivity analysis and activation sparsity statistics, complete the feature numerical solution by means of the supporting tools of the deep learning framework, traverse the convolutional layers layer by layer to screen out redundant convolution kernels that meet the constraints of multiple indicators at the same time and record and generate a pruning list. Then determine the pruning granularity adapted to the hardware computing dimension according to the specification parameters of the Cube computing unit, implement structured pruning on the visual branch by combining the pruning granularity with the pruning list, uniformly constrain the clipping scale to adapt to the inherent hardware specification of the Cube unit, eliminate the computing power loss caused by the mismatch between the network structure and the specification of the hardware acceleration unit, and finally output the visual branch pruning structure matched with the NPU hardware topology.

[0034] Optionally, aligning the pruning granularity with the dimension of the Cube computing unit according to the specification parameters of the Cube computing unit, and pruning the visual branch of the original multi-modal large model according to the pruning granularity to obtain a visual branch pruning structure aligned with the NPU hardware topology includes: Acquire gradient information, activation value statistics, downstream task performance data, and NPU Cube computing unit specifications of the visual branch. Through gradient sensitivity analysis and activation sparsity statistics, calculate the gradient sensitivity score and activation sparsity value of each convolutional kernel in the visual branch. Based on gradient sensitivity scores and activation sparsity values, pruned convolutional kernels are selected layer by layer in the visual branch. When the gradient sensitivity score of the target convolutional kernel is lower than a preset first threshold, the activation sparsity is higher than a preset second threshold, and the performance degradation after removal is less than a preset third threshold, the target convolutional kernel is determined to be a pruned convolutional kernel. Record the index and parameters of each pruned convolution kernel to form a convolution kernel pruning list; Based on the Cube computing unit specifications, after aligning the pruning granularity with the Cube computing unit dimensions, structured pruning is performed according to the pruning list to obtain the visual branch structure after preliminary pruning. The visual branch structure after initial pruning is subjected to mixed precision quantization to obtain a visual branch pruning structure aligned with the NPU hardware topology.

[0035] Here, gradient information refers to the set of partial derivatives of the loss function generated during the backpropagation phase of the model with respect to the weights of the visual branch convolutional kernels and intermediate features. This can be used to quantify the influence weight of each convolutional kernel on the model's prediction results. Activation value statistics refer to the statistical distribution of the activation tensors output by each channel of the visual branch during the forward propagation of the multimodal large model network. This can be used to characterize the effective activation degree of each channel. Downstream task performance data refers to quantitative indicators such as accuracy and similarity obtained after the visual branch output features of the multimodal large model participate in cross-modal alignment and image feature extraction tasks. This can be used to evaluate the accuracy loss caused by convolutional kernel pruning. Gradient sensitivity score refers to a quantitative indicator calculated from gradient information. This can be used to characterize the fluctuation range of the model's prediction error caused by small changes in convolutional kernel parameters; a lower score indicates a lower contribution from the convolutional kernel. Activation sparsity value refers to a quantitative proportion indicator converted from activation value statistics. This can represent the overall proportion of near-zero values ​​within a single channel activation tensor; a higher value indicates less effective computation per channel. The first preset threshold can be a threshold used to determine the gradient sensitivity score of low-contribution convolutional kernels; convolutional kernels below this threshold can be included in the pruning candidate range. The second preset threshold can be a threshold used to determine the activation sparsity of invalid activation channels; channels above this threshold can be identified as having a large amount of redundant computation. The third preset threshold can be a threshold used to determine the performance degradation after pruning; convolutional kernels with performance degradation less than this threshold after pruning can be discarded.

[0036] Specifically, gradient information, activation value statistics, downstream task performance data, and NPU Cube computing unit specifications are collected synchronously from the visual branch. Gradient sensitivity analysis is performed based on the gradient information, and activation sparsity statistics are performed based on the activation value statistics. The gradient sensitivity score and activation sparsity value corresponding to all convolutional kernels within the visual branch are calculated. Gradient sensitivity analysis generates gradient information by calculating the partial derivatives of the loss function with respect to the convolutional kernel parameters and intermediate activation features, and then converts it into gradient sensitivity score. Activation sparsity statistics are obtained by counting the number of zeros and minima in the activation tensors of each channel during forward propagation, and then converting them into activation sparsity values. Both types of analysis and statistics can be implemented using open-source tools that support deep learning frameworks, such as TorchPrune (model pruning tool) in the PyTorch (deep learning training framework) ecosystem and TFMO (TensorFlow Model Optimization, model optimization toolset) in the TensorFlow (deep learning development framework) ecosystem. Subsequently, based on the solved gradient sensitivity score and activation sparsity value, the pruning convolution kernel selection and determination can be carried out layer by layer from shallow to deep along the visual branch convolutional layer. The gradient sensitivity calculation, activation sparsity statistics, trial pruning performance evaluation, multi-threshold joint determination, and candidate convolution kernel labeling process are executed layer by layer. Layered processing can avoid the memory overflow problem caused by synchronous evaluation of all layers. To determine whether a target convolutional kernel is a pruned convolutional kernel, three quantization constraints must be met simultaneously: the target convolutional kernel's gradient sensitivity score must be lower than a preset first threshold, its activation sparsity must be higher than a preset second threshold, and the performance degradation of the downstream task corresponding to removing the convolutional kernel must be less than a preset third threshold. Each threshold can be customized according to deployment requirements. For example, the preset first threshold can be set to the top 80% percentile value after sorting all convolutional kernel gradient sensitivity scores from high to low, the preset second threshold can be set to 70%, and the preset third threshold can be set to a 0.5% decrease in cross-modal alignment cosine similarity. The corresponding judgment conditions are that the convolutional kernel's gradient sensitivity score is in the bottom 20% low sensitivity range of all convolutional kernels, the percentage of values ​​with absolute values ​​less than 1e-5 in the channel activation tensor exceeds 70%, and the decrease in cross-modal alignment cosine similarity between visual and text features after trial pruning does not exceed 0.5%, so as to ensure that removing the convolutional kernel will not significantly degrade the performance of downstream tasks such as image feature extraction and cross-modal alignment.After completing the layer-by-layer screening and judgment, the index, corresponding gradient sensitivity score, activation sparsity value, and downstream task performance impact data of all pruned convolutional kernels are recorded one by one. This data is then integrated to generate a complete convolutional kernel pruning list. The pruning granularity is set according to the pre-obtained Cube computing unit specifications, strictly aligning the pruning granularity with the inherent computational dimensions of the Cube computing unit. Combined with the convolutional kernel pruning list, structured pruning operations are performed on the original visual branches to obtain the preliminary pruned visual branch structure. Aligning with the Cube computing unit dimensions ensures that subsequent matrix operations fully match the hardware acceleration unit specifications, avoiding idle computing power. Finally, hybrid precision quantization processing is performed on the preliminary pruned visual branch structure. For example, a W4A8 (INT4 weights + INT8 activations) quantization scheme can be used, combined with compression of visual features to reduce the perceptual accuracy loss caused by quantization. Ultimately, a visual branch pruning structure completely aligned with the NPU hardware topology is generated.

[0037] S230. Based on the memory level characteristic parameters and NPU hardware memory constraints, redundant attention heads and feedforward network intermediate layers in the text branches of the original multimodal large model are pruned to obtain a memory-adapted text branch pruning structure.

[0038] Among them, memory-level characteristic parameters refer to quantitative parameters characterizing the capacity, bandwidth, and data scheduling rules of each level of storage media in the NPU hardware. These parameters can be used to limit the peak memory limit for inference in multimodal large models and the throughput indicators of interactions between storage levels. NPU hardware memory constraints refer to hard resource boundary conditions such as the maximum available capacity of the NPU hardware's built-in high-speed storage and off-chip storage, as well as the amount of data read and written in a single operation. Attention heads refer to independent basic computational subunits of the multi-head attention mechanism in the text branches of multimodal large models. Each attention head can independently complete text sequence queries, key-value feature mapping, and long-distance semantic dependency modeling. Feedforward network intermediate layers refer to hidden feature transformation layers set between the input feature projection layer and the output feature projection layer in the feedforward network of multimodal large models. These layers can be used to complete the mapping, transformation, and dimensional reconstruction of high-dimensional text features and are the core network structure for realizing semantic feature representation in text branches.

[0039] Understandably, by using the NPU memory hierarchy characteristics and NPU hardware memory constraints as the hardware optimization benchmark, hardware-adapted structured pruning can be performed on the original multimodal large model text branches. This can be achieved by jointly evaluating gradient sensitivity, activation sparsity, and downstream task performance degradation to identify redundant attention heads with weak modeling capabilities and low task contributions in the text branches. At the same time, redundant feedforward network intermediate layers with large parameter counts and caching overhead can be targeted for pruning, simplifying the redundant computational structure and feature mapping dimension of the text branches. This reduces the weight storage occupation and intermediate tensor caching overhead during the text inference process, ultimately resulting in a text branch pruning structure that adapts to the upper limit of NPU hardware memory resources and hierarchical scheduling characteristics.

[0040] Optionally, based on memory hierarchy characteristic parameters and NPU hardware memory constraints, redundant attention heads and intermediate layers of the feedforward network in the text branches of the original multimodal large model are pruned to obtain a memory-adapted text branch pruning structure, including: Based on the NPU's memory level characteristics, hardware memory constraints, gradient information of text branches, activation statistics, and downstream task performance data, calculate the gradient sensitivity score, activation sparsity of the attention weight matrix, and performance degradation of downstream tasks after removing the attention head for each attention head in the text branch. Based on the calculation results and the preset matching thresholds, redundant attention heads are identified in each attention head. Combined with the NPU hardware memory bandwidth and capacity constraints, redundant attention heads are pruned and the intermediate layer of the feedforward network is pruned to obtain the text branch structure after preliminary pruning. A text modality quantization calibration dataset is constructed. The text modality quantization calibration dataset is input into the pre-pruned text branch structure and forward propagation is performed to obtain the activation value statistics of each layer of the text branch structure. Based on the activation value statistics, dynamic range calibration and outlier suppression are performed to calculate the quantization parameters of each layer of the text branch structure. Based on each quantization parameter, the attention projection layer weights associated with cross-modal alignment are located in the text branch structure, and the weights of each layer except the attention projection layer weights are quantized to the second target precision to obtain the quantized text branch parameters. Construct a downstream task validation set, perform quantization accuracy verification on the downstream task validation set using the quantized text branch parameters, calculate the performance degradation before and after quantization of each layer of the text branch, and identify the layers whose performance degradation exceeds a preset threshold as precision-sensitive layers. Detect whether the overall accuracy index of each layer of the text branch decreases beyond a preset threshold; If so, a callback of at least one level of quantization precision is performed for at least one precision-sensitive layer; wherein, the callback start point of each precision-sensitive layer is the second target precision, the callback end point is the precision before quantization, and there is at least one precision level between the callback start point and the callback end point. Return to the operation of checking whether the overall precision index of each layer of the text branch has decreased beyond the preset threshold, until the overall precision index of each layer of the text branch meets the preset threshold requirement, and obtain the memory-adapted text branch pruning structure.

[0041] The attention weight matrix activation sparsity refers to the proportion of near-zero values ​​in the activation tensor generated by forward inference of the attention head weight matrix of the text branch in a multimodal large model. It can be used to characterize the effectiveness of the semantic activation of the attention head. The text modality quantization calibration dataset refers to a dedicated dataset covering text samples of various lengths, built based on the actual application scenarios of multimodal large models to adapt to the text branch quantization accuracy calibration requirements. It can be used to collect the distribution characteristics of activation values ​​at each layer of the network for solving quantization parameters. Quantization parameters refer to the core calibration parameters used to map floating-point precision to integer precision during model quantization. These parameters may include scaling factors (Scale) and zero-point offsets (ZeroPoint), and can be used to standardize numerical quantization mapping rules. The second target precision refers to the preset basic quantization precision level of the model weights, which can be set to INT4 (4-bit integer) precision, and is the execution precision standard for regular weight quantization of text branches. Text branch parameters refer to the set of all trainable network parameters included in the text branch of the multimodal large model, such as attention weights, feedforward network weights, and projection weights of each layer. Downstream task validation sets refer to dedicated datasets used to evaluate the text branch inference performance of multimodal large models, covering core downstream application scenarios such as text-to-image question answering, cross-modal retrieval, and visual dialogue. These datasets can be used to quantitatively evaluate the inference performance of the model after quantization. Precision-sensitive layers refer to network layers whose downstream task performance indicators decay beyond a preset threshold after multimodal large model quantization. These network layers are highly sensitive to changes in quantization precision.

[0042] Specifically, hardware-adaptive pruning and hierarchical quantization optimization are performed on the text branches of the original multimodal large model. First, the memory level characteristic parameters and hardware memory constraint parameters of the NPU can be obtained. Gradient information, activation value statistics, and downstream task performance data corresponding to the text branches are collected simultaneously. Based on multi-dimensional data, the gradient sensitivity score, activation sparsity of the attention weight matrix, and the performance degradation of the downstream task after removing each attention head are calculated for each attention head, thereby achieving a quantitative evaluation of the task contribution and redundancy of each attention head. Subsequently, the above-calculated quantification results are compared with the corresponding preset thresholds to identify redundant attention heads in the text branches. At the same time, combined with the inherent constraints of NPU hardware memory bandwidth and storage capacity, targeted pruning operations are performed on redundant attention heads and the intermediate layers of the feedforward network are optimized to simplify the redundant network structure of the text branches, resulting in the preliminary pruned text branch structure. Construct a text modality quantization calibration dataset adapted to multiple scenarios and length distributions. For example, the dataset can include short texts of less than 32 tokens, medium-length texts ranging from [32 tokens to 512 tokens], and long texts ranging from [512 tokens to 2048 tokens], comprehensively covering real-world inference input scenarios. Input this calibration dataset into the pre-pruned text branch structure for forward propagation, collecting the complete activation value statistics of each network layer in the text branch. Perform dynamic range calibration and outlier suppression based on the activation value distribution characteristics of each layer. For example, the mean plus three times the standard deviation (…) can be used… The decision rule truncates abnormal extreme values ​​of activation values ​​to avoid compressing the effective feature range with extreme values. Based on the calibrated activation value range, the specific quantization parameters for each layer are calculated, providing data support for accurate quantization. According to the obtained quantization parameters for each layer, differentiated mixed-precision quantization is performed on the text branch structure. This accurately locates the attention projection layer weights responsible for visual-text cross-modal alignment in the text branch, preserving their high-precision characteristics to maintain cross-modal semantic consistency. Simultaneously, the weights of the remaining network layers, except for the attention projection layer weights, are quantized to a preset second target precision (i.e., INT4 precision), generating the quantized text branch parameters. To verify the quantization optimization effect, a corresponding downstream task validation set can be constructed. The quantized text branch parameters are then fed into the downstream task validation set to complete inference testing, performing quantization precision comparison verification. The performance degradation of each layer in the text branch before and after quantization is statistically analyzed. Network layers whose performance degradation exceeds a preset threshold are judged as precision-sensitive layers. For example, an increase in perplexity exceeding 5% can be used as the precision anomaly judgment threshold.

[0043] Furthermore, it can detect whether the overall precision index of the text branch decreases beyond a preset threshold. If the overall precision does not meet the preset requirements, multi-level quantization precision callback optimization is performed for at least one precision-sensitive layer. The precision callback starts with the second target precision and ends with the original floating-point precision before quantization. The start and end points can include multiple switchable precision levels such as INT8, achieving gradient precision repair. The overall precision detection and precision callback operations are iteratively executed until the overall precision index of each layer of the text branch meets the preset threshold requirements, ultimately resulting in a memory-adaptive text branch pruning structure that balances NPU memory resource constraints, inference efficiency, and cross-modal semantic accuracy.

[0044] S240. Based on the characteristic parameters calculated using the mixed precision, the alignment key dimension and non-aligned dimension of the cross-modal fusion branch in the original multimodal large model are determined through gradient sensitivity analysis. After performing low-rank decomposition compression on the non-aligned dimension, quantization with the first target precision is applied to obtain the compressed structure of the cross-modal fusion branch.

[0045] Among them, the mixed-precision computational characteristic parameters can be hardware specification parameters used to characterize the NPU hardware's native support for parallel computation of multiple numerical precisions, and can be used to guide the configuration of differentiated precision for different feature branches and different network layers in multimodal large models. Alignment key dimensions can refer to feature dimensions in cross-modal fusion branches that contribute significantly to image-text matching loss, have dense and stable feature activations, and can directly determine the semantic matching effect between images and text. Non-aligned dimensions can refer to redundant feature dimensions in cross-modal fusion branches that do not dominate image-text semantic alignment and have a weak impact on cross-modal matching loss, possessing room for compression and optimization. The first target precision can refer to the preset quantization precision level for non-aligned dimensions in cross-modal fusion branches, which can be set to INT16 (16-bit integer) precision.

[0046] Understandably, the NPU's mixed-precision computing characteristic parameters are used as the basis for differentiated precision configuration. Gradient sensitivity analysis is used to quantify the contribution of each feature dimension to the cross-modal alignment loss of text and images. Based on this, the alignment key dimension and non-alignment dimension within the cross-modal fusion branch are divided. The alignment key dimension is preserved and protected and does not participate in the compression process. For redundant non-alignment dimensions, a low-rank decomposition method can be used to complete parameter sparsity compression. Then, the first target precision quantization process is applied to the compressed non-alignment dimension network, while maintaining the high-precision representation capability of the alignment key dimension. Finally, a cross-modal fusion branch compression structure that balances cross-modal semantic alignment accuracy and hardware computing efficiency is obtained.

[0047] Optionally, based on the characteristic parameters calculated using mixed precision, the alignment key dimensions and unaligned dimensions of the cross-modal fusion branches in the original multimodal large model are determined through gradient sensitivity analysis. After performing low-rank decomposition compression on the unaligned dimensions, quantization with the first target precision is applied to obtain the compressed structure of the cross-modal fusion branches, including: Based on the obtained weight matrix, gradient information, and activation value statistics of the cross-modal fusion branch, the contribution of each feature dimension in the cross-modal fusion branch to the cross-modal alignment loss is calculated through gradient sensitivity analysis and activation value sparsity statistics. The alignment key dimensions and non-alignment dimensions of the cross-modal fusion branch are determined according to the preset contribution threshold, and the corresponding dimension labeling information is generated respectively. Based on the dimension labeling information, the projection layer weight matrix of the cross-modal fusion branch is divided into aligned dimension submatrices and unaligned dimension submatrices; Singular value decomposition is performed on the unaligned dimension submatrix to obtain the corresponding singular values. The rank parameter of the low-rank decomposition is determined based on the cumulative explained variance ratio of the singular values. The unaligned dimension submatrix is ​​then expressed as the product of two low-rank matrices according to the rank parameter. Perform low-rank adaptation training on two low-rank matrices and an aligned dimension submatrix; During low-rank adaptation training, the parameters of the aligned dimension submatrix are frozen, the parameters of the two low-rank matrices are updated, and after training, the first target precision quantization is applied to the two low-rank matrices of the unaligned dimension to obtain the sparsified weights of the cross-modal fusion branch. Based on the sparse weights, the computation process of the cross-modal fusion branch is reorganized into a two-step matrix multiplication form adapted to the NPU hardware architecture, resulting in a compressed structure for the cross-modal fusion branch.

[0048] The contribution threshold refers to a quantitative critical value used to determine the importance of cross-modal feature dimensions to the image-text alignment task. If the contribution of a feature dimension to the cross-modal alignment loss is higher than this threshold, it is considered a key alignment dimension; if it is lower, it is considered a compressible non-alignment dimension. The projection layer weight matrix refers to the two-dimensional weight matrix used in the cross-modal fusion module to achieve the mapping and transformation of visual and textual feature dimensions; it is the core parameter carrier of the cross-modal fusion branch. Singular values ​​refer to the diagonal matrix elements obtained after orthogonal diagonalization of the weight matrix. They can be used to quantify the effective information amplitude and importance of each dimension of the matrix. The larger the value, the higher the proportion of effective information from the original data carried by the corresponding feature dimension, and the stronger the feature representation ability. The cumulative explained variance ratio (CRR) of singular values ​​refers to a quantitative indicator used to evaluate the information fidelity of the low-rank reconstruction of the matrix. By statistically analyzing the sum of the squared information energy of the preceding core principal components after matrix decomposition, it accounts for the proportion of the sum of the squared information energy of all principal components, accurately reflecting the degree to which the low-rank structure retains the effective feature information of the original matrix. It is the core criterion for determining the low-rank decomposition dimension. The rank parameter can refer to the core parameter used to constrain the dimension of the low-rank matrix during low-rank decomposition. It determines the parameter to retain the original weight matrix and can directly control the compression ratio and information retention accuracy of the model parameters. Sparse weights can refer to the novel weight parameters of the cross-modal fusion branch obtained after low-rank decomposition, parameter fine-tuning, and quantization. They are composed of a high-precision aligned dimension submatrix and a low-precision low-rank matrix.

[0049] Specifically, by acquiring the projection layer weight matrix, network gradient information, and feature activation value statistics of the cross-modal fusion branch, and relying on gradient sensitivity analysis and activation value sparsity statistics, the contribution of each feature dimension to the image-text cross-modal alignment loss can be accurately quantified. A preset contribution threshold is constructed by statistically analyzing the contribution distribution of all feature dimensions of the cross-modal fusion branch to the cross-modal alignment loss, forming a basic statistical threshold baseline. Combined with the allowable attenuation range of the model's cross-modal alignment accuracy, candidate threshold intervals that meet accuracy fidelity requirements are selected. Then, based on the preset model parameter compression ratio, the final contribution threshold is determined from the candidate threshold intervals, ensuring that while maximizing the compression of redundant parameters, the core cross-modal alignment capability is preserved. This allows for accurate classification of key alignment dimensions and non-alignment dimensions to adapt to different task scenarios. By comparing the contribution of each dimension with the preset contribution threshold, key alignment dimensions that play a decisive role in cross-modal semantic alignment and non-alignment dimensions that have little impact on the alignment loss and possess compression redundancy are identified. Dedicated dimension labeling information is generated for each type of dimension, achieving accurate classification and definition of cross-modal feature dimensions. Subsequently, based on the aforementioned dimension labeling information, the projection layer weight matrix of the cross-modal fusion branch is split into an aligned dimension submatrix that retains the original parameters and an unaligned dimension submatrix to be compressed and optimized, thereby achieving physical isolation between key semantic information and redundant parameters.

[0050] Furthermore, singular value decomposition can be performed on the unaligned dimension submatrices obtained from the splitting to obtain a series of corresponding singular values. The decomposition formula can be: ,in For left singular vector matrix, For singular value diagonal matrices, Let be the transpose of the right singular vector. The optimal low-rank decomposition rank parameter is determined based on the cumulative explained variance ratio of singular values. The formula for calculating the cumulative explained variance ratio is: ,in, Take the rank parameter The corresponding cumulative explained variance ratio To preset the number of low-rank decomposition dimensions, The magnitude of the i-th principal component information obtained after matrix decomposition and sorting. This represents the total number of principal components after the weight matrix decomposition. For example, it can be selected to satisfy the cumulative explained variance ratio. hour The minimum value of the rank parameter is used as the rank parameter. While preserving the core effective information of the original matrix, the parameter size is compressed to the maximum extent. For example, a typical value for the rank parameter can be set to [8, 64], which is much lower than the conventional original feature dimension of [1024, 4096]. Based on the determined rank parameter, the high-dimensional unaligned submatrix is ​​approximately equivalent to the product of two low-rank matrices, realizing the low-rank sparse representation reconstruction of the unaligned weights. Low-rank adaptation training is performed on the aligned submatrix obtained from the splitting and the two sets of low-rank matrices. During training, a fixed parameter constraint strategy is implemented, freezing the parameters of the aligned submatrix throughout the process and preventing them from participating in gradient updates. Only backpropagation is performed and the parameters of the two sets of low-rank matrices are iteratively updated, ensuring that the cross-modal aligned core feature expression does not shift and the semantic alignment capability is not compressed or degraded. After the low-rank adaptation training converges, the first target precision quantization processing is applied to the two sets of low-rank matrices corresponding to the unaligned dimension. For example, INT8 precision quantization compression can be used, ultimately forming a cross-modal fusion branch sparse weight composed of a high-precision aligned submatrix and a low-precision low-rank quantized matrix. Finally, based on the sparse weights, the overall computational logic of the cross-modal fusion branch is reconstructed. The original single high-dimensional matrix operation is reorganized into a two-step matrix multiplication operation of the Cube computing unit adapted to NPU hardware. That is, the matrix operation between the low-rank matrix and the input features is completed first, and the mapping operation of the low-rank matrix is ​​completed. By reconstructing the inference process to adapt to the NPU hardware computing architecture, the hardware batch matrix acceleration capability and on-chip cache resources are fully utilized to reduce inference latency and power consumption. In this way, a cross-modal fusion branch compression structure with high cross-modal alignment accuracy, low computational overhead and adaptation to NPU resource constraints is finally obtained.

[0051] S250 outputs a simplified multimodal large model that includes visual branch pruning structure, text branch pruning structure, and cross-modal fusion branch compression structure.

[0052] It is understandable that by assembling and integrating the visual branch pruning structure obtained through NPU hardware adaptation and structured pruning, the text branch pruning structure obtained through memory adaptation pruning and quantization optimization, and the cross-modal fusion branch compression structure obtained through dimensionality protection and low-rank sparse compression, a streamlined multimodal large model with overall optimization can be output.

[0053] S260. Using the original multimodal large model as the teacher model, a cross-modal knowledge distillation method is adopted to perform lightweight training on the simplified multimodal large model according to the NPU hardware constraint parameters, so as to obtain a lightweight multimodal large model adapted to the NPU hardware, and obtain the matching lightweight model parameters and compression configuration information.

[0054] Optionally, the original multimodal large model is used as the teacher model. A cross-modal knowledge distillation method is employed to perform lightweight training on the simplified multimodal large model based on NPU hardware constraint parameters, resulting in a lightweight multimodal large model adapted to the NPU hardware. The matching lightweight model parameters and compression configuration information are then obtained, including: The original multimodal large model is used as the teacher model, and the simplified multimodal large model is used as the student model. The first-layer text query visual attention matrix and output distribution data of the teacher model are obtained. Based on the first-level text query visual attention matrix of the teacher model, the first-level attention distillation is performed to enable the student model to learn the corresponding attention weight distribution, thus obtaining the first-level attention distillation loss. Based on the visual attention matrix of the first-level text query in the teacher model, visual token importance distillation is performed to calculate the importance score of each visual token. High-scoring visual tokens are selected and retained according to the preset scoring criteria to obtain the visual token distillation loss and token selection results. Obtain the output distribution data of the student model and construct a loss function by combining it with the output distribution data of the teacher model. Calculate the inverse relative entropy divergence loss based on the output distribution data of the student model and minimize this loss function to obtain the output distribution distillation loss. Based on the first-layer attention distillation loss, visual token distillation loss, output distribution distillation loss, and token selection results, combined with NPU hardware constraint parameters, joint iterative training is performed to obtain a lightweight multimodal large model adapted to NPU hardware, lightweight model parameters, and compressed configuration information.

[0055] The first-layer text query visual attention matrix refers to the weight matrix generated in the first attention layer of the multimodal large model by associating and matching visual features with text features as query vectors. It can be used to characterize the initial cross-modal association strength and alignment relationship between text tokens and visual tokens. The output distribution data refers to the probability distribution data generated by the downstream task output dimension after the multimodal large model completes feature inference. It can be used to reflect the model's semantic prediction and feature fitting results for the input samples. The first-layer attention distillation loss refers to the difference quantification loss between the student model's first-layer cross-modal attention weight distribution and the teacher model's first-layer text query visual attention matrix. It can be used to supervise the student model's learning of the teacher model's core cross-modal alignment attention rules. The visual token distillation loss refers to the difference quantification loss between the student model and the teacher model in visual token importance scoring and effective visual token selection matching. It can be used to constrain the student model to accurately select high-value visual features and eliminate redundant visual features. The token selection result refers to the set of high-semantic-contribution visual tokens retained after selection based on visual token importance scores, along with the corresponding simplified visual feature results. This provides effective visual feature input basis for the model's lightweight inference. Distributional distillation loss refers to the inverse relative entropy divergence loss calculated by combining the student model's output distribution with the teacher model's output distribution. It can be used to quantify the fitting differences in the semantic distributions of the teacher and student model outputs, enabling accurate transfer of model output knowledge. Lightweight multimodal large models refer to streamlined multimodal large models that have undergone branch pruning, quantization compression, low-rank sparsity optimization, and multi-dimensional knowledge distillation joint optimization to adapt to NPU hardware resource constraints while retaining the original multimodal core capabilities.

[0056] Specifically, the uncompressed, original multimodal large model is used as the high-precision teacher model, while the simplified multimodal large model, after branch pruning, quantization, and sparse compression, is used as the student model to be optimized. The first-layer text query visual attention matrix and model output distribution data generated during the teacher model's inference process are collected as the supervised data source for knowledge distillation. The first-layer text query visual attention matrix carries the initial association matching weights between visual and text features, serving as the core feature basis for multimodal cross-semantic alignment. Based on the obtained first-layer text query visual attention matrix of the teacher model, first-layer attention distillation training is performed to guide the structurally simplified student model to fit and learn the core attention weight distribution features of the teacher model. This constrains the student model's initial cross-modal alignment logic to remain consistent with the teacher model, allowing the calculation of the first-layer attention distillation loss. Additionally, visual token importance distillation can be performed using the visual attention matrix of the first-layer text query in the teacher model. By statistically analyzing the average weight value of each visual token being attended by the text tokens, the importance score of each visual token is quantified. Based on a preset scoring standard, all visual tokens are filtered and sorted, retaining effective visual tokens with high scores and high semantic contributions, while eliminating or compressing invalid visual tokens with low scores and primarily background redundancy. This yields the corresponding visual token distillation loss and visual token selection results. Furthermore, output distribution data generated by the student model inference can be collected, and a loss function adapted to cross-modal distillation scenarios can be constructed by combining it with the teacher model output distribution data. The inverse relative entropy divergence loss is constructed and calculated based on the student model output distribution. The output distribution distillation loss is obtained by iteratively minimizing this loss function. The inverse relative entropy divergence loss can be calculated as follows, for example, the loss function expression is... In the formula For mathematical expectation, Output distribution for the student model. The output distribution of the teacher model is provided. Finally, three types of multi-dimensional distillation supervision losses—first-layer attention distillation loss, visual token distillation loss, and output distribution distillation loss—are integrated. Combined with the visual token selection results and NPU hardware constraint parameters, multi-objective joint iterative training is conducted. While distilling and learning transfers high-precision knowledge from the teacher model, it continuously adapts to the boundary constraints of hardware resources such as NPU computing power, memory, and storage, iteratively optimizing the student model structure and parameters. The final trained model is a lightweight multimodal large model that is fully adapted to NPU hardware deployment, possessing both high accuracy and low inference overhead, and can output corresponding lightweight model parameters and model compression configuration information.

[0057] S270. Based on the lightweight model parameters and compression configuration information, the lightweight multimodal large model is compiled and optimized to obtain a multi-stream parallel instruction sequence and deployment package adapted to NPU hardware.

[0058] Optionally, based on the lightweight model parameters and compression configuration information, the lightweight multimodal large model is compiled and optimized to obtain a multi-stream parallel instruction sequence and deployment package adapted to NPU hardware, including: Using lightweight model parameters and compressed configuration information, the lightweight multimodal large model is parsed, path tracing is performed to construct the corresponding computation graph, and topological boundary marking is performed on the nodes of the corresponding visual branch, text branch and cross-modal fusion branch in the computation graph to obtain the multimodal branch topological marking. Extract and label the dynamic control flow structure of the computation graph to obtain control flow labels. Use the multimodal branch topology labels and control flow labels to perform path-aware shape derivation to obtain shape constraint information. By utilizing multimodal branch topology tags, control flow tags, and shape constraint information, compatibility analysis and downgrade mapping are performed on each operator in the computation graph. Hardware fusion rewriting is performed on three key operators: multi-head attention, rotation position encoding, and cross-modal query fusion. Dynamic bucketing is performed on shape constraints to obtain an optimized computation graph and shape bucket configuration after operator adaptation. By optimizing the computation graph and shape bucket configuration, the graph optimization engine performs operator fusion, memory reuse and data arrangement optimization operations on the entire computation graph to generate a hardware instruction sequence. The hardware instruction sequence is then optimized by resolving memory access conflicts and pipelined parallel scheduling to obtain a multi-stream parallel instruction sequence adapted to NPU hardware. An offline model file that can be executed by the NPU hardware is generated based on the multi-stream parallel instruction sequence. Combined with model parameters, shape configuration and runtime dependency information, a deployment package adapted to the NPU hardware is generated.

[0059] The computation graph can refer to a directed acyclic graph structure composed of all network operators, tensor data, and data dependencies in a multimodal large-scale model. It can be used to fully represent the execution logic and topological relationships of the forward inference of the multimodal model. Multimodal branch topology tags refer to the exclusive tagging information generated for the boundary nodes and topological structures of visual branches, text branches, and modality fusion branches in the computation graph. This can be used to distinguish the computational range of subgraphs of different modalities. Control flow tags refer to graph-level tags generated after structured identification of dynamic execution structures such as conditional branches, loop iterations, and multi-way jumps in the inference process of a multimodal large-scale model. This can be used to represent the dynamic execution logic of the computation graph. Shape constraint information refers to the dimensional value range and dimensional constraint relationships of each network tensor under different input conditions, derived from the control flow execution path. This can be used to represent the legal space of the model's dynamic input size. Rotation position encoding refers to the position representation method of text attention computation in a multimodal large-scale model. Through rotation transformation, the relative position information of the text sequence is embedded into the query and key vector to achieve semantic modeling of sequence position. Cross-modal query fusion refers to the cross-modal feature aggregation computation process that retrieves visual features using learnable query vectors and achieves cross-association and fusion of visual and textual features. Shape bucket configuration refers to multiple combinations of typical input shapes obtained after clustering and discretizing continuous dynamic input dimensions; it is a core configuration parameter for matching the compiler pre-compilation with the runtime model version. Hardware instruction sequence refers to the set of underlying machine instructions for operator operations, data memory access, and data layout adjustment adapted to NPU hardware after compilation optimization. Multi-stream parallel instruction sequence refers to the final hardware instruction sequence that supports multi-modal multi-computation stream parallel execution and efficient reuse of hardware units after eliminating memory access conflicts and completing pipeline scheduling optimization. Shape configuration refers to a complete set of parameter configurations including the model's legal input resolution, sequence length, tensor dimension constraints, and corresponding shape bucket mapping relationships; it can be used to support dynamic adaptation during multi-modal large model inference runtime.

[0060] Specifically, based on the obtained lightweight model parameters and corresponding compressed configuration information, the lightweight multimodal large model is structurally analyzed. A complete model computation graph is constructed through execution path tracing. Based on modal functional boundaries, the topological nodes of visual branches, text branches, and cross-modal fusion branches within the computation graph are precisely delineated and marked, generating multimodal branch topological markers to distinguish the scope of each modal subgraph. Building upon the multimodal topological markers, all dynamic control flow structures, such as conditional branches, loop iterations, and multi-path jumps, are extracted from the computation graph. Standardized node markings are applied to these dynamic control flow structures to obtain the control flow markers for the computation graph. Combining the multimodal branch topological markers and control flow markers, path-aware tensor dimension derivation is performed according to different control flow execution paths. The range of tensor dimension values ​​and dimensional constraint relationships under each execution path are obtained, yielding complete model shape constraint information. Based on the aforementioned multimodal branch topology tags, control flow tags, and shape constraint information, hardware compatibility analysis is performed on all operators in the computation graph. Adaptive degradation mapping is executed for incompatible operators. Simultaneously, for three core cross-modal computation operators—multi-head attention, rotation-position encoding, and cross-modal query fusion—NPU-specific operator fusion rewriting optimizations are performed. For example, rotation-position encoding can be optimized using rotation transformation formulas. and To achieve location information embedding, where, , These are the m-th query vector and the n-th key vector of the original text sequence, respectively. The orthogonal rotation matrix is ​​related to the sequence position. For rotation angle parameters, and This refers to the query vector and key vector after incorporating location information. For example, cross-modal query fusion can use a cross-attention calculation formula. Complete feature aggregation, where, For learnable text query vectors and These are the visual feature key vector and value vector, respectively. The scaling factor is the feature dimension. The normalized activation function can be used to output weighted aggregated cross-modal fusion features. Multiple serial basic operators are merged into a single hardware-optimized fusion operator. Simultaneously, dynamic shape bucketing can be performed based on the overall model shape constraints. For example, K-Means clustering combined with the elbow rule can be used to divide the model into 10 typical input shape buckets, completing the discretized adaptation configuration of the model input size. Finally, the computational graph structure and corresponding shape bucket configuration after operator adaptation optimization are obtained. Based on the optimized computational graph and shape bucket configuration after operator adaptation, the NPU hardware-matched graph optimization engine is called to perform global optimization on the entire computational graph, completing operator fusion, tensor memory reuse, and hardware-adapted data arrangement optimization. Furthermore, a two-layer optimization is performed sequentially on the original hardware instruction sequence. First, memory access conflicts are resolved. For example, memory access blocking can be eliminated by introducing an offset base that is coprime to the amount of memory space to disperse memory access addresses and by dividing the memory space into modal-specific memory space groups. Then, instruction-level pipeline parallel scheduling and timing orchestration are performed. By constructing an instruction definition-to-use chain to analyze data dependencies, the pipeline is divided into multi-stage stages containing memory access, computation, and write operations. The execution timing and resource allocation relationships of multimodal instructions are reconstructed, ultimately resulting in a multi-stream parallel instruction sequence that can fully utilize NPU hardware resources. Based on the optimized multi-stream parallel instruction sequence, an offline model file (e.g., an offline model in .om format) that can be directly parsed and executed by the NPU hardware is compiled. At the same time, lightweight model parameters, model shape configuration, and various runtime dependency resources required for the inference stage are integrated. This allows for the encapsulation of a multimodal lightweight model deployment package that is fully adapted to the NPU hardware architecture and can be directly deployed and run.

[0061] S280: Based on the multi-stream parallel instruction sequence and deployment package adapted to the NPU hardware, deploy the lightweight multimodal large model to the NPU hardware, and complete the environment configuration for loading and running the lightweight multimodal large model and the initialization of the multi-stream computing channels.

[0062] The technical solution of this invention first obtains NPU hardware constraint parameters, including Cube computing unit specification parameters, memory level characteristic parameters, and mixed-precision computing characteristic parameters. Based on the Cube computing unit specification parameters, the pruning granularity and Cube computing unit dimensions are aligned. The original multimodal large model's visual branch is pruned according to the corresponding pruning granularity to obtain a visual branch pruning structure aligned with the NPU hardware topology. Based on the memory level characteristic parameters and NPU hardware memory constraints, redundant attention heads and intermediate layers of the feedforward network in the original multimodal large model's text branch are pruned to obtain a memory-adapted text branch pruning structure. Relying on the mixed-precision computing characteristic parameters, gradient sensitivity analysis is used to determine the alignment key dimensions and non-alignment dimensions of the original multimodal large model's cross-modal fusion branch. Low-rank decomposition compression and first-target precision quantization are performed on the non-alignment dimensions to obtain a compressed cross-modal fusion branch structure. By integrating the visual branch pruning structure, text branch pruning structure, and cross-modal fusion branch compressed structure, a simplified multimodal large model is output. Using the original multimodal large model as the teacher model, a cross-modal knowledge distillation method combined with NPU hardware constraint parameters is employed to perform lightweight training on the simplified multimodal large model, resulting in a lightweight multimodal large model adapted to NPU hardware. The corresponding lightweight model parameters and compression configuration information are then determined. Based on these parameters and configuration, the lightweight multimodal large model undergoes compilation optimization to obtain a multi-stream parallel instruction sequence and deployment package adapted to NPU hardware. Using this sequence, the lightweight multimodal large model is deployed to the NPU hardware, completing model loading, runtime environment configuration, and multi-stream computation channel initialization, thus achieving hardware adaptation optimization and deployment of the multimodal large model. This technical solution effectively reduces the number of model parameters, computational load, and memory consumption while fully preserving the model's cross-modal semantic alignment capability and inference accuracy. Furthermore, through compilation optimization, dynamic shape bucketing, and pipelined parallel scheduling strategies, it significantly reduces model inference latency and compilation overhead, enabling efficient, high-precision, and highly stable deployment of multimodal large models on NPU hardware, expanding the application scenarios for lightweight models.

[0063] Example 3 Figure 3 This is a schematic diagram of a multimodal large model deployment device for a neural network processor (NPU) provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a model compression processing module 310, a model lightweight training module 320, a model compilation optimization processing module 330, and a model deployment module 340.

[0064] The model compression processing module 310 is used to obtain the original multimodal large model and NPU hardware constraint parameters. Based on the NPU hardware constraint parameters, the visual branch, text branch and cross-modal fusion branch of the original multimodal large model are compressed to obtain a simplified multimodal large model.

[0065] The lightweight model training module 320 is used to take the original multimodal large model as the teacher model, and use the cross-modal knowledge distillation method to perform lightweight training on the simplified multimodal large model according to the NPU hardware constraint parameters to obtain a lightweight multimodal large model adapted to the NPU hardware, and obtain the matching lightweight model parameters and compression configuration information.

[0066] The model compilation optimization processing module 330 is used to perform compilation optimization processing on the lightweight multimodal large model according to the lightweight model parameters and compression configuration information, so as to obtain a multi-stream parallel instruction sequence and deployment package adapted to NPU hardware.

[0067] The model deployment module 340 is used to deploy the lightweight multimodal large model to the NPU hardware according to the multi-stream parallel instruction sequence and deployment package adapted to the NPU hardware, and to complete the environment configuration for loading and running the lightweight multimodal large model and the initialization of the multi-stream computing channels.

[0068] The technical solution of this invention obtains the original multimodal large model and NPU hardware constraint parameters, and performs targeted compression processing on the visual branch, text branch, and cross-modal fusion branch of the original multimodal large model according to the NPU hardware constraint parameters to obtain a simplified multimodal large model with a streamlined structure. Then, using the original multimodal large model as the teacher model, a cross-modal knowledge distillation method is employed, combined with the NPU hardware constraint parameters, to perform lightweight training on the simplified multimodal large model, further obtaining a lightweight multimodal large model adapted to the NPU hardware, and simultaneously acquiring the matching lightweight model parameters and compression configuration information. Subsequently, based on the obtained lightweight model parameters and compression configuration information, the lightweight multimodal large model undergoes overall compilation optimization processing to generate a multi-stream parallel instruction sequence and corresponding deployment package adapted to the NPU hardware's operating characteristics. Finally, relying on the multi-stream parallel instruction sequence and deployment package adapted to NPU hardware, the lightweight multimodal large model is deployed to the NPU hardware. This completes the model loading, runtime environment configuration, and multi-stream computation channel initialization of the lightweight multimodal large model, achieving end-to-end lightweight adaptation and deployment of the multimodal large model for NPU hardware. This technical solution solves the problems of insufficient memory and computing power, difficulty in adapting framework operators, inefficient multimodal inference, and impaired semantic alignment when deploying multimodal large models on NPU hardware through hardware-aware cross-modal lightweighting and compilation-based collaborative optimization. It effectively adapts to NPU hardware resource constraints, ensuring inference stability while balancing cross-modal semantic inference accuracy and real-time inference performance.

[0069] Optionally, the model compression processing module 310 can be specifically used to obtain NPU hardware constraint parameters, including Cube computing unit specification parameters, memory level characteristic parameters, and mixed-precision computing characteristic parameters. Based on the Cube computing unit specification parameters, the pruning granularity is aligned with the Cube computing unit dimensions, and the visual branch of the original multimodal large model is pruned according to the pruning granularity to obtain a visual branch pruning structure aligned with the NPU hardware topology. Based on the memory level characteristic parameters and NPU hardware memory constraints, redundant attention heads and feedforward network intermediate layers in the text branch of the original multimodal large model are pruned to obtain a memory-adapted text branch pruning structure. Based on the mixed-precision computing characteristic parameters, the alignment key dimensions and non-alignment dimensions of the cross-modal fusion branch in the original multimodal large model are determined through gradient sensitivity analysis. Low-rank decomposition compression is performed on the non-alignment dimensions, followed by quantization with the first target precision to obtain the cross-modal fusion branch compressed structure. The output is a simplified multimodal large model containing the visual branch pruning structure, text branch pruning structure, and cross-modal fusion branch compressed structure.

[0070] Optionally, the model compression processing module 310 can also be specifically used to acquire gradient information, activation value statistics, downstream task performance data, and NPU Cube computing unit specification parameters of the visual branch. Through gradient sensitivity analysis and activation sparsity statistics, it calculates the gradient sensitivity score and activation sparsity value of each convolutional kernel in the visual branch. Based on the gradient sensitivity score and activation sparsity value, it filters and prunes convolutional kernels layer by layer in the visual branch. When the gradient sensitivity score of the target convolutional kernel is lower than a preset first threshold, the activation sparsity is higher than a preset second threshold, and the performance degradation after removal is less than a preset third threshold, the target convolutional kernel is determined to be a pruned convolutional kernel. The index and parameters of each pruned convolutional kernel are recorded to form a convolutional kernel pruning list. Based on the Cube computing unit specification parameters, the pruning granularity is aligned with the Cube computing unit dimension. Structured pruning is then performed according to the pruning list to obtain the initially pruned visual branch structure. Mixed precision quantization is then performed on the initially pruned visual branch structure to obtain a visual branch pruned structure aligned with the NPU hardware topology.

[0071] Optionally, the model compression processing module 310 can also be specifically used to calculate the gradient sensitivity score, attention weight matrix activation sparsity, and downstream task performance degradation value of each attention head in the text branch, based on the NPU's memory level characteristic parameters, hardware memory constraints, gradient information of the text branch, activation value statistics, and downstream task performance data. Based on the calculation results and preset matching thresholds, redundant attention heads are identified among the attention heads. Combining the NPU's hardware memory bandwidth and capacity constraints, redundant attention heads are pruned, and intermediate layers of the feedforward network are trimmed to obtain the pre-pruned text branch structure. A text modality quantization calibration dataset is constructed. This dataset is input into the pre-pruned text branch structure for forward propagation to obtain the activation value statistics of each layer in the text branch structure. Based on these activation value statistics, dynamic range calibration and outlier suppression are performed to calculate the quantization parameters of each layer in the text branch structure. Based on the quantization parameters, the attention projection layer weights associated with cross-modal alignment are located in the text branch structure. The weights of each layer except the attention projection layer weights are quantized to the second target precision, resulting in quantized text branch parameters. A downstream task validation set is constructed, and the quantized text branch parameters are validated on this set. The performance degradation of each layer before and after quantization is statistically analyzed, and layers with performance degradation exceeding a preset threshold are identified as precision-sensitive layers. The overall precision index of each layer of the text branch is then checked to see if the degradation exceeds the preset threshold. If so, at least one level of quantization precision is called back for at least one precision-sensitive layer. The callback start point for each precision-sensitive layer is the second target precision, and the callback end point is the pre-quantization precision. At least one precision level is included between the callback start point and the callback end point. The process of checking whether the overall precision index of each layer of the text branch has decreased beyond the preset threshold is then repeated until the overall precision index of each layer of the text branch meets the preset threshold requirement, resulting in a memory-adapted text branch pruning structure.

[0072] Optionally, the model compression processing module 310 can also be specifically used to calculate the contribution of each feature dimension in the cross-modal fusion branch to the cross-modal alignment loss based on the obtained weight matrix, gradient information, and activation value statistics of the cross-modal fusion branch, through gradient sensitivity analysis and activation value sparsity statistics. It determines the alignment key dimension and non-alignment dimension of the cross-modal fusion branch according to a preset contribution threshold and generates corresponding dimension labeling information. Based on the dimension labeling information, the projection layer weight matrix of the cross-modal fusion branch is divided into aligned dimension sub-matrices and non-aligned dimension sub-matrices. Singular value decomposition is performed on the non-aligned dimension sub-matrices to obtain the corresponding singular values. The rank parameter of the low-rank decomposition is determined based on the cumulative explained variance ratio of the singular values. The non-aligned dimension sub-matrices are then represented as the product of two low-rank matrices according to the rank parameter. Low-rank adaptation training is performed on two low-rank matrices and an aligned dimension submatrix. During low-rank adaptation training, the parameters of the aligned dimension submatrix are frozen, while the parameters of the two low-rank matrices are updated. After training, quantization with the first target precision is applied to the two unaligned dimension low-rank matrices to obtain the sparsified weights of the cross-modal fusion branch. Based on the sparsified weights, the computation flow of the cross-modal fusion branch is reorganized into a two-step matrix multiplication form adapted to the NPU hardware architecture, resulting in a compressed structure for the cross-modal fusion branch.

[0073] Optionally, the lightweight model training module 320 can be specifically used to use the original multimodal large model as the teacher model and the simplified multimodal large model as the student model, obtaining the first-layer text query visual attention matrix and output distribution data of the teacher model. Based on the first-layer text query visual attention matrix of the teacher model, first-layer attention distillation is performed, enabling the student model to learn the corresponding attention weight distribution, resulting in the first-layer attention distillation loss. Based on the first-layer text query visual attention matrix of the teacher model, visual token importance distillation is performed, calculating the importance score of each visual token, and selecting and retaining high-scoring visual tokens according to a preset scoring standard, resulting in the visual token distillation loss and token selection results. The output distribution data of the student model is obtained, and a loss function is constructed by combining it with the output distribution data of the teacher model. The inverse relative entropy divergence loss is calculated based on the student model's output distribution data and minimized, resulting in the output distribution distillation loss. Based on the first-layer attention distillation loss, visual token distillation loss, output distribution distillation loss, and token selection results, combined with NPU hardware constraint parameters, joint iterative training is performed to obtain a lightweight multimodal large model adapted to the NPU hardware, lightweight model parameters, and compression configuration information.

[0074] Optionally, the model compilation and optimization processing module 330 can be specifically used to parse a lightweight multimodal large model using lightweight model parameters and compressed configuration information, perform path tracing to construct the corresponding computation graph, and perform topological boundary marking on nodes in the computation graph corresponding to visual branches, text branches, and cross-modal fusion branches to obtain multimodal branch topological markings. The dynamic control flow structure of the computation graph is extracted and marked to obtain control flow markings. Path-aware shape derivation is performed using the multimodal branch topological markings and control flow markings to obtain shape constraint information. Using the multimodal branch topological markings, control flow markings, and shape constraint information, compatibility analysis and downgrade mapping are performed on each operator in the computation graph. Hardware fusion rewriting is performed on three key operators: multi-head attention, rotation position encoding, and cross-modal query fusion. Dynamic bucketing processing is performed on shape constraints to obtain an optimized computation graph and shape bucket configuration after operator adaptation. By optimizing the computation graph and shape bucket configuration, the graph optimization engine performs operator fusion, memory reuse, and data arrangement optimization operations on the entire computation graph to generate a hardware instruction sequence. The hardware instruction sequence is then subjected to memory access conflict resolution optimization and pipeline parallel scheduling orchestration to obtain a multi-stream parallel instruction sequence adapted to NPU hardware. Based on the multi-stream parallel instruction sequence, an offline model file that can be executed by NPU hardware is generated. Combining the model parameters, shape configuration, and runtime dependency information, a deployment package adapted to NPU hardware is generated.

[0075] The multimodal large model deployment device for neural network processors (NPUs) provided in this embodiment of the invention can execute the multimodal large model deployment method for neural network processors (NPUs) provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0076] Example 4 Figure 4 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0077] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0078] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0079] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as multimodal large model deployment methods for neural network processors (NPUs).

[0080] In some embodiments, the multimodal large model deployment method for a neural network processor (NPU) can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method XXX described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the multimodal large model deployment method for a neural network processor (NPU) by any other suitable means (e.g., by means of firmware).

[0081] That is, the original multimodal large model and NPU hardware constraint parameters are obtained, and the visual branch, text branch and cross-modal fusion branch of the original multimodal large model are compressed based on the NPU hardware constraint parameters to obtain a simplified multimodal large model. Using the original multimodal large model as the teacher model, a cross-modal knowledge distillation method is adopted to perform lightweight training on the simplified multimodal large model according to the NPU hardware constraint parameters, so as to obtain a lightweight multimodal large model adapted to the NPU hardware, and obtain the matching lightweight model parameters and compression configuration information. Based on the lightweight model parameters and compression configuration information, the lightweight multimodal large model is compiled and optimized to obtain a multi-stream parallel instruction sequence and deployment package adapted to NPU hardware. Based on the multi-stream parallel instruction sequence and deployment package adapted to NPU hardware, the lightweight multimodal large model is deployed to the NPU hardware, completing the environment configuration for loading and running the lightweight multimodal large model and the initialization of multi-stream computing channels.

[0082] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0083] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0084] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0085] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0086] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0087] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0088] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0089] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for deploying multimodal large models for neural network processors (NPUs), characterized in that, include: Obtain the original multimodal large model and NPU hardware constraint parameters. Based on the NPU hardware constraint parameters, compress the visual branch, text branch and cross-modal fusion branch of the original multimodal large model to obtain a simplified multimodal large model. Using the original multimodal large model as the teacher model, a cross-modal knowledge distillation method is adopted to perform lightweight training on the simplified multimodal large model according to the NPU hardware constraint parameters, so as to obtain a lightweight multimodal large model adapted to the NPU hardware, and obtain the matching lightweight model parameters and compression configuration information. Based on the lightweight model parameters and compression configuration information, the lightweight multimodal large model is compiled and optimized to obtain a multi-stream parallel instruction sequence and deployment package adapted to NPU hardware. Based on the multi-stream parallel instruction sequence and deployment package adapted to NPU hardware, the lightweight multimodal large model is deployed to the NPU hardware, completing the environment configuration for loading and running the lightweight multimodal large model and the initialization of multi-stream computing channels.

2. The method according to claim 1, characterized in that, Obtain the original multimodal large model and NPU hardware constraint parameters. Based on the NPU hardware constraint parameters, compress the visual branch, text branch, and cross-modal fusion branch of the original multimodal large model to obtain a simplified multimodal large model, including: Obtain NPU hardware constraint parameters, including Cube computing unit specification parameters, memory level characteristic parameters, and mixed precision computing characteristic parameters; The pruning granularity is aligned with the dimensions of the Cube computing unit according to the Cube computing unit specifications, and the visual branches of the original multimodal large model are pruned according to the pruning granularity to obtain a visual branch pruning structure aligned with the NPU hardware topology. Based on the memory hierarchy characteristic parameters and NPU hardware memory constraints, redundant attention heads and feedforward network intermediate layers in the text branches of the original multimodal large model are pruned to obtain a memory-adapted text branch pruning structure. Based on the mixed precision calculation characteristic parameters, the alignment key dimension and non-alignment dimension of the cross-modal fusion branch in the original multimodal large model are determined by gradient sensitivity analysis. After performing low-rank decomposition compression on the non-alignment dimension, quantization of the first target precision is applied to obtain the cross-modal fusion branch compression structure. The output includes a simplified multimodal large model with visual branch pruning structure, text branch pruning structure, and cross-modal fusion branch compression structure.

3. The method according to claim 2, characterized in that, Based on the Cube computing unit specifications, the pruning granularity is aligned with the Cube computing unit dimensions, and the visual branches of the original multimodal large model are pruned according to the pruning granularity to obtain a visual branch pruning structure aligned with the NPU hardware topology, including: Acquire gradient information, activation value statistics, downstream task performance data, and NPU Cube computing unit specifications of the visual branch. Through gradient sensitivity analysis and activation sparsity statistics, calculate the gradient sensitivity score and activation sparsity value of each convolutional kernel in the visual branch. Based on the gradient sensitivity score and activation sparsity value, pruned convolutional kernels are screened layer by layer in the visual branch. When the gradient sensitivity score of the target convolutional kernel is lower than the preset first threshold, the activation sparsity is higher than the preset second threshold, and the performance degradation value after removal is less than the preset third threshold, the target convolutional kernel is determined to be a pruned convolutional kernel. Record the index and parameters of each pruned convolution kernel to form a convolution kernel pruning list; Based on the Cube computing unit specifications, after aligning the pruning granularity with the Cube computing unit dimensions, structured pruning is performed according to the pruning list to obtain the visual branch structure after preliminary pruning. The visual branch structure after initial pruning is subjected to mixed precision quantization to obtain a visual branch pruning structure aligned with the NPU hardware topology.

4. The method according to claim 2, characterized in that, Based on memory hierarchy characteristics and NPU hardware memory constraints, redundant attention heads and intermediate layers of the feedforward network are pruned from the text branches of the original multimodal large model to obtain a memory-adapted text branch pruning structure, including: Based on the NPU's memory level characteristics, hardware memory constraints, gradient information of text branches, activation statistics, and downstream task performance data, calculate the gradient sensitivity score, activation sparsity of the attention weight matrix, and performance degradation of downstream tasks after removing the attention head for each attention head in the text branch. Based on the calculation results and the preset matching thresholds, redundant attention heads are identified in each attention head. Combined with the NPU hardware memory bandwidth and capacity constraints, redundant attention heads are pruned and the intermediate layer of the feedforward network is pruned to obtain the text branch structure after preliminary pruning. A text modality quantization calibration dataset is constructed. The text modality quantization calibration dataset is input into the pre-pruned text branch structure and forward propagation is performed to obtain the activation value statistics of each layer of the text branch structure. Based on the activation value statistics, dynamic range calibration and outlier suppression are performed to calculate the quantization parameters of each layer of the text branch structure. Based on the quantization parameters, the attention projection layer weights associated with cross-modal alignment are located in the text branch structure, and the weights of each layer except the attention projection layer weights are quantized to the second target precision to obtain the quantized text branch parameters. Construct a downstream task validation set, perform quantization accuracy verification on the downstream task validation set using the quantized text branch parameters, calculate the performance degradation before and after quantization of each layer of the text branch, and identify the layers whose performance degradation exceeds a preset threshold as precision-sensitive layers. Detect whether the overall accuracy index of each layer of the text branch decreases beyond a preset threshold; If so, a callback of at least one level of quantization precision is performed for at least one precision-sensitive layer; wherein, the callback start point of each precision-sensitive layer is the second target precision, the callback end point is the precision before quantization, and there is at least one precision level between the callback start point and the callback end point. Return to the operation of checking whether the overall precision index of each layer of the text branch has decreased beyond the preset threshold, until the overall precision index of each layer of the text branch meets the preset threshold requirement, and obtain the memory-adapted text branch pruning structure.

5. The method according to claim 2, characterized in that, Based on the mixed-precision calculation characteristic parameters, the alignment key dimensions and unaligned dimensions of the cross-modal fusion branches in the original multimodal large model are determined through gradient sensitivity analysis. After performing low-rank decomposition compression on the unaligned dimensions, quantization with the first target precision is applied to obtain the compressed structure of the cross-modal fusion branches, including: Based on the obtained weight matrix, gradient information, and activation value statistics of the cross-modal fusion branch, the contribution of each feature dimension in the cross-modal fusion branch to the cross-modal alignment loss is calculated through gradient sensitivity analysis and activation value sparsity statistics. The alignment key dimensions and non-alignment dimensions of the cross-modal fusion branch are determined according to the preset contribution threshold, and the corresponding dimension labeling information is generated respectively. Based on the dimensional labeling information, the projection layer weight matrix of the cross-modal fusion branch is divided into aligned dimensional submatrices and unaligned dimensional submatrices; Singular value decomposition is performed on the unaligned dimension submatrix to obtain the corresponding singular values. The rank parameter of the low-rank decomposition is determined based on the cumulative explained variance ratio of the singular values. The unaligned dimension submatrix is ​​then expressed as the product of two low-rank matrices according to the rank parameter. Perform low-rank adaptation training on the two low-rank matrices and the aligned dimension submatrix; During low-rank adaptation training, the parameters of the aligned dimension submatrix are frozen, the parameters of the two low-rank matrices are updated, and after training, the first target precision quantization is applied to the two low-rank matrices of the unaligned dimension to obtain the sparsified weights of the cross-modal fusion branch. Based on the sparse weights, the computation process of the cross-modal fusion branch is reorganized into a two-step matrix multiplication form adapted to the NPU hardware architecture, resulting in a compressed structure for the cross-modal fusion branch.

6. The method according to claim 1, characterized in that, Using the original multimodal large model as the teacher model, a cross-modal knowledge distillation method is employed to perform lightweight training on the simplified multimodal large model based on NPU hardware constraints, resulting in a lightweight multimodal large model adapted to the NPU hardware. The matching lightweight model parameters and compression configuration information are then obtained, including: The original multimodal large model is used as the teacher model, and the simplified multimodal large model is used as the student model. The first-layer text query visual attention matrix and output distribution data of the teacher model are obtained. Based on the first-level text query visual attention matrix of the teacher model, the first-level attention distillation is performed to enable the student model to learn the corresponding attention weight distribution, thus obtaining the first-level attention distillation loss. Based on the visual attention matrix of the first-level text query in the teacher model, visual token importance distillation is performed to calculate the importance score of each visual token. High-scoring visual tokens are selected and retained according to the preset scoring criteria to obtain the visual token distillation loss and token selection results. Obtain the output distribution data of the student model and construct a loss function by combining it with the output distribution data of the teacher model. Calculate the inverse relative entropy divergence loss based on the output distribution data of the student model and minimize this loss function to obtain the output distribution distillation loss. Based on the first-layer attention distillation loss, visual token distillation loss, output distribution distillation loss, and token selection results, combined with NPU hardware constraint parameters, joint iterative training is performed to obtain a lightweight multimodal large model adapted to NPU hardware, lightweight model parameters, and compressed configuration information.

7. The method according to claim 1, characterized in that, Based on the lightweight model parameters and compression configuration information, the lightweight multimodal large model is compiled and optimized to obtain a multi-stream parallel instruction sequence and deployment package adapted to NPU hardware, including: Using lightweight model parameters and compressed configuration information, the lightweight multimodal large model is parsed, path tracing is performed to construct the corresponding computation graph, and topological boundary marking is performed on the nodes of the corresponding visual branch, text branch and cross-modal fusion branch in the computation graph to obtain the multimodal branch topological marking. Extract and label the dynamic control flow structure of the computation graph to obtain control flow labels. Use the multimodal branch topology labels and control flow labels to perform path-aware shape derivation to obtain shape constraint information. By utilizing multimodal branch topology tags, control flow tags, and shape constraint information, compatibility analysis and downgrade mapping are performed on each operator in the computation graph. Hardware fusion rewriting is performed on three key operators: multi-head attention, rotation position encoding, and cross-modal query fusion. Dynamic bucketing is performed on shape constraints to obtain an optimized computation graph and shape bucket configuration after operator adaptation. Using the optimized computation graph and shape bucket configuration, the graph optimization engine performs operator fusion, memory reuse and data arrangement optimization operations on the entire computation graph to generate a hardware instruction sequence. Then, the hardware instruction sequence is sequentially optimized for memory access conflict resolution and pipeline parallel scheduling to obtain a multi-stream parallel instruction sequence adapted to NPU hardware. An offline model file that can be executed by the NPU hardware is generated based on the multi-stream parallel instruction sequence. Combined with model parameters, shape configuration and runtime dependency information, a deployment package adapted to the NPU hardware is generated.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the implementation method for multimodal large-scale deployment of NPU according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method for multimodal large-scale deployment of an NPU as described in any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method for multimodal large-scale deployment of NPU according to any one of claims 1-7.

Images