Deployment method and device of multi-modal large model, electronic equipment and storage medium

Abstract

The invention provides a deployment method and device of a multi-modal large model, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the field of deep learning and model deployment. According to the specific implementation scheme, the method comprises the steps of splitting a first multi-modal large model into a visual part and a language part; determining a first static graph model corresponding to the visual part and a second static graph model corresponding to the language part; and deploying the first multi-modal large model based on the first static graph model and the second static graph model. According to the embodiment of the invention, occupation of multi-modal large model deployment on hardware resources can be optimized, so that the reasoning speed is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the field of deep learning and model deployment. Background Art

[0002] Multimodal large models have demonstrated excellent performance in natural language processing, computer vision, and speech recognition. These models provide more comprehensive and accurate understanding and generation capabilities by integrating multiple data modalities (such as text, images, and audio). However, due to their large number of parameters and complex computing requirements, they face many challenges in actual deployment. Summary of the invention

[0003] The present disclosure provides a method, device, electronic device and storage medium for deploying a multimodal large model.

[0004] According to one aspect of the present disclosure, a method for deploying a multimodal large model is provided, comprising:

[0005] Split the first multimodal large model into visual and language parts;

[0006] Determine a first static graphic model corresponding to the visual part and a second static graphic model corresponding to the language part;

[0007] Based on the first static graph model and the second static graph model, the first multimodal large model is deployed.

[0008] According to another aspect of the present disclosure, a deployment device for a multimodal large model is provided, comprising:

[0009] A model splitting module, used for splitting the first multimodal large model into a visual part and a language part;

[0010] A static graph export module, used to determine a first static graph model corresponding to the visual part and a second static graph model corresponding to the language part;

[0011] A deployment module is used to deploy the first multimodal large model based on the first static graph model and the second static graph model.

[0012] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0018] According to the embodiments of the present disclosure, the occupation of hardware resources by the deployment of multimodal large models can be optimized, the utilization rate of hardware resources can be improved, and thus the reasoning speed can be improved.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 It is a flowchart of a method for deploying a multimodal large model provided in an embodiment of the present disclosure;

[0022] Figure 2 is a schematic diagram of an application example of a deployment method of a multimodal large model according to an embodiment of the present disclosure;

[0023] Figure 3 is a schematic diagram of another application example of the method for deploying a multimodal large model according to an embodiment of the present disclosure;

[0024] Figure 4 It is a schematic block diagram of a deployment device of a multimodal large model provided by an embodiment of the present disclosure;

[0025] Figure 5 is a schematic block diagram of a deployment device for a multimodal large model provided by another embodiment of the present disclosure;

[0026] Figure 6 It is a block diagram of an electronic device used to implement the method for deploying a multimodal large model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0028] In order to facilitate understanding of the deployment method of the multimodal large model provided by the embodiment of the present disclosure, the relevant technologies of the embodiment of the present disclosure are explained below. The following related technologies can be arbitrarily combined with the technical solutions of the embodiment of the present disclosure as optional solutions, and they all belong to the protection scope of the embodiment of the present disclosure.

[0029] At present, the default reasoning method of most multimodal large models is based on 16-bit floating point (FP16). Although FP16 provides higher accuracy, it will cause the model to occupy a large amount of video memory during reasoning, and increase the reasoning time of the model, making the model unusable in a resource-constrained environment. In order to solve this problem, the model will be quantized in related technologies, and the most commonly used method is 8-bit integer (INT8) quantization. There are two ways to implement INT8 quantization, one is to quantize only the weights (called WINT8), and the other is to quantize both activations and weights (called W8A8). In theory, the use of INT8 quantization can not only reduce the video memory occupied during model reasoning, but also accelerate the model reasoning process. However, in practice, the model using INT8 requires a dequantization process in the reasoning stage, which leads to an increase in calculation time after quantization. Therefore, a solution is needed that can both reduce the video memory occupied during reasoning and accelerate the model process. The following introduces some mainstream model reasoning frameworks.

[0030] (1)LLM.int8

[0031] LLM.int8 is an adaptive mixed-precision quantization method that uses different precisions to represent different types of data to improve computational efficiency and accuracy. In the LLM.int8 scheme, each element in the input feature is first classified according to its numerical size and importance. Those elements identified as outliers are specially marked, while the rest are treated as regular values. For regular values, 8-bit quantization is used for processing, which can significantly reduce storage and computational requirements. For outliers, their original 16-bit precision is retained to ensure that these important features do not lose too much information due to quantization during the calculation process. Through this mixed-precision decomposition method, the method successfully maintains the model prediction performance while significantly reducing the memory usage during the inference process.

[0032] (2) AirLLM

[0033] The core technology of AirLLM lies in its efficient memory management mechanism. It adopts advanced caching strategies and data flow scheduling algorithms to achieve effective reuse of model weights and intermediate results, thereby greatly reducing the demand for video memory space. The technical highlights behind AirLLM also include the intelligent splitting and rearrangement of the computational graph, which further reduces unnecessary memory usage. Through the method of hierarchical reasoning, AirLLM can load only the necessary layer data from disk when executing a specific layer, and release the memory after the calculation is completed, so that only the necessary sub-models remain in memory at any given time. In addition, AirLLM also applies block quantization technology to further compress sub-models, reducing disk loading time and memory usage.

[0034] (3) VLLM

[0035] VLLM is an open source large-model inference acceleration framework that aims to provide efficient memory management and computational optimization for large language models during the inference phase. The core technology of VLLM lies in the PagedAttention algorithm it adopts. This algorithm effectively manages the key and value tensors in the attention mechanism, solving the problem of low memory utilization in traditional autoregressive models. PagedAttention divides the KV cache into fixed-size blocks and stores these blocks in discontinuous memory space, thereby avoiding the problems of memory fragmentation and over-reservation, making the memory utilization close to the theoretical optimal value.

[0036] Using the LLM.int8 solution can reduce the model's memory usage during inference, but the inference speed is significantly reduced. This is because the inference stage requires a dequantization process, which increases the calculation time after quantization. In addition, AirLLM does not support multimodal models, and VLLM does not support quantization of multimodal models. The following will detail the shortcomings of different solutions.

[0037] Since LLM.int8 requires special processing of outliers and uses mixed precision calculations, this may increase the computational complexity during inference. In particular, on some hardware accelerators, the inference speed may be reduced compared to when using FP16 due to the difficulty in efficiently implementing outlier decomposition and mixed precision operations.

[0038] Since AirLLM needs to frequently load and release model layer data from disk during inference, this may increase the latency of I / O (input / output) operations, especially when the disk read speed is slow. Therefore, in real-time scenarios or applications with high inference speed requirements, the performance of AirLLM may be affected to a certain extent. In addition, the performance and effect of AirLLM are also affected by external factors, such as disk read and write speed, network latency, etc. In actual applications, these factors may cause additional performance fluctuations and uncertainties.

[0039] Currently, VLLM mainly supports some mainstream large-scale language models. For new or non-mainstream models, some customization may be required to adapt. This means that users may need to consider the range of models it supports when choosing VLLM. Since VLLM uses a series of innovative technical means to optimize reasoning performance, its internal implementation is relatively complex. This may pose a certain technical threshold for users when using and customizing VLLM.

[0040] The technical solution of the embodiment of the present disclosure can solve at least one of the above technical problems.

[0041] Figure 1 It is a flowchart of a method for deploying a multimodal large model provided by an embodiment of the present disclosure. The method can be applied to a deployment device for a multimodal large model, and the device can be deployed in an electronic device. The electronic device is, for example, a single or multi-machine terminal, server or other processing device. Among them, the terminal can be a user equipment (User Equipment, UE) such as a mobile device, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in a memory. For example Figure 1 As shown, the method may include:

[0042] S110, splitting the first multimodal large model into a visual part and a language part;

[0043] S120, determining a first static graphic model corresponding to the visual part and a second static graphic model corresponding to the language part;

[0044] S130: Deploy the first multimodal large model based on the first static graph model and the second static graph model.

[0045] In the above method, the first multimodal large model can be a large model that can process input information of multiple modes. Among them, multiple modes are, for example, images, texts, etc. A large model refers to a machine learning model with ultra-large-scale parameters (usually more than one billion) and super computing resources, which can process massive data and complete various complex tasks.

[0046] Exemplarily, the first multimodal large model can be used to process a first image and / or a first text, and output a second image and / or a second text.

[0047] In the disclosed embodiment, the first multimodal large model is a model that needs to be deployed on a hardware device. Here, deployment refers to configuring the trained model on a hardware device, which may be referred to as target hardware. Optionally, the deployment process may include one or more processes such as model structure optimization, model compression, reasoning optimization, and hardware adaptation. In practical applications, model deployment can be achieved by generating a prediction program for running on the target hardware and installing it on the target hardware.

[0048] In the disclosed embodiment, the first multimodal large model is deployed by determining a static graph model. The static graph model is a computational graph model that is defined first and then run. For the static graph model, the structure of the entire computational graph can be defined first, including the layers, nodes, and connection methods of the network, and then the data can be passed into the graph for calculation. Deployment based on the static graph model can statically allocate resources and optimize the computing process, so it has higher efficiency and performance during execution.

[0049] In the related art, it is not supported to export the complete multimodal large model as a static graph model, which limits the deployment and reasoning efficiency of the multimodal large model in practical applications to a certain extent. In the embodiment of the present disclosure, the first multimodal large model is split into a visual model and a language model, and then the corresponding static graph models, i.e., the first static graph model and the second static graph model, are determined for the visual model and the language model respectively. According to the above method of the embodiment of the present disclosure, the occupation of hardware resources by the deployment of the multimodal large model is optimized, and the utilization rate of hardware resources is improved, which not only improves the reasoning speed of the multimodal large model, but also reduces the hardware resource consumption of the multimodal large model during actual deployment, and successfully solves the problem that the related art does not support the direct export of the multimodal large model as a static graph. At the same time, the split visual part and language part are more flexible and can be more easily integrated into various application scenarios, providing new possibilities for the practical application of multimodal tasks.

[0050] Figure 2 FIG. 1 is a schematic diagram of an application example of a method for deploying a multimodal large model according to an embodiment of the present disclosure. Figure 2As shown, the first multimodal large model 200 includes a visual encoder 210 and a language model 220. The input information of the first multimodal large model 200 may include an image X v and the text sequence X q , the visual encoder 210 in the first multimodal large model 200 performs the image X v and the text sequence X q Execute encoding process Z v The processing results are H v and H q In the embodiment of the present disclosure, the encoding process Z v The processing results are concatenated and regarded as data in the form of a text sequence, which is input into the language model 220, and then processed by the language model 220 to obtain the output information X a In this way, the first multimodal large model 200 can be split into a visual part (including the visual encoder 210) and a language part (including the language model 220), and each of them can be exported as a static graph model.

[0051] Optionally, based on the training framework or version of the first multimodal large model, the corresponding static graph export tool can be used to determine the first static graph model corresponding to the visual part and the second static graph model corresponding to the language part. For example, when the first multimodal large model is a model based on the Paddle framework, the static graph export function provided by the Paddle framework can be used to export the two parts into corresponding static graph models.

[0052] In some embodiments, the method for deploying a multimodal large model further includes: obtaining a second multimodal large model to be deployed; when the training framework of the second multimodal large model is different from the target framework, based on the weight conversion rule between the training framework of the second multimodal large model and the target framework, processing the weight information in the second multimodal large model to obtain the first multimodal large model. Optionally, the above steps can be performed before splitting the first multimodal large model into a visual part and a language part.

[0053] Optionally, the second multimodal large model may be a large model capable of processing input information of multiple modes, wherein the multiple modes are, for example, images, texts, etc. Exemplarily, the second multimodal large model may be used to process the third image and / or the third text, and output a fourth image and / or a fourth text.

[0054] In the disclosed embodiment, when the training framework of the second multimodal large model to be deployed is different from the target framework, the weight information in the second multimodal large model can be converted to convert the second multimodal large model into the first multimodal large model corresponding to the target framework. Optionally, when the training framework of the second multimodal large model is the same as the target framework, the second multimodal large model is used as the above-mentioned first multimodal large model for deployment.

[0055] Exemplarily, the target framework is a framework that provides a static graph export tool. For example, the target framework is the Paddle framework. If the second multimodal large model to be deployed is trained based on the PyTorch framework, the weight conversion tool between the PyTorch framework and the Paddle framework is used. Based on the rules defined in the tool, the model weights of the PyTorch version of the second multimodal large model are converted to the model weights of the Paddle version to obtain the first multimodal large model. The first multimodal large model is split into a visual part and a language part, and the corresponding static graph models are exported respectively using the static graph export function of the Paddle framework. It can be understood that the second multimodal large model can also be trained based on other deep learning frameworks such as TensorFlow, Caffe, etc.

[0056] According to the above embodiments, the deployment of multimodal large models of different training frameworks can be supported, thereby improving the inference speed of the multimodal large models for different training frameworks and reducing the resource consumption of the multimodal large models during actual deployment.

[0057] In some embodiments, the above-mentioned step S120, determining the first static graph model corresponding to the visual part and the second static graph model corresponding to the language part, includes: compiling and installing a custom operator for hardware adaptation of the model; determining the first static graph model based on the custom operator and the visual part; determining the second static graph model based on the custom operator and the language part.

[0058] Optionally, the custom operator can be an operator created based on the interface provided by the target framework, such as a PaddleNLP custom operator.

[0059] According to the above embodiment, the first multimodal large model provides high-level information such as model structure and parameters, and pre-compiles and installs custom operators to perform underlying hardware adaptation to prepare the environment for static graph export, thereby ensuring the effect of static resource allocation and optimization of computing processes, and improving deployment efficiency and reasoning performance.

[0060] In some embodiments, the custom operator is further used to convert parameters in the visual part and / or the language part according to the graphics card precision of the target hardware.

[0061] Exemplarily, if the target hardware includes a graphics card that does not support BF16 precision, the compiled and installed custom operator can convert the parameters (including weights and / or activations, etc.) of BF16 precision in the visual part and / or language part of the first multimodal large model into FP16 precision.

[0062] According to the above embodiment, the custom operator makes judgments on different models of graphics cards, so that the operator can be applicable to more models of graphics cards, improves compatibility with different hardware, and thus improves the flexibility of model deployment.

[0063] In some embodiments, the custom operator includes a save function related operator; the save function related operator is used to add different identification information to the intermediate output results of different task flows.

[0064] Exemplarily, the save function related operators may include a save_output operator.

[0065] Optionally, based on the saving function-related operators, when performing static graph reasoning, the intermediate output results of different task flows will be associated with different identification information. When the intermediate calculation results are saved to disk, the intermediate calculation results will overwrite the original intermediate calculation results with the same identification information on the disk, but will not overwrite the intermediate calculation results with different identification information.

[0066] In actual applications, during static graph reasoning, the model needs to save the intermediate calculation results on disk, which may cause the model intermediate results to be overwritten in high-concurrency scenarios, making the final output of the model incorrect. According to the embodiment of the present disclosure, identification information is added to distinguish the results of different task flows, which effectively avoids the problem of overwriting the model intermediate results and ensures that the model can still output accurate and stable results in high-concurrency scenarios.

[0067] In some embodiments, the first multimodal large model is deployed based on the first static graph model and the second static graph model, including: quantizing the second static graph model to obtain a third static graph model; obtaining a prediction program based on the first static graph model and the third static graph model; wherein the prediction program is used to deploy the first multimodal large model on the target hardware for reasoning.

[0068] Exemplarily, after exporting the static graph model, the language part can be quantized before generating the prediction program (prediction code). Optionally, the prediction program can be generated based on the model development kit provided by the target framework. For example, it can be generated using PaddleMIX, a multimodal large model development kit provided by the Paddle framework.

[0069] According to the above embodiment, a quantization strategy can be implemented on the language part of the first multimodal large model, so as to further optimize the performance of the model during reasoning, reduce the memory usage and improve the reasoning speed.

[0070] In some embodiments, quantizing the second static graph model to obtain the third static graph model includes: quantizing weight information of the second static graph model to obtain the third static graph model.

[0071] Exemplarily, the second static graph model may be quantized by WINT8 quantization (quantizing weights only). Compared with the W8A8 quantization method, WINT8 quantization has obvious advantages in maintaining model performance. Although W8A8 quantization can achieve lower video memory occupancy, it may sacrifice certain model accuracy in some cases, thereby affecting the quality of the inference results. WINT8 quantization can achieve dual optimization of video memory occupancy and inference speed while maintaining high model accuracy.

[0072] Therefore, according to the above embodiment, dual optimization of video memory occupancy and inference speed can be achieved.

[0073] In some embodiments, the deployment method of the multimodal large model also includes: based on the prediction program, creating a KV (key-value) cache when the first multimodal large model is loaded for the first time; wherein the KV cache is used to store pre-calculated KV information for calling when the first multimodal large model performs inference calculations.

[0074] Specifically, by setting relevant codes in the prediction program, after the prediction program is configured on the target hardware, a KV cache is created when the first multimodal large model is first loaded on the target hardware.

[0075] In the related art, there are some convenient prediction program development tools based on static graph reasoning, which provide strong support for developers to deploy and test multimodal large models in practical applications. However, the provided prediction program has a potential performance bottleneck. Each time a calculation is performed, the program generates a temporary KV cache tensor. This approach increases the video memory usage during reasoning, which is obviously a problem that cannot be ignored for application scenarios with limited resources or sensitive to delays. In order to solve this problem, the embodiment of the present disclosure has carried out in-depth optimization of the prediction code. Specifically, by setting the required KV cache to be created when the model is loaded for the first time, only the KV cache created for the first time needs to be passed in for subsequent calculations, avoiding the need to regenerate a new KV cache for each calculation, thereby reducing the video memory usage during model reasoning.

[0076] Figure 3: is a schematic diagram of another application example of the deployment method of a multimodal large model according to an embodiment of the present disclosure. In this application example, high-performance multimodal model deployment is implemented based on the Paddle framework, which can optimize user experience and meet the needs of specific application scenarios. Specifically, the method includes:

[0077] S310. For models trained using PyTorch, a model conversion tool is provided to convert the model weights of the PyTorch version into the model weights of the Paddle version.

[0078] S320. Determine the Paddle inference model (converted based on step S310 or directly obtained through Paddle).

[0079] S330. Compile and install the improved operator to prepare the environment.

[0080] S340, convert the dynamic graph model to a static graph model. The static graph reasoning method of the Paddle framework can improve the reasoning speed because it completes the optimization and compilation of the computational graph before model reasoning, thereby reducing the computational overhead during the reasoning process. This conversion step is the core of achieving high-performance model deployment. In order to further improve the reasoning efficiency and performance of the model, this application example also performs WINT8 quantization on the static graph model.

[0081] S350: Generate a prediction program, and optimize Paddle's calculation logic during inference in the prediction program, further reducing the model's video memory usage.

[0082] S360, the model is deployed on the target hardware, and predictive reasoning is performed based on the input data to obtain the prediction results.

[0083] The static graph reasoning method provided by the Paddle framework has significant advantages over the dynamic reasoning method of PyTorch, especially in terms of reasoning speed. This feature is particularly important for reducing user waiting time and improving system response efficiency, especially in scenarios such as health questions and answers that require instant feedback. Therefore, the above application examples can be applied to model deployment in such scenarios, which can significantly improve user experience.

[0084] It can be seen that the embodiments of the present disclosure are suitable for the quantitative deployment of multimodal models in high-performance scenarios, and have significant advantages in many aspects. First, in terms of reasoning speed, the embodiments of the present disclosure can significantly reduce the time required for reasoning. This feature makes it particularly outstanding in scenarios with low latency requirements, such as real-time application scenarios such as health questions and answers, which can provide users with a smoother and more immediate experience. Secondly, the embodiments of the present disclosure can also significantly reduce the video memory occupancy of the model. In the multimodal model reasoning process, video memory occupancy is a key limiting factor. By optimizing video memory management, the embodiments of the present disclosure allow more computing resources to be used for actual reasoning tasks rather than being limited by video memory occupancy, thereby improving reasoning efficiency. Finally, the embodiments of the present disclosure have wide applicability and can support more types of multimodal models, while most of the existing technical solutions support reasoning acceleration of the language model part, which makes the embodiments of the present disclosure more flexible and versatile in the field of multimodal model deployment.

[0085] According to an embodiment of the present disclosure, the present disclosure also provides a deployment device for a multi-modal large model, Figure 4 A schematic block diagram of a deployment device for a multimodal large model provided by an embodiment of the present disclosure is shown. Figure 4 As shown, the device comprises:

[0086] A model splitting module 410, for splitting the first multimodal large model into a visual part and a language part;

[0087] A static graph export module 420, configured to determine a first static graph model corresponding to the visual part and a second static graph model corresponding to the language part;

[0088] The deployment module 430 is used to deploy the first multimodal large model based on the first static graph model and the second static graph model.

[0089] In some embodiments, Figure 5 As shown, the device also includes:

[0090] A model acquisition module 510, used to acquire a second multimodal large model to be deployed;

[0091] The model conversion module 520 is used to process the weight information in the second multimodal large model based on the weight conversion rule between the training framework of the second multimodal large model and the target framework to obtain the first multimodal large model when the training framework of the second multimodal large model is different from the target framework.

[0092] In some embodiments, the static image export module 420 is used to:

[0093] Compile and install custom operators for hardware adaptation of the model;

[0094] Determine the first static graph model based on the custom operator and the visual part;

[0095] Based on the custom operator and the language part, the second static graph model is determined.

[0096] In some embodiments, the custom operator is further used to convert parameters in the visual part and / or the language part according to the graphics card accuracy of the target hardware.

[0097] In some embodiments, the custom operator includes a save function related operator; the save function related operator is used to add different identification information to the intermediate output results of different task flows.

[0098] In some embodiments, the deployment module 430 is used to:

[0099] quantizing the second static graph model to obtain a third static graph model;

[0100] Based on the first static graph model and the third static graph model, a prediction program is obtained; wherein the prediction program is used to deploy the first multimodal large model on target hardware for reasoning.

[0101] In some embodiments, the deployment module 430 is used to:

[0102] The weight information of the second static graph model is quantized to obtain the third static graph model.

[0103] In some embodiments, Figure 5 As shown, the device also includes:

[0104] The cache creation module 530 is used to create a key-value KV cache based on the prediction program when the first multimodal large model is loaded for the first time; wherein the KV cache is used to store pre-calculated KV information for calling when the first multimodal large model performs inference calculations.

[0105] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0106] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0107] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0108] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0109] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0110] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0111] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the deployment method of the multimodal large model. For example, in some embodiments, the deployment method of the multimodal large model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the deployment method of the multimodal large model described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the method for deploying a multimodal large model in any other appropriate manner (eg, by means of firmware).

[0112] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0114] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0115] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0116] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0117] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0118] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0119] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for deploying a multimodal large model, comprising: Split the first multimodal large model into visual and language parts; Determine a first static graphic model corresponding to the visual part and a second static graphic model corresponding to the language part; Based on the first static graph model and the second static graph model, the first multimodal large model is deployed.

2. The method according to claim 1, further comprising: Obtaining a second multimodal large model to be deployed; When the training framework and the target framework of the second multimodal large model are different, the weight information in the second multimodal large model is processed based on the weight conversion rule between the training framework and the target framework of the second multimodal large model to obtain the first multimodal large model.

3. The method according to claim 1 or 2, wherein: The determining the first static graphic model corresponding to the visual part and the second static graphic model corresponding to the language part includes: Compile and install custom operators for hardware adaptation of the model; Determine the first static graph model based on the custom operator and the visual part; Based on the custom operator and the language part, the second static graph model is determined.

4. The method according to claim 3, wherein: The custom operator is also used to convert parameters in the visual part and / or the language part according to the graphics card accuracy of the target hardware.

5. The method according to claim 3 or 4, wherein: The custom operator includes a save function related operator; the save function related operator is used to add different identification information to the intermediate output results of different task flows.

6. The method according to any one of claims 1 to 5, wherein: The deploying the first multimodal large model based on the first static graph model and the second static graph model includes: quantizing the second static graph model to obtain a third static graph model; Based on the first static graph model and the third static graph model, a prediction program is obtained; wherein the prediction program is used to deploy the first multimodal large model on target hardware for reasoning.

7. The method according to claim 6, wherein: The step of quantizing the second static graph model to obtain a third static graph model includes: The weight information of the second static graph model is quantized to obtain the third static graph model.

8. The method according to claim 6 or 7, further comprising: Based on the prediction program, when the first multimodal large model is loaded for the first time, a key-value KV cache is created; wherein the KV cache is used to store pre-calculated KV information for calling when the first multimodal large model performs inference calculations.

9. A deployment device for a multimodal large model, comprising: A model splitting module, used for splitting the first multimodal large model into a visual part and a language part; A static image export module, used to determine a first static image model corresponding to the visual part and a second static image model corresponding to the language part; A deployment module is used to deploy the first multimodal large model based on the first static graph model and the second static graph model.

10. The apparatus according to claim 9, further comprising: A model acquisition module, used to acquire a second multimodal large model to be deployed; A model conversion module is used to process the weight information in the second multimodal large model based on the weight conversion rule between the training framework of the second multimodal large model and the target framework to obtain the first multimodal large model when the training framework of the second multimodal large model is different from the target framework.

11. The device according to claim 9 or 10, wherein: The static image export module is used for: Compile and install custom operators for hardware adaptation of the model; Determine the first static graph model based on the custom operator and the visual part; Based on the custom operator and the language part, the second static graph model is determined.

12. The device according to claim 11, wherein The custom operator is also used to convert parameters in the visual part and / or the language part according to the graphics card accuracy of the target hardware.

13. The device according to claim 11 or 12, wherein: The custom operator includes a save function related operator; the save function related operator is used to add different identification information to the intermediate output results of different task flows.

14. The device according to any one of claims 9 to 13, wherein: The deployment module is used to: quantizing the second static graph model to obtain a third static graph model; Based on the first static graph model and the third static graph model, a prediction program is obtained; wherein the prediction program is used to deploy the first multimodal large model on target hardware for reasoning.

15. The device according to claim 14, wherein: The deployment module is used to: The weight information of the second static graph model is quantized to obtain the third static graph model.

16. The device according to claim 15 or 16, further comprising: A cache creation module is used to create a key-value KV cache based on the prediction program when the first multimodal large model is loaded for the first time; wherein the KV cache is used to store pre-calculated KV information for calling when the first multimodal large model performs inference calculations.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

19. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images