Large model deployment method, device and equipment for heterogeneous platform and program product

Abstract

The invention discloses a large model deployment method and device for a heterogeneous platform, equipment and a program product, and relates to the technical field of artificial intelligence. A large model to be deployed is divided into a plurality of sub-models, and according to the operator type and the operator calculation amount contained in each sub-model, the operator support conditions of various types of chips on the heterogeneous platform are combined, so that the large model deployment efficiency is improved. The method comprises the following steps: determining a target chip to be deployed by each sub-model, determining a quantization strategy of each sub-model according to a quantization support condition of the target chip, quantifying the sub-model according to the quantization strategy, and generating a deployable sub-model from the quantized sub-model and the quantization strategy so as to be deployed into a corresponding target chip on a heterogeneous platform. And automatic deployment of a large model is realized. According to the method, the target chip to be deployed by the sub-model can be automatically selected in the large model deployment process, the quantization strategy can be automatically selected, and compared with a method of manually setting a deployment scheme, the deployment efficiency can be greatly improved, and the end-side deployment difficulty of the large model can be reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a large model deployment method, device, equipment and program product for a heterogeneous platform. Background Art

[0002] With the development of big model technology, we have entered the stage of big model application. One big model implementation solution is to deploy and reason about the big model on the end-side device.

[0003] Existing large models are usually deployed on chips on heterogeneous platforms. Heterogeneous platforms contain multiple different types of heterogeneous chips, which can use the computing power of heterogeneous chips to complete the end-side deployment and subsequent reasoning of large models. The current end-side deployment solution for large models generally relies on manual settings, that is, manual analysis of various types of chips on heterogeneous platforms, and customized deployment of different models on different chips. Obviously, this solution requires high professional requirements for staff and has low work efficiency. Summary of the invention

[0004] In view of the above problems, this application is proposed to provide a large model deployment method, device, equipment and program product for a heterogeneous platform to achieve automated large model deployment, reduce dependence on manual labor, and improve deployment efficiency. The specific solution is as follows:

[0005] In a first aspect, a large model deployment method for a heterogeneous platform is provided, comprising:

[0006] Split the large model to be deployed into several sub-models;

[0007] Determine the target chip to be deployed for each sub-model according to the operator type and the computational amount of the operator included in each sub-model and the operator support of various types of chips on the heterogeneous platform;

[0008] Determine the quantization strategy for each sub-model based on the quantization support of the target chip where each sub-model is to be deployed;

[0009] Quantize the sub-models according to the quantization strategy of each sub-model to obtain quantized sub-models;

[0010] The quantized sub-model and the corresponding quantization strategy are used to generate a deployable sub-model so as to be deployed to the corresponding target chip on the heterogeneous platform.

[0011] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the large model to be deployed is a hybrid expert large model, and the process of splitting the large model to be deployed into several sub-models includes:

[0012] Analyze the hybrid expert macromodel to obtain the operators contained in the macromodel and the structural relationships between the operators;

[0013] According to the structural relationship between operators, searching starts from the input of the hybrid expert large model to identify the target gating operator, and there are more than two feedforward neural networks behind the target gating operator;

[0014] Splitting the hybrid expert large model from input to the target gating operator into head sub-models;

[0015] Continue to search backward and identify that the outputs of more than two feedforward networks are concentrated on the same weight summation operator. Perform splitting before the weight summation operator, and use each path from the weight summation operator to the target gating operator as an expert sub-model. Split the weight summation operator and the subsequent part into tail sub-models.

[0016] In a possible design, in another implementation of the first aspect of the embodiment of the present application, according to the operator type and the calculation amount of the operator included in each sub-model, combined with the operator support of various types of chips on the heterogeneous platform, the process of determining the target chip to be deployed for each sub-model includes:

[0017] For each of the sub-models:

[0018] According to the operator types included in the sub-model and in combination with the operator support of various types of chips on the heterogeneous platform, determine each candidate chip that can support the sub-model;

[0019] Determining the computation amount of the sub-model according to the computation amount of the operators included in the sub-model;

[0020] If the computation amount of the sub-model is greater than the set computation amount threshold, combining the candidate chips that can support the sub-model, and selecting the target chip to be deployed by the sub-model from the candidate chips according to the first chip selection order;

[0021] If the computational complexity of the sub-model is not greater than the set computational complexity threshold, then in combination with the candidate chips that can support the sub-model, the target chip to be deployed by the sub-model is selected from the candidate chips in a second chip selection order, and the first chip selection order is different from the second chip selection order.

[0022] In one possible design, in another implementation of the first aspect of the embodiment of the present application, the first chip selection order and the second chip selection order each include a central processing unit CPU, a graphics processing unit GPU and a neural network processor NPU. In the first chip selection order, the order from high to low priority is NPU, GPU, CPU; in the second chip selection order, the order from high to low priority is CPU, GPU, NPU.

[0023] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of determining the quantization strategy of each sub-model according to the quantization support of the target chip to be deployed by each sub-model includes:

[0024] Determine a set of candidate quantization strategies for each operator in the sub-model based on the quantization support of each operator by the target chip to be deployed in each sub-model;

[0025] A target quantization strategy is selected from the candidate quantization strategy set of each operator, and the quantization strategy of the sub-model is composed of the target quantization strategies of each operator.

[0026] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the process of selecting a target quantization strategy from a set of candidate quantization strategies for each operator includes:

[0027] In response to an operation of selecting a target quantization mode from a configured quantization mode list, selecting a target quantization strategy matching the target quantization mode from a set of candidate quantization strategies for each operator according to a quantization strategy selection rule corresponding to the target quantization mode;

[0028] The quantization mode list includes more than two quantization modes, and different quantization modes represent different degrees of accuracy reduction caused by quantization, and different model volumes and calculation amounts after quantization.

[0029] In a possible design, in another implementation manner of the first aspect of the embodiments of the present application, the quantization mode list includes at least three quantization modes: a precision mode, a performance mode, and a balance mode;

[0030] The precision mode characterization and the original floating point model are compared, and the precision reduction caused by quantization does not exceed a first ratio;

[0031] The performance model characterizes that the accuracy reduction caused by quantization compared with the original floating-point model does not exceed a second ratio and the volume and amount of calculation of the quantized model are reduced to the greatest extent;

[0032] The balance mode characterization reduces the accuracy caused by quantization by no more than a third ratio compared to the original floating-point model and reduces the volume and computational complexity of the quantized model to the greatest extent;

[0033] The first ratio is smaller than the third ratio, and the third ratio is smaller than the second ratio.

[0034] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the process of generating a deployable sub-model from the quantized sub-model and the corresponding quantization strategy includes:

[0035] The quantized sub-model and the corresponding quantization strategy are used to generate a sub-model in a set intermediate format, where the intermediate format is a set format different from the format supported by various types of chips.

[0036] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the method further includes:

[0037] When the model is deployed, the format of each sub-model is converted into a format supported by the target chip to which the sub-model is to be deployed, and deployed to the corresponding target chip on the heterogeneous platform.

[0038] In a possible design, in another implementation of the first aspect of the embodiment of the present application, after deploying the sub-model on the target chip, the method further includes:

[0039] Counting the occupancy of each chip on the heterogeneous platform under the current inference allocation;

[0040] According to the occupancy of each chip under the current inference allocation and the configured performance balancing strategy, the sub-models deployed on each chip are adjusted.

[0041] In a second aspect, a large model deployment device for a heterogeneous platform is provided, comprising:

[0042] A model splitting unit is used to split the large model to be deployed into several sub-models;

[0043] A chip selection unit, used to determine the target chip to be deployed for each sub-model according to the operator type and the calculation amount of the operator contained in each sub-model, combined with the operator support of various types of chips on the heterogeneous platform;

[0044] A quantization strategy determination unit, used to determine the quantization strategy of each sub-model according to the quantization support of the target chip to be deployed by each sub-model;

[0045] A model quantization unit, used for quantizing the sub-model according to the quantization strategy of each sub-model to obtain a quantized sub-model;

[0046] The deployable model generation unit is used to generate a deployable sub-model from the quantized sub-model and the corresponding quantization strategy, so as to facilitate deployment to the corresponding target chip on the heterogeneous platform.

[0047] In a third aspect, an electronic device is provided, comprising: a memory and a processor;

[0048] The memory is used to store programs;

[0049] The processor is used to execute the program to implement the large model deployment method for a heterogeneous platform as described in any one of the first aspects of the present application.

[0050] In a fourth aspect, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the large model deployment method for a heterogeneous platform described in any one of the first aspects of the present application is implemented.

[0051] In a fifth aspect, a computer program product is provided, including a computer program, which, when executed by a processor, implements the large model deployment method for a heterogeneous platform as described in any one of the first aspects of the present application.

[0052] By means of the above technical scheme, the present application provides an automated deployment scheme for the landing end side of a large model, splitting the large model to be deployed into several sub-models, and determining the target chip to be deployed for each sub-model according to the operator type and operator calculation amount contained in each sub-model, combined with the operator support of various types of chips on the heterogeneous platform, and determining the quantization strategy of each sub-model according to the quantization support of the target chip, quantizing the sub-model according to the quantization strategy to obtain the quantized sub-model, and generating a deployable sub-model with the quantized sub-model and the quantization strategy, so as to facilitate deployment to the corresponding target chip on the heterogeneous platform, and realize the automated deployment of the large model. The present application can automatically select the target chip to be deployed for the sub-model in the automated deployment process of the large model, and automatically select the quantization strategy. Compared with the method of manually setting the deployment plan, it has lower dependence on manual labor, can greatly improve the deployment efficiency, and reduce the difficulty of the end-side deployment of the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0054] Figure 1 A schematic diagram of a system architecture for implementing a large model deployment method for a heterogeneous platform provided in an embodiment of the present application;

[0055] Figure 2 A schematic diagram of a terminal structure provided in an embodiment of the present application;

[0056] Figure 3 A schematic diagram of a server structure provided in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of a process of deploying a large model on a heterogeneous platform provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of splitting a large MOE model provided in an embodiment of the present application;

[0059] Figure 6 A schematic diagram of a chip selector processing method provided in an embodiment of the present application;

[0060] Figure 7 A schematic diagram of a process of processing a large model by a server in an offline state to generate a deployable sub-model provided in an embodiment of the present application;

[0061] Figure 8 A schematic diagram of a process of deploying and reasoning a sub-model through a terminal in an online state provided in an embodiment of the present application;

[0062] Fig. 9 A schematic diagram of the structure of a large model deployment device for a heterogeneous platform provided in an embodiment of the present application;

[0063] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] Before introducing this application solution, the relevant concepts involved in this article are first explained:

[0065] Large models: In the field of artificial intelligence, large models usually refer to large-scale pre-trained models. Such models are called "large" because they can be pre-trained on a large amount of data and can be transferred to a variety of downstream tasks. Their full English name is Large Pre-Trained Models or Large-Scale Pre-Training Models. Large models are characterized by their large scale and contain billions or even more parameters, which help them learn complex patterns in the data. The emerging capabilities of large models include but are not limited to: contextual learning, instruction following, code generation, step-by-step reasoning capabilities, etc. Large models can include large language models (LLMs) and multimodal large models. Large language models are mainly used to process text modal data. Multimodal large models further integrate multimodal capabilities on the basis of large language models and can process information in multiple modalities, such as images, text, audio, etc.

[0066] MOE big model: The full name is Mixture of Experts, which is a big model architecture used to improve the performance and efficiency of deep learning models. Its core idea is to introduce multiple expert models, and only select and activate a part of the expert models for processing for each input data, thereby reducing the amount of calculation and improving the training and reasoning speed. Common MOE big models include GPT-4.

[0067] CPU: The full name is Central Processing Unit. The CPU is the core component of the computer system, responsible for executing program instructions, processing data, and controlling the operation of the computer. It is composed of components such as the arithmetic unit, controller, register group, and bus interface, and is the key to the computer's ability to complete various complex tasks.

[0068] GPU: The full name is Graphics Processing Unit. GPU is a microprocessor specially used for image processing. It has powerful parallel processing capabilities and can efficiently complete complex graphics and image processing tasks.

[0069] NPU: The full name is Neural Processing Unit. NPU is a new type of processor designed for deep learning and neural network computing. It can efficiently execute neural network algorithms and accelerate the calculation process of deep learning tasks. With the continuous development of artificial intelligence technology, NPU plays an increasingly important role in image recognition, speech recognition, natural language processing and other fields.

[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0071] This application provides a large model deployment method for a heterogeneous platform, which can be applied to Figure 1 The system architecture shown in FIG. 1 may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 A server is included as an example for explanation).

[0072] Either the terminal 100 or the server 200 can be used alone to execute the large model deployment method for the heterogeneous platform provided in the embodiment of the present application. In addition, the terminal 100 and the server 200 can also be used in collaboration to execute the large model deployment method for the heterogeneous platform provided in the embodiment of the present application. The large model deployment method for the heterogeneous platform of the present application can be divided into large model splitting, chip selection, quantization strategy selection, model quantization processing to generate deployable model files in an offline state, and deployment and reasoning of model files on the terminal side in an online state. In an optional implementation, each method step in an offline state can be implemented on the server 200, and each method step in an online state can be implemented on the terminal 100 (the terminal 100 is used as a heterogeneous platform).

[0073] Next describe Figure 1 The product form of the mid-terminal 100;

[0074] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a learning machine, a wearable device, a vehicle-mounted device, a conference terminal, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.

[0075] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0076] refer to Figure 2As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160, a speaker 161, a microphone 162, an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0077] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0078] Among them, other input devices 132 can receive input data and so on.

[0079] The display unit 140 can be used to display information input by the user or information provided to the user, various menus of the terminal 100, interactive interfaces, file display and / or playback of any multimedia file. In the embodiment of the present application, the display unit 140 can be used to display various interactive interfaces, processing results, etc. in the large model deployment method of the heterogeneous platform.

[0080] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0081] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0082] Among them, the memory 120 can be used to store software codes related to the large model deployment method of the heterogeneous platform, the processor 170 can execute the steps of the large model deployment method of the heterogeneous platform, and can also schedule other units (such as the above-mentioned input unit 130 and the display unit 140) to implement corresponding functions.

[0083] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0084] In the embodiment of the present application, the RF unit 110 can send data to the server 200 and receive the processing result sent by the server 200. Exemplarily, the RF unit 110 sends the large model download instruction to the server 200, and the server 200 executes the large model deployment method of the heterogeneous platform to obtain the sub-model to be deployed, and sends it to the terminal 100, and the terminal 100 deploys the sub-model to the local target chip and performs subsequent reasoning tasks.

[0085] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0086] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0087] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0088] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0089] Next describe Figure 1 The product form of the server 200;

[0090] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0091] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0092] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0093] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0094] Among them, the memory 204 can be used to store software codes related to the large model deployment method of the heterogeneous platform, the processor 202 can execute the steps of the large model deployment method of the heterogeneous platform, and can also schedule other units to implement corresponding functions.

[0095] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0096] The embodiment of the present application provides a large model deployment method for a heterogeneous platform, taking the method including an offline implementation part and an online implementation part as an example, wherein the offline implementation part can be applied to Figure 1 The server 200 in the online implementation part can be applied to Figure 1 Next, refer to the terminal 100. Figure 4 The large model deployment method of the heterogeneous platform specifically includes the following steps:

[0097] Step S100: split the large model to be deployed into several sub-models.

[0098] Specifically, in order to fully utilize the computing power of various types of chips on a heterogeneous platform, when deploying a large model to the end side of a heterogeneous platform, it can be split into several sub-models to be deployed on different chips respectively.

[0099] In this step, the large model to be deployed can be various types of large models. For example, it can be a hybrid expert MOE large model. The MOE large model can be split into a head sub-model, several expert sub-models in the middle part, and a tail sub-model.

[0100] Of course, the above is only an optional way to split a large model. In addition, other optional splitting strategies can be used to split the large model into several sub-models to facilitate subsequent deployment on different chips.

[0101] Each sub-model after the large model is split can contain more than one operator. Usually, each sub-model contains multiple operators. Operators are elements that perform specific mathematical operations or transformations, which define how data is transformed from one state to another. Operators can be divided into many types, including but not limited to: linear operators, such as fully connected operators and convolution operators; non-linear operators, such as activation functions Relu, Sigmoid, etc.; pooling operators, such as max pooling, average pooling, etc.; normalization operators, other special operators, etc.

[0102] Step S110: Determine the target chip to be deployed for each sub-model according to the operator type and the computational complexity of each sub-model and the operator support of various types of chips on the heterogeneous platform.

[0103] Specifically, after splitting the large model into several sub-models, each sub-model contains more than one operator. The type of each operator in each sub-model and the amount of calculation of the operator can be determined. The amount of calculation of the operator can be calculated based on the input and output dimensions of the operator, the number of operations (such as the number of addition, subtraction, multiplication, and division operations), etc.

[0104] Different types of chips on heterogeneous platforms may support different operators. The operator types supported by the chip can be determined based on the list of supported chips pre-equipped in each chip. The operator types supported by the chip can be understood as the operator types that the chip can process.

[0105] Furthermore, different types of chips have different computing power. When determining the target chip to deploy the sub-model, the computing power of the operators in the sub-model can be considered. For example, for a sub-model with a large computing power, it can be preferentially allocated to a chip with a higher computing power.

[0106] In this step, according to the pre-configured allocation strategy, combined with the operator types and computational complexity of the sub-model, as well as the operator support of various types of chips on the heterogeneous platform, a corresponding target chip can be allocated to each sub-model as the chip to be deployed by the sub-model.

[0107] Step S120: Determine the quantization strategy of each sub-model according to the quantization support of the target chip to be deployed by each sub-model.

[0108] Specifically, before deploying the sub-model to the target chip, the sub-model can be quantized to reduce the size and amount of calculation when the model is deployed.

[0109] Different types of chips may have different quantization support for different types of operators. For example, for a certain NPU chip, the channel-by-channel convolution depthwise_conv operator in the depthwise separable convolution can support fp16 operations and int16 operations, but does not support 8-bit operations. For CPU chips, all quantization schemes are generally supported.

[0110] After determining the target chip to be deployed for each sub-model in the previous step, the specific quantization strategy of each sub-model can be determined according to the quantization support of the target chip, that is, the quantization scheme adopted for each operator contained in the sub-model.

[0111] Step S130: quantize the sub-model according to the quantization strategy of each sub-model to obtain a quantized sub-model.

[0112] Specifically, after the quantization strategy of each sub-model is determined, the sub-model can be quantized according to the quantization strategy to obtain a quantized sub-model.

[0113] Step S140: Generate a deployable sub-model using the quantized sub-model and the corresponding quantization strategy, so as to facilitate deployment to the corresponding target chip on the heterogeneous platform.

[0114] Specifically, a compressed sub-model that can be deployed on a target chip can be generated based on the quantized sub-model and the corresponding quantization strategy, so that the sub-model can be deployed to the corresponding target chip on a heterogeneous platform during the model deployment phase.

[0115] In one possible implementation, when generating a deployable sub-model, a sub-model file in the format supported by a certain type of chip (such as a CPU chip) can be generated according to the model format supported by the default model format. In the subsequent model deployment stage, if the format supported by the target chip to which the sub-model is to be deployed is exactly the format of the current sub-model, it can be deployed directly; if the format supported by the target chip to which the sub-model is to be deployed is not the format of the current sub-model, the format of the sub-model needs to be converted to the format supported by the target chip. For example, when generating a deployable sub-model, a format supported by the CPU can be generated. If the sub-model needs to be deployed on the NPU, the format of the sub-model needs to be converted to a format supported by the NPU before deployment.

[0116] In another possible implementation, the present application may design an intermediate format that is different from the model formats supported by various types of chips, and the model in the intermediate format may be simply and quickly converted to the format supported by the target chip when loaded into the memory. The present application will not further describe the specific format content of the intermediate format, and those skilled in the art may design the specific format content according to business needs.

[0117] On this basis, in this step, the quantized sub-model and the corresponding quantization strategy are used to generate a sub-model in a set intermediate format.

[0118] Furthermore, when the model is subsequently deployed, the format of each sub-model can be converted into a format supported by the target chip on which the sub-model is to be deployed, and deployed to the corresponding target chip on the heterogeneous platform.

[0119] The embodiment of the present application provides an automated deployment method for a large model on the terminal side. This method can automatically select the target chip on which the sub-model is to be deployed during the automated deployment of the large model, and automatically select the quantization strategy, so that the deployment of the large model on the terminal side device becomes more streamlined and automated, which greatly reduces the customized manpower and cost investment in the deployment of the large model on the terminal side device, and improves the deployment efficiency.

[0120] Furthermore, the sub-models of the same structure in the intermediate format generated by the embodiment of the application can complete the optimal structure conversion and deployment to the target chip in a relatively short time, ensuring the best reasoning structure support on different chips. The same format model can support reasoning on different chips.

[0121] In some possible implementations, taking the large model to be deployed as a hybrid expert MOE large model as an example, the process of splitting the large model to be deployed into several sub-models in the aforementioned step S100 is described, and the process may include the following steps:

[0122] S1. Analyze the hybrid expert large model to obtain the operators contained in the large model and the structural relationship between the operators.

[0123] S2. According to the structural relationship between operators, the search starts from the input of the mixed expert large model to identify the target gating operator. There are more than two feedforward networks FFN behind the target gating operator.

[0124] Based on the characteristics of the MOE large model, in the MOE architecture, the routing module Router is a key component responsible for assigning input data to different expert sub-models for processing. The input data is first passed to the Router, which calculates the degree of response (i.e., score) of each expert sub-model to the input data based on its internal mapping mechanism (such as the linear layer and the softmax layer). Based on the calculated score, the Router selects the expert sub-model with the highest score (under the Top-1 strategy) or multiple sub-models (under the Top-k strategy) to process the input data. If the Top-k strategy is adopted, the processing results of multiple sub-models will be aggregated in a certain way (such as weighted average) to form the final output result. Specifically, the output results of multiple expert sub-models are effectively aggregated through the weight sum operator weight sum.

[0125] Router includes gating operators, such as soft-gated softmax operator, hard-gated argmax operator, TopK gated softmax+top-k operator, etc.

[0126] In this step, according to the structural relationship between the operators obtained by analysis, the search starts from the input of the MOE large model to identify the target gating operator. There are more than two feedforward networks FFN after the target gating operator, indicating that the target gating operator belongs to the expert sub-model. Therefore, the target gating operator and the subsequent FFN can be used as the splitting position to obtain the head sub-model.

[0127] S3. Divide the hybrid expert large model from the input to the target gating operator part into the head sub-model.

[0128] like Figure 5 As shown in the figure, it illustrates the structure of the MOE large model, from left to right, it is the input to output direction of the model. After identifying the target gating operator Router, it can be split between Router and FFN to obtain the head sub-model.

[0129] S4. Continue to search backward and identify that the outputs of more than two feedforward neural networks FFN are concentrated on the same weight sum operator weight sum. Split before weight sum, and use each path from weight sum to the target gating operator Router as an expert sub-model, and split weight sum and the part after it into tail tail sub-models.

[0130] like Figure 5As shown in FIG. 1 , a partition is performed between FFN and weight sum, and each path from weight sum to the target gating operator Router is used as an expert sub-model, namely FFN1, FFN2…FFNE, with a total of E expert sub-models.

[0131] After the above steps, we can obtain the split head sub-model, several expert sub-models and the tail sub-model.

[0132] In some possible implementations, the implementation process of step S100 may be specifically implemented by a preconfigured model segmenter.

[0133] In some possible implementations, the process of determining the target chip to be deployed for each sub-model in the aforementioned step S110 is described based on the operator types and operator computational complexity included in each sub-model and combined with the operator support of various types of chips on the heterogeneous platform.

[0134] For each sub-model obtained after the splitting in step S100, the target chip to be deployed for the sub-model may be determined according to the following steps, which specifically include the following steps:

[0135] S1. According to the operator types included in the sub-model and the operator support of various types of chips on the heterogeneous platform, determine the candidate chips that can support the sub-model.

[0136] Specifically, different types of chips may have different support for operators. You can obtain the operator support list of each type of chip configured, which includes the types of operators supported by the current type of chip. On this basis, according to the types of operators included in the sub-model, analyze and determine the candidate chips on the heterogeneous platform that can support each operator in the sub-model, and the candidate chips include at least one type.

[0137] S2. Determine the computational complexity of the sub-model according to the computational complexity of the operators included in the sub-model.

[0138] Specifically, after the computational effort of each operator in the sub-model is determined, the computational effort of the sub-model may be obtained by comprehensively calculating the sum of the computational effort of each operator in the sub-model.

[0139] S3. If the computational complexity of the sub-model is greater than the set computational complexity threshold, then in combination with the candidate chips that can support the sub-model, a target chip to be deployed for the sub-model is selected from the candidate chips according to the first chip selection order.

[0140] Specifically, different types of chips have different computing power and process different tasks. For sub-models with larger computing power, they can be preferentially assigned to chips with stronger computing power.

[0141] In this embodiment, a computational amount threshold can be pre-set to measure whether the computational amount of the sub-model is large enough. If it is determined that the computational amount of the current sub-model is greater than the computational amount threshold, the target chip to be deployed for the sub-model can be selected from the candidate chips that can support the sub-model according to the configured first chip selection order.

[0142] S4. If the computational complexity of the sub-model is not greater than the set computational complexity threshold, then combining the candidate chips that can support the sub-model, select a target chip for the sub-model to be deployed from the candidate chips according to the second chip selection order.

[0143] The first chip selection sequence is different from the second chip selection sequence.

[0144] The first chip selection order and the second chip selection order can be preset by the staff. When setting the chip selection order, factors such as the computing power of different types of chips and the tasks to be processed can be considered.

[0145] In an optional example, taking three types of chips, CPU, GPU and NPU, as examples, an optional setting method of the first chip selection order and the second chip selection order is introduced, namely:

[0146] In the first chip selection sequence, the priority order from high to low is NPU, GPU, CPU; in the second chip selection sequence, the priority order from high to low is CPU, GPU, NPU.

[0147] The chip selection order in the above example is mainly based on the consideration that when the computational amount of the sub-model is large (greater than the set computational amount threshold), the NPU is more suitable for inference calculations of the neural network model and has stronger computing power, so the NPU can be effectively selected. If the NPU does not support the current sub-model, the GPU can be selected as a fallback. If the GPU also does not support the current sub-model, the CPU can be used.

[0148] On the contrary, when the computational complexity of the sub-model is small (not greater than the set computational complexity threshold), a CPU with smaller computing power can be used. If the CPU does not support the current sub-model, the GPU can be used. If the GPU does not support the current sub-model, the NPU can be used.

[0149] The above-mentioned embodiment of the present application illustrates a chip selection scheme, which can determine the target chip to be deployed for each sub-model based on the operator types and the computational amount of the operators included in the sub-model, combined with the operator support of each chip. While ensuring that the target chip allocated to each sub-model can normally infer and calculate the sub-model, other factors such as the computing power of the chip can also be comprehensively considered to achieve a more reasonable allocation of sub-models.

[0150] In some possible implementations, step S110 may be implemented by a preconfigured chip selector. Figure 6 As shown, the chip selector may include a computing power calculation module, an operator detection module, and a sorting module. Among them, the computing power calculation module is specifically used to calculate the computational amount of the sub-model. The operator detection module is specifically used to refer to the configured operator support list to determine whether each type of chip supports the current sub-model. The sorting module is specifically used to prioritize each type of chip according to the size of the computational amount of the sub-model and whether each type of chip supports the current sub-model, and select the target chip to be deployed for the sub-model in the sorting order.

[0151] Figure 6 Only three types of chips, NPU, GPU and CPU, are used as examples for explanation. When the computational workload of the sub-model is large, NPU is preferred; when the computational workload is medium, GPU is preferred; and when the computational workload is small, CPU is preferred.

[0152] In some possible implementations, the process of determining the quantization strategy of each sub-model according to the quantization support of the target chip to be deployed by each sub-model in the aforementioned step S120 is described. An optional implementation method may include the following steps:

[0153] S1. Determine a set of candidate quantization strategies for each operator in the sub-model according to the quantization support of each operator by the target chip to be deployed in each sub-model.

[0154] Specifically, the target chip may have different quantization support for different operators. For example, a certain NPU chip can support fp16 operations and int16 operations for the depthwise_conv operator, but does not support 8-bit operations. Therefore, in this step, the target chip's support for quantization of each operator is referenced to determine the candidate quantization strategies supported by the target chip for each operator in the sub-model, and the candidate quantization strategy set for each operator is obtained. Using the above example to illustrate, the candidate quantization strategy set for the depthwise_conv operator includes: fp16 operations, int16 operations.

[0155] S2. Select a target quantization strategy from the candidate quantization strategy set of each operator, and the target quantization strategies of each operator constitute the quantization strategy of the sub-model.

[0156] Specifically, since the candidate quantization strategy set of the operator includes the candidate quantization strategies of the operator supported by the chip, a target quantization strategy can be selected therefrom, and then the target quantization strategies of each operator constitute the quantization strategy of the sub-model.

[0157] The quantization strategy selection method provided in this embodiment can automatically select the quantization strategy of each operator in the sub-model, ensuring that the sub-model can be correctly quantized.

[0158] For the process of selecting a target quantization strategy from the candidate quantization strategy set of each operator in the above step S2, there may be multiple selection methods, such as randomly selecting one from the candidate quantization strategy set as the target quantization strategy, or adopting other selection rules.

[0159] This embodiment provides an optional selection method. Specifically:

[0160] Considering that in different application scenarios, users may have different requirements for model quantization, such as requirements for accuracy after quantization, volume of the model after quantization, and amount of calculation required. Therefore, the present application may be pre-configured with a quantization mode list, which includes more than two different quantization modes, and different quantization modes represent different degrees of accuracy reduction caused by quantization, and different model volume sizes and amounts of calculation after quantization.

[0161] In an optional implementation, the present application provides examples of three quantization modes, which are defined as precision mode, performance mode, and balance mode.

[0162] The precision mode characterization, compared with the original floating-point model, the precision reduction caused by quantization does not exceed a first ratio.

[0163] The performance model characterizes that, compared with the original floating-point model, the accuracy reduction caused by quantization does not exceed the second ratio and the volume and computational complexity of the quantized model are reduced to the greatest extent.

[0164] The balanced mode representation shows that compared with the original floating-point model, the accuracy reduction caused by quantization does not exceed the third ratio and the volume and calculation amount of the quantized model are reduced to the greatest extent.

[0165] The first ratio is smaller than the third ratio, and the third ratio is smaller than the second ratio. An optional value setting method is that the first ratio is 0.5%, the second ratio is 2%, and the third ratio is 1%.

[0166] On this basis, the present application can respond to the operation of selecting a target quantization mode from the configured quantization mode list, and select a target quantization strategy that matches the target quantization mode from the candidate quantization strategy set of each operator according to the quantization strategy selection rule corresponding to the target quantization mode.

[0167] Different quantization modes can be configured with corresponding quantization strategy selection rules, which can be set by the staff. Taking the above-mentioned precision mode as an example, its corresponding quantization strategy selection rule can be to preferentially select the quantization strategy with the smallest quantization loss from the candidate quantization strategy set of the operator.

[0168] The sub-model quantization strategy selection method provided in this embodiment can realize the automatic selection of the model quantization strategy. At the same time, it can support the user to select the desired quantization mode, and then automatically select the matching target quantization strategy according to the quantization mode.

[0169] In a possible implementation, steps S120-S140 may be implemented by a preconfigured quantization selector. The quantization selector may include a quantization operator detection module, a quantization information loss detection module, and a quantization model generation module.

[0170] The quantization operator detection module is used to determine a set of candidate quantization strategies for each operator in the sub-model according to the quantization support of each operator by the target chip to be deployed in each sub-model.

[0171] The quantization information loss detection module is used to, in response to the operation of selecting a target quantization mode from the configured quantization mode list, select a target quantization strategy matching the target quantization mode from a set of candidate quantization strategies for each operator according to a quantization strategy selection rule corresponding to the target quantization mode.

[0172] The quantization model generation module is used to quantize the sub-model according to the quantization strategy of each sub-model to obtain the quantized sub-model.

[0173] Combination Figure 7 Taking the offline deployment of the MOE large model as an example, it introduces the process of achieving offline deployment through model segmentation, chip selector and quantization selector.

[0174] For the large MOE model to be deployed, it is first divided into a head sub-model, a tail sub-model and several expert sub-models in the middle through a model splitter: FFN1 sub-model, FFN2 sub-model...

[0175] Furthermore, for each sub-model after division, the target chip to be deployed for each sub-model is selected through the chip selector, and the quantization strategy of each sub-model is selected through the quantization selector.

[0176] Optionally, in the process of selecting the quantization strategy of each sub-model through the quantization selector, the target chip to be deployed for each sub-model can be adjusted by returning to the chip selector according to the quantization strategy of the sub-model. For example, if the quantization selector finds that for a sub-model, the corresponding target chip does not support quantization of one or more operators in the sub-model, it can return to the chip selector and reassign the target chip to be deployed to the sub-model.

[0177] Finally, after the quantization selector determines the quantization strategy of each sub-model, the sub-model is quantized according to the quantization strategy to obtain the quantized sub-model, such as Figure 7 HdM.uni, F1M.uni, F2M.uni…TlM.uni as shown.

[0178] In some embodiments of the present application, after the sub-model is deployed to the corresponding target chip on the heterogeneous platform, a process of balancing the performance of each chip on the heterogeneous platform may be further added.

[0179] Specifically, the occupancy of each chip under the current inference allocation on the heterogeneous platform can be counted.

[0180] According to the occupancy of each chip under the current inference allocation and the configured performance balancing strategy, the sub-models deployed on each chip are adjusted.

[0181] The above-mentioned operation of balancing the performance of each chip on the heterogeneous platform can be achieved through a configured performance equalizer, which can count the occupancy of each chip under the current inference allocation in real time. Optionally, the occupancy can be obtained through the official performance occupancy tool of the chip, for example, the memory occupancy rate of the chip can be obtained through the performance occupancy tool analysis. In addition, another performance occupancy indicator Pi is provided in the embodiment of the present application:

[0182] Pi=S / L×100%

[0183] Among them, S represents the inference time, L represents the inference window length, and Pi represents the time required for the model to process text or data of unit length.

[0184] In some possible implementations, when calculating the occupancy of each chip under the current inference allocation, the results obtained by analyzing the official performance occupancy tool of the chip and the performance occupancy index Pi can be comprehensively considered to obtain the overall occupancy of the chip under the current inference allocation.

[0185] In the embodiment of the present application, a performance balancing strategy can be pre-configured to adjust the sub-models deployed on each chip according to the chip occupancy to achieve performance balance between the chips.

[0186] Among them, the performance balancing strategy can be set by the staff according to business needs. For example, for a GPU chip, if it needs to run image display and the GPU memory usage does not exceed 60%, and the Pi of the sub-model running on the GPU does not exceed 100%, there is no need to reallocate the sub-model on the GPU. On the contrary, if the GPU needs to run image display and the GPU memory usage exceeds 60, or the Pi of the sub-model running on the GPU exceeds 100%, the sub-model on the GPU needs to be reallocated.

[0187] Of course, there are other ways to set the performance balancing strategy, which are not described in detail in this application. The staff will set it specifically according to business needs to ensure the performance balance of each chip on the heterogeneous platform.

[0188] Combination Figure 8 , which illustrates the framework for online deployment and reasoning of sub-models on terminal devices.

[0189] Each deployable sub-model generated by the server, such as HdM.uni, F1M.uni, F2M.uni…TlM.uni, can be loaded into the memory through the model loading module and transformed into a format supported by the target chip.

[0190] Call the model inference module to complete the inference of the sub-model. During the inference process, the model inference module can receive external input, perform inference through the deployed model, and output the inference result.

[0191] In this embodiment, a memory access distributor MAD can be further introduced on the terminal. The memory access distributor is configured with various memory acceleration strategies for optimizing general chips (such as Figure 8 The memory interaction speed of chip1, chip2, chip3 in the example and heterogeneous computing chips can be improved to solve the memory bottleneck of large model computing.

[0192] The following describes a large model deployment device for a heterogeneous platform provided in an embodiment of the present application. The large model deployment device for a heterogeneous platform described below and the large model deployment method for a heterogeneous platform described above can refer to each other.

[0193] See also Fig. 9 , Fig. 9 A schematic diagram of the structure of a large model deployment device for a heterogeneous platform disclosed in an embodiment of the present application.

[0194] like Fig. 9 As shown, the device may include:

[0195] A model splitting unit 11 is used to split the large model to be deployed into several sub-models;

[0196] A chip selection unit 12 is used to determine the target chip to be deployed for each sub-model according to the operator type and the calculation amount of the operator included in each sub-model, combined with the operator support of various types of chips on the heterogeneous platform;

[0197] A quantization strategy determination unit 13, used to determine the quantization strategy of each sub-model according to the quantization support of the target chip to be deployed by each sub-model;

[0198] A model quantization unit 14, configured to quantize the sub-model according to the quantization strategy of each sub-model to obtain a quantized sub-model;

[0199] The deployable model generation unit 15 is used to generate a deployable sub-model from the quantized sub-model and the corresponding quantization strategy, so as to facilitate deployment to the corresponding target chip on the heterogeneous platform.

[0200] In a possible implementation, the large model to be deployed is a hybrid expert large model, and the process of the model splitting unit splitting the large model to be deployed into a plurality of sub-models includes:

[0201] Analyze the hybrid expert macromodel to obtain various operators included in the macromodel and the structural relationships between the operators;

[0202] According to the structural relationship between operators, searching starts from the input of the hybrid expert large model to identify the target gating operator, and there are more than two feedforward neural networks behind the target gating operator;

[0203] Splitting the hybrid expert large model from the input to the target gating operator part into a head sub-model;

[0204] Continue to search backward and identify that the outputs of more than two feedforward networks are concentrated on the same weight sum operator weight sum, split before the weight sum, and take each path from the weight sum to the target gating operator as an expert sub-model, and split the weight sum and the part after it into a tail sub-model.

[0205] In a possible implementation, the chip selection unit determines the target chip to be deployed for each sub-model according to the operator type and the computational amount of the operator included in each sub-model and the operator support of each type of chip on the heterogeneous platform, including:

[0206] For each of the sub-models:

[0207] According to the operator types included in the sub-model and in combination with the operator support of various types of chips on the heterogeneous platform, determine each candidate chip that can support the sub-model;

[0208] Determining the computation amount of the sub-model according to the computation amount of the operators included in the sub-model;

[0209] If the computation amount of the sub-model is greater than the set computation amount threshold, combining the candidate chips that can support the sub-model, and selecting the target chip to be deployed by the sub-model from the candidate chips according to the first chip selection order;

[0210] If the computational complexity of the sub-model is not greater than the set computational complexity threshold, then in combination with the candidate chips that can support the sub-model, the target chip to be deployed by the sub-model is selected from the candidate chips in a second chip selection order, and the first chip selection order is different from the second chip selection order.

[0211] In one possible implementation, the first chip selection order and the second chip selection order each include a central processing unit CPU, a graphics processing unit GPU and a neural network processor NPU. In the first chip selection order, the order of priority from high to low is NPU, GPU, CPU; in the second chip selection order, the order of priority from high to low is CPU, GPU, NPU.

[0212] In a possible implementation, the process of the quantization strategy determination unit determining the quantization strategy of each sub-model according to the quantization support of the target chip to be deployed by each sub-model includes:

[0213] Determine a set of candidate quantization strategies for each operator in the sub-model based on the quantization support of each operator by the target chip to be deployed in each sub-model;

[0214] A target quantization strategy is selected from the candidate quantization strategy set of each operator, and the quantization strategy of the sub-model is composed of the target quantization strategies of each operator.

[0215] In a possible implementation, the process of the quantization strategy determination unit selecting a target quantization strategy from a set of candidate quantization strategies for each operator includes:

[0216] In response to an operation of selecting a target quantization mode from a configured quantization mode list, selecting a target quantization strategy matching the target quantization mode from a set of candidate quantization strategies for each operator according to a quantization strategy selection rule corresponding to the target quantization mode;

[0217] The quantization mode list includes more than two quantization modes, and different quantization modes represent different degrees of accuracy reduction caused by quantization, and different model volumes and calculation amounts after quantization.

[0218] In a possible implementation, the quantization mode list includes at least three quantization modes: a precision mode, a performance mode, and a balance mode;

[0219] The precision mode characterization and the original floating point model are compared, and the precision reduction caused by quantization does not exceed a first ratio;

[0220] The performance model characterizes that the accuracy reduction caused by quantization compared with the original floating-point model does not exceed a second ratio and the volume and amount of calculation of the quantized model are reduced to the greatest extent;

[0221] The balance mode characterization reduces the accuracy caused by quantization by no more than a third ratio compared to the original floating-point model and reduces the volume and computational complexity of the quantized model to the greatest extent;

[0222] The first ratio is smaller than the third ratio, and the third ratio is smaller than the second ratio.

[0223] In a possible implementation, the process of the deployable model generation unit generating a deployable sub-model from the quantized sub-model and the corresponding quantization strategy includes:

[0224] The quantized sub-model and the corresponding quantization strategy are used to generate a sub-model in a set intermediate format, where the intermediate format is a set format different from the format supported by various types of chips.

[0225] In a possible implementation, the device of the present application may further include:

[0226] The online deployment unit is used to convert the format of each sub-model into a format supported by the target chip to which the sub-model is to be deployed, and deploy it to the corresponding target chip on the heterogeneous platform when performing model deployment.

[0227] In a possible implementation, the device of the present application may further include:

[0228] The performance balancing unit is used to count the occupancy of each chip under the current inference allocation on the heterogeneous platform after the sub-model is deployed on the target chip; according to the occupancy of each chip under the current inference allocation, the sub-model deployed on each chip is adjusted according to the configured performance balancing strategy.

[0229] The present application also provides an electronic device in an embodiment. Fig.10As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, tablet computers, teaching large screens, wearable devices, etc. Fig.10 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0230] like Fig.10 As shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603, so as to implement the large model deployment method of the heterogeneous platform of the aforementioned embodiment of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0231] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Fig.10 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0232] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any of the large model deployment methods for heterogeneous platforms provided in the embodiments of the present application.

[0233] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any large model deployment method for a heterogeneous platform provided in an embodiment of the present application.

[0234] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0235] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0236] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0237] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0238] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

Claims

1. A large model deployment method for a heterogeneous platform, characterized in that: include: Split the large model to be deployed into several sub-models; Determine the target chip to be deployed for each sub-model according to the operator type and the computational amount of the operator included in each sub-model and the operator support of various types of chips on the heterogeneous platform; Determine the quantization strategy for each sub-model based on the quantization support of the target chip where each sub-model is to be deployed; Quantize the sub-models according to the quantization strategy of each sub-model to obtain quantized sub-models; The quantized sub-model and the corresponding quantization strategy are used to generate a deployable sub-model so as to be deployed to the corresponding target chip on the heterogeneous platform.

2. The method according to claim 1, characterized in that If the large model to be deployed is a hybrid expert large model, the process of splitting the large model to be deployed into a plurality of sub-models includes: Analyze the hybrid expert macromodel to obtain various operators included in the macromodel and the structural relationships between the operators; According to the structural relationship between operators, searching starts from the input of the hybrid expert large model to identify the target gating operator, and there are more than two feedforward neural networks behind the target gating operator; Splitting the hybrid expert large model from input to the target gating operator into head sub-models; Continue to search backward and identify that the outputs of more than two feedforward networks are concentrated on the same weight summation operator. Perform splitting before the weight summation operator, and use each path from the weight summation operator to the target gating operator as an expert sub-model. Split the weight summation operator and the subsequent part into tail sub-models.

3. The method according to claim 1, characterized in that The process of determining the target chip to be deployed for each sub-model according to the operator type and the calculation amount of the operator included in each sub-model and the operator support of various types of chips on the heterogeneous platform includes: For each of the sub-models: According to the operator types included in the sub-model and in combination with the operator support of various types of chips on the heterogeneous platform, determine each candidate chip that can support the sub-model; Determining the computation amount of the sub-model according to the computation amount of the operators included in the sub-model; If the computation amount of the sub-model is greater than the set computation amount threshold, combining the candidate chips that can support the sub-model, and selecting the target chip to be deployed by the sub-model from the candidate chips according to the first chip selection order; If the computational complexity of the sub-model is not greater than the set computational complexity threshold, then in combination with the candidate chips that can support the sub-model, the target chip to be deployed by the sub-model is selected from the candidate chips in a second chip selection order, and the first chip selection order is different from the second chip selection order.

4. The method according to claim 3, characterized in that The first chip selection sequence and the second chip selection sequence each include a central processing unit CPU, a graphics processing unit GPU and a neural network processor NPU. In the first chip selection sequence, the priority is NPU, GPU, CPU in descending order; in the second chip selection sequence, the priority is CPU, GPU, NPU in descending order.

5. The method according to claim 1, characterized in that The process of determining the quantization strategy of each sub-model according to the quantization support of the target chip to be deployed by each sub-model includes: Determine a set of candidate quantization strategies for each operator in the sub-model based on the quantization support of each operator by the target chip to be deployed in each sub-model; A target quantization strategy is selected from the candidate quantization strategy set of each operator, and the quantization strategy of the sub-model is composed of the target quantization strategies of each operator.

6. The method according to claim 5, characterized in that The process of selecting a target quantization strategy from the set of candidate quantization strategies for each operator includes: In response to an operation of selecting a target quantization mode from a configured quantization mode list, selecting a target quantization strategy matching the target quantization mode from a set of candidate quantization strategies for each operator according to a quantization strategy selection rule corresponding to the target quantization mode; The quantization mode list includes more than two quantization modes, and different quantization modes represent different degrees of accuracy reduction caused by quantization, and different model volumes and calculation amounts after quantization.

7. The method according to claim 6, characterized in that The quantization mode list includes at least three quantization modes: precision mode, performance mode and balance mode; The precision mode characterization and the original floating point model are compared, and the precision reduction caused by quantization does not exceed a first ratio; The performance model characterizes that the accuracy reduction caused by quantization compared with the original floating-point model does not exceed a second ratio and the volume and amount of calculation of the quantized model are reduced to the greatest extent; The balance mode characterization reduces the accuracy caused by quantization by no more than a third ratio compared to the original floating-point model and reduces the volume and computational complexity of the quantized model to the greatest extent; The first ratio is smaller than the third ratio, and the third ratio is smaller than the second ratio.

8. The method according to claim 1, characterized in that The process of generating a deployable sub-model from the quantized sub-model and the corresponding quantization strategy includes: The quantized sub-model and the corresponding quantization strategy are used to generate a sub-model in a set intermediate format, where the intermediate format is a set format different from the format supported by various types of chips.

9. The method according to any one of claims 1 to 8, characterized in that: Also includes: When the model is deployed, the format of each sub-model is converted into a format supported by the target chip to which the sub-model is to be deployed, and deployed to the corresponding target chip on the heterogeneous platform.

10. The method according to claim 9, characterized in that After deploying the sub-model on the target chip, the method further includes: Counting the occupancy of each chip on the heterogeneous platform under the current inference allocation; According to the occupancy of each chip under the current inference allocation and the configured performance balancing strategy, the sub-models deployed on each chip are adjusted.

11. A large model deployment device for a heterogeneous platform, characterized in that: include: A model splitting unit is used to split the large model to be deployed into several sub-models; A chip selection unit, used to determine the target chip to be deployed for each sub-model according to the operator type and the computational amount of the operator contained in each sub-model, combined with the operator support of various types of chips on the heterogeneous platform; A quantization strategy determination unit, used to determine the quantization strategy of each sub-model according to the quantization support of the target chip to be deployed by each sub-model; A model quantization unit, used for quantizing the sub-model according to the quantization strategy of each sub-model to obtain a quantized sub-model; The deployable model generation unit is used to generate a deployable sub-model from the quantized sub-model and the corresponding quantization strategy, so as to facilitate deployment to the corresponding target chip on the heterogeneous platform.

12. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the large model deployment method for a heterogeneous platform as described in any one of claims 1 to 10.

13. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the large model deployment method for a heterogeneous platform as described in any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the large model deployment method for a heterogeneous platform as described in any one of claims 1 to 10 is implemented.

Images