Large model reasoning method, device, equipment and program product

Abstract

The present application discloses a large model reasoning method, device, equipment and program product, which relates to the field of artificial intelligence technology. The present application selects a sparse large model as the large model to be deployed during deployment, further splits it into several sub-models, and stores them in the memory of the terminal. When executing a large model reasoning task, only the target sub-model required by the current reasoning task can be loaded from the memory to the reasoning chip to perform the reasoning task, without loading all the sparse large models, which greatly reduces the number of model parameters required to be loaded, reduces the amount of memory access, and reduces the time consumption of reasoning, effectively alleviating the memory wall problem.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a large model reasoning method, device, equipment and program product. Background Art

[0002] With the development of big model technology, we have entered the stage of big model application. One big model implementation solution is to deploy and reason about the big model on the end-side device.

[0003] The process of deploying large models on edge devices for inference is prone to memory wall problems. The memory wall refers to the phenomenon that the performance of the CPU or NPU (neural network processor) is severely restricted due to limited memory capacity or transmission bandwidth. This problem is particularly prominent when large models are deployed on edge devices for inference, because large models usually require a large amount of memory to store model parameters and intermediate data. Currently, there is no systematic solution to the memory wall problem of edge devices where large models are deployed. Summary of the invention

[0004] In view of the above problems, this application is proposed to provide a large model reasoning method, device, equipment and program product to alleviate the memory wall problem when the large model is implemented on the terminal side. The specific solution is as follows:

[0005] In a first aspect, a large model reasoning method is provided, including:

[0006] Determine the sparse large model to be deployed on the client side;

[0007] Splitting the sparse large model into a plurality of sub-models and storing them in the memory of the terminal;

[0008] When executing a large model reasoning task, the target sub-model is loaded from the memory to the reasoning chip according to the target sub-model required by the current reasoning task, and the reasoning task is executed using the loaded target sub-model.

[0009] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the sparse large model is a hybrid expert large model, and the process of splitting the sparse large model into a plurality of sub-models includes:

[0010] Analyze the hybrid expert macromodel to obtain various operators included in the macromodel and the structural relationships between the operators;

[0011] According to the structural relationship between operators, searching starts from the input of the hybrid expert large model to identify the target gating operator, and there are more than two feedforward neural networks behind the target gating operator;

[0012] Splitting the hybrid expert large model from input to the target gating operator into head sub-models;

[0013] Continue to search backward and identify that the outputs of more than two feedforward networks are concentrated on the same weight summation operator. Perform splitting before the weight summation operator, and use each path from the weight summation operator to the target gating operator as an expert sub-model. Split the weight summation operator and the subsequent part into tail sub-models.

[0014] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the large model reasoning task includes a pre-processing process of the reasoning task and a large model reasoning calculation process;

[0015] When executing a large model reasoning task, the process of loading the target sub-model from the memory to the reasoning chip according to the target sub-model required by the current reasoning task includes:

[0016] In response to the inference task start instruction, while executing the pre-processing of the inference task, the target sub-model is pre-loaded from the memory to the inference chip according to the target sub-model required by the current inference task. The pre-processing is the process of pre-processing the input data before executing the large model inference calculation.

[0017] In a possible design, in another implementation of the first aspect of the embodiment of the present application, when executing a large model reasoning task, according to the target sub-model required by the current reasoning task, the process of loading the target sub-model from the memory to the reasoning chip includes:

[0018] When executing a large model reasoning task, the relevant data of the target sub-model is sparsely processed in the memory according to the target sub-model required by the current reasoning task, the sparsely processed model data is transferred from the memory to the reasoning chip, and the sparsely processed model data is restored in the reasoning chip to obtain the restored target sub-model.

[0019] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of storing the split sub-models into the memory of the terminal includes:

[0020] According to the inference target of the current inference task, determine the target arrangement mode of the data related to each sub-model in the memory, and for the inference target, the target arrangement mode has a higher cache affinity than other arrangements;

[0021] Each sub-model is stored according to the target arrangement.

[0022] In a possible design, in another implementation of the first aspect of the embodiment of the present application, before splitting the sparse large model into a plurality of sub-models, the method further includes:

[0023] Performing pruning and / or quantization processing on the sparse large model to obtain a processed sparse large model;

[0024] The process of splitting the sparse large model into several sub-models specifically includes:

[0025] The processed sparse large model is split into several sub-models.

[0026] In a possible design, in another implementation of the first aspect of the embodiment of the present application, after splitting the sparse large model into a plurality of sub-models and before storing them in the memory of the terminal, the method further includes:

[0027] Pruning and / or quantization are performed on each of the split sub-models to obtain each processed sub-model for storage in the memory of the terminal.

[0028] In a second aspect, a large model reasoning device is provided, comprising:

[0029] A to-be-deployed model determination unit, used to determine a sparse large model to be deployed on the terminal side;

[0030] A model splitting unit, used for splitting the sparse large model into a plurality of sub-models and storing them in the memory of the terminal;

[0031] The model loading unit is used to load the target sub-model from the memory to the inference chip according to the target sub-model required by the current inference task when executing a large model inference task, and use the loaded target sub-model to execute the inference task.

[0032] In a third aspect, an electronic device is provided, comprising: a memory and a processor;

[0033] The memory is used to store programs;

[0034] The processor is used to execute the program to implement the various steps of the large model reasoning method described in any one of the first aspects of the present application.

[0035] In a fourth aspect, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the various steps of the large model reasoning method described in any one of the first aspects of the present application are implemented.

[0036] In a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the various steps of the large model reasoning method described in any one of the aforementioned first aspects of the present application.

[0037] Through the above technical solution, the present application provides a memory wall solution for the landing of large models on the terminal side device. During deployment, a sparse large model is selected as the large model to be deployed. The sparse large model refers to a model that only uses part of the parameters for calculation. Not all parameters in the model are involved in the calculation, but some parameters are dynamically activated according to task requirements. In this application, a sparse large model is selected as the large model to be deployed, and it is further split into several sub-models and stored in the memory of the terminal. When executing a large model reasoning task, only the target sub-model required by the current reasoning task can be loaded from the memory to the inference chip to perform the reasoning task. There is no need to load all the sparse large models, which greatly reduces the number of model parameters required to be loaded, reduces the amount of memory access, and reduces the time consumption of reasoning, effectively alleviating the memory wall problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0039] Figure 1 A schematic diagram of an implementation system architecture of the large model reasoning method provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of a terminal structure provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of a server structure provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of a large model reasoning method flow chart provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of splitting a large MOE model provided in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of another large model reasoning method flow chart provided in an embodiment of the present application;

[0045] Figure 7 A schematic diagram of a memory prefetching timing provided in an embodiment of the present application;

[0046] Figure 8 A scenario example diagram of memory sorting based on cache affinity optimization provided in an embodiment of the present application;

[0047] Fig. 9 A schematic diagram of the structure of a large model reasoning device provided in an embodiment of the present application;

[0048] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] Before introducing this application solution, the relevant concepts involved in this article are first explained:

[0050] Large models: In the field of artificial intelligence, large models usually refer to large-scale pre-trained models. Such models are called "large" because they can be pre-trained on a large amount of data and can be transferred to a variety of downstream tasks. Their full English name is Large Pre-Trained Models or Large-Scale Pre-Training Models. Large models are characterized by their large scale and contain billions or even more parameters, which help them learn complex patterns in the data. The emerging capabilities of large models include but are not limited to: contextual learning, instruction following, code generation, step-by-step reasoning capabilities, etc. Large models can include large language models (LLMs) and multimodal large models. Large language models are mainly used to process text modal data. Multimodal large models further integrate multimodal capabilities on the basis of large language models and can process information in multiple modalities, such as images, text, audio, etc.

[0051] Sparse large models: Sparse large models refer to models that only use some parameters for calculation. While these models have a large number of parameters, they use sparsity technology to make most of the parameters inactive in specific tasks, thereby improving computing efficiency and performance. Sparse large models usually use the MOE architecture, which is the full name of Mixture of Experts. It is a large model architecture used to improve the performance and efficiency of deep learning models. The core idea is to introduce multiple expert models, and only select and activate a part of the expert models for processing for each input data, thereby reducing the amount of calculation and increasing the training and reasoning speed. Common MOE large models include GPT-4.

[0052] CPU: The full name is Central Processing Unit. The CPU is the core component of the computer system, responsible for executing program instructions, processing data, and controlling the operation of the computer. It is composed of components such as the arithmetic unit, controller, register group, and bus interface, and is the key to the computer's ability to complete various complex tasks.

[0053] NPU: The full name is Neural Processing Unit. NPU is a new type of processor designed for deep learning and neural network computing. It can efficiently execute neural network algorithms and accelerate the calculation process of deep learning tasks. With the continuous development of artificial intelligence technology, NPU plays an increasingly important role in image recognition, speech recognition, natural language processing and other fields.

[0054] Cache affinity: Cache affinity is an important concept in computer science, especially in multi-core processor architecture. It mainly refers to the tendency of a process or thread to run on a specific CPU core or processor during execution and maintain the association with the cache content on that core or processor for as long as possible. This affinity helps to improve the cache hit rate and reduce the number of cache misses, thereby reducing the frequency of processor access to memory and improving the overall performance of the system.

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] This application provides a large model reasoning method that can be applied to Figure 1 The system architecture shown in FIG. 1 may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 A server is included as an example for explanation).

[0057] The terminal 100 can be used alone to execute the large model reasoning method provided in the embodiment of the present application. In addition, the terminal 100 and the server 200 can also be used in collaboration to execute the large model reasoning method provided in the embodiment of the present application.

[0058] Next describe Figure 1 The product form of the mid-terminal 100;

[0059] The terminal 100 in the embodiment of the present application is a terminal device that needs to deploy a large model. It can be a mobile phone, a tablet computer, a learning machine, a wearable device, a vehicle-mounted device, a conference terminal, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiment of the present application does not impose any restrictions on this.

[0060] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0061] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160, a speaker 161, a microphone 162, an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0062] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0063] Among them, other input devices 132 can receive input data and so on.

[0064] The display unit 140 may be used to display information input by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file display, and / or playback of any multimedia file. In an embodiment of the present application, the display unit 140 may be used to display various interactive interfaces, processing results, etc. in the large model reasoning method.

[0065] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0066] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on independent chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0067] Among them, the memory 120 can be used to store software codes related to the large model reasoning method, the processor 170 can execute the steps of the large model reasoning method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to implement corresponding functions.

[0068] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0069] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0070] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0071] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0072] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0073] Next describe Figure 1 The product form of the server 200;

[0074] Figure 3A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0075] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0076] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0077] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0078] Among them, the memory 204 can be used to store software codes related to the large model reasoning method, the processor 202 can execute the steps of the large model reasoning method, and can also schedule other units to implement corresponding functions.

[0079] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0080] For the sake of convenience, in the following embodiments of the present application, a mobile phone is used as an example of a terminal device to be deployed with a large model. It is assumed that the mobile phone chip uses Kirin 9010, and the NPU chip is used as the inference chip. The parameters are assumed to be as follows:

[0081] NPU: Dual Big Core + Tiny Core NPUs, TOPS can only use a single core;

[0082] DDR5:4800MHz,32GB / s.

[0083] The present application embodiment provides a large model reasoning method, which is applied to Figure 1 For example, the computer device in the example may be the terminal 100, or a system consisting of the terminal 100 and the server 200. Figure 4 , the large model reasoning method specifically includes the following steps:

[0084] Step S100: Determine a large sparse model to be deployed on the terminal side.

[0085] Specifically, for computationally intensive large models, they pose a greater obstacle to solving the memory wall problem. According to research and evaluation, the amount of inference calculations for sparse large models of the same parameter magnitude is much lower than that for computationally intensive large model structures. Therefore, in this embodiment, a sparse large model is selected as the large model to be deployed.

[0086] The comparison of inference efficiency between computationally intensive large models and sparse large models is as follows:

[0087] The LLaMA3 structure is used as a computationally intensive large model. The total number of parameters of the model is 0.9B, the computation amount is 4GFLOPS / token, and the memory access amount is 1B FLOAT (1FLOAT=4byte). The inference efficiency under the memory wall is shown in Table 1:

[0088] Table 1

[0089]

[0090] This application can select a sparse large model with the same number of parameters, and take the MOE hybrid expert large model as an example of the large model to be deployed. Assuming that the number of experts in the MOE hybrid expert large model is 32, the total number of parameters is 0.96B, the number of activated experts is 4, the maximum number of activated parameters is 0.128B, the calculation amount is 0.4 GFLOPS / token, the access memory is 1B FLOAT, and the inference efficiency under the memory wall is as shown in Table 2:

[0091] Table 2

[0092]

[0093] Comparing the above computationally intensive large model and the sparse large model, it can be seen that due to the memory wall, the inference time remains basically unchanged whether the amount of computation increases or decreases.

[0094] Since the sparse large model can be further split and can contain more expert models with the same amount of computation, it can cope with more reasoning scenarios. Therefore, in this embodiment, the sparse large model is selected as the large model to be deployed.

[0095] Step S110: split the sparse large model into several sub-models and store them in the memory of the terminal.

[0096] Specifically, taking the MOE mixed expert large model as an example of a sparse large model, in this step the MOE large model can be split into a head sub-model, several expert sub-models in the middle part, and a tail sub-model.

[0097] Of course, the above is only an optional way to split the large model. In addition, other optional splitting strategies can also be used to split the large model into several sub-models.

[0098] Each sub-model after the large model is split can contain more than one operator. Usually, each sub-model contains multiple operators. Operators are elements that perform specific mathematical operations or transformations, which define how data is transformed from one state to another. Operators can be divided into many types, including but not limited to: linear operators, such as fully connected operators and convolution operators; non-linear operators, such as activation functions Relu, Sigmoid, etc.; pooling operators, such as max pooling, average pooling, etc.; normalization operators, other special operators, etc.

[0099] Each sub-model after the sparse large model is split can be stored in the terminal's memory. For example, each sub-model after the sparse large model is split can be stored in a ROM hard disk, and then loaded from the hard disk into the CPU's memory after the terminal device is started. When performing the inference task later, the selected target sub-model can be loaded from the memory to the inference chip.

[0100] Step S120: When executing a large model reasoning task, load the target sub-model from the memory to the reasoning chip according to the target sub-model required by the current reasoning task, and use the loaded target sub-model to execute the reasoning task.

[0101] Specifically, the process of executing the reasoning task of the large model can determine the target sub-model required to execute the current reasoning task according to the type of the current reasoning task. Taking the MOE large model as an example, assuming that it contains a head sub-model, 32 expert sub-models in the middle part, and a tail sub-model, and the number of activated experts is 4, then when executing different reasoning tasks, the expert models required may not be exactly the same, so only the 4 expert sub-models used in the current reasoning task can be loaded, without loading all 32 expert sub-models.

[0102] After determining the target sub-model required for the current reasoning task, the target sub-model can be loaded from the memory to the reasoning chip. Taking the NPU chip as the reasoning chip as an example, the target sub-model can be loaded into the NPU, and then the target sub-model can be used in the NPU to perform the reasoning task.

[0103] After the current reasoning task is executed, the next reasoning task is executed, that is, the process of step S120 is repeated until all reasoning tasks are completed.

[0104] The embodiment of the present application provides a solution to the memory wall when a large model is deployed on the terminal side device. During deployment, a sparse large model is selected as the large model to be deployed. A sparse large model refers to a model that only uses some parameters for calculation. Not all parameters in the model are involved in the calculation, but some parameters are dynamically activated according to task requirements. In this application, a sparse large model is selected as the large model to be deployed, and it is further split into several sub-models and stored in the memory of the terminal. When executing a large model reasoning task, only the target sub-model required by the current reasoning task can be loaded from the memory to the reasoning chip to perform the reasoning task. There is no need to load all the sparse large models, which greatly reduces the number of model parameters required to be loaded, reduces the amount of memory access, and reduces the reasoning time, effectively alleviating the memory wall problem.

[0105] In some possible implementations, taking the hybrid expert MOE large model as an example, the process of splitting the sparse large model into several sub-models in the aforementioned step S110 is described. The process may include the following steps:

[0106] S1. Analyze the MOE big model to obtain the various operators contained in the MOE big model and the structural relationship between the operators.

[0107] S2. According to the structural relationship between operators, the search starts from the input of the MOE large model and the target gating operator is identified. There are more than two feedforward networks FFN behind the target gating operator.

[0108] Based on the characteristics of the MOE large model, in the MOE architecture, the routing module Router is a key component responsible for assigning input data to different expert sub-models for processing. The input data is first passed to the Router, which calculates the degree of response (i.e., score) of each expert sub-model to the input data based on its internal mapping mechanism (such as the linear layer and the softmax layer). Based on the calculated score, the Router selects the expert sub-model with the highest score (under the Top-1 strategy) or multiple sub-models (under the Top-k strategy) to process the input data. If the Top-k strategy is adopted, the processing results of multiple sub-models will be aggregated in a certain way (such as weighted average) to form the final output result. Specifically, the output results of multiple expert sub-models are effectively aggregated through the weight sum operator weight sum.

[0109] Router includes gating operators, such as soft-gated softmax operator, hard-gated argmax operator, TopK gated softmax+top-k operator, etc.

[0110] In this step, according to the structural relationship between the operators obtained by analysis, the search starts from the input of the MOE large model to identify the target gating operator. There are more than two feedforward networks FFN after the target gating operator, indicating that the target gating operator belongs to the expert sub-model. Therefore, the target gating operator and the subsequent FFN can be used as the splitting position to obtain the head sub-model.

[0111] S3. Divide the MOE large model from the input to the target gating operator part into the head sub-model.

[0112] like Figure 5 As shown in the figure, it illustrates the structure of the MOE large model, from left to right, it is the input to output direction of the model. After identifying the target gating operator Router, it can be split between Router and FFN to obtain the head sub-model.

[0113] S4. Continue to search backward and identify that the outputs of more than two feedforward neural networks FFN are concentrated on the same weight sum operator weight sum. Split before weight sum, and use each path from weight sum to the target gating operator Router as an expert sub-model, and split weight sum and the part after it into tail tail sub-models.

[0114] like Figure 5 As shown in FIG. 1 , a partition is performed between FFN and weight sum, and each path from weight sum to the target gating operator Router is used as an expert sub-model, namely FFN1, FFN2…FFNE, with a total of E expert sub-models.

[0115] After the above steps, we can obtain the split head sub-model, several expert sub-models and the tail sub-model.

[0116] In this embodiment, the MOE large model is taken as an example to introduce the process of segmenting it. The MOE large model can be segmented into several sub-models, which are then stored in the memory respectively for subsequent calls.

[0117] In some possible implementations, in order to further alleviate the memory wall problem, the present embodiment may further increase the process of pruning and / or quantizing the sparse large model to be deployed, thereby further reducing the size and computational complexity of the large model.

[0118] The above-mentioned process of pruning and quantizing the sparse large model can be adopted one by one, or the pruning and quantizing processes can be performed successively. The execution order of the pruning and quantizing processes can be selected arbitrarily.

[0119] Furthermore, the process of pruning and quantization may be performed before the aforementioned step S110 of splitting the sparse large model into several sub-models, or after step S110, and this application does not impose strict limitations on this.

[0120] Reference Figure 6 , which illustrates a large model reasoning method process, including the following steps:

[0121] Step S200: Determine a large sparse model to be deployed on the terminal side.

[0122] Step S210: Pruning the sparse large model.

[0123] Step S220: quantize the sparse large model.

[0124] Step S230: split the sparse large model into several sub-models and store them in the memory of the terminal.

[0125] Step S240: When executing a large model reasoning task, load the target sub-model from the memory to the reasoning chip according to the target sub-model required by the current reasoning task, and use the loaded target sub-model to execute the reasoning task.

[0126] In this embodiment, pruning and quantization processing are performed on the sparse large model in sequence, and the processed sparse large model is further split, and each split sub-model is stored in the memory of the terminal. When executing the reasoning task, according to the target sub-model required by the current reasoning task, the corresponding target sub-model is loaded from the memory to the reasoning chip to execute the reasoning task. Compared with the previous embodiment, the pruning and quantization processing process of the large model is further added in this embodiment. Through pruning and quantization processing, the sparse large model is compressed, the model size and memory usage are reduced, and the memory pressure is reduced, which can further alleviate the memory wall problem.

[0127] In some possible implementations, a method for pruning large sparse models is introduced.

[0128] In the reasoning process of sparse large models, the multi-head attention mechanism (MHA) is one of the core components, which enhances the expressiveness of the model by processing multiple attention heads in parallel. However, the key-value (KV) cache of MHA poses a major obstacle to reasoning efficiency, especially when processing long sequences, the size of the KV cache will increase significantly, resulting in excessive memory usage, affecting reasoning speed and cost.

[0129] To this end, this application uses the MLA algorithm proposed by DeepSeek-V2 to optimize the MHA algorithm, aiming to reduce the demand for KV cache through low-rank KV joint compression while maintaining or improving model performance. Specifically, the MLA algorithm compresses the original KV cache into a smaller latent space through low-rank decomposition, thereby significantly reducing the dimension and size of the cache.

[0130] The implementation of the MLA algorithm mainly involves the following steps:

[0131] Low-rank decomposition: The original KV cache matrix is ​​compressed into a smaller latent space through low-rank decomposition (such as matrix multiplication, factor decomposition, etc.).

[0132] Latent space calculation: Calculate and normalize the attention scores in the latent space to obtain the weight of each attention head.

[0133] Output reconstruction: Reconstruct the output of each attention head based on the attention weights in the latent space and the original input.

[0134] The experimental results of DeepSeek-V2 show that compared with the MHA-based model, the MLA algorithm significantly reduces the KV cache while maintaining comparable model performance and showing better performance on some tasks. In addition, the MLA algorithm also reduces training costs and improves inference speed.

[0135] In this embodiment, after selecting MLA to optimize memory and calculation, the number of parameters is optimized to about 2 / 7 under the condition of the same effect.

[0136] Taking the MOE large model in Table 2 as an example, the number of experts in the MOE model is 32. After MLA optimization, the total number of parameters is 0.32B, the number of activated experts is 4, the maximum number of activated parameters is 0.058B, the computing amount is 0.04 GFLOPS / token, and the memory access amount is 0.32B FLOAT. The inference efficiency under the memory wall is shown in Table 3:

[0137] Table 3

[0138]

[0139] From the above, we can see that after pruning, the amount of calculation and memory access have been reduced, and the final reasoning time has also been greatly reduced.

[0140] Of course, the above only illustrates an optional pruning algorithm. In addition, other pruning algorithms can also be used to prune the sparse large model.

[0141] In some possible implementations, the process of quantizing a large sparse model usually involves dynamic quantization and int8 quantization. Depending on the support of the NPU, int4 quantization can also be selected to obtain better performance. Some storage-computing integrated NPUs can even support extreme 1-bit quantization. In this embodiment, the int8 quantization solution is taken as an example.

[0142] For the large model after the above pruning process, after int8 quantization, the number of experts in the MOE model is 32, the total number of parameters is 0.32B, the number of activated experts is 4, the maximum number of activated parameters is 0.058B, the computing amount is 0.04 GFLOPS / token, the memory access amount is 0.32B byte, and the inference efficiency under the memory wall is as shown in Table 4:

[0143] Table 4

[0144]

[0145] After the above quantization processing, the memory access amount is further reduced and the reasoning time is further reduced.

[0146] After completing the above quantization processing, the sparse large model can be further split. After splitting, it is not necessary to load the full amount of expert models, and on-demand loading can be achieved. Since the number of activated experts is 4, a maximum of 4 expert models can be loaded each time. The overall computational workload of the model remains unchanged, and the memory access volume is reduced from 0.32B byte to 0.064B byte (experimental measurement). The inference efficiency under the memory wall is shown in Table 5 below:

[0147] Table 5

[0148]

[0149] From the above, we can see that after splitting the model, there is no need to load the entire expert model each time for inference, which can further reduce the amount of memory access and further reduce the inference time.

[0150] The model pruning, quantization, and splitting methods described in the above embodiments can reduce model memory. In this embodiment, the data transmission process of the inference chip is further managed to improve memory bandwidth utilization.

[0151] This embodiment provides three memory transfer management solutions, which are described as follows:

[0152] First, memory prefetching.

[0153] The large model reasoning task includes the large model reasoning calculation process and other non-inference calculation processes, among which the non-inference calculation process examples include the pre-processing process of the reasoning task and the post-processing process of the reasoning task. The pre-processing of the reasoning task can be understood as the pre-processing process of the input data. The post-processing process of the reasoning task can be understood as the post-processing process of the reasoning result. Depending on the reasoning task, the pre-processing and post-processing operations may also be different. Taking the speech recognition reasoning task as an example, the pre-processing process may include acoustic feature extraction, etc.

[0154] In the case where the large model inference task includes pre-processing of the inference task, in this embodiment, when responding to the inference task start instruction, while executing the pre-processing of the inference task, the target sub-model can be pre-loaded from the memory to the inference chip according to the target sub-model required by the current inference task. When the program runs to the inference calculation, the loading of part or all of the target sub-model has been completed, which can improve the utilization of the memory bandwidth.

[0155] Furthermore, if the previous reasoning task (defined as task B) of the current reasoning task (defined as task A) includes a post-processing process, the target sub-model required for task A can be pre-loaded from the memory to the inference chip while task B is performing post-processing.

[0156] Combination Figure 7 As shown in Figure 1, the inference task includes three steps: pre-processing, model inference, and post-processing. During the post-processing and pre-processing, the target sub-model required for the next inference task can be pre-loaded from the memory to the inference chip (i.e. Figure 7 ”Prefetch 1” and “Prefetch 2” in the READ_MODE function) to improve the utilization of memory bandwidth.

[0157] Second, memory sparse transmission.

[0158] Specifically, in order to reduce memory usage during transmission, when loading the target sub-model from the memory to the inference chip, the target sub-model related data can be first sparsely processed in the memory, and then the sparsely processed model data can be transmitted from the memory to the inference chip. Thereafter, the sparsely processed model data can be restored in the inference chip to obtain the restored target sub-model.

[0159] Through the above-mentioned sparse processing, the memory bandwidth occupied during transmission can be reduced.

[0160] Third, cache affinity optimization.

[0161] When the inference chip loads target sub-model related data from the CPU memory, it can first load the data into the cache. When it needs to access certain data in the future, it can first check whether the data is already in the cache. If it is in the cache, it can read the data directly from the cache, which is much faster than reading data from the memory.

[0162] According to the locality principle (temporal locality and spatial locality) of program access to data, temporal locality means that the data that was recently accessed is likely to be accessed again in the near future; spatial locality means that after a data item is accessed, its adjacent data items may also be accessed.

[0163] On this basis, through reasonable data layout, such as placing data that are frequently accessed together in adjacent memory locations, the cache hit rate can be improved.

[0164] Specifically, by adjusting the storage location of data in memory, the cache can be used more effectively when accessing the data. When a certain data needs to be accessed, it is first checked whether the data is already in the cache (i.e., cache hit). If the data is in the cache, it can be read directly from the cache, which is much faster than reading from memory.

[0165] In order to improve the hit rate of the cache, the embodiment of the present application considers placing the data that are frequently accessed together in adjacent memory locations. In this way, when one of the data is accessed, it is likely that the entire cache line (usually 64 bytes) containing the data will be loaded into the cache. Since these data are adjacent in the memory, they are likely to be located in the same or adjacent cache lines. When these data need to be accessed later, they are likely to be in the cache, thereby improving the hit rate of the cache.

[0166] In this embodiment, the process of storing the split sub-models into the memory of the terminal in the aforementioned step S110 may specifically include:

[0167] According to the inference target of the current inference task, the target arrangement of the data related to each sub-model in the memory is determined. For this inference target, the target arrangement has a higher cache affinity than other arrangements. Further, each sub-model is stored according to the target arrangement.

[0168] In this embodiment, a memory arrangement method with higher affinity for the inference target cache of the inference chip is constructed to improve the efficiency of memory reuse and increase the memory efficiency under the spatiotemporal reuse of memory.

[0169] Reference Figure 8As shown in , assume that the reasoning goal of the current reasoning task is to calculate the product of matrix A and matrix B. Based on this reasoning goal, a memory arrangement with higher cache affinity can be constructed. The memory arrangement after cache affinity optimization is as follows: Figure 8 as shown in .

[0170] The large model reasoning device provided in the embodiment of the present application is described below. The large model reasoning device described below and the large model reasoning method described above can be referenced to each other.

[0171] See also Fig. 9 , Fig. 9 A schematic diagram of the structure of a large model reasoning device disclosed in an embodiment of the present application.

[0172] like Fig. 9 As shown, the device may include:

[0173] A to-be-deployed model determination unit 11, used to determine a sparse large model to be deployed on the terminal side;

[0174] A model splitting unit 12 is used to split the sparse large model into a plurality of sub-models and store them in the memory of the terminal;

[0175] The model loading unit 13 is used to load the target sub-model from the memory to the inference chip according to the target sub-model required by the current inference task when executing a large model inference task, and use the loaded target sub-model to execute the inference task.

[0176] In a possible implementation, the sparse large model is a hybrid expert large model, and the process of the model splitting unit splitting the sparse large model into a plurality of sub-models includes:

[0177] Analyze the hybrid expert macromodel to obtain various operators included in the macromodel and the structural relationships between the operators;

[0178] According to the structural relationship between operators, searching starts from the input of the hybrid expert large model to identify the target gating operator, and there are more than two feedforward neural networks behind the target gating operator;

[0179] Splitting the hybrid expert large model from input to the target gating operator into head sub-models;

[0180] Continue to search backward and identify that the outputs of more than two feedforward networks are concentrated on the same weight summation operator. Perform splitting before the weight summation operator, and use each path from the weight summation operator to the target gating operator as an expert sub-model. Split the weight summation operator and the subsequent part into tail sub-models.

[0181] In a possible implementation, the large model reasoning task includes a pre-processing process of the reasoning task and a large model reasoning calculation process. When the model loading unit executes the large model reasoning task, the process of loading the target sub-model from the memory to the reasoning chip according to the target sub-model required by the current reasoning task includes:

[0182] In response to the inference task start instruction, while executing the pre-processing of the inference task, the target sub-model is pre-loaded from the memory to the inference chip according to the target sub-model required by the current inference task. The pre-processing is the process of pre-processing the input data before executing the large model inference calculation.

[0183] In a possible implementation, when the model loading unit executes a large model reasoning task, the process of loading the target sub-model from the memory to the reasoning chip according to the target sub-model required by the current reasoning task includes:

[0184] When executing a large model reasoning task, the relevant data of the target sub-model is sparsely processed in the memory according to the target sub-model required by the current reasoning task, the sparsely processed model data is transferred from the memory to the reasoning chip, and the sparsely processed model data is restored in the reasoning chip to obtain the restored target sub-model.

[0185] In a possible implementation, the process in which the model splitting unit stores each split sub-model into the memory of the terminal includes:

[0186] According to the inference target of the current inference task, determine the target arrangement mode of the data related to each sub-model in the memory, and for the inference target, the target arrangement mode has a higher cache affinity than other arrangements;

[0187] Each sub-model is stored according to the target arrangement.

[0188] In a possible implementation, the device of the present application may further include:

[0189] The first processing unit is used to perform pruning and / or quantization processing on the sparse large model before splitting the sparse large model into a plurality of sub-models to obtain a processed sparse large model. On this basis, the process of the model splitting unit splitting the sparse large model into a plurality of sub-models specifically includes:

[0190] The processed sparse large model is split into several sub-models.

[0191] In a possible implementation, the device of the present application may further include:

[0192] The second processing unit is used to perform pruning and / or quantization processing on each of the split sub-models after splitting the sparse large model into several sub-models and before storing them in the terminal's memory, so as to obtain each processed sub-model for storage in the terminal's memory.

[0193] The present application also provides an electronic device in an embodiment. Fig.10 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, tablet computers, teaching large screens, wearable devices, etc. Fig.10 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0194] like Fig.10 As shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603 to implement the large model reasoning method of the aforementioned embodiment of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0195] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Fig.10 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0196] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any large model reasoning method provided in the embodiment of the present application.

[0197] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When one or more computer programs are executed by an electronic device, the electronic device can implement any large model reasoning method provided in the embodiment of the present application.

[0198] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0199] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0200] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0201] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0202] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

Claims

1. A large model reasoning method, characterized in that: include: Determine the sparse large model to be deployed on the client side; The sparse large model is a mixed expert large model; Analyze the hybrid expert macromodel to obtain various operators included in the macromodel and the structural relationships between the operators; According to the structural relationship between operators, searching starts from the input of the hybrid expert large model to identify the target gating operator, and there are more than two feedforward neural networks behind the target gating operator; Splitting the hybrid expert large model from input to the target gating operator into head sub-models; Continue to search backward, identify that the outputs of more than two feedforward networks are concentrated on the same weight summation operator, split before the weight summation operator, use each path from the weight summation operator to the target gating operator as an expert sub-model, split the weight summation operator and the subsequent part into tail sub-models, and store the obtained sub-models in the memory of the terminal; When executing a large model reasoning task, the target sub-model is loaded from the memory to the reasoning chip according to the target sub-model required by the current reasoning task, and the reasoning task is executed in the reasoning chip using the loaded target sub-model.

2. The method according to claim 1, characterized in that The large model reasoning task includes a pre-processing process of the reasoning task and a large model reasoning calculation process; When executing a large model reasoning task, the process of loading the target sub-model from the memory to the reasoning chip according to the target sub-model required by the current reasoning task includes: In response to the inference task start instruction, while executing the pre-processing of the inference task, the target sub-model is pre-loaded from the memory to the inference chip according to the target sub-model required by the current inference task. The pre-processing is the process of pre-processing the input data before executing the large model inference calculation.

3. The method according to claim 1, characterized in that When executing a large model reasoning task, the process of loading the target sub-model from the memory to the reasoning chip according to the target sub-model required by the current reasoning task includes: When executing a large model reasoning task, the relevant data of the target sub-model is sparsely processed in the memory according to the target sub-model required by the current reasoning task, the sparsely processed model data is transferred from the memory to the reasoning chip, and the sparsely processed model data is restored in the reasoning chip to obtain the restored target sub-model.

4. The method according to claim 1, characterized in that: The process of storing the split sub-models into the terminal's memory includes: According to the inference target of the current inference task, determine the target arrangement mode of the data related to each sub-model in the memory, and for the inference target, the target arrangement mode has a higher cache affinity than other arrangements; Each sub-model is stored according to the target arrangement.

5. The method according to any one of claims 1 to 4, characterized in that: Before splitting the sparse large model into a plurality of sub-models, the method further includes: Performing pruning and / or quantization processing on the sparse large model to obtain a processed sparse large model; The process of splitting the sparse large model into several sub-models specifically includes: The processed sparse large model is split into several sub-models.

6. The method according to any one of claims 1 to 4, characterized in that: After splitting the sparse large model into a plurality of sub-models and before storing them in the memory of the terminal, the method further includes: Pruning and / or quantization are performed on each of the split sub-models to obtain each processed sub-model for storage in the memory of the terminal.

7. A large model reasoning device, characterized in that: include: A to-be-deployed model determination unit, used to determine a sparse large model to be deployed on the terminal side; The sparse large model is a mixed expert large model; The model splitting unit is used to parse the hybrid expert large model to obtain the various operators contained in the large model and the structural relationship between the operators; according to the structural relationship between the operators, start searching from the input of the hybrid expert large model to identify the target gating operator, and there are more than two feedforward neural networks after the target gating operator; divide the hybrid expert large model from the input to the target gating operator into a head sub-model; continue to search backward, identify that the outputs of more than two feedforward networks are concentrated on the same weight summation operator, split before the weight summation operator, and take each path from the weight summation operator to the target gating operator as an expert sub-model, divide the weight summation operator and the subsequent part into tail sub-models, and store the obtained sub-models in the memory of the terminal; The model loading unit is used to load the target sub-model from the memory to the inference chip according to the target sub-model required by the current inference task when executing a large model inference task, and perform the inference task in the inference chip using the loaded target sub-model.

8. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the large model reasoning method as described in any one of claims 1 to 6.

9. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the large model reasoning method as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the large model reasoning method as described in any one of claims 1 to 6 is implemented.

Images