A Heterogeneous Scheduling Method and System for Mobile LLM Based on Hybrid Parallelism

Abstract

The application discloses a mobile terminal LLM heterogeneous scheduling method based on hybrid parallelism, comprising: obtaining a scheduling plan; performing model inference based on an offline transformed large language model (LLM) according to the obtained scheduling plan; wherein the offline transformation of the large language model (LLM) comprises: pre-executing a benchmark test task to determine the computing power ratio of a mobile terminal heterogeneous computing device; extracting static information of the large language model (LLM), combining the determined computing power ratio, formulating a hybrid parallel strategy, and dividing the large language model (LLM) into multiple computing blocks; setting a synchronization point for each computing block, integrating the computing results, and completing the offline transformation of the original computing graph of the large language model (LLM) to a hybrid parallel computing graph. The application applies the hybrid parallel strategy to the mobile terminal heterogeneous scene, improves the inference performance of the LLM on the mobile terminal, and solves the problems of insufficient use of the existing inference framework computing device and imbalance between heterogeneous devices.

Description

Technical Field

[0001] This invention belongs to the field of deep learning inference optimization technology, and relates to a heterogeneous scheduling method for mobile large language model (LLM) based on hybrid parallelism, and particularly to an LLM inference optimization method for mobile heterogeneous computing devices (such as CPU, GPU, DSP, NPU). Background Technology

[0002] With the widespread application of Large Language Models (LLMs) in natural language processing tasks, efficiently performing LLM inference on resource-constrained mobile devices has become a significant challenge. Mobile devices are typically equipped with various heterogeneous computing units (such as CPUs, GPUs, DSPs, and NPUs), which vary significantly in terms of computing power, memory bandwidth, and energy efficiency. Traditional single-device inference methods cannot fully utilize these heterogeneous resources, resulting in low inference efficiency.

[0003] Existing LLM inference optimization methods mainly focus on techniques such as model pruning, quantization, and knowledge distillation, but these methods often overlook the parallel computing potential of heterogeneous devices. Therefore, an LLM inference scheduling method that can adapt to the characteristics of heterogeneous mobile devices is needed to maximize computational efficiency and reduce inference latency.

[0004] The terms used in this invention specification have the following meanings:

[0005] CPU: Central Processing Unit;

[0006] GPU: Graphics Processing Unit;

[0007] DSP: Digital Signal Processor;

[0008] NPU: Neural Network Processing Unit;

[0009] LLM: Large Language Model;

[0010] MHA: Multi-Head Attention;

[0011] MLP: Multi-layer Perceptron;

[0012] Softmax: Normalized exponential function;

[0013] LayerNorm: Layer normalization;

[0014] Concat operation: merge operation;

[0015] Reduce operation: an accumulation operation. Summary of the Invention

[0016] To address the issues of insufficient utilization of computing devices and imbalance between heterogeneous devices in existing inference frameworks, this invention provides a heterogeneous scheduling method for mobile LLM based on hybrid parallelism. By applying the hybrid parallelism strategy to heterogeneous mobile scenarios, the inference performance of LLM on mobile devices is improved.

[0017] According to one aspect of the present invention, a heterogeneous scheduling method for mobile LLM based on hybrid parallelism is provided, comprising:

[0018] Obtain the scheduling plan;

[0019] According to the obtained scheduling plan, model inference is performed based on the offline transformed Large Language Model (LLM); wherein, the offline transformation of the Large Language Model (LLM) includes:

[0020] Pre-execute benchmark testing tasks to determine the computing power ratio of heterogeneous computing devices on mobile devices;

[0021] Extract static information from the large language model LLM, and combine it with a determined computing power ratio to formulate a hybrid parallel strategy to divide the large language model LLM into multiple computing blocks;

[0022] Synchronization points are set for each computation block, and computation results are integrated to complete the offline transformation of the original computation graph of the Large Language Model (LLM) into a hybrid parallel computation graph.

[0023] As a further technical solution, the offline transformation of the Large Language Model (LLM) also includes:

[0024] Create an actuator unit for each computing device, and each actuator unit executes a set of computational subgraphs.

[0025] As a further technical solution, a hybrid parallel strategy is formulated, including:

[0026] Multi-head attention (MHA) input projection block: The weight matrix of query Q, key K, and value V is divided according to the number of attention heads and the computing power ratio. Each computing device corresponds to a portion of the matrix and performs the corresponding general matrix multiplication.

[0027] Multi-head attention (MHA) attention block: merges the results of query Q, key K, and value V obtained from the input projection block and performs normalized exponential function (Softmax function) calculation;

[0028] Multi-head attention MHA output projection block: The output weight matrix is ​​divided according to the number of attention heads and the computing power ratio. Each computing device corresponds to a portion of the matrix and performs the corresponding general matrix multiplication.

[0029] Multilayer Perceptron (MLP) blocks: The up-dimensional matrix and the down-dimensional matrix are divided according to the column and row dimensions, respectively. Each computing device is assigned and performs its own general matrix multiplication and activation function calculation.

[0030] Before the input of Multi-Head Attention (MHA), after the output of Multi-Layer Perceptron (MLP), and at the junction of MHA and MLP: Sequence-level parallelism is adopted. Based on the computing power ratio of the devices, the sequence is divided according to the sequence dimension, and each device performs its own layer normalization and residual connection calculation.

[0031] As a further technical solution, a synchronization point is set for each computation block to integrate the computation results, including:

[0032] For blocks employing tensor parallelism, an accumulation operation is inserted at the end to aggregate the computation results from different computing devices, and the aggregated result is used as the input for subsequent blocks.

[0033] For blocks that use sequence-level parallelism, a merge operation is inserted at the end to concatenate the computation results.

[0034] As a further technical solution, static information of the large language model LLM is extracted, including:

[0035] Analyze the large language model LLM to extract static information, including model hierarchical dependencies, the number of parameters in computationally intensive modules, and the input and output dimensions.

[0036] As a further technical solution, when performing model inference based on the offline-transformed large language model LLM, it also includes:

[0037] The computational subgraph is divided based on the established parallel strategy. The divided computational subgraphs are mapped to the corresponding executor units according to their division shares. The computational subgraphs corresponding to the synchronization points are executed by the CPU executor unit by default.

[0038] As a further technical solution, when performing model inference based on the offline-transformed large language model LLM, it also includes:

[0039] At the synchronization point, after all actuator units have completed their computation tasks, the computation results are integrated.

[0040] According to one aspect of the present invention, a heterogeneous scheduling system for mobile LLM based on hybrid parallelism is provided, comprising:

[0041] The acquisition module is used to obtain the scheduling plan;

[0042] The heterogeneous scheduling module is used to perform model inference based on the offline-transformed Large Language Model (LLM) according to the acquired scheduling plan; wherein, the offline transformation of the Large Language Model (LLM) includes:

[0043] Pre-execute benchmark testing tasks to determine the computing power ratio of heterogeneous computing devices on mobile devices;

[0044] Extract static information from the large language model LLM, and combine it with a determined computing power ratio to formulate a hybrid parallel strategy to divide the large language model LLM into multiple computing blocks;

[0045] Synchronization points are set for each computation block, and computation results are integrated to complete the offline transformation of the original computation graph of the Large Language Model (LLM) into a hybrid parallel computation graph.

[0046] According to one aspect of the present invention, an execution device is provided, including an abstract interface to heterogeneous memory and a processor, wherein the memory stores data required for computational tasks associated with the processor, and the processor executes an allocated computational subgraph to perform the aforementioned hybrid parallel-based mobile LLM heterogeneous scheduling method.

[0047] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0048] This invention innovatively applies a hybrid parallel strategy to heterogeneous mobile scenarios, solving problems such as insufficient utilization of computing devices and imbalance between heterogeneous devices in existing inference frameworks, thereby improving the inference performance of LLM on mobile devices. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating the heterogeneous scheduling method for mobile LLM based on hybrid parallelism provided in an embodiment of the present invention.

[0051] Figure 2 This is a schematic diagram of the offline transformation process of the Large Language Model (LLM) provided in an embodiment of the present invention.

[0052] Figure 3 This is a schematic diagram illustrating the offline transformation principle of the Large Language Model (LLM) provided in an embodiment of the present invention.

[0053] Figure 4This is a schematic diagram of a heterogeneous scheduling system for mobile LLM based on hybrid parallelism, provided in an embodiment of the present invention. Detailed Implementation

[0054] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0056] This invention discloses a heterogeneous scheduling method for mobile LLM based on hybrid parallelism. By innovatively applying the hybrid parallelism strategy to the heterogeneous scenario of mobile devices, it solves the problems of insufficient utilization of computing devices and imbalance between heterogeneous devices in existing inference frameworks, thereby improving the inference performance of LLM on mobile devices.

[0057] It should be noted that hybrid parallelism is a method that combines tensor-level parallelism and sequence-level parallelism. Tensor-level parallelism is the parallel computation of tensor computations (such as matrix multiplication) in LLM by dividing the weight matrix according to a certain proportion and distributing it to different computing devices. Sequence-level parallelism divides the input sequence dimension to multiple devices and is an extension of data parallelism.

[0058] The scheduling method described in this invention first acquires the static information of the model and uses the pre-execution of benchmark data to determine the computing power ratio of different computing devices. Then, based on the computing power ratio, a partitioning strategy is formulated for the model to be executed. Following this, the model is divided into different blocks according to the LLM structure, and different types of parallel strategies are executed for different blocks. For MHA and MLP, tensor-level parallelism is the primary strategy, while for other connection blocks, sequence-level parallelism is the primary strategy. A synchronization point is set at the end of each block to integrate the computation results.

[0059] Please see Figure 1 The mobile terminal LLM heterogeneous scheduling method based on hybrid parallelism provided in this embodiment of the invention includes the following steps:

[0060] Step S1: Obtain the scheduling plan.

[0061] Step S2: Based on the obtained scheduling plan, perform model inference based on the offline transformed Large Language Model (LLM).

[0062] During model inference, computational subgraphs are partitioned based on the parallel strategy established during offline transformation of the Large Language Model (LLM). For tensor-level parallel components such as MHA and MLP, each weight matrix and its associated local computation correspond to a set of computational subgraphs. These subgraphs are then mapped to the corresponding executor units according to their partition shares. The same applies to modules related to sequence-level parallelism. The subgraph corresponding to a synchronization point is executed by the CPU executor unit by default. At a synchronization point, execution must wait for all other executor units to complete their computational tasks before proceeding.

[0063] Execute step S2 layer by layer until the model inference is complete.

[0064] In this embodiment of the invention, when performing model inference, the model inference is performed by binding the executor to the subgraph according to the scheduling plan.

[0065] Please see Figure 2 The offline transformation of the Large Language Model (LLM) provided in this embodiment of the invention includes the following steps:

[0066] Step 1: Pre-execute benchmark tasks to evaluate the computing power ratio of mobile heterogeneous computing devices (CPU, GPU, DSP, NPU) by measuring the execution time on each device.

[0067] Step 2: Analyze the LLM model structure and extract static information, including model hierarchical dependencies, the number of parameters and input / output dimensions of computationally intensive modules (MHA, MLP).

[0068] Step 3: Based on the computing power ratio and model structure, formulate a hybrid parallel strategy to divide the LLM into multiple computing blocks, with the partitioning method as follows: Figure 3 As shown.

[0069] For the MHA part, it is divided into an input projection block, an attention block, and an output projection block. The input projection block divides the weight matrices of Q, K, and V according to the number of attention heads and based on the computing power ratio. Each computing device corresponds to a portion of the matrix and performs the corresponding general matrix multiplication. The attention block merges the calculation results of Q, K, and V from the input projection block and performs normalized exponential function (Softmax function) calculation. The output projection block divides the output weight matrix according to the number of attention heads and based on the computing power ratio. Using the result of the attention block as input, each computing device corresponds to a portion of the matrix and performs the corresponding general matrix multiplication to complete the remaining output projection calculation of the MHA block.

[0070] For MLP blocks, the increased-dimensional matrix and the reduced-dimensional matrix are divided according to the column and row dimensions respectively. After each computing device is assigned, its own general matrix multiplication and activation function calculation are performed.

[0071] For the part before MHA input, after MLP output, and at the boundary between MHA and MLP, sequence-level parallelism is adopted. Based on the computing power ratio of the devices, the sequence is divided according to the sequence dimension of the sequence, and each device performs its own LayerNorm and residual connection calculations.

[0072] Step 4: Set a synchronization point for each computation block and integrate the computation results through shared memory; for blocks that use tensor-level parallelism, such as MHA and MLP, insert a Reduce operation at the end to aggregate the computation results from different devices, and the aggregated result will be used as the input for subsequent blocks; for other blocks that use sequence-level parallelism, insert a Concat operation at the end to concatenate the computation results.

[0073] After steps 3 and 4, the original computation graph has been modified to include a hybrid parallel computing mode; this part of the work is entirely static. That is, in steps 3 and 4, the original model structure is transformed offline.

[0074] Step 5: Create an executor unit for each computing device. Each executor unit can execute a set of computational subgraphs. These include executor units for CPU, GPU, DSP, NPU, etc.

[0075] At this point, the content of steps 1-5 completely corresponds to... Figure 2 The process is as follows, and steps 1-5 are processed offline.

[0076] After the above steps, the computation ratio of heterogeneous devices is determined by pre-executing benchmark tasks, and then the model structure is transformed into a structure that includes parallel mode and memory partitioning is performed. Finally, model inference is executed.

[0077] The implementation of the various embodiments of the present invention is based on programmed processing by a device with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of the present invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of the present invention provide a hybrid parallel mobile LLM heterogeneous scheduling system, which is used to execute the hybrid parallel mobile LLM heterogeneous scheduling method in the above method embodiments.

[0078] See Figure 4 The system includes: an acquisition module for acquiring a scheduling plan; and a heterogeneous scheduling module for performing model inference based on the acquired scheduling plan and the offline-transformed Large Language Model (LLM). The offline transformation of the LLM includes: pre-executing a benchmark test task to determine the computing power ratio of the mobile heterogeneous computing devices; extracting the static information of the LLM, and, based on the determined computing power ratio, formulating a hybrid parallel strategy to divide the LLM into multiple computing blocks; setting synchronization points for each computing block, integrating the computational results, and completing the offline transformation of the original computational graph of the LLM into a hybrid parallel computational graph.

[0079] The mobile LLM heterogeneous scheduling system based on hybrid parallelism provided in this invention addresses the problems of insufficient utilization of computing devices and imbalance between heterogeneous devices in existing inference frameworks. Figure 4 Several modules within the framework improve the inference performance of LLM on mobile devices by applying a hybrid parallel strategy to heterogeneous mobile scenarios.

[0080] It should be noted that the system embodiments provided by this invention, in addition to implementing the methods in the above method embodiments, are also used to implement the methods in other method embodiments provided by this invention. The difference lies only in setting corresponding functional modules, and their principles are basically the same as those of the above system embodiments provided by this invention. As long as those skilled in the art, based on the above system embodiments and referring to the specific technical solutions in other method embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and improve the modules in the above system embodiments while ensuring the practicality of the technical solutions, they can obtain corresponding system-like embodiments for implementing the methods in other method-like embodiments. For example:

[0081] Based on the above system embodiments, as a preferred embodiment, the mobile LLM heterogeneous scheduling system based on hybrid parallelism provided in this embodiment of the invention further includes offline transformation of the large language model LLM, which further includes:

[0082] Create an actuator unit for each computing device, and each actuator unit executes a set of computational subgraphs.

[0083] Based on the above system embodiments, as a preferred embodiment, the mobile LLM heterogeneous scheduling system based on hybrid parallelism provided in this embodiment of the invention further includes, in the offline transformation of the large language model LLM, a hybrid parallel strategy is formulated, and the following is also included:

[0084] Multi-head attention (MHA) input projection block: The weight matrix of query Q, key K, and value V is divided according to the number of attention heads and the computing power ratio. Each computing device corresponds to a portion of the matrix and performs the corresponding general matrix multiplication.

[0085] Multi-head attention (MHA) attention block: merges the results of query Q, key K, and value V obtained from the input projection block and performs normalized exponential function (softmax function) calculation;

[0086] Multi-head attention MHA output projection block: The output weight matrix is ​​divided according to the number of attention heads and the computing power ratio. Each computing device corresponds to a portion of the matrix and performs the corresponding general matrix multiplication.

[0087] Multilayer Perceptron (MLP) blocks: The up-dimensional matrix and the down-dimensional matrix are divided according to the column and row dimensions, respectively. Each computing device is assigned and performs its own general matrix multiplication and activation function calculation.

[0088] Before the input of Multi-Head Attention (MHA), after the output of Multi-Layer Perceptron (MLP), and at the junction of MHA and MLP: Sequence-level parallelism is adopted. Based on the computing power ratio of the devices, the sequence is divided according to the sequence dimension, and each device performs its own layer normalization and residual connection calculation.

[0089] Based on the above system embodiments, as a preferred embodiment, the mobile LLM heterogeneous scheduling system based on hybrid parallelism provided in this embodiment of the invention, in the offline transformation of the Large Language Model (LLM), sets a synchronization point for each computation block and integrates the computation results, and further includes:

[0090] For blocks employing tensor parallelism, an accumulation operation is inserted at the end to aggregate the computation results from different computing devices, and the aggregated result is used as the input for subsequent blocks.

[0091] For blocks that use sequence-level parallelism, a merge operation is inserted at the end to concatenate the computation results.

[0092] Based on the above system embodiments, as a preferred embodiment, the mobile LLM heterogeneous scheduling system based on hybrid parallelism provided in this embodiment of the invention further includes extracting the static information of the large language model LLM during offline transformation of the large language model LLM, and also includes:

[0093] Analyze the large language model LLM to extract static information, including model hierarchical dependencies, the number of parameters in computationally intensive modules, and the input and output dimensions.

[0094] Based on the above system embodiments, as a preferred embodiment, the mobile terminal LLM heterogeneous scheduling system based on hybrid parallelism provided in this embodiment of the invention further includes a heterogeneous scheduling module configured to execute the following instructions:

[0095] The computational subgraph is divided based on the established parallel strategy. The divided computational subgraphs are mapped to the corresponding executor units according to their division shares. The computational subgraphs corresponding to the synchronization points are executed by the CPU executor unit by default.

[0096] Based on the above system embodiments, as a preferred embodiment, the mobile terminal LLM heterogeneous scheduling system based on hybrid parallelism provided in this embodiment of the invention further includes a heterogeneous scheduling module configured to execute the following instructions:

[0097] At the synchronization point, after all actuator units have completed their computation tasks, the computation results are integrated.

[0098] Based on the same inventive concept as the foregoing embodiments, this embodiment of the invention also provides an execution device, namely an abstract interface for heterogeneous memory and processors, wherein the memory stores data required for computational tasks in the associated processor, and the processor executes the allocated computational subgraph to execute the aforementioned mobile LLM heterogeneous scheduling method based on hybrid parallelism.

[0099] In embodiments of the present invention, the memory can be non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), or it can be volatile memory, such as random-access memory (RAM). Memory is any other medium capable of carrying or storing desired program code having an instruction or data structure form and accessible by a computer, but is not limited thereto. The memory in embodiments of the present invention can also be a circuit or any other device capable of implementing a storage function for storing program instructions and / or data.

[0100] In this embodiment of the invention, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in this embodiment of the invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in this embodiment of the invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0101] Based on the same inventive concept as the foregoing embodiments, the computer instructions cause the computer to execute the hybrid parallel-based mobile LLM heterogeneous scheduling method as follows:

[0102] Obtain the scheduling plan;

[0103] According to the obtained scheduling plan, model inference is performed based on the offline transformed Large Language Model (LLM); wherein, the offline transformation of the Large Language Model (LLM) includes:

[0104] Pre-execute benchmark testing tasks to determine the computing power ratio of heterogeneous computing devices on mobile devices;

[0105] Extract static information from the large language model LLM, and combine it with a determined computing power ratio to formulate a hybrid parallel strategy to divide the large language model LLM into multiple computing blocks;

[0106] Synchronization points are set for each computation block, and computation results are integrated to complete the offline transformation of the original computation graph of the Large Language Model (LLM) into a hybrid parallel computation graph.

[0107] In summary, this invention discloses a heterogeneous scheduling method for mobile LLM based on hybrid parallelism. First, static information of the model is acquired, and the computing power ratio of different computing devices is determined using pre-execution of benchmark data. Then, a partitioning strategy is formulated for the model to be executed based on the computing power ratio. Next, the model is divided into different blocks according to the LLM structure, and different types of parallelism strategies are executed for different blocks. Tensor-level parallelism is mainly used for MHA and MLP, while sequence-level parallelism is mainly used for other connection blocks. Synchronization points are set at the end of each block to integrate the computation results. This invention innovatively applies a hybrid parallelism strategy to a heterogeneous mobile scenario, solving problems such as insufficient utilization of computing devices and imbalance between heterogeneous devices in existing inference frameworks, thus improving the inference performance of LLM on mobile devices.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A mobile terminal LLM heterogeneous scheduling method based on hybrid parallel, characterized in that, The method comprises the following steps: obtaining a scheduling plan; based on the obtained scheduling plan, performing model inference based on the offline transformed large language model LLM; wherein the offline transformation of the large language model LLM comprises: pre-executing a benchmark test task to determine the computing power ratio of the mobile heterogeneous computing device; extracting static information of the large language model LLM, and formulating a hybrid parallel strategy based on the determined computing power ratio to divide the large language model LLM into multiple computing blocks; wherein formulating the hybrid parallel strategy comprises: a multi-head attention MHA input projection block: the weight matrix of the query Q, key K and value V is divided according to the number of attention heads and given the computing power ratio, each computing device corresponds to a divided part and performs corresponding general matrix multiplication; a multi-head attention MHA attention block: the results of the query Q, key K and value V obtained in the input projection block are combined and normalized exponential function calculation is performed; a multi-head attention MHA output projection block: the output weight matrix is divided according to the number of attention heads and given the computing power ratio, each computing device corresponds to a divided part and performs corresponding general matrix multiplication; a multi-layer perception MLP block: the dimension increasing matrix and the dimension decreasing matrix are respectively divided according to the column and row dimensions, and each computing device is allocated to perform respective general matrix multiplication and activation function calculation; the part before the multi-head attention MHA input, after the multi-layer perception MLP output and at the junction of the multi-head attention MHA and the multi-layer perception MLP: sequence-level parallel is adopted, the sequence dimension is divided according to the device computing power ratio, and each device respectively performs respective layer normalization and residual connection calculation; setting a synchronization point for each computing block, integrating the calculation results, and completing the offline transformation of the original computation graph of the large language model LLM to the hybrid parallel computation graph.

2. The mobile end LLM heterogeneous scheduling method based on hybrid parallel according to claim 1, characterized in that, The offline transformation of the large language model LLM further comprises: creating an executor unit for each computing device, and each executor unit executes a group of computing subgraphs.

3. The heterogeneous scheduling method for mobile end LLM based on hybrid parallelism according to claim 1, characterized in that, setting a synchronization point for each computing block, integrating the calculation results, comprising: for the blocks using tensor-level parallelism, an accumulation operation is inserted at the end to aggregate the calculation results of different computing devices, and the aggregated results are used as the input of the subsequent blocks; for the blocks using sequence-level parallelism, a merging operation is inserted at the end to splice the calculation results.

4. The mobile end LLM heterogeneous scheduling method based on hybrid parallel according to claim 1, characterized in that, extracting static information of the large language model LLM comprises: parsing the large language model LLM to extract static information, including model level dependency relationship, parameter quantity and input / output dimension of computation-intensive module.

5. The heterogeneous scheduling method for mobile LLM based on hybrid parallelism according to claim 1, wherein, When performing model inference based on the offline transformed large language model LLM, it further comprises: based on the formulated parallel strategy, dividing the computing subgraphs, mapping the divided computing subgraphs into corresponding executor units according to the division share, and the computing subgraphs corresponding to the synchronization points are executed by the CPU executor unit by default.

6. The mobile end LLM heterogeneous scheduling method based on hybrid parallel according to claim 5, characterized in that, When performing model inference based on the offline transformed large language model LLM, it further comprises: at the synchronization point, waiting for all executor units to complete the calculation task, and then integrating the calculation results.

7. A mobile terminal LLM heterogeneous scheduling system based on hybrid parallel, characterized in that, The method comprises the following steps: obtaining a scheduling plan; The heterogeneous scheduling module is configured to perform model inference based on the obtained scheduling plan and the offline transformed large language model (LLM). Pre-execution benchmarking tasks are performed to determine the computing power ratio of the mobile heterogeneous computing device. Static information of the large language model (LLM) is extracted, and a hybrid parallel strategy is formulated based on the determined computing power ratio to divide the large language model (LLM) into multiple computing blocks. The hybrid parallel strategy includes: a multi-head attention (MHA) input projection block that splits the weight matrix of the query (Q), key (K), and value (V) according to the number of attention heads and the computing power ratio, each computing device corresponding to a split portion and performing the corresponding general matrix multiplication; a multi-head attention (MHA) attention block that combines the results of the query (Q), key (K), and value (V) obtained in the input projection block and performs normalization exponential function calculation; a multi-head attention (MHA) output projection block that splits the output weight matrix according to the number of attention heads and the computing power ratio, each computing device corresponding to a split portion and performing the corresponding general matrix multiplication; a multi-layer perception (MLP) block that splits the dimension-increasing matrix and the dimension-reducing matrix according to the column and row dimensions, respectively, each computing device being assigned to perform the corresponding general matrix multiplication and activation function calculation; the parts before the multi-head attention (MHA) input, after the multi-layer perception (MLP) output, and at the junction of the multi-head attention (MHA) and the multi-layer perception (MLP) adopt sequence-level parallelism, divide according to the sequence dimension based on the device computing power ratio, and each device performs the corresponding layer normalization and residual connection calculation.

8. An execution device, characterized by, Synchronization points are set for each computing block, the computing results are integrated, and the offline transformation of the original computation graph of the large language model (LLM) to the hybrid parallel computation graph is completed. An abstract interface for heterogeneous memory and processors is included, the memory stores data required for computing tasks in the associated processor, and the processor executes the allocated computing subgraph to perform the hybrid parallel-based mobile LLM heterogeneous scheduling method of any one of claims 1 to 6.

Images