Edge generation type large language model collaborative reasoning method for ubiquitous computing power network

Abstract

The invention provides an edge generation type large language model collaborative reasoning method for a ubiquitous computing power network. The edge generation type large language model collaborative reasoning method comprises the steps that generation type large language model information and edge device information are obtained; carrying out parallel reasoning modeling and solving on edge equipment assembly line arrangement and a generative large language model assembly line; performing parallel pre-filling based on an assembly line divided by the input sequence; the invention relates to assembly line parallel autoregression decoding based on an outline. According to the method, by means of computing resources of multi-edge-end equipment in a ubiquitous computing power network environment, a pre-filling stage of the generative large language model is accelerated through a sequence division-based pipeline parallel reasoning technology, and meanwhile, an autoregression decoding stage of the generative large language model is accelerated by utilizing an outline-based pipeline parallel decoding method; low-delay and resource-efficient distributed collaborative reasoning of the generative large language model on multiple edge devices is realized; the privacy data security of the user is effectively protected, and the method has better adaptability and robustness aiming at the resource characteristics of the complex edge environment.

Description

Technical Field

[0001] The present invention relates to the field of deep learning parallel reasoning technology, and in particular to an edge-generated large language model collaborative reasoning method for ubiquitous computing power networks. Background Art

[0002] Generative large language models have attracted widespread attention due to their outstanding capabilities in a variety of artificial intelligence and natural language processing tasks. These models are usually deployed in cloud data centers equipped with powerful server-class GPUs and have promoted the development of edge intelligence applications such as question-answering robots and smart home AI assistants.

[0003] Although generative large language models were originally created in data centers, due to the sensitivity and privacy of user data, in recent years, deploying generative large language models on more accessible edge platforms rather than uploading requests to the remote cloud of commercial companies has become an emerging trend. To ensure privacy protection, the generative large language models used by individual users should be fully or mainly deployed on local edge devices (such as personal consumer PC graphics cards, smartphones, laptops, smart home IoT devices, etc.) for reasoning.

[0004] However, deploying computationally intensive and resource-intensive generative large language models on the edge faces huge challenges. This is because the computing resources of a single edge device are very limited. Often, directly deploying generative large language models on these devices will lead to problems such as long inference delays and high memory usage. To solve the above problems, existing edge-side collaborative large language model inference systems often use the following four parallel inference architectures.

[0005] One is data parallelism; it divides the workload by the sample dimension, and each device maintains a complete copy of the model and performs inference independently. However, for single-sample inference requests (such as in a smart home environment, a user issues a task instruction to a smart assistant), data parallelism cannot utilize multiple edge devices at the same time.

[0006] The second is sequence parallelism; sequence parallelism divides the input sequence into multiple subsequences for parallel processing. However, sequence parallelism requires each device to maintain a complete model copy, which will cause huge memory overhead. At the same time, the sequence parallel reasoning process requires two tensor synchronizations in each Transformer decoder layer, which will introduce intolerable communication overhead.

[0007] The third is tensor parallelism. Tensor parallelism divides the weights of a large generative language model onto different devices, with each device retaining only some parameters. However, it is necessary to perform tensor synchronization after the multi-head attention module and the feedforward neural network module in each Transformer decoder layer. This also introduces intolerable communication overhead, making device communication in low-speed edge networks a bottleneck for the inference system.

[0008] The fourth is pipeline parallelism; pipeline parallelism divides the large language model horizontally into multiple continuous stages along the layer granularity. Each stage is mapped to an independent edge device, and multiple input samples are injected at the same time to improve parallelism. However, for single-sample user requests, pipeline parallelism will degenerate into serial reasoning and cannot utilize the computing resources of multiple edge devices at the same time. Summary of the invention

[0009] In view of the shortcomings of the prior art, the present invention provides an edge-based generative large language model collaborative reasoning method for ubiquitous computing networks. The present invention can achieve low-latency and resource-efficient distributed collaborative reasoning of generative large language models on multiple edge devices.

[0010] The technical solution of the present invention is: an edge-generated large language model collaborative reasoning method for ubiquitous computing power network, comprising the following steps:

[0011] S1), obtaining generative large language model information;

[0012] S2), collect edge device performance parameter information;

[0013] S3), modeling and solving edge device pipeline orchestration and generative large language model pipeline parallel reasoning;

[0014] S4), pipeline parallel pre-filling based on input sequence partitioning;

[0015] S5),Outline-based pipelined parallel autoregressive decoding.

[0016] Preferably, in step S1), the generative large language model is installed on each edge device to the local computing environment, and configuration parameters of the generative large language model are measured and collected.

[0017] Preferably, in step S1), the generative large language model inference task is performed offline using the test sample data, and the number of model layers of the generative large language model, the parameter information of each layer, and the memory space occupied by the parameters of each layer of the generative large language model and the intermediate features between adjacent layers during runtime are recorded; for a generative large language model containing L consecutive model layers, the memory space occupied by the parameters of the lth layer is w l .

[0018] Preferably, in step S2), each edge device performs the generative large language model reasoning task offline using the test input samples, records the relevant runtime data, and estimates the local computing resource parameters of the edge device.

[0019] Preferably, in step S2), the recorded runtime data includes the time required for the forward propagation of each Transformer decoder layer of the runtime generative large language model, and the ordered set of all edge devices participating in the collaborative reasoning is defined as Ordered Sets Sort by computing performance from weak to strong, and define D n ={d1,d2,…,d n} is an ordered set The first n edge devices in the subset; define is the execution time required to execute model layer l on edge device n.

[0020] Preferably, in step S3), the configuration parameters of the generative large language model collected in step S1) and step S2) and the performance parameter information of each edge device are combined; the generative large language model is modeled using pipeline parallel architecture reasoning, and a dynamic programming algorithm is designed to solve the optimal edge device pipeline orchestration scheme and generative large language model partitioning scheme; finally, according to the edge device pipeline orchestration scheme and model partitioning results, the edge device is deployed and the generative large language model is deployed.

[0021] Preferably, in step S3), the edge device pipeline is specifically arranged as follows:

[0022] The edge devices participating in collaborative reasoning are organized into ordered sets The edge devices are deployed in a pipeline form into multiple pipeline stages in the order of; and each edge device only communicates with the edge devices responsible for its previous stage and the next stage, and the total number of pipeline stages is equal to the total number of edge devices.

[0023] Preferably, in step S3), the generative large language model pipeline parallel reasoning is modeled and solved, specifically including the following steps:

[0024] S31) Divide all model layers of the generative large language model into multiple layer subsets connected in series, and define each layer subset as an independent stage; define the Transformer decoder layer in each stage as a stage model; and define A(i→j,D n ) is to divide the model layer i to model layer j into n stage models and assign them to the set D n The inference time required for the slowest model stage under the optimal segmentation scheme in the process;

[0025] The optimal segmentation solution is solved by dynamic programming algorithm, namely:

[0026]

[0027] In the formula, Indicates edge device d n The inference time required to infer the stage model from model layer i to model layer j;

[0028] If assigned to edge device d n If the total memory overhead required during the inference process of the stage model exceeds the memory budget of the edge device, then set T(i→j,n)=+∞;

[0029] S32), after completing the dynamic programming arrangement in step S31), all L-layer model layers of the generative large language model are divided into an ordered device set According to the obtained solution, the target generative large language model is cut along the model layer granularity, divided into multiple stage models and installed accordingly in the local environment of the edge device responsible for each stage.

[0030] Preferably, in step S31), the total memory overhead required during the inference process includes the generative large language model parameters and intermediate cache results generated during the inference process.

[0031] Preferably, in step S4), the pipeline parallel pre-filling based on the input sequence division specifically includes the following steps:

[0032] S41), cutting the natural language inference request sent by the user into multiple subsequences s1, s2, ..., s n , and multiple subsequences s1,s2,…,s n At the same time, it is injected into the pipeline of the edge device; the number of splits is set according to the needs. In this embodiment, the number of stages is doubled, and the pipeline is Figure 2 Phase 1 in.

[0033] S42), through the generative large language model for each subsequence s i Perform inference and convert each subsequence s i The intermediate results of the self-attention network layer reasoning in the generative large language model are cached for subsequent subsequence reasoning.

[0034] Preferably, in step S42), in the pipeline parallel reasoning process based on sequence partitioning, since the generative large language model includes a network layer of a self-attention mechanism, the network layer needs to calculate the attention value of each word in the reasoning input sequence and all the previous words, therefore, each subsequence s i In the inference calculation process, it depends on all the previous subsequences s1, s2, ...s i-1 The intermediate calculation results.

[0035] Preferably, in step S5), the generative large language model is prompted by a preset outline instruction to generate a response outline containing multiple key points for the target question, and the generative large language model is allowed to decode and expand the multiple key points in parallel in a pipeline parallel manner.

[0036] Preferably, step S5) specifically includes the following steps:

[0037] S51), presetting an outline generation instruction, in the pre-filling stage, combining the preset outline generation instruction with the user's question as an input of the generative large language model to guide the generative large language model to generate an outline for a reply answer;

[0038] S52), after the pre-filling stage is completed, the generative large language model is allowed to serially generate an outline of the answer, and each point of the answer outline is individually packaged into an independent decoding extension request, each request will prompt the generative large language model to perform decoding extension only for the specific outline point;

[0039] S53), injecting the decoding extension request into the edge collaborative reasoning pipeline to achieve efficient pipeline parallel processing;

[0040] S54), when all decoding extension requests are completed, the output results of each request are concatenated to obtain the final answer.

[0041] The beneficial effects of the present invention are:

[0042] 1. The present invention uses the computing resources of multiple edge devices in a ubiquitous computing network environment to accelerate the pre-filling stage of the generative large language model through a pipeline parallel reasoning technology based on sequence partitioning, and uses an outline-based pipeline parallel decoding method to accelerate the autoregressive decoding stage of the generative large language model, thereby achieving low-latency and resource-efficient distributed collaborative reasoning of the generative large language model on multiple edge devices;

[0043] 2. The present invention can retain the data generated by processing user data on the user device, effectively protecting the user's privacy data security. At the same time, it does not need to communicate a large amount of data with the cloud data center, avoiding the problem of unstable backbone network and unreliable service quality;

[0044] 3. The present invention has better adaptability and robustness to the resource characteristics of complex edge environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the process of the present invention;

[0046] Figure 2 A schematic diagram of the process of pre-filling a large language model in a pipeline parallel generation based on sequence partitioning according to the present invention;

[0047] Figure 3 It is a schematic diagram of the outline-based pipeline parallel autoregressive decoding method of the present invention. DETAILED DESCRIPTION

[0048] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings:

[0049] like Figure 1 As shown, this embodiment provides an edge-generative large language model collaborative reasoning method for a ubiquitous computing network, including the following steps:

[0050] S1), obtaining generative large language model information; this embodiment installs the generative large language model on each edge device to the local computing environment, and measures and collects the configuration parameters of the generative large language model;

[0051] By using the test sample data to perform the generative large language model inference task offline, and record the number of model layers of the generative large language model, the parameter information of each layer, and the memory space occupied by the parameters of each layer of the generative large language model and the intermediate features between adjacent layers during runtime; for a generative large language model containing L consecutive model layers, the memory space occupied by the parameters of the lth layer is w l .

[0052] S2), collect edge device information; each edge device of this embodiment uses the test input sample to perform the generative large language model reasoning task offline, record the relevant runtime data, and estimate the local computing resource parameters of the edge device;

[0053] The recorded runtime data includes the time required for the forward propagation of each Transformer decoder layer of the runtime generative large language model, and defines the ordered set of all edge devices participating in collaborative reasoning as Ordered Sets Sort by computing performance from weak to strong, and define D n ={d1,d2,…,d n} is an ordered set The first n edge devices in the subset; define is the execution time required to execute model layer l on edge device n.

[0054] S3), modeling and solving edge device pipeline orchestration and generative large language model pipeline parallel reasoning;

[0055] This embodiment combines the configuration parameters of the generative large language model collected in step S1) and step S2) and the performance parameter information of each edge device; the generative large language model is modeled using pipeline parallel architecture reasoning, and a dynamic programming algorithm is designed to solve the optimal edge device pipeline orchestration scheme and generative large language model partitioning scheme; finally, according to the edge device pipeline orchestration scheme and model partitioning results, the edge device is deployed and the generative large language model is deployed.

[0056] Among them, the edge device pipeline arrangement is specifically as follows: the edge devices participating in collaborative reasoning are arranged according to the ordered set The edge devices are deployed in a pipeline form into multiple pipeline stages in the order of; and each edge device only communicates with the edge devices responsible for its previous stage and the next stage, and the total number of pipeline stages is equal to the total number of edge devices.

[0057] Model and solve the parallel reasoning of the generative large language model pipeline, which includes the following steps:

[0058] S31) Divide all model layers of the generative large language model into multiple layer subsets connected in series, and define each layer subset as an independent stage; define the Transformer decoder layer in each stage as a stage model; and define A(i→j,D n ) is to divide the model layer i to model layer j into n stage models and assign them to the set D n The inference time required for the slowest model stage under the optimal segmentation scheme in the process;

[0059] The optimal segmentation solution is solved by dynamic programming algorithm, namely:

[0060]

[0061] In the formula, Indicates edge device d n The inference time required to infer the stage model from model layer i to model layer j;

[0062] If assigned to edge device d n If the total memory overhead required during the inference process exceeds the memory budget of the edge device, then set T(i→j,n)=+∞; the total memory overhead required during the inference process includes the parameters of the generative large language model. and intermediate cache results generated during the inference process.

[0063] S32), after completing the dynamic programming arrangement in step S31), all L-layer model layers of the generative large language model are divided into an ordered device set According to the obtained solution, the target generative large language model is cut along the model layer granularity, divided into multiple stage models and installed accordingly in the local environment of the edge device responsible for each stage.

[0064] S4), pipeline parallel pre-filling based on input sequence partitioning; Figure 2 As shown, the specific steps include:

[0065] S41), cutting the natural language inference request sent by the user into multiple subsequences s1, s2, ..., s n , and multiple subsequences s1,s2,…,s n Simultaneously injected into the pipeline of edge devices;

[0066] S42), through the generative large language model for each subsequence s i Perform inference and convert each subsequence s i The intermediate results of the self-attention network layer reasoning in the generative large language model are cached for subsequent subsequence reasoning. In this embodiment, in the pipeline parallel reasoning process based on sequence division, since the generative large language model contains a network layer with a self-attention mechanism, the network layer needs to calculate the attention value of each word in the reasoning input sequence and all the previous words. Therefore, each subsequence s i In the inference calculation process, it depends on all the previous subsequences s1, s2, ...s i-1 The intermediate calculation results.

[0067] S5), outline-based pipeline parallel autoregressive decoding; prompting the generative large language model to generate a response outline containing multiple key points for the target question through the preset outline instructions, and allowing the generative large language model to decode and expand multiple key points in parallel in the form of pipeline parallelism; specifically including the following steps:

[0068] S51), presetting an outline generation instruction, in the pre-filling stage, combining the preset outline generation instruction with the user's question as an input of the generative large language model to guide the generative large language model to generate an outline for a reply answer;

[0069] S52), after the pre-filling stage is completed, the generative large language model is allowed to serially generate an outline of the answer, and each point of the answer outline is individually packaged into an independent decoding extension request, each request will prompt the generative large language model to perform decoding extension only for the specific outline point;

[0070] S53), injecting the decoding extension request into the edge collaborative reasoning pipeline to achieve efficient pipeline parallel processing;

[0071] S54), when all decoding extension requests are completed, the output results of each request are spliced ​​together to obtain the final answer. The decoding process can be seen in Figure 3 shown.

[0072] Example 2

[0073] This example builds a homogeneous edge device cluster consisting of four NVIDIA Jetson Xavier NXs. All edge devices use the Pytorch deep learning framework to perform automatic differentiation and forward and backward propagation of deep neural network model calculation graphs, and use GLOO as the distributed communication backend. This example obtains two generative large language models, Llama2-7B and Llama-13B, and verifies the inference speed on the LiMA dataset.

[0074] This implementation verifies two open-source generative large language models, Llama2-7B and Llama-13B, in two bandwidth environments of 100Mbps and 1000Mbps, respectively. The results show that this embodiment always performs well under different network bandwidth environments. Since the serial parallel method requires each device to save the complete model weights, the serial parallel method has a memory overflow problem in some experiments. The method of this embodiment divides the model weights onto different devices and carefully schedules the computing load on different devices, so there will be no memory overflow problem.

[0075] The above embodiments and descriptions are only for illustrating the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, all of which fall within the scope of the present invention to be protected.

Claims

1. The edge-generative large language model collaborative reasoning method for ubiquitous computing power networks is characterized by: The steps include: S1), obtaining generative large language model information by installing the generative large language model on each edge device to the local computing environment, and measuring and collecting configuration parameters of the generative large language model; S2) Collect performance parameter information of edge devices. Each edge device uses test input samples to perform generative large language model reasoning tasks offline, and records relevant runtime data to estimate local computing resource parameters of edge devices. S3), modeling and solving edge device pipeline orchestration and generative large language model pipeline parallel reasoning; S4) Pipeline parallel pre-filling based on input sequence division, cutting the natural language reasoning request issued by the user into multiple sub-sequences, and injecting them into the edge device pipeline for reasoning at the same time; S5) Outline-based pipeline parallel autoregressive decoding, which prompts the generative large language model to generate a response outline containing multiple key points for the target question through preset outline instructions, and allows the generative large language model to decode and expand multiple key points in parallel in a pipeline parallel form.

2. The edge-generative large language model collaborative reasoning method for ubiquitous computing network according to claim 1 is characterized by: In step S1), the generative large language model inference task is performed offline using the test sample data, and the number of model layers of the generative large language model, the parameter information of each layer, and the memory space occupied by the parameters of each layer of the generative large language model and the intermediate features between adjacent layers during runtime are recorded; for a generative large language model containing L consecutive model layers, the memory space occupied by the parameters of the lth layer is w l .

3. The edge-generative large language model collaborative reasoning method for ubiquitous computing network according to claim 2 is characterized by: In step S2), when the generative large language model is running, the time required for the forward propagation of each Transformer decoder layer is recorded, and the ordered set of all edge devices participating in collaborative reasoning is defined as Ordered Sets Sort by computing performance from weak to strong, and define D n ={d1,d2,…,d n } is an ordered set The first n edge devices in the subset; define is the execution time required to execute model layer l on edge device n.

4. The edge-generated large language model collaborative reasoning method for ubiquitous computing network according to claim 3 is characterized by: In step S3), the configuration parameters of the generative large language model collected in step S1) and step S2) and the performance parameter information of each edge device are combined; the generative large language model is modeled using pipeline parallel architecture reasoning, and a dynamic programming algorithm is designed to solve the optimal edge device pipeline orchestration scheme and generative large language model partitioning scheme; finally, according to the edge device pipeline orchestration scheme and model partitioning results, the edge device is deployed and the generative large language model is deployed.

5. The edge-generative large language model collaborative reasoning method for ubiquitous computing network according to claim 4 is characterized by: In step S3), the edge device pipeline is specifically arranged as follows: The edge devices participating in collaborative reasoning are organized into ordered sets The edge devices are deployed in a pipeline form into multiple pipeline stages in the order of; and each edge device only communicates with the edge devices responsible for its previous stage and the next stage, and the total number of pipeline stages is equal to the total number of edge devices.

6. The edge-generative large language model collaborative reasoning method for ubiquitous computing power network according to claim 5 is characterized by: In step S3), the generative large language model pipeline parallel reasoning is modeled and solved, which specifically includes the following steps: S31) Divide all model layers of the generative large language model into multiple layer subsets connected in series, and define each layer subset as an independent stage; define the Transformer decoder layer in each stage as a stage model; and define A(i→j,D n ) is to divide the model layer i to model layer j into n stage models and assign them to the set D n The inference time required for the slowest model stage under the optimal segmentation scheme in the process; The optimal segmentation solution is solved by dynamic programming algorithm, namely: In the formula, Represents edge device d n The inference time required to infer the stage model from model layer i to model layer j; If assigned to edge device d n If the total memory overhead required during the inference process of the stage model exceeds the memory budget of the edge device, then set T(i→j,n)=+∞; S32), after completing the dynamic programming arrangement in step S31), all L-layer model layers of the generative large language model are divided into an ordered device set According to the obtained solution, the target generative large language model is cut along the model layer granularity, divided into multiple stage models and installed accordingly in the local environment of the edge device responsible for each stage.

7. The edge-generative large language model collaborative reasoning method for ubiquitous computing power network according to claim 6 is characterized by: In step S31), the total memory overhead required during the inference process includes the generative large language model parameters and intermediate cache results generated during the inference process.

8. The edge-generative large language model collaborative reasoning method for ubiquitous computing network according to claim 1 is characterized by: In step S4), the pipeline parallel pre-filling based on the input sequence division specifically includes the following steps: S41), cutting the natural language inference request sent by the user into multiple subsequences s1, s2, ..., s n , and multiple subsequences s1,s2,…,s n Simultaneously injected into the pipeline of edge devices; S42), through the generative large language model for each subsequence s i Perform inference and convert each subsequence s i The intermediate results of the self-attention network layer reasoning in the generative large language model are cached for subsequent subsequence reasoning.

9. The edge-generative large language model collaborative reasoning method for ubiquitous computing network according to claim 8, characterized in that: In step S42), in the pipeline parallel reasoning process based on sequence partitioning, since the generative large language model contains a network layer with a self-attention mechanism, the network layer needs to calculate the attention value of each word in the reasoning input sequence and all the previous words. Therefore, each subsequence s i In the inference calculation process, it depends on all the previous subsequences s1, s2, ...s i-1 The intermediate calculation results.

10. The edge-generated large language model collaborative reasoning method for ubiquitous computing network according to claim 1, characterized in that: In step S5), the outline-based pipeline parallel autoregressive decoding specifically includes the following steps: S51), presetting an outline generation instruction, in the pre-filling stage, combining the preset outline generation instruction with the user's question as an input of the generative large language model to guide the generative large language model to generate an outline for a reply answer; S52), after the pre-filling stage is completed, the generative large language model is allowed to serially generate an outline of the answer, and each point of the answer outline is individually packaged into an independent decoding extension request, each request will prompt the generative large language model to perform decoding extension only for the specific outline point; S53), injecting the decoding extension request into the edge collaborative reasoning pipeline to achieve efficient pipeline parallel processing; S54), when all decoding extension requests are completed, the output results of each request are concatenated to obtain the final answer.

Images