Edge-Generative Large Language Model Collaborative Reasoning Method for Ubiquitous Computing Networks

Abstract

The present invention provides an edge-based generative large language model collaborative reasoning method for a ubiquitous computing power network, comprising obtaining generative large language model information and edge device information; modeling and solving edge device pipeline orchestration and generative large language model pipeline parallel reasoning; pipeline parallel pre-filling based on input sequence division; and pipeline parallel autoregressive decoding based on an outline. The present invention, with the help of computing resources of multiple edge devices in a ubiquitous computing power network environment, accelerates the pre-filling stage of the generative large language model through a pipeline parallel reasoning technology based on sequence division, while utilizing an outline-based pipeline parallel decoding method to accelerate the autoregressive decoding stage of the generative large language model, thereby achieving low-latency and resource-efficient distributed collaborative reasoning of the generative large language model on multiple edge devices; effectively protecting user privacy data security, the present invention has better adaptability and robustness for the resource characteristics of complex edge environments.

Description

Technical Field

[0001] The present invention relates to the field of deep learning parallel reasoning technology, and in particular to an edge-generated large language model collaborative reasoning method for ubiquitous computing power networks. Background Art

[0002] Large generative language models have attracted widespread attention due to their impressive performance across a variety of artificial intelligence and natural language processing tasks. These models are typically deployed in cloud data centers equipped with powerful server-class GPUs and have driven the development of edge intelligence applications such as question-answering bots and smart home AI assistants.

[0003] Although generative large language models were originally developed in data centers, due to the sensitivity and privacy of user data, deploying them on more accessible edge platforms, rather than uploading requests to remote cloud platforms hosted by commercial companies, has become an emerging trend in recent years. To ensure privacy, generative large language models used by individual users should be deployed entirely or primarily on local edge devices (such as consumer PC graphics cards, smartphones, laptops, smart home IoT devices, etc.) for inference.

[0004] However, deploying large, computationally intensive and resource-intensive generative language models at the edge presents significant challenges. This is because the computing resources of individual edge devices are very limited. Directly deploying large generative language models on these devices often leads to problems such as excessive inference latency and high memory usage. To address these issues, existing edge-based collaborative large language model inference systems often adopt the following four parallel inference architectures.

[0005] One is data parallelism, which divides the workload by the sample dimension, with each device maintaining a complete copy of the model and performing inference independently. However, for single-sample inference requests (such as in a smart home environment, where a user issues a task instruction to an intelligent assistant), data parallelism cannot simultaneously utilize multiple edge devices.

[0006] The second is sequence parallelism, which divides the input sequence into multiple subsequences for parallel processing. However, sequence parallelism requires each device to maintain a complete copy of the model, which incurs significant memory overhead. Sequential parallel inference also requires tensor synchronization to be performed twice at each Transformer decoder layer, introducing intolerable communication overhead.

[0007] The third is tensor parallelism. Tensor parallelism divides the weights of a large generative language model across different devices, with each device retaining only some parameters. However, tensor synchronization is required after each Transformer decoder layer, following the multi-head attention module and the feedforward neural network module. This also introduces intolerable communication overhead, making device communication in low-speed edge networks a bottleneck for the inference system.

[0008] The fourth is pipeline parallelism; pipeline parallelism divides the large language model horizontally into multiple continuous stages along the layer granularity. Each stage is mapped to an independent edge device and multiple input samples are injected simultaneously to improve parallelism. However, for single-sample user requests, pipeline parallelism will degenerate into serial inference and cannot utilize the computing resources of multiple edge devices at the same time. Summary of the Invention

[0009] In response to the shortcomings of the existing technology, the present invention provides an edge-based generative large language model collaborative reasoning method for ubiquitous computing networks. The present invention can achieve low-latency and resource-efficient distributed collaborative reasoning of generative large language models on multiple edge devices.

[0010] The technical solution of the present invention is: an edge-generative large language model collaborative reasoning method for ubiquitous computing power networks, comprising the following steps:

[0011] S1), obtain generative large language model information;

[0012] S2), collecting edge device performance parameter information;

[0013] S3) Model and solve the edge device pipeline orchestration and the parallel reasoning of the generative large language model pipeline;

[0014] S4), pipeline parallel pre-filling based on input sequence partitioning;

[0015] S5), outline-based pipelined parallel autoregressive decoding.

[0016] Preferably, in step S1), the generative large language model is installed on each edge device to the local computing environment, and the configuration parameters of the generative large language model are measured and collected.

[0017] Preferably, in step S1), the generative large language model inference task is performed offline using the test sample data, and the number of model layers of the generative large language model, the parameter information of each layer, and the memory space occupied by the parameters of each layer of the generative large language model and the intermediate features between adjacent layers during runtime are recorded; for a generative large language model containing L consecutive model layers, the memory space occupied by the parameters of the lth layer is w l .

[0018] Preferably, in step S2), each edge device uses test input samples to perform generative large language model reasoning tasks offline, records relevant runtime data, and estimates local computing resource parameters of the edge device.

[0019] Preferably, in step S2), the recorded runtime data includes the time required for the forward propagation of each Transformer decoder layer of the runtime generative large language model, and the ordered set of all edge devices participating in collaborative reasoning is defined as Ordered Set Sort by computing performance from weak to strong and define D n ={d1,d2,…,d n} is an ordered set The subset of the first n edge devices in ; define is the execution time required to execute model layer l on edge device n.

[0020] Preferably, in step S3), the configuration parameters of the generative large language model collected in step S1) and step S2) and the performance parameter information of each edge device are combined; the generative large language model is modeled using pipeline parallel architecture reasoning, and a dynamic programming algorithm is designed to solve the optimal edge device pipeline orchestration scheme and generative large language model partitioning scheme; finally, according to the edge device pipeline orchestration scheme and model partitioning results, the edge device is deployed and the generative large language model is deployed.

[0021] Preferably, in step S3), the edge device pipeline is specifically arranged as follows:

[0022] The edge devices participating in collaborative reasoning are arranged into ordered sets The edge devices are deployed in a pipeline form into multiple pipeline stages in the order of ; and each edge device only communicates with the edge devices responsible for its previous stage and the next stage. The total number of pipeline stages is equal to the total number of edge devices.

[0023] Preferably, in step S3), the parallel reasoning modeling and solving of the generative large language model pipeline is performed, specifically including the following steps:

[0024] S31) Divide all model layers of the generative large language model into multiple layer subsets connected in series, and define each layer subset as an independent stage; define the Transformer decoder layer in each stage as a stage model; and define A(i→j,D n ) is to divide the model layer i to model layer j into n stage models and assign them to the set D n The inference time required for the slowest model stage under the optimal segmentation scheme in the process;

[0025] The optimal segmentation solution is solved by dynamic programming algorithm, namely:

[0026]

[0027] Where, Indicates edge device d n The inference time required to infer the stage model from model layer i to model layer j;

[0028] If assigned to edge device d n If the total memory overhead required by the stage model in the inference process exceeds the memory budget of the edge device, then set T(i→j,n)=+∞;

[0029] S32) After completing the dynamic programming arrangement in step S31), all L-layer model layers of the generative large language model are divided into an ordered device set. According to the obtained solution, the target generative large language model is cut along the model layer granularity, divided into multiple stage models and installed accordingly in the local environment of the edge device responsible for each stage.

[0030] As a preference, in step S31), the total memory overhead required during the inference process includes the generative large language model parameters and intermediate cache results generated during the inference process.

[0031] Preferably, in step S4), the parallel pre-filling of the pipeline based on the input sequence division specifically includes the following steps:

[0032] S41), cutting the natural language inference request sent by the user into multiple subsequences s1, s2, ..., s n , and multiple subsequences s1,s2,…,s n At the same time, it is injected into the pipeline of the edge device; the number of splits is set according to the needs. This embodiment uses twice the number of stages, where the pipeline is Figure 2 Phase 1 in.

[0033] S42), through the generative large language model for each subsequence s i Perform inference and convert each subsequence s i The intermediate results of the self-attention network layer reasoning in the generative large language model are cached for subsequent subsequence reasoning.

[0034] Preferably, in step S42), in the pipeline parallel reasoning process based on sequence partitioning, since the generative large language model contains a network layer of a self-attention mechanism, the network layer needs to calculate the attention value of each word in the reasoning input sequence and all the words before it. Therefore, each subsequence s i In the inference calculation process, it depends on all the previous subsequences s1, s2, ...s i-1 The intermediate calculation results.

[0035] Preferably, in step S5), the generative large language model is prompted by a preset outline instruction to generate a response outline containing multiple key points for the target question, and the generative large language model is allowed to decode and expand the multiple key points in parallel in a pipelined parallel manner.

[0036] Preferably, step S5) specifically includes the following steps:

[0037] S51), presetting an outline generation instruction, in the pre-filling stage, combining the preset outline generation instruction with the user's question as input to the generative large language model to guide the generative large language model to generate an outline for a reply answer;

[0038] S52) After the pre-filling phase is completed, the generative large language model is first serially generated into an outline of the answer, and each point of the answer outline is individually packaged into an independent decoding expansion request. Each request will prompt the generative large language model to decode and expand only the key points of the specific outline;

[0039] S53) injecting the decoding expansion request into the edge collaborative inference pipeline to achieve efficient pipeline parallel processing;

[0040] S54) When all decoding extension requests are completed, the output results of each request are concatenated to obtain the final answer.

[0041] The beneficial effects of the present invention are:

[0042] 1. This invention leverages the computing resources of multiple edge devices in a ubiquitous computing network environment to accelerate the pre-population phase of a large generative language model through a pipelined parallel reasoning technique based on sequence partitioning. It also utilizes an outline-based pipelined parallel decoding method to accelerate the autoregressive decoding phase of the large generative language model, achieving low-latency and resource-efficient distributed collaborative reasoning of the large generative language model on multiple edge devices.

[0043] 2. The present invention can retain the data generated by processing user data on the user device, effectively protecting the user's privacy data security. At the same time, it does not require large-scale data communication with the cloud data center, avoiding the problems of unstable backbone network and unreliable service quality;

[0044] 3. The present invention has better adaptability and robustness to the resource characteristics of complex edge environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of the process of the present invention;

[0046] Figure 2 This is a flow chart of the pre-filling of a large language model using a pipelined parallel generative approach based on sequence partitioning according to the present invention;

[0047] Figure 3 Schematic diagram of the outline-based pipelined parallel autoregressive decoding method of the present invention. DETAILED DESCRIPTION

[0048] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0049] like Figure 1 As shown, this embodiment provides an edge-generative large language model collaborative reasoning method for a ubiquitous computing network, including the following steps:

[0050] S1) Obtaining generative large language model information; This embodiment installs the generative large language model on each edge device to the local computing environment, and measures and collects the generative large language model configuration parameters;

[0051] By using the test sample data to perform the generative large language model inference task offline, and record the number of model layers of the generative large language model, the parameter information of each layer, and the memory space occupied by the parameters of each layer of the generative large language model and the intermediate features between adjacent layers during runtime; for a generative large language model containing L consecutive model layers, the memory space occupied by the parameters of the lth layer is w l .

[0052] S2) Collect edge device information; each edge device in this embodiment uses the test input sample to perform the generative large language model reasoning task offline, record the relevant runtime data, and estimate the local computing resource parameters of the edge device;

[0053] The recorded runtime data includes the time required for the forward propagation of each Transformer decoder layer of the runtime generative large language model, and defines the ordered set of all edge devices participating in collaborative reasoning as Ordered Set Sort by computing performance from weak to strong and define D n ={d1,d2,…,d n} is an ordered set The subset of the first n edge devices in ; define is the execution time required to execute model layer l on edge device n.

[0054] S3) Model and solve the edge device pipeline orchestration and the parallel reasoning of the generative large language model pipeline;

[0055] This embodiment combines the configuration parameters of the generative large language model collected in step S1) and step S2) and the performance parameter information of each edge device; models the generative large language model using pipeline parallel architecture reasoning, and designs a dynamic programming algorithm to solve the optimal edge device pipeline orchestration scheme and generative large language model partitioning scheme; finally, based on the edge device pipeline orchestration scheme and model partitioning results, deploys the edge device and the generative large language model.

[0056] Among them, the edge device pipeline arrangement is specifically as follows: the edge devices participating in collaborative reasoning are arranged according to the ordered set The edge devices are deployed in a pipeline form into multiple pipeline stages in the order of ; and each edge device only communicates with the edge devices responsible for its previous stage and the next stage. The total number of pipeline stages is equal to the total number of edge devices.

[0057] Modeling and solving the parallel reasoning of a large generative language model pipeline includes the following steps:

[0058] S31) Divide all model layers of the generative large language model into multiple layer subsets connected in series, and define each layer subset as an independent stage; define the Transformer decoder layer in each stage as a stage model; and define A(i→j,D n ) is to divide the model layer i to model layer j into n stage models and assign them to the set D n The inference time required for the slowest model stage under the optimal segmentation scheme in the process;

[0059] The optimal segmentation solution is solved by dynamic programming algorithm, namely:

[0060]

[0061] Where, Indicates edge device d n The inference time required to infer the stage model from model layer i to model layer j;

[0062] If assigned to edge device d n If the total memory overhead required during inference exceeds the memory budget of the edge device, then set T(i→j,n)=+∞; the total memory overhead required during inference includes the parameters of the generative large language model. and intermediate cache results generated during the inference process.

[0063] S32) After completing the dynamic programming arrangement in step S31), all L-layer model layers of the generative large language model are divided into an ordered device set. According to the obtained solution, the target generative large language model is cut along the model layer granularity, divided into multiple stage models and installed accordingly in the local environment of the edge device responsible for each stage.

[0064] S4), pipeline parallel pre-filling based on input sequence division; Figure 2 As shown, the specific steps include:

[0065] S41), cutting the natural language inference request sent by the user into multiple subsequences s1, s2, ..., s n , and multiple subsequences s1,s2,…,s n Simultaneously inject into the pipeline of edge devices;

[0066] S42), through the generative large language model for each subsequence s i Perform inference and convert each subsequence s i The intermediate results of the self-attention network layer reasoning of the generative large language model are cached for subsequent subsequence reasoning. In this embodiment, in the pipeline parallel reasoning process based on sequence division, since the generative large language model contains a network layer with a self-attention mechanism, the network layer needs to calculate the attention value of each word in the reasoning input sequence and all the words before it. Therefore, each subsequence s i In the inference calculation process, it depends on all the previous subsequences s1, s2, ...s i-1 The intermediate calculation results.

[0067] S5) Outline-based pipelined parallel autoregressive decoding: Prompting the generative large language model to generate a response outline containing multiple key points for the target question through preset outline instructions, and allowing the generative large language model to decode and expand multiple key points in parallel in a pipelined parallel manner; specifically, the following steps are included:

[0068] S51), presetting an outline generation instruction, in the pre-filling stage, combining the preset outline generation instruction with the user's question as input to the generative large language model to guide the generative large language model to generate an outline for a reply answer;

[0069] S52) After the pre-filling phase is completed, the generative large language model is first serially generated into an outline of the answer, and each point of the answer outline is individually packaged into an independent decoding expansion request. Each request will prompt the generative large language model to decode and expand only the key points of the specific outline;

[0070] S53) injecting the decoding expansion request into the edge collaborative inference pipeline to achieve efficient pipeline parallel processing;

[0071] S54) When all decoding extension requests are completed, the output results of each request are spliced ​​together to obtain the final answer. The decoding process can be seen in Figure 3 shown.

[0072] Example 2

[0073] This example built a cluster of homogeneous edge devices consisting of four NVIDIA Jetson Xavier NXs. All edge devices used the Pytorch deep learning framework for automatic differentiation and forward and backward propagation of deep neural network model computation graphs, using Gloo as the distributed communication backend. This example generated two large generative language models, Llama2-7B and Llama-13B, and verified their inference speed on the LiMA dataset.

[0074] This implementation verifies two open-source generative large language models, Llama2-7B and Llama-13B, under 100Mbps and 1000Mbps bandwidth environments, respectively. The results show that this embodiment consistently performs well under different network bandwidth environments. Since the serial parallel method requires each device to save the complete model weights, the serial parallel method has memory overflow problems in some experiments. The method of this embodiment divides the model weights onto different devices and carefully schedules the computing load on different devices, so memory overflow problems will not occur.

[0075] The above embodiments and descriptions are only for explaining the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, which shall fall within the scope of the invention to be protected.

Claims

1. An edge-generative large language model collaborative reasoning method for ubiquitous computing networks, characterized by: The steps include: S1) Obtaining generative large language model information by installing the generative large language model on each edge device to the local computing environment and measuring and collecting the generative large language model configuration parameters; S2) Collect performance parameter information of edge devices. Each edge device uses test input samples to perform generative large language model inference tasks offline, records relevant runtime data, and estimates local computing resource parameters of the edge device. S3) Model and solve the edge device pipeline orchestration and the parallel reasoning of the generative large language model pipeline; S4) Pipeline parallel pre-filling based on input sequence division divides the natural language inference request issued by the user into multiple sub-sequences, which are simultaneously injected into the edge device pipeline for inference; specifically, the following steps are included: S41), cutting the natural language inference request sent by the user into multiple subsequences s1, s2, ..., s n , and multiple subsequences s1,s2,…,s n Simultaneously inject into the pipeline of edge devices; S42), through the generative large language model for each subsequence s i Perform inference and convert each subsequence s i Cache the intermediate results of the self-attention network layer reasoning in the generative large language model for subsequent subsequence reasoning; S5) Outline-based pipelined parallel autoregressive decoding: Using preset outline instructions, the generative large language model is prompted to generate a response outline containing multiple key points for the target question, and the generative large language model is allowed to decode and expand multiple key points simultaneously in a pipelined parallel manner; specifically, the steps include: S51), presetting an outline generation instruction, in the pre-filling stage, combining the preset outline generation instruction with the user's question as input to the generative large language model to guide the generative large language model to generate an outline for a reply answer; S52) After the pre-filling phase is completed, the generative large language model is first serially generated into an answer outline, and each sub-point of the answer outline is individually packaged into an independent decoding expansion request. Each request will prompt the generative large language model to decode and expand only the key points of the answer outline; S53) injecting the decoding expansion request into the edge collaborative inference pipeline to achieve efficient pipeline parallel processing; S54) When all decoding extension requests are completed, the output results of each request are concatenated to obtain the final answer.

2. The edge-generative large language model collaborative reasoning method for ubiquitous computing power networks according to claim 1 is characterized by: In step S1), the generative large language model inference task is performed offline using the test sample data, and the number of model layers of the generative large language model, the parameter information of each layer, and the memory space occupied by the parameters of each layer of the generative large language model and the intermediate features between adjacent layers are recorded during runtime; for a generative large language model containing L consecutive model layers, the memory space occupied by the parameters of the lth layer is w l .

3. The edge-generative large language model collaborative reasoning method for ubiquitous computing power networks according to claim 2 is characterized by: In step S2), when the generative large language model is running, the time required for the forward propagation of each Transformer decoder layer is recorded, and the ordered set of all edge devices participating in collaborative reasoning is defined as Ordered Set Sort by computing performance from weak to strong and define is an ordered set The subset of the first n edge devices in ; define is the execution time required to execute model layer l on edge device n.

4. The edge-generative large language model collaborative reasoning method for ubiquitous computing power networks according to claim 3 is characterized by: In step S3), the configuration parameters of the generative large language model collected in steps S1) and S2) and the performance parameter information of each edge device are combined; the generative large language model is modeled using pipeline parallel architecture reasoning, and a dynamic programming algorithm is designed to solve the optimal edge device pipeline orchestration scheme and generative large language model partitioning scheme; finally, based on the edge device pipeline orchestration scheme and model partitioning results, the edge device and the generative large language model are deployed.

5. The edge-generative large language model collaborative reasoning method for ubiquitous computing power networks according to claim 4 is characterized by: In step S3), the edge device pipeline is specifically arranged as follows: The edge devices participating in collaborative reasoning are organized into ordered sets The edge devices are deployed in a pipeline form into multiple pipeline stages in the order of ; and each edge device only communicates with the edge devices responsible for its previous stage and the next stage. The total number of pipeline stages is equal to the total number of edge devices.

6. The edge-generative large language model collaborative reasoning method for ubiquitous computing power networks according to claim 5 is characterized by: In step S3), the parallel reasoning model of the generative large language model pipeline is modeled and solved, which specifically includes the following steps: S31) Divide all model layers of the generative large language model into multiple layer subsets connected in series, and define each layer subset as an independent stage; define the Transformer decoder layer in each stage as a stage model; and define A(i→j,D n ) is to divide the model layer i to model layer j into n stage models and assign them to the set D n The inference time required for the slowest model stage under the optimal segmentation scheme in the process; The optimal segmentation solution is solved by dynamic programming algorithm, namely: Where, Indicates edge device d n The inference time required to infer the stage model from model layer i to model layer j; is the execution time required to execute model layer l on edge device n; If assigned to edge device d n If the total memory overhead required by the stage model in the inference process exceeds the memory budget of the edge device, then set T(i→j,n)=+∞; S32) After completing the dynamic programming arrangement in step S31), all L-layer model layers of the generative large language model are divided into an ordered device set. According to the obtained solution, the target generative large language model is cut along the model layer granularity, divided into multiple stage models and installed accordingly in the local environment of the edge device responsible for each stage.

7. The edge-generative large language model collaborative reasoning method for ubiquitous computing power networks according to claim 6 is characterized by: In step S31), the total memory overhead required during the inference process includes the generative large language model parameters And the intermediate cache results generated during the inference process, among which, for a large language model with L consecutive model layers, the memory space occupied by the parameters of the lth layer is w l .

8. The edge-generative large language model collaborative reasoning method for ubiquitous computing network according to claim 1 is characterized by: In step S42), in the pipeline parallel reasoning process based on sequence partitioning, since the generative large language model contains a network layer with a self-attention mechanism, the network layer needs to calculate the attention value of each word in the reasoning input sequence and all the words before it. Therefore, each subsequence s i In the inference calculation process, it depends on all the previous subsequences s1, s2, ...s i-1 The intermediate calculation results.

Images