Cloud-edge collaborative inference method and inference system

Abstract

The invention discloses a cloud-side collaborative inference method and inference system, and the method comprises the steps: edge equipment receives a user task request, and sends a task notification to cloud equipment; the cloud equipment controls a deployed cloud large model to execute a user task, and sends a task intermediate result to the edge equipment; and the edge device executes the user task based on the task intermediate result and the user prompt to obtain a task result, and feeds back the task result to the user side. According to the application, the edge device cooperates with the cloud device, the edge device shares the task intermediate result in the reasoning process of the cloud device, and the user prompting user task with high real-time performance and low resource consumption is processed based on the task intermediate result, so that on one hand, the computing power of the cloud device can be utilized; and on the other hand, communication delay and data transmission quantity can be reduced, so that high delay caused by the fact that data transmission occupies a large amount of bandwidth can be reduced on the premise of ensuring that a user task can be executed.

Description

Technical Field

[0001] The present application relates to the field of network communication technology, and in particular to a cloud-edge collaborative reasoning method and reasoning system. Background Art

[0002] At present, large language models (LLMs) (such as GPT-3, ChatGPT, BERT, etc.) are widely used in various fields, and they provide important impetus and key support for the development of 6G. LLMs generally have billions or even hundreds of billions of parameters, requiring a large amount of memory, storage, and computing resources to train and reason about them. In addition, large models based on the transformer architecture involve very complex matrix calculations, attention mechanism calculations, etc. in their reasoning process.

[0003] Therefore, in practical applications, LLM is generally deployed on cloud devices, and the reasoning process of LLM is implemented through cloud computing, that is, users send reasoning requests to the cloud in a unified manner, that is, call the deployed large model through API or other interfaces to obtain high-accuracy task results. However, to perform large model reasoning through cloud devices, a large amount of data needs to be transmitted between the user end and the cloud device. With the annual increase in network traffic, especially when real-time tasks or large-scale devices need to be connected, most of the network bandwidth will be occupied, causing high latency, which will affect the timeliness of task results. At the same time, the data transmission process is vulnerable to network attacks (such as distributed denial of service (DDOS) attacks, attacks on domain name systems and security vulnerabilities, etc.), which will pose a serious threat to the stability of the communication system, data integrity and privacy protection, thereby causing economic losses.

[0004] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0005] The technical problem to be solved by this application is to provide a cloud-edge collaborative reasoning method and reasoning system to address the shortcomings of the existing technology.

[0006] In order to solve the above technical problems, the first aspect of the present application provides a cloud-edge collaborative reasoning method, wherein the cloud-edge collaborative reasoning method specifically includes:

[0007] The edge device receives the user's task request and sends a task notification to the cloud device;

[0008] Based on the task notification, the cloud device controls the deployed cloud large model to execute the preset task, and sends the intermediate task results obtained during the task execution process to the edge device;

[0009] Based on the intermediate result of the task and the user prompt corresponding to the user task request, the edge device controls the deployed edge large model to execute the user task to obtain the task result, and feeds back the task result to the user end.

[0010] The cloud-edge collaborative reasoning method, wherein before sending the task notification to the cloud device, the method further includes:

[0011] Detecting the task type corresponding to the user task request;

[0012] When the task type is a dedicated task, generating a task notification for executing the user task based on a system prompt;

[0013] When the task type is a non-proprietary task, a task notification for executing the user task based on a user prompt corresponding to the user task request is generated, and the user prompt is added to the task notification.

[0014] The cloud-edge collaborative reasoning method, wherein the cloud device controls the deployed cloud large model to execute the preset task based on the task notification, and sends the task intermediate results obtained during the task execution to the edge device, specifically includes:

[0015] The cloud device obtains a task prompt based on the task notification, wherein the task prompt is a system prompt or a user prompt, and the system prompt is background knowledge corresponding to the user task;

[0016] inputting the task prompt into a cloud-based large model deployed by the cloud-based device;

[0017] Whenever a network layer in the cloud-based large model is executed, the cloud-based device sends the intermediate result of the task corresponding to the network layer to the edge device.

[0018] The cloud-edge collaborative reasoning method, wherein the edge device controls the deployed edge big model to execute the user task to obtain the task result based on the task intermediate result and the user prompt corresponding to the user task request, specifically includes:

[0019] The edge device determines the network layer corresponding to the intermediate result of the task;

[0020] The edge device selects the current network layer in the deployed edge big model based on the network layer corresponding to the intermediate result of the task;

[0021] The edge device inputs the task result of the previous network layer of the current network layer and the intermediate result of the task into the current network layer, and outputs the task result through the current network layer. If the current network layer is the frontmost network layer, the task result of the previous network layer of the current network layer is a user prompt; if the current network layer is the last network layer, the task result output by the current network layer is a task result for feedback to the user end.

[0022] The cloud-edge collaborative reasoning method, wherein the edge device receives a user task request and sends a task notification to the cloud device, specifically includes:

[0023] The edge device receives the user's task request and detects its own available device resources;

[0024] When the available device resources do not meet the preset requirements, a task notification is sent to the cloud device;

[0025] When the available device resources meet the preset requirements, the cloud large model deployed by the cloud device is downloaded from the cloud device, and the user task corresponding to the user task request is executed through the cloud large model.

[0026] In the cloud-edge collaborative reasoning method, when the available device resources do not meet the preset requirements, sending a task notification to the cloud device specifically includes:

[0027] The edge device detects the distance between itself and the cloud device;

[0028] When the interval distance is less than or equal to the preset distance threshold, a task notification is sent to the cloud device, and the cloud device is informed by the task notification to collaboratively execute the user task corresponding to the user task request with the edge device, wherein the motion device executes the preset task based on the system prompt, and the edge device executes the preset task based on the task intermediate result and the user prompt corresponding to the user task request until the user task is completed.

[0029] The cloud-edge collaborative reasoning method, wherein the method further includes:

[0030] When the interval distance is greater than a preset distance threshold, issuing an execution notification to the cloud device, and informing the cloud device through the execution notification to independently execute the user task corresponding to the user task request based on the user prompt;

[0031] The edge device receives the task result from the cloud device and feeds the task result back to the user end.

[0032] The cloud-edge collaborative reasoning method, wherein the intermediate result of the task is the key vector and value vector cached by the network layer in the cloud-based large model, and the contextual knowledge of the user task is obtained through the key vector and value vector.

[0033] The second aspect of the present application provides a cloud-edge collaborative reasoning system, wherein the cloud-edge collaborative reasoning system specifically includes a cloud device and at least one edge device, the edge device is used to receive user task requests and send task notifications to the cloud device, the cloud device is used to control the cloud large model deployed by it to execute preset tasks based on the task notifications, and send the task intermediate results obtained during the task execution process to the edge device; the edge device is also used to control the edge large model deployed by it to execute the user task to obtain the task result based on the task intermediate result and the user prompt corresponding to the user task request, and feed back the task result to the user end.

[0034] In the cloud-edge collaborative reasoning system, the intermediate result of the task is the key vector and value vector cached by the network layer in the cloud-based large model, and the contextual knowledge of the user task is obtained through the key vector and value vector.

[0035] Beneficial effects:

[0036] 1) Through the collaborative work of edge devices and cloud devices, the powerful computing resources and data storage capabilities of the cloud large model can be fully utilized, while also having the timely response capabilities of the edge large model;

[0037] 2) Sharing value vectors and key vectors (i.e., KV) between the cloud-based large model and the edge-based large model alleviates the problem of insufficient memory in the edge-side model while accelerating the reasoning capability of the entire task system;

[0038] 4) Privacy layered processing is realized. Edge device users do not need to upload data to cloud devices. The edge-end big model can provide personalized reasoning solutions for different regions based on the reasoning of the cloud-end big model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 A schematic diagram of the principles of the cloud-edge collaborative reasoning system provided in an embodiment of the present application.

[0041] Figure 2 A flowchart of the cloud-edge collaborative reasoning method provided in an embodiment of the present application.

[0042] Figure 3A flowchart of an example of a cloud-edge collaborative reasoning method provided in an embodiment of the present application.

[0043] Figure 4 A flowchart of another example of the cloud-edge collaborative reasoning method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The embodiments of the present application provide a cloud-edge collaborative reasoning method and reasoning system. To make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0045] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0046] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0047] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0048] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0049] This embodiment provides a cloud-edge collaborative reasoning method, which applies a cloud-edge collaborative reasoning system, and the cloud-edge collaborative reasoning system can be deployed in a 6G network environment. Specifically, Figure 1 As shown, the cloud-edge collaborative reasoning system may include cloud devices and edge devices, the cloud devices communicate with edge devices, and the edge devices communicate with several user terminals (for example, mobile phones, tablets, notebooks, cars, etc.). The user terminal sends a reasoning request to the edge device, and the reasoning request may use prompts in natural language (such as questions or task descriptions). After receiving the reasoning request, the edge device will send a task notification to the cloud device, obtain the task intermediate result through the edge device, and then the edge device determines the task result based on the task intermediate result and the user prompt in the reasoning request, and feeds back the task result to the user terminal to achieve collaborative reasoning between the cloud device and the edge device.

[0050] like Figure 2 As shown, the cloud-edge collaborative reasoning method provided in the embodiment of the present application specifically includes:

[0051] S10. The edge device receives the user's task request and sends a task notification to the cloud device.

[0052] Specifically, the user task request is sent by the user end to the edge device to request the edge device to perform the user task, wherein the user task request may carry natural language prompts, which are used as user prompts to inform the edge device of the task problem or task description of the user task to be performed. For example, Figure 1 As shown, the edge device is connected to user terminals User1, User2, ..., UserN, User1 sends Promp1 to the edge device, User2 sends Promp2 to the edge device, ..., UserN sends PrompN, Promp1 to the edge device.

[0053] The task notification is used to inform the cloud device that a task needs to be performed, wherein the task notification is formed by the edge device when receiving the user reasoning request. The task notification can be used only as a notification instruction to execute the user task through the large model, or it can carry the prompt words required for the large model to execute the user task process. That is to say, when the user task request is received, the task notification can be generated and the prompt word category can be determined according to the implementation requirements, and the task notification can be updated based on the prompt word category, and then the updated task notification is sent to the cloud as a task notification. Among them, the prompt word category can be a user prompt category or a system prompt category. When the prompt word list is a system prompt category, the task performed based on the system prompt is used as a preset task. When the prompt word list is a user prompt category, the task performed based on the user prompt is used as a preset task.

[0054] Exemplarily, before sending the task notification to the cloud device, the method further includes:

[0055] Detecting the task type corresponding to the user task request;

[0056] When the task type is a dedicated task, generating a task notification for executing the user task based on a system prompt;

[0057] When the task type is a non-proprietary task, a task notification for executing the user task based on a user prompt corresponding to the user task request is generated, and the user prompt is added to the task notification.

[0058] Specifically, the task type is used to reflect whether the user task corresponding to the user task request belongs to a proprietary task system. The proprietary task system includes a set of user tasks that can be executed based on system prompts pre-stored in cloud devices, that is, for user tasks in the proprietary task system, the system prompts can be directly used as background knowledge, and the system prompts can be used as preset tasks. The preset tasks are executed based on the system prompts through the cloud large model. On the contrary, for user tasks in non-proprietary task systems, it is necessary to use user prompts as task prompts to execute user tasks, that is, use user prompts as preset tasks, and execute preset tasks based on user prompts through the cloud large model. This application utilizes the powerful computing power of cloud devices to complete large-scale user tasks through the collaborative work of cloud devices and edge devices, where user tasks can be reasoning tasks and generation tasks, etc. For example, user tasks are complex traffic scene prediction, route generation, voice request processing, etc.

[0059] like Figure 3 As shown, after obtaining the task type, it will be determined whether the task type is a proprietary task (i.e., a proprietary task network is required). If the task type is a proprietary task, the cloud device is notified to execute the preset task based on the system prompt (i.e., a task notification for executing the user task based on the system prompt is generated, so that the cloud device obtains the task intermediate result based on the system prompt). Conversely, if the task type is a non-proprietary task, the cloud device is informed through the task notification to execute the preset task based on the user prompt corresponding to the user task request (i.e., a task notification for executing the user task based on the user prompt corresponding to the user task request is generated, and the user prompt is added to the task notification, so that the cloud device obtains the task intermediate result based on the user prompt).

[0060] It should be noted that in actual applications, after determining the prompt word category according to the task type, the task notification can be sent to the cloud device first, and then the prompt word can be sent to the cloud device according to the prompt word category. Specifically, when the prompt word category is the system prompt category, a prompt notification is sent to inform the cloud device to use the system prompt to execute the user task. When the prompt word category is the user prompt, the user prompt is sent as a prompt notification to the cloud device to inform the cloud device to perform task reasoning based on the user prompt.

[0061] Furthermore, after receiving the user task request, in addition to directly sending task notifications to the cloud device to collaborate with the cloud device, the edge device can also determine whether to complete the user task independently based on its own available device resources to further reduce data transmission during the user task execution process. Figure 3 As shown, the edge device receives the user task request and sends the task notification to the cloud device, specifically including:

[0062] The edge device receives the user's task request and detects its own available device resources;

[0063] When the available device resources do not meet the preset requirements, a task notification is sent to the cloud device;

[0064] When the available device resources meet the preset requirements, the cloud large model deployed by the cloud device is downloaded from the cloud device, and the user task corresponding to the user task request is executed through the cloud large model.

[0065] Specifically, the available device resources are the resources that the edge device can use to execute the user task corresponding to the user task request, that is, the available device resources can be equal to the device resources configured for the edge device, or can be less than the device resources configured for the edge device. The preset requirements are pre-set and used as the basis for the edge device to independently execute the user task. For example, the preset requirement is that the available device resources are greater than the preset resource threshold, etc. Therefore, after obtaining the available device resources, it will be determined whether the available device resources meet the preset requirements. When the preset requirements are met, it means that the resources of the edge device are sufficient, so that the user task can be executed locally on the edge device in priority to reduce network overhead and delay. On the contrary, when the preset requirements are not met, it means that the edge device cannot execute the user task independently. At this time, it can choose to cooperate with the cloud device to execute the user task, and share the computing burden of the user task through dynamic routing to complete the user task.

[0066] In one implementation, when it is determined that a cloud device is selected to collaboratively perform a user task, it is necessary to transmit the intermediate results of the task through dynamic routing. When the edge device is too far away from the cloud device, the communication path between the edge device and the cloud device will be longer, which will take a long time to transmit and will be more susceptible to network attacks. For this reason, when choosing a cloud device to collaboratively perform a user task, you can directly select the edge device and the cloud device to collaboratively perform the user task, or you can choose to have the cloud device perform the user task independently. Based on this, if Figure 4 As shown, when the available device resources do not meet the preset requirements, sending a task notification to the cloud device specifically includes:

[0067] The edge device detects the distance between itself and the cloud device;

[0068] When the interval distance is less than or equal to a preset distance threshold, a task notification is sent to the cloud device, and the cloud device is informed by the task notification to collaboratively execute the user task corresponding to the user task request with the edge device.

[0069] Specifically, the interval distance may be the physical distance between the edge device and the cloud device, for example, it may be calculated based on the location information of the edge device and the location information of the cloud device, etc. The preset distance threshold is pre-set, and when the interval distance is less than or equal to the preset distance threshold, it indicates that the user task can be completed jointly between the edge device and the cloud device, that is, a task notification is sent to the cloud device, and the cloud device and the edge device are informed through the task notification to collaboratively perform the user task corresponding to the user task request, wherein the motion device performs the preset task based on the system prompt, and the edge device performs the user task based on the intermediate result of the task and the user prompt corresponding to the user task request.

[0070] In addition, when comparing the interval distance with the preset distance threshold, there may be a situation where the interval distance is greater than the preset distance threshold. When the interval distance is greater than the preset distance threshold, the cloud device will independently complete the user task to minimize the data that needs to be transmitted. To this end, the method also includes: when the interval distance is greater than the preset distance threshold, issuing an execution notification to the cloud device, informing the cloud device through the execution notification to independently execute the user task corresponding to the user task request based on the user prompt; the edge device receives the task result of the cloud device and feeds back the task result to the user end.

[0071] It should be noted that when the available device resources meet the preset requirements, in addition to determining whether the cloud device and the edge device collaboratively perform the user task or the cloud device performs the user task based on the interval distance, it is also possible to determine whether the device resources meet another preset requirement. When the other preset requirement is met, the cloud device and the edge device collaboratively perform the user task. When the other preset requirement is not met, the cloud device performs the user task. Among them, another preset requirement may be that the available device resources are less than the target resource threshold, wherein the target resource threshold is less than the preset resource threshold. In other words, when the resources of the edge device are limited or the task is complex, cloud-edge collaborative reasoning is selected to share the computing burden through dynamic routing. When the resources of the edge device are extremely limited, choose to rely entirely on cloud reasoning to ensure that the user task can be completed.

[0072] S20. Based on the task notification, the cloud device controls the deployed cloud large model to execute the preset task, and sends the intermediate results of the task obtained during the task execution process to the edge device.

[0073] Specifically, the cloud device is deployed with a cloud big model, and the edge device is deployed with an edge big model, wherein both the cloud big model and the edge big model can be existing large language models, for example, GPT-3, ChatGPT, BERT, etc. In the embodiment of the present application, both the cloud big model and the edge big model are transformer models using a decoder structure, which are used to perform autoregressive generation tasks, such as text generation, language modeling, etc. In the cloud big model and the edge big model, the decoder generates a token at each time step, and only uses the previously generated token as context knowledge during the generation process, so that the intermediate results of the task generated by the cloud big model based on the system prompt can be used as the context information when the edge big model performs the user task based on the user prompt, so that the cloud device and the edge device share different parts of the reasoning task, thereby significantly reducing the communication overhead.

[0074] Furthermore, the decoder includes several decoding layers, each of which is configured with a multi-head attention mechanism. The I / O bytes and FLOPs (computational amount) of each decoding layer can be expressed as:

[0075] FLOPs = 8bmkd

[0076] I / O bytes = 8bmk + 4bmkd + 4bkd

[0077] h=k·d

[0078] Among them, b represents the batch size, m represents the sequence length, k represents the number of attention heads, d represents the hidden size of each attention head, and h represents the overall hidden size.

[0079] Correspondingly, the arithmetic intensity of the cloud-based large model can be expressed as:

[0080]

[0081] Among them, intensity represents arithmetic intensity.

[0082] From the expression of arithmetic intensity, we can know that when the arithmetic intensity is low, data transmission (I / O bytes) becomes the key factor limiting the model performance of large cloud models, which means that each calculation requires a large amount of data to be transferred from the memory or cache to the computing unit; and when the arithmetic intensity is high, the amount of calculation (FLOPs) becomes a bottleneck. To this end, this application can solve the problems that arise when the arithmetic intensity is high or low through the layered collaboration of edge devices and cloud devices.

[0083] Furthermore, for a large cloud model deployed on a cloud device, when considering a single attention head and the length of the system prompt is s, the calculation of the query vector, key vector, and value vector of the lth layer can be expressed as follows:

[0084]

[0085] Among them, l∈{1,2,…,N} represents the number of transformer layers, represents the input item of the lth layer, represents the query vector of layer l, represents the key vector of layer l, represents the value vector of the lth layer, and Both represent the weights of the lth layer.

[0086] The attention output of layer l is expressed as:

[0087]

[0088] in, represents the normalization coefficient of the attention weight of the lth layer, represents the output item of the lth layer, and (·)T represents the transpose. Represents the normalization coefficient of the attention weight of the l-1th layer.

[0089] Furthermore, the cloud-based large model deployed on the cloud device stores the intermediate results of each layer and The edge big model deployed on the edge device can be downloaded and used. At the same time, the cloud big model output It will be used as the input of the next layer and the attention calculation will continue until the entire reasoning process is completed: It can be seen that the intermediate result of the task is the key vector and value vector cached by the network layer in the cloud-based large model, and the contextual knowledge of the user task is obtained through the key vector and value vector.

[0090] In one implementation, the cloud device controls the deployed cloud big model to execute the preset task based on the task notification, and sends the task intermediate result obtained during the task execution to the edge device, which specifically includes:

[0091] The cloud device obtains a task prompt based on the task notification, wherein the task prompt is a system prompt or a user prompt;

[0092] Inputting the task prompt into the cloud big model deployed by the cloud device, and executing the user task through the cloud big model;

[0093] Whenever a network layer in the cloud-based large model is executed, the cloud-based device sends the intermediate result of the task corresponding to the network layer to the edge device.

[0094] Specifically, when the task prompt is a system prompt, the system prompt will be used as the input item of the first network layer in the cloud-based large model. The first network layer constructs a query vector, a key vector, and a value vector based on the system prompt, and forms an output item based on the query vector, the key vector, and the value vector. After obtaining the output item, the key vector and the value vector of the first network layer are stored and the output item is used as the input item of the second network layer, and so on, until the last network layer. Among them, the key vector and the value vector stored in each network layer are used as the intermediate result of the task for the edge device to download, and the contextual knowledge is provided to the edge device through the intermediate result of the task. Similarly, when the task prompt is a user prompt, the user prompt is used as the input item of the first network layer in the cloud-based large model, and then the above process is executed to obtain the intermediate results of the task of each network layer, which is specifically explained here.

[0095] S30. The edge device controls the deployed edge big model to execute the user task to obtain the task result based on the intermediate result of the task and the user prompt corresponding to the user task request, and feeds back the task result to the user end.

[0096] Specifically, after obtaining the intermediate results of the task, the edge big model will execute the user task based on the intermediate results of the task and the user prompts. When the edge big model executes the user task, the first network layer in the edge big model takes the user prompts and the obtained intermediate results of the task as input items, and each network layer except the first network layer takes the output items of the previous network layer and the intermediate results of the task as input items. In addition, for each network layer, the intermediate results of the task maintained by the previous network layer of the corresponding network layer in the cloud big model are taken as input items.

[0097] Based on this, the edge device controls the deployed edge big model to execute the user task based on the task intermediate result and the user prompt corresponding to the user task request to obtain the task result, which specifically includes:

[0098] The edge device determines the network layer corresponding to the intermediate result of the task;

[0099] The edge device selects the current network layer in the deployed edge big model based on the network layer corresponding to the intermediate result of the task;

[0100] The edge device inputs the task result of the previous network layer of the current network layer and the intermediate result of the task into the current network layer, and outputs the task result through the current network layer. If the current network layer is the frontmost network layer, the task result of the previous network layer of the current network layer is a user prompt; if the current network layer is the last network layer, the task result output by the current network layer is a task result for feedback to the user end.

[0101] Specifically, the edge big model deployed on the edge device will perform user tasks based on user prompts, and when the length of the user prompt is s and the length of the generated response is t-1, the query, key and value of the lth layer of the edge big model can be expressed as:

[0102]

[0103] in, represents the query vector of the lth layer of the edge model, represents the key vector of the lth layer of the edge model, represents the value vector of the lth layer of the edge model, represents the input item of the lth layer of the edge model, and Average weight.

[0104] Correspondingly, the attention output of the lth layer of the edge large model can be expressed as:

[0105]

[0106] in, represents the attention output of the lth layer of the edge large model, L = s + u + t-1, Represents the attention weight normalization coefficient.

[0107] Based on this, given a batch of user reasoning requests from the user side, the edge device is responsible for processing and scheduling the user reasoning requests, and collaboratively executes the user tasks corresponding to the user reasoning requests through the cloud device and the edge device. Among them, the cloud large model deployed by the cloud device and the edge large model deployed by the edge device collaborate layer by layer. Finally, when generating step t, the attention output o of the edge large model t It can be expressed as:

[0108]

[0109] in, represents the weight coefficient of the large cloud model, Represents the weight coefficient of the edge large model, q t represents the query vector, k i represents the key vector, vi represents a value vector, and s represents the length of the user prompt.

[0110] In addition, cloud devices and edge devices use a layer-by-layer pipeline parallel approach to collaboratively process tasks. Specifically, the cloud-based large model calculates each network layer in sequence. After the calculation of the first network layer is completed, the key vector and value vector cached in the first network layer are transmitted to the edge device, and the cloud-based large model calculates the second network layer. After the edge device receives the key vector and value vector transmitted by the cloud, the edge large model begins to calculate the first network layer based on the user prompt and the received value vector and key vector, and waits for the cloud device to transmit the key vector and value vector of the second network layer. After receiving the key vector and value vector cached in the second network layer of the cloud-based large model, the edge large model calculates the second network layer based on the output of the first network layer and the key vector and value vector cached in the second network layer of the cloud-based large model, and waits for the cloud device to transmit the key vector and value vector of the third network layer, and so on, the last network layer of the edge large model. In this way, calculation and transmission can achieve maximum overlap, reduce waiting time, and ultimately improve the throughput of the entire architecture.

[0111] The embodiment of the present application uploads the complex computing task part (system prompt) to the cloud device for execution. During the reasoning process, the edge device downloads the value vector and key vector of each layer of the cloud cache to obtain the task-related contextual knowledge, thereby completing the lightweight task (user prompt) sent by the user end. In this way, on the one hand, the resource-constrained edge model does not need to repeatedly calculate long context reasoning, increasing the batch processing of user requests; on the other hand, the KV cache of the large cloud model can be reused in multiple edge devices in the proprietary task system (for example, 6G proprietary tasks, etc.).

[0112] In addition, when cloud devices and edge devices are used to perform tasks collaboratively, dynamic routing decisions can be used to build communication paths between edge devices and cloud devices, where dynamic routing decisions can flexibly select the optimal path based on the real-time environment and the resource status of the device. Specifically, the communication path determination process can be: first, collect the real-time status information of each router and its related links, including data traffic characteristics, topology information, and resource usage information; obtain the congestion information representation of the intelligent agent and its surroundings and the weighted local feature representation of the current node to obtain bandwidth load information, where the weighted local feature representation of the current node is obtained by weighting based on neighbor node information, and finally, through dynamic routing strategies, determine the communication path based on real-time status information and bandwidth load. Of course, in practical applications, other methods can also be used to determine the communication path, such as the shortest path method.

[0113] In summary, this embodiment provides a cloud-edge collaborative reasoning method and reasoning system, the method includes an edge device receiving a user task request and sending a task notification to a cloud device; the cloud device controls the deployed cloud large model to execute a preset task based on the task notification, and sends the task intermediate result obtained during the task execution to the edge device; the edge device controls the deployed edge large model to execute the user task to obtain the task result based on the task intermediate result and the user prompt corresponding to the user task request, and feeds the task result back to the user end. This application collaborates with the edge device and the cloud device, and the cloud device processes the background knowledge related to the user task through the system prompt to obtain the task intermediate result used to reflect the context knowledge, and then the edge device shares the task intermediate result in the cloud device reasoning process, and processes the user prompt user task with high real-time performance and low resource consumption based on the task intermediate result, so that on the one hand, the computing power of the cloud device can be utilized, and on the other hand, the communication delay and data transmission volume can be reduced, thereby reducing the high latency caused by the large bandwidth occupied by data transmission under the premise of ensuring that the user task can be executed, and also reducing the data leakage caused by network attacks, thereby avoiding economic losses caused by data leakage.

[0114] Based on the above cloud-edge collaborative reasoning method, this embodiment provides a cloud-edge collaborative reasoning system, such as Figure 1 As shown, the cloud-edge collaborative reasoning system specifically includes a cloud device and at least one edge device. The edge device is used to receive user task requests and send task notifications to the cloud device. The cloud device is used to control the cloud large model deployed by it to execute preset tasks based on the task notifications, and send the task intermediate results obtained during the task execution process to the edge device; the edge device is also used to control the edge large model deployed by it to execute the user task to obtain the task result based on the task intermediate result and the user prompt corresponding to the user task request, and feed back the task result to the user end.

[0115] In one implementation, in the cloud-edge collaborative reasoning system, the intermediate result of the task is the key vector and value vector cached by the network layer in the cloud-based large model, and the contextual knowledge of the user task is obtained through the key vector and value vector.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A cloud-edge collaborative reasoning method, characterized in that: The cloud-edge collaborative reasoning method specifically includes: The edge device receives the user's task request and sends a task notification to the cloud device; Based on the task notification, the cloud device controls the deployed cloud large model to execute the preset task, and sends the intermediate task results obtained during the task execution process to the edge device; Based on the intermediate result of the task and the user prompt corresponding to the user task request, the edge device controls the deployed edge large model to execute the user task to obtain the task result, and feeds back the task result to the user end.

2. The cloud-edge collaborative reasoning method according to claim 1, characterized in that: Before sending the task notification to the cloud device, the method further includes: Detecting the task type corresponding to the user task request; When the task type is a dedicated task, generating a task notification for executing the user task based on a system prompt; When the task type is a non-proprietary task, a task notification for executing the user task based on a user prompt corresponding to the user task request is generated, and the user prompt is added to the task notification.

3. The cloud-edge collaborative reasoning method according to claim 1, characterized in that: The cloud device controls the deployed cloud large model to execute the preset task based on the task notification, and sends the task intermediate result obtained during the task execution to the edge device, specifically including: The cloud device obtains a task prompt based on the task notification, wherein the task prompt is a system prompt or a user prompt, and the system prompt is background knowledge corresponding to the user task; inputting the task prompt into a cloud-based large model deployed by the cloud-based device; Whenever a network layer in the cloud-based large model is executed, the cloud-based device sends the intermediate result of the task corresponding to the network layer to the edge device.

4. The cloud-edge collaborative reasoning method according to claim 1 or 3, characterized in that: The edge device controls the deployed edge big model to execute the user task to obtain the task result based on the task intermediate result and the user prompt corresponding to the user task request, specifically including: The edge device determines the network layer corresponding to the intermediate result of the task; The edge device selects the current network layer in the deployed edge big model based on the network layer corresponding to the intermediate result of the task; The edge device inputs the task result of the previous network layer of the current network layer and the intermediate result of the task into the current network layer, and outputs the task result through the current network layer. If the current network layer is the frontmost network layer, the task result of the previous network layer of the current network layer is a user prompt; if the current network layer is the last network layer, the task result output by the current network layer is a task result for feedback to the user end.

5. The cloud-edge collaborative reasoning method according to claim 1, characterized in that: The edge device receives the user task request and sends the task notification to the cloud device specifically including: The edge device receives the user's task request and detects its own available device resources; When the available device resources do not meet the preset requirements, a task notification is sent to the cloud device; When the available device resources meet the preset requirements, the cloud large model deployed by the cloud device is downloaded from the cloud device, and the user task corresponding to the user task request is executed through the cloud large model.

6. The cloud-edge collaborative reasoning method according to claim 5, characterized in that: When the available device resources do not meet the preset requirements, sending the task notification to the cloud device specifically includes: The edge device detects the distance between itself and the cloud device; When the interval distance is less than or equal to a preset distance threshold, a task notification is sent to the cloud device, and the cloud device and the edge device are informed by the task notification to collaboratively execute the user task corresponding to the user task request, wherein the motion device executes the preset task based on the system prompt, and the edge device executes the user task based on the intermediate result of the task and the user prompt corresponding to the user task request.

7. The cloud-edge collaborative reasoning method according to claim 6, characterized in that: The method further comprises: When the interval distance is greater than a preset distance threshold, issuing an execution notification to the cloud device, and informing the cloud device through the execution notification to independently execute the user task corresponding to the user task request based on the user prompt; The edge device receives the task result from the cloud device and feeds the task result back to the user end.

8. The cloud-edge collaborative reasoning method according to claim 1, characterized in that: The intermediate result of the task is the key vector and value vector cached by the network layer in the cloud-based large model, and the contextual knowledge of the user task is obtained through the key vector and value vector.

9. A cloud-edge collaborative reasoning system, characterized in that: The cloud-edge collaborative reasoning system specifically includes a cloud device and at least one edge device, wherein the edge device is used to receive a user task request and send a task notification to the cloud device, and the cloud device is used to control the deployed cloud large model to execute a preset task based on the task notification, and send the task intermediate result obtained during the task execution process to the edge device; The edge device is also used to control the deployed edge large model to execute the user task to obtain the task result based on the intermediate result of the task and the user prompt corresponding to the user task request, and feed back the task result to the user end.

10. The cloud-edge collaborative reasoning system according to claim 9, characterized in that: The intermediate result of the task is the key vector and value vector cached by the network layer in the cloud-based large model, and the contextual knowledge of the user task is obtained through the key vector and value vector.

Images