Large model deployment method and system based on edge device and cloud collaboration

Abstract

The embodiment of the invention provides a large model deployment method and system based on edge device and cloud collaboration.The method comprises the steps that a single large model is dynamically divided into two reasoning modules which are collaborative in function according to the real-time resource states of the edge side and the cloud side, the first reasoning module is deployed on the edge device, and the second reasoning module is deployed on the cloud side; the second reasoning module is deployed on the cloud server; processing the received reasoning request through the first reasoning module, and generating a decision signal according to a processing result; and based on the decision signal, dynamically selecting to independently complete reasoning on the edge side or request the second reasoning module to participate in collaborative reasoning so as to generate a final reasoning result.

Description

Technical Field

[0001] This document relates to the field of edge computing technology, and in particular to a method and system for deploying large-scale models of edge devices and cloud collaboration. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models with massive parameters have demonstrated outstanding performance in natural language processing, content generation, and intelligent question answering. However, the deployment and inference services of such models face severe challenges. Traditional deployment paradigms mainly include full cloud deployment and full edge deployment, both of which have inherent drawbacks. The full cloud deployment model is insufficient because, although it can utilize the abundant computing power of the cloud to ensure the complete performance of the model, all computing tasks must be completed on remote servers via the network, resulting in huge network bandwidth overhead and significant communication latency. Especially in high-concurrency or large-data-volume application scenarios, network transmission becomes a performance bottleneck, making it difficult for the system to meet real-time interaction requirements and incurring high bandwidth costs for users. The full edge deployment model is insufficient because, although it uses model compression and other technologies to deploy the model locally to achieve instant response and data privacy protection, the inherently limited computing power and memory resources of edge devices force the model to be significantly simplified. This inevitably leads to a serious decline in model accuracy, inference ability, and performance in handling complex tasks, making it impossible for many advanced applications to be effectively carried out at the edge and sacrificing the quality of intelligent services. There is an irreconcilable contradiction between the two in terms of deployment efficiency, bandwidth consumption, inference performance and real-time performance. Moving all large models to the cloud ensures performance but sacrifices response speed and bandwidth cost; moving all large models to the edge ensures real-time performance but sacrifices inference accuracy and the ability to handle complex tasks. Summary of the Invention

[0003] This specification provides one or more embodiments of a large-scale deployment method for edge device and cloud collaboration, including: Based on the real-time resource status of the edge and cloud sides, a single large model is dynamically divided into two functionally collaborative inference modules. The first inference module is deployed on the edge device, and the second inference module is deployed on the cloud server. The first inference module processes the received inference request and generates a decision signal based on the processing result. Based on the decision signal, the system dynamically selects to complete the reasoning independently at the edge or to request the second reasoning module to participate in collaborative reasoning in order to generate the final reasoning result.

[0004] Furthermore, the real-time resource status includes one or more of the following: computing power of edge devices, memory usage, cloud server load, and network bandwidth.

[0005] Furthermore, the specific steps of dynamically dividing a single large model into two functionally collaborative inference modules are as follows: The large model is divided according to the network layer depth. The first inference module contains the first N layers of the model, and the second inference module contains the remaining layers of the model, where N is a positive integer dynamically determined according to the real-time resource status.

[0006] Furthermore, the specific steps of generating a decision signal based on the processing result are as follows: An intermediate result is generated by the first inference module, and the confidence evaluation value of the intermediate result is calculated. The confidence evaluation value is used as the decision signal.

[0007] Furthermore, the specific method for dynamically selecting whether to complete reasoning independently at the edge or to request the second reasoning module to participate in collaborative reasoning is as follows: If the confidence assessment value is higher than a preset threshold, the first inference module generates the final inference result based on the intermediate result; If the confidence assessment value is lower than or equal to the preset threshold, the intermediate result is uploaded to the second inference module, which then performs further processing on the intermediate result to generate the final inference result and return it to the edge side.

[0008] Furthermore, when the decision requires the second inference module to participate in collaborative inference, the uploaded intermediate results are adaptively compressed based on the current network conditions.

[0009] Furthermore, the method also includes: The inference requests processed at the edge and their corresponding final inference results are asynchronously uploaded to the cloud for updating and optimizing the large model.

[0010] This specification provides one or more embodiments of a large-scale model deployment system for edge device and cloud collaboration, used to implement the aforementioned large-scale model deployment method for edge device and cloud collaboration, including: The dynamic split management unit is used to dynamically divide a single large model into two functionally collaborative inference modules based on the real-time resource status of the edge and cloud sides. An edge-side inference unit, deployed on an edge device, is used to carry the first inference module and execute local decisions; The cloud-side inference unit is deployed on a cloud server and is used to host the second inference module; The collaborative communication unit is used to manage data transmission and decision signal transmission between the edge-side inference unit and the cloud-side inference unit.

[0011] This specification provides one or more embodiments of an electronic device, including: Processor; and, A memory is configured to store computer-executable instructions, which, when executed, cause the processor to implement the steps of the fault diagnosis method based on wind turbine vibration data described above.

[0012] This specification provides one or more embodiments of a storage medium for storing computer-executable instructions, which, when executed, implement the steps of the fault judgment method based on wind turbine vibration data described above.

[0013] This invention, through its dynamic splitting-collaborative inference core architecture, creatively breaks the binary dilemma of full cloud deployment versus full edge deployment, achieving complementary advantages between the two. While maintaining near-cloud-based high-precision inference capabilities for large models, it significantly reduces reliance on network bandwidth and improves system response efficiency. By dynamically splitting the model, this application effectively solves the performance loss problem of native deployment of large models in resource-constrained edge environments, as well as the inherent latency and bandwidth costs of centralized cloud deployment. It achieves an optimal balance between deployment efficiency and inference performance, providing an efficient and feasible solution for the widespread deployment of large models in scenarios with high real-time requirements and variable network conditions, such as the Internet of Things and mobile computing.

[0014] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating a method for deploying a large-scale model of edge device and cloud collaboration, provided for one or more embodiments of this specification; Figure 2 A schematic diagram illustrating the composition of a large-scale deployment system for edge device and cloud collaboration, provided for one or more embodiments of this specification; Figure 3 This is a schematic diagram of the structure of an electronic device provided for one or more embodiments of this specification. Detailed Implementation

[0017] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0018] Method Implementation Examples According to embodiments of the present invention, a method for deploying a large-scale model in collaboration between edge devices and the cloud is provided. Figure 1 A flowchart illustrating a large-scale deployment method for edge device and cloud collaboration provided in one or more embodiments of this specification, such as... Figure 1 As shown, the large-scale deployment method for edge device and cloud collaboration according to an embodiment of the present invention specifically includes: S1. Based on the real-time resource status of the edge and cloud sides, a single large model is dynamically divided into two functionally collaborative inference modules, wherein the first inference module is deployed on the edge device and the second inference module is deployed on the cloud server.

[0019] In scenarios where large models are deployed collaboratively on edge devices and in the cloud, the core lies in achieving efficient model splitting and collaborative inference by dynamically adapting resource status. First, the real-time resource status of both the edge and cloud sides is perceived. This real-time resource status includes one or more of the following: computing power of the edge device, memory utilization, cloud server load, and network bandwidth. Using this real-time resource status as the basis for decision-making ensures that the splitting strategy can respond to resource fluctuations in real time.

[0020] Based on the aforementioned resource status, a single large model is dynamically partitioned into two functionally closely coordinating inference modules. Since most large models employ a layered, progressive structure, with the initial network layers typically responsible for extracting basic features and subsequent layers focusing on advanced semantic understanding, the model is split according to the depth characteristics of its network layers. The first inference module contains the first N layers of the model, deployed on edge devices, and handles initial processing tasks. The second inference module contains the remaining network layers, deployed on a cloud server, and handles complex deep inference. The value of N is not fixed but a positive integer dynamically determined based on the real-time resource status. When the edge device has sufficient computing power and low memory usage, the value of N increases, allowing the edge to handle more basic feature extraction work and reducing pressure on the cloud. If edge device resources are limited or network bandwidth is scarce, the value of N decreases, retaining only the most critical first few layers at the edge and offloading most computational tasks to the cloud.

[0021] S2. The first inference module processes the received inference request and generates a decision signal based on the processing result.

[0022] When an edge device receives an inference request from a real-world scenario, the first inference module deployed on the edge device performs preliminary feature extraction and computation on the data contained in the inference request based on the first N layers of the large model's network structure. This transforms the raw data into information with certain semantic or feature representation meaning, generating an intermediate result. After generating the intermediate result, the first inference module further calculates the confidence evaluation value of the intermediate result and uses this evaluation value as a decision signal. The confidence evaluation value is a key indicator for measuring the reliability and effectiveness of the intermediate result. It includes analyzing whether the feature data corresponding to the intermediate result is complete, whether there is any abnormal interference information, or judging the degree of matching between the current intermediate result and the effective features in the training samples based on the processing effect of the first N layers of the large model's network on similar feature data during training. The higher the matching degree, the more reliable the intermediate result. A high confidence rating indicates that the intermediate results are highly reliable, and the edge device is likely to obtain an accurate final inference result by virtue of its own first inference module, without having to occupy network resources to request cloud collaboration; while a low confidence rating indicates that the intermediate results are not reliable enough, and it is difficult to guarantee the inference accuracy by relying solely on edge processing. The intermediate results need to be uploaded to the cloud for further in-depth processing by the second inference module.

[0023] S3. Based on the decision signal, dynamically select to complete the reasoning independently at the edge or request the second reasoning module to participate in collaborative reasoning to generate the final reasoning result.

[0024] After the first inference module generates a decision signal centered on the confidence assessment value, it enters the dynamic selection stage of the inference path. A threshold is used to determine the subject undertaking the inference task, thus achieving an optimal balance between real-time performance and accuracy. The preset threshold is not a fixed value, but is comprehensively set based on the needs of specific application scenarios, the computing power limit of edge devices, cloud resource scheduling strategies, and feedback from historical inference data. When the confidence assessment value is higher than this preset threshold, it means that the intermediate results generated by the first inference module on the edge side have sufficient reliability and can complete high-precision inference without relying on cloud resources. At this time, the first inference module will directly generate the final inference result based on the existing intermediate results, without interacting with the cloud. If the confidence assessment value is lower than or equal to the preset threshold, the edge-cloud collaborative inference mode is automatically triggered. The edge device uploads the intermediate results generated by the first inference module to the second inference module in the cloud. Before uploading, the system first performs real-time detection of the current network conditions, including key indicators such as real-time network bandwidth transmission rate, network latency, and packet loss rate, and adaptively compresses the intermediate results based on these indicators. When network bandwidth is sufficient, a low compression rate is used to preserve the detailed features of intermediate results to the maximum extent and avoid the loss of accuracy in subsequent cloud processing due to compression. When network bandwidth is tight or network latency is high, the compression rate will be automatically increased. Through feature quantization, sparse coding and other methods, the amount of data will be reduced to 1 / 5 or even 1 / 10 of the original amount while ensuring that core features are not lost. This will significantly reduce the network resource consumption of data transmission and avoid inference task timeouts due to data congestion.

[0025] After the compressed intermediate results are transmitted to the cloud, the second inference module performs further in-depth processing on the intermediate results based on the remaining network layers of the large model it carries, generates the final inference result, and returns it to the edge side.

[0026] The method also includes asynchronously uploading the inference requests processed on the edge side and their corresponding final inference results to the cloud for updating and optimizing the large model.

[0027] The final inference result is uploaded asynchronously during periods of network idleness and computing power redundancy on the edge device, avoiding interference with the real-time response performance at the edge. The uploaded content includes not only the raw data of the inference request but also the final inference result generated at the edge, as well as key intermediate information from the inference process, such as confidence assessment values ​​and decision path records. Based on the uploaded data, the cloud performs targeted updates and optimizations to the large model, including: fine-tuning with small samples using the correspondence between inference requests and final results from the edge feedback, making model parameters more suitable for the actual data distribution in edge scenarios; and optimizing the model's hierarchical logic and confidence threshold settings by combining decision records from the inference process, adjusting the N-value calculation strategy for dynamic splitting, making the model more intelligent in resource allocation and inference path selection.

[0028] The beneficial effects of this invention are as follows: This invention, through its dynamic splitting-collaborative inference core architecture, creatively breaks the binary dilemma of full cloud deployment versus full edge deployment, achieving complementary advantages between the two. While maintaining near-cloud-based high-precision inference capabilities for large models, it significantly reduces reliance on network bandwidth and improves system response efficiency. By dynamically splitting the model, this application effectively solves the performance loss problem of native deployment of large models in resource-constrained edge environments, as well as the inherent latency and bandwidth costs of centralized cloud deployment. It achieves an optimal balance between deployment efficiency and inference performance, providing an efficient and feasible solution for the widespread deployment of large models in scenarios with high real-time requirements and variable network conditions, such as the Internet of Things and mobile computing.

[0029] System Implementation Examples According to embodiments of the present invention, a large-scale deployment system for edge devices and cloud collaboration is provided. Figure 2 This specification provides a schematic diagram illustrating the composition of a large-scale deployment system for edge device and cloud collaboration, as shown in one or more embodiments. Figure 2 As shown, the large-scale model deployment system for edge device and cloud collaboration according to an embodiment of the present invention specifically includes: The dynamic split management unit 20 is used to dynamically divide a single large model into two functionally collaborative inference modules based on the real-time resource status of the edge side and the cloud side. Edge-side inference unit 22, deployed on an edge device, is used to carry the first inference module and execute local decisions; The cloud-side inference unit 24 is deployed on a cloud server and is used to carry the second inference module; The collaborative communication unit 26 is used to manage the data transmission and decision signal transmission between the edge-side inference unit and the cloud-side inference unit.

[0030] The embodiments of the present invention are system embodiments corresponding to the above method embodiments. The specific operation of each module can be understood by referring to the description of the method embodiments, and will not be repeated here.

[0031] Device Example 1 This invention provides an electronic device, such as... Figure 3 As shown, it includes: a memory 30, a processor 32, and a computer program stored in the memory 30 and executable on the processor 32. When the computer program is executed by the processor 32, it performs the following method steps: S1. Based on the real-time resource status of the edge and cloud sides, a single large model is dynamically divided into two functionally collaborative inference modules, wherein the first inference module is deployed on the edge device and the second inference module is deployed on the cloud server. S2. The first inference module processes the received inference request and generates a decision signal based on the processing result; S3. Based on the decision signal, dynamically select to complete the reasoning independently at the edge or request the second reasoning module to participate in collaborative reasoning to generate the final reasoning result.

[0032] Device Example 2 This invention provides a computer-readable storage medium storing an information transmission implementation program. When executed by a processor 32, the program performs the following method steps: S1. Based on the real-time resource status of the edge and cloud sides, a single large model is dynamically divided into two functionally collaborative inference modules, wherein the first inference module is deployed on the edge device and the second inference module is deployed on the cloud server. S2. The first inference module processes the received inference request and generates a decision signal based on the processing result; S3. Based on the decision signal, dynamically select to complete the reasoning independently at the edge or request the second reasoning module to participate in collaborative reasoning to generate the final reasoning result.

[0033] The computer-readable storage media described in this embodiment include, but are not limited to, ROM, RAM, disk, or optical disk.

[0034] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for deploying a large-scale model involving edge devices and the cloud, characterized in that, include: Based on the real-time resource status of the edge and cloud sides, a single large model is dynamically divided into two functionally collaborative inference modules. The first inference module is deployed on the edge device, and the second inference module is deployed on the cloud server. The first inference module processes the received inference request and generates a decision signal based on the processing result. Based on the decision signal, the system dynamically selects to complete the reasoning independently at the edge or to request the second reasoning module to participate in collaborative reasoning in order to generate the final reasoning result.

2. The method according to claim 1, characterized in that, The real-time resource status includes one or more of the following: edge device computing power, memory usage, cloud server load, and network bandwidth.

3. The method according to claim 1, characterized in that, The specific steps of dynamically dividing a single large model into two functionally collaborative inference modules are as follows: The large model is divided according to the network layer depth. The first inference module contains the first N layers of the model, and the second inference module contains the remaining layers of the model, where N is a positive integer dynamically determined according to the real-time resource status.

4. The method according to claim 1, characterized in that, The specific steps for generating a decision signal based on the processing result are as follows: An intermediate result is generated by the first inference module, and the confidence evaluation value of the intermediate result is calculated. The confidence evaluation value is used as the decision signal.

5. The method according to claim 4, characterized in that, The specific method for determining whether to dynamically select whether to complete inference independently on the edge side or to request the second inference module to participate in collaborative inference is as follows: If the confidence assessment value is higher than a preset threshold, the first inference module generates the final inference result based on the intermediate result; If the confidence assessment value is lower than or equal to the preset threshold, the intermediate result is uploaded to the second inference module, which then performs further processing on the intermediate result to generate the final inference result and return it to the edge side.

6. The method according to claim 5, characterized in that, When the decision requires the second inference module to participate in collaborative inference, the uploaded intermediate results are adaptively compressed based on the current network conditions.

7. The method according to claim 1, characterized in that, The method further includes: The inference requests processed at the edge and their corresponding final inference results are asynchronously uploaded to the cloud for updating and optimizing the large model.

8. A large-scale model deployment system for edge device and cloud collaboration, used to implement the large-scale model deployment method for edge device and cloud collaboration according to any one of claims 1-7, characterized in that, include: The dynamic split management unit is used to dynamically divide a single large model into two functionally collaborative inference modules based on the real-time resource status of the edge and cloud sides. An edge-side inference unit, deployed on an edge device, is used to carry the first inference module and execute local decisions; The cloud-side inference unit is deployed on a cloud server and is used to host the second inference module; The collaborative communication unit is used to manage data transmission and decision signal transmission between the edge-side inference unit and the cloud-side inference unit.

9. An electronic device, characterized in that, include: processor; as well as, A memory configured to store computer-executable instructions, which, when executed, cause the processor to implement the steps of the large-scale deployment method for edge device and cloud collaboration as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, Used to store computer-executable instructions, which, when executed, implement the steps of the large-scale deployment method for edge device and cloud collaboration as described in any one of claims 1 to 7.

Images