Cloud edge collaborative data distribution system
By using components such as scheduling modules and data distribution modules in the cloud edge collaborative data distribution system, the data distribution strategy is dynamically adjusted, and the response time and bandwidth limitation of data distribution in the coordinated work of cloud computing and edge computing is solved, achieving efficient and secure data processing and transmission improve system performance and user experience.
Patent Information
- Application Number
- CN202411368729.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2025-01-14
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the collaborative work between cloud computing and edge computing, data distribution has response time and bandwidth limitations, and lacks an intelligent scheduling mechanism, resulting in waste of resources and inefficiency.
A cloud-edge collaborative data distribution system was designed, including cloud servers, edge devices, data distribution modules and scheduling modules. Through the scheduling module, the data distribution module is dynamically adjusted according to real-time data traffic, network conditions and edge device load, and the data distribution module is used. Components such as data processing unit and data encryption unit realize efficient data flow and processing.
It significantly reduces data processing delays, optimizes network resource utilization, improves data transmission security and system performance, reduces network load and congestion, and improves user experience.
Smart Images
Figure CN119316426A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data communication technology, and specifically to a cloud-edge collaborative data distribution system. Background Art
[0002] As an emerging computing model, cloud computing can provide powerful storage and computing capabilities. However, in some scenarios, especially in applications with high real-time requirements, the response time and bandwidth limitations of cloud computing often become bottlenecks. At the same time, the rise of edge computing provides a new solution for data processing. Edge computing can effectively reduce latency and improve the real-time performance of data processing by delegating data processing tasks to edge devices closer to the data source.
[0003] However, there are still many technical challenges in the collaboration between cloud computing and edge computing. In terms of data distribution, how to achieve efficient data flow between the cloud and the edge and ensure the timeliness, accuracy and security of data is an urgent problem to be solved. In addition, the current distribution system often lacks an intelligent scheduling mechanism and cannot dynamically adjust the data distribution strategy according to the real-time data flow and network conditions, resulting in waste of resources and low efficiency. Summary of the invention
[0004] 1. Technical issues to be resolved
[0005] In response to the shortcomings of the existing technology, the present invention provides a cloud-edge collaborative data distribution system, which has the advantages of being able to fully utilize the powerful capabilities of cloud computing and the low latency advantages of edge computing, achieve efficient distribution and processing of data, optimize the utilization of network resources, and solve the above-mentioned technical problems.
[0006] (II) Technical solution
[0007] In order to achieve the above-mentioned purpose of making full use of the powerful capabilities of cloud computing and the low latency advantages of edge computing to achieve efficient distribution and processing of data, the present invention provides the following technical solutions: a cloud-edge collaborative data distribution system, including a cloud server, an edge device, a data distribution module and a scheduling module, wherein the cloud server is used to store and process data, the edge device is arranged near the data source, and is used to receive, process and forward data, the data distribution module is used to achieve efficient flow of data between the cloud server and the edge device, and the scheduling module dynamically adjusts the data distribution strategy according to the real-time data flow, network status and load of the edge device. The specific algorithm formula is as follows:
[0008]
[0009] Among them, S ij represents the data distribution decision score from edge device i to cloud server j, Di is the amount of data to be processed by edge device i, L j is the current load of cloud server j, N ij is the network delay time between edge device i and cloud server j, w1, w2, and w3 are weight coefficients, representing the relative importance of data volume, edge device load, and network delay in the distribution decision, respectively.
[0010] Preferably, the scheduling module further includes a traffic prediction unit, which uses a machine learning algorithm to predict future data traffic patterns. The specific algorithm formula is as follows:
[0011]
[0012] in, is the predicted data flow at time t+Δt, F(t-a+1) is the historical data flow value at time t-a+1, where a ranges from 1 to p, p is the number of historical data points, and X b (t) is the bth feature variable at time t, where b ranges from 1 to m, m is the number of feature variables, and β a is the regression coefficient of historical data flow, γ x is the regression coefficient of the characteristic variable, β o is the intercept term, and ∈ is the error term, which represents the difference between the model prediction and the actual value.
[0013] Preferably, the data distribution module includes a data receiving unit, a data processing unit and a data forwarding unit. The data receiving unit is used to receive data from a cloud server and an edge device, the data processing unit is used to process and analyze the received data, and the data forwarding unit is used to forward the processed data to a target device or storage location.
[0014] Preferably, the data processing unit adopts a parallel processing architecture for processing multiple types of data streams simultaneously.
[0015] Preferably, the scheduling module further includes a load balancing unit for monitoring the real-time load status of the edge device.
[0016] Preferably, the data distribution module further comprises a data compression unit and a data encryption unit, which are used to compress the data before data transmission, and the data encryption unit is used to encrypt the data during data transmission and storage.
[0017] Preferably, the system further includes an adaptive cache management module, which determines the cache strategy of data in the edge device according to the access frequency and lifetime of the data. The specific algorithm formula is as follows:
[0018]
[0019] Among them, C represents the usage of cache capacity, H k is the access frequency of the kth data, E k is the survival time of the kth data, and V is the total amount of data.
[0020] Preferably, the system further includes a user management module, and the user management module is used to allow an administrator to monitor the system status and adjust the data distribution strategy.
[0021] Compared with the prior art, the present invention provides a cloud-edge collaborative data distribution system with the following features:
[0022] Beneficial effects:
[0023] 1. The present invention provides a cloud-edge collaborative data distribution system, which utilizes the low-latency characteristics of edge computing. Data can be initially processed and quickly responded to on edge devices close to the data source, reducing the waiting time for data transmission to the cloud, thereby significantly reducing the overall processing delay. By setting a scheduling module, the data distribution strategy is dynamically adjusted according to real-time data traffic, network conditions and the load of edge devices, ensuring that data can flow efficiently between cloud servers and edge devices.
[0024] 2. A cloud-edge collaborative data distribution system provided by the present invention, through a data distribution strategy, combines the advantages of edge devices and cloud servers, reasonably allocates data processing tasks, effectively reduces network load, optimizes the use of network bandwidth, and further reduces the data transmission volume and improves the security of data transmission through data compression and encryption units. At the same time, it reduces the demand for network bandwidth, making network resources more efficiently utilized. Through the adaptive cache management module, it ensures that high-frequency access data is effectively cached, reduces repeated requests to the cloud, reduces network congestion, and improves the overall performance of the system and user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a schematic diagram of the framework of the system of the present invention; DETAILED DESCRIPTION
[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0027] See also Figure 1,A cloud-edge collaborative data distribution system includes a cloud server, an edge device, a data distribution module and a scheduling module. The cloud server is used to store and process data. The edge device is set near the data source and is used to receive, process and forward data. The data distribution module is used to achieve efficient data flow between the cloud server and the edge device. The scheduling module dynamically adjusts the data distribution strategy according to the real-time data flow, network status and load of the edge device. The specific algorithm formula is as follows:
[0028]
[0029] Among them, S ij represents the data distribution decision score from edge device i to cloud server j, D i is the amount of data to be processed by edge device i, L j is the current load of cloud server j, N ij is the network delay time between edge device i and cloud server j, w1, w2, and w3 are weight coefficients, representing the relative importance of data volume, edge device load, and network delay in the distribution decision, respectively.
[0030] Specifically, the scheduling module also includes a traffic prediction unit, which uses a machine learning algorithm to predict future data traffic patterns. The specific algorithm formula is as follows:
[0031]
[0032] in, is the predicted data flow at time t+Δt, F(t-a+1) is the historical data flow value at time t-a+1, where a ranges from 1 to p, p is the number of historical data points, and X b (t) is the bth feature variable at time t, where b ranges from 1 to m, m is the number of feature variables, and β a is the regression coefficient of historical data flow, γ x is the regression coefficient of the characteristic variable, β o is the intercept term, ∈ is the error term, which represents the difference between the model prediction and the actual value. The advantage is that in this model formula, not only the autoregressive part of the historical data flow is considered, but also feature engineering is added to include other factors that may affect the data flow. In this way, the dynamic changes of data flow can be captured more comprehensively and the accuracy of the prediction can be improved. By considering the historical data flow values and multiple feature variables, the model is able to capture the trends and patterns of data flow changes over time, thereby improving the accuracy of the prediction.
[0033] Specifically, the data distribution module includes a data receiving unit, a data processing unit and a data forwarding unit. The data receiving unit is used to receive data from the cloud server and the edge device, the data processing unit is used to process and analyze the received data, and the data forwarding unit is used to forward the processed data to the target device or storage location. The advantage is that the data receiving unit receives the data from both the cloud and the edge according to the distribution strategy formulated by the scheduling module, and then sends the data to the data processing unit. The data processing unit performs processing including data cleaning, format conversion, aggregation, etc. on the received data, and then sends the processed data to the data forwarding unit. The processed data is forwarded to the target device or storage location by the data forwarding unit, which ensures that the data can reach the destination quickly, reduces the residence time of the data in the system, and improves the overall data distribution efficiency.
[0034] Specifically, the data processing unit adopts a parallel processing architecture for processing multiple types of data streams simultaneously. The advantage is that the use of a parallel processing architecture means that the system can process multiple types of data streams simultaneously, which greatly improves the efficiency of data processing and the throughput of the system, especially when processing large amounts of or diverse data, and avoids idle resources and processing bottlenecks caused by single-threaded processing.
[0035] Specifically, the scheduling module further includes a load balancing unit for monitoring the real-time load of edge devices. The advantage is that the load balancing unit can monitor the real-time load of edge devices, so as to reasonably allocate tasks and data flows, which ensures that edge devices will not reduce performance or fail due to overload, and can also make full use of the processing power of the device to avoid waste of resources. At the same time, the scheduling module dynamically adjusts the data distribution strategy according to the real-time data flow, network conditions and load conditions of edge devices; as part of the scheduling module, the load balancing unit can monitor the load of edge devices in real time, which helps the system make intelligent decisions according to the current load status and optimize data distribution. Considering the network delay time, the load balancing unit will consider the network conditions when allocating tasks, which helps to reduce data transmission time and improve network efficiency. At the same time, the load balancing unit can quickly respond to the load changes of edge devices and adjust the data distribution strategy in time, thereby improving the overall response speed and processing capacity of the system.
[0036] Specifically, the data distribution module further includes a data compression unit and a data encryption unit, which are used to compress data before data transmission, and the data encryption unit is used to encrypt data during data transmission and storage. The advantage is that the data compression unit compresses data before data transmission, which can significantly reduce the amount of data to be transmitted, thereby speeding up the transmission speed of data in the network, reducing bandwidth consumption, and improving the overall transmission efficiency. The compressed data occupies less storage space, which helps to reduce storage costs. The data encryption unit encrypts data during data transmission and storage, ensuring the security of data during transmission and storage, preventing data from being accessed or stolen by unauthorized persons, and protecting user privacy and corporate secrets.
[0037] Specifically, the system also includes an adaptive cache management module, which determines the cache strategy of data in the edge device according to the access frequency and survival time of the data. The specific algorithm formula is as follows:
[0038]
[0039] Among them, C represents the usage of cache capacity, H k is the access frequency of the kth data, E k is the survival time of the kth data, and V is the total amount of data. The advantage is that the adaptive cache management module determines the cache strategy of data on the edge device according to the access frequency and survival time of the data, which means that the system can intelligently identify which data is accessed more frequently, so as to cache these data first and improve the data access speed. Through the cache capacity usage mentioned in the formula, the adaptive cache management module can dynamically adjust the allocation of cache space to ensure that the cache space is used efficiently and avoid the waste of cache space. By caching commonly used data on the edge device, the number of requests to the cloud server can be reduced, reducing the network load.
[0040] Specifically, the system further includes a user management module, which is used to allow administrators to monitor system status and adjust data distribution policies. The advantage is that the user management module allows administrators to monitor system status, which helps to timely discover and deal with security threats and ensure the safe operation of the system. Through effective user management and policy adjustment, users can be ensured to have a better service experience, such as reducing delays and providing more stable services by optimizing data distribution.
[0041] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A cloud-edge collaborative data distribution system, comprising a cloud server, an edge device, a data distribution module and a scheduling module, characterized in that: The cloud server is used to store and process data. The edge device is set near the data source to receive, process and forward data. The data distribution module is used to achieve efficient data flow between the cloud server and the edge device. The scheduling module dynamically adjusts the data distribution strategy according to the real-time data flow, network status and edge device load. The specific algorithm formula is as follows: Among them, S ij represents the data distribution decision score from edge device i to cloud server j, D i is the amount of data to be processed by edge device i, L j is the current load of cloud server j, N ij is the network delay time between edge device i and cloud server j, w1, w2, and w3 are weight coefficients, representing the relative importance of data volume, edge device load, and network delay in the distribution decision, respectively.
2. A cloud-edge collaborative data distribution system according to claim 1, characterized in that: The scheduling module also includes a traffic prediction unit, which uses a machine learning algorithm to predict future data traffic patterns. The specific algorithm formula is as follows: in, is the predicted data flow at time t+Δt, F(t-a+1) is the historical data flow value at time t-a+1, where a ranges from 1 to p, p is the number of historical data points, and X b (t) is the bth feature variable at time t, where b ranges from 1 to m, m is the number of feature variables, and β a is the regression coefficient of historical data flow, γ x is the regression coefficient of the characteristic variable, β o is the intercept term, and ∈ is the error term, which represents the difference between the model prediction and the actual value.
3. The cloud-edge collaborative data distribution system according to claim 1, characterized in that: The data distribution module includes a data receiving unit, a data processing unit and a data forwarding unit. The data receiving unit is used to receive data from a cloud server and an edge device, the data processing unit is used to process and analyze the received data, and the data forwarding unit is used to forward the processed data to a target device or storage location.
4. A cloud-edge collaborative data distribution system according to claim 3, characterized in that: The data processing unit adopts a parallel processing architecture to process multiple types of data streams simultaneously.
5. The cloud-edge collaborative data distribution system according to claim 1, characterized in that: The scheduling module further includes a load balancing unit for monitoring the real-time load status of edge devices.
6. The cloud-edge collaborative data distribution system according to claim 1, characterized in that: The data distribution module further includes a data compression unit and a data encryption unit, which are used to compress the data before data transmission. The data encryption unit is used to encrypt the data during data transmission and storage.
7. The cloud-edge collaborative data distribution system according to claim 1, characterized in that: The system also includes an adaptive cache management module, which determines the cache strategy of data in the edge device according to the access frequency and survival time of the data. The specific algorithm formula is as follows: Among them, C represents the usage of cache capacity, H k is the access frequency of the kth data, E k is the survival time of the kth data, and V is the total amount of data.
8. The cloud-edge collaborative data distribution system according to claim 1, characterized in that: The system further includes a user management module, which is used to allow an administrator to monitor system status and adjust data distribution strategies.