An adaptive cloud edge-end collaboration method and device

Abstract

The application relates to the technical field of Internet of Things time series data processing, and specifically provides a self-adaptive cloud edge-end cooperation method which has the following steps: S1, cloud side cloud service storage and calculation, edge side nearby calculation of a time series data source, and the end side is a collection end directly connected with the device; S2, self-adaptive time series data distribution, automatically adjusting the time span of time series data storage at the cloud side, the edge side and the end side; S3, self-adaptive time series data pre-computation, reasonably selecting an optimal computation node for time series data pre-computation according to the available resources of the cloud edge-end; S4, self-adaptive time series data compression, reasonably selecting an optimal node for time series data compression according to the available resources of the cloud edge-end; S5, self-adaptive time series data query analysis, automatically selecting an optimal computation node for time series data query analysis according to the distribution of the time series data. Compared with the prior art, the application can automatically optimize, optimize time series data processing, and reduce system maintenance and optimization costs.

Description

Technical Field

[0001] This invention relates to the field of IoT time-series data processing technology, specifically providing an adaptive cloud-edge-device collaboration method and system. Background Technology

[0002] With the rapid development of IoT-related technologies and applications, more and more devices are becoming smart devices and connecting to the IoT. A massive influx of sensor data into the IoT environment not only demands cloud services from IoT time-series data management platforms but also places higher demands on the real-time performance and efficiency of time-series data processing. However, traditional cloud computing architectures, being geographically distant from time-series data sources, struggle to achieve optimal computation, failing to meet the performance requirements of different levels of IoT applications and fully utilizing computing resources located near the time-series data. Therefore, more and more enterprises are turning their attention to edge computing as an extension of the cloud to accelerate data analysis, enabling faster and better decision-making and making production processes more sustainable. Edge computing refers to computation close to the time-series data source, providing the nearest and fastest time-series data services locally. Its core concept is to entrust the storage, transmission, and computation of time-series data to edge nodes. Applications originate at the edge, resulting in faster network service responses and meeting the needs of various industries in real-time business and application intelligence. However, in complex IoT application environments, uneven distribution of system resources, time-series data, and application load across the cloud, edge, and device sides leads to unreasonable use of system resources and poor performance in time-series data processing. To improve system performance, when there is no data in the cloud or available system resources are scarce, tasks such as time-series data query analysis, data pre-computation, and compression can be offloaded to the edge or even the device side through cloud-edge-device collaboration. Similarly, if there is no historical data or available system resources are scarce on the edge side, these tasks can be migrated to the cloud or device side through cloud-edge-device collaboration. Furthermore, if there is no historical data or available system resources are scarce on the device side, these tasks can also be submitted to the edge or even the cloud side through cloud-edge-device collaboration.

[0003] With the rise of edge computing, IoT time-series data management platforms are becoming increasingly complex. The uneven distribution of storage, computing, and network resources across the cloud, edge, and device sides presents a significant challenge. A key technical challenge for IoT time-series data management is how to fully utilize these system resources, dynamically select the optimal nodes for storage, computing, and query analysis, and improve time-series data processing performance. Furthermore, given the diverse nature of time-series data and application requirements across various IoT application industries, optimizing time-series data deployment and computation to improve service quality is another crucial aspect of IoT time-series data management. Summary of the Invention

[0004] This invention addresses the shortcomings of the prior art by providing a highly practical adaptive cloud-edge-device collaboration method.

[0005] A further technical objective of this invention is to provide a reasonably designed, safe, and applicable adaptive cloud-edge-device collaborative device.

[0006] The technical solution adopted by this invention to solve its technical problem is:

[0007] An adaptive cloud-edge-device collaboration method has the following steps:

[0008] S1. Cloud-edge-device refers to the cloud side, edge side, and device side. The cloud side performs cloud service storage and computation, the edge side performs local computation of time-series data sources, and the device side is the acquisition end that is directly connected to the device.

[0009] S2. Adaptive time-series data distribution, automatically adjusting the time span of time-series data stored on the cloud side, edge side, and terminal side respectively;

[0010] S3. Adaptive time series data pre-computation: Based on the available resources in the cloud, edge, and terminal, the optimal computing node is selected for time series data pre-computation.

[0011] S4. Adaptive time-series data compression: Based on the available resources in the cloud, edge, and terminal, the optimal node is selected for time-series data compression.

[0012] S5. Adaptive time series data query and analysis: Based on the distribution of time series data, it automatically selects the optimal computing node for time series data query and analysis.

[0013] Furthermore, in step S1, after the edge side collects the sensor data from the device, it performs a short-term cache in memory and synchronizes the timing data to the edge side.

[0014] After receiving data, the edge device stores the time-series data for a certain period of time and periodically synchronizes the time-series data to the cloud; the cloud device stores the time-series data as historical data for a long period of time.

[0015] Furthermore, step S2 further includes:

[0016] S2-1. The smallest unit of data distribution in the time series engine is called ZObjectTable. Each ZObjectTable has a timestamp and multiple sensor data.

[0017] S2-2, Determine the time span during which each ZObjectTable is stored in the client-side memory;

[0018] S2-3. Determine the time span for storing each ZObjectTable on the edge-side disk;

[0019] S2-4. Determine the maximum latency for storing each ZObjectTable on the cloud-side disk;

[0020] S2-5. The lifecycle of each ZObjectTable in the cloud disk is determined by the user based on the needs of IoT applications and disk resources. Historical data that exceeds the lifecycle will be automatically deleted periodically.

[0021] Furthermore, step S3 further includes:

[0022] S3-1. Automatically add or delete pre-calculated data;

[0023] S3-2. Automatically select the optimal pre-computed node.

[0024] Furthermore, step S3-1 includes:

[0025] S3-1-1 When the same query calculation occurs continuously within a period of time, time series data pre-calculation is automatically created. The calculation is performed when the time series data is entered into the database, and the results are saved. During the time series data query and analysis process, the pre-calculation results will be automatically matched. If the match is successful, the pre-calculation results will be read directly, and then subsequent query calculations will be performed according to the query and analysis requirements, and the results will be returned in the end.

[0026] S3-1-2. If the pre-calculated data has not been used for a long period of time, it will be automatically deleted.

[0027] S3-1-3. If the pre-calculated load efficiency value is less than 1, it means that the pre-calculation has reduced the overall system load. Therefore, even under overload conditions, the pre-calculation will continue to be used.

[0028] If the pre-calculated load efficiency value is greater than 1, it means that the pre-calculation has increased the load overall. When overloaded, the pre-calculation with a lower load efficiency value needs to be deleted.

[0029] The specific definition of the pre-calculated CPU load efficiency value of P is as follows:

[0030]

[0031] Where Period is the time period for evaluating the pre-computed load efficiency value; Interval is the time interval for aggregation and grouping; m is the number of query requests involving pre-computed P within the Period, and the corresponding query request is Q. i , i∈[1,m]; QSpan(Q i ) represents the time span of each query; Uc The CPU resources consumed in a single pre-calculation;

[0032] The disk efficiency value for pre-calculated P is defined as follows:

[0033]

[0034] Among them, D c The disk read / write resources consumed in reading data from the disk without prior calculation; D write When pre-computation is used, the read / write resources consumed in writing the pre-computed data to disk; D read The disk read / write resources consumed when reading the pre-calculation results from the disk when pre-calculation is used;

[0035] S3-1-4. When CPU overload occurs, automatically delete the pre-calculated CPU load efficiency value with the lowest value until the CPU is no longer overloaded.

[0036] S3-1-5. When disk overload occurs, automatically delete the pre-calculated disks with the lowest load efficiency values ​​until the disk is no longer overloaded.

[0037] Furthermore, step S3-2 includes:

[0038] S3-2-1. If there is a pre-calculation query requirement on the endpoint, the pre-calculation is performed on the endpoint and the pre-calculation result is cached. At this time, if there is also a pre-calculation query requirement on the edge, the pre-calculation result on the endpoint will be copied to the edge for storage.

[0039] If there is also a pre-computation query requirement on the cloud side, the pre-computation results received on the edge side will be copied to the cloud side for storage. If the load on the edge side is high, the pre-computation will be arranged on the edge side. When the edge side performs a pre-computation query, the request will be sent to the edge side for pre-computation query.

[0040] S3-2-2 If there is no pre-computation query requirement on the endpoint side, but there is a pre-computation query requirement on the edge side, then a node with more available resources will be selected from the endpoint side and the edge side for pre-computation. If there are more available resources on the endpoint side, then the endpoint side will perform pre-computation and send the pre-computation results to the edge side for storage.

[0041] At this time, if there is also a pre-computation query requirement on the cloud side, the pre-computation results received on the edge side will also be copied to the cloud side for storage. If the load on the edge side and the edge side is relatively high, the pre-computation will be arranged on the cloud side. When the edge side performs a pre-computation query, the request will be sent to the cloud side for pre-computation query.

[0042] S3-2-3 If there is no pre-computation query requirement on the endpoint and edge sides, but there is a pre-computation query requirement on the cloud side, then a node with more available resources will be selected from the endpoint, edge, and cloud sides for pre-computation.

[0043] If a node on the endpoint or edge is selected for pre-computation, the pre-computation results also need to be synchronously copied to the cloud for storage.

[0044] S3-2-4 When a node on one side has a pre-computation query requirement, it will usually perform the pre-computation on its own node or by the upstream node and synchronize the pre-computation result to the node on its own side. However, the system load of the node on its own side and the upstream node is high and it is not suitable to perform pre-computation. Therefore, it will be considered to perform the pre-computation by the downstream node. When the node on its own side performs a pre-computation query, it will send the request to the downstream node for pre-computation query.

[0045] Without prior calculation, the time consumed during query analysis at this node is defined as follows:

[0046]

[0047] Among them, UT local The average time consumed to perform calculations on a set of data within an interval at this node; DT local This represents the average time taken to read a set of data within an interval from disk into memory at this node.

[0048] The time consumed during pre-computation query analysis at downstream nodes is defined as follows:

[0049]

[0050] Among them, DT remote NS is the average time consumed by downstream nodes to read a set of pre-computed interval results from disk into memory; NS is the time consumed to send the query request to downstream nodes; Size is the average time consumed by downstream nodes to read the pre-computed interval results from disk into memory. result The size of each pre-calculated result; NR is the transmission rate at which the pre-calculated results are sent from downstream nodes to this node;

[0051] If ET local >ET remote This means that querying from a remote node for pre-computation performs better than querying raw data locally. Without affecting the overall performance of downstream nodes, downstream nodes will be selected for pre-computation, thereby improving query and analysis performance.

[0052] Furthermore, step S4 includes:

[0053] S4-1, Adaptive time-series data compression;

[0054] Further includes:

[0055] S4-1-1 When time-series data is written to the database, it is first cached in memory. When the cache is full, it is periodically written to disk. Before writing to disk, the row storage data of ZObjectTable is converted into column storage format.

[0056] S4-1-2. If the time-series data changes slowly and the amount of data cached before being written to disk is small, the data will not be compressed and the uncompressed data will be written to disk directly.

[0057] S4-1-3. Perform secondary data compression on historical data files;

[0058] S4-1-4. Based on data characteristics and application characteristics, perform automatic sampling before archiving time-series data without affecting application query and analysis.

[0059] When the load is high or the disk space is limited, sampling is performed before data archiving. Each ZObjectTable has a minimum archiving sampling period to ensure that the sampled historical data can meet the query requirements of IoT applications.

[0060] When the load is high, the archive sampling frequency of the ZObjectTable, which is updated frequently, is reduced, thereby reducing disk space usage and CPU time and disk read time during query analysis.

[0061] S4-2. Automatically select the optimal compression node and compression time;

[0062] Further includes:

[0063] S4-2-1. Select the node with the most available resources from the end-side, edge-side, and cloud-side nodes, compress the time-series data, and send the compressed data to the downstream nodes for storage.

[0064] S4-2-2 When performing secondary compression on historical data files, automatically select the time period with the lowest load for compression.

[0065] Furthermore, step S5 includes:

[0066] S5-1: Automatically select the optimal node for data query;

[0067] Further includes:

[0068] S5-1-1. If data exists locally, query data from the local machine first.

[0069] S5-1-2. If there is data locally, and if the local node is busy, select one or more of the best nodes from the nodes with data to query the data.

[0070] S5-1-3. If there is no data locally, select one or more of the best nodes from the nodes with data to perform data query. First, the selected node must be able to meet the query requirements; second, the selected node has the fastest data query speed.

[0071] S5-2, Adaptive Distributed Time Series Data Query;

[0072] Further includes:

[0073] S5-2-1. When querying historical data over a long period of time, and when the data on the device, edge, and cloud sides are from different time periods, the request is split into multiple requests based on the load, each query is performed separately, and then the results are aggregated at the query request node and returned.

[0074] S5-2-1 When a single query request involves multiple ZObjectTables, the query request is sent to nodes on different sides to perform time-series data queries respectively;

[0075] S5-3, Automatically adjust data distribution;

[0076] Further includes:

[0077] S5-3-1. If, within a certain time frame, data needs to be queried from other nodes frequently because there is no ZObjectTable time-series data locally, the data distribution of ZObjectTable will be automatically adjusted so that the required historical data is stored locally, allowing subsequent queries to be performed locally and improving query speed.

[0078] S5-3-2. If there is no local query for historical data in ZObjectTable for a long period of time, the data distribution of ZObjectTable will be automatically adjusted to shorten the time span of local historical data storage.

[0079] An adaptive cloud-edge-device collaboration device includes: at least one memory and at least one processor;

[0080] The at least one memory is used to store a machine-readable program;

[0081] The at least one processor is used to call the machine-readable program to execute the adaptive cloud-edge-device collaboration method.

[0082] Compared with existing technologies, the adaptive cloud-edge-device collaboration method and apparatus of the present invention have the following outstanding advantages:

[0083] This invention, based on the characteristics of time-series data, the characteristics of IoT applications, and the available system resources at the cloud, edge, and device levels, achieves adaptive time-series data distribution, adaptive time-series data pre-computation, adaptive time-series data compression, and adaptive time-series data query and analysis through cloud-edge-device collaboration. This optimizes system resource utilization, data distribution, data computation, and query analysis, thereby improving time-series data processing performance. Through automatic tuning, it optimizes time-series data processing and reduces system maintenance and tuning costs.

[0084] Among them, the adaptive time-series data distribution automatically optimizes the distribution of time-series data in the cloud, edge, and terminal, so that time-series data query and analysis requests are performed locally as much as possible, thereby speeding up time-series data query and analysis.

[0085] Adaptive time-series data pre-computation can automatically add time-series data pre-computation based on the characteristics of IoT applications. When querying and analyzing data, the data will be directly obtained from the pre-computation results, which can greatly speed up the time-series data query and analysis speed. At the same time, it can select the optimal node for pre-computation based on the available system resources of cloud, edge and terminal, make reasonable use of system resources, and improve the performance of time-series data processing.

[0086] Adaptive time-series data compression can automatically select the optimal compression algorithm based on the characteristics of time-series data and IoT applications, thereby improving the compression ratio and storing longer-term historical time-series data. At the same time, it can select the optimal node for time-series data compression based on the available system resources in the cloud, edge, and terminal, making reasonable use of resources and improving time-series data processing capabilities.

[0087] Adaptive time-series data query analysis can automatically select the optimal node for data query analysis based on the distribution of time-series data, thereby improving the performance of data query analysis. At the same time, it can also automatically adjust the distribution of time-series data according to the characteristics of time-series data query analysis, so that time-series data query analysis can be completed locally as quickly as possible, thereby improving the performance of time-series data query analysis. Attached Figure Description

[0088] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0089] Appendix Figure 1 This is a schematic diagram of the topology of an adaptive cloud-edge-device collaboration method;

[0090] Appendix Figure 2 This is an overall architecture diagram of an adaptive cloud-edge-device collaboration method. Detailed Implementation

[0091] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0092] The following is a preferred embodiment:

[0093] like Figure 1 As shown, an adaptive cloud-edge-device collaboration method in this embodiment has the following steps:

[0094] S1. Cloud-edge-device refers to the cloud side, edge side, and device side. The cloud side performs cloud service storage and computation, the edge side performs local computation of time-series data sources, and the device side is the acquisition end that is directly connected to the device.

[0095] The specific steps are as follows:

[0096] The cloud side provides cloud service storage and computing, the edge side supports local computing near the time-series data source, and the endpoint is the acquisition terminal directly connected to the device. For cost control reasons, the resources of cloud-side, edge-side, and endpoint nodes differ significantly, including CPU, memory, disk, and network resources. Cloud-side nodes, abbreviated as C nodes, have the most abundant resources, strong computing capabilities, and large amounts of memory and disk space. Edge-side nodes, abbreviated as E nodes, have a certain amount of resources, but limited resources; they have some computing capabilities, but limited memory and disk space. Endpoint nodes, abbreviated as T nodes, have very limited resources; they have some computing capabilities, but little memory and no disk space for storing long-term historical time-series data; they are mainly responsible for data acquisition, providing a cache of recent time-series data in memory. In specific deployments, endpoint nodes can be omitted depending on the actual situation, and the edge side will directly connect to the device for time-series data acquisition.

[0097] The cloud-side (C-side) provides cloud storage management for IoT time-series data from all edge sides in a cloud service manner, supporting global massive time-series data query and analysis services across multiple edge sides.

[0098] The edge side (E side) provides storage management close to the time-series data source and has local time-series data query and analysis capabilities.

[0099] The terminal side (T side) connects directly to smart devices, providing sensor data acquisition, caching, and data transmission services, as well as query and analysis services for recently cached data. However, it does not provide disk storage management for historical data.

[0100] After the T-side collects sensor data from the device, it performs a short-term cache in memory and synchronizes the time-series data to the E-side. After receiving the data, the E-side stores the time-series data for a certain period of time and periodically synchronizes the time-series data to the C-side. The C-side stores the time-series data as historical data for a long period of time.

[0101] For the same time-series data, the data stored on the T side, E side, and C side may have some overlap or redundancy, but data for any time period will inevitably be stored on one of the sides, unless the time period is a long time ago, in which case the IoT application will no longer need to access that data.

[0102] One E-side node can correspond to multiple T-side nodes; the E-side may also not have any T-side nodes, in which case the E-side will directly collect sensor data from the device.

[0103] S2. Adaptive time-series data distribution, automatically adjusting the time span of time-series data stored on the cloud side, edge side, and terminal side respectively;

[0104] The specific steps are as follows:

[0105] The adaptive cloud-edge-device collaborative system supports adaptive time-series data distribution. It automatically adjusts the time span of time-series data stored on the cloud, edge, and device sides based on available system resources, including memory space and disk space. At the same time, it dynamically adjusts the distribution of time-series data according to the data access needs of IoT applications on different sides, so that time-series data query and analysis can be completed locally without sending requests to nodes on other sides, thereby accelerating time-series data query and analysis.

[0106] (1) The smallest unit of data distribution in the time series engine is called a ZObjectTable. Each ZObjectTable has a timestamp and multiple sensor data. During data acquisition, all sensor data in the ZObjectTable are collected simultaneously, and a current timestamp is added. All sensor data in the ZObjectTable are then written into the time series engine at once. The time span for storing different ZObjectTables on the cloud side, edge side, and endpoint side can vary, depending on the available system resources and the data query and analysis requests for each ZObjectTable.

[0107] (2) The time span in which each ZObjectTable is stored in memory on the T side is determined as follows:

[0108] If there are no recent historical data query and analysis requests for the ZObjectTable on the T side within a certain period, the recent historical data will not be cached on the T side, and the time-series data collected on the T side will be directly synchronized to the E side for storage. During implementation, short-term caching can be considered based on network transmission performance; the actual memory space used is not within this consideration. "Recent period" refers to system configuration parameters, such as one week.

[0109] If, within a recent period, there are query and analysis requests from side T for recent historical data of this ZObjectTable, then its time span is determined as follows:

[0110]

[0111] Among them, O i Represents a ZObjectTable object; Freq(O i ) represents O i The sampling frequency; N i Indicates the recent period of time regarding O i The number of queries; Q j , j∈[1, N i ] indicates the recent period of time regarding O i The query request; Qint(Q j ) represents Q j The time span from the earliest time to the current time; Size(O i ) is each O i Data size; M i O i Maximum available memory space.

[0112] When a large number of historical data queries from ZObjectTables occur on the T side, the memory requirements of these ZObjectTables will surge, potentially leading to insufficient total memory on the T side; the total memory requirement on the T side is M. re for:

[0113]

[0114] Where C is the total number of ZObjectTables on the T side.

[0115] Assume that the total memory available for caching time-series data on side T is M. av So when M re >M av At this time, it is necessary to adjust the time span of ZObjectTable to reduce the overall memory requirements.

[0116] The method for readjusting memory usage is as follows:

[0117] Sort all ZObjectTables in descending order of the time when the last historical data on side T was accessed. The resulting reordered set is: Among them l i For O i After the indexes are rearranged, assuming k is the dividing line, the following conditions are met:

[0118]

[0119] So, after readjusting memory usage, O i The time span has been updated to:

[0120]

[0121] (3) The time span for each ZObjectTable to be stored on the E-side disk is determined as follows:

[0122] If neither side E nor side T requests query or analysis of historical data for the ZObjectTable within a certain period, the historical data will not be stored on the disk of side E. Data received by side E from side T will be directly synchronized to side C for storage. During implementation, short-term disk data temporary storage can be considered based on network transmission performance; the actual disk space used is not within this scope.

[0123] If there are recent query and analysis requests for historical data of this ZObjectTable from either side E or T, the time span shall be determined as follows:

[0124]

[0125] Among them, D i O i Maximum available disk space.

[0126] When a large number of historical data queries from ZObjectTables occur on the E or T side, the disk space requirements of these ZObjectTables will surge, potentially leading to insufficient total disk space on the E side; the total disk space requirement on the E side is D. re for:

[0127]

[0128] Assume that the total disk space available for caching time-series data on side E is D. av So when D re >D av At this time, it is necessary to adjust the time span of ZObjectTable to reduce the overall disk space requirement.

[0129] The method for readjusting disk space usage is as follows:

[0130] Sort all ZObjectTables in descending order of the last time the historical data on the E side was accessed. The resulting reordered set is: Among them l i For O i After the indexes are rearranged, assuming k is the dividing line, the following conditions are met:

[0131]

[0132] So, after readjusting disk space usage, O i The time span has been updated to:

[0133]

[0134] (4) The maximum latency for storing each ZObjectTable on the C-side disk is determined as follows:

[0135] The system provides n fixed data update frequencies Freq k k∈[1,n], satisfying Its corresponding maximum delay is

[0136] If the ZObjectTable has no historical data query and analysis requests in the near future, its update frequency is set to Freq(0). i )=Freq n Its maximum storage latency on the C side is Delay(O) i ) = Delay n .

[0137] If this ZObjectTable has historical data query and analysis requests in the near future, assuming the minimum delay for querying historical data is MinDelay i This means that all queries will not query MinDelay. i If the data is historical, then its update frequency is set to:

[0138]

[0139] Then its maximum latency for storage on the C side is if Then set its update frequency to Freq(O) i =Freq1, and its maximum storage delay on the C side is Delay(O) i =Delay1.

[0140] The lifespan of each ZObjectTable on the C-side disk is determined by the user based on the needs of IoT applications and disk resources. Historical data that exceeds its lifespan will be automatically deleted periodically.

[0141] S3. Adaptive time series data pre-computation: Based on the available resources in the cloud, edge, and terminal, the optimal computing node is selected for time series data pre-computation.

[0142] The specific steps are as follows:

[0143] The adaptive cloud-edge-device collaborative system supports adaptive time-series data pre-computation. Based on the time-series data query and analysis needs of IoT applications, it automatically identifies time-series data that can be optimized for pre-computation and automatically creates pre-computation data. Pre-computation is performed in advance when time-series data is entered into the database, and the pre-computation results are saved. In subsequent time-series data query and analysis processes, the pre-computation results can be directly queried, or secondary calculations can be performed based on the pre-computation results, thereby improving the performance of time-series data query and analysis. At the same time, based on the available resources of the cloud, edge, and device, the optimal computing node is rationally selected for time-series data pre-computation, improving query and analysis performance while optimizing system resource utilization.

[0144] (1) Automatically add or delete pre-calculated data.

[0145] When the same query calculations occur repeatedly over a period of time, time-series data pre-calculation is automatically created. This calculation is performed when the time-series data is imported into the database, and the results are saved. During the time-series data query and analysis process, the pre-calculated results are automatically matched. If a match is found, the pre-calculated results are directly read, and subsequent query calculations are performed according to the query and analysis requirements, ultimately returning the final results.

[0146] If pre-computed data is not used for a long period of time, it will be automatically deleted to reduce the additional load brought by pre-computed data.

[0147] Each pre-calculated load efficiency value is defined as: increased system load / decreased system load, reflecting whether the pre-calculation increased or decreased the overall system load. If the pre-calculated load efficiency value is less than 1, it means that the pre-calculation decreased the overall system load, and therefore, the pre-calculation will continue to be used even when the system is overloaded. If the pre-calculated load efficiency value is greater than 1, it means that the pre-calculation increased the overall system load, and when the system is overloaded, the pre-calculation with the lower load efficiency value should be deleted. Most calculations in time series data query analysis are aggregation calculations, and they are all grouped and aggregated according to certain time intervals.

[0148] The specific definition of the pre-calculated CPU load efficiency value of P is as follows:

[0149]

[0150] Where Period is the time period for evaluating the pre-computed load efficiency value; Interval is the time interval for aggregation and grouping; m is the number of query requests involving pre-computed P within the Period, and the corresponding query request is Q. i , i∈[1,m]; QSpan(Q i ) represents the time span of each query; U c This refers to the CPU resources consumed in a single pre-calculation.

[0151] The disk efficiency value for pre-calculated P is defined as follows:

[0152]

[0153] Among them, D c The disk read / write resources consumed in reading data from the disk without prior calculation; D write When pre-computation is used, the read / write resources consumed in writing the pre-computed data to disk; D read This refers to the disk read / write resources consumed when reading the pre-calculation results from the disk when pre-calculation is used.

[0154] When the system experiences CPU overload, the pre-calculated values ​​with the lowest CPU load efficiency are automatically deleted until the system is no longer experiencing CPU overload.

[0155] When the system experiences disk overload, the pre-calculated disks with the lowest load efficiency values ​​are automatically deleted until the system is no longer experiencing disk overload.

[0156] (1) Automatically select the optimal pre-computation node

[0157] If there is a pre-computation query requirement on side T, the pre-computation is performed on side T, and the pre-computation results are cached. If there is also a pre-computation query requirement on side E, the pre-computation results on side T will be copied to side E for storage; similarly, if there is also a pre-computation query requirement on side C, the pre-computation results received on side E will also be copied to side C for storage. If the system load on side T is high, the pre-computation can be scheduled to be performed on side E. When side T performs a pre-computation query, the request is sent to side E for the pre-computation query.

[0158] If there is no pre-computation query requirement on side T, but there is on side E, then a node with more available resources will be selected from sides T and E for pre-computation. If side T has more available resources, then side T will perform pre-computation and send the pre-computation results to side E for storage. If side C also has a pre-computation query requirement at this time, the pre-computation results from side E will also be copied to side C for storage. If the system load on sides T and E is high, pre-computation can be scheduled to be performed on side C; when side E performs a pre-computation query, the request will be sent to side C for pre-computation query processing.

[0159] If there is no pre-computation query requirement on sides T and E, but there is a pre-computation query requirement on side C, then a node with more available resources will be selected from sides T, E, and C for pre-computation. If a node on side T or E is selected for pre-computation, the pre-computation results will also need to be synchronously copied to side C for storage.

[0160] When a node on one side has a pre-computation query requirement, it usually performs the pre-computation on its own node or by an upstream node, and then synchronizes the pre-computation results to the local node. However, the system load of the local and upstream nodes is high, making pre-computation unsuitable. It is more appropriate to have the downstream node perform the pre-computation. When the local node performs a pre-computation query, it sends the request to the downstream node for pre-computation query processing.

[0161] Without prior calculation, the time consumed during query analysis at this node is defined as follows:

[0162]

[0163] Among them, UT local The average time consumed to perform calculations on a set of data within an interval at this node; DT local This represents the average time taken to read a set of data within an interval from disk into memory at this node.

[0164] The time consumed during pre-computation query analysis at downstream nodes is defined as follows:

[0165]

[0166] Among them, DT remote NS is the average time consumed by downstream nodes to read a set of pre-computed interval results from disk into memory; NS is the time consumed to send the query request to downstream nodes; Size is the average time consumed by downstream nodes to read the pre-computed interval results from disk into memory. result NR represents the size of each pre-calculated result; NR is the transmission rate at which the pre-calculated results are sent from downstream nodes to this node.

[0167] If ET local>ET remote This means that querying from a remote node for pre-computation performs better than querying raw data locally. Without affecting the overall performance of downstream nodes, downstream nodes will be selected for pre-computation, thereby improving query and analysis performance.

[0168] S4. Adaptive time-series data compression: Based on the available resources in the cloud, edge, and terminal, the optimal node is selected for time-series data compression.

[0169] The specific steps are as follows:

[0170] The adaptive cloud-edge-device collaborative system supports adaptive time-series data compression. Based on the characteristics of time-series data types and change frequencies, it selects appropriate compression algorithms to compress time-series data. At the same time, based on the available resources of the cloud, edge, and device, it rationally selects the optimal node for time-series data compression, thereby reducing disk data storage space and optimizing system resource utilization.

[0171] (1) Adaptive temporal data compression algorithm;

[0172] When time-series data is written to the database, it is first cached in memory. When the cache is full, or periodically, a write operation is performed to disk. Before writing to disk, the row-based data of ZObjectTable is converted into column-based data format, and then different compression algorithms are selected to compress the time-series data according to the data type:

[0173] Integer: The compression algorithm is delta(delta-of-delta) + zig-zag + simple8b.

[0174] Boolean type: uses bit-packing compression algorithm;

[0175] Timestamp type: Delta-Of-Delta+Zigzag compression algorithm;

[0176] Floating-point type: Employs the XOR compression algorithm suitable for compressing floating-point data;

[0177] String type: Uses the LZ4 compression algorithm for better decompression efficiency;

[0178] Blob data type: Employs the LZ4 compression algorithm for better decompression efficiency.

[0179] If the time-series data changes slowly and the amount of data cached before being written to disk is small, the data will not be compressed and the uncompressed data will be written to disk directly.

[0180] The historical data files are compressed a second time to further reduce the amount of data on the disk.

[0181] Based on data and application characteristics, automatic sampling is performed before archiving time-series data, without affecting application query and analysis. This is a form of lossy data compression that can significantly reduce disk storage space. Normally, only the data collected is stored. When system load is high or disk space is limited, sampling can be performed before archiving. For example, if the collection frequency is one data point per second, after archiving sampling, only one data point is stored every five seconds. Each ZObjectTable has a minimum archiving sampling period to ensure that its sampled historical data meets the query requirements of IoT applications. When system load is high, the archiving sampling frequency of some ZObjectTables with higher update frequencies can be reduced, thereby reducing disk space usage and also reducing CPU time and disk read time during query and analysis.

[0182] (2) Automatically select the optimal compression node and compression time;

[0183] Select the node with the most available system resources from the T-side, E-side, and C-side nodes, compress the time-series data, and send the compressed data to the downstream nodes for storage.

[0184] When performing secondary compression on historical data files, the system automatically selects the time period with the lowest system load for compression, for example, performing secondary compression on historical data files at night when the load is low.

[0185] S5. Adaptive time series data query and analysis: Based on the distribution of time series data, it automatically selects the optimal computing node for time series data query and analysis.

[0186] The specific steps are as follows:

[0187] The adaptive cloud-edge-device collaborative system supports adaptive time-series data query and analysis. Based on the distribution of time-series data, it automatically selects the optimal computing node for time-series data query and analysis. At the same time, when it is necessary to frequently remotely query and analyze time-series data on other nodes because the data is not stored locally, it will automatically adjust the data distribution as long as available resources allow, to ensure that the time-series data required for query and analysis is available locally, thereby accelerating time-series data query and analysis.

[0188] (1) Automatically select the optimal node for data query;

[0189] If local data is available, it will be retrieved first. If local data is available but the local node is busy, one or more optimal nodes with data can be selected for querying. If no local data is available, one or more optimal nodes with data will be selected for querying. First, the selected node must be able to meet the query requirements; second, the selected node must have the fastest data query speed.

[0190] (2) Adaptive distributed time-series data query;

[0191] When querying historical data over a long period of time, and when there are data from different time periods on the T, E, and C sides, the request is split into multiple requests based on the system load. Data is queried separately for each request, and then the results are aggregated at the query request node and returned.

[0192] When a single query request involves multiple ZObjectTables, the query request can be sent to nodes on different sides to perform time-series data queries.

[0193] (3) Automatically adjust data distribution;

[0194] If, within a certain time frame, data is frequently needed to be queried from other nodes because a certain ZObjectTable lacks local time-series data, the data distribution of that ZObjectTable will be automatically adjusted to store the required historical data locally, allowing subsequent queries to be performed locally and improving query speed.

[0195] If there is no local query for historical data of a certain ZObjectTable for a long period of time, the data distribution of the ZObjectTable can be automatically adjusted to shorten the time span of local historical data storage.

[0196] like Figure 2 As shown, an adaptive cloud-edge-device collaborative system includes a cloud-edge-device collaborative module, an information acquisition and synchronization module, an automatic learning module, a data query and analysis module, a local time series engine, a time series data acquisition module, and a time series data synchronization module.

[0197] The information acquisition and synchronization module is responsible for collecting and identifying the system's operating environment, including time-series data characteristics, IoT application query and analysis characteristics, and the usage of various system resources. This module synchronizes the collected operating environment data across the cloud, edge, and device via the network, ensuring that each node has the latest operating environment data from all three nodes (cloud, edge, and device).

[0198] The automatic learning module periodically analyzes and processes the collected operating environment data, extracts data features, extracts pre-calculations that can be optimized, and calculates the load information of each node in the cloud-edge-device interface, generating the operating environment information required by the cloud-edge-device collaboration module.

[0199] The cloud-edge-device collaboration module uses the runtime environment information generated by automatic learning to make decisions, and sends the decisions to the local time series engines on the cloud, edge, and device sides through cloud-edge-device collaboration, including adjusting the distribution of time series data, adjusting pre-computation, and adjusting compression strategies.

[0200] The local time series engine is responsible for the calculation, storage and query of local time series data, providing data ingestion services, data query and analysis services, and decision control services for the cloud-edge-device collaboration module. It can adjust the distribution of time series data, adjust pre-computation, and adjust compression strategies according to the requirements of the cloud-edge-device collaboration module.

[0201] The data query and analysis module provides cloud-edge-device collaborative data query and analysis services. It retrieves the required time-series data objects and time spans based on the data query and analysis request. The cloud-edge-device collaboration then determines which nodes / nodes on the cloud / edge will perform the historical data query for which time period, based on data distribution, pre-computation information, and available system resources. After obtaining the decision information from the cloud-edge-device system modules, the data query and analysis module directly queries the corresponding time period from both the local time-series engine and other time-series engines, merges the retrieved data, and returns the final result.

[0202] The time-series data acquisition module is deployed on the T side (or on the E side if there is no T side), directly connected to the terminal device, and collects data from the terminal device and writes it into the local time-series engine.

[0203] The timing data synchronization module is responsible for synchronizing timing data from the T side to the E side, and from the E side to the C side.

[0204] The data collection and synchronization process is as follows:

[0205] Most time-series data is collected from smart / terminal devices in the Internet of Things (IoT) environment. However, some time-series data is virtual and does not correspond to any device sensor data. These virtual time-series data are time-series results obtained by calculating other time-series data and are stored and managed as virtual ZObjectTables in the same way as regular time-series data.

[0206] Typically, time-series data is collected by nodes on side T and written to the local time-series engine. The time-series engine on side T then sends the data to side E in real time through the time-series data synchronization module. After receiving the data, the time-series data synchronization module on side E writes it to the local time-series engine on side E. The time-series engine on side E synchronizes the time-series data to side C according to the synchronization frequency of each ZObjectTable. After receiving the data, the time-series data synchronization module on side C writes it to the local time-series engine on side C.

[0207] The nodes in the cloud-edge-device collaboration will periodically clear the historical data of each node according to the data distribution strategy of cloud-edge-device collaboration. However, each node will not delete the data for the time period required to be stored in the data distribution strategy, thereby ensuring the correctness and integrity of the data queried through cloud-edge-device collaboration.

[0208] The adaptive cloud-edge-device collaborative processing flow is as follows:

[0209] Cloud, edge, and terminal nodes regularly collect time-series data characteristics, IoT application query and analysis characteristics, and system resource usage, and synchronize them with other nodes so that all cloud, edge, and terminal nodes have current system operating environment data.

[0210] Some runtime environment data can be directly used by the cloud-edge-device collaboration module for decision analysis, but some directly collected runtime environment data cannot be used directly. Therefore, the automatic learning module will periodically calculate and analyze the runtime environment data and update the runtime environment information that the cloud-edge-device collaboration module can directly use, including: statistical information of each ZObjectTable in application data query analysis; calculations that frequently occur in application data query analysis for each ZObjectTable; and statistical information on the resource usage of cloud-edge-device nodes, including CPU, memory, disk, and network.

[0211] The cloud-edge-device collaboration module periodically and automatically adjusts the current adaptive time-series data distribution, adaptive time-series data pre-computation, and adaptive time-series data compression based on the latest operating environment information. When a decision adjustment occurs, the cloud-edge-device collaboration module notifies the local time-series engine to make corresponding adjustments.

[0212] When an IoT application submits a time-series data query and analysis request, the cloud-edge-device collaboration module will also make a decision based on the current data distribution and the latest operating environment information, and formulate a distributed query plan for this time-series data query and analysis.

[0213] The adaptive load reduction process after system overload is as follows:

[0214] Upon detecting system overload, the first step is to check if data distribution adjustments can be made to some ZObjectTables to optimize queries. If so, data distribution adjustments will be made. Secondly, pre-computations with the lowest efficiency values ​​(less than 1) will be stopped. If the system remains overloaded, after multiple adaptive load reductions, all pre-computations with efficiency values ​​less than 1 will be stopped. Finally, if the system is still overloaded after stopping all pre-computations, the compression algorithm will be adjusted to reduce data compression. During the compression algorithm adjustment process, file-based secondary compression will be stopped first, followed by stopping data compression for some ZObjectTables. If all time-series data is no longer compressed, the archive sampling rate will be adjusted to reduce archiving and query load.

[0215] Once the system overload is relieved, the system will first automatically resume the compression algorithm, and then resume the stopped pre-computation.

[0216] This embodiment of an adaptive cloud-edge-device collaboration device includes: at least one memory and at least one processor;

[0217] The at least one memory is used to store a machine-readable program;

[0218] The at least one processor is used to call the machine-readable program to execute the adaptive cloud-edge-device collaboration method.

[0219] The above-described specific embodiments are merely specific examples of the present invention. The patent protection scope of the present invention includes, but is not limited to, the above-described specific embodiments. Any appropriate changes or substitutions made by those skilled in the art to an adaptive cloud-edge-device collaboration method and apparatus that conform to the present invention should fall within the patent protection scope of the present invention.

[0220] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An adaptive cloud-edge-device collaboration method, characterized in that, It has the following steps: S1. Cloud-edge-device refers to the cloud side, edge side, and device side. The cloud side performs cloud service storage and computation, the edge side performs local computation of time-series data sources, and the device side is the acquisition end that is directly connected to the device. S2. Adaptive time-series data distribution, automatically adjusting the time span of time-series data stored on the cloud side, edge side, and terminal side respectively; Further includes: S2-1. The smallest unit of data distribution in the time series engine is called ZObjectTable. Each ZObjectTable has a timestamp and multiple sensor data. S2-2, Determine the time span during which each ZObjectTable is stored in the client-side memory; S2-3. Determine the time span for storing each ZObjectTable on the edge-side disk; S2-4. Determine the maximum latency for storing each ZObjectTable on the cloud-side disk; S2-5. The lifecycle of each ZObjectTable in the cloud disk is determined by the user based on the needs of IoT applications and disk resources. Historical data that exceeds the lifecycle will be automatically deleted periodically. S3. Adaptive time series data pre-computation: Based on the available resources in the cloud, edge, and terminal, the optimal computing node is selected for time series data pre-computation. Further includes: S3-1, Automatically add / delete pre-calculated data; include: S3-1-1 When the same query calculation occurs continuously within a period of time, time series data pre-calculation is automatically created. The calculation is performed when the time series data is entered into the database, and the results are saved. During the time series data query and analysis process, the pre-calculation results will be automatically matched. If the match is successful, the pre-calculation results will be read directly, and then subsequent query calculations will be performed according to the query and analysis requirements, and the results will be returned in the end. S3-1-2. If the pre-calculated data has not been used for a long period of time, it will be automatically deleted. S3-1-3. If the pre-calculated load efficiency value is less than 1, it means that the pre-calculation has reduced the overall system load. Therefore, even under overload conditions, the pre-calculation will continue to be used. If the pre-calculated load efficiency value is greater than 1, it means that the pre-calculation has increased the load overall. When overloaded, the pre-calculation with a relatively low load efficiency value needs to be deleted. The specific definition of the pre-calculated CPU load efficiency value of P is as follows: ； in, The time period for evaluating pre-calculated load efficiency values; m is the time interval for aggregation grouping; m is the time interval in the aggregation grouping. The number of query requests involving pre-calculation P within the time period, and the corresponding number of query requests. ; The time span for each query; The CPU resources consumed in a single pre-calculation; The disk efficiency value for pre-calculated P is defined as follows: ； in, Disk read / write resources consumed to read data from disk without prior calculation; When using pre-computation, the read / write resources consumed in writing the pre-computed data to disk are considered. The disk read / write resources consumed when reading the pre-calculation results from the disk when pre-calculation is used; S3-1-4. When CPU overload occurs, automatically delete the pre-calculated CPU load efficiency value with the lowest value until the CPU is no longer overloaded. S3-1-5. When disk overload occurs, automatically delete the pre-calculated disks with the lowest load efficiency values ​​until the disk is no longer overloaded. S3-2. Automatically select the optimal pre-calculated node; include: S3-2-1. If there is a pre-calculation query requirement on the endpoint, the pre-calculation is performed on the endpoint and the pre-calculation result is cached. At this time, if there is also a pre-calculation query requirement on the edge, the pre-calculation result on the endpoint will be copied to the edge for storage. If there is also a pre-computation query requirement on the cloud side, the pre-computation results received on the edge side will be copied to the cloud side for storage. If the load on the edge side is higher than the threshold, the pre-computation will be scheduled to be performed on the edge side. When the edge side performs a pre-computation query, the request will be sent to the edge side for pre-computation query. S3-2-2 If there is no pre-computation query requirement on the endpoint side, but there is a pre-computation query requirement on the edge side, then a node with relatively more available resources will be selected from the endpoint side and the edge side for pre-computation. If there are relatively more available resources on the endpoint side, then the endpoint side will perform pre-computation and send the pre-computation results to the edge side for storage. At this time, if there is also a pre-computation query requirement on the cloud side, the pre-computation result on the edge side will also be copied to the cloud side for storage. If the load on the edge side and the edge side is higher than the threshold, the pre-computation will be arranged on the cloud side. When the edge side performs a pre-computation query, the request will be sent to the cloud side for pre-computation query. S3-2-3 If there is no pre-computation query requirement on the endpoint and edge sides, but there is a pre-computation query requirement on the cloud side, then a node with more available resources will be selected from the endpoint, edge, and cloud sides for pre-computation. If a node on the endpoint or edge is selected for pre-computation, the pre-computation results also need to be synchronously copied to the cloud for storage. S3-2-4 When a node on one side has a pre-computation query requirement, it will usually perform the pre-computation on its own node or by the upstream node and synchronize the pre-computation result to the node on its own side. However, if the system load of the node on its own side and the upstream node is higher than the threshold, it is not suitable to perform pre-computation. In this case, the downstream node will be considered to perform the pre-computation. When the node on its own side performs a pre-computation query, the request will be sent to the downstream node for pre-computation query. Without prior calculation, the time consumed during query analysis at this node is defined as follows: ； in, To perform a set at this node The average time consumed in performing calculations on data within a given time frame; To set a group at this node The average time taken to read data from disk into memory within a given time frame; The time consumed during pre-computation query analysis at downstream nodes is defined as follows: ； in, For downstream nodes, transfer a set of data from the disk The average time consumed to read the pre-calculated results into memory; The time consumed in sending the query request to the downstream node; The size of each pre-calculated result; The transmission rate at which the pre-calculated results are sent from downstream nodes to this node; if This means that querying from a remote location for pre-computation performs better than querying raw data locally. Without affecting the overall performance of downstream nodes, downstream nodes will be selected for pre-computation, thereby improving query and analysis performance. S4. Adaptive time-series data compression: Based on the available resources in the cloud, edge, and terminal, the optimal node is selected for time-series data compression. S5. Adaptive time series data query and analysis: Based on the distribution of time series data, it automatically selects the optimal computing node for time series data query and analysis. include: S5-1: Automatically select the optimal node for data query; Further includes: S5-1-1. If data exists locally, query data from the local machine first. S5-1-2. If there is data locally and the local node is busy, select one or more optimal nodes from the nodes with data to query the data. S5-1-3. If there is no data locally, select one or more of the best nodes from the nodes with data to perform data query. First, the selected node must be able to meet the query requirements; second, the selected node has the fastest data query speed. S5-2, Adaptive Distributed Time Series Data Query; Further includes: S5-2-1. When querying historical data over a long period of time, and when the data on the device, edge, and cloud sides are from different time periods, the request is split into multiple requests based on the load, each query is performed separately, and then the results are aggregated at the query request node and returned. S5-2-2 When a single query request involves multiple ZObjectTables, the query request is sent to nodes on different sides to perform time-series data queries respectively; S5-3, Automatically adjust data distribution; Further includes: S5-3-1. If, within a certain time frame, data needs to be queried from other nodes frequently because there is no ZObjectTable time-series data locally, the data distribution of ZObjectTable will be automatically adjusted so that the required historical data is stored locally, allowing subsequent queries to be performed locally. S5-3-2. If there is no local query for historical data in ZObjectTable for a long period of time, the data distribution of ZObjectTable will be automatically adjusted to shorten the time span of local historical data storage.

2. The adaptive cloud-edge-device collaboration method according to claim 1, characterized in that, In step S1, after the edge side collects the sensor data from the device, it performs a short-term cache in memory and synchronizes the time-series data to the edge side. After receiving data, the edge device stores the time-series data for a certain period of time and periodically synchronizes the time-series data to the cloud; the cloud device stores the time-series data as historical data for a long period of time.

3. The adaptive cloud-edge-device collaboration method according to claim 2, characterized in that, Step S4 includes: S4-1, Adaptive time-series data compression; Further includes: S4-1-1 When time-series data is written to the database, it is first cached in memory. When the cache is full, it is periodically written to disk. Before writing to disk, the row storage data of ZObjectTable is converted into column storage format. S4-1-2. If the time-series data changes slowly and the amount of data cached before being written to disk is less than the set threshold, the data will not be compressed and the uncompressed data will be written to disk directly. S4-1-3. Perform secondary data compression on historical data files; S4-1-4. Based on data characteristics and application characteristics, perform automatic sampling before archiving time-series data without affecting application query and analysis. When the load is higher than the threshold or the disk space is lower than the threshold, sampling is performed before data archiving. Each ZObjectTable has a minimum archiving sampling period to ensure that the historical data after sampling can meet the query requirements of IoT applications. When the load exceeds the threshold, the archive sampling frequency of the ZObjectTable, which has a relatively high update frequency, is reduced, thereby reducing disk space usage and reducing CPU time and disk read time during query analysis. S4-2. Automatically select the optimal compression node and compression time; Further includes: S4-2-1. Select the node with the most available resources from the end-side, edge-side, and cloud-side nodes, compress the time-series data, and send the compressed data to the downstream nodes for storage. S4-2-2 When performing secondary compression on historical data files, automatically select the time period with the lowest load for compression.

4. An adaptive cloud-edge-device collaborative device, characterized in that, include: At least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to invoke the machine-readable program to perform the method according to any one of claims 1 to 3.

Images