Data processing method and system based on cloud edge collaboration

Abstract

The embodiment of the invention provides a data processing method and system based on cloud edge collaboration. According to the method, a collaborative scheduling module is utilized, and processing tasks can be dynamically scheduled according to data processing requirements and network states. In addition, an intelligent cache module is adopted, a cache mechanism is established between the cloud processing module and the edge computing module, and a distributed loading module is utilized to divide and distribute a large data set to realize parallel processing. By means of the mode, dynamic adjustment and optimization of collaborative scheduling, intelligent caching and distributed loading strategies of the data can be achieved, cloud computing and edge computing are combined, a cloud-edge collaborative architecture is formed, and the data processing efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and system based on cloud-edge collaboration. Background Art

[0002] With the rapid development of big data and artificial intelligence technologies, intelligent computing has become a key force in promoting social progress. In the context of intelligent computing, the importance of data processing and analysis has become increasingly prominent, and higher requirements have been placed on data processing efficiency. Edge computing and cloud-edge collaboration, as emerging technologies, are gradually becoming the focus of the industry, and their importance is self-evident.

[0003] At present, traditional data processing systems mainly use cloud processing modules, edge computing modules and communication modules to achieve collaborative processing and efficient transmission of data.

[0004] However, traditional data processing systems are slow when processing massive, high-dimensional, and complex-structured data, resulting in low data processing efficiency. Summary of the invention

[0005] The embodiments of the present invention provide a data processing method and system based on cloud-edge collaboration, which can improve data processing efficiency.

[0006] In a first aspect, an embodiment of the present invention provides a data processing method based on cloud-edge collaboration, the method comprising:

[0007] Using the collaborative scheduling module, attribute analysis and task priority division are performed on at least one task received by the cloud and the edge;

[0008] Determining the computing resource status of the cloud and the edge;

[0009] Based on the task priority and the computing resource status, determining a target scheduling position corresponding to the task, wherein the target scheduling position includes the cloud and the edge;

[0010] dispatching the at least one task to the target dispatching location;

[0011] Use the intelligent cache module to obtain the user's historical access data;

[0012] Based on the historical access data, determining at least one hotspot identification index, wherein the hotspot identification index is used to reflect the cache priority corresponding to each data item;

[0013] Determining a target cache location based on the historical access data and the at least one hot spot identification index, wherein the target cache location includes a high-speed cache area and a low-speed cache area;

[0014] Based on the at least one task, the cloud processing module and the edge computing module are used to process the data at the target cache location respectively.

[0015] Preferably,

[0016] The collaborative scheduling module is used to perform attribute analysis and task priority division on at least one task received by the cloud and the edge, including:

[0017] Determine a first execution script using the collaborative scheduling module, wherein the first execution script is used to determine at least one of the computational complexity, data type, and real-time requirement information of each of the tasks;

[0018] Determine at least one of the computational complexity, the data type, and the real-time requirement information of the at least one task based on the first execution script;

[0019] The task priority is determined based on the computational complexity, the data type, the real-time requirement information, and a first formula, where the first formula is:

[0020] P = α1 × C + β1 × T1 + γ × R;

[0021] Among them, α1 is the weight coefficient of the computational complexity, β1 is the weight coefficient of the data type, γ is the weight coefficient of the real-time requirement, C is the computational complexity, T1 is the data type, R is the real-time requirement, and P is the task priority.

[0022] Preferably,

[0023] The determining, based on the historical access data, at least one hotspot identification index, wherein the hotspot identification index is used to reflect the cache priority corresponding to each data item, includes:

[0024] Based on the historical access data, determine the access frequency and access time interval of the data;

[0025] At least one hotspot identification index is determined based on the access frequency of the data and the access time interval and the following second formula, where the second formula is:

[0026]

[0027] Among them, α2 is the weight coefficient of the access frequency of the data, β2 is the weight coefficient of the access time interval, F is the access frequency of the data, T2 is the access time interval, and H is the hotspot identification index.

[0028] Preferably,

[0029] The determining of the computing resource status of the cloud and the edge includes:

[0030] Determine the occupancy of computing resources on the cloud and the edge, the remaining amount of storage resources, and the real-time status of network bandwidth;

[0031] Preferably,

[0032] Before the intelligent cache module is used to obtain the user's historical access data, the method further includes:

[0033] Using the distributed loading module, determine if there is a large data set;

[0034] When the large data set exists, dividing the large data set into at least one data subset;

[0035] The at least one divided data subset is distributed to the computing nodes corresponding to the cloud and the edge end for data storage.

[0036] In a second aspect, an embodiment of the present invention provides a data processing system based on cloud-edge collaboration, characterized in that the system includes:

[0037] A collaborative scheduling module, configured to perform attribute analysis and task priority division on at least one task received by the cloud and the edge, determine the computing resource status of the cloud and the edge, determine a target scheduling position corresponding to the task based on the task priority and the computing resource status, wherein the target scheduling position includes the cloud and the edge; and schedule the at least one task to the target scheduling position;

[0038] An intelligent cache module, configured to obtain historical access data of a user, determine at least one hot spot identification index based on the historical access data, wherein the hot spot identification index is used to reflect the cache priority corresponding to each data item, and determine a target cache location based on the historical access data and the at least one hot spot identification index, wherein the target cache location includes a high-speed cache area and a low-speed cache area;

[0039] A processing module is used to perform data processing on the data of the target cache location based on the at least one task of the collaborative scheduling module.

[0040] Preferably,

[0041] The collaborative scheduling module is used to execute:

[0042] Determine a first execution script using the collaborative scheduling module, wherein the first execution script is used to determine at least one of the computational complexity, data type, and real-time requirement information of each of the tasks;

[0043] Based on the first execution script, determining the computational complexity, the data type, and the real-time requirement information of the at least one task;

[0044] The task priority is determined based on the computational complexity, the data type, the real-time requirement information, and a first formula, where the first formula is:

[0045] P = α1 × C + β1 × T1 + γ × R;

[0046] Among them, α1 is the weight coefficient of the computational complexity, β1 is the weight coefficient of the data type, γ is the weight coefficient of the real-time requirement, C is the computational complexity, T1 is the data type, R is the real-time requirement, and P is the task priority.

[0047] Preferably,

[0048] The intelligent cache module is used to execute:

[0049] Based on the historical access data, determine the access frequency and access time interval of the data;

[0050] At least one hotspot identification index is determined based on the access frequency of the data and the access time interval and the following second formula, where the second formula is:

[0051]

[0052] Among them, α2 is the weight coefficient of the access frequency of the data, β2 is the weight coefficient of the access time interval, F is the access frequency of the data, T2 is the access time interval, and H is the hotspot identification index.

[0053] Preferably,

[0054] The collaborative scheduling module is used to execute:

[0055] Determine the occupancy of computing resources on the cloud and the edge, the remaining amount of storage resources, and the real-time status of network bandwidth;

[0056] Preferably,

[0057] Further comprising: a distributed loading module;

[0058] The distributed loading module is used to execute:

[0059] Determine if a large dataset exists;

[0060] When the large data set exists, dividing the large data set into at least one data subset;

[0061] The at least one divided data subset is distributed to the computing nodes corresponding to the cloud and the edge end for data storage.

[0062] In a third aspect, an embodiment of the present invention provides a data processing system based on cloud-edge collaboration, including: at least one memory and at least one processor;

[0063] The at least one memory is used to store a machine-readable program;

[0064] The at least one processor is used to call the machine-readable program to execute any method described in the first aspect.

[0065] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes any one of the methods described in the first aspect.

[0066] The embodiment of the present invention provides a data processing method based on cloud-edge collaboration. By using a collaborative scheduling module, processing tasks can be dynamically scheduled according to data processing requirements and network status. In addition, the method also uses an intelligent cache module to establish a cache mechanism between the cloud processing module and the edge computing module, and uses a distributed loading module to divide and distribute large data sets to achieve parallel processing. Through the above methods, dynamic adjustment and optimization of data collaborative scheduling, intelligent caching and distributed loading strategies can be achieved, cloud computing and edge computing are combined to form a cloud-edge collaborative architecture, which can improve the efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0068] Figure 1 It is a flow chart of a data processing method based on cloud-edge collaboration provided by one embodiment of the present invention;

[0069] Figure 2 is a flowchart of another data processing method based on cloud-edge collaboration provided by an embodiment of the present invention;

[0070] Figure 3 is a schematic diagram of a data processing system based on cloud-edge collaboration provided by an embodiment of the present invention;

[0071] Figure 4It is a schematic diagram of another data processing system based on cloud-edge collaboration provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0073] like Figure 1 As shown, an embodiment of the present invention provides a data processing method based on cloud-edge collaboration, which may include the following steps:

[0074] Step 101: using a collaborative scheduling module to perform attribute analysis and task priority division on at least one task received by the cloud and the edge;

[0075] Step 102: Determine the computing resource status of the cloud and edge;

[0076] Step 103: Determine a target scheduling location corresponding to the task based on the task priority and computing resource status, wherein the target scheduling location includes the cloud and the edge;

[0077] Step 104: dispatching at least one task to a target dispatching position;

[0078] Step 105: Obtain the user's historical access data using the intelligent cache module;

[0079] Step 106: determining at least one hotspot identification index based on the historical access data, wherein the hotspot identification index is used to reflect the cache priority corresponding to each data item;

[0080] Step 107: Determine a target cache location based on the historical access data and at least one hot spot identification index, wherein the target cache location includes a high-speed cache area and a low-speed cache area;

[0081] Step 108: Based on at least one task, use the cloud processing module and the edge computing module to process the data at the target cache location respectively.

[0082] In an embodiment of the present invention, a data processing method based on cloud-edge collaboration is provided. By using a collaborative scheduling module, processing tasks can be dynamically scheduled according to data processing requirements and network status. In addition, the method also uses an intelligent cache module to establish a cache mechanism between the cloud processing module and the edge computing module, and uses a distributed loading module to divide and distribute large data sets to achieve parallel processing. Through the above methods, dynamic adjustment and optimization of data collaborative scheduling, intelligent caching, and distributed loading strategies can be achieved, cloud computing and edge computing are combined to form a cloud-edge collaborative architecture, which can improve the efficiency of data processing.

[0083] In order to realize dynamic scheduling of tasks, in one embodiment of the present invention, the collaborative scheduling module in step 101 in the above embodiment performs attribute analysis and task priority division for at least one task received by the cloud and the edge, including:

[0084] Determine a first execution script using the collaborative scheduling module, wherein the first execution script is used to determine at least one of the computational complexity, data type, and real-time requirement information of each of the tasks;

[0085] Determine at least one of the computational complexity, the data type, and the real-time requirement information of the at least one task based on the first execution script;

[0086] The task priority is determined based on the computational complexity, the data type, the real-time requirement information, and a first formula, where the first formula is:

[0087] P = α1 × C + β1 × T1 + γ × R;

[0088] Among them, α1 is the weight coefficient of the computational complexity, β1 is the weight coefficient of the data type, γ is the weight coefficient of the real-time requirement, C is the computational complexity, T1 is the data type, R is the real-time requirement, and P is the task priority.

[0089] Specifically, the computational complexity (C) can be determined according to the size of the data volume of each task. For example, different computational complexity values ​​can be assigned to each task according to the size of the data volume. The data type (T1) can determine the type of the task according to the extension of the task file. For example, .doc corresponds to the task as word text. Different values ​​can be assigned to different data types. The real-time requirement (R) can be predefined by the user. For example, a delay setting can be performed by writing code or script. The delay setting can obtain the processing time of the task and improve the data processing efficiency. In addition, the determination of the three weight coefficients α1, β1, and γ can also be predefined manually. For example, if the user believes that the computational complexity is more important in task processing, α1 can be set to a higher value. Similarly, if the user believes that the data type is more important in task processing, β1 can be set to a higher value.

[0090] In an embodiment of the present invention, in order to achieve collaborative scheduling of cloud and edge tasks, a collaborative scheduling module can be used to reasonably allocate computing tasks to various computing nodes, reducing the waiting time of each task and the waste of computing resources. At the same time, in terms of collaborative scheduling, this solution adopts a scheduling strategy based on task attribute analysis and task priority division. Through the first execution script, the tasks received by the cloud and edge can be analyzed in detail to determine the key information such as the computational complexity, data type, and real-time requirements corresponding to each task. Then, based on this information and the above-mentioned first formula, the priority corresponding to each task can be determined to ensure that high-priority tasks can be given priority, thereby meeting the user's requirements for task processing speed.

[0091] In order to achieve collaborative scheduling of tasks, it is also necessary to monitor the resource status of the cloud and edge in real time, including the occupancy of computing resources, the remaining amount of storage resources, and the real-time status of network bandwidth. Through real-time monitoring of this information, the processing capacity of each node can be evaluated. Then, based on the task priority and resource evaluation results, tasks with large computing volume and high real-time requirements are preferentially scheduled to the cloud for processing. The cloud has powerful computing power and rich resource pools to ensure that these tasks are processed quickly and accurately. For tasks with small computing volume and low real-time requirements, they are scheduled to the edge for processing.

[0092] In order to improve the efficiency of data exchange between the cloud and the edge, in one embodiment of the present invention, the step 106 in the above embodiment of determining at least one hotspot identification index based on the historical access data includes:

[0093] Based on the historical access data, determine the access frequency and access time interval of the data;

[0094] At least one hotspot identification index is determined based on the access frequency of the data and the access time interval and the following second formula, where the second formula is:

[0095]

[0096] Among them, α2 is the weight coefficient of the access frequency of the data, β2 is the weight coefficient of the access time interval, F is the access frequency of the data, T2 is the access time interval, and H is the hotspot identification index.

[0097] In an embodiment of the present invention, in order to improve the efficiency of data exchange between the cloud and the edge, and to realize intelligent caching technology, hot data and unpopular data can be identified, and corresponding caching strategies can be formulated. First, a large amount of historical access data of users is obtained, and machine learning algorithms such as cluster analysis and association rule mining can be used to train and analyze these data to reveal the access rules of the data, thereby determining the access frequency (F) and access time interval (T) of the data. In addition, the access frequency and access time interval of the data can also be determined according to the user's behavior pattern. At the same time, for hot data, a cache priority can be assigned to each data item according to the hot spot identification index, and high-priority data is stored in the cache, while low-priority data is stored in the slow cache; for unpopular data, an on-demand loading strategy can be adopted. When these data are accessed, we load the data from the data source and calculate the ratio of its loading cost (C) and access frequency (F) to determine whether to store it in the cache. Only when the ratio is lower than a certain preset threshold is it stored in the cache to save cache resources.

[0098] In order to achieve collaborative scheduling of tasks, in one embodiment of the present invention, the determining of the computing resource status of the cloud and the edge in step 102 in the above embodiment includes:

[0099] Determine the occupancy of computing resources on the cloud and the edge, the remaining amount of storage resources, and the real-time status of network bandwidth.

[0100] In the embodiment of the present invention, in order to realize the coordinated scheduling of tasks, it is also necessary to monitor the resource status of the cloud and edge in real time, including the occupancy of computing resources (U), the remaining amount of storage resources (S), and the real-time status of network bandwidth (B), etc. Through the real-time monitoring of this information, the processing capacity of each node can be evaluated.

[0101] In order to improve the efficiency of cloud-edge collaboration, in one embodiment of the present invention, before using the intelligent cache module to obtain the user's historical access data in step 105, the following steps are further included:

[0102] Using the distributed loading module, determine if there is a large data set;

[0103] When the large data set exists, dividing the large data set into at least one data subset;

[0104] The at least one divided data subset is distributed to the computing nodes corresponding to the cloud and the edge end for data storage.

[0105] In an embodiment of the present invention, distributed loading technology can be used to further improve the efficiency of cloud-edge collaboration, thereby achieving higher performance and stability in the entire data processing process. Distributed loading technology, with its unique data subset partitioning and parallel processing mechanism, has shown significant advantages in the processing of large data sets. First, in the data subset partitioning stage, a large data set can be divided into multiple data subsets according to the specific needs of data processing and the characteristics of the data set. This not only ensures the integrity of the data, but also makes each data subset relatively independent, providing a solid foundation for subsequent distributed processing. By reasonably setting the size and number of subsets, the computing resources of the cloud and edge can be fully utilized to optimize the use of computing resources. At the same time, the distributed file system or cloud storage service can be used to distribute the divided data subsets to the cloud and edge computing nodes for data storage.

[0106] like Figure 2 As shown, in order to more clearly illustrate the technical solution and advantages of the present invention, a data processing method based on cloud-edge collaboration is provided in detail for an embodiment of the present invention, which may specifically include the following steps:

[0107] Step 201: using the collaborative scheduling module, determining a first execution script, wherein the first execution script is used to determine at least one of the computational complexity, data type, and real-time requirement information of each task received by the cloud and the edge;

[0108] Specifically, edge computing, as an emerging computing model, can sink computing tasks and data storage from the cloud to the edge of the network, thereby reducing network latency and increasing data processing speed. However, edge computing also faces problems such as limited computing resources and data security. Therefore, combining cloud computing with edge computing to form a cloud-edge collaborative architecture is the key to achieving efficient data processing and transmission. Step 202: Based on the first execution script, determine at least one of the computational complexity, data type, and real-time requirement information of at least one task;

[0109] Step 203: Determine the task priority based on the computational complexity, data type, real-time requirement information, and the first formula;

[0110] Specifically, the first formula is: P = α1×C+β1×T1+γ×R; wherein α1 is the weight coefficient of computational complexity, β1 is the weight coefficient of data type, γ is the weight coefficient of real-time requirement, C is computational complexity, T1 is data type, R is real-time requirement, and P is task priority.

[0111] Step 204: Determine the occupancy of computing resources, remaining amount of storage resources and real-time status of network bandwidth of the cloud and edge;

[0112] Step 205: Based on the task priority, the occupancy of computing resources, the remaining amount of storage resources, and the real-time status of network bandwidth, determine the target scheduling position corresponding to the task, where the target scheduling position includes the cloud and the edge;

[0113] Step 206: dispatching at least one task to a target dispatching position;

[0114] Step 207: using the distributed loading module, determining whether there is a large data set;

[0115] Step 208: When there is a large data set, divide the large data set into at least one data subset;

[0116] Step 209: Distribute the divided at least one data subset to the computing nodes corresponding to the cloud and the edge for data storage;

[0117] Specifically, by using distributed storage technology, data is stored in multiple nodes in a dispersed manner, and data loading efficiency can be improved through parallel loading. Distributed loading technology can further improve the efficiency of cloud-edge collaboration, thereby achieving higher performance and stability in the entire data processing process. Distributed loading technology has shown significant advantages in processing large data sets with its unique data subset partitioning and parallel processing mechanism.

[0118] For example, hashing algorithms or clustering techniques can be used to efficiently divide large data sets into multiple subsets. In the process of distributing data subsets, load balancing algorithms can be used to ensure data load balancing between nodes and avoid resource bottlenecks. At the same time, in order to ensure the efficiency and security of data during transmission, data compression and encryption technologies can also be applied. Data compression reduces the amount of data transmitted and improves transmission efficiency; encryption technology ensures the security of data during transmission and prevents data leakage and illegal access.

[0119] During the data loading phase, cloud and edge nodes can use multi-threading and asynchronous I / O technology to achieve parallel loading of data subsets. By optimizing data reading strategies and caching mechanisms, disk I / O and network transmission delays are reduced, further improving data loading speed. At the same time, in order to ensure the accuracy and reliability of data loading, data verification and error recovery mechanisms can also be used. Once data errors or damage are found, they can be repaired and replaced in a timely manner to ensure data integrity and accuracy.

[0120] Step 210: Obtain the user's historical access data using the intelligent cache module;

[0121] Step 211: Determine the access frequency and access time interval of the data based on the historical access data;

[0122] Step 212: determining at least one hotspot identification index based on the access frequency and access time interval of the data and a second formula;

[0123] Specifically, the second formula is: Among them, α2 is the weight coefficient of the access frequency of the data, β2 is the weight coefficient of the access time interval, F is the access frequency of the data, T2 is the access time interval, and H is the hotspot identification index.

[0124] Step 213: determining a target cache location based on the historical access data and at least one hot spot identification index, wherein the target cache location includes a high-speed cache area and a low-speed cache area;

[0125] Specifically, through intelligent caching strategies, we can improve the cache hit rate of data, reduce the number of repeated data transmission and loading, and thus achieve resource reuse. In order to ensure the consistency of cached data, we have established a cache synchronization mechanism. The triggering conditions for cache synchronization can be expressed as:

[0126] Trigger=△ Data >Threshold>

[0127] Among them, △ Data Indicates the amount of change in cloud data. Threshold is the threshold that triggers cache synchronization.

[0128] Step 214: Based on at least one task, use the cloud processing module and the edge computing module to process the data at the target cache location respectively.

[0129] Specifically, when data processing is completed, each node can upload the processing results to the central node for summary and integration. The central node can use efficient data merging algorithms and distributed computing frameworks to quickly compare, verify and merge processing results from different nodes. This process makes full use of the parallel processing capabilities of the distributed computing framework to achieve rapid generation and integration of processing results. At the same time, data optimization technology is used to compress and deduplicate the merged results, further reducing storage space and transmission costs. The processing results can be fed back to users or upper-level applications in the form of API interfaces or data reports. Users or upper-level applications can make corresponding decisions and operations based on these results, improving the efficiency and accuracy of the entire data processing process.

[0130] like Figure 3 As shown, an embodiment of the present invention provides a data processing system based on cloud-edge collaboration, the system comprising:

[0131] The collaborative scheduling module 301 is used to execute:

[0132] Performing attribute analysis and task priority classification on at least one task received by the cloud and the edge;

[0133] Determining the computing resource status of the cloud and the edge;

[0134] Based on the task priority and the computing resource status, determine a target scheduling position corresponding to the task, wherein the target scheduling position includes the cloud and the edge; schedule at least one task to the target scheduling position;

[0135] The intelligent cache module 302 is used to perform:

[0136] Get the user's historical access data;

[0137] Based on the historical access data, determining at least one hotspot identification index, wherein the hotspot identification index is used to reflect the cache priority corresponding to each data item;

[0138] Determining a target cache location based on the historical access data and the at least one hot spot identification index, wherein the target cache location includes a high-speed cache area and a low-speed cache area;

[0139] The processing module 303 is used to perform data processing on the data at the target cache location based on the at least one task of the collaborative scheduling module.

[0140] Specifically, data transmission between modules can be established by setting up a communication module. The processing module may include a cloud processing module and an edge computing module.

[0141] based on Figure 3In the data processing system based on cloud-edge collaboration shown in the figure, the collaborative scheduling module 301 in the above embodiment is also used to execute:

[0142] Determine a first execution script using the collaborative scheduling module, wherein the first execution script is used to determine at least one of the computational complexity, data type, and real-time requirement information of each of the tasks;

[0143] Determine at least one of the computational complexity, the data type, and the real-time requirement information of the at least one task based on the first execution script;

[0144] The task priority is determined based on the computational complexity, the data type, the real-time requirement information, and a first formula, where the first formula is:

[0145] P = α1 × C + β1 × T1 + γ × R;

[0146] Among them, α1 is the weight coefficient of the computational complexity, β1 is the weight coefficient of the data type, γ is the weight coefficient of the real-time requirement, C is the computational complexity, T1 is the data type, R is the real-time requirement, and P is the task priority.

[0147] based on Figure 3 In the data processing system based on cloud-edge collaboration shown in the figure, the intelligent cache module 302 in the above embodiment is also used to execute:

[0148] Based on the historical access data, determine the access frequency and access time interval of the data;

[0149] At least one hotspot identification index is determined based on the access frequency of the data and the access time interval and the following second formula, where the second formula is:

[0150]

[0151] Among them, α2 is the weight coefficient of the access frequency of the data, β2 is the weight coefficient of the access time interval, F is the access frequency of the data, T2 is the access time interval, and H is the hotspot identification index.

[0152] based on Figure 3 The data processing system based on cloud-edge collaboration is shown, and the collaborative scheduling module 301 in the above embodiment is also used to execute:

[0153] Determine the occupancy of computing resources on the cloud and the edge, the remaining amount of storage resources, and the real-time status of network bandwidth.

[0154] based on Figure 3 A data processing system based on cloud-edge collaboration is shown in FIG. Figure 4 As shown, in one embodiment of the present invention, it further includes: a distributed loading module 304;

[0155] The distributed loading module is used to execute:

[0156] Determine if a large dataset exists;

[0157] When the large data set exists, dividing the large data set into at least one data subset;

[0158] The at least one divided data subset is distributed to the computing nodes corresponding to the cloud and the edge end for data storage.

[0159] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on a report merging device. In other embodiments of the present invention, a report merging device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0160] The information interaction, execution process and other contents between the units in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention. For specific contents, please refer to the description in the embodiment of the method of the present invention, and no further description is given here.

[0161] An embodiment of the present invention further provides a data processing system based on cloud-edge collaboration, including: at least one memory and at least one processor;

[0162] at least one memory for storing a machine-readable program;

[0163] At least one processor is used to call a machine-readable program to execute a cloud-edge collaborative data processing method in any embodiment of the present invention.

[0164] An embodiment of the present invention also provides a computer-readable medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes a cloud-edge collaborative data processing method in any embodiment of the present invention.

[0165] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program code stored in the storage medium.

[0166] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0167] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0168] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0169] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0170] Each embodiment of the present invention has at least the following beneficial effects:

[0171] 1. In an embodiment of the present invention, a data processing method based on cloud-edge collaboration is provided. By using a collaborative scheduling module, processing tasks can be dynamically scheduled according to data processing requirements and network status. In addition, the method also uses an intelligent cache module to establish a cache mechanism between the cloud processing module and the edge computing module, and uses a distributed loading module to divide and distribute large data sets to achieve parallel processing. The above methods can realize dynamic adjustment and optimization of data collaborative scheduling, intelligent caching, and distributed loading strategies, combine cloud computing with edge computing, and form a cloud-edge collaborative architecture, which can improve the efficiency of data processing;

[0172] 2. In an embodiment of the present invention, in order to achieve collaborative scheduling of cloud and edge tasks, a collaborative scheduling module can be used to reasonably allocate computing tasks to various computing nodes, thereby reducing the waiting time of each task and the waste of computing resources. At the same time, in terms of collaborative scheduling, this solution adopts a scheduling strategy based on task attribute analysis and task priority division. Through the first execution script, the tasks received by the cloud and edge can be analyzed in detail to determine key information such as the computational complexity, data type, and real-time requirements corresponding to each task. Then, based on this information and the above-mentioned first formula, the priority corresponding to each task can be determined to ensure that high-priority tasks can be given priority, thereby meeting the user's requirements for task processing speed;

[0173] 3. In the embodiment of the present invention, in order to improve the efficiency of data exchange between the cloud and the edge, and to realize intelligent caching technology, hot data and unpopular data can be identified, and corresponding caching strategies can be formulated. First, a large amount of historical access data of users is obtained, and machine learning algorithms such as cluster analysis and association rule mining can be used to train and analyze these data to reveal the access rules of the data, thereby determining the access frequency (F) and access time interval (T) of the data. In addition, the access frequency and access time interval of the data can also be determined according to the user's behavior pattern. At the same time, for hot data, a cache priority (P) can be assigned to each data item according to the hot spot identification index, and high-priority data is stored in the cache, while low-priority data is stored in the slow cache; for unpopular data, an on-demand loading strategy can be adopted. When these data are accessed, we load the data from the data source and calculate the ratio of its loading cost (C) and access frequency (F) to determine whether to store it in the cache. Only when the ratio is lower than a certain preset threshold is it stored in the cache to save cache resources.

[0174] It should be noted that not all steps and modules in the above-mentioned processes and system structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0175] In the above embodiments, the hardware unit can be realized by mechanical means or electrical means. For example, a hardware unit can include permanent dedicated circuits or logic (such as special processors, FPGA or ASIC) to complete the corresponding operation. The hardware unit can also include programmable logic or circuits (such as general-purpose processors or other programmable processors), which can be temporarily set by software to complete the corresponding operation. Concrete implementation (mechanical means or dedicated permanent circuits or temporarily set circuits) can be determined based on cost and time considerations.

[0176] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data processing method based on cloud-edge collaboration, characterized in that: The method includes: Using the collaborative scheduling module, attribute analysis and task priority division are performed on at least one task received by the cloud and the edge; Determining the computing resource status of the cloud and the edge; Based on the task priority and the computing resource status, determining a target scheduling position corresponding to the task, wherein the target scheduling position includes the cloud and the edge; dispatching the at least one task to the target dispatching location; Use the intelligent cache module to obtain the user's historical access data; Based on the historical access data, determining at least one hotspot identification index, wherein the hotspot identification index is used to reflect the cache priority corresponding to each data item; Determining a target cache location based on the historical access data and the at least one hot spot identification index, wherein the target cache location includes a high-speed cache area and a low-speed cache area; Based on the at least one task, the cloud processing module and the edge computing module are used to process the data at the target cache location respectively.

2. The method according to claim 1, characterized in that: The collaborative scheduling module is used to perform attribute analysis and task priority division on at least one task received by the cloud and the edge, including: Determine a first execution script using the collaborative scheduling module, wherein the first execution script is used to determine at least one of the computational complexity, data type, and real-time requirement information of each of the tasks; Determine at least one of the computational complexity, the data type, and the real-time requirement information of the at least one task based on the first execution script; The task priority is determined based on the computational complexity, the data type, the real-time requirement information, and a first formula, where the first formula is: P = α1 × C + β1 × T1 + γ × R; Among them, α1 is the weight coefficient of the computational complexity, β1 is the weight coefficient of the data type, γ is the weight coefficient of the real-time requirement, C is the computational complexity, T1 is the data type, R is the real-time requirement, and P is the task priority.

3. The method according to claim 1, characterized in that The determining, based on the historical access data, at least one hotspot identification index, wherein the hotspot identification index is used to reflect the cache priority corresponding to each data item, includes: Based on the historical access data, determine the access frequency and access time interval of the data; At least one hotspot identification index is determined based on the access frequency of the data and the access time interval and the following second formula, where the second formula is: Among them, α2 is the weight coefficient of the access frequency of the data, β2 is the weight coefficient of the access time interval, F is the access frequency of the data, T2 is the access time interval, and H is the hotspot identification index.

4. The method according to any one of claims 1 to 3, characterized in that: The determining of the computing resource status of the cloud and the edge includes: Determine at least one of the occupancy of computing resources of the cloud and the edge, the remaining amount of storage resources, and the real-time status of network bandwidth; and / or, Before the intelligent cache module is used to obtain the user's historical access data, the method further includes: Using the distributed loading module, determine if there is a large data set; When the large data set exists, dividing the large data set into at least one data subset; The at least one divided data subset is distributed to the computing nodes corresponding to the cloud and the edge end for data storage.

5. The data processing system based on cloud-edge collaboration is characterized by: The system includes: a collaborative scheduling module, configured to perform attribute analysis and task priority division on at least one task received by the cloud and the edge, determine the computing resource status of the cloud and the edge, determine a target scheduling position corresponding to the task based on the task priority and the computing resource status, wherein the target scheduling position includes the cloud and the edge; and schedule the at least one task to the target scheduling position; An intelligent cache module, configured to obtain historical access data of a user, determine at least one hot spot identification index based on the historical access data, wherein the hot spot identification index is used to reflect the cache priority corresponding to each data item, and determine a target cache location based on the historical access data and the at least one hot spot identification index, wherein the target cache location includes a high-speed cache area and a low-speed cache area; A processing module is used to perform data processing on the data of the target cache location based on the at least one task of the collaborative scheduling module.

6. The system according to claim 5, characterized in that The collaborative scheduling module is used to execute: Determine a first execution script using the collaborative scheduling module, wherein the first execution script is used to determine at least one of the computational complexity, data type, and real-time requirement information of each of the tasks; Determine at least one of the computational complexity, the data type, and the real-time requirement information of the at least one task based on the first execution script; The task priority is determined based on the computational complexity, the data type, the real-time requirement information, and a first formula, where the first formula is: P = α1 × C + β1 × T1 + γ × R; Among them, α1 is the weight coefficient of the computational complexity, β1 is the weight coefficient of the data type, γ is the weight coefficient of the real-time requirement, C is the computational complexity, T1 is the data type, R is the real-time requirement, and P is the task priority.

7. The system according to claim 5, characterized in that The intelligent cache module is used to execute: Based on the historical access data, determine the access frequency and access time interval of the data; At least one hotspot identification index is determined based on the access frequency of the data and the access time interval and the following second formula, where the second formula is: Among them, α2 is the weight coefficient of the access frequency of the data, β2 is the weight coefficient of the access time interval, F is the access frequency of the data, T2 is the access time interval, and H is the hotspot identification index.

8. The system according to any one of claims 5 to 7, characterized in that: The collaborative scheduling module is used to execute: Determine the occupancy of computing resources on the cloud and the edge, the remaining amount of storage resources, and the real-time status of network bandwidth; and / or, Further comprising: a distributed loading module; The distributed loading module is used to execute: Determine if a large dataset exists; When the large data set exists, dividing the large data set into at least one data subset; The at least one divided data subset is distributed to the computing nodes corresponding to the cloud and the edge end for data storage.

9. The data processing system based on cloud-edge collaboration is characterized by: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 4.

10. A computer readable medium, characterized in that The computer readable medium stores computer instructions, which, when executed by a processor, cause the processor to perform any one of the methods of claims 1 to 4.

Images