Edge cloud collaborative data processing system and method based on deep learning

Abstract

The application provides a kind of edge cloud collaborative data processing system and method based on deep learning, related to edge cloud collaborative technical field, the task characteristic vector of edge terminal equipment is extracted from data task flow;Further construct task dependency graph, determine the allocation label of subtask processing through the resource allocation granularity of subtask in task dependency graph and data task flow;Confidence adjustment is carried out to each allocation label, to obtain the execution confidence of subtask in edge terminal equipment, further load transfer analysis is carried out to edge terminal equipment, to obtain the load transfer amount of edge terminal equipment;When load transfer amount is less than preset load transfer threshold, the subtask in data task flow is executed asynchronously in edge terminal equipment, otherwise, the data task flow in edge terminal equipment is sent to cloud server for batch processing through edge cache server, can be realized in the condition of limited heterogeneous computing resources, the task load between edge terminal equipment and cloud server is divided.

Description

Technical Field

[0001] This application relates to the field of edge cloud collaboration technology, and more specifically, to an edge cloud collaborative data processing system and method based on deep learning. Background Technology

[0002] Edge-cloud collaboration technology is a distributed computing architecture that integrates the advantages of edge computing and cloud computing. It aims to optimize the utilization efficiency of computing resources and system response performance. This technology completes some computing tasks and data processing at edge nodes close to the data source, enabling rapid response to latency-sensitive applications and significantly reducing network load and latency during data transmission. Meanwhile, cloud servers handle complex computations, massive data storage, and in-depth analysis tasks, leveraging their powerful computing capabilities. The edge and cloud use a dynamic scheduling mechanism to rationally allocate and collaboratively process tasks, adjusting the computing load based on task characteristics, resource availability, and network environment to ensure optimal overall system performance. Furthermore, edge-cloud collaboration technology supports heterogeneous devices and diverse application scenarios, flexibly adapting to the needs of multiple fields such as the Internet of Things, smart manufacturing, smart cities, and autonomous driving. Through collaborative design, edge-cloud collaboration not only improves computing efficiency and service quality but also enhances system scalability and fault tolerance, promoting the popularization and development of intelligent applications.

[0003] With the rapid development of the Internet of Things (IoT) and smart terminals, edge computing is widely used in data processing and real-time response. Edge terminal devices, being close to the data source, can effectively reduce response latency and alleviate cloud load. However, their computing power, storage capacity, and energy consumption are limited, making it difficult to complete complex and computationally intensive tasks. Cloud servers, on the other hand, have abundant computing resources and storage space, making them suitable for handling large-scale and complex tasks. However, uploading all tasks to the cloud leads to resource waste and transmission overhead. Furthermore, the tasks to be processed vary significantly in terms of computational complexity, real-time requirements, data dependencies, and resource consumption, resulting in prominent task heterogeneity. Different tasks need to be rationally allocated between the edge and the cloud to ensure overall system performance. However, how to efficiently allocate task load based on task characteristics under the limited resources of edge terminal devices to maximize resource utilization and minimize response latency remains a serious challenge. Therefore, how to allocate task load between edge terminal devices and cloud servers under the condition of limited heterogeneous computing resources has become a difficult problem for the industry. Summary of the Invention

[0004] This application provides an edge cloud collaborative data processing system and method based on deep learning, which can realize the division of task load between edge terminal devices and cloud servers under the condition of limited heterogeneous computing resources.

[0005] In a first aspect, this application provides a deep learning-based edge cloud collaborative data processing method, comprising the following steps:

[0006] Receive the data task streams to be processed from all edge terminal devices, and extract the task feature vector of each edge terminal device from each data task stream;

[0007] A task dependency graph is constructed based on all task feature vectors and a preset deep learning model. The allocation label for processing subtasks in each data task flow is determined by the task dependency graph and the resource allocation granularity of subtasks in each data task flow.

[0008] Based on the computing resource status and bandwidth utilization of the cloud server, confidence adjustment is performed on each allocation tag to obtain the execution confidence of subtasks in each data task flow on the edge terminal device. Load transfer analysis is performed on each edge terminal device using all execution confidence to obtain the load transfer amount of each edge terminal device.

[0009] When the load transfer amount is less than the preset load transfer threshold, the subtasks in the data task flow are executed asynchronously within the edge terminal device. When the load transfer amount is greater than or equal to the preset load transfer threshold, the data task flow in the edge terminal device is sent to the cloud server for batch processing through the edge cache server.

[0010] In this embodiment, extracting the task feature vector of each edge terminal device from each data task stream specifically includes:

[0011] Each data task stream is segmented into frames to obtain multiple sub-tasks for each edge terminal device;

[0012] The feature information of each subtask is extracted, and then the task feature vector of each edge terminal device is determined based on the feature information of all task segments.

[0013] In this embodiment, constructing a task dependency graph based on all task feature vectors and a preset deep learning model specifically includes:

[0014] Input all task feature vectors into a pre-defined deep learning model to determine the data dependency strength of different subtasks in each data task stream;

[0015] Construct a task dependency graph based on the strength of data dependencies between different subtasks in all data task flows.

[0016] In this embodiment, determining the allocation label for processing subtasks in each data task flow through the task dependency graph and the resource allocation granularity of subtasks in each data task flow specifically includes:

[0017] Determine the granularity of resource allocation for subtasks in each data task stream;

[0018] The task dependency graph is used to evaluate the priority of subtasks in each data task flow to obtain the execution priority of subtasks in each data task flow. The execution priority indicates the urgency of the subtasks being scheduled for execution.

[0019] The allocation label for processing subtasks in each data task flow is determined based on the resource allocation granularity and execution priority of subtasks in each data task flow.

[0020] In this embodiment, confidence adjustment is performed on each allocation tag based on the computing resource status and bandwidth utilization of the cloud server to obtain the execution confidence of subtasks in each data task stream on the edge terminal device. Specifically, this includes:

[0021] Obtain the computing resource status and bandwidth utilization of the cloud server;

[0022] The execution feasibility score of each edge terminal device's data task flow is calculated based on the computing resource status and bandwidth utilization of the cloud server.

[0023] The confidence level of the execution of subtasks in each data task flow is obtained by evaluating the assignment labels during the processing of subtasks in each data task flow through the execution feasibility score of each data task flow on the edge terminal device.

[0024] In this embodiment, load transfer analysis is performed on each edge terminal device using all execution confidence levels to obtain the specific load transfer amount for each edge terminal device, including:

[0025] By analyzing the execution confidence of subtasks in each data task stream on the edge terminal device, the load status index of each edge terminal device is obtained.

[0026] The load transfer amount for each edge terminal device is obtained by evaluating the transfer of subtasks in the data task flow of each edge terminal device based on the load status index of each edge terminal device.

[0027] In this embodiment, asynchronously executing subtasks in the data task flow within the edge terminal device specifically includes:

[0028] Create a task queue within the edge terminal device;

[0029] An asynchronous scheduling mechanism is adopted to dynamically allocate the execution of subtasks in the task queue within the time window when resources are available.

[0030] In this embodiment, sending the data task stream from the edge terminal device to the cloud server for batch processing via the edge caching server specifically includes:

[0031] Compress some subtasks of the data task flow in the edge terminal device, and transmit the compressed data of some subtasks to the cloud server through the edge cache server;

[0032] After receiving the compressed data, the cloud server uses a distributed batch processing framework for scheduling and execution.

[0033] In this embodiment, the allocation tag represents a marker indicating the execution location selected during subtask processing in the data task flow, wherein the execution location is an edge terminal device or a cloud server.

[0034] Secondly, this application provides a deep learning-based edge cloud collaborative data processing system, the edge cloud collaborative data processing system comprising:

[0035] The receiving module is used to receive the data task streams to be processed in all edge terminal devices and extract the task feature vector of each edge terminal device from each data task stream.

[0036] The feature processing module is used to construct a task dependency graph based on all task feature vectors and a preset deep learning model, and to determine the allocation label when processing subtasks in each data task flow through the task dependency graph and the resource allocation granularity of subtasks in each data task flow.

[0037] The feature processing module is also used to adjust the confidence of each assigned label based on the computing resource status and bandwidth utilization of the cloud server, to obtain the execution confidence of subtasks in each data task flow on the edge terminal device, and to perform load transfer analysis on each edge terminal device through all execution confidence to obtain the load transfer amount of each edge terminal device.

[0038] The execution module is used to asynchronously execute subtasks in the data task flow within the edge terminal device when the load transfer amount is less than the preset load transfer threshold, and to send the data task flow in the edge terminal device to the cloud server for batch processing through the edge cache server when the load transfer amount is greater than or equal to the preset load transfer threshold.

[0039] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects:

[0040] First, all data task streams to be processed within the edge terminal devices are received, and the task feature vector of each edge terminal device is extracted from each data task stream. A task dependency graph is constructed based on all task feature vectors and a preset deep learning model. The allocation label for each subtask in each data task stream is determined using the task dependency graph and the resource allocation granularity of the subtasks within each data task stream. Confidence adjustment is performed on each allocation label based on the computing resource status and bandwidth utilization of the cloud server to obtain the execution confidence of each subtask in the data task stream on the edge terminal devices. Load transfer analysis is performed on each edge terminal device using all execution confidence values ​​to obtain the load transfer amount for each edge terminal device. When the load transfer amount is less than a preset load transfer threshold, the subtasks in the data task stream are executed asynchronously within the edge terminal device. When the load transfer amount is greater than or equal to the preset load transfer threshold, the data task stream from the edge terminal device is sent to the cloud server for batch processing via the edge cache server.

[0041] Therefore, this application performs load transfer analysis on each edge terminal device using all execution confidence levels to obtain the load transfer amount for each edge terminal device. Firstly, by constructing a task dependency graph, the system can comprehensively identify the logical relationships between subtasks. Through deep learning models' understanding of task characteristics, the task dependency graph can accurately depict the internal structural complexity and processing paths of subtasks, thus providing a clear basis for the reasonable division and scheduling of subtasks. Under conditions of limited heterogeneous computing resources, the task dependency graph can identify which subtasks are suitable for local processing on edge devices and which require collaborative execution in the cloud. Secondly, by determining allocation labels through the task dependency graph and the resource allocation granularity of subtasks in the data task flow, a comprehensive quantification and labeling of the processing requirements and dependencies of each subtask can be achieved. The allocation label, as the core basis for task scheduling, reflects the appropriate location (i.e., edge or cloud) of the subtask under specific resource constraints. Confidence adjustments to the allocation label based on the computing resource status and bandwidth utilization of the cloud server can dynamically reflect the actual load and network status of the current system resources, thereby more accurately assessing the execution possibility and reliability of each subtask on the edge terminal device. Row confidence, as a crucial indicator for task scheduling, determines whether edge terminal devices possess sufficient resources to efficiently complete sub-tasks, avoiding performance degradation caused by resource overload or communication bottlenecks. Then, load balancing analysis is performed on each edge terminal device to accurately assess its current computational pressure and resource usage. The load balancing amount reflects whether the device is overloaded and the extent to which tasks need to be transferred to the cloud. In heterogeneous computing resource-constrained environments, this avoids task processing delays or failures due to insufficient resources, achieving load balancing between edge terminal devices and cloud servers. This effectively utilizes the powerful computing resources of cloud servers, ensuring the overall efficiency and stability of task processing and promoting optimized system resource allocation and collaborative operation. Finally, when the load balancing amount is less than a preset threshold, the system chooses to execute sub-tasks asynchronously within the edge terminal device, fully leveraging the real-time response advantages and low latency characteristics of edge devices to improve the immediacy and efficiency of task processing. When the load balancing amount is greater than or equal to the threshold, tasks are forwarded to the cloud for batch processing via the edge cache server, utilizing the powerful computing and storage capabilities of the cloud to alleviate resource pressure on edge devices and avoid overload risks. This dynamic switching mechanism based on load thresholds enables intelligent allocation and collaborative processing of task load between the edge and the cloud, ensuring that the system can guarantee response latency and maximize resource utilization under conditions of limited heterogeneous resources, thereby improving the overall system stability and performance.

[0042] In summary, the proposed solution can achieve the allocation of task load between edge terminal devices and cloud servers under conditions of limited heterogeneous computing resources. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this embodiment of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of the edge cloud collaborative data processing method based on deep learning provided in this application;

[0045] Figure 2 This is an exemplary flowchart for determining the task dependency graph provided in this application;

[0046] Figure 3 This is an exemplary flowchart for determining the execution confidence level provided in this application;

[0047] Figure 4 This is a module structure diagram of the edge cloud collaborative data processing system based on deep learning provided in this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] This application provides an edge cloud collaborative data processing system and method based on deep learning. Its core is to determine the collaborative adaptability of scheduling resources to logistics tasks by analyzing the execution status characteristics and scheduling strategies of the logistics tasks; to divide transportation resources into multiple resource allocation units, determine the resource collaboration relationships between these units, and determine the scheduling cost of each resource allocation unit for the logistics tasks based on these collaboration relationships and the scheduling cycle constraints of the logistics network; to perform a fuzzy evaluation of the scheduling adaptability relationship between resource allocation and task requirements in the logistics network using all scheduling costs and collaboration adaptability, and determine a scheduling optimization index based on fuzzy logic; and to perform load-balanced scheduling of logistics tasks in each resource allocation unit based on the scheduling optimization index. This application uses fuzzy logic to comprehensively evaluate the resource collaboration relationships and scheduling adaptability in complex logistics scheduling, thereby achieving load balancing in logistics scheduling.

[0050] Example 1: To better understand the above technical solution, the following will provide a detailed description of the technical solution in conjunction with the accompanying drawings and specific implementation methods. (Refer to...) Figure 1As shown in the figure, this is an exemplary flowchart of a deep learning-based edge cloud collaborative data processing method according to this embodiment of the application. The deep learning-based edge cloud collaborative data processing method includes the following steps:

[0051] In step S1, the data task streams to be processed in all edge terminal devices are received, and the task feature vector of each edge terminal device is extracted from each data task stream.

[0052] It should be noted that, in this application, the data task flow refers to an ordered set of data processing tasks generated on an edge terminal device that have a certain processing order and dependency relationship. The data task flow consists of multiple ordered subtasks, each of which is responsible for performing specific processing operations on the input data and has dependencies and resource requirements during execution.

[0053] In this embodiment, extracting the task feature vector of each edge terminal device from each data task stream can be achieved through the following steps:

[0054] Each data task stream is segmented into frames to obtain multiple sub-tasks for each edge terminal device;

[0055] The feature information of each subtask is extracted, and then the task feature vector of each edge terminal device is determined based on the feature information of all task segments.

[0056] It should be noted that the task feature vector described in this application represents a vector representation composed of the feature information of the subtasks to be processed within the edge terminal device.

[0057] In specific implementation, firstly, existing time series segmentation techniques are used to segment each data task stream, thereby obtaining multiple task segments corresponding to each data task stream on the edge terminal device. Each of these task segments is then treated as a subtask, resulting in multiple subtasks for each edge terminal device. Next, feature engineering techniques are used to extract the feature information of each subtask. The feature information consists of task computational complexity, resource consumption, execution latency, memory usage, and input / output data size. Finally, the vector composed of feature information belonging to the same data task stream is used as the task feature vector of the corresponding edge terminal device, thus obtaining the task feature vector of each edge terminal device.

[0058] In step S2, a task dependency graph is constructed based on all task feature vectors and a preset deep learning model. The allocation label for processing subtasks in each data task flow is determined by the task dependency graph and the resource allocation granularity of subtasks in each data task flow.

[0059] In this embodiment, reference Figure 2As shown, this diagram is an exemplary flowchart for determining the task dependency graph in an embodiment of this application. In this embodiment, the task dependency graph can be constructed based on all task feature vectors and a preset deep learning model using the following steps:

[0060] In step S21, all task feature vectors are input into a preset deep learning model to determine the data dependency strength of different subtasks in each data task flow.

[0061] In step S22, a task dependency graph is constructed based on the data dependency strength between different subtasks in all data task flows.

[0062] It should be noted that the pre-trained deep learning model in this application is a graph neural network model. Graph neural network models are suitable for deep learning models that handle task dependencies. In edge computing scenarios, each subtask is regarded as a node in the graph, and the feature information in the task feature vector is used as the node input. The dependencies between different subtasks in the data task flow are learned through a multi-layer message passing mechanism.

[0063] It should also be noted that the data dependency strength mentioned in this application refers to the correlation strength when data is exchanged between subtasks; the task dependency graph refers to the dependency correlation graph between subtasks in the edge computing environment during task execution.

[0064] In practice, firstly, all task feature vectors are input into a pre-defined deep learning model (i.e., a graph neural network model). The multi-layer message passing mechanism of the graph neural network model is used to extract the adjacency weights of the subtasks corresponding to the feature information of each task feature vector. The obtained adjacency weights are used as the data dependency strength of the subtasks, thereby obtaining the data dependency strength of different subtasks in each data task flow. Then, all subtasks are used as nodes and the data dependency strengths of all subtasks are used as edge weights to construct a graph structure, and the obtained graph structure is used as the task dependency graph.

[0065] In this embodiment, determining the allocation label for processing subtasks in each data task flow using the task dependency graph and the resource allocation granularity of subtasks in each data task flow can be achieved through the following steps:

[0066] Determine the granularity of resource allocation for subtasks in each data task stream;

[0067] The task dependency graph is used to evaluate the priority of subtasks in each data task flow to obtain the execution priority of subtasks in each data task flow. The execution priority indicates the urgency of the subtasks being scheduled for execution.

[0068] The allocation label for processing subtasks in each data task flow is determined based on the resource allocation granularity and execution priority of subtasks in each data task flow.

[0069] It should be noted that the resource allocation granularity described in this application refers to the minimum number of allocatable resource units for subtasks in the data task flow during resource scheduling and allocation.

[0070] Additionally, it should be noted that the allocation label mentioned in this application represents a marker for selecting the execution location when processing subtasks in a data task flow, wherein the execution location is an edge terminal device or a cloud server.

[0071] In specific implementation, firstly, static code analysis is performed on the subtasks in each data task flow to obtain the resource requirements of each subtask. Then, combined with the smallest allocation unit of system resource management in the edge terminal device (e.g., number of CPU cores, memory block size, and bandwidth segment), the resource requirements of each subtask in each data task flow are converted into the number of standard resource units. The obtained standard resource unit numbers are used as the resource allocation granularity of each subtask, thus obtaining the resource allocation granularity of each data task flow. Secondly, the critical path method is used to calculate the critical path length of each subtask node in the task dependency graph. Based on the critical path length, a priority sorting algorithm is used to sort all subtask nodes in the data task flow, and the sorting result is used as the execution priority of the corresponding subtask in the data task flow, thus obtaining... The execution priority of each subtask in each data task flow is determined. Then, the resource allocation granularity of each subtask in each data task flow is added to its execution priority, and all the sums are used as the location allocation value for the corresponding subtask. Thus, the location allocation values ​​for all subtasks in all data task flows are calculated. The average of all location allocation values ​​is then used as the location allocation equilibrium value. The execution location of subtasks with location allocation values ​​greater than the location allocation equilibrium value is marked as a cloud server, and the obtained mark is used as the allocation label when processing the subtask. The execution location of subtasks with location allocation values ​​less than or equal to the location allocation equilibrium value is marked as an edge terminal device, and the obtained mark is used as the allocation label when processing the subtask. Thus, the allocation label for processing subtasks in each data task flow is obtained.

[0072] In step S3, confidence adjustment is performed on each allocation tag based on the computing resource status and bandwidth utilization of the cloud server to obtain the execution confidence of subtasks in each data task flow on the edge terminal device. Load transfer analysis is performed on each edge terminal device based on all execution confidence to obtain the load transfer amount of each edge terminal device.

[0073] In this embodiment, reference Figure 3As shown, this diagram is an exemplary flowchart for determining execution confidence in an embodiment of this application. In this embodiment, confidence adjustment is performed on each allocation tag based on the computing resource status and bandwidth utilization of the cloud server. The execution confidence of subtasks in each data task stream on the edge terminal device can be obtained through the following steps:

[0074] In step S31, the computing resource status and bandwidth utilization of the cloud server are obtained;

[0075] In step S32, the execution feasibility score of the data task flow of each edge terminal device is calculated based on the computing resource status and bandwidth utilization of the cloud server.

[0076] In step S33, the confidence assessment of the allocation label during the processing of subtasks in each data task flow is performed by using the execution feasibility score of each data task flow of each edge terminal device, so as to obtain the execution confidence of subtasks in each data task flow on the edge terminal device.

[0077] It should be noted that the computing resource status of the cloud server in this application represents the overall situation of the currently available computing resources of the cloud server. The computing resource status specifically includes CPU utilization, memory utilization, and storage capacity. The bandwidth utilization represents a parameter that measures the efficiency of the current network transmission resource utilization of the cloud server.

[0078] Additionally, it should be noted that the execution feasibility score mentioned in this application refers to the feasibility score parameter for the execution of data task flows on edge terminal devices.

[0079] In specific implementation, firstly, the computational complexity and resource consumption of subtasks in the data task flow of each edge terminal device are obtained. Then, the CPU utilization rate of the cloud server's computing resources is divided by the memory utilization rate, and the difference between this value and the computational complexity of the subtask is used as the computational suitability of the subtask. Next, the resource consumption of the subtask is subtracted from the storage capacity of the cloud server, and the result is used as the storage margin indicator for the subtask. Finally, the computational suitability of the subtask is divided by the storage margin indicator, and the result is used as the execution feasibility of the subtask. The execution feasibility of all subtasks in each edge terminal device data task flow is summed, and the summed value is used as the execution feasibility score of each edge terminal device data task flow. Next, for each data task flow, the allocation labels of all subtasks in the data task flow are filtered to select the total number of edge terminal devices. The total number is multiplied by the execution feasibility score of the edge terminal device data task flow, and the multiplied value is used as the execution confidence of the subtasks in the data task flow on the edge terminal devices, thereby obtaining the execution confidence of the subtasks in each data task flow on the edge terminal devices.

[0080] It should be noted that the execution confidence level mentioned in this application represents the degree of confidence that the edge terminal device can successfully complete the subtask processing under the current system resource conditions.

[0081] In this embodiment, the load transfer analysis of each edge terminal device based on all execution confidence levels can be achieved through the following steps:

[0082] By analyzing the execution confidence of subtasks in each data task stream on the edge terminal device, the load status index of each edge terminal device is obtained.

[0083] The load transfer amount for each edge terminal device is obtained by evaluating the transfer of subtasks in the data task flow of each edge terminal device based on the load status index of each edge terminal device.

[0084] It should be noted that the load state index mentioned in this application represents an indicator of the degree of computing load undertaken by edge terminal devices under conditions of limited heterogeneous computing resources.

[0085] In specific implementation, firstly, for each data task flow, the execution confidence of subtasks in the data task flow on edge terminal devices can be summed, and the summed value is used as the load state index of the corresponding edge terminal device, thus obtaining the load state index of each edge terminal device; secondly, for each edge terminal device, the resource consumption and task computational complexity of subtasks in the data task flow of the edge terminal device are obtained, and the load state index of the edge terminal device is used as a weighting coefficient to perform a weighted summation of the resource consumption and task computational complexity of subtasks in the data task flow of the edge terminal device, and the weighted summation value is used as the transfer evaluation quantity of the subtasks. Then, the average of the transfer evaluation quantities of all subtasks is calculated, and the average value is used as the transfer evaluation equilibrium value. The set of subtasks corresponding to the transfer evaluation quantities greater than the transfer evaluation equilibrium value is used as the subtask cloud execution set of the edge terminal device, and the total number of subtasks in the subtask cloud execution set is used as the load transfer quantity of the edge terminal device, thus obtaining the load transfer quantity of each edge terminal device.

[0086] It should be noted that the load transfer amount mentioned in this application refers to the number of tasks that need to be transferred from edge devices to cloud servers for execution when computing resources are limited.

[0087] In step S4, when the load transfer amount is less than the preset load transfer threshold, the subtasks in the data task flow are executed asynchronously in the edge terminal device. When the load transfer amount is greater than or equal to the preset load transfer threshold, the data task flow in the edge terminal device is sent to the cloud server for batch processing through the edge cache server.

[0088] It should be noted that the load transfer threshold mentioned in this application represents the boundary value for determining whether an edge terminal device needs to transfer its task flow to a cloud server for processing. The average load transfer amount of all edge terminal devices can be used as the load transfer threshold.

[0089] In this embodiment, asynchronous execution of subtasks in the data task flow within the edge terminal device can be achieved using the following steps:

[0090] Create a task queue within the edge terminal device;

[0091] An asynchronous scheduling mechanism is adopted to dynamically allocate the execution of subtasks in the task queue within the time window when resources are available.

[0092] In specific implementation, firstly, a task queue for managing pending subtasks is built inside the edge terminal device. The task queue can be implemented through a first-in-first-out structure or a priority queue to store each subtask to be scheduled for execution. The task queue represents a centralized organization and scheduling container for the subtasks to be executed, used to achieve orderly management of tasks. Secondly, an asynchronous scheduling mechanism is built based on the current resource status of the edge terminal device (e.g., CPU idle rate, available memory, etc.). The asynchronous scheduling mechanism can perform non-blocking scheduling and dynamic allocation of subtasks in the task queue within the resource availability time window through event-driven or thread pool methods, to achieve concurrent or intermittent processing of tasks. The asynchronous scheduling mechanism refers to the scheduling logic that intelligently determines whether a task is executed based on resource availability without blocking the main process.

[0093] In this embodiment, sending the data task stream from the edge terminal device to the cloud server for batch processing via the edge caching server can be achieved using the following steps:

[0094] Compress some subtasks of the data task flow in the edge terminal device, and transmit the compressed data of some subtasks to the cloud server through the edge cache server;

[0095] After receiving the compressed data, the cloud server uses a distributed batch processing framework for scheduling and execution.

[0096] In specific implementation, firstly, the cloud execution set of subtasks from the edge terminal device is obtained. Existing data compression algorithms are used to compress the cloud execution set of subtasks, and the compressed data is used as the compressed subtask data. The edge caching server deployed between the edge network and the cloud is used as a relay node to cache, package, and forward the compressed subtask data to the cloud server. Secondly, after receiving the compressed subtask data, the cloud server uses the Apache Spark distributed batch processing framework to decompress, schedule, and execute the compressed subtask data. The distributed batch processing framework refers to a software platform that can uniformly manage, distribute, and execute batch data tasks in parallel on a multi-node cluster.

[0097] Therefore, this application performs load transfer analysis on each edge terminal device using all execution confidence levels to obtain the load transfer amount for each edge terminal device. Firstly, by constructing a task dependency graph, the system can comprehensively identify the logical relationships between subtasks. Through deep learning models' understanding of task characteristics, the task dependency graph can accurately depict the internal structural complexity and processing paths of subtasks, thus providing a clear basis for the reasonable division and scheduling of subtasks. Under conditions of limited heterogeneous computing resources, the task dependency graph can identify which subtasks are suitable for local processing on edge devices and which require collaborative execution in the cloud. Secondly, by determining allocation labels through the task dependency graph and the resource allocation granularity of subtasks in the data task flow, a comprehensive quantification and labeling of the processing requirements and dependencies of each subtask can be achieved. The allocation label, as the core basis for task scheduling, reflects the appropriate location (i.e., edge or cloud) of the subtask under specific resource constraints. Confidence adjustments to the allocation label based on the computing resource status and bandwidth utilization of the cloud server can dynamically reflect the actual load and network status of the current system resources, thereby more accurately assessing the execution possibility and reliability of each subtask on the edge terminal device. Row confidence, as a crucial indicator for task scheduling, determines whether edge terminal devices possess sufficient resources to efficiently complete sub-tasks, avoiding performance degradation caused by resource overload or communication bottlenecks. Then, load balancing analysis is performed on each edge terminal device to accurately assess its current computational pressure and resource usage. The load balancing amount reflects whether the device is overloaded and the extent to which tasks need to be transferred to the cloud. In heterogeneous computing resource-constrained environments, this avoids task processing delays or failures due to insufficient resources, achieving load balancing between edge terminal devices and cloud servers. This effectively utilizes the powerful computing resources of cloud servers, ensuring the overall efficiency and stability of task processing and promoting optimized system resource allocation and collaborative operation. Finally, when the load balancing amount is less than a preset threshold, the system chooses to execute sub-tasks asynchronously within the edge terminal device, fully leveraging the real-time response advantages and low latency characteristics of edge devices to improve the immediacy and efficiency of task processing. When the load balancing amount is greater than or equal to the threshold, tasks are forwarded to the cloud for batch processing via the edge cache server, utilizing the powerful computing and storage capabilities of the cloud to alleviate resource pressure on edge devices and avoid overload risks. This dynamic switching mechanism based on load thresholds enables intelligent allocation and collaborative processing of task load between the edge and the cloud, ensuring that the system can guarantee response latency and maximize resource utilization under conditions of limited heterogeneous resources, thereby improving the overall system stability and performance.

[0098] In summary, the proposed solution can achieve the allocation of task load between edge terminal devices and cloud servers under conditions of limited heterogeneous computing resources.

[0099] Example 2: This application provides an edge cloud collaborative data processing system based on deep learning, referencing... Figure 4 As shown in the figure, this is a schematic diagram of a deep learning-based edge cloud collaborative data processing system according to this embodiment of the present application. The deep learning-based edge cloud collaborative data processing system includes:

[0100] The receiving module 100 is used to receive the data task streams to be processed in all edge terminal devices and extract the task feature vector of each edge terminal device from each data task stream.

[0101] The feature processing module 200 is used to construct a task dependency graph based on all task feature vectors and a preset deep learning model, and to determine the allocation label when processing subtasks in each data task flow through the task dependency graph and the resource allocation granularity of subtasks in each data task flow.

[0102] The feature processing module 200 is also used to perform confidence adjustment on each assigned tag based on the computing resource status and bandwidth utilization of the cloud server, to obtain the execution confidence of subtasks in each data task flow on the edge terminal device, and to perform load transfer analysis on each edge terminal device through all execution confidence to obtain the load transfer amount of each edge terminal device.

[0103] The execution module 300 is used to asynchronously execute subtasks in the data task flow within the edge terminal device when the load transfer amount is less than the preset load transfer threshold, and to send the data task flow in the edge terminal device to the cloud server for batch processing through the edge cache server when the load transfer amount is greater than or equal to the preset load transfer threshold.

[0104] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0105] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compactdisc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0106] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

Claims

1. A deep learning-based edge cloud collaborative data processing method, characterized in that, Includes the following steps: Receive the data task streams to be processed from all edge terminal devices, and extract the task feature vector of each edge terminal device from each data task stream; A task dependency graph is constructed based on all task feature vectors and a preset deep learning model. The allocation label for processing subtasks in each data task flow is determined by the task dependency graph and the resource allocation granularity of subtasks in each data task flow. Specifically, determining the allocation label for processing subtasks in each data task flow through the task dependency graph and the resource allocation granularity of subtasks in each data task flow includes: Determine the resource allocation granularity of subtasks in each data task flow, whereby the resource allocation granularity represents the minimum number of allocatable resource units for a subtask in the data task flow during resource scheduling and allocation. The task dependency graph is used to evaluate the priority of subtasks in each data task flow to obtain the execution priority of subtasks in each data task flow. The execution priority indicates the urgency of the subtasks being scheduled for execution. The allocation label for processing subtasks in each data task flow is determined based on the resource allocation granularity and execution priority of subtasks in each data task flow. Based on the computing resource status and bandwidth utilization of the cloud server, confidence adjustment is performed on each allocation tag to obtain the execution confidence of subtasks in each data task flow on the edge terminal device. Load transfer analysis is performed on each edge terminal device using all execution confidence to obtain the load transfer amount of each edge terminal device. Specifically, the confidence adjustment of each allocation label based on the computing resource status and bandwidth utilization of the cloud server, to obtain the execution confidence of subtasks in each data task stream on the edge terminal device, includes: Obtain the computing resource status and bandwidth utilization of the cloud server; The execution feasibility score of each edge terminal device's data task flow is calculated based on the computing resource status and bandwidth utilization of the cloud server. The confidence assessment of the allocation label during the processing of subtasks in each data task flow is performed by scoring the execution feasibility of each data task flow of each edge terminal device, thereby obtaining the execution confidence of subtasks in each data task flow on the edge terminal device. The execution confidence represents the degree of credibility of the edge terminal device in successfully completing the subtask processing under the current system resource state. When the load transfer amount is less than the preset load transfer threshold, the subtasks in the data task flow are executed asynchronously within the edge terminal device. When the load transfer amount is greater than or equal to the preset load transfer threshold, the data task flow in the edge terminal device is sent to the cloud server for batch processing through the edge cache server.

2. The method as described in claim 1, characterized in that, Extracting the task feature vector for each edge terminal device from each data task stream specifically includes: Each data task stream is segmented into frames to obtain multiple sub-tasks for each edge terminal device; The feature information of each subtask is extracted, and then the task feature vector of each edge terminal device is determined based on the feature information of all task segments.

3. The method as described in claim 1, characterized in that, Constructing a task dependency graph based on all task feature vectors and a pre-defined deep learning model specifically includes: Input all task feature vectors into a pre-defined deep learning model to determine the data dependency strength of different subtasks in each data task stream; Construct a task dependency graph based on the strength of data dependencies between different subtasks in all data task flows.

4. The method as described in claim 1, characterized in that, Load transfer analysis is performed on each edge terminal device using all execution confidence levels to obtain the specific load transfer amount for each edge terminal device, including: By analyzing the execution confidence of subtasks in each data task stream on the edge terminal device, the load status index of each edge terminal device is obtained. The load transfer amount for each edge terminal device is obtained by evaluating the transfer of subtasks in the data task flow of each edge terminal device based on the load status index of each edge terminal device.

5. The method as described in claim 1, characterized in that, Asynchronous execution of subtasks in a data task flow within an edge terminal device specifically includes: Create a task queue within the edge terminal device; An asynchronous scheduling mechanism is adopted to dynamically allocate the execution of subtasks in the task queue within the time window when resources are available.

6. The method as described in claim 1, characterized in that, Sending data task streams from edge terminal devices to cloud servers for batch processing via edge caching servers specifically includes: Compress some subtasks of the data task flow in the edge terminal device, and transmit the compressed data of some subtasks to the cloud server through the edge cache server; After receiving the compressed data, the cloud server uses a distributed batch processing framework for scheduling and execution.

7. The method as described in claim 1, characterized in that, The allocation tag represents a marker indicating the execution location selected during subtask processing in the data task flow, where the execution location is either an edge terminal device or a cloud server.

8. A deep learning-based edge cloud collaborative data processing system, used to execute the deep learning-based edge cloud collaborative data processing method as described in any one of claims 1 to 7, characterized in that, The edge cloud collaborative data processing system includes: The receiving module is used to receive the data task streams to be processed in all edge terminal devices and extract the task feature vector of each edge terminal device from each data task stream. The feature processing module is used to construct a task dependency graph based on all task feature vectors and a preset deep learning model, and to determine the allocation label when processing subtasks in each data task flow through the task dependency graph and the resource allocation granularity of subtasks in each data task flow. The feature processing module is also used to adjust the confidence of each assigned label based on the computing resource status and bandwidth utilization of the cloud server, to obtain the execution confidence of subtasks in each data task flow on the edge terminal device, and to perform load transfer analysis on each edge terminal device through all execution confidence to obtain the load transfer amount of each edge terminal device. The execution module is used to asynchronously execute subtasks in the data task flow within the edge terminal device when the load transfer amount is less than the preset load transfer threshold, and to send the data task flow in the edge terminal device to the cloud server for batch processing through the edge cache server when the load transfer amount is greater than or equal to the preset load transfer threshold.

Images