Edge cloud and distributed storage hybrid computing task scheduling method and system

Abstract

The invention relates to the technical field of edge cloud and distributed storage hybrid computing, and discloses an edge cloud and distributed storage hybrid computing task scheduling system, which comprises an edge computing module, a distributed storage module, a task scheduling module, a network communication module, a monitoring and management module, a security and privacy module, a cloud edge collaboration module and an application layer interface module. According to the hybrid computing task scheduling system based on the edge cloud and the distributed storage, computing and storage resources are collaboratively optimized, a computing task is sunk to an edge node close to a data source, the time for transmitting data to the cloud is shortened, the data are fragmented and stored in a plurality of edge nodes, parallel reading and writing are supported, the single-point bottleneck is avoided, and the computing efficiency is improved. And the real-time response capability is improved, and decision lag caused by network delay is avoided, so that the resource scheduling efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of edge cloud and distributed storage hybrid computing technology, and specifically to an edge cloud and distributed storage hybrid computing task scheduling method and system. Background Art

[0002] Hybrid computing of edge cloud and distributed storage is a computing model that combines the advantages of edge computing and distributed storage. It aims to meet the application requirements of low latency, high bandwidth, and high reliability through resource collaborative optimization, sink computing power to the edge of the network, reduce data transmission delay, and is suitable for scenarios with high real-time requirements. It stores data in multiple nodes in a distributed manner, improves data reliability and access efficiency through redundant backup and load balancing, and combines the advantages of both to process real-time data at the edge node. At the same time, it uses distributed storage to achieve data persistence and global sharing. It is dynamically allocated to the edge or cloud according to the task type, and data is read preferentially from local edge storage or distributed storage to reduce cross-node transmission. Edge node overload is avoided through task migration or resource reservation.

[0003] As people's lives continue to change, the use of hybrid computing of edge cloud and distributed storage is becoming more and more common. However, the resource scheduling efficiency of traditional algorithms is low when used, so a new system is proposed to solve this technical problem. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In response to the shortcomings of the existing technology, the present invention provides a method and system for scheduling hybrid computing tasks of edge cloud and distributed storage, which has the advantages of improving resource scheduling efficiency and solves the problems mentioned in the above background technology.

[0006] (2) Technical solution

[0007] To achieve the purpose of improving resource scheduling efficiency mentioned in the above background technology, the present invention provides the following technical solution: an edge cloud and distributed storage hybrid computing task scheduling system, characterized by: including an edge computing module, a distributed storage module, a task scheduling module, a network communication module, a monitoring and management module, a security and privacy module, a cloud-edge collaboration module, and an application layer interface module;

[0008] Edge computing module: performs computing tasks on edge nodes close to data sources to reduce data transmission delays;

[0009] Distributed storage module: provides data storage and management capabilities across multiple nodes, supporting high data availability and scalability;

[0010] Task scheduling module: dynamically assigns tasks to appropriate computing nodes based on task requirements, resource status, and network conditions to optimize overall system performance;

[0011] Network communication module: responsible for data transmission and communication between edge nodes, storage nodes and the cloud;

[0012] Monitoring and management module: real-time monitoring of system operation status, collection of performance indicators, support for fault diagnosis and system optimization;

[0013] Security and privacy module: ensures the security of system data and computing processes, and prevents data leakage and malicious attacks;

[0014] Cloud-edge collaboration module: This module enables efficient collaboration between edge nodes and the cloud, supporting elastic expansion and fault-tolerant processing of tasks.

[0015] Application layer interface module: provides a unified interface for upper-layer applications, simplifies the development process, and supports rapid integration.

[0016] Preferably, the edge computing module includes localized computing execution, task scheduling and load balancing, data caching and preprocessing, security and privacy protection, as well as hybrid computing task scheduling strategies and the collaboration of distributed storage and edge computing.

[0017] Through the above technical solutions, localized computing execution: data is processed directly on the edge node, reducing the delay in data transmission to the cloud;

[0018] Task scheduling and load balancing: Dynamically allocate computing tasks based on the resource status and task priority of edge nodes to avoid node overload;

[0019] Data caching and preprocessing: Cache frequently used data at edge nodes and preprocess raw data to reduce the amount of data transmitted to the cloud;

[0020] Fault tolerance and recovery mechanism: ensures that when an edge node fails, tasks can be quickly migrated to other available nodes to ensure system reliability;

[0021] Security and privacy protection: Sensitive data processing is performed on edge nodes to avoid data leakage risks while supporting encrypted transmission and access control;

[0022] Hybrid computing task scheduling strategies are divided into task hierarchical scheduling, resource-aware scheduling, and data dependency management;

[0023] Collaboration between distributed storage and edge computing: Data is sharded and stored in multiple edge nodes and the cloud, hot data is cached at the edge nodes to reduce access to the cloud, and distributed transactions or eventual consistency models are used to ensure data consistency between edge nodes and the cloud.

[0024] Preferably, the distributed storage module includes data sharding and redundant storage, data caching and prefetching, data consistency and transaction support, fault tolerance and recovery mechanism, security and privacy protection, as well as hybrid storage architecture design and collaboration between distributed storage and edge computing.

[0025] Through the above technical solutions, data sharding and redundant storage are achieved: data shards are stored in multiple edge nodes and the cloud, and replication or erasure coding technology is used to ensure data reliability;

[0026] Data caching and prefetching: Cache hot data at edge nodes to reduce access latency to the cloud. Based on task execution history and prediction models, prefetch relevant data to edge nodes in advance.

[0027] Data consistency and transaction support: Provides strong consistency and eventual consistency models to meet the data consistency requirements of different tasks and ensure the atomicity of cross-node data operations;

[0028] Fault tolerance and recovery mechanism: Real-time monitoring of storage node status, automatic detection and repair of faulty nodes, support for data reconstruction and migration, and ensure system availability in the event of node failure;

[0029] Security and privacy protection: Data is encrypted and stored to prevent data leakage, and access control and audit logs are supported to record data access behavior;

[0030] Hybrid storage architecture design: Tiered storage, automatically migrating data to the appropriate storage tier based on data access frequency and timeliness;

[0031] Collaboration between distributed storage and edge computing: Dynamically adjust data storage locations based on task requirements, support task-level data locality optimization, reduce data transmission overhead, consider data storage locations when scheduling computing tasks, prioritize assigning tasks to nodes where the data is located, support coordinated scheduling of data prefetching and computing tasks, and reduce task execution latency.

[0032] Preferably, the task scheduling module includes task classification and priority management, resource-aware scheduling, data locality optimization, fault tolerance and recovery, energy consumption optimization, and scheduling algorithm design and coordination of distributed storage and task scheduling.

[0033] Through the above technical solutions, we can achieve classification and priority management: set task priority according to task type and user needs, support dynamic priority adjustment, and ensure that high-priority tasks are executed first;

[0034] Resource-aware scheduling: Real-time monitoring of computing resource usage at edge nodes and in the cloud. Dynamically assigning tasks to nodes with sufficient resources based on resource status to avoid overload.

[0035] Data locality optimization: Combined with the data distribution information of the distributed storage module, tasks are assigned to the nodes where the data is located first, reducing data transmission delays. The data pre-fetch mechanism is supported to load the data required for the task into the edge node cache in advance.

[0036] Fault tolerance and recovery: monitors task execution status, detects node failures or task failures, and automatically reschedules failed tasks to other available nodes to ensure task completion.

[0037] Energy consumption optimization: Considering the energy consumption constraints of edge nodes, prioritize scheduling to low-power nodes or enable energy-saving mode;

[0038] Scheduling algorithm design: Using deep reinforcement learning models, we dynamically learn the optimal scheduling strategy based on historical scheduling data and real-time environment status. We combine multiple scheduling algorithms to dynamically select the optimal scheduling strategy based on task characteristics and system status.

[0039] Collaboration between distributed storage and task scheduling: When scheduling tasks, both the data storage location and computing resource distribution are considered. Based on the task execution history and prediction model, data is pre-fetched to the edge node cache in advance.

[0040] Preferably, the network communication module includes task and data transmission, status synchronization and monitoring, security and authentication, fault tolerance and retransmission mechanism, as well as communication protocol selection and communication architecture design.

[0041] Through the above technical solutions, task and data transmission: support the real-time transmission of task instructions, calculation results and intermediate data, optimize the data transmission path, reduce delay and bandwidth usage;

[0042] Status synchronization and monitoring: Real-time synchronization of status information of edge nodes, cloud and storage systems, supporting node health monitoring and fault warning;

[0043] Security and authentication: Provides data encryption, identity authentication, and access control to prevent data leakage and malicious attacks;

[0044] Fault tolerance and retransmission mechanism: Automatically detects network failures, supports data retransmission and task migration, and ensures communication reliability.

[0045] Preferably, the monitoring and management module includes real-time monitoring, fault detection and alarm, automated operation and maintenance, visual management and operation, as well as monitoring data collection and data analysis and alarm.

[0046] Through the above technical solutions, real-time monitoring is divided into resource monitoring, task monitoring, and network monitoring;

[0047] Fault detection and alarm: Fault types are divided into hardware faults, software faults, and network faults. When a fault occurs, the administrator is notified via email, SMS, or message queues.

[0048] Automated operation and maintenance: Dynamically allocate edge nodes and cloud resources based on task requirements, support load balancing to avoid resource overload, and regularly back up key data to support rapid recovery;

[0049] Visual management and operation: Real-time display of system topology, resource usage, and task status, providing historical data query and report generation functions.

[0050] Preferably, the security and privacy module includes identity authentication and access control, data encryption and privacy protection, security auditing and log management, threat detection and response, as well as encryption technology and privacy protection technology.

[0051] Through the above technical solutions, identity authentication and access control: multi-factor authentication is used between edge nodes, users and cloud services to limit user access rights to data and tasks, dynamically adjust permissions based on task requirements, and automatically revoke permissions after the task is completed;

[0052] Data encryption and privacy protection: Encrypt data in transit and desensitize sensitive information before sharing;

[0053] Security auditing and log management: Records all user and system operation logs, including task scheduling, data access, and permission changes. Log data is encrypted and stored to prevent tampering and leakage. Machine learning algorithms are used to analyze logs and detect abnormal behavior.

[0054] Threat detection and response: Use intrusion detection systems and intrusion prevention systems to detect network attacks and malicious behavior. When threats are detected, they automatically isolate infected nodes, block malicious traffic, or notify administrators.

[0055] Preferably, the cloud-edge collaboration module includes resource collaboration, data collaboration, intelligent collaboration, application management collaboration, as well as resource scheduling algorithms and data transmission optimization.

[0056] Through the above technical solutions, resource collaboration is achieved: when edge node resources are insufficient, computing tasks are offloaded to the cloud through cloud-edge collaboration, or idle resources of other edge nodes are utilized. When edge node storage is insufficient, some data is migrated to the cloud or distributed storage system to achieve tiered storage of hot and cold data. The data transmission path is dynamically adjusted according to network conditions, giving priority to low-latency, high-bandwidth network links.

[0057] Data collaboration: Ensures data consistency between edge nodes and the cloud, supports incremental and full synchronization, and caches cloud data to edge nodes in advance based on task requirements to reduce data access latency. Data storage, archiving, and deletion policies are managed separately in the cloud and on edge nodes.

[0058] Intelligent collaboration: Conduct large-scale model training in the cloud, distribute the trained models to edge nodes for real-time inference, and train the models locally on the edge nodes, uploading only the model parameters to the cloud for aggregation to protect data privacy.

[0059] Application management collaboration: Dynamically deploy application services on the cloud or edge nodes based on task requirements, and use containerization technology to achieve unified application orchestration and scheduling between the cloud and the edge. When an edge node fails, tasks are automatically switched to the cloud or other edge nodes.

[0060] Resource scheduling algorithm: Based on the task's sensitivity to latency, it prioritizes allocation to low-latency edge nodes or the cloud. It optimizes the task allocation strategy by combining the computing costs of edge nodes and the cloud. When edge node resources are limited, it reduces energy consumption by offloading to the cloud.

[0061] Data transmission optimization: compress and fragment data before transmission to reduce transmission bandwidth usage, deploy cache services at edge nodes to cache hot data and reduce cloud access, and encrypt data during transmission to ensure data security.

[0062] Preferably, the application layer interface module includes a task scheduling interface, a data management interface, a resource monitoring interface, an intelligent decision-making interface, and an interface gateway design and an interface abstraction layer.

[0063] Through the above technical solution, the task scheduling interface supports users to submit computing tasks, specify task types, resource requirements, and data dependencies, and query task execution status in real time. After the task is completed, the computing results or storage path can be obtained through the interface.

[0064] Data management interface: supports uploading local data to distributed storage systems, downloading data from edge nodes and the cloud, setting data storage policies, and restricting data access permissions based on roles or policies;

[0065] Resource monitoring interface: obtains the resource usage of edge nodes and cloud servers in real time. When resource usage exceeds the threshold, an alarm message is pushed through the interface.

[0066] Intelligent decision-making interface: Train the model locally on the edge node and only upload the model parameters to the cloud for aggregation.

[0067] Preferably, the method for scheduling edge cloud and distributed storage hybrid computing tasks includes the following steps:

[0068] Step 1: Extract the computational complexity, data dependency, latency sensitivity, and data locality properties of the task and complete the modeling;

[0069] Step 2: Collect the CPU / GPU utilization, memory usage, network bandwidth, and remaining storage space of edge devices in real time, and track the I / O throughput, data replica distribution, and storage node load of the storage cluster;

[0070] Step 3: Schedule tasks with strong data locality to edge nodes. Dynamically allocate tasks based on the real-time load of edge nodes and storage nodes, giving priority to tasks with no dependencies or whose dependencies are already satisfied.

[0071] Step 4: Use the scheduling algorithm to assign tasks to appropriate resource nodes, and use technologies such as data compression, fragmented transmission, and edge caching to optimize data transmission efficiency;

[0072] Step 5: After the task is completed, obtain the calculation results or storage path from the execution node, analyze indicators such as task execution time, resource utilization, and data transmission delay, and evaluate the scheduling effect.

[0073] Compared with the existing technology, the present invention provides a method and system for scheduling edge cloud and distributed storage hybrid computing tasks, which has the following beneficial effects:

[0074] 1. The hybrid computing task scheduling system of edge cloud and distributed storage in the present invention optimizes computing and storage resources in a coordinated manner, sinks computing tasks to edge nodes close to data sources, reduces the time for data transmission to the cloud, stores data in fragments on multiple edge nodes, supports parallel reading and writing, avoids single-point bottlenecks, improves real-time response capabilities, avoids decision lags caused by network delays, and thus improves the efficiency of resource scheduling.

[0075] 2. In the present invention, data is sharded and backed up among multiple nodes to avoid single point failure and ensure data availability. When an edge node or storage node fails, the system automatically migrates tasks to other available nodes to ensure service continuity.

[0076] 3. In this invention, sensitive data is processed and stored at the edge node, reducing the risk of uploading data to the cloud and the risk of data leakage. Data is fully encrypted during transmission and storage to prevent man-in-the-middle attacks and improve the overall security of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 Schematic diagram of the system flow of the present invention;

[0078] Figure 2 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0079] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0080] See also Figure 1-Figure 2 ,Edge cloud and distributed storage hybrid computing task scheduling system, including edge computing module, distributed storage module, task scheduling module, network communication module, monitoring and management module, security and privacy module, cloud-edge collaboration module, and application layer interface module;

[0081] Edge computing module: performs computing tasks on edge nodes close to data sources to reduce data transmission delays;

[0082] Distributed storage module: provides data storage and management capabilities across multiple nodes, supporting high data availability and scalability;

[0083] Task scheduling module: dynamically assigns tasks to appropriate computing nodes based on task requirements, resource status, and network conditions to optimize overall system performance;

[0084] Network communication module: responsible for data transmission and communication between edge nodes, storage nodes and the cloud;

[0085] Monitoring and management module: real-time monitoring of system operation status, collection of performance indicators, support for fault diagnosis and system optimization;

[0086] Security and privacy module: ensures the security of system data and computing processes, and prevents data leakage and malicious attacks;

[0087] Cloud-edge collaboration module: This module enables efficient collaboration between edge nodes and the cloud, supporting elastic expansion and fault-tolerant processing of tasks.

[0088] Application layer interface module: provides a unified interface for upper-layer applications, simplifies the development process, and supports rapid integration.

[0089] Specifically, such as Figure 1 As shown, the edge computing module includes localized computing execution, task scheduling and load balancing, data caching and preprocessing, security and privacy protection, as well as hybrid computing task scheduling strategies and the collaboration of distributed storage and edge computing.

[0090] Through the above technical solutions, localized computing execution: data is processed directly on the edge node, reducing the delay in data transmission to the cloud;

[0091] Task scheduling and load balancing: Dynamically allocate computing tasks based on the resource status and task priority of edge nodes to avoid node overload;

[0092] Data caching and preprocessing: Cache frequently used data at edge nodes and preprocess raw data to reduce the amount of data transmitted to the cloud;

[0093] Fault tolerance and recovery mechanism: ensures that when an edge node fails, tasks can be quickly migrated to other available nodes to ensure system reliability;

[0094] Security and privacy protection: Sensitive data processing is performed on edge nodes to avoid data leakage risks while supporting encrypted transmission and access control;

[0095] Hybrid computing task scheduling strategies are divided into task hierarchical scheduling, resource-aware scheduling, and data dependency management;

[0096] Collaboration between distributed storage and edge computing: Data is sharded and stored in multiple edge nodes and the cloud, hot data is cached at the edge nodes to reduce access to the cloud, and distributed transactions or eventual consistency models are used to ensure data consistency between edge nodes and the cloud.

[0097] Specifically, such as Figure 1 As shown, the distributed storage module includes data sharding and redundant storage, data caching and prefetching, data consistency and transaction support, fault tolerance and recovery mechanism, security and privacy protection, as well as hybrid storage architecture design and collaboration between distributed storage and edge computing.

[0098] Through the above technical solutions, data sharding and redundant storage are achieved: data shards are stored in multiple edge nodes and the cloud, and replication or erasure coding technology is used to ensure data reliability;

[0099] Data caching and prefetching: Cache hot data at edge nodes to reduce access latency to the cloud. Based on task execution history and prediction models, prefetch relevant data to edge nodes in advance.

[0100] Data consistency and transaction support: Provides strong consistency and eventual consistency models to meet the data consistency requirements of different tasks and ensure the atomicity of cross-node data operations;

[0101] Fault tolerance and recovery mechanism: Real-time monitoring of storage node status, automatic detection and repair of faulty nodes, support for data reconstruction and migration, and ensure system availability in the event of node failure;

[0102] Security and privacy protection: Data is encrypted and stored to prevent data leakage, and access control and audit logs are supported to record data access behavior;

[0103] Hybrid storage architecture design: Tiered storage, automatically migrating data to the appropriate storage tier based on data access frequency and timeliness;

[0104] Collaboration between distributed storage and edge computing: Dynamically adjust data storage locations based on task requirements, support task-level data locality optimization, reduce data transmission overhead, consider data storage locations when scheduling computing tasks, prioritize assigning tasks to nodes where the data is located, support coordinated scheduling of data prefetching and computing tasks, and reduce task execution latency.

[0105] Specifically, such as Figure 1 As shown, the task scheduling module includes task classification and priority management, resource-aware scheduling, data locality optimization, fault tolerance and recovery, energy consumption optimization, as well as scheduling algorithm design and coordination of distributed storage and task scheduling.

[0106] Through the above technical solutions, we can achieve classification and priority management: set task priority according to task type and user needs, support dynamic priority adjustment, and ensure that high-priority tasks are executed first;

[0107] Resource-aware scheduling: Real-time monitoring of computing resource usage at edge nodes and in the cloud. Dynamically assigning tasks to nodes with sufficient resources based on resource status to avoid overload.

[0108] Data locality optimization: Combined with the data distribution information of the distributed storage module, tasks are assigned to the nodes where the data is located first, reducing data transmission delays. The data pre-fetch mechanism is supported to load the data required for the task into the edge node cache in advance.

[0109] Fault tolerance and recovery: monitors task execution status, detects node failures or task failures, and automatically reschedules failed tasks to other available nodes to ensure task completion.

[0110] Energy consumption optimization: Considering the energy consumption constraints of edge nodes, prioritize scheduling to low-power nodes or enable energy-saving mode;

[0111] Scheduling algorithm design: Using deep reinforcement learning models, we dynamically learn the optimal scheduling strategy based on historical scheduling data and real-time environment status. We combine multiple scheduling algorithms to dynamically select the optimal scheduling strategy based on task characteristics and system status.

[0112] Collaboration between distributed storage and task scheduling: When scheduling tasks, both the data storage location and computing resource distribution are considered. Based on the task execution history and prediction model, data is pre-fetched to the edge node cache in advance.

[0113] Specifically, such as Figure 1As shown, the network communication module includes task and data transmission, state synchronization and monitoring, security and authentication, fault tolerance and retransmission mechanism, as well as communication protocol selection and communication architecture design.

[0114] Through the above technical solutions, task and data transmission: support the real-time transmission of task instructions, calculation results and intermediate data, optimize the data transmission path, reduce delay and bandwidth usage;

[0115] Status synchronization and monitoring: Real-time synchronization of status information of edge nodes, cloud and storage systems, supporting node health monitoring and fault warning;

[0116] Security and authentication: Provides data encryption, identity authentication, and access control to prevent data leakage and malicious attacks;

[0117] Fault tolerance and retransmission mechanism: Automatically detects network failures, supports data retransmission and task migration, and ensures communication reliability.

[0118] Specifically, such as Figure 1 As shown, the monitoring and management module includes real-time monitoring, fault detection and alarm, automated operation and maintenance, visual management and operation, as well as monitoring data collection and data analysis and alarm.

[0119] Through the above technical solutions, real-time monitoring is divided into resource monitoring, task monitoring, and network monitoring;

[0120] Fault detection and alarm: Fault types are divided into hardware faults, software faults, and network faults. When a fault occurs, the administrator is notified via email, SMS, or message queues.

[0121] Automated operation and maintenance: Dynamically allocate edge nodes and cloud resources based on task requirements, support load balancing to avoid resource overload, and regularly back up key data to support rapid recovery;

[0122] Visual management and operation: Real-time display of system topology, resource usage, and task status, providing historical data query and report generation functions.

[0123] Specifically, such as Figure 1 As shown, the security and privacy module includes identity authentication and access control, data encryption and privacy protection, security audit and log management, threat detection and response, as well as encryption technology and privacy protection technology.

[0124] Through the above technical solutions, identity authentication and access control: multi-factor authentication is used between edge nodes, users and cloud services to limit user access rights to data and tasks, dynamically adjust permissions based on task requirements, and automatically revoke permissions after the task is completed;

[0125] Data encryption and privacy protection: Encrypt data in transit and desensitize sensitive information before sharing;

[0126] Security auditing and log management: Records all user and system operation logs, including task scheduling, data access, and permission changes. Log data is encrypted and stored to prevent tampering and leakage. Machine learning algorithms are used to analyze logs and detect abnormal behavior.

[0127] Threat detection and response: Use intrusion detection systems and intrusion prevention systems to detect network attacks and malicious behavior. When threats are detected, they automatically isolate infected nodes, block malicious traffic, or notify administrators.

[0128] Specifically, such as Figure 1 As shown, the cloud-edge collaboration module includes resource collaboration, data collaboration, intelligent collaboration, application management collaboration, as well as resource scheduling algorithms and data transmission optimization.

[0129] Through the above technical solutions, resource collaboration is achieved: when edge node resources are insufficient, computing tasks are offloaded to the cloud through cloud-edge collaboration, or idle resources of other edge nodes are utilized. When edge node storage is insufficient, some data is migrated to the cloud or distributed storage system to achieve tiered storage of hot and cold data. The data transmission path is dynamically adjusted according to network conditions, giving priority to low-latency, high-bandwidth network links.

[0130] Data collaboration: Ensures data consistency between edge nodes and the cloud, supports incremental and full synchronization, and caches cloud data to edge nodes in advance based on task requirements to reduce data access latency. Data storage, archiving, and deletion policies are managed separately in the cloud and on edge nodes.

[0131] Intelligent collaboration: Conduct large-scale model training in the cloud, distribute the trained models to edge nodes for real-time inference, and train the models locally on the edge nodes, uploading only the model parameters to the cloud for aggregation to protect data privacy.

[0132] Application management collaboration: Dynamically deploy application services on the cloud or edge nodes based on task requirements, and use containerization technology to achieve unified application orchestration and scheduling between the cloud and the edge. When an edge node fails, tasks are automatically switched to the cloud or other edge nodes.

[0133] Resource scheduling algorithm: Based on the task's sensitivity to latency, it prioritizes allocation to low-latency edge nodes or the cloud. It optimizes the task allocation strategy by combining the computing costs of edge nodes and the cloud. When edge node resources are limited, it reduces energy consumption by offloading to the cloud.

[0134] Data transmission optimization: compress and fragment data before transmission to reduce transmission bandwidth usage, deploy cache services at edge nodes to cache hot data and reduce cloud access, and encrypt data during transmission to ensure data security.

[0135] Specifically, such as Figure 1 As shown, the application layer interface module includes a task scheduling interface, a data management interface, a resource monitoring interface, an intelligent decision-making interface, and an interface gateway design and an interface abstraction layer.

[0136] Through the above technical solution, the task scheduling interface supports users to submit computing tasks, specify task types, resource requirements, and data dependencies, and query task execution status in real time. After the task is completed, the computing results or storage path can be obtained through the interface.

[0137] Data management interface: supports uploading local data to distributed storage systems, downloading data from edge nodes and the cloud, setting data storage policies, and restricting data access permissions based on roles or policies;

[0138] Resource monitoring interface: obtains the resource usage of edge nodes and cloud servers in real time. When resource usage exceeds the threshold, an alarm message is pushed through the interface.

[0139] Intelligent decision-making interface: Train the model locally on the edge node and only upload the model parameters to the cloud for aggregation.

[0140] like Figure 2 As shown, the edge cloud and distributed storage hybrid computing task scheduling method includes the following steps:

[0141] Step 1: Extract the computational complexity, data dependency, latency sensitivity, and data locality properties of the task and complete the modeling;

[0142] Step 2: Collect the CPU / GPU utilization, memory usage, network bandwidth, and remaining storage space of edge devices in real time, and track the I / O throughput, data replica distribution, and storage node load of the storage cluster;

[0143] Step 3: Schedule tasks with strong data locality to edge nodes. Dynamically allocate tasks based on the real-time load of edge nodes and storage nodes, giving priority to tasks with no dependencies or whose dependencies are already satisfied.

[0144] Step 4: Use the scheduling algorithm to assign tasks to appropriate resource nodes, and use technologies such as data compression, fragmented transmission, and edge caching to optimize data transmission efficiency;

[0145] Step 5: After the task is completed, obtain the calculation results or storage path from the execution node, analyze indicators such as task execution time, resource utilization, and data transmission delay, and evaluate the scheduling effect.

[0146] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. Edge cloud and distributed storage hybrid computing task scheduling system, characterized by: Including edge computing module, distributed storage module, task scheduling module, network communication module, monitoring and management module, security and privacy module, cloud-edge collaboration module, and application layer interface module; Edge computing module: performs computing tasks on edge nodes close to data sources to reduce data transmission delays; Distributed storage module: provides data storage and management capabilities across multiple nodes, supporting high data availability and scalability; Task scheduling module: dynamically assigns tasks to appropriate computing nodes based on task requirements, resource status, and network conditions to optimize overall system performance; Network communication module: responsible for data transmission and communication between edge nodes, storage nodes and the cloud; Monitoring and management module: real-time monitoring of system operation status, collection of performance indicators, support for fault diagnosis and system optimization; Security and privacy module: ensures the security of system data and computing processes, and prevents data leakage and malicious attacks; Cloud-edge collaboration module: This module enables efficient collaboration between edge nodes and the cloud, supporting elastic expansion and fault-tolerant processing of tasks. Application layer interface module: provides a unified interface for upper-layer applications, simplifies the development process, and supports rapid integration.

2. The edge cloud and distributed storage hybrid computing task scheduling system according to claim 1 is characterized by: The edge computing module includes localized computing execution, task scheduling and load balancing, data caching and preprocessing, security and privacy protection, as well as hybrid computing task scheduling strategies and the collaboration of distributed storage and edge computing.

3. The edge cloud and distributed storage hybrid computing task scheduling system according to claim 1, characterized in that: The distributed storage module includes data sharding and redundant storage, data caching and prefetching, data consistency and transaction support, fault tolerance and recovery mechanism, security and privacy protection, as well as hybrid storage architecture design and collaboration between distributed storage and edge computing.

4. The edge cloud and distributed storage hybrid computing task scheduling system according to claim 1, characterized in that: The task scheduling module includes task classification and priority management, resource-aware scheduling, data locality optimization, fault tolerance and recovery, energy consumption optimization, as well as scheduling algorithm design and coordination of distributed storage and task scheduling.

5. The edge cloud and distributed storage hybrid computing task scheduling system according to claim 1, characterized in that: The network communication module includes task and data transmission, status synchronization and monitoring, security and authentication, fault tolerance and retransmission mechanism, as well as communication protocol selection and communication architecture design.

6. The edge cloud and distributed storage hybrid computing task scheduling system according to claim 1, characterized in that: The monitoring and management module includes real-time monitoring, fault detection and alarm, automated operation and maintenance, visual management and operation, as well as monitoring data collection, data analysis and alarm.

7. The edge cloud and distributed storage hybrid computing task scheduling system according to claim 1, characterized in that: The security and privacy module includes identity authentication and access control, data encryption and privacy protection, security auditing and log management, threat detection and response, as well as encryption technology and privacy protection technology.

8. The edge cloud and distributed storage hybrid computing task scheduling system according to claim 1, characterized in that: The cloud-edge collaboration module includes resource collaboration, data collaboration, intelligent collaboration, application management collaboration, as well as resource scheduling algorithms and data transmission optimization.

9. The edge cloud and distributed storage hybrid computing task scheduling system according to claim 1, characterized in that: The application layer interface module includes a task scheduling interface, a data management interface, a resource monitoring interface, an intelligent decision-making interface, and an interface gateway design and an interface abstraction layer.

10. The edge cloud and distributed storage hybrid computing task scheduling method according to claim 1, characterized in that: The following steps are involved: Step 1: Extract the computational complexity, data dependency, latency sensitivity, and data locality properties of the task and complete the modeling; Step 2: Collect the CPU / GPU utilization, memory usage, network bandwidth, and remaining storage space of edge devices in real time, and track the I / O throughput, data replica distribution, and storage node load of the storage cluster; Step 3: Schedule tasks with strong data locality to edge nodes. Dynamically allocate tasks based on the real-time load of edge nodes and storage nodes, giving priority to tasks with no dependencies or whose dependencies are already satisfied. Step 4: Use the scheduling algorithm to assign tasks to appropriate resource nodes, and use technologies such as data compression, fragmented transmission, and edge caching to optimize data transmission efficiency; Step 5: After the task is completed, obtain the calculation results or storage path from the execution node, analyze indicators such as task execution time, resource utilization, and data transmission delay, and evaluate the scheduling effect.

Images