Efficient edge computing and cloud collaborative data scheduling system

Abstract

The invention provides an efficient edge computing and cloud collaborative data scheduling system, and relates to the field of electric digital data processing, and the efficient edge computing and cloud collaborative data scheduling system comprises an edge access module, an edge computing module, a collaborative scheduling module and a cloud computing module, the edge computing module is used for executing a low-delay computing task and intelligent reasoning at an edge end, the collaborative scheduling module is used for performing scheduling analysis on resources, and the cloud computing module is used for performing large model training and cross-regional strategy optimization; according to the system, a unified mathematical modeling method based on a spiral fluctuation theory is adopted, accurate description of a system state is realized through a spiral fluctuation basic function, and a spiral decreasing multi-dimensional cost evaluation model and a fluctuation resonance inverse migration value algorithm are combined; the key technical problems of inaccurate task cost evaluation, uneven load distribution and the like in traditional edge computing scheduling are solved.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing, and more specifically to a data scheduling system that integrates efficient edge computing and cloud collaboration. Background Technology

[0002] With the rapid development of cloud computing, edge computing, and artificial intelligence technologies, resource scheduling and task management in distributed computing environments have become core technologies for improving system performance and resource utilization. Traditional centralized scheduling methods often suffer from problems such as high scheduling latency, poor load balancing, and insufficient system scalability when facing large-scale, high-concurrency computing tasks. Especially in edge computing scenarios, due to the limited node resources, dynamic changes in network conditions, and diverse task types, traditional scheduling algorithms struggle to achieve optimal resource allocation and task distribution.

[0003] The following three algorithms are commonly used in existing technologies:

[0004] 1. Greedy scheduling based on heuristic / cost model:

[0005] Approach: Calculate the overall cost J for each subtask: J = α·latency + β·bandwidth + γ·energy consumption +

[0006] δ·Failure risk, real-time selection of the optimal execution position.

[0007] Advantages: Simple to implement, quick to make decisions, easy to explain; friendly to sudden traffic surges; low project implementation cost.

[0008] Disadvantages: Weak global optimality; parameters need to be manually adjusted; prone to jitter in highly dynamic or multi-tenant game scenarios.

[0009] 2. Mathematical Programming / Combinatorial Optimization:

[0010] Approach: Model task placement, bandwidth allocation, migration, and SLA constraints as integer linear programming or constrained programming; use a solver or metaheuristic to find an approximate global optimum.

[0011] Advantages: It can consider multiple constraints simultaneously; it can provide near-global optimal solutions and provable bounds.

[0012] Disadvantages: The solution time increases exponentially with the number of tasks / nodes; it requires "scrolling window + warm start + pruning" to achieve real-time results.

[0013] 3. Deep reinforcement learning scheduling:

[0014] Approach: Input the system state into the policy network and output placement / migration actions; train with long-term rewards.

[0015] Advantages: Strong adaptability to nonlinear and time-varying environments; can learn long-term optimal migration / prefetch strategies across time periods; can be fine-tuned online.

[0016] Disadvantages: Requires cost for training and exploration; weak stability and interpretability; may be inferior to rule-based methods in cold start scenarios.

[0017] The following are the main problems in the prior art:

[0018] 1. Insufficient task scheduling accuracy:

[0019] Traditional scheduling algorithms usually use heuristic methods based on greedy strategies, only considering single-dimensional optimization objectives such as shortest job first or round-robin scheduling, making it difficult to achieve global optimality under multi-dimensional constraint conditions. These methods often result in low resource utilization and prolonged task completion time when dealing with task sets with complex dependency relationships.

[0020] 2. Limitations of load balancing algorithms:

[0021] Existing load balancing techniques are mainly based on static weight allocation or simple round-robin mechanisms, lacking real-time perception ability of system dynamic state. In high-load or burst traffic scenarios, hot nodes may be overloaded while other nodes are idle, leading to decreased overall system performance and unstable service quality.

[0022] 3. Insufficient optimization of resource utilization:

[0023] Traditional systems usually use fixed allocation strategies for resource allocation, which cannot be dynamically adjusted according to task characteristics and system state. This static allocation method cannot fully utilize system resources in the face of heterogeneous computing environments and diverse task demands, resulting in waste of computing, storage, and network resources.

[0024] 4. Incomplete real-time guarantee mechanism:

[0025] Existing scheduling systems lack effective priority management and deadline guarantee mechanisms when handling tasks with high real-time requirements. In edge computing scenarios, due to network latency and node processing capacity uncertainty, it is difficult to provide reliable real-time guarantees for critical tasks.

[0026] Now many data scheduling systems have been developed, after a large amount of retrieval and reference, it is found that the existing data scheduling systems have systems such as the system disclosed in CN111506408B, these system methods generally include: 1) in view of the present situation of the increasing number of data and computing tasks under the edge computing environment, real-time monitoring of sensor data and migratable nodes is carried out, the number and capacity of data and migratable nodes are determined; 2) real-time monitoring of computing task requests, bundling data and computing tasks based on dependency relationship; 3) the data-task association items are aggregated to form an overall data-task association set, and the overall data association degree is maximized as the target to optimize it into an optimal migration set; 4) cutting the optimal migration set based on the migratable node capacity, forming a local optimal migration set; 5) obtaining the data and computing tasks in the local optimal migration set, migrating them to the target node with corresponding capacity for execution, and repeating steps 1-4. But the system is relatively simple in task cost evaluation, and cannot obtain the optimal migration scheme. SUMMARY

[0027] The purpose of the present application is to propose a high-efficiency edge computing and cloud collaborative data scheduling system in view of the existing deficiencies.

[0028] The present application adopts the following technical solutions:

[0029] A high-efficiency edge computing and cloud collaborative data scheduling system, comprising an edge access module, an edge computing module, a collaborative scheduling module and a cloud computing module;

[0030] The edge access module is used to send raw data into the system, the edge computing module is used to perform low-delay computing tasks and intelligent inference at the edge, the collaborative scheduling module is used to analyze resource scheduling, and the cloud computing module is used for large model training and cross-regional strategy optimization;

[0031] In the system, the whole process of data is disassembled into a multi-level processing link from edge collection to cloud training, the edge side focuses on low delay and real-time performance, the cloud undertakes high computing power and global optimization, the collaborative scheduling module is located in the center and plays the role of resource coordination and strategy scheduling, thereby realizing edge-cloud integrated collaboration.

[0032] The edge access module comprises a data acquisition unit, a protocol adaptation unit and an edge preprocessing unit, the data acquisition unit is used to receive raw event data from the terminal and perform identification processing, the protocol adaptation unit is used to map different protocols into a unified internal event model, and the edge preprocessing unit is used to perform feature processing on event data at the edge;

[0033] The edge access module is equivalent to a data entry layer, and the data acquisition unit is responsible for multi-source access, such as sensors, terminal devices or industrial controllers, and provides a basis for subsequent tracking and scheduling through identification processing. The protocol adaptation unit solves the protocol differences of heterogeneous devices, realizes the unified access of different manufacturers and different standard devices, and the edge preprocessing unit undertakes preliminary data cleaning and feature extraction, reduces the amount of redundant data transmission, and provides standardized input for edge computing.

[0034] The edge computing module includes a task decomposition unit, a fast cache unit and a local inference unit. The task decomposition unit is used to split a task request into multiple subtasks. The fast cache unit is used to store hotspot data, model segments and checkpoints. The local inference unit is used to run a lightweight model for real-time inference.

[0035] The edge computing module is the "first computing link" of the system. The task decomposition unit can split complex tasks into subtasks that can be executed in parallel according to computing load, data granularity and other indicators, improving concurrent processing capability. The fast cache unit efficiently caches hotspot data and key segments during model execution, reducing repeated requests and delays. The local inference unit deploys lightweight AI models, such as compressed neural networks, for real-time scene inference, meeting millisecond-level response requirements.

[0036] The collaborative scheduling module includes a resource monitoring unit, a task allocation unit and a task migration unit. The resource monitoring unit is used to collect real-time resource indicators and calculate health scores. The task allocation unit is used to evaluate multi-dimensional costs for each subtask to be executed and output placement decisions. The task migration unit is used to set trigger events and migrate tasks.

[0037] The collaborative scheduling module is the "scheduling center" of the entire system. The resource monitoring unit continuously tracks the computing power, bandwidth, storage and other conditions of edge nodes and cloud nodes, and quantifies them in the form of health scores. The task allocation unit generates optimal scheduling decisions based on multi-dimensional indicators such as delay, energy consumption and bandwidth consumption. The task migration unit ensures that tasks can automatically migrate to appropriate nodes when nodes are overloaded, links are congested or policies are updated, improving the stability and fault tolerance of the system.

[0038] The cloud computing module includes a global optimization unit, a large-scale storage unit and a centralized training unit. The global optimization unit is used to aggregate cross-domain operation indicators and historical data and optimize global strategies. The large-scale storage unit is used to set data directories and store corresponding data. The centralized training unit is used to train large models on labeled data and manage model versions.

[0039] The cloud computing module assumes the role of a "global brain". The global optimization unit optimizes the resource scheduling and model allocation as a whole by using massive historical data and cross-regional operation indexes. The large-scale storage unit constructs a data lake and a hierarchical storage architecture to ensure efficient management of historical data, feature data, and model parameters. The centralized training unit is responsible for training high-precision and large-scale models and continuously updating them. At the same time, the model version management is performed to ensure that the edge nodes can dynamically obtain the optimal model.

[0040] Further, the task allocation unit includes a task evaluation processor, an allocation decision processor, and a priority processor. The task evaluation processor is used to calculate the execution cost of a task. The allocation decision processor selects a suitable execution location based on the evaluation results. The priority processor is used to set the priority of a task and ensure that critical tasks are scheduled first.

[0041] Further, the task evaluation processor calculates the spiral fluctuation state value Φ(r, θ, t) of the system according to the following formula:

[0042]

[0043] where r is the spiral radius, θ is the spiral angle, t is the time variable, A0 is the basic amplitude coefficient, λ is the spiral attenuation coefficient, w is the angular frequency, P is the spiral period length, θ0 is the initial phase offset, ε is the resonance coupling strength, and φ is the resonance phase period.

[0044] Further, the task evaluation processor calculates the cost value C(T i , t) of a task according to the following formula:

[0045]

[0046] where w k is the importance weight of the 4 scheduling dimensions, T i represents the i-th task, and Ψ(T i , t) represents the urgency of the i-th task at time t.

[0047] The 4 scheduling dimensions are respectively computing, network, storage, and latency.

[0048] Further, the task migration unit includes a migration detection processor, a state saving processor, and a migration recovery processor. The migration detection processor is used to determine whether task migration is needed. The state saving processor is used to save the execution state and checkpoint of a task before migration. The migration recovery processor is used to restore the task on the target node and continue execution.

[0049] The migration detection processor calculates the inverse migration value V(d j, t):

[0050]

[0051] wherein, d j denotes the jth data item, V(d j ) denotes the base value of the jth data item, RT is a set of related tasks, a is a task feedback coefficient, C max is the maximum task cost;

[0052] When the sum of the inverse migration values of all data items involved in the task is less than the migration threshold, it indicates that the task needs to be migrated.

[0053] The beneficial effects achieved by the present application are:

[0054] The system can effectively improve the key performance indicators such as task completion time, system throughput, load balancing stability, etc. by proposing a spiral wave function to unify the dynamic evolution process of the edge computing system, designing a spiral decreasing multi-dimensional cost evaluation algorithm, and innovating a wave resonance migration value model.

[0055] In order to further understand the features and technical contents of the present application, please refer to the following detailed description and drawings of the present application. However, the provided drawings are only used for reference and explanation, and are not used to limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 It is a schematic diagram of the overall structure framework of the present application;

[0057] Figure 2 It is a schematic diagram of the edge access module of the present application;

[0058] Figure 3 It is a schematic diagram of the edge computing module of the present application;

[0059] Figure 4 It is a schematic diagram of the cooperative scheduling module of the present application;

[0060] Figure 5 It is a schematic diagram of the cloud computing module of the present application;

[0061] Figure 6 It is a comparison chart of the task scheduling performance of the present application and the ordinary system;

[0062] Figure 7 It is a comparison chart of the load balancing degree of the present application and the ordinary system. DETAILED DESCRIPTION

[0063] The present application is described in detail by specific embodiments, and the advantages and effects of the present application can be understood by those skilled in the art from the disclosure. The present application can be implemented or applied by other different embodiments, and various modifications and changes can be made based on different views and applications without departing from the spirit of the present application. In addition, the drawings of the present application are only simple schematic illustrations, not actual size depictions, and the foregoing is declared. The following embodiments will further illustrate the related technical content of the present application, but the disclosed content is not used to limit the protection scope of the present application.

[0064] Embodiment one.

[0065] The present embodiment provides a high-efficiency edge computing and cloud collaborative data scheduling system, which combines Figure 1 , including an edge access module, an edge computing module, a collaborative scheduling module and a cloud computing module;

[0066] The edge access module is used to send raw data into the system, the edge computing module is used to perform low-delay computing tasks and intelligent inference at the edge, the collaborative scheduling module is used to analyze resource scheduling, and the cloud computing module is used to perform large model training and cross-region strategy optimization;

[0067] The edge access module includes a data acquisition unit, a protocol adaptation unit and an edge preprocessing unit, the data acquisition unit is used to receive raw event data from the terminal and perform identification processing, the protocol adaptation unit is used to map different protocols to a unified internal event model, and the edge preprocessing unit is used to perform feature processing on event data at the edge;

[0068] The edge computing module includes a task decomposition unit, a fast cache unit and a local inference unit, the task decomposition unit is used to split a task request into multiple subtasks, the fast cache unit is used to store hot data, model fragments and checkpoints, and the local inference unit is used to run a lightweight model for real-time inference;

[0069] The collaborative scheduling module includes a resource monitoring unit, a task allocation unit and a task migration unit, the resource monitoring unit is used to collect real-time resource indicators and calculate a health score, the task allocation unit is used to evaluate multi-dimensional costs for each to-be-executed subtask and output a placement decision, and the task migration unit is used to set a trigger event and perform task migration;

[0070] The cloud computing module comprises a global optimization unit, a large-scale storage unit and a centralized training unit, the global optimization unit is used for aggregating cross-domain operation indexes and historical data and optimizing global strategies, the large-scale storage unit is used for setting data directories and storing corresponding data, and the centralized training unit is used for large model training on labeled data and management of model versions.

[0071] The task allocation unit comprises a task evaluation processor, an allocation decision processor and a priority processor, the task evaluation processor is used for calculating the execution cost of a task, the allocation decision processor is used for selecting a suitable execution location based on the evaluation result, and the priority processor is used for setting the priority of a task and ensuring that a critical task is scheduled in priority.

[0072] The task evaluation processor calculates the spiral fluctuation state value Φ(r, θ, t) of the system according to the following formula:

[0073]

[0074] Wherein, r is the spiral radius, θ is the spiral angle, t is the time variable, A0 is the basic amplitude coefficient, λ is the spiral attenuation coefficient, w is the angular frequency, P is the spiral period length, θ0 is the initial phase offset, ε is the resonance coupling strength, and φ is the resonance phase period.

[0075] The task evaluation processor calculates the cost value C(T i , t) of the task according to the following formula:

[0076]

[0077] Wherein, w k is the importance weight of 4 scheduling dimensions, T i represents the i-th task, and Ψ(T i , t) represents the urgency of the i-th task at time t.

[0078] The 4 scheduling dimensions are respectively calculation, network, storage and time delay.

[0079] The task migration unit comprises a migration detection processor, a state saving processor and a migration recovery processor, the migration detection processor is used for judging whether task migration is needed, the state saving processor is used for saving the execution state and checkpoint of the task before migration, and the migration recovery processor is used for recovering the task at the target node and continuing execution;

[0080] The migration detection processor calculates the inverse migration value V(d j , t) of the data item involved in the task according to the following formula:

[0081]

[0082] wherein, d j denotes the jth data item, V(d j ) denotes the base value of the jth data item, RT is the set of related tasks, a is the task feedback coefficient, C max is the maximum task cost;

[0083] When the sum of the inverse migration values of all data items involved in the task is less than the migration threshold, it indicates that the task needs to be migrated.

[0084] Embodiment Two.

[0085] This embodiment includes all the contents of Embodiment One, and provides a high-efficiency edge computing and cloud collaboration data scheduling system, including an edge access module, an edge computing module, a collaborative scheduling module, and a cloud computing module.

[0086] The edge access module is used to send raw data into the system, the edge computing module is used to perform low-latency computing tasks and intelligent inference at the edge, the collaborative scheduling module is used to analyze resource scheduling, and the cloud computing module is used for large model training and cross-regional strategy optimization.

[0087] In combination with Figure 2 , the edge access module includes a data acquisition unit, a protocol adaptation unit, and an edge preprocessing unit, the data acquisition unit is used to receive raw event data from the terminal and perform identification processing, the protocol adaptation unit is used to map different protocols into a unified internal event model, and the edge preprocessing unit is used to perform feature processing on event data at the edge.

[0088] In combination with Figure 3 , the edge computing module includes a task decomposition unit, a fast cache unit, and a local inference unit, the task decomposition unit is used to split the task request into multiple sub-tasks, the fast cache unit is used to store hot data, model fragments, and checkpoints, and the local inference unit is used to run a lightweight model for real-time inference.

[0089] In combination with Figure 4 , the collaborative scheduling module includes a resource monitoring unit, a task allocation unit, and a task migration unit, the resource monitoring unit is used to collect real-time resource indicators and calculate a health score, the task allocation unit is used to evaluate multi-dimensional costs for each to-be-executed sub-task and output placement decisions, and the task migration unit is used to set trigger events and migrate tasks.

[0090] In combination with Figure 5The cloud computing module comprises a global optimization unit, a large-scale storage unit and a centralized training unit, the global optimization unit is used for aggregating cross-domain operation indexes and historical data and optimizing a global strategy, the large-scale storage unit is used for setting a data directory and storing corresponding data, and the centralized training unit is used for large model training on labeled data and management of model versions;

[0091] The data collection unit comprises a data receiving processor, a timestamp processor and an integrity verification processor, the data receiving processor is used for receiving original data messages from terminals, the timestamp processor is used for attaching a unified system timestamp when data is received, and the integrity verification processor is used for signature verification on received data packets;

[0092] The protocol adaptation unit comprises a protocol parsing processor, a format conversion processor and a compatibility mapping processor, the protocol parsing processor is used for decoding data of different protocols, the format conversion processor is used for uniformly converting the decoded data into an internal standard format, and the compatibility mapping processor is used for establishing a field mapping table for a proprietary protocol to ensure system compatibility;

[0093] The edge preprocessing unit comprises a data filtering processor, a data compression processor and a feature extraction processor, the data filtering processor is used for removing redundant data and invalid data, the data compression processor is used for lossless compression of large-volume data streams, and the feature extraction processor is used for extracting basic statistical features and used for feature vector representation;

[0094] The task decomposition unit comprises a task parsing processor, a task segmentation processor and a dependency relationship processor, the task parsing processor is used for parsing task descriptions to identify task types and computing requirements, the task segmentation processor is used for splitting a complex task into multiple subtasks, and the dependency relationship processor is used for recording the dependency order between subtasks;

[0095] The fast cache unit comprises a memory cache processor, a disk cache processor and a cache eviction processor, the memory cache processor is used for temporarily storing high-frequency access data in memory, the disk cache processor is used for storing large-scale data on disk, and the cache eviction processor is used for dynamically cleaning low-value caches;

[0096] The local inference unit comprises a model loading processor, an inference calculation processor and a result verification processor, the model loading processor is used for loading lightweight models that can be run on edge devices, the inference calculation processor performs inference calculation of tasks based on the model, and the result verification processor is used for confidence detection on inference results;

[0097] The resource monitoring unit comprises a performance detection processor, a network monitoring processor and an energy consumption monitoring processor, the performance detection processor is used for collecting the usage of computing resources such as CPU, GPU and memory, the network monitoring processor is used for monitoring bandwidth, time delay and packet loss rate, and the energy consumption monitoring processor is used for monitoring device energy consumption and temperature state;

[0098] The task allocation unit comprises a task evaluation processor, an allocation decision processor and a priority processor, the task evaluation processor is used for calculating the execution cost of a task, the allocation decision processor selects a suitable execution location based on the evaluation result, and the priority processor is used for setting priority for a task and ensuring that a critical task is scheduled in priority;

[0099] The task evaluation processor calculates the spiral fluctuation state value Φ(r, θ, t) of the system according to the following formula:

[0100]

[0101] Wherein, r is the spiral radius, θ is the spiral angle, t is the time variable, A0 is the basic amplitude coefficient, λ is the spiral attenuation coefficient, w is the angular frequency, P is the spiral period length, θ0 is the initial phase offset, ε is the resonance coupling strength, and φ is the resonance phase period.

[0102] The spiral radius refers to the degree of deviation of the current load state of the edge node from the optimal working point;

[0103] The spiral angle refers to the working cycle phase in which the edge computing system is currently located;

[0104] The basic amplitude coefficient refers to the basic scheduling strength of the edge computing system;

[0105] The spiral attenuation coefficient refers to the attenuation speed of the scheduling influence with the distance from the core node;

[0106] The angular frequency refers to the basic frequency of the load change of the edge computing system;

[0107] The spiral period length refers to the complete scheduling period in which the edge node returns from high load to low load;

[0108] The initial phase offset refers to the time difference in the start of the load cycle of different edge nodes;

[0109] The resonance coupling strength refers to the strength coefficient of the mutual influence of the loads between the edge nodes;

[0110] The resonance phase period refers to the phase period of the load resonance between the edge nodes;

[0111] The task evaluation processor calculates the cost value C(T i , t) of the task according to the following formula:

[0112]

[0113] wherein w k is the importance weight of the 4 scheduling dimensions, T i represents the i-th task, and Ψ(T i , t) represents the urgency of the i-th task at time t.

[0114] The 4 scheduling dimensions are respectively computation, network, storage and latency.

[0115] The task migration unit comprises a migration detection processor, a state saving processor and a migration recovery processor, the migration detection processor is configured to determine whether task migration is needed, the state saving processor is configured to save the execution state and checkpoint of the task before migration, and the migration recovery processor is configured to recover the task at the target node and continue execution.

[0116] The migration detection processor calculates the inverse migration value V(d j , t) of the data item involved by the task according to the following formula:

[0117]

[0118] wherein d j is the j-th data item, V(d j ) is the basic value of the j-th data item, RT is the related task set, a is the task feedback coefficient, and C max is the maximum task cost.

[0119] When the sum of the inverse migration values of all data items involved by the task is less than the migration threshold, it indicates that the task needs to be migrated.

[0120] The global optimization unit comprises a resource summary processor, an optimization calculation processor and a strategy issuing processor, the resource summary processor is configured to summarize the running state and load condition of the edge nodes, the optimization calculation processor is configured to run a global scheduling algorithm to generate an optimization scheme across regions, and the strategy issuing processor is configured to issue the optimization result to the edge nodes.

[0121] The large-scale storage unit comprises a data storage processor, an index retrieval processor and a backup management processor, the data storage processor is configured to store raw data and intermediate results from the edge, the index retrieval processor is configured to index and quickly retrieve historical data, and the backup management processor is configured to perform version management and hierarchical archiving on data.

[0122] The centralized training unit comprises a training data processor, a model training processor and a model publishing processor, the training data processor is configured to clean and label edge backhaul data to construct a training sample set, the model training processor is configured to perform centralized training and verification of a deep learning model, and the model publishing processor is configured to compress and version manage the trained model and deliver it to the edge node.

[0123] Embodiment three.

[0124] This embodiment includes all the contents of embodiment two, and provides a high-efficiency edge computing and cloud collaboration data scheduling system, comprising an edge access module, an edge computing module, a collaborative scheduling module and a cloud computing module.

[0125] The edge access module is configured to send raw data into the system, the edge computing module is configured to perform low-latency computing tasks and intelligent inference at the edge, the collaborative scheduling module is configured to analyze resource scheduling, and the cloud computing module is configured to perform large model training and cross-region strategy optimization.

[0126] This embodiment adds an edge intelligent analysis module and a dynamic load balancing module to improve the intelligence level and load processing capacity of the system.

[0127] The edge intelligent analysis module is configured to perform deep learning analysis on the data collected by the edge node to realize anomaly detection, trend prediction and intelligent decision support.

[0128] The dynamic load balancing module is configured to monitor system load distribution in real time, dynamically adjust task allocation strategies, and ensure optimal utilization of system resources.

[0129] The edge intelligent analysis module comprises a data preprocessing unit, a feature engineering unit, a model inference unit and a result evaluation unit.

[0130] The data preprocessing unit comprises a data cleaning processor, an outlier detection processor and a data standardization processor, the data cleaning processor is configured to remove noise and error values in the data, the outlier detection processor is configured to identify and process abnormal points in the data, and the data standardization processor is configured to normalize data of different dimensions.

[0131] The feature engineering unit comprises a feature selection processor, a feature transformation processor and a feature construction processor, the feature selection processor is configured to select the most representative feature subset from the original features, the feature transformation processor is configured to perform mathematical transformation on the features to improve model performance, and the feature construction processor is configured to create new derived features by combining existing features.

[0132] The model inference unit includes a lightweight model loader, an inference engine, and a result post-processor, the lightweight model loader is used to load a compressed neural network model suitable for running on edge devices, the inference engine is used to perform real-time inference calculation, and the result post-processor is used to format and confidence evaluation on the inference result;

[0133] The result evaluation unit includes an accuracy evaluation processor, a consistency verification processor, and a reliability scoring processor, the accuracy evaluation processor is used to evaluate the accuracy level of the inference result, the consistency verification processor is used to verify the consistency of the inference result with historical data, and the reliability scoring processor is used to assign a reliability score to the inference result;

[0134] The dynamic load balancing module includes a load monitoring unit, a balancing strategy unit, and an adaptive adjustment unit;

[0135] The load monitoring unit includes a real-time load collector, a load trend analyzer, and a bottleneck identifier, the real-time load collector is used to continuously collect the resource usage of CPU, memory, network, etc. of each node, the load trend analyzer is used to analyze the load change trend and predict the future load level, and the bottleneck identifier is used to automatically identify the system performance bottleneck point;

[0136] The balancing strategy unit includes a strategy selection processor, a weight calculation processor, and a threshold management processor, the strategy selection processor is used to select the most suitable load balancing strategy according to the current system state, the weight calculation processor is used to dynamically calculate the load distribution weight of each node, and the threshold management processor is used to manage the trigger threshold and adjustment parameters of load balancing;

[0137] The adaptive adjustment unit includes an adjustment trigger, a parameter optimizer, and an effect verifier, the adjustment trigger is used to monitor the load imbalance and trigger the adjustment action, the parameter optimizer is used to optimize the adjustment parameters based on historical data and current state, and the effect verifier is used to verify the adjustment effect and feedback optimization;

[0138] The embodiment also adds a fault tolerance recovery module to improve the reliability and fault tolerance capability of the system;

[0139] The fault tolerance recovery module includes a fault detection unit, a fault isolation unit, and an automatic recovery unit;

[0140] The fault detection unit includes a health monitoring processor, an anomaly detection processor, and a fault diagnosis processor, the health monitoring processor is used to continuously monitor the running state of each component, the anomaly detection processor is used to detect system anomalies based on statistical learning methods, and the fault diagnosis processor is used to analyze the fault cause and determine the fault type;

[0141] The fault isolation unit comprises an isolation strategy processor, a resource re-allocation processor and a communication switching processor, the isolation strategy processor is used to formulate an isolation scheme of the fault node, the resource re-allocation processor is used to re-allocate tasks and resources of the fault node to healthy nodes, and the communication switching processor is used to switch a communication path to avoid the fault node;

[0142] The automatic recovery unit comprises a recovery strategy processor, a data synchronization processor and a service restart processor, the recovery strategy processor is used to formulate a recovery plan of the fault node, the data synchronization processor is used to perform data synchronization after the node is recovered, and the service restart processor is used to restart the fault service and verify normal operation thereof;

[0143] The i, j and k appearing in the above are ordinal numbers for representing serial numbers and have no actual meaning.

[0144] Part of the code information of the system is as follows:

[0145]

[0146]

[0147]

[0148] The system and the ordinary system are actually tested, and the effect comparison chart of collected data is obtained Figure 6 and Figure 7

[0149] The disclosed content is only a preferred feasible embodiment of the application, and does not limit the protection scope of the application, so that equivalent technical changes made by applying the content of the application specification and drawings are included in the protection scope of the application, and furthermore, elements can be updated as technology develops.​

Claims

1. A high-efficiency edge computing and cloud collaborative data scheduling system, characterized in that, The edge access module, the edge computing module, the collaborative scheduling module and the cloud computing module are comprised. The edge access module is used for sending raw data into the system, the edge computing module is used for performing low-delay computing tasks and intelligent inference at the edge, the collaborative scheduling module is used for scheduling analysis of resources, and the cloud computing module is used for large model training and cross-region strategy optimization. The edge access module comprises a data acquisition unit, a protocol adaptation unit and an edge preprocessing unit, the data acquisition unit is used for receiving raw event data from a terminal and performing identification processing, the protocol adaptation unit is used for mapping different protocols into a unified internal event model, and the edge preprocessing unit is used for performing feature processing on event data at the edge. The edge computing module comprises a task decomposition unit, a fast cache unit and a local inference unit, the task decomposition unit is used for splitting a task request into multiple subtasks, the fast cache unit is used for storing hot data, model fragments and checkpoints, and the local inference unit is used for running a lightweight model to perform real-time inference. The collaborative scheduling module comprises a resource monitoring unit, a task allocation unit and a task migration unit, the resource monitoring unit is used for collecting real-time resource indicators and calculating a health score, the task allocation unit is used for evaluating multi-dimensional costs of each to-be-executed subtask and outputting placement decisions, and the task migration unit is used for setting trigger events and migrating tasks. The cloud computing module comprises a global optimization unit, a large-scale storage unit and a centralized training unit, the global optimization unit is used for aggregating cross-domain running indicators and historical data and optimizing a global strategy, the large-scale storage unit is used for setting a data directory and storing corresponding data, and the centralized training unit is used for performing large model training on labeled data and managing model versions. 2.The data scheduling system of claim 1, wherein, The task allocation unit comprises a task evaluation processor, an allocation decision processor and a priority processor, the task evaluation processor is used for calculating execution costs of tasks, the allocation decision processor is used for selecting appropriate execution locations based on evaluation results, and the priority processor is used for setting priorities for tasks and ensuring that critical tasks are scheduled first. 3.The data scheduling system of claim 2, wherein, The task evaluation processor calculates a spiral fluctuation state value Φ(r, θ, t) of the system according to the following formula: wherein r is a spiral radius, θ is a spiral angle, t is a time variable, A0 is a basic amplitude coefficient, λ is a spiral attenuation coefficient, w is an angular frequency, P is a spiral period length, θ0 is an initial phase offset, ε is a resonance coupling strength, and φ is a resonance phase period. 4.The data scheduling system of claim 3, wherein, The task evaluation processor calculates the generation cost C(T i , t) of the task according to the following equation: wherein w k are importance weights for the 4 scheduling dimensions, T i represents the ith task, Ψ(T i , t) represents the urgency of the ith task at time t; The four scheduling dimensions are respectively computing, network, storage and latency. 5.The data scheduling system of claim 4, wherein, The task migration unit comprises a migration detection processor, a state saving processor and a migration recovery processor, the migration detection processor is used for judging whether task migration is needed, the state saving processor is used for saving execution states and checkpoints of tasks before migration, and the migration recovery processor is used for resuming tasks at a target node and continuing execution. The migration detection processor calculates the inverse migration value V(d, t) of a task- involved data item according to the following formula: j V(d, t) = -log (1 - P(d, t)) wherein d j denotes the jth data item, V(d j ) denotes the base value of the jth data item, RT is the set of related tasks, a is the task feedback coefficient, C max is the maximum task cost; When the sum of inverse migration values of all data items of a task is less than a migration threshold, it is indicated that the task needs to be migrated.

Images