An efficient edge computing and cloud collaborative data scheduling system

Abstract

The application provides a kind of high-efficiency edge computing and cloud coordination data scheduling system, it is related to electric digital data processing field, including edge access module, edge computing module, coordination scheduling module and cloud computing module, the edge access module is used to send original data into system, the edge computing module is used to execute low-delay computing task and intelligent inference in edge, the coordination scheduling module is used to analyze the scheduling of resources, the cloud computing module is used for large model training and cross-region strategy optimization;The system adopts unified mathematical modeling method based on spiral wave theory, realizes the accurate description of system state through spiral wave basic function, combines spiral decreasing multi-dimensional cost evaluation model and wave resonance inverse migration value algorithm, solves the key technical problems of inaccurate task cost evaluation and uneven load distribution in traditional edge computing scheduling.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing, and more specifically to a data scheduling system that integrates efficient edge computing and cloud collaboration. Background Technology

[0002] With the rapid development of cloud computing, edge computing, and artificial intelligence technologies, resource scheduling and task management in distributed computing environments have become core technologies for improving system performance and resource utilization. Traditional centralized scheduling methods often suffer from problems such as high scheduling latency, poor load balancing, and insufficient system scalability when facing large-scale, high-concurrency computing tasks. Especially in edge computing scenarios, due to the limited node resources, dynamic changes in network conditions, and diverse task types, traditional scheduling algorithms struggle to achieve optimal resource allocation and task distribution.

[0003] The following three algorithms are commonly used in existing technologies:

[0004] 1. Greedy scheduling based on heuristic / cost model:

[0005] Approach: Calculate the overall cost J for each subtask: J = α·latency + β·bandwidth + γ·energy consumption +

[0006] δ·Failure risk, real-time selection of the optimal execution position.

[0007] Advantages: Simple to implement, quick to make decisions, easy to explain; friendly to sudden traffic surges; low project implementation cost.

[0008] Disadvantages: Weak global optimality; parameters need to be manually adjusted; prone to jitter in highly dynamic or multi-tenant game scenarios.

[0009] 2. Mathematical Programming / Combinatorial Optimization:

[0010] Approach: Model task placement, bandwidth allocation, migration, and SLA constraints as integer linear programming or constrained programming; use a solver or metaheuristic to find an approximate global optimum.

[0011] Advantages: It can consider multiple constraints simultaneously; it can provide near-global optimal solutions and provable bounds.

[0012] Disadvantages: The solution time increases exponentially with the number of tasks / nodes; it requires "scrolling window + warm start + pruning" to achieve real-time results.

[0013] 3. Deep reinforcement learning scheduling:

[0014] Approach: Input the system state into the policy network and output placement / migration actions; train with long-term rewards.

[0015] Advantages: Strong adaptability to nonlinear and time-varying environments; able to learn long-term optimal migration / prefetch strategies across time periods; can be fine-tuned online.

[0016] Disadvantages: Training and exploration require resources; stability and interpretability are relatively weak; it may be inferior to rule-based methods during cold starts.

[0017] The existing technology mainly has the following key problems:

[0018] 1. Insufficient task scheduling precision:

[0019] Traditional scheduling algorithms typically employ heuristic methods based on greedy strategies, considering only a single-dimensional optimization objective, such as shortest job first or round-robin scheduling. This makes it difficult to achieve global optimum under multi-dimensional constraints. When dealing with task sets with complex dependencies, these methods often result in low resource utilization and prolonged task completion times.

[0020] 2. Limitations of load balancing algorithms:

[0021] Existing load balancing technologies are mainly based on static weight allocation or simple round-robin mechanisms, lacking the ability to perceive the dynamic status of the system in real time. Under high load or sudden traffic scenarios, hot nodes are prone to overload while other nodes are idle, resulting in a decline in overall system performance and unstable service quality.

[0022] 3. Insufficient optimization of resource utilization:

[0023] Traditional systems typically employ fixed allocation strategies when allocating resources, failing to dynamically adjust based on task characteristics and system status. This static allocation approach struggles to fully utilize system resources in heterogeneous computing environments and diverse task requirements, resulting in a waste of computing, storage, and network resources.

[0024] 4. Inadequate real-time performance guarantee mechanism:

[0025] Existing scheduling systems lack effective priority management and deadline guarantee mechanisms when handling tasks with high real-time requirements. This is especially true in edge computing scenarios, where network latency and the uncertainty of node processing capabilities make it difficult to provide reliable real-time guarantees for critical tasks.

[0026] Many data scheduling systems have been developed. Extensive research and reference have revealed existing systems such as the one disclosed in publication number CN111506408B. These systems generally include the following methods: 1) Real-time monitoring of sensor data and migrateable nodes to determine the amount of data and the number and capacity of migrateable nodes, given the increasing volume of data and computing tasks in edge computing environments; 2) Real-time monitoring of computing task requests, binding data and computing tasks based on dependencies; 3) Aggregating data-task associations into an overall data-task association set, optimizing it into the optimal migration set with the goal of maximizing overall data association; 4) Dividing the optimal migration set into locally optimal migration sets based on the capacity of migrateable nodes; 5) Obtaining the data and computing tasks from the locally optimal migration sets, migrating them to target nodes with corresponding capacity, and repeating steps 1-4. However, this system's assessment of task costs is relatively simplistic and cannot yield the optimal migration solution. Summary of the Invention

[0027] The purpose of this invention is to address the shortcomings by proposing a high-efficiency data scheduling system that combines edge computing and cloud collaboration.

[0028] The present invention adopts the following technical solution:

[0029] A high-efficiency edge computing and cloud-coordinated data scheduling system includes an edge access module, an edge computing module, a coordinated scheduling module, and a cloud computing module;

[0030] The edge access module is used to send raw data into the system, the edge computing module is used to perform low-latency computing tasks and intelligent inference at the edge, the collaborative scheduling module is used to schedule and analyze resources, and the cloud computing module is used to perform large model training and cross-regional strategy optimization.

[0031] In this system, the entire data flow is broken down into a multi-level processing link from edge acquisition to cloud training. The edge side focuses on low latency and real-time performance, while the cloud side undertakes high computing power and global optimization. The collaborative scheduling module is located in the center and plays the role of resource coordination and strategy scheduling, thereby realizing edge-cloud integrated collaboration.

[0032] The edge access module includes a data acquisition unit, a protocol adaptation unit, and an edge preprocessing unit. The data acquisition unit is used to receive raw event data from the terminal and perform identification processing. The protocol adaptation unit is used to map different protocols to a unified internal event model. The edge preprocessing unit is used to perform feature processing on the event data at the edge.

[0033] The edge access module is equivalent to the data entry layer. The data acquisition unit is responsible for multi-source access, such as sensors, terminal devices or industrial controllers, and provides a basis for subsequent tracking and scheduling through identification processing. The protocol adaptation unit solves the protocol differences of heterogeneous devices and realizes unified access of devices from different manufacturers and with different standards. The edge preprocessing unit undertakes preliminary data cleaning and feature extraction, reduces the amount of redundant data transmission, and provides standardized input for edge computing.

[0034] The edge computing module includes a task decomposition unit, a fast caching unit, and a local inference unit. The task decomposition unit is used to split task requests into multiple sub-tasks. The fast caching unit is used to store hot data, model fragments, and checkpoints. The local inference unit is used to run lightweight models for real-time inference.

[0035] The edge computing module is the "first computing link" of the system. The task decomposition unit can break down complex tasks into parallel subtasks based on indicators such as computing load and data granularity to improve concurrent processing capabilities. The fast caching unit efficiently caches hot data and key segments during model execution to reduce duplicate requests and latency. The local inference unit deploys lightweight AI models, such as compressed neural networks, for real-time scene inference to meet millisecond-level response requirements.

[0036] The collaborative scheduling module includes a resource monitoring unit, a task allocation unit, and a task migration unit. The resource monitoring unit is used to collect real-time resource indicators and calculate health scores. The task allocation unit is used to evaluate the multi-dimensional cost of each subtask to be executed and output placement decisions. The task migration unit is used to set trigger events and migrate tasks.

[0037] The collaborative scheduling module is located in the "scheduling center" of the entire system. The resource monitoring unit continuously tracks the computing power, bandwidth, storage and other conditions of edge nodes and cloud nodes and quantifies them by health scoring. The task allocation unit generates the optimal scheduling decision based on multi-dimensional indicators such as latency, energy consumption and bandwidth consumption. The task migration unit ensures that when a node is overloaded, a link is blocked or the policy is updated, the task can be automatically migrated to a suitable node, thereby improving the stability and fault tolerance of the system.

[0038] The cloud computing module includes a global optimization unit, a large-scale storage unit, and a centralized training unit. The global optimization unit is used to summarize cross-domain operation indicators and historical data and optimize global strategies. The large-scale storage unit is used to set up data directories and store corresponding data. The centralized training unit is used to train large models on labeled data and manage model versions.

[0039] The cloud computing module acts as the "global brain," while the global optimization unit uses massive historical data and cross-regional operating metrics to optimize resource scheduling and model allocation. The large-scale storage unit builds a data lake and hierarchical storage architecture to ensure efficient management of historical data, feature data, and model parameters. The centralized training unit is responsible for training high-precision, large-scale models and continuously iterating and updating them, while also managing model versions to ensure that edge nodes can dynamically acquire the optimal model.

[0040] Furthermore, the task allocation unit includes a task evaluation processor, an allocation decision processor, and a priority processor. The task evaluation processor is used to calculate the execution cost of the task. The allocation decision processor selects a suitable execution location based on the evaluation result. The priority processor is used to set priorities for tasks and ensure that critical tasks are scheduled first.

[0041] Furthermore, the task evaluation processor calculates the spiral fluctuation state value Φ(r, θ, t) of the system according to the following formula:

[0042]

[0043] Where r is the helical radius, θ is the helical angle, t is the time variable, A0 is the basic amplitude coefficient, λ is the helical attenuation coefficient, w is the angular frequency, P is the helical period length, θ0 is the initial phase shift, ε is the resonant coupling strength, and φ is the resonant phase period.

[0044] Furthermore, the task evaluation processor calculates the task's cost C(T) according to the following formula. i ,t):

[0045]

[0046] Among them, w k T represents the importance weights for the four scheduling dimensions. i Let Ψ(T) represent the i-th task. i ,t) represents the urgency of the i-th task at time t;

[0047] The four scheduling dimensions refer to computing, networking, storage, and latency.

[0048] Furthermore, the task migration unit includes a migration detection processor, a state saving processor, and a migration recovery processor. The migration detection processor is used to determine whether task migration is required. The state saving processor is used to save the execution state and checkpoints of the task before migration. The migration recovery processor is used to restore the task on the target node and continue execution.

[0049] The migration detection processor calculates the reverse migration value V(d) of the data items involved in the task according to the following formula. j,t):

[0050]

[0051] Where, d j Refers to the j-th data item, V(d) j ) refers to the basic value of the j-th data item, RT is the set of related tasks, α is the task feedback coefficient, and C max For the maximum cost of the task;

[0052] When the sum of the reverse migration values ​​of all data items involved in a task is less than the migration threshold, it indicates that the task needs to be migrated.

[0053] The beneficial effects achieved by this invention are:

[0054] This system proposes a unified modeling function based on spiral fluctuations to model the dynamic evolution of edge computing systems, designs a spiral decreasing multidimensional cost evaluation algorithm, and innovates a fluctuation resonance migration value model. This can effectively improve key performance indicators such as task completion time, system throughput, and load balancing stability.

[0055] To further understand the features and technical content of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are for reference and illustration only and are not intended to limit the present invention. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the overall structural framework of the present invention;

[0057] Figure 2 This is a schematic diagram of the edge access module of the present invention;

[0058] Figure 3 This is a schematic diagram of the edge computing module of the present invention;

[0059] Figure 4 This is a schematic diagram of the collaborative scheduling module of the present invention;

[0060] Figure 5 This is a schematic diagram of the cloud computing module structure of the present invention;

[0061] Figure 6 This is a comparison chart of the task scheduling performance of the present invention and ordinary systems;

[0062] Figure 7 This is a comparison chart of the load balancing performance of the present invention and a conventional system. Detailed Implementation

[0063] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can understand the advantages and effects of the present invention from the content disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and various details in this specification can also be modified and changed based on different viewpoints and applications without departing from the spirit of the present invention. Furthermore, the accompanying drawings of the present invention are for simple illustrative purposes only and are not depictions of actual dimensions; this is stated beforehand. The following embodiments will further describe the relevant technical content of the present invention in detail, but the disclosed content is not intended to limit the scope of protection of the present invention.

[0064] Example 1.

[0065] This embodiment provides a high-efficiency data scheduling system that combines edge computing and cloud collaboration. Figure 1 It includes an edge access module, an edge computing module, a collaborative scheduling module, and a cloud computing module;

[0066] The edge access module is used to send raw data into the system, the edge computing module is used to perform low-latency computing tasks and intelligent inference at the edge, the collaborative scheduling module is used to schedule and analyze resources, and the cloud computing module is used to perform large model training and cross-regional strategy optimization.

[0067] The edge access module includes a data acquisition unit, a protocol adaptation unit, and an edge preprocessing unit. The data acquisition unit is used to receive raw event data from the terminal and perform identification processing. The protocol adaptation unit is used to map different protocols to a unified internal event model. The edge preprocessing unit is used to perform feature processing on the event data at the edge.

[0068] The edge computing module includes a task decomposition unit, a fast caching unit, and a local inference unit. The task decomposition unit is used to split task requests into multiple sub-tasks. The fast caching unit is used to store hot data, model fragments, and checkpoints. The local inference unit is used to run lightweight models for real-time inference.

[0069] The collaborative scheduling module includes a resource monitoring unit, a task allocation unit, and a task migration unit. The resource monitoring unit is used to collect real-time resource indicators and calculate health scores. The task allocation unit is used to evaluate the multi-dimensional cost of each subtask to be executed and output placement decisions. The task migration unit is used to set trigger events and migrate tasks.

[0070] The cloud computing module includes a global optimization unit, a large-scale storage unit, and a centralized training unit. The global optimization unit is used to summarize cross-domain operation indicators and historical data and optimize global strategies. The large-scale storage unit is used to set up data directories and store corresponding data. The centralized training unit is used to train large models on labeled data and manage model versions.

[0071] The task allocation unit includes a task evaluation processor, an allocation decision processor, and a priority processor. The task evaluation processor is used to calculate the execution cost of the task. The allocation decision processor selects a suitable execution location based on the evaluation result. The priority processor is used to set priorities for tasks and ensure that critical tasks are scheduled first.

[0072] The task evaluation processor calculates the spiral fluctuation state value Φ(r, θ, t) of the system according to the following formula:

[0073]

[0074] Where r is the helical radius, θ is the helical angle, t is the time variable, A0 is the basic amplitude coefficient, λ is the helical attenuation coefficient, w is the angular frequency, P is the helical period length, θ0 is the initial phase shift, ε is the resonant coupling strength, and φ is the resonant phase period.

[0075] The task evaluation processor calculates the cost C(T) of the task according to the following formula. i ,t):

[0076]

[0077] Among them, w k T represents the importance weights for the four scheduling dimensions. i Let Ψ(T) represent the i-th task. i ,t) represents the urgency of the i-th task at time t;

[0078] The four scheduling dimensions refer to computing, networking, storage, and latency.

[0079] The task migration unit includes a migration detection processor, a state saving processor, and a migration recovery processor. The migration detection processor is used to determine whether task migration is required. The state saving processor is used to save the execution state and checkpoints of the task before migration. The migration recovery processor is used to restore the task at the target node and continue execution.

[0080] The migration detection processor calculates the reverse migration value V(d) of the data items involved in the task according to the following formula. j ,t):

[0081]

[0082] Where, d j Refers to the j-th data item, V(d) j ) refers to the basic value of the j-th data item, RT is the set of related tasks, α is the task feedback coefficient, and C max For the maximum cost of the task;

[0083] When the sum of the reverse migration values ​​of all data items involved in a task is less than the migration threshold, it indicates that the task needs to be migrated.

[0084] Example 2.

[0085] This embodiment includes all the contents of Embodiment 1, and provides an efficient data scheduling system for edge computing and cloud collaboration, including an edge access module, an edge computing module, a collaborative scheduling module and a cloud computing module;

[0086] The edge access module is used to send raw data into the system, the edge computing module is used to perform low-latency computing tasks and intelligent inference at the edge, the collaborative scheduling module is used to schedule and analyze resources, and the cloud computing module is used to perform large model training and cross-regional strategy optimization.

[0087] Combination Figure 2 The edge access module includes a data acquisition unit, a protocol adaptation unit, and an edge preprocessing unit. The data acquisition unit is used to receive raw event data from the terminal and perform identification processing. The protocol adaptation unit is used to map different protocols to a unified internal event model. The edge preprocessing unit is used to perform feature processing on event data at the edge.

[0088] Combination Figure 3 The edge computing module includes a task decomposition unit, a fast caching unit, and a local inference unit. The task decomposition unit is used to split task requests into multiple sub-tasks. The fast caching unit is used to store hot data, model fragments, and checkpoints. The local inference unit is used to run lightweight models for real-time inference.

[0089] Combination Figure 4 The collaborative scheduling module includes a resource monitoring unit, a task allocation unit, and a task migration unit. The resource monitoring unit is used to collect real-time resource indicators and calculate health scores. The task allocation unit is used to evaluate the multi-dimensional cost of each subtask to be executed and output placement decisions. The task migration unit is used to set trigger events and migrate tasks.

[0090] Combination Figure 5The cloud computing module includes a global optimization unit, a large-scale storage unit, and a centralized training unit. The global optimization unit is used to summarize cross-domain operation indicators and historical data and optimize global strategies. The large-scale storage unit is used to set up data directories and store corresponding data. The centralized training unit is used to train large models on labeled data and manage model versions.

[0091] The data acquisition unit includes a data receiving processor, a timestamp processor, and an integrity verification processor. The data receiving processor is used to receive raw data packets from the terminal. The timestamp processor is used to append a unified system timestamp when the data is received. The integrity verification processor is used to perform signature verification on the received data packets.

[0092] The protocol adaptation unit includes a protocol parsing processor, a format conversion processor, and a compatibility mapping processor. The protocol parsing processor is used to decode data from different protocols. The format conversion processor is used to unify the parsed data into an internal standard format. The compatibility mapping processor is used to establish a field mapping table for proprietary protocols to ensure system compatibility.

[0093] The edge preprocessing unit includes a data filtering processor, a data compression processor, and a feature extraction processor. The data filtering processor is used to remove redundant and invalid data, the data compression processor is used to perform lossless compression on large data streams, and the feature extraction processor is used to extract basic statistical features and use them for feature vector representation.

[0094] The task decomposition unit includes a task parsing processor, a task splitting processor, and a dependency processor. The task parsing processor is used to parse the task description to identify the task type and computational requirements. The task splitting processor is used to split a complex task into multiple subtasks. The dependency processor is used to record the dependency order between subtasks.

[0095] The fast cache unit includes a memory cache processor, a disk cache processor, and a cache eviction processor. The memory cache processor is used to temporarily store frequently accessed data in memory, the disk cache processor is used to persist large-scale data to disk, and the cache eviction processor is used to dynamically clean up low-value cache.

[0096] The local inference unit includes a model loading processor, an inference computation processor, and a result verification processor. The model loading processor is used to load a lightweight model that can run on the edge device. The inference computation processor performs inference computation based on the model to perform the task. The result verification processor is used to perform confidence detection on the inference results.

[0097] The resource monitoring unit includes a performance monitoring processor, a network monitoring processor, and an energy consumption monitoring processor. The performance monitoring processor is used to collect data on the usage of computing resources such as CPU, GPU, and memory. The network monitoring processor is used to monitor bandwidth, latency, and packet loss rate. The energy consumption monitoring processor is used to monitor device energy consumption and temperature status.

[0098] The task allocation unit includes a task evaluation processor, an allocation decision processor, and a priority processor. The task evaluation processor is used to calculate the execution cost of the task. The allocation decision processor selects a suitable execution location based on the evaluation result. The priority processor is used to set priorities for tasks and ensure that critical tasks are scheduled first.

[0099] The task evaluation processor calculates the spiral fluctuation state value Φ(r, θ, t) of the system according to the following formula:

[0100]

[0101] Where r is the helical radius, θ is the helical angle, t is the time variable, A0 is the basic amplitude coefficient, λ is the helical attenuation coefficient, w is the angular frequency, P is the helical period length, θ0 is the initial phase shift, ε is the resonant coupling strength, and φ is the resonant phase period.

[0102] The spiral radius refers to the degree to which the current load state of the edge node deviates from the optimal operating point;

[0103] The spiral angle refers to the current phase of the edge computing system's operating cycle;

[0104] The basic amplitude coefficient refers to the basic scheduling intensity of the edge computing system;

[0105] The spiral decay coefficient refers to the rate at which the scheduling influence decays as the distance from the core node increases;

[0106] Angular frequency refers to the fundamental frequency of load changes in an edge computing system;

[0107] Spiral cycle length refers to the complete scheduling cycle of an edge node returning from high load to low load;

[0108] Initial phase offset refers to the difference in load cycle start-up time between different edge nodes;

[0109] Resonant coupling strength refers to the strength coefficient of the mutual influence of loads between edge nodes;

[0110] The resonant phase period refers to the phase period during which the load resonates between edge nodes;

[0111] The task evaluation processor calculates the cost C(T) of the task according to the following formula. i ,t):

[0112]

[0113] Among them, w k T represents the importance weights for the four scheduling dimensions. i Let Ψ(T) represent the i-th task. i ,t) represents the urgency of the i-th task at time t;

[0114] The four scheduling dimensions refer to computation, network, storage, and latency.

[0115] The task migration unit includes a migration detection processor, a state saving processor, and a migration recovery processor. The migration detection processor is used to determine whether task migration is required. The state saving processor is used to save the execution state and checkpoints of the task before migration. The migration recovery processor is used to restore the task at the target node and continue execution.

[0116] The migration detection processor calculates the reverse migration value V(d) of the data items involved in the task according to the following formula. j ,t):

[0117]

[0118] Where, d j Refers to the j-th data item, V(d) j ) refers to the basic value of the j-th data item, RT is the set of related tasks, α is the task feedback coefficient, and C max For the maximum cost of the task;

[0119] When the sum of the reverse migration values ​​of all data items involved in a task is less than the migration threshold, it means that the task needs to be migrated.

[0120] The global optimization unit includes a resource aggregation processor, an optimization calculation processor, and a policy distribution processor. The resource aggregation processor is used to aggregate the running status and load of edge nodes. The optimization calculation processor is used to run a global scheduling algorithm to generate cross-regional optimization schemes. The policy distribution processor is used to distribute the optimization results to edge nodes.

[0121] The large-scale storage unit includes a data storage processor, an index retrieval processor, and a backup management processor. The data storage processor is used to store raw data and intermediate results from the edge. The index retrieval processor is used to index and quickly retrieve historical data. The backup management processor is used to perform version management and hierarchical archiving of data.

[0122] The centralized training unit includes a training data processor, a model training processor, and a model deployment processor. The training data processor is used to clean and label the data returned from the edge to construct a training sample set. The model training processor is used to perform centralized training and verification of the deep learning model. The model deployment processor is used to compress and manage the version of the trained model and distribute it to the edge nodes.

[0123] Example 3.

[0124] This embodiment includes all the contents of Embodiment 2, and provides an efficient data scheduling system for edge computing and cloud collaboration, including an edge access module, an edge computing module, a collaborative scheduling module and a cloud computing module;

[0125] The edge access module is used to send raw data into the system, the edge computing module is used to perform low-latency computing tasks and intelligent inference at the edge, the collaborative scheduling module is used to schedule and analyze resources, and the cloud computing module is used to perform large model training and cross-regional strategy optimization.

[0126] This embodiment adds an edge intelligent analysis module and a dynamic load balancing module to improve the system's intelligence level and load handling capacity;

[0127] The edge intelligent analysis module is used to perform deep learning analysis on the data collected from edge nodes to achieve anomaly detection, trend prediction and intelligent decision support.

[0128] The dynamic load balancing module is used to monitor the system load distribution in real time, dynamically adjust the task allocation strategy, and ensure the optimal utilization of system resources.

[0129] The edge intelligent analysis module includes a data preprocessing unit, a feature engineering unit, a model inference unit, and a result evaluation unit;

[0130] The data preprocessing unit includes a data cleaning processor, an outlier detection processor, and a data standardization processor. The data cleaning processor is used to remove noise and erroneous values ​​from the data. The outlier detection processor is used to identify and process outliers in the data. The data standardization processor is used to normalize data of different dimensions.

[0131] The feature engineering unit includes a feature selection processor, a feature transformation processor, and a feature construction processor. The feature selection processor is used to select the most representative feature subset from the original features. The feature transformation processor is used to perform mathematical transformations on the features to improve model performance. The feature construction processor is used to create new derived features by combining existing features.

[0132] The model inference unit includes a lightweight model loader, an inference engine, and a result post-processor. The lightweight model loader is used to load a compressed neural network model suitable for running on edge devices. The inference engine is used to perform real-time inference computation. The result post-processor is used to format the inference results and evaluate their confidence.

[0133] The result evaluation unit includes an accuracy evaluation processor, a consistency check processor, and a reliability scoring processor. The accuracy evaluation processor is used to evaluate the accuracy level of the inference results, the consistency check processor is used to verify the consistency between the inference results and historical data, and the reliability scoring processor is used to assign a reliability score to the inference results.

[0134] The dynamic load balancing module includes a load monitoring unit, a balancing strategy unit, and an adaptive adjustment unit.

[0135] The load monitoring unit includes a real-time load collector, a load trend analyzer, and a bottleneck identifier. The real-time load collector is used to continuously collect the usage of resources such as CPU, memory, and network of each node. The load trend analyzer is used to analyze the load change trend and predict the future load level. The bottleneck identifier is used to automatically identify the system performance bottleneck.

[0136] The load balancing strategy unit includes a strategy selection processor, a weight calculation processor, and a threshold management processor. The strategy selection processor is used to select the most suitable load balancing strategy based on the current system state. The weight calculation processor is used to dynamically calculate the load distribution weight of each node. The threshold management processor is used to manage the trigger threshold and adjustment parameters of load balancing.

[0137] The adaptive adjustment unit includes an adjustment trigger, a parameter optimizer, and an effect verifier. The adjustment trigger is used to monitor load imbalance and trigger adjustment actions. The parameter optimizer is used to optimize adjustment parameters based on historical data and the current state. The effect verifier is used to verify the adjustment effect and perform feedback optimization.

[0138] This embodiment also adds a fault-tolerant recovery module to improve the system's reliability and fault tolerance.

[0139] The fault-tolerant recovery module includes a fault detection unit, a fault isolation unit, and an automatic recovery unit;

[0140] The fault detection unit includes a health monitoring processor, an anomaly detection processor, and a fault diagnosis processor. The health monitoring processor is used to continuously monitor the operating status of each component. The anomaly detection processor is used to detect system anomalies based on statistical learning methods. The fault diagnosis processor is used to analyze the cause of the fault and determine the fault type.

[0141] The fault isolation unit includes an isolation strategy processor, a resource reallocation processor, and a communication switching processor. The isolation strategy processor is used to formulate an isolation scheme for the faulty node. The resource reallocation processor is used to reallocate the tasks and resources of the faulty node to healthy nodes. The communication switching processor is used to switch communication paths to avoid the faulty node.

[0142] The automatic recovery unit includes a recovery strategy processor, a data synchronization processor, and a service restart processor. The recovery strategy processor is used to formulate a recovery plan for the faulty node, the data synchronization processor is used to perform data synchronization after the node is recovered, and the service restart processor is used to restart the faulty service and verify its normal operation.

[0143] The i, j, and k mentioned above are ordinal numbers used to represent sequence numbers and have no actual meaning.

[0144] The following is a partial code snippet from this system:

[0145]

[0146]

[0147]

[0148] This system will now be tested against ordinary systems, and the collected data will be organized accordingly. Figure 6 and Figure 7 Comparison of effects.

[0149] The content disclosed above is only a preferred and feasible embodiment of the present invention, and is not intended to limit the scope of protection of the present invention. Therefore, all equivalent technical changes made based on the content of the present invention specification and drawings are included within the scope of protection of the present invention. Furthermore, the elements therein can be updated as technology develops.

Claims

1. A high-efficiency edge computing and cloud-based collaborative data scheduling system, characterized in that, It includes an edge access module, an edge computing module, a collaborative scheduling module, and a cloud computing module; The edge access module is used to send raw data into the system, the edge computing module is used to perform low-latency computing tasks and intelligent inference at the edge, the collaborative scheduling module is used to schedule and analyze resources, and the cloud computing module is used to perform large model training and cross-regional strategy optimization. The edge access module includes a data acquisition unit, a protocol adaptation unit, and an edge preprocessing unit. The data acquisition unit is used to receive raw event data from the terminal and perform identification processing. The protocol adaptation unit is used to map different protocols to a unified internal event model. The edge preprocessing unit is used to perform feature processing on the event data at the edge. The edge computing module includes a task decomposition unit, a fast caching unit, and a local inference unit. The task decomposition unit is used to split task requests into multiple sub-tasks. The fast caching unit is used to store hot data, model fragments, and checkpoints. The local inference unit is used to run lightweight models for real-time inference. The collaborative scheduling module includes a resource monitoring unit, a task allocation unit, and a task migration unit. The resource monitoring unit is used to collect real-time resource indicators and calculate health scores. The task allocation unit is used to evaluate the multi-dimensional cost of each subtask to be executed and output placement decisions. The task migration unit is used to set trigger events and migrate tasks. The cloud computing module includes a global optimization unit, a large-scale storage unit, and a centralized training unit. The global optimization unit is used to summarize cross-domain operation indicators and historical data and optimize global strategies. The large-scale storage unit is used to set up data directories and store corresponding data. The centralized training unit is used to train large models on labeled data and manage model versions. The task allocation unit includes a task evaluation processor, an allocation decision processor, and a priority processor. The task evaluation processor is used to calculate the execution cost of the task. The allocation decision processor selects a suitable execution location based on the evaluation result. The priority processor is used to set priorities for tasks and ensure that critical tasks are scheduled first. The task evaluation processor calculates the spiral fluctuation state value of the system according to the following formula. : ； Where r is the helix radius. Let A0 be the helix angle, t be the time variable, and A0 be the basic amplitude coefficient. Where is the helical attenuation coefficient, w is the angular frequency, and P is the helical period length. For the initial phase shift, The resonant coupling strength, The resonant phase period; the helix radius refers to the degree to which the current load state of the edge node deviates from the optimal operating point; the helix angle refers to the current working cycle phase of the edge computing system; the basic amplitude coefficient refers to the basic scheduling intensity of the edge computing system; the helix attenuation coefficient refers to the rate attenuation of the scheduling influence as the distance from the core node increases; the angular frequency refers to the basic frequency of load changes in the edge computing system; the helix period length refers to the complete scheduling cycle of the edge node returning from high load to low load; the initial phase offset refers to the time difference between the load cycle start-up of different edge nodes; the resonant coupling strength refers to the strength coefficient of the mutual influence between the loads of edge nodes; the resonant phase period refers to the phase period during which the loads of edge nodes resonate. The task evaluation processor calculates the cost of the task according to the following formula. : ； Among them, w k T represents the importance weights for the four scheduling dimensions. i This represents the i-th task. This indicates the urgency of the i-th task at time t; The four scheduling dimensions refer to computation, network, storage, and latency. The task migration unit includes a migration detection processor, a state saving processor, and a migration recovery processor. The migration detection processor is used to determine whether task migration is required. The state saving processor is used to save the execution state and checkpoints of the task before migration. The migration recovery processor is used to restore the task on the target node and continue its execution. The migration detection processor calculates the reverse migration value of the data items involved in the task according to the following formula. : ； Where, d j Refers to the j-th data item, V(d) j ) refers to the basic value of the j-th data item, and RT is the set of related tasks. C is the task feedback coefficient. max For the maximum cost of the task; When the sum of the reverse migration values ​​of all data items involved in a task is less than the migration threshold, it indicates that the task needs to be migrated.

Images