Data processing method, system and device, medium and product

Abstract

The invention discloses a data processing method, system and device, a medium and a product, and belongs to the technical field of computers.The method comprises the steps that a real-time distribution threshold value is obtained, and the real-time distribution threshold value is obtained by periodically executing an iterative coordinate rising algorithm by a target server; a target function is solved according to the state information of the multiple first edge nodes, and the target function is constructed based on task analysis precision and time delay cost; analyzing the data processing task based on a first analysis model to obtain an initial processing result and a confidence coefficient; the confidence coefficient is compared with a real-time distribution threshold value, if the confidence coefficient is larger than the real-time distribution threshold value, the initial processing result serves as a first processing result, otherwise, the data processing task is unloaded to the target server to be processed through a second analysis model of the target server, and a second processing result is obtained. And low-delay and high-precision collaborative gains for performing data processing tasks at edge nodes can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular to a data processing method, system, device, medium and product. BACKGROUND

[0002] In deterministic network scenarios such as industrial control and intelligent transportation, operation and maintenance data (such as device logs and performance indicators) generated on the edge side exhibit massive, heterogeneous and high-concurrency characteristics. Immediate and accurate analysis and processing of these data is a prerequisite for realizing intelligent operation and maintenance and business assurance. However, the limited computing power of edge nodes cannot support complex high-precision data analysis models, and if all data is unconditionally uploaded to the cloud, it will inevitably introduce uncontrollable network transmission delays. Therefore, how to achieve a dynamic optimal balance between the analysis accuracy and processing delay of data processing tasks has become a core challenge in the current data processing field.

[0003] The closest prior art to the present application is a cloud-edge collaborative task offloading method based on a fixed threshold. This method presets a fixed threshold, and when the confidence level of the edge model output is lower than the threshold, the task is offloaded to the cloud for in-depth analysis using the more powerful model of the cloud; otherwise, the processing is completed locally on the edge.

[0004] However, the biggest defect of the above-mentioned prior art is its static and rigid decision-making mechanism. The fixed offloading threshold cannot perceive and adapt to the dynamic changes of network status and system load. For example, when the network condition is good, the analysis accuracy may be sacrificed due to the overly conservative threshold, and when the network is congested or the cloud is overloaded, blind task offloading may occur due to the overly aggressive threshold, resulting in unpredictable long delays in task transmission and queuing. Therefore, this method has poor adaptability in complex and variable real network environments, and the data processing effect is not good. SUMMARY

[0005] The present application provides a data processing method, system, device, medium and product, which can solve the problem that a static data processing offloading strategy cannot adapt to dynamically changing network and load environments, resulting in a suboptimal balance between data processing accuracy and delay.

[0006] The present application provides a data processing method, comprising: obtaining a real-time offloading threshold, wherein the real-time offloading threshold is obtained by a target server by periodically performing an iterative coordinate ascent algorithm and solving a target function based on state information of a plurality of first edge nodes, the target function being constructed based on task analysis accuracy and delay cost, and the target server being in communication connection with the plurality of first edge nodes; The data processing task is analyzed based on the first analysis model to obtain the initial processing results and confidence level; The confidence level is compared with the real-time diversion threshold. If the confidence level is greater than the real-time diversion threshold, the initial processing result is taken as the first processing result. Otherwise, the data processing task is offloaded to the target server for processing by the second analysis model of the target server to obtain the second processing result.

[0007] In this embodiment of the invention, the first edge node performs local traffic offloading decisions, delegating some computational offloading judgments to the network edge. This reduces reliance on real-time communication with the central server, decreases network round-trip latency, and improves system response speed. A "real-time traffic offloading threshold," periodically calculated by the server, is obtained. This threshold is no longer statically set but dynamically refreshed based on the global node status, improving the system's adaptability to environmental changes. This threshold is obtained by the server using an iterative coordinate ascent algorithm based on a target function constructed by the server based on "task analysis accuracy and latency cost." Accuracy assurance and latency control are incorporated into a unified optimization objective, achieving an optimal trade-off between accuracy and overhead through multi-dimensional iterative optimization using the coordinate ascent method. The edge node first processes the task using the first analysis model to obtain "initial processing result + confidence level," fully utilizing the lightweight computing power of the edge node for rapid initial task judgment, reducing unnecessary offloading. By comparing the confidence level with the real-time traffic offloading threshold, a decision is made on whether to adopt the result locally or offload it to the server, using "confidence level - threshold" as the criterion. This logic is simple, execution overhead is low, and traffic offloading decisions can be completed in milliseconds. Overall, this embodiment can quickly complete the intelligent routing of tasks at the edge, which reduces the average end-to-end latency of tasks and ensures the overall analysis accuracy through global optimization on the server side, achieving a synergistic gain of "low latency + high accuracy".

[0008] Furthermore, the real-time traffic splitting threshold is obtained by the target server periodically executing an iterative coordinate ascent algorithm and solving the objective function based on the state information of multiple first edge nodes, specifically as follows: In each iteration, the target server sequentially iterates over each first edge node as the current optimization node, fixes the first diversion threshold of all first edge nodes except the current optimization node, updates the first diversion threshold of the current optimization node, and obtains the second diversion threshold of the current optimization node when the function value of the objective function is maximized. The next first edge node is iteratively optimized based on the second diversion threshold and each of the first diversion thresholds until all first edge nodes have been traversed, and then the next iteration begins. When the threshold change of all edge nodes is less than a preset convergence threshold, the iteration stops, and the real-time diversion threshold of the first edge node is obtained.

[0009] In this way, the server sequentially uses a single first edge node as the "current optimization node" in each iteration, breaking down the joint optimization of multiple nodes into a series of single-node sub-problems, significantly reducing the computational cost per iteration and making it suitable for periodic online execution. By fixing the first shunting threshold of the remaining nodes, only updating the threshold of the current node, and iterating through all nodes before entering the next round, a "block coordinate ascent" strategy is adopted to ensure that the objective function is monotonically constant and has strong theoretical convergence. At the same time, it is easy to implement and debug in engineering. The iteration stops when the threshold change of all edge nodes is less than the preset convergence threshold, providing a clear convergence criterion and preventing the waste of computational resources caused by infinite iteration. Overall, this embodiment provides a convergent, implementable, and computationally controllable global threshold optimization mechanism for large-scale edge node groups through block iteration using the coordinate ascent method, making the real-time shunting threshold theoretically approach the Pareto optimality of "accuracy-latency".

[0010] Furthermore, the value of the objective function is obtained by calculating the objective cost and the objective benefit, specifically as follows: For each of the first edge nodes, predictions are made based on historical task data to obtain the predicted task data for the current period. The predicted task data is then divided according to the first diversion threshold to obtain the predicted task unloading volume and the predicted task processing volume. The target cost is calculated based on the predicted task unloading volume and the status information, and the target revenue corresponding to the task analysis accuracy is calculated based on the predicted task processing volume. The function value is calculated based on the target cost and the target benefit for each of the first edge nodes.

[0011] This approach utilizes historical task data from each edge node to predict the "current cycle task volume," based on real historical statistics, avoiding blind assumptions about the load and improving the reliability of subsequent cost / benefit assessments. The predicted tasks are divided into "predicted task processing volume" and "predicted task unloading volume" according to the current first diversion threshold, pre-quantifying the ratio of local processing to upload processing and providing input for the cost-benefit model. The "target cost" (latency caused by unloading) and "target benefit" (analytical accuracy obtained from local processing) are calculated separately, breaking down the abstract target into two quantifiable and measurable indicators, facilitating numerical comparisons in the optimization algorithm. The objective function value is obtained by integrating the costs and benefits of all nodes, aggregating the local indicators of distributed nodes into a system-level evaluation, ensuring that the optimization results reflect global benefits. Overall, this embodiment, through the "prediction-division-evaluation" link, maps dynamically changing load conditions, network states, and accuracy requirements into a unified function value, providing an optimizable quantitative target for the coordinate ascent algorithm, making threshold decisions rational and evidence-based.

[0012] Furthermore, the target cost is calculated based on the predicted task unloading amount and the status information, specifically as follows: Periodically acquire status information, wherein the status information includes network latency information from the first edge node to the target server and the current load status of the target server; Based on the predicted task offloading amount, the network latency information, and the current load status, a first latency is obtained through a queuing theory model, and a first latency cost is calculated based on the first latency and the first latency weight coefficient, wherein the first latency is used to characterize the latency of offloading the predicted task offloading amount to the target server; The second latency cost is calculated based on the predicted task processing volume and the second latency weighting coefficient. The sum of the first latency cost and the second latency cost is then calculated to obtain the target cost. The second latency cost is used to characterize the latency corresponding to the first edge node processing the predicted task.

[0013] This approach collects "network latency information + current server load status" before each cycle begins, allowing real-time monitoring of link quality and cloud load, preventing erroneous offloading due to outdated information. Based on predicted task offloading volume, network latency, and server load, a queuing theory model is used to estimate the "first latency." Queuing theory abstracts "network transmission + cloud queuing" into formulaizable latency components, improving estimation accuracy. "Weight coefficients" are assigned to the first and second latencies and summed to obtain the target cost, allowing the system to flexibly adjust the importance of "offloading latency vs. local processing latency" according to business scenarios, enhancing the method's versatility. Overall, this embodiment incorporates complex cross-layer latency factors into a single cost formula, enabling subsequent optimization algorithms to accurately measure the time cost difference between the "offloading-local" and "online" modes, providing a quantitative basis for balancing analysis accuracy and real-time performance.

[0014] Furthermore, the data processing method further includes: The edge nodes use a support vector machine model or a gradient boosting decision tree model as the first analysis model to analyze the data processing task; The target server uses any one of the following models as the second analysis model: graph neural network model, long short-term memory network model, or deep learning model based on Transformer, to analyze the data processing tasks unloaded from each edge node.

[0015] This approach employs SVM or GBDT as the primary analysis model for edge nodes. SVM / GBDT models are small and fast, making them suitable for resource-constrained edge devices, while GBDT has low feature engineering requirements and is easy to deploy. On the server side, Graph Neural Networks (GNNs) or LSTMs are used as the secondary analysis model. GNNs can fuse inter-node relationship information, while LSTMs excel at capturing temporal features. Both can leverage the abundant computing power of the cloud to achieve higher accuracy. Overall, this embodiment forms a hierarchical model architecture of "lightweight initial edge detection + complex cloud-based fine-tuning," ensuring real-time edge detection while fully utilizing cloud-based deep learning models to improve accuracy after offloading, achieving complementary "edge-cloud" capabilities.

[0016] Furthermore, updating the first shunting threshold of the current optimized node specifically involves: The first diversion threshold of the current optimized node is updated using a numerical search method.

[0017] This approach uses a numerical search method to update the branching threshold of the current optimization node. It doesn't require a differentiable objective function; only the function value is needed for the search, reducing constraints on the model's form. It can incorporate strategies such as the golden section, binary search, or exhaustive search, making implementation simple. Overall, this embodiment enables the coordinate ascent algorithm to find the (locally) optimal threshold in a single coordinate optimization phase, ensuring monotonically increasing function values, accelerating overall convergence, and facilitating implementation in engineering code.

[0018] Another embodiment of the present invention also provides a data processing system, comprising: Threshold module, initial processing module, and splitting processing module; The threshold module is used to obtain a real-time traffic splitting threshold, wherein the real-time traffic splitting threshold is obtained by the target server through periodically executing an iterative coordinate ascent algorithm and solving the objective function based on the state information of multiple first edge nodes. The objective function is constructed based on task analysis accuracy and latency cost. The target server is communicatively connected to multiple first edge nodes. The initial processing module is used to analyze the data processing task based on the first analysis model to obtain the initial processing results and confidence level. The traffic splitting module is used to compare the confidence level with the real-time traffic splitting threshold. If the confidence level is greater than the real-time traffic splitting threshold, the initial processing result is used as the first processing result. Otherwise, the data processing task is offloaded to the target server for processing by the second analysis model of the target server to obtain the second processing result.

[0019] Another embodiment of the present invention provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the steps of the data processing method of the present invention.

[0020] Another embodiment of the present invention provides a computer-readable storage medium item, including: a stored computer program, which, when the computer program is running, controls the device where the computer-readable storage medium is located to perform the steps of the data processing method of the present invention.

[0021] Another embodiment of the present invention also provides a computer program product stored in a storage medium, the computer program product being executed by at least one processor to implement the steps of the data processing method as described in any one of the first aspects. Attached Figure Description

[0022] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is one of the flowcharts illustrating a data processing method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a processing architecture provided by an embodiment of the present invention; Figure 3 This is a second schematic flowchart of a data processing method provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the accuracy versus loss curve of a second analysis model provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the confusion matrix of a second analysis model provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the confusion matrix of a first analysis model provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the ROC curve and PR curve of a second analysis model provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the ROC curve and PR curve of a first analysis model provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of a data processing system provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0026] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0027] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0028] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0029] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0030] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0031] See Figure 1 To address the problem that existing static data processing offloading strategies cannot adapt to dynamically changing network and load environments, resulting in an inability to achieve an optimal balance between data processing accuracy and latency, an embodiment of the present invention provides a data processing method, comprising: Step 101: Obtain the real-time traffic splitting threshold, wherein the real-time traffic splitting threshold is obtained by the target server periodically executing an iterative coordinate ascent algorithm and solving the objective function based on the state information of multiple first edge nodes. The objective function is constructed based on task analysis accuracy and latency cost. The target server communicates with multiple first edge nodes.

[0032] In this embodiment, the method is applied to a device such as... Figure 2 Taking the two-level processing architecture of cloud-edge layered collaborative analysis as an example, the architecture includes: Edge Layer: On the first edge node close to the data source (such as 5G base station, industrial park gateway), lightweight analysis models are deployed. These models are specially designed and trained, characterized by extremely fast computing speed and low resource consumption. Their main task is to perform real-time preliminary screening and rapid analysis of local data streams, and can accurately identify about 70%-80% of common or clearly patterned faults. • Central Cloud Layer: A high-precision analysis model is deployed on the central cloud server. This model is highly complex, has strong computing power, and can aggregate global information from all first edge nodes. Its advantage lies in having a global view, enabling it to analyze complex and interconnected faults caused by multiple factors that span different regions, with extremely high analysis accuracy.

[0033] This layered architecture allows most simple, high-frequency analysis tasks to be handled locally at the edge, avoiding unnecessary data transmission and laying the foundation for rapid response.

[0034] Based on this layered architecture, it provides, for example Figure 3The second schematic diagram of a data processing method illustrates that the first edge node to which this method is applied is designated as the first edge node. This first edge node receives the real-time traffic splitting threshold from the target server with which it is communicatively connected. This real-time traffic splitting threshold is generated by the target server, acting as a central coordinator, through a periodic, distributed collaborative optimization process. The core objective of this optimization process is to collaboratively optimize the task scheduling strategy of all edge nodes within the entire system. The optimization is based on a global perception of the state information of multiple (or even all) edge nodes (this information at least reflects the current or predicted state of the system's computation and network). The optimization objective is to maximize or minimize a preset system-level objective function. The concept behind constructing this objective function is to quantitatively balance the gains in analytical accuracy brought about by task scheduling decisions with the latency costs incurred.

[0035] It should be noted that the two-level cloud-edge architecture of the present invention can be expanded into a multi-level architecture. For example, regional aggregation nodes can be introduced to form a three-level system of "edge-region-central cloud" to adapt to the operation and maintenance needs of ultra-large-scale networks.

[0036] Step 102: Analyze the data processing task based on the first analysis model to obtain the initial processing results and confidence level.

[0037] In this embodiment, a pre-deployed, lightweight analysis model (i.e., the first analysis model) is run locally on the first edge node to quickly analyze the real-time data processing tasks. This analysis process synchronously outputs two key pieces of information: a preliminary analysis conclusion (such as "a certain port is congested") and a confidence score. This score (usually between 0 and 1) is a quantitative expression of the model's "confidence" in its analysis conclusion; the higher the score, the greater the likelihood that the model considers its conclusion correct.

[0038] Step 103: Compare the confidence level with the real-time diversion threshold. If the confidence level is greater than the real-time diversion threshold, the initial processing result is taken as the first processing result. Otherwise, the data processing task is offloaded to the target server for processing by the second analysis model of the target server to obtain the second processing result.

[0039] In this embodiment, the confidence score is compared with the real-time traffic splitting threshold held by the first edge node: if the confidence score > the traffic splitting threshold, it indicates that the local model has sufficient confidence in this analysis, directly adopts the local analysis results, and immediately executes corresponding operational actions (such as adjusting traffic strategies). The entire processing flow is closed-loop at the edge, and the response time can reach the millisecond level. If the confidence level is less than or equal to the triage threshold, it indicates that the local model considers the current situation to be complex or rare, and its analysis conclusions may be unreliable. In this case, the "unload" mechanism is automatically triggered to package the relevant data of the task and send it to the central cloud server via the network.

[0040] After receiving a task, the high-precision model in the central cloud performs in-depth analysis based on global information, providing a more authoritative final analysis conclusion. This mechanism ensures efficient processing of simple tasks and accurate analysis of complex tasks, and the key to decision-making lies in the dynamically changing real-time triage threshold.

[0041] As an example of an embodiment of the present invention, the real-time diversion threshold is obtained by the target server through periodically executing an iterative coordinate ascent algorithm and solving the objective function based on the state information of multiple first edge nodes. Specifically, in each iteration, the target server iterates on each first edge node as the current optimization node, fixes the first diversion threshold of all first edge nodes except the current optimization node, updates the first diversion threshold of the current optimization node, and obtains the second diversion threshold of the current optimization node when the function value of the objective function is maximized. It then iterates and optimizes the next first edge node based on the second diversion threshold and each of the first diversion thresholds until all first edge nodes have been traversed, and enters the next iteration. When the threshold change of all edge nodes is less than a preset convergence threshold, the iteration stops, and the real-time diversion threshold of the first edge node is obtained.

[0042] In this embodiment, the central cloud server periodically (e.g., every 5 seconds) runs a core adaptive threshold optimization algorithm (C-DAOT) to calculate the current optimal "triage threshold" for each edge node. Finding the optimal threshold for all nodes simultaneously is a highly complex multidimensional optimization problem. Therefore, the C-DAOT algorithm employs an efficient "iterative coordinate ascent" strategy to solve it. The algorithm includes steps 201-204, as follows: Step 201, Initialization Phase: The algorithm starts with a set of default or previous period thresholds. Taking the first decision period as an example, the target server assigns a first diversion threshold as an initial value to all first edge nodes connected to it. This initial value can be a uniform default value (e.g., 0.5) or an empirical value set based on historical data.

[0043] Step 202, Iterative Optimization Phase: The server enters a multi-round iterative loop. The goal of each iteration is to generate a better set of thresholds. In each iteration, the server sequentially designates each first edge node as the "current optimization node" in a predetermined order (e.g., node ID order) and performs the following steps: a. Fixed environment: When optimizing the current node, the server temporarily keeps the first traffic splitting threshold of all other first edge nodes in the network unchanged, which is equivalent to creating a stable "decision-making environment" for the optimization of the current node; b. Univariate search: After fixing the thresholds of all other nodes, the problem is simplified to a univariate optimization problem. The server can systematically or intelligently adjust the first shunting threshold of the current optimization node within a feasible threshold range (usually between 0 and 1). c. Evaluation and Decision: For each candidate value of the first diversion threshold, the server will recalculate the function value of the objective function based on the latest acquired state information (including network latency of all nodes, server load, etc.). This function value quantifies the overall efficiency of the entire system under the current threshold combination (i.e., the comprehensive trade-off between accuracy gains and latency costs). The server selects the threshold that maximizes the function value of the objective function from all candidate values, and uses it as the update result of the current optimization node in this iteration (i.e., the second diversion threshold). d. Update and rotation: The server internally replaces the first traffic splitting threshold with the second traffic splitting threshold to form a new threshold combination. Then, the server sets the next first edge node as the "current optimized node" and repeats steps a to c.

[0044] Step 203, Convergence Judgment Phase: After one round of iteration is completed, that is, after all edge nodes have been optimized once, the server will check the convergence of the algorithm, calculate the change between the new threshold of each node and the old threshold in the previous round of iteration. If the change of the threshold of all nodes is less than a preset, very small convergence threshold, the algorithm is considered to be stable, and the globally optimal or near-optimal solution has been found, and the iteration stops.

[0045] Step 204, Result Output Stage: The final set of thresholds obtained after the iteration stops is the optimal threshold combination for this decision cycle. The target server distributes them to the corresponding first edge nodes to guide the task diversion decision in the next cycle.

[0046] It should be noted that, in addition to the block coordinate ascent method, other distributed optimization algorithms or game theory methods can also be used to solve this collaborative optimization problem. For example, each edge node can be modeled as a rational agent, and by designing a reasonable incentive mechanism, it can converge to the global optimum in the process of pursuing its own utility maximization.

[0047] As an example of an embodiment of the present invention, the function value of the objective function is obtained by calculating the objective cost and the objective benefit, specifically as follows: for each first edge node, prediction is performed based on historical task data to obtain the predicted task data for the current period, and the predicted task data is divided according to the first diversion threshold to obtain the predicted task unloading amount and the predicted task processing amount; the objective cost is calculated based on the predicted task unloading amount and the status information, and the objective benefit corresponding to the task analysis accuracy is calculated based on the predicted task processing amount; the function value is calculated based on the objective cost and the objective benefit for each first edge node.

[0048] In this embodiment, for each first edge node, the target server uses a prediction algorithm (such as time series analysis or machine learning model) based on its historical task data (e.g., arrival time, number of tasks) to estimate the total number of tasks that the node will generate in the current decision-making cycle, i.e., the predicted task data. Based on the currently evaluated triage threshold (i.e., the candidate threshold in the algorithm iteration), the system simulates the node's decision-making behavior, dividing the predicted total task volume into two parts: predicted task processing volume (i.e., the number of tasks expected to be processed on the local edge node with a confidence level greater than the threshold); and predicted task offloading volume (i.e., the number of tasks expected to be offloaded to the target server with a confidence level less than or equal to the threshold). The target cost primarily represents the time delay cost incurred by task processing. Its core component is the end-to-end latency cost introduced by offloading tasks. Combining the predicted task offloading volume, real-time network latency information, and the current load status of the target server, the processing latency of offloading tasks is estimated using a modeling method (e.g., queuing theory). Ultimately, the target cost is a function of the estimated latency and a preset latency cost weighting coefficient, where the first weighting coefficient reflects the task's sensitivity to latency.

[0049] like Figure 4The diagram illustrates the change curves of accuracy and loss for a second analytical model. The loss represents the prediction error generated by each of the first and second analytical models when processing data. This diagram is used to simultaneously and intuitively display and compare the performance trends of cloud and edge models during training. The X-axis (horizontal axis) in the curve typically represents the training period or number of training steps, indicating the time progression of model training. The left Y-axis (vertical axis) corresponds to the accuracy curve, with values ​​ranging from 0% to 100% (or 0 to 1), where higher is better. The right Y-axis (vertical axis) corresponds to the loss curve, with values ​​starting from 0 and extending downwards, where lower is better. Therefore, the target benefit used in this embodiment mainly represents the improvement in analytical accuracy brought about by task processing. Its core part is the accuracy guarantee benefit obtained by processing tasks locally (avoiding the risk of task failure due to network problems), and the accuracy improvement benefit obtained by unloading tasks to high-precision models. The calculation of this benefit is directly related to the amount of task processing. Based on the predicted amount of task processing (local processing) and the predicted amount of task unloading (high-precision processing in the cloud), combined with the preset accuracy benefit weighting coefficient, the total benefit is calculated. Among them, the second weighting coefficient reflects the importance that the task attaches to analytical accuracy.

[0050] After completing the above calculations, a corresponding target cost and target revenue will be obtained for each first edge node. The target server adds up the target revenues of all edge nodes to obtain the total system revenue; then it adds up the target costs of all edge nodes to obtain the total system cost. The final function value is the difference between the total system revenue and the total system cost. This value intuitively reflects the "net efficiency" that the entire cloud-edge collaborative system can achieve under a certain group splitting threshold scheme.

[0051] As an example of an embodiment of the present invention, the target cost is calculated based on the predicted task offloading amount and the status information, specifically as follows: Status information is periodically acquired, wherein the status information includes network latency information from the first edge node to the target server and the current load status of the target server; based on the predicted task offloading amount, the network latency information, and the current load status, a first latency is obtained through a queuing theory model, and a first latency cost is calculated based on the first latency and a first latency weight coefficient, wherein the first latency is used to characterize the latency of offloading the predicted task offloading amount to the target server; a second latency cost is calculated based on the predicted task processing amount and a second latency weight coefficient, and the sum of the first latency cost and the second latency cost is calculated to obtain the target cost, wherein the second latency cost is used to characterize the latency corresponding to the first edge node processing the predicted task.

[0052] In this embodiment, to more scientifically predict the processing latency after unloading tasks to the cloud, a mature queuing theory model (such as the M / M / c model) can be introduced into the utility calculation model for refined modeling. This model no longer simply looks at the current task backlog in the cloud, but comprehensively considers the total unloading rate of all edge nodes (average task arrival rate) and the processing capacity of the cloud server (service rate), thereby scientifically predicting the expected queuing time for any new task after it arrives in the cloud. This predictive latency assessment makes the system's optimization decisions more forward-looking and effectively avoids cloud resource congestion caused by decision lag. Specifically, the target server periodically collects the latest status information from all first edge nodes and itself. This information constitutes the dynamic basis for cost calculation and mainly includes two categories: network latency information, obtained by measuring the current latency and / or jitter of the data transmission path between each edge node and the target server; and server load status, obtained by monitoring the target server's own real-time computing resource utilization, such as CPU utilization, memory usage, and the number of tasks currently being processed or queued. To overcome the lag in static measurements, the system employs a queuing theory model as its core tool. This model treats the target server as one or more "service desks" and the predicted task offloading volume from all edge nodes as a "customer arrival flow." By inputting the aggregated total predicted offloading task volume, measured network latency, and server processing capacity into the queuing theory model, the system scientifically estimates the total expected time for any new offloading task to travel from leaving the edge node to completion of processing in the cloud, i.e., the "first latency." This predicted value includes network transmission time and queuing time in the cloud. The estimated first latency is multiplied by a preset first latency weighting coefficient to obtain the first latency cost. For the predicted task volume to be processed locally (i.e., the predicted task processing volume), a second latency cost is assigned. This cost is typically obtained by multiplying the predicted task processing volume by a preset second latency weighting coefficient. This second latency weighting coefficient can be a fixed unit cost or a baseline latency value estimated based on the average processing speed of edge nodes, representing the inherent, relatively fixed, and low time cost of processing tasks at the edge. Finally, the system adds the two costs together to obtain the target cost. This target cost is a global estimate that quantifies the total latency cost expected to be incurred in executing all predicted tasks (including local processing and cloud offloading) under the current threshold scheme and system state.

[0053] As an example of an embodiment of the present invention, the data processing method further includes: the edge nodes using a support vector machine model or a gradient boosting decision tree model as the first analysis model to analyze the data processing task; and the target server using any one of a graph neural network model, a long short-term memory network model, or a Transformer-based deep learning model as the second analysis model to analyze the data processing task unloaded by each edge node.

[0054] In this embodiment, as Figure 5 The diagram shown is a schematic representation of the confusion matrix of the second analysis model. Figure 5 It is evident that the model not only boasts high overall accuracy but also demonstrates exceptional precision in identifying various specific fault types, particularly addressing the rare occurrence of "missed alarms." The second analysis model preferentially employs advanced models based on the Transformer architecture (such as FT-Transformer). The core advantage of the Transformer model lies in its powerful sequence information capture capability, enabling it to accurately pinpoint the root cause of problems (root cause analysis) from long, complex alarm event streams. It is particularly adept at handling complex fault scenarios requiring long-term dependencies and global correlation information. For example... Figure 6 The diagram shown is a schematic representation of the confusion matrix of the first analytical model. Figure 6 It can be seen that although the model has limited capabilities, it has a strong ability to identify some common and typical faults (the strengths of the main diagonal). The first analysis model preferably adopts gradient boosting decision tree (such as LightGBM). This type of model has extremely fast inference speed and small memory consumption, making it very suitable for deployment on resource-constrained edge devices. At the same time, it has a good processing effect on structured network monitoring data (such as performance counters and alarm log features). Figure 5 and Figure 6 This indicates that the cloud-edge models excel at different types of problems and complement each other. Furthermore, to comprehensively evaluate the analytical performance of the first and second analysis models, model performance evaluation curves, including ROC curves and PR curves, were plotted, as shown below. Figure 6 and Figure 7 As shown.

[0055] As alternatives, edge node models can also use other lightweight machine learning models, such as support vector machines (SVM) and decision trees; cloud models can also be replaced with other deep learning models with powerful feature extraction capabilities, such as graph neural networks (GNN) and long short-term memory networks (LSTM).

[0056] As an example of an embodiment of the present invention, updating the first diversion threshold of the current optimization node specifically involves using a numerical search method to update the first diversion threshold of the current optimization node.

[0057] In this embodiment, during the iterative optimization process, when the algorithm locks onto other nodes and seeks the optimal threshold for a single node (i.e., solving a one-dimensional subproblem), an efficient numerical search method can be used instead of simple traversal. For example, the Golden-section Search, like a binary search, can quickly approach the optimal threshold point that maximizes the utility function by making only a few "trials" within the [0, 1] interval. This ensures that the core optimization algorithm can complete calculations in milliseconds within each decision cycle, meeting the needs of real-time decision-making.

[0058] It should be noted that, before deploying this invention to an actual production network, it is strongly recommended to build a high-fidelity digital twin verification platform for thorough, secure, and low-cost system verification and parameter optimization. This platform can construct a virtual network environment highly consistent with the physical world using protocol-level network simulation software (such as OMNeT++) and real computing nodes (hardware-in-the-loop). In this virtual environment, we can simulate various extreme network events, such as traffic bursts, link interruptions, and widespread device failures, to comprehensively evaluate the performance, stability, and resilience of this invention under various pressures, thereby adjusting the system to its optimal state before deployment.

[0059] As an example of an embodiment of the present invention, taking an intelligent operation and maintenance scenario including 10 edge nodes (simulating deployment on 5G base stations in different areas of a city) and a central cloud server as an example, the steps 1-3 are as follows: Step 1, Initialization and Stable Operation, specifically includes steps 11-12, as follows: Step 11, System Startup: After the system using this method goes online, the central cloud server uses the core optimization algorithm to calculate the initial traffic splitting threshold of 0.85 for all 10 nodes based on the initial stable network state (e.g., the network latency from each node to the cloud is 20ms), and then distributes it.

[0060] Step 12, Stable Operation (Time t = 0s to t = 20s): During this period, the network condition is good. Most analysis tasks are processed at the edge nodes, resulting in relatively high confidence levels. Only complex tasks with confidence levels below 0.85 are offloaded to the cloud. The system as a whole operates in a "high accuracy pursuit" mode, running smoothly and efficiently.

[0061] Step 2, Unexpected Events and Adaptive Adjustment, specifically includes steps 21-23, as follows: Step 21: Monitor sudden events in the scenario (time t = 20s): A large-scale sports event is held in the city center, causing a surge in user traffic in the area. The uplink of the base stations deployed in the edge node 3 and edge node 4 in the area begins to be congested, and the network latency to the central cloud deteriorates sharply from 20ms to 120ms.

[0062] Step 22: Start of the new decision-making cycle (time t = 25s), specifically including steps 221-224, as follows: Step 221, State Awareness: The central cloud server enters a new 5-second decision cycle. It collects the latest network status from all nodes and immediately detects that the latency of nodes 3 and 4 has become severely abnormal.

[0063] Step 222: Utility Model Recalculation: When the core optimization algorithm starts running, the "latency cost" becomes enormous in the utility calculation for nodes 3 and 4 due to the high latency of 120ms. At this point, the "accuracy gain" brought by offloading the task to the cloud is far from enough to compensate for its huge "time cost".

[0064] Step 223, Optimization and Results: During iterative solution processing, in order to maximize the overall system utility, the algorithm automatically shifts its optimal solution towards "reducing latency at all costs." Therefore, the new optimal traffic splitting thresholds calculated for nodes 3 and 4 will be significantly reduced, for example, from 0.85 to 0.40. Simultaneously, since the cloud is expected to receive fewer tasks from nodes 3 and 4, resulting in a lighter load, the algorithm may fine-tune the thresholds for the other eight normal nodes, raising them to 0.88 to further improve the analysis accuracy in these areas.

[0065] Step 224, Threshold Distribution and Behavior Adjustment: The central cloud will distribute this new set of differentiated thresholds (0.40 for nodes 3 / 4 and 0.88 for the rest).

[0066] Step 23, New System Steady State (Time t > 25s): After nodes 3 and 4 received and applied the low threshold of 0.40, their task offloading strategies changed dramatically. Now, the vast majority of tasks (as long as the confidence level is higher than 0.40) are forced to be processed locally, greatly reducing data transmission to congested links. While this may sacrifice the analytical accuracy for a very small number of complex tasks, it ensures that the vast majority of operational tasks in this area receive timely responses, avoiding the risk of business interruption due to excessive latency. The other 8 nodes continue to operate smoothly in "high accuracy" mode.

[0067] Step 3, System Data Analysis: Edge nodes have faster inference speeds and tend to analyze common errors, enabling rapid troubleshooting of common errors. Cloud nodes, through deeper training and specific data augmentation, can achieve higher accuracy in troubleshooting error categories with fewer samples.

[0068] like Figure 9 As shown, based on the above-described method embodiments, an embodiment of the present invention also provides a data processing system 900, including: a threshold module 901, an initial processing module 902, and a splitting processing module 903; The threshold module 901 is used to obtain a real-time traffic splitting threshold, wherein the real-time traffic splitting threshold is obtained by the target server through periodically executing an iterative coordinate ascent algorithm and solving the objective function based on the state information of multiple first edge nodes. The objective function is constructed based on task analysis accuracy and latency cost. The target server is communicatively connected to multiple first edge nodes. The initial processing module 902 is used to analyze the data processing task based on the first analysis model to obtain the initial processing results and confidence level. The traffic splitting module 903 is used to compare the confidence level with the real-time traffic splitting threshold. If the confidence level is greater than the real-time traffic splitting threshold, the initial processing result is used as the first processing result. Otherwise, the data processing task is offloaded to the target server for processing by the second analysis model of the target server to obtain the second processing result.

[0069] It is understood that the above system embodiments correspond to the method embodiments of the present invention, and can implement the data processing method provided by any of the above method embodiments of the present invention.

[0070] It should be noted that the system embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0071] For ease of description and brevity, the system embodiments of the present invention include all the implementation methods described in the above data processing method embodiments, and will not be repeated here.

[0072] Based on the above-described embodiments of the data processing method, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the data processing method of any embodiment of the present invention.

[0073] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0074] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0075] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0076] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the data processing method described in any of the above-described method embodiments of the present invention.

[0077] Based on the above-described method embodiments, this invention also provides a computer program / program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of any of the above-described method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0078] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0079] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, Executed by the first edge node, including: The real-time traffic splitting threshold is obtained by the target server through periodically executing an iterative coordinate ascent algorithm and solving the objective function based on the state information of multiple first edge nodes. The objective function is constructed based on task analysis accuracy and latency cost. The target server communicates with multiple first edge nodes. The data processing task is analyzed based on the first analysis model to obtain the initial processing results and confidence level; The confidence level is compared with the real-time diversion threshold. If the confidence level is greater than the real-time diversion threshold, the initial processing result is taken as the first processing result. Otherwise, the data processing task is offloaded to the target server for processing by the second analysis model of the target server to obtain the second processing result.

2. The data processing method as described in claim 1, characterized in that, The real-time traffic splitting threshold is obtained by the target server periodically executing an iterative coordinate ascent algorithm and solving the objective function based on the state information of multiple first edge nodes, specifically: In each iteration, the target server sequentially iterates over each first edge node as the current optimization node, fixes the first diversion threshold of all first edge nodes except the current optimization node, updates the first diversion threshold of the current optimization node, and obtains the second diversion threshold of the current optimization node when the function value of the objective function is maximized. The next first edge node is iteratively optimized based on the second diversion threshold and each of the first diversion thresholds until all first edge nodes have been traversed, and then the next iteration begins. When the threshold change of all edge nodes is less than a preset convergence threshold, the iteration stops, and the real-time diversion threshold of the first edge node is obtained.

3. The data processing method as described in claim 2, characterized in that, The value of the objective function is obtained by calculating the objective cost and the objective benefit, specifically: For each of the first edge nodes, predictions are made based on historical task data to obtain the predicted task data for the current period. The predicted task data is then divided according to the first diversion threshold to obtain the predicted task unloading volume and the predicted task processing volume. The target cost is calculated based on the predicted task unloading volume and the status information, and the target revenue corresponding to the task analysis accuracy is calculated based on the predicted task processing volume. The function value is calculated based on the target cost and the target benefit for each of the first edge nodes.

4. The data processing method as described in claim 3, characterized in that, The target cost is calculated based on the predicted task unloading volume and the status information, specifically as follows: Periodically acquire status information, wherein the status information includes network latency information from the first edge node to the target server and the current load status of the target server; Based on the predicted task offloading amount, the network latency information, and the current load status, a first latency is obtained through a queuing theory model, and a first latency cost is calculated based on the first latency and the first latency weight coefficient, wherein the first latency is used to characterize the latency of offloading the predicted task offloading amount to the target server; The second latency cost is calculated based on the predicted task processing volume and the second latency weighting coefficient. The sum of the first latency cost and the second latency cost is then calculated to obtain the target cost. The second latency cost is used to characterize the latency corresponding to the first edge node processing the predicted task.

5. The data processing method as described in claim 1, characterized in that, The data processing method further includes: The edge nodes use a support vector machine model or a gradient boosting decision tree model as the first analysis model to analyze the data processing task; The target server uses any one of the following models as the second analysis model: graph neural network model, long short-term memory network model, or deep learning model based on Transformer, to analyze the data processing tasks unloaded from each edge node.

6. The data processing method as described in claim 2, characterized in that, The update of the first traffic splitting threshold of the current optimized node is specifically as follows: The first diversion threshold of the current optimized node is updated using a numerical search method.

7. A data processing system, characterized in that, include: Threshold module, initial processing module, and splitting processing module; The threshold module is used to obtain a real-time traffic splitting threshold, wherein the real-time traffic splitting threshold is obtained by the target server through periodically executing an iterative coordinate ascent algorithm and solving the objective function based on the state information of multiple first edge nodes. The objective function is constructed based on task analysis accuracy and latency cost. The target server is communicatively connected to multiple first edge nodes. The initial processing module is used to analyze the data processing task based on the first analysis model to obtain the initial processing results and confidence level. The traffic splitting module is used to compare the confidence level with the real-time traffic splitting threshold. If the confidence level is greater than the real-time traffic splitting threshold, the initial processing result is used as the first processing result. Otherwise, the data processing task is offloaded to the target server for processing by the second analysis model of the target server to obtain the second processing result.

8. A terminal device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the data processing method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, include: A stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the data processing method as described in any one of claims 1-6.

10. A computer program product, characterized in that, include: Computer instructions, when executed by a processor, implement the steps of the data processing method as described in any one of claims 1 to 6.

Images