A data processing method and system based on cloud computing

Abstract

The present invention relates to the field of cloud computing technology and discloses a cloud computing-based data processing method and system. Addressing the dynamic changes in multi-node data streams and communication delays, this method uses time window filtering, bandwidth occupancy analysis, and other techniques to identify and eliminate invalid gradient data. It also employs an asynchronous update mechanism and delayed node isolation technology to address nodes with excessive communication delays. This system optimizes computing resource allocation by combining parallel node screening and asynchronous task distribution. Finally, it utilizes a delay compensation strategy to coordinate the gradient update time of each node, achieving efficient synchronization of gradient data in a distributed computing environment. This method effectively addresses the issue of data inconsistency caused by communication delays in distributed environments, improves the accuracy and efficiency of gradient synchronization, and provides reliable technical support for the training of large-scale distributed machine learning models.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing, and in particular to a data processing method and system based on cloud computing. Background Art

[0002] As a core technology for data processing and intelligent analysis, cloud computing is crucial for driving technological innovation in the big data era. Its powerful computing capabilities and flexible resource allocation make it a key player in massive data processing, model training, and real-time analysis.

[0003] However, current cloud computing-based data processing methods still have significant limitations when faced with large-scale data. For example, distributed computing frameworks rely on data sharding storage, but task scheduling may cause large amounts of cross-node data transmission due to mismatches between data and computing nodes. Data migration across racks or data centers will significantly increase network I / O latency. Traditional schedulers are based on static resource pool divisions and cannot perceive task priorities or data flow changes in real time. In bursty traffic scenarios, fixed resource quotas will cause some tasks to starve. Stream processing systems need to maintain operator states for fault tolerance, but dynamic data flows may lead to state explosion (such as windows not being cleared in time) or long recovery times.

[0004] As a result, many solutions suffer from inefficiencies, poor resource scheduling, and an inability to effectively handle dynamically changing data streams when processing data in distributed environments. These shortcomings often lead to performance bottlenecks when handling complex tasks, and even affect the accuracy of results due to data latency. Summary of the Invention

[0005] The present invention provides a data processing method and system based on cloud computing to improve the efficiency and accuracy of cloud computing.

[0006] In a first aspect, in order to solve the above technical problems, the present invention provides a data processing method based on cloud computing, comprising:

[0007] In a multi-node environment, obtaining a timestamp deviation between the generation time of each gradient data and the current system clock from each node, and marking the corresponding gradient data if the timestamp deviation exceeds a preset deviation threshold, thereby obtaining the marked gradient data;

[0008] filtering invalid portions of the marked gradient data to obtain a preliminarily filtered gradient data group, performing inter-node communication coordination on the preliminarily filtered gradient data group, determining whether a communication status satisfies synchronization requirements, and caching current gradient data if the communication status indicates a delay exceeding a preset delay threshold;

[0009] Identify and isolate nodes with communication delays based on the cached current gradient data, obtain the available computing resource distribution information, determine the set of nodes that can currently participate in parallel computing, confirm the list of available nodes, and assign tasks to nodes with low latency;

[0010] Refresh the availability status of the node set currently capable of participating in parallel computing in real time, determine the computing completion time of each node, perform time compensation processing on the data of delayed nodes, and obtain the adjusted gradient data update time.

[0011] Preferably, obtaining the timestamp deviation between the generation time of each piece of gradient data and the current system clock from each node in a multi-node environment includes:

[0012] Collecting data streams from each node, and if the data streams change dynamically, adjusting the data collection period according to the frequency of uploading by each node to obtain an initial data set including a timestamp;

[0013] Obtaining the current system clock in real time, and calculating the difference between the timestamp and the timestamp of the current system clock;

[0014] If the timestamp difference exceeds the preset deviation threshold, the generation time deviation of each gradient data is calculated to obtain a deviation value set;

[0015] If any value in the deviation value set exceeds the tolerance range of network reliability, the data synchronization strategy is adjusted to obtain the final timestamp deviation value.

[0016] Preferably, if the timestamp deviation value exceeds a preset deviation threshold, marking the corresponding gradient data to obtain the marked gradient data includes:

[0017] If the timestamp deviation value exceeds the preset deviation threshold, marking the corresponding gradient data to obtain marked intermediate data;

[0018] Counting the data packet loss rate of the intermediate data during the communication process;

[0019] If the data packet loss rate of any gradient data exceeds a preset loss threshold, the integrity of the gradient data is determined; if incomplete, the gradient data is marked as invalid, thereby obtaining the gradient data marked as invalid.

[0020] Preferably, filtering the invalid portion of the marked gradient data to obtain a preliminarily filtered gradient data set includes:

[0021] Eliminate the gradient data marked as invalid to obtain a preliminary cleaned data group;

[0022] Performing real-time detection on the communication channel to obtain the bandwidth occupancy of the communication channel;

[0023] If the bandwidth occupancy exceeds a preset occupancy threshold, load adjustment is performed on the communication channel to obtain adjusted channel load data;

[0024] The preliminarily cleaned data groups are secondary filtered according to the adjusted channel load data to obtain the preliminarily filtered gradient data group.

[0025] Preferably, the inter-node communication coordination of the preliminarily filtered gradient data group is performed to determine whether the communication status meets the synchronization requirement, and if the communication status shows that the delay exceeds a preset delay threshold, the current gradient data is cached, including:

[0026] Tracking the processing of the communication request by each node, obtaining the response time of each node, and comparing the response time with a preset response threshold;

[0027] If there is a node that exceeds the response threshold, it is marked as abnormal and the corresponding abnormal data group is obtained;

[0028] Transferring the abnormal data group to a temporary storage area for caching to obtain a cached data set;

[0029] According to the cached data set and the update status of the communication status, the synchronization conditions between the nodes are dynamically adjusted to obtain the adjusted synchronization parameters.

[0030] Preferably, identifying and isolating nodes with communication delays based on cached current gradient data, obtaining available computing resource distribution information, and determining a set of nodes currently available for parallel computing include:

[0031] Performing block processing on the cached current gradient data to obtain a block data group set;

[0032] Performing real-time detection on the inter-node communication status of the data packet set, marking the corresponding node if the transmission duration exceeds a delay threshold, and obtaining a list of marked delayed nodes;

[0033] According to the marked delay node list, unmarked nodes in the current computing distribution are scanned to obtain available resource data therefrom, and the available resource data are prioritized to obtain the set of nodes that can currently participate in parallel computing.

[0034] Preferably, confirming the list of available nodes and assigning the task to a node with lower latency includes:

[0035] According to the communication status and the preset communication threshold, the nodes that meet the conditions are judged and a filtered list of available nodes is obtained;

[0036] If it is detected that the communication status of some nodes is unstable, the task ratio of these nodes is reduced first, and the adjusted allocation plan is determined;

[0037] According to the adjusted allocation scheme, at least one data subset is obtained from the data group set, the data groups suitable for allocation are determined and mapped to the nodes in the available node list, and the task allocation ratio is dynamically adjusted according to the real-time data of communication delay during distribution to obtain the final task allocation scheme.

[0038] Preferably, the real-time refreshing of the availability status of the set of nodes currently capable of participating in parallel computing, determining the computing completion time of each node, and performing time compensation processing on the data of delayed nodes include:

[0039] Receive communication delays between nodes, and if the communication delay is higher than the preset delay threshold, mark the node as a high-delay node, and obtain a high-delay node list;

[0040] Performing time compensation on the gradient data of the high-delay node;

[0041] The compensation time is dynamically allocated according to the node performance data and the load data to obtain the adjusted gradient data update time.

[0042] Preferably,

[0043] After performing time compensation processing on the data of the delayed node and obtaining the adjusted gradient data update time, the following steps are also included:

[0044] Acquire historical data of each node, extract fluctuation characteristics from the historical data and obtain a communication state sequence, and generate a node communication state set;

[0045] Comparing the communication status of each node with a preset communication threshold, if the communication status of one node is lower than the preset communication threshold, marking it as a node to be adjusted;

[0046] Calibrate the gradient update time of the node to be adjusted, extract the deviation value from the calibrated time data, and determine the node time synchronization solution;

[0047] Integrate the gradient data of each node according to the node time synchronization scheme;

[0048] The gradient data of each node is uniformly verified according to the adjusted gradient data update time to obtain the update time deviation of each node. If the update time deviation is lower than the preset update threshold, the data synchronization is confirmed to be completed, and a synchronized gradient data set is obtained.

[0049] In a second aspect, the present invention provides a data processing system based on cloud computing, comprising:

[0050] The detection end is used to obtain the timestamp deviation between the generation time of each gradient data and the current system clock from each node in a multi-node environment. If the timestamp deviation exceeds a preset deviation threshold, the corresponding gradient data is marked to obtain the marked gradient data;

[0051] The processing end is configured to filter out invalid portions of the marked gradient data, obtain a preliminarily filtered gradient data group, coordinate inter-node communication on the preliminarily filtered gradient data group, determine whether the communication status meets synchronization requirements, and cache the current gradient data if the communication status indicates that the delay exceeds a preset delay threshold; identify and isolate nodes with communication delays based on the cached current gradient data, obtain available computing resource distribution information, determine the set of nodes currently available for parallel computing, confirm a list of available nodes, and assign tasks to nodes with low delays;

[0052] The update end is used to refresh the availability status of the node set currently capable of participating in parallel computing in real time, determine the computing completion time of each node, perform time compensation processing on the data of delayed nodes, and obtain the adjusted gradient data update time.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] (1) The present invention obtains the timestamp deviation value between the generation time of each gradient data and the current system clock from each node in a multi-node environment. If the timestamp deviation value exceeds a preset deviation threshold, the corresponding gradient data is marked to obtain the marked gradient data. From data collection, delay monitoring to deviation calibration, a complete technical closed loop is formed to ensure the real-time and accuracy of data stream processing in a multi-node environment.

[0055] (2) The present invention filters out the invalid part in the marked gradient data, obtains a preliminary filtered gradient data group, performs inter-node communication coordination on the preliminary filtered gradient data group, and determines whether the communication status meets the synchronization requirements. If the communication status shows that the delay exceeds the preset delay threshold, the current gradient data is cached. In the inter-node communication coordination, a complete logical chain from response time measurement to delay abnormality marking to cache processing is realized, providing technical guarantee for the reliable transmission of gradient data groups.

[0056] (3) The present invention identifies nodes with communication delays based on the cached current gradient data and isolates them, obtains the available computing resource distribution information, determines the set of nodes that can currently participate in parallel computing, confirms the list of available nodes, assigns tasks to nodes with low delays, and records the task allocation amount, execution time and optimization adjustment data of each node, forming a closed-loop process from node screening, task distribution to execution optimization.

[0057] (4) The present invention refreshes the availability status of the set of nodes currently capable of participating in parallel computing in real time, determines the computing completion time of each node, performs time compensation processing on the data of delayed nodes, obtains the adjusted gradient data update time, and forms a complete closed-loop logic from time determination, delay compensation to consistency verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of a data processing method based on cloud computing provided by an embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram of the structure of a data processing system based on cloud computing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0061] Reference Figure 1 , an embodiment of the present invention provides a flow chart of a data processing method based on cloud computing, comprising the following steps:

[0062] S1, in a multi-node environment, obtains the timestamp deviation between the generation time of each gradient data and the current system clock from each node. If the timestamp deviation exceeds a preset deviation threshold, the corresponding gradient data is marked to obtain the marked gradient data;

[0063] S2, filtering out invalid parts in the marked gradient data to obtain a preliminarily filtered gradient data group, performing inter-node communication coordination on the preliminarily filtered gradient data group, and determining whether the communication status meets the synchronization requirement; if the communication status shows that the delay exceeds a preset delay threshold, caching the current gradient data;

[0064] S3, based on the cached current gradient data, identifies and isolates nodes with communication delays, obtains information about the distribution of available computing resources, determines the set of nodes that can currently participate in parallel computing, confirms the list of available nodes, and assigns tasks to nodes with low latency;

[0065] S4, refreshing the availability status of the set of nodes currently capable of participating in parallel computing in real time, determining the computing completion time of each node, performing time compensation processing on the data of delayed nodes, and obtaining the adjusted gradient data update time.

[0066] In step S1, in a multi-node environment, a timestamp deviation value between the generation time of each gradient data and the current system clock is obtained from each node. If the timestamp deviation value exceeds a preset deviation threshold, the corresponding gradient data is marked to obtain the marked gradient data.

[0067] In the field of cloud computing, multi-node environments, gradient data, and timestamp skew are key concepts in distributed system design and data processing. A multi-node environment refers to a distributed cluster consisting of multiple computing nodes (physical machines / virtual machines / containers), where nodes collaborate through network communication to complete tasks. In distributed machine learning, gradient data refers to the direction and magnitude of model parameter optimization (i.e., the partial derivatives of the loss function with respect to the parameters) and is typically used to synchronize model updates. Timestamp skew refers to the difference in event timestamps caused by inconsistent clocks between different nodes in a distributed system.

[0068] Preferably, obtaining the timestamp deviation between the generation time of each piece of gradient data and the current system clock from each node in a multi-node environment includes:

[0069] Collecting data streams from each node, and if the data streams change dynamically, adjusting the data collection period according to the frequency of uploading by each node to obtain an initial data set including a timestamp;

[0070] Obtaining the current system clock in real time, and calculating the difference between the timestamp and the timestamp of the current system clock;

[0071] If the timestamp difference exceeds the preset deviation threshold, the generation time deviation of each gradient data is calculated to obtain a deviation value set;

[0072] If any value in the deviation value set exceeds the tolerance range of network reliability, the data synchronization strategy is adjusted to obtain the final timestamp deviation value.

[0073] In the field of cloud computing, generation time skew, skew value sets, and data synchronization strategies are core concepts closely related to distributed system time management, state consistency, and data coordination. Generation time skew refers to the difference between the generation timestamp of data / events and the actual time of occurrence in a distributed system. Skew value sets refer to the deviation of local metrics from the global average due to uneven data distribution or node performance differences in distributed computing. Data synchronization strategies refer to the protocols and methods for coordinating data consistency across multiple replicas or nodes in a distributed system. The core trade-off is consistency and availability.

[0074] Specifically, in a multi-node distributed computing framework, we first use a pre-built cluster environment to dynamically distribute and collect data streams using an Apache Kafka-based message queue system. Assume that the cluster contains 10 computing nodes, and each node generates approximately 100MB of gradient data per second, along with timestamp information. For example, node 1 generates gradient data at 2023-10-01 10:00:00.123. The data stream is distributed to the master node through the Kafka topic partitioning mechanism. The master node records the receipt time as 2023-10-01 10:00:01.223, and the preliminary calculated transmission delay is 1.1 seconds.

[0075] Next, communication delay monitoring technology was used to measure the inter-node delay in real time using the Ping-Pong mechanism. The specific method was that the master node sent a 64-byte test packet to each node every 5 seconds and recorded the round-trip time (RTT). Assuming that the RTT of node 1 was 200ms and that of node 2 was 300ms, the average of the past 10 RTTs was calculated, and the average delay of node 1 was 210ms. Further analysis showed that the standard deviation of the delay fluctuation was 15ms, which determined its communication stability.

[0076] Next, the time deviation of each data generation is calculated based on the timestamp of the gradient data and the system clock. The master node's system clock is 2023-10-01 10:00:02.000. Combined with the data reception time and average latency of Node 1, the time deviation is estimated to be 1.79 seconds (i.e., 2.000 - 0.123 - 0.210). If the deviation exceeds the preset threshold of 1.5 seconds, the time synchronization protocol NTP is triggered to calibrate the node clock to ensure subsequent data consistency. To form a logical chain, assuming that the deviation is calibrated, the system further analyzes the gradient data quality. If a node's deviation exceeds the standard five times in a row, its data weight is reduced to 0.5 to optimize the global model aggregation effect.

[0077] Through the above method, a complete technical closed loop is formed from data collection, delay monitoring to deviation calibration to ensure the real-time and accuracy of data stream processing in a multi-node environment.

[0078] Preferably, if the timestamp deviation value exceeds a preset deviation threshold, marking the corresponding gradient data to obtain the marked gradient data includes:

[0079] If the timestamp deviation value exceeds the preset deviation threshold, marking the corresponding gradient data to obtain marked intermediate data;

[0080] Counting the data packet loss rate of the intermediate data during the communication process;

[0081] If the data packet loss rate of any gradient data exceeds a preset loss threshold, the integrity of the gradient data is determined; if incomplete, the gradient data is marked as invalid, thereby obtaining the gradient data marked as invalid.

[0082] In cloud computing, packet loss rate (PLR) refers to the ratio of the number of packets lost during network transmission to the total number of packets sent, typically expressed as a percentage. It is a key indicator of network communication reliability and quality, directly impacting the performance, consistency, and user experience of distributed systems.

[0083] Specifically, in a multi-node distributed computing framework, the system uses a time window filtering mechanism to address timestamp deviations in gradient data. Assume a cluster consists of eight nodes, each generating 50MB of gradient data per second, with a timestamp format of 2023-11-01 09:00:00.456. The master node sets a one-second time window to collect the gradient data uploaded by each node and filter out data with a deviation exceeding 1.2 seconds.

[0084] For example, if the timestamp for node 3 is 2023-11-01 09:00:00.500 and the master node receives it at 2023-11-01 09:00:02.000, the deviation is calculated as 2.000 - 0.500 = 1.5 seconds, exceeding the threshold by 1.2 seconds. Therefore, the data is marked for analysis. The system then calculates the packet loss rate, using a window size of 20 transmissions, and records the loss rate for the most recent 20 transmissions from node 3. Assuming 8 transmissions were lost, the loss rate is calculated as 8 / 20 = 0.4. Combined with historical data, the average loss rate for node 3 is 0.35, with a standard deviation of 0.05, indicating minimal fluctuation in the loss rate and a relatively stable communication link. Based on this, the system marks data with an excessive deviation and a loss rate higher than 0.3 as invalid, generating an invalid dataset.

[0085] For example, three of the ten gradient data sets from node 3 were marked as invalid due to a deviation of 1.5 seconds and a loss rate of 0.4. To ensure model training quality, the system filtered out the invalid data, and the remaining data was aggregated globally. The weight assigned to node 3's data was reduced to 0.6, prioritizing data from other nodes. The invalid data sets were stored in the Redis cache for subsequent analysis of the cause of the communication anomaly, forming a closed-loop processing process from deviation detection, loss rate analysis, and data filtering.

[0086] In step S2, invalid portions of the marked gradient data are filtered to obtain a preliminarily filtered gradient data set. Inter-node communication coordination is performed on the preliminarily filtered gradient data set to determine whether the communication status meets synchronization requirements. If the communication status indicates that the delay exceeds a preset delay threshold, the current gradient data is cached.

[0087] Preferably, filtering the invalid portion of the marked gradient data to obtain a preliminarily filtered gradient data set includes:

[0088] Eliminate the gradient data marked as invalid to obtain a preliminary cleaned data group;

[0089] Performing real-time detection on the communication channel to obtain the bandwidth occupancy of the communication channel;

[0090] If the bandwidth occupancy exceeds a preset occupancy threshold, load adjustment is performed on the communication channel to obtain adjusted channel load data;

[0091] The preliminarily cleaned data groups are secondary filtered according to the adjusted channel load data to obtain the preliminarily filtered gradient data group.

[0092] Specifically, in a distributed computing cluster, for gradient data sets marked as invalid, the system first performs a pruning operation on the original data to remove the invalid portions. It then combines bandwidth occupancy analysis to detect the usage of inter-node communication channels, ultimately generating a preliminarily filtered gradient data set. Consider a 10-node cluster, with each node generating 100MB of gradient data per minute. Node 5 has 15 data points, with 4 of them marked as invalid. The system automatically compares the original data set with the invalid data set, removing these 4 points from node 5's original data. The remaining 11 points, representing 73.3% of the original data, are retained, generating a preliminarily filtered data set totaling 1100MB.

[0093] At the same time, the system activates a bandwidth utilization analysis module to monitor the usage of inter-node communication channels in real time. It calculates the bandwidth utilization of each node over the past 30 minutes. Assuming that node 5 has an average bandwidth utilization of 85.6% and a peak of 92.1%, compared to the cluster average of 78.2%, this indicates that node 5 is experiencing high communication pressure, potentially impacting data transmission stability. To further optimize data processing, the system correlates bandwidth utilization with data transmission success rate. The calculated transmission success rate for node 5 is 88.5%, lower than the cluster average of 91.3%. Therefore, the system automatically adjusts node 5's data priority, reducing its participation in subsequent aggregation to 0.7. This analysis result is recorded in the log database for future communication optimization. This initial filtering of the gradient data set ensures data volume while also providing a more reliable foundation for subsequent data aggregation, combined with the bandwidth utilization analysis results. This completes the logical chain from invalid data removal to communication status assessment.

[0094] Preferably, the inter-node communication coordination of the preliminarily filtered gradient data group is performed to determine whether the communication status meets the synchronization requirement, and if the communication status shows that the delay exceeds a preset delay threshold, the current gradient data is cached, including:

[0095] Tracking the processing of the communication request by each node, obtaining the response time of each node, and comparing the response time with a preset response threshold;

[0096] If there is a node that exceeds the response threshold, it is marked as abnormal and the corresponding abnormal data group is obtained;

[0097] Transferring the abnormal data group to a temporary storage area for caching to obtain a cached data set;

[0098] According to the cached data set and the update status of the communication status, the synchronization conditions between the nodes are dynamically adjusted to obtain the adjusted synchronization parameters.

[0099] In cloud computing, synchronized parameters typically refer to shared parameters or state variables that need to be maintained consistently across multiple nodes or processes in distributed systems or parallel computing environments. Maintaining the consistency of these parameters is crucial to the correctness, performance, and fault tolerance of the system.

[0100] Specifically, in a distributed computing cluster, when coordinating inter-node communication for a preliminarily filtered gradient data set, the system first uses node response speed measurement technology to obtain the response time of each node to the communication request.

[0101] For example, in a cluster consisting of 12 nodes, each node processes 100 communication requests per second. The system timestamps 1,000 test requests sent by node 7 and calculates its average response time to be 12.5 milliseconds, with a peak time of 18.7 milliseconds. The cluster average response time is 10.2 milliseconds, indicating that node 7's response speed is slow.

[0102] Next, the system determines the communication status based on the synchronization condition threshold (set to 15 milliseconds). Node 7's 18.7 millisecond peak exceeds the threshold, automatically adding a delay anomaly tag to Node 7's current data packet, marking it at a 20% rate and involving 200MB of data. For data marked as delay anomaly, the system activates a cache trigger mechanism, storing this data in the distributed cache system. Using a consistent hashing algorithm, it distributes the data to three cache nodes, setting the cache validity period to 5 minutes. The cache hit rate is also recorded, currently at 92.3%. To ensure the integrity of the logical chain, the system further analyzes the relationship between Node 7's latency and the cluster communication load, calculating a correlation coefficient of 0.82 between Node 7's latency and cluster load, indicating that high load may be the primary cause of the delay.

[0103] Therefore, the system automatically adjusts the communication priority of node 7 to 0.6 and writes the adjustment record to the distributed log system for reference in subsequent task scheduling optimization.

[0104] Through the above processing, the system realizes a complete logical chain from response time measurement to delay anomaly marking and then to cache processing in inter-node communication coordination, providing technical guarantee for the reliable transmission of gradient data groups.

[0105] In step S3, nodes with communication delays are identified and isolated based on the cached current gradient data, the available computing resource distribution information is obtained, the set of nodes that can currently participate in parallel computing is determined, the list of available nodes is confirmed, and tasks are assigned to nodes with low delays.

[0106] Among them, in the field of cloud computing, the available computing resource distribution information, the set of nodes that can currently participate in parallel computing, and the list of available nodes are the core concepts of resource scheduling and distributed task execution. The three are closely related but have different focuses.

[0107] Available Resource Distribution (Available Resource Distribution) refers to the real-time resource status of all nodes in the cluster. Active Parallel Computing Nodes (Active Parallel Computing Nodes) refers to the subset of nodes that can immediately participate in distributed task computations. Available Nodes List (Available Nodes List) refers to the set of all currently available nodes in the cluster, typically a superset of the Active Parallel Computing Nodes set.

[0108] Preferably, identifying and isolating nodes with communication delays based on cached current gradient data, obtaining available computing resource distribution information, and determining a set of nodes currently available for parallel computing include:

[0109] Performing block processing on the cached current gradient data to obtain a block data group set;

[0110] Performing real-time detection on the inter-node communication status of the data packet set, marking the corresponding node if the transmission duration exceeds a delay threshold, and obtaining a list of marked delayed nodes;

[0111] According to the marked delay node list, unmarked nodes in the current computing distribution are scanned to obtain available resource data therefrom, and the available resource data are prioritized to obtain the set of nodes that can currently participate in parallel computing.

[0112] Specifically, in a distributed computing environment, for the gradient data group processed by cache, the system first handles the communication delay problem between nodes through an asynchronous update mechanism, and uses delayed node isolation technology to automatically identify nodes with excessive communication delays and temporarily isolate them.

[0113] For example, in a cluster containing 15 nodes, the system detects that the average communication delay of node 3 reaches 25.3 milliseconds, which is much higher than the cluster average of 8.7 milliseconds and exceeds the preset threshold of 20 milliseconds. The system then calls the isolation protocol to temporarily suspend the communication task of node 3, setting the isolation time to 3 minutes. At the same time, it generates an isolation log to record the delay peak data and isolation timestamp for subsequent analysis.

[0114] During this process, the system uses resource allocation query technology to scan the distribution of available computing resources within the cluster and calculate the set of nodes currently eligible for parallel computing. For example, it selects 10 nodes from the remaining 14 nodes with loads below 60%. Node 5, with a CPU utilization of 42.1% and a memory usage of 35.6%, meets the requirements for parallel computing. The system further prioritizes these nodes, taking into account node latency and resource usage. Node 5 receives an overall score of 85.4, ranking first. The computational tasks for the gradient data set are then assigned to this set of nodes to ensure even distribution. To complete the logical chain, the system also analyzes the cause of latency for isolated node 3 and finds a high correlation of 0.78 between latency and network bandwidth utilization, suggesting that bandwidth bottlenecks are the primary factor. This automatically triggers bandwidth optimization scheduling, limiting node 3's network traffic to 70% of its original capacity. The optimized data is then returned to monitoring, forming a closed-loop process from latency identification to node isolation to resource reallocation.

[0115] Preferably, confirming the list of available nodes and assigning the task to a node with lower latency includes:

[0116] According to the communication status and the preset communication threshold, the nodes that meet the conditions are judged and a filtered list of available nodes is obtained;

[0117] If it is detected that the communication status of some nodes is unstable, the task ratio of these nodes is reduced first, and the adjusted allocation plan is determined;

[0118] According to the adjusted allocation scheme, at least one data subset is obtained from the data group set, the data groups suitable for allocation are determined and mapped to the nodes in the available node list, and the task allocation ratio is dynamically adjusted according to the real-time data of communication delay during distribution to obtain the final task allocation scheme.

[0119] Specifically, in a distributed computing environment, the system optimizes the identified set of parallel nodes, and through parallel node screening technology, evaluates the communication delay and computing power of each node in a cluster of 12 nodes to generate a list of available nodes.

[0120] For example, the system collected node communication delay data in real time and found that the average delay of node 7 was 6.2 milliseconds and that of node 9 was 7.8 milliseconds, both lower than the preset threshold of 10 milliseconds. The delay of node 11 was 12.4 milliseconds, which exceeded the threshold and was eliminated. Finally, 8 nodes were screened out, among which the CPU utilization of node 7 was 38.5% and the memory usage was 31.2%, meeting the conditions for efficient computing.

[0121] Next, the system utilized asynchronous task distribution technology to break the pending task into 32 subtasks, distributing them based on node latency and load. Node 7 was assigned four subtasks with a weight coefficient of 0.85, reflecting its high priority. During task distribution, the system retrieved gradient data from the cache, extracting a total of 2.6GB of pending data. After analyzing data access frequency, it confirmed that 80% of the data was concentrated in three hotspot partitions. The hotspot data was preferentially loaded into the local caches of Nodes 7 and 9, reducing access latency by approximately 15%. To ensure reasonable allocation, considering node computing power and data distribution, the estimated completion time for Node 7 was 42.3 seconds, and for Node 9 was 45.1 seconds, keeping the total planned execution time within 50 seconds.

[0122] Subsequently, the system tracked task progress and discovered that node 9's task execution efficiency was lower than expected. Analysis revealed that its memory bandwidth utilization was 85.2%, with a correlation coefficient of 0.72, suggesting a memory bottleneck. This automatically triggered memory optimization scheduling, adjusting node 9's cache allocation to 65%, improving execution efficiency by approximately 10%. Ultimately, the system generated a detailed task execution plan log, recording each node's task allocation, execution time, and optimization adjustment data, forming a closed-loop process from node screening, task distribution, to execution optimization.

[0123] In step S4, the availability status of the set of nodes currently capable of participating in parallel computing is refreshed in real time, the computing completion time of each node is determined, time compensation processing is performed on the data of the delayed nodes, and the adjusted gradient data update time is obtained.

[0124] Preferably, the real-time refreshing of the availability status of the set of nodes currently capable of participating in parallel computing, determining the computing completion time of each node, and performing time compensation processing on the data of delayed nodes include:

[0125] Receive communication delays between nodes, and if the communication delay is higher than the preset delay threshold, mark the node as a high-delay node, and obtain a high-delay node list;

[0126] Performing time compensation on the gradient data of the high-delay node;

[0127] The compensation time is dynamically allocated according to the node performance data and the load data to obtain the adjusted gradient data update time.

[0128] Specifically, in a distributed computing environment, the system determines the computation completion time of each node. Combined with historical data analysis, it concludes that the completion time for node A is 25.6 seconds, and for node B is 31.8 seconds, resulting in a 6.2-second difference between the nodes. Subsequently, to address the impact of communication delays, the system employs a delay compensation strategy. It detects that the communication delay between node B and the central server is 3.4 milliseconds, while that for node A is 1.7 milliseconds, a significant delay difference. Therefore, the system adjusts node B's data transmission time, calculating a compensation coefficient of 1.2. This automatically advances node B's data transmission time by 1.2 times (3.4 milliseconds), or 4.08 milliseconds, ensuring data arrival synchronization with node A. The calculation of the compensation coefficient is typically related to resource allocation, service reliability, fault recovery, and quality of service (QoS) assurance.

[0129] For reference, after step S4, it also includes S5:

[0130] In step S5, historical data of each node is obtained, fluctuation characteristics are extracted from the historical data, and a communication state sequence is obtained to generate a node communication state set;

[0131] Comparing the communication status of each node with a preset communication threshold, if the communication status of one node is lower than the preset communication threshold, marking it as a node to be adjusted;

[0132] Calibrate the gradient update time of the node to be adjusted, extract the deviation value from the calibrated time data, and determine the node time synchronization solution;

[0133] Integrate the gradient data of each node according to the node time synchronization scheme;

[0134] The gradient data of each node is uniformly verified according to the adjusted gradient data update time to obtain the update time deviation of each node. If the update time deviation is lower than the preset update threshold, the data synchronization is confirmed to be completed, and a synchronized gradient data set is obtained.

[0135] In cloud computing, update time skew refers to the inconsistency in the timestamps of updates to the same data across different nodes or replicas within a distributed system. This creates challenges in coordinating data consistency. This skew can lead to data conflicts, inconsistent states, or performance degradation.

[0136] Specifically, in a distributed computing environment, the system analyzes the network transmission performance of each node through communication status logs. The system records the communication status data between nodes C and D over the past 10 minutes and finds that the average packet transmission time for node C is 15.2 milliseconds, while that for node D is 18.7 milliseconds, a difference of 3.5 milliseconds. Setting a window size of 5 minutes, the system extracts the packet loss rate for node C to be 0.8% and that for node D to be 1.3%. Combining this with historical data, the system calculates the transmission stability coefficient for node D to be 0.85, lower than the 0.92 for node C. To improve data synchronization accuracy, the system allocates an additional 12.5% ​​of bandwidth to node D based on the current network load of 58.3%. The adjusted transmission time is expected to be reduced to 16.4 milliseconds. The system then calculates the gradient update value based on the adjusted transmission time. The gradient contribution value for node C is 0.73, and that for node D is 0.69. Combining the data from the two nodes yields a global gradient update value of 0.71.

[0137] The system further analyzed the direction of model parameter adjustment and, using gradient descent, set a learning rate of 0.01 and calculated parameter updates of 0.0071, determined that the model weights were adjusted in a positive direction. To ensure synchronized output, the system used a timestamp verification mechanism to detect that the output time of node C was 22.4 seconds and that of node D was 22.6 seconds, with a time difference of 0.2 seconds, meeting the synchronization accuracy requirements. If an anomaly is detected, the system automatically triggers data verification and analyzes bandwidth utilization, which is currently 65.2%. The remaining bandwidth supports 4.7% of data retransmission requirements, ensuring that all nodes complete synchronization within the next cycle, forming a closed-loop process from status recording, bandwidth adjustment, and parameter optimization.

[0138] The system calibrates the gradient update time of the adjusted data, setting the update window to 2.5 seconds. It detects that the gradient data update time for node A is 27.3 seconds, and that for node B after adjustment is 27.5 seconds, reducing the time difference to 0.2 seconds, meeting the consistency requirements. To further ensure coordination and consistency, the system introduces a gradient synchronization verification mechanism, analyzing the time distribution of the overall gradient update and finding that the update time difference for 99.5% of the nodes is within 0.3 seconds, determining that the overall consistency meets the standard. At the same time, for the remaining 0.5% of abnormal nodes, the system automatically triggers the data retransmission mechanism. Combined with bandwidth utilization analysis, the current bandwidth utilization is 62.7%, and the retransmission bandwidth requirement is 8.2%. This ensures that abnormal nodes complete the update within the next cycle, forming a complete closed-loop logic from time determination, delay compensation, to consistency verification.

[0139] In summary, the present invention discloses a cloud computing-based data processing method. This method addresses the dynamic changes in multi-node data streams and communication delays. It uses time window filtering, bandwidth occupancy analysis, and other techniques to identify and eliminate invalid gradient data. It also employs an asynchronous update mechanism and delayed node isolation technology to handle nodes with excessive communication delays. It optimizes computing resource allocation by combining parallel node screening and asynchronous task distribution. Finally, it utilizes a delay compensation strategy to coordinate the gradient update time of each node, achieving efficient synchronization of gradient data in a distributed computing environment. This method effectively addresses the data inconsistency issue caused by communication delays in distributed environments, improves the accuracy and efficiency of gradient synchronization, and provides reliable technical support for the training of large-scale distributed machine learning models.

[0140] Reference Figure 2 , an embodiment of the present invention provides a schematic diagram of a data processing system structure based on cloud computing, including:

[0141] The detection end is used to obtain the timestamp deviation between the generation time of each gradient data and the current system clock from each node in a multi-node environment. If the timestamp deviation exceeds a preset deviation threshold, the corresponding gradient data is marked to obtain the marked gradient data;

[0142] The processing end is configured to filter out invalid portions of the marked gradient data, obtain a preliminarily filtered gradient data group, coordinate inter-node communication on the preliminarily filtered gradient data group, determine whether the communication status meets synchronization requirements, and cache the current gradient data if the communication status indicates that the delay exceeds a preset delay threshold; identify and isolate nodes with communication delays based on the cached current gradient data, obtain available computing resource distribution information, determine the set of nodes currently available for parallel computing, confirm a list of available nodes, and assign tasks to nodes with low delays;

[0143] The update end is used to refresh the availability status of the node set currently capable of participating in parallel computing in real time, determine the computing completion time of each node, perform time compensation processing on the data of delayed nodes, and obtain the adjusted gradient data update time.

[0144] It should be noted that the cloud computing-based data processing system provided in an embodiment of the present invention is used to execute all the process steps of the cloud computing-based data processing method in the above embodiment. The working principles and beneficial effects of the two correspond one to one, so they will not be repeated here.

[0145] The embodiment of the present invention further provides a terminal device. The terminal device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned embodiments of the data processing method based on cloud computing are implemented, such as Figure 1 Alternatively, the processor implements the functions of the modules / units in the above-mentioned system embodiments when executing the computer program.

[0146] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0147] The terminal device may be a computing device such as a desktop computer, laptop, PDA, or smart tablet. The terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that the aforementioned components are merely examples of terminal devices and do not constitute a limitation of the terminal device. The terminal device may include more or fewer components than those described above, or a combination of certain components or different components. For example, the terminal device may also include input / output devices, network access devices, buses, and the like.

[0148] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. The processor is the control center of the terminal device and connects various parts of the entire terminal device using various interfaces and lines.

[0149] The memory can be used to store the computer programs and / or modules. The processor implements the various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and accessing the data stored in the memory. The memory may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function or an image playback function); the data storage area may store data generated based on the use of the mobile phone (such as audio data, a phone book, etc.). Furthermore, the memory may include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0150] If the module / unit integrated into the terminal device is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the process steps in the above-mentioned method embodiments by using a computer program to instruct the relevant hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or system capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium can be appropriately increased or decreased based on the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, based on legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0151] It should be noted that the system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the system embodiment provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0152] The specific embodiments described above further illustrate the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A data processing method based on cloud computing, characterized in that: include: In a multi-node environment, obtaining a timestamp deviation between the generation time of each gradient data and the current system clock from each node, and marking the corresponding gradient data if the timestamp deviation exceeds a preset deviation threshold, thereby obtaining the marked gradient data; filtering invalid portions of the marked gradient data to obtain a preliminarily filtered gradient data group, performing inter-node communication coordination on the preliminarily filtered gradient data group, determining whether a communication status satisfies synchronization requirements, and caching current gradient data if the communication status indicates a delay exceeding a preset delay threshold; Identify and isolate nodes with communication delays based on the cached current gradient data, obtain the available computing resource distribution information, determine the set of nodes that can currently participate in parallel computing, confirm the list of available nodes, and assign tasks to nodes with low latency; Refresh the availability status of the node set currently capable of participating in parallel computing in real time, determine the computing completion time of each node, perform time compensation processing on the data of delayed nodes, and obtain the adjusted gradient data update time.

2. The cloud computing-based data processing method according to claim 1, characterized in that: The method of obtaining the timestamp deviation between the generation time of each gradient data and the current system clock from each node in a multi-node environment includes: Collecting data streams from each node, and if the data streams change dynamically, adjusting the data collection period according to the frequency of uploading by each node to obtain an initial data set including a timestamp; Obtaining the current system clock in real time, and calculating the difference between the timestamp and the timestamp of the current system clock; If the timestamp difference exceeds the preset deviation threshold, the generation time deviation of each gradient data is calculated to obtain a deviation value set; If any value in the deviation value set exceeds the tolerance range of network reliability, the data synchronization strategy is adjusted to obtain the final timestamp deviation value.

3. The cloud computing-based data processing method according to claim 2, characterized in that: If the timestamp deviation value exceeds a preset deviation threshold, marking the corresponding gradient data to obtain the marked gradient data includes: If the timestamp deviation value exceeds the preset deviation threshold, marking the corresponding gradient data to obtain marked intermediate data; Counting the data packet loss rate of the intermediate data during the communication process; If the data packet loss rate of any gradient data exceeds a preset loss threshold, the integrity of the gradient data is determined; if incomplete, the gradient data is marked as invalid, thereby obtaining the gradient data marked as invalid.

4. The cloud computing-based data processing method according to claim 3, characterized in that: The filtering of the invalid portion of the marked gradient data to obtain a preliminarily filtered gradient data set includes: Eliminate the gradient data marked as invalid to obtain a preliminary cleaned data group; Performing real-time detection on the communication channel to obtain the bandwidth occupancy of the communication channel; If the bandwidth occupancy exceeds a preset occupancy threshold, load adjustment is performed on the communication channel to obtain adjusted channel load data; The preliminarily cleaned data groups are secondary filtered according to the adjusted channel load data to obtain the preliminarily filtered gradient data group.

5. The cloud computing-based data processing method according to claim 4, characterized in that: The inter-node communication coordination of the preliminarily filtered gradient data group is performed to determine whether the communication status meets the synchronization requirement, and if the communication status shows that the delay exceeds a preset delay threshold, the current gradient data is cached, including: Tracking the processing of the communication request by each node, obtaining the response time of each node, and comparing the response time with a preset response threshold; If there is a node that exceeds the response threshold, it is marked as abnormal and the corresponding abnormal data group is obtained; Transferring the abnormal data group to a temporary storage area for caching to obtain a cached data set; According to the cached data set and the update status of the communication status, the synchronization conditions between the nodes are dynamically adjusted to obtain the adjusted synchronization parameters.

6. The cloud computing-based data processing method according to any one of claims 1 to 5, characterized in that: The method of identifying and isolating nodes with communication delays based on the cached current gradient data, obtaining available computing resource distribution information, and determining a set of nodes that can currently participate in parallel computing includes: Performing block processing on the cached current gradient data to obtain a block data group set; Performing real-time detection on the inter-node communication status of the data packet set, marking the corresponding node if the transmission duration exceeds a delay threshold, and obtaining a list of marked delayed nodes; According to the marked delay node list, unmarked nodes in the current computing distribution are scanned to obtain available resource data therefrom, and the available resource data are prioritized to obtain the set of nodes that can currently participate in parallel computing.

7. The cloud computing-based data processing method according to claim 6, characterized in that: Confirming the list of available nodes and assigning tasks to nodes with lower latency includes: According to the communication status and the preset communication threshold, the nodes that meet the conditions are judged and a filtered list of available nodes is obtained; If it is detected that the communication status of some nodes is unstable, the task ratio of these nodes is reduced first, and the adjusted allocation plan is determined; According to the adjusted allocation scheme, at least one data subset is obtained from the data group set, the data groups suitable for allocation are determined and mapped to the nodes in the available node list, and the task allocation ratio is dynamically adjusted according to the real-time data of communication delay during distribution to obtain the final task allocation scheme.

8. The cloud computing-based data processing method according to claim 6, characterized in that: The real-time refreshing of the availability status of the set of nodes currently capable of participating in parallel computing, determining the computing completion time of each node, and performing time compensation processing on the data of delayed nodes include: Receive communication delays between nodes, and if the communication delay is higher than the preset delay threshold, mark the node as a high-delay node, and obtain a high-delay node list; Performing time compensation on the gradient data of the high-delay node; The compensation time is dynamically allocated according to the node performance data and the load data to obtain the adjusted gradient data update time.

9. The cloud computing-based data processing method according to claim 6, characterized in that: After performing time compensation processing on the data of the delayed node and obtaining the adjusted gradient data update time, the following steps are also included: Acquire historical data of each node, extract fluctuation characteristics from the historical data and obtain a communication state sequence, and generate a node communication state set; Comparing the communication status of each node with a preset communication threshold, if the communication status of one node is lower than the preset communication threshold, marking it as a node to be adjusted; Calibrate the gradient update time of the node to be adjusted, extract the deviation value from the calibrated time data, and determine the node time synchronization solution; Integrate the gradient data of each node according to the node time synchronization scheme; The gradient data of each node is uniformly verified according to the adjusted gradient data update time to obtain the update time deviation of each node. If the update time deviation is lower than the preset update threshold, the data synchronization is confirmed to be completed, and a synchronized gradient data set is obtained.

10. A data processing system based on cloud computing, characterized in that: include: The detection end is used to obtain the timestamp deviation between the generation time of each gradient data and the current system clock from each node in a multi-node environment. If the timestamp deviation exceeds a preset deviation threshold, the corresponding gradient data is marked to obtain the marked gradient data; The processing end is configured to filter out invalid portions of the marked gradient data, obtain a preliminarily filtered gradient data group, coordinate inter-node communication on the preliminarily filtered gradient data group, determine whether the communication status meets synchronization requirements, and cache the current gradient data if the communication status indicates that the delay exceeds a preset delay threshold; identify and isolate nodes with communication delays based on the cached current gradient data, obtain available computing resource distribution information, determine the set of nodes currently available for parallel computing, confirm a list of available nodes, and assign tasks to nodes with low delays; The update end is used to refresh the availability status of the node set currently capable of participating in parallel computing in real time, determine the computing completion time of each node, perform time compensation processing on the data of delayed nodes, and obtain the adjusted gradient data update time.

Images