Cloud computing data acquisition and transmission method and system of distributed architecture

Abstract

The invention relates to the field of data processing, in particular to a cloud computing data acquisition and transmission method and system of a distributed architecture, and the method comprises the steps: obtaining real-time state information of each data source in a cloud computing environment in response to an acquisition instruction; based on the real-time state information, analyzing the acquisition instruction into a plurality of acquisition sub-tasks, and distributing the acquisition sub-tasks to corresponding acquisition nodes according to the real-time state information; executing the collection subtask through the collection node to obtain original data, and locally processing the original data to generate protocol data; identifying attribute information and a real-time network state of the protocol data, and determining a transmission strategy for each acquisition node according to the attribute information and the real-time network state of the protocol data; and based on the transmission strategy, sending the protocol data to the central node through the acquisition node. The transmission strategy is intelligently determined by integrating the internal attribute of the protocol data and the external real-time network state, and the adaptive process of optimal balance can be made according to the specific situation.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a distributed architecture cloud computing data acquisition and transmission method and system. Background Technology

[0002] Distributed architecture cloud computing data acquisition and transmission is designed to meet the needs of massive, multi-source, and heterogeneous data acquisition. It involves deploying a group of geographically or logically dispersed acquisition nodes in a cloud platform or hybrid IT environment to work together to complete the process of collecting and transmitting data from the source to the central storage or processing system.

[0003] First, existing methods often treat data acquisition, task scheduling, local processing, and data transmission as relatively isolated stages. Decisions in each stage (such as task allocation, processing methods, and transmission protocol selection) typically rely on local, static, or pre-defined rules, lacking a collaborative optimization mechanism based on global real-time state awareness. This makes it difficult for the system to adapt to dynamic changes in data source status, node load, and network conditions. Decisions in each stage may hinder each other, failing to form an efficient closed loop, thus limiting the overall system's adaptability and resource utilization efficiency.

[0004] Secondly, at the task scheduling and resource management level, existing distributed data acquisition systems lack sufficient elasticity and resilience when facing heterogeneous node performance, dynamic load fluctuations, and sudden failures. Specifically, most methods employ static or semi-static task allocation strategies, failing to perform fine-grained dynamic matching and adjustment based on data source characteristics and real-time node load. When some nodes become overloaded due to handling complex tasks or encountering resource contention, the system lacks effective prediction and intervention methods, potentially leading to task execution stagnation or data acquisition delays. Furthermore, for running tasks, existing solutions typically lack lightweight, low-overhead state preservation and migration mechanisms, making it difficult to seamlessly transfer tasks between nodes to cope with failures or achieve load balancing, thus impacting the overall reliability and continuity of the system. Summary of the Invention

[0005] In order to solve one or more problems in the prior art, the main objective of this application is to provide a distributed architecture cloud computing data acquisition and transmission method and system.

[0006] To achieve the aforementioned objectives, this application proposes a distributed architecture cloud computing data acquisition and transmission method, the method comprising: In response to the collection command, it obtains real-time status information of various data sources in the cloud computing environment; Based on the real-time status information, the acquisition command is parsed into multiple acquisition sub-tasks, and each acquisition sub-task is distributed to the corresponding acquisition node according to the real-time status information. Receive the acceptance response information returned by each acquisition node for the assigned acquisition subtask, and mark the acquisition subtask as assigned based on the acceptance response information; Based on the labeling results, the acquisition subtask is executed through the acquisition node corresponding to the acquisition subtask to obtain the raw data, and the raw data is processed locally to generate the reduced data. Identify the attribute information and real-time network status of the protocol data, and determine a transmission strategy for each acquisition node based on the attributes of the protocol data and the real-time network status. Based on the transmission strategy, the protocol data is sent to the central node through the acquisition node.

[0007] This application also provides a distributed architecture cloud computing data acquisition and transmission system, including: The acquisition module is used to respond to acquisition commands and acquire real-time status information of various data sources in the cloud computing environment; The parsing module is used to parse the acquisition command into multiple acquisition sub-tasks based on the real-time status information, and distribute each acquisition sub-task to the corresponding acquisition node according to the real-time status information. The receiving module is used to receive the acceptance response information returned by each acquisition node for the assigned acquisition sub-task, and mark the acquisition sub-task as assigned based on the acceptance response information. The generation module is used to execute the acquisition subtask through the acquisition node corresponding to the acquisition subtask based on the labeling results to obtain the raw data, and process the raw data locally to generate the reduced data. The identification module is used to identify the attribute information and real-time network status of the protocol data, and determine the transmission strategy for each acquisition node based on the attributes of the protocol data and the real-time network status. The sending module is used to send the protocol data to the central node through the acquisition node based on the transmission strategy.

[0008] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0009] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0010] The distributed architecture cloud computing data acquisition and transmission method and system of this application systematically improves the overall performance and reliability of distributed acquisition and transmission in dynamic cloud environments by constructing a closed-loop process of perception, decision-making, execution, and feedback. First, by acquiring and utilizing real-time status information from the data source for task decomposition and dynamic distribution, a shift from static preset to dynamic adaptation is achieved, solving the problems of uneven node load and resource idleness caused by scheduling lag in traditional methods. Second, the raw data is processed locally at the acquisition nodes in real time to generate reduced data, shifting the computational burden forward and significantly reducing the amount of invalid data transmitted over the network, alleviating network bandwidth pressure and reducing the processing load on the central node. Finally, by comprehensively considering the inherent attributes of the reduced data (such as data volume and timeliness) and the external real-time network status, the transmission strategy is intelligently determined, making data transmission no longer fixed or blind, but an adaptive process that makes the optimal trade-off between latency, reliability, and throughput based on the specific context. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating a distributed architecture cloud computing data acquisition and transmission method according to an embodiment of this application. Figure 2 This is a flowchart illustrating a distributed architecture cloud computing data acquisition and transmission method according to an embodiment of this application. Figure 3 This is a schematic block diagram of a distributed architecture cloud computing data acquisition and transmission system according to an embodiment of this application; Figure 4 This is a schematic block diagram of the structure of a computer device according to an embodiment of this application; Figure 5 This is a schematic diagram illustrating the enhancement mechanism of a distributed architecture cloud computing data acquisition and transmission method according to an embodiment of this application.

[0012] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0014] Reference Figure 1 This application provides a distributed architecture cloud computing data acquisition and transmission method, the method comprising: S1. Responding to the collection command, obtain real-time status information of each data source in the cloud computing environment; S2. Based on the real-time status information, the acquisition instruction is parsed into multiple acquisition sub-tasks, and each acquisition sub-task is distributed to the corresponding acquisition node according to the real-time status information. S3. Receive the acceptance response information returned by each acquisition node for the assigned acquisition sub-task, and mark the acquisition sub-task as assigned based on the acceptance response information. S4. Based on the labeling results, the acquisition subtask is executed through the acquisition node corresponding to the acquisition subtask to obtain the raw data, and the raw data is processed locally to generate the specification data. S5. Identify the attribute information and real-time network status of the protocol data, and determine a transmission strategy for each acquisition node based on the attributes of the protocol data and the real-time network status. S6. Based on the transmission strategy, the protocol data is sent to the central node through the acquisition node.

[0015] As described in steps S1-S4 above, the method provided in this embodiment can be executed by a distributed data acquisition system deployed in a cloud environment. This system includes a central scheduling server, multiple distributed acquisition nodes, and a central data aggregation server. The central scheduling server is responsible for receiving and processing acquisition instructions, parsing and distributing tasks; the acquisition nodes are deployed in network locations close to the data source to execute specific acquisition tasks and perform local data processing; the central data aggregation server receives the protocol data uploaded by each acquisition node and performs unified storage and management. The acquisition instructions can originate from a user-operated management console, a request initiated by a third-party business system through an application programming interface (API), or a scheduled task or event-triggered task automatically generated by the system according to a preset strategy. Step S1 establishes a dynamically adaptive data foundation. Upon receiving the acquisition instruction, acquisition does not begin immediately, but rather the system first actively probes the real-time operating status of the target data source. This includes, but is not limited to, the current data update frequency of the data source, the approximate size of the data to be acquired, and dynamic indicators such as the network reachability and response latency of the data source. This process can be accomplished through lightweight heartbeat detection, metadata querying, or communication with the data source agent. This approach changes the traditional scheduling model based on static configuration or historical experience, ensuring that all subsequent decisions are based on the real-time environmental state. It enables visibility into data source changes, providing early warnings and decision-making support for anomalies such as data surges and source failures. Step S2, based on the real-time status information obtained in Step S1, refines the macro-level data collection task. For example, a database table with multiple partitions can be decomposed into sub-tasks of varying sizes based on the real-time differences in data volume across partitions; a high-frequency updated data stream can be divided into sub-tasks segmented by time windows. After decomposition, dynamic matching and distribution are performed based on the requirements of each sub-task and the real-time load, geographical location, and processing capacity of the collection nodes, rather than simple polling or random allocation. By decomposing large tasks into parallel-executable sub-tasks and dynamically allocating them based on node status, the problem of some nodes being overloaded while others are idle is avoided, maximizing the overall computing power of the distributed cluster. Differentiated collection strategies are adopted for data sources with different characteristics, such as allocating more resources or more compact collection windows to high-frequency update sources, thereby optimizing the overall completion time of the collection task. Step S3 is the two-way confirmation and state synchronization step in the distributed interaction. After the distribution of the collection subtasks is completed in step S2, it is not assumed that the task has been successfully received and accepted; instead, it enters a state of waiting for confirmation. After receiving the distributed subtask, each collection node will determine whether it can accept the task based on its current resource status, task queue length, and connection reachability with the data source. Nodes that can accept the task return a clear technical signal to the system, namely, an acceptance response.Upon receiving this information, the system performs a deterministic data operation: updating the status field of the acquisition subtask to "assigned" in the global task record and associating it with the acquisition node identifier that returned the response. In step S4, after acquiring the raw data, the acquisition node does not forward it directly but executes predefined or distributed lightweight processing logic locally. This processing may include data cleaning, invalid record filtering, format standardization, and preliminary aggregation based on time or dimension. The aim is to reduce data redundancy and invalid parts near the data source, extracting reduced data with higher information density. Local processing removes a large amount of unnecessary raw impurities and redundant information, significantly reducing the amount of data that actually needs to be transmitted over the network, saving valuable bandwidth resources, and is particularly beneficial for cross-regional and cross-cloud data transmission scenarios. Offloading some computing tasks to edge nodes allows central nodes to focus on core data storage, analysis, and services.

[0016] As described in steps S5-S6 above, step S4 achieves intelligent and contextualized transmission strategies. It not only focuses on network conditions but also considers the attributes of the data itself as key decision factors. For example, if a batch of protocol data is identified as having high real-time performance and small data packet characteristics, and the current network latency is high but bandwidth is sufficient, a transmission strategy prioritizing latency reduction may be selected. The strategy library can include various combinations of protocol selections, fragment sizes, compression algorithms, and retransmission mechanisms. It can dynamically select the most suitable transmission method based on both the data type and network conditions, thereby ensuring both low latency for critical data and reliability for high-throughput data in complex and ever-changing cloud network environments. Through strategy optimization, lower-cost transmission paths or modes can be selected while meeting business needs, such as transmitting large volumes of data during off-peak hours. Step S5 is the strategy execution phase. Based on the transmission strategy determined in step S4, the collection node calls the corresponding network library and parameter configurations to efficiently and reliably transmit the locally processed protocol data to the designated central aggregation node. This finalizes the intelligent decisions made in the preceding steps, ensuring that the optimized data can be effectively aggregated. By combining previous local processing with intelligent strategies, this step ultimately achieves high-quality, high-efficiency, and highly adaptable end-to-end transmission from the data source to the central storage, which is the final manifestation of the overall performance advantages of the system.

[0017] As described above, by constructing a closed-loop process of perception, decision-making, execution, and feedback, the overall efficiency and reliability of distributed data acquisition and transmission in a dynamic cloud environment are systematically improved. First, by acquiring and utilizing real-time status information from the data source for task decomposition and dynamic distribution, a shift from static presets to dynamic adaptation is achieved, resolving the issues of uneven node load and resource idleness caused by scheduling lag in traditional methods. Second, real-time processing of raw data at the acquisition nodes to generate reduced data shifts the computational burden forward, significantly reducing the amount of invalid data transmitted over the network, alleviating network bandwidth pressure, and lowering the processing load on the central node. Finally, by comprehensively considering the inherent attributes of the reduced data (such as data volume and timeliness) and the external real-time network status to intelligently determine the transmission strategy, data transmission is no longer fixed or blind, but rather an adaptive process that makes the optimal trade-off between latency, reliability, and throughput based on the specific context.

[0018] Reference Figure 2 In one embodiment, the steps of parsing the acquisition command into multiple acquisition sub-tasks based on the real-time status information and distributing each acquisition sub-task to the corresponding acquisition node according to the real-time status information include: S21. Analyze the operating characteristics of each data source reflected in the real-time status information, and parse the acquisition instruction into multiple acquisition sub-tasks according to the operating characteristics of each data source, wherein the acquisition task of one data source is parsed into at least one acquisition sub-task. S22. Obtain real-time load information for each acquisition node; S23. Combining the real-time status information and the real-time load information, determine a target acquisition node for each acquisition subtask; S24. When determining the target acquisition node, the acquisition subtasks corresponding to data sources with high data update frequency are preferentially assigned to acquisition nodes with lower real-time load.

[0019] As described in the steps above, S21 involves differentiated task decomposition. Instead of mechanically and uniformly dividing the collection instructions, it delves into the unique operational characteristics of each data source. These characteristics include data generation patterns, data increments per unit time, and interface access characteristics. For example, a service that continuously generates high-speed log streams is characterized by a high update frequency and can be parsed into continuous streaming subtasks based on time windows; a database table storing massive amounts of historical data is characterized by large capacity but static nature and can be parsed into multiple parallel batch processing subtasks based on its physical or logical partitions. Here, each data source corresponds to at least one subtask, ensuring that the granularity of task decomposition is sufficiently fine to match the actual structure of the data source. This aligns the task decomposition method with the inherent characteristics of the data source, avoiding the unreasonable overhead or performance bottlenecks caused by a one-size-fits-all decomposition strategy for data sources with different characteristics. Fine-grained subtasks are easier to monitor, retry, and migrate, enhancing the controllability of the collection process for individual data sources. S22 aims to establish a real-time dynamic view of cluster resources. Continuously collect core performance metrics from each data collection node, such as CPU utilization, memory usage, network I / O throughput, and the length of the currently executing task queue, to accurately characterize the current workload and available capacity of each node. Upgrade scheduling decisions from being based on static configuration or simple polling to being based on the real-time capacity of nodes. This is a key prerequisite for achieving elastic scheduling and avoiding blind allocation that leads to local overload. S23 is the core computational process for scheduling decisions. As a decision-making hub, it considers both task requirements and resource supply information. It matches the resource requirements of each subtask with the real-time supply capacity of each node, and selects the most suitable execution node for each subtask using a pre-defined scheduling algorithm, thus establishing a one-to-one binding relationship. This represents a qualitative leap from simply having tasks to allocate to allocating tasks to the most suitable node. This improves the rationality of resource allocation and optimizes overall throughput and response time at a higher level. S24 defines a specific and optimized scheduling strategy rule. It identifies the key data source characteristic of high update frequency and associates it with the key resource state of low node load. This rule means that for streaming tasks that need to be processed quickly and continuously to keep up with the data generation rate, nodes with idle processing capacity will be proactively sought and allocated to ensure that these time-sensitive tasks can be processed immediately, avoiding data backlog. This specifically addresses the core pain point of high-frequency data sources being prone to acquisition delays due to busy nodes, ensuring timely response to the fastest-changing data. This rule guides the transfer of pressure from busy nodes to idle nodes, a proactive and preventative load balancing method that helps prevent localized hotspots. In this embodiment, the scheduling system is clearly aware of the tasks and the differences in urgency among different tasks. By strategically associating high-frequency data sources with low-load nodes, it directly targets the core contradiction caused by instantaneous load fluctuations and uneven data flow rates in the cloud environment.These steps work together to ensure that the computing power of the distributed acquisition cluster can be accurately and flexibly directed to where it is most needed, improving overall efficiency and stability when facing heterogeneous and dynamic workloads.

[0020] In one embodiment, the steps of identifying the attribute information and real-time network status of the protocol data, and determining a transmission strategy for each acquisition node based on the attributes of the protocol data and the real-time network status, include: Analyze the data volume of the specification data and identify the preset timeliness level of the specification data; Monitor the network latency and packet loss rate of the transmission link between the acquisition node and the central node; The transmission strategy is determined based on the data volume, timeliness level, network latency, and packet loss rate. The transmission strategy includes: if the timeliness level of the protocol data is the first level, then determine to use the User Datagram Protocol for transmission; If the amount of the protocol data is greater than a preset size threshold, and the network latency is higher than a preset latency threshold or the packet loss rate is higher than a preset packet loss threshold, then the protocol data is determined to be fragmented and a breakpoint resume task is established for each fragment. If the timeliness level of the protocol data is lower than the first level (second level) and its data volume is not greater than the size threshold, then it is determined that the transmission control protocol will be used for transmission.

[0021] As described above, a multi-dimensional, condition-driven, and rule-defined transmission strategy decision engine has been established. This addresses the shortcomings of traditional methods, such as fixed and rigid transmission strategies or simple switching logic, which make it difficult to achieve optimal transmission efficiency, reliability, and cost for data of different values ​​in complex cloud network environments. Step one involves profiling the transmission object. First, the volume of the data to be transmitted is quantified. Simultaneously, based on the business logic supported by the data, a preset timeliness level is assigned. This level is typically determined during task definition; for example, real-time monitoring data is defined as requiring millisecond-level delivery (Level 1), while end-of-day batch report data is defined as allowing minute-level latency (Level 2). These two attributes constitute the inherent transmission requirements of the data. Step two involves real-time diagnostics of the transmission environment. By actively sending probe packets or analyzing historical transmission data, key performance indicators of the current network path are continuously measured. Network latency reflects the one-way time of data packets from transmission to reception, directly determining the real-time experience of transmission; packet loss rate reflects the reliability of the network link, and packet loss will trigger retransmission, affecting effective throughput and completion time. These two indicators objectively characterize the current health status and carrying capacity of the transmission channel. Ensure the strategy can adapt to dynamic changes such as network jitter and congestion. Step 3 defines a clear condition-action decision logic: For high-time-sensitivity data: When the protocol data is identified as belonging to the first time-sensitivity level, the decision logic prioritizes speed. User Datagram Protocol (UDP), due to its connectionless, handshake-free, and low header overhead characteristics, can provide the lowest transmission latency. Although it does not guarantee reliable delivery, this strategy is the optimal choice for services such as real-time streaming media and status heartbeats that allow for a small amount of data loss but have extremely high speed requirements. For large data volumes and poor network quality: When the data volume is large and the network environment is poor, the decision logic prioritizes reliability and recoverability. Directly transmitting giant data packets in a poor network is extremely inefficient, as a single packet loss can easily lead to the entire data being retransmitted. Therefore, the strategy is determined to be fragmentation and the establishment of breakpoint resume tasks. Fragmentation breaks large tasks into smaller parts, reducing the risk of single transmissions; establishing independent breakpoint resume tasks for each fragment ensures that if any fragment transmission fails, only that fragment needs to be retransmitted, rather than the entire data body, greatly improving the transmission success rate and overall efficiency in unstable networks. For general data and normal network conditions: When data timeliness requirements are moderate, data volume is small, and network conditions are at a normal level, the decision logic selects a balanced and reliable transmission control protocol. Transmission control protocols provide connection-oriented, acknowledgment-based reliable transmission. Although the handshake and acknowledgment mechanisms introduce some latency and overhead, for most routine business data that requires guaranteed complete and error-free delivery, this is the most stable and universal choice. It effectively coordinates the often contradictory transmission goals of low latency, high reliability, and high throughput. For example, it enables reliable transmission of large amounts of data over unreliable networks through data fragmentation and retransmission, and makes reasonable trade-offs between reliability and latency through protocol selection.The multi-dimensional, fine-grained strategy decision-making mechanism in this embodiment ensures that resources can be accurately adapted to data streams of different values, namely: making critical data run faster, making large-scale data transmit more stably, and making ordinary data flow in the most economical way.

[0022] In one embodiment, the method further includes: Real-time monitoring of the resource consumption rate of each acquisition node when executing acquisition sub-tasks, and prediction of the time when the remaining available resources of each acquisition node are exhausted; Based on the prediction results, if it is detected that the remaining available resources of a certain acquisition node are exhausted earlier than the theoretical completion time of its current task, a migration instruction is generated. According to the migration instruction, the unfinished collection subtasks on the corresponding collection node, along with the intermediate data already collected, will be migrated to another collection node with sufficient resources to continue execution.

[0023] As described above, a predictive task migration mechanism is introduced to endow the distributed acquisition system with proactive elasticity and resilience, addressing the shortcomings of existing technologies where task interruptions and data acquisition failures occur due to node resource exhaustion or performance degradation, and the lack of effective self-recovery capabilities. Step one implements a proactive diagnosis of node health status. For each acquisition node, its key resource indicators, such as instantaneous values ​​of CPU utilization, memory usage, local disk I / O, or specific cache usage, are continuously tracked. By calculating the changing trends of these indicators within a recent time window—that is, the resource consumption rate—predictive algorithms, such as linear regression or time series analysis, can be used to extrapolate the specific time point at which existing resources will be exhausted at the current consumption rate. This exhaustion time is a dynamic prediction. Step two compares the predicted node endurance time with the estimated remaining execution time of all running tasks on the current node. The theoretical completion time can be estimated based on the total data volume, processed progress, and average processing speed. When it is determined that resources cannot support task completion, i.e., the predicted exhaustion time is earlier than the theoretical completion time, a migration instruction is automatically generated. This quantitative judgment logic ensures the necessity and economy of migration actions, avoiding unnecessary migration overhead. It ensures that migration only occurs when there is a genuine risk of resource conflict, avoiding erroneous migrations due to short-term resource fluctuations and improving stability. Proactively removing tasks from nodes that are about to be stalled ensures that individual collection subtasks can be executed completely, improving the reliability of data collection. Step three is the execution process of stateful migration. Migration is not simply restarting a task, but involves context transfer. First, an identical collection subtask instance is initialized on the target node, and then the complete execution context of the task on the source node is transferred. This includes: task configuration, established data source connection status, intermediate data buffer content that has been collected but not yet processed or sent, and the task's internal progress pointer. After migration, the target node will seamlessly take over the work from the breakpoint. To the external data source and the central node, this collection task appears as a seamless executor switch. This avoids duplicate data collection or collection gaps caused by task restarts, ensuring the continuity of the data stream. In this embodiment, traditional methods typically rely on monitoring alerts and manual restarts, or on the scheduler redistributing tasks after a node failure. This inevitably leads to task interruptions, data loss, or duplication. The mechanism implemented in this embodiment achieves three major shifts: from fault response to risk warning; from task restart to state transition; and from static deployment to dynamic flow. Through predictive judgment and context transition, the resource bottlenecks or potential failures of individual acquisition nodes no longer constitute a bottleneck in the entire acquisition process.

[0024] In one embodiment, the steps of real-time monitoring of the resource consumption rate of each acquisition node when executing acquisition sub-tasks and predicting the exhaustion time of the remaining available resources of each acquisition node include: Set the first and second time windows; The resource consumption rate of the acquisition node is monitored through a first time window and a second time window, wherein the length of the first time window is shorter than that of the second time window; Calculate the first average consumption rate of the acquisition node within the first time window and the second average consumption rate within the second time window, respectively. Calculate the difference between the first average consumption rate and the second average consumption rate; Based on the magnitude of the difference, the weights of the first average consumption rate and the second average consumption rate in the prediction model are dynamically adjusted, wherein the larger the difference, the higher the weight of the second average consumption rate used in the prediction model. Based on the adjusted prediction model, the predicted consumption rate of the acquisition node is calculated, and its exhaustion time is predicted according to the predicted consumption rate and the remaining available resources of the acquisition node.

[0025] As mentioned above, to address prediction distortion issues such as over-migrating and migration lag caused by drastic and non-periodic fluctuations (glitch) in resource consumption rates in dynamic shared environments, Step 1 establishes two observation windows with different time scales. The first time window is shorter, used to capture recent, instantaneous changes in resource consumption; the second time window is longer, used to observe long-term, stable trends in resource consumption. For example, the first time window can be set to 5 seconds, and the second time window to 60 seconds. The dual-window design is to simultaneously perceive short-term fluctuations and long-term trends. Step 2 continuously collects resource usage data and calculates the consumption rate within each of the two time windows. The rate changes drastically within the short window, potentially spiked instantaneously due to sudden tasks or neighbor interference; the rate changes smoothly within the long window, smoothing out short-term fluctuations and better reflecting the continuous consumption level of the task itself. This allows for the separation of short-term noise in resource consumption from the true long-term consumption trend. Step 3 averages the monitoring data within each window to obtain two representative rate values: the first average consumption rate reflects the short-term average consumption level, and the second average consumption rate reflects the long-term average consumption level. The average calculation filters out some random fluctuations within the window. Step four calculates the difference between the two average rates, the magnitude of which directly reflects the degree of fluctuation in the resource consumption rate. If the difference is small, it indicates that the consumption rate is stable, and the short-term and long-term trends are consistent; if the difference is large, it indicates that there have been drastic fluctuations recently, and short-term consumption has deviated significantly from the long-term trend. The difference, as a key indicator, objectively measures the stability or instability of resource consumption, providing a basis for dynamically adjusting the weights. Step five is the adaptive core of the prediction algorithm. The prediction model usually uses a weighted average or similar method to combine short-term and long-term rates to predict the future consumption rate. In this step, the weight allocation is not fixed, but dynamically adjusted according to the difference calculated in step four. The specific rule is: the larger the difference, the greater the short-term fluctuation, so the weight of the short-term rate is reduced, while the weight of the long-term rate is increased, because the long-term trend is more reliable; conversely, a small difference indicates stability, so a higher weight can be given to the short-term rate to quickly respond to trend changes. When drastic short-term fluctuations occur, the algorithm automatically reduces its confidence in the short-term rate and relies more on the long-term trend, thereby avoiding incorrect predictions due to short-term interference. When the consumption rate changes steadily, the algorithm assigns sufficient weight to short-term rates to promptly capture trend shifts and avoid prediction lag. Step six uses the dynamically adjusted weights to calculate the final predicted consumption rate, and then uses this rate to estimate how long the remaining resources can sustain the system. For example, if the remaining memory is 100MB and the predicted consumption rate is 10MB per second, then the exhaustion time is 10 seconds. Through adaptive weight adjustment, the predicted consumption rate neither overreacts to short-term noise nor is it slow to react to changes in the actual trend, thus obtaining a prediction value closer to reality and improving the accuracy of exhaustion time prediction.Traditional forecasting methods may use only a single time window or a multi-window fusion with fixed weights. When faced with drastic non-periodic fluctuations, they are prone to triggering false alarms due to short-term spikes or failing to respond promptly to true trend jumps due to excessive smoothing. The solution in this embodiment forms a closed-loop feedback adjustment mechanism through a series of steps including dual-window observation, difference evaluation, and dynamic weight adjustment: it adjusts the sensitivity of the forecasting strategy in real time based on the observed degree of instability.

[0026] In one embodiment, based on the prediction results, if it is detected that the remaining available resources of a certain acquisition node are exhausted earlier than the theoretical completion time of its current task, a migration instruction is generated, specifically including: A first migration threshold and a second migration threshold are set, wherein the first migration threshold is greater than the second migration threshold; When the predicted exhaustion time is earlier than the theoretical completion time but later than the first migration threshold, the acquisition node is marked as an early warning state, and at least one candidate migration node is pre-selected for the task on the acquisition node. The migration instruction is generated when the predicted exhaustion time is earlier than the second migration threshold; If a data acquisition node is in an early warning state, and the predicted exhaustion time for several consecutive monitoring cycles is later than the first migration threshold, then the early warning state of that node is lifted.

[0027] As mentioned above, to address the potential for excessive migration or command oscillations caused by short-term normal fluctuations in resource forecast indicators, thereby ensuring resilience while improving decision-making stability and economy, Step 1 defines two key time boundaries for migration decisions. The first migration threshold is a relatively lenient warning line, while the second migration threshold is a more urgent emergency action line. Both are earlier than the theoretical completion time of the task, but the second threshold is earlier and closer to the current moment than the first threshold. For example, if the theoretical completion time is 100 seconds later, the first threshold might be set 30 seconds later, and the second threshold 10 seconds later. The dual threshold setting constructs a decision buffer from warning to mandatory action, providing more granular operational space for dealing with resource fluctuations. Step 2 predicts that resources may be exhausted before the task is completed, but the exhaustion time is still some distance away, and migration is not immediately triggered. Instead, the node is first marked as being in a warning state, indicating a potential risk. At the same time, based on the cluster status, one or more suitable candidate migration nodes are proactively identified for the task on that node, and communication connections may be established in advance or some resources may be reserved. This is an intermediate state of preparation without execution. By pre-evaluating and pre-connecting target nodes, the node selection and handshake phases can be skipped once migration is needed, accelerating the migration process and reducing task downtime. Step three: When predictions indicate that resources will be exhausted in a very short time, and the risk is deemed imminent, there is no time to wait and see. At this point, based on the pre-selection results from step two or a rapid re-decision, a formal migration instruction is generated, initiating the substantive task and state migration process. The second migration threshold is the trigger point for initiating the actual migration operation. Taking action before resources are truly exhausted prevents the worst-case scenario of node crashes and task failures due to hesitation or delay. Step four: A state recovery mechanism. If a node in a warning state shows that its exhaustion time has returned to a safe range, later than the first threshold, for multiple consecutive monitoring cycles (e.g., three consecutive times), it indicates that the previous resource shortage was temporary and has been alleviated. Therefore, its warning state is lifted, and pre-selected candidate node resources are released. Requiring multiple consecutive cycles to meet the safety conditions, rather than a single instance, effectively filters out frequent fluctuations in prediction values ​​near the threshold, avoiding system instability caused by repeated oscillations between migration instruction generation and cancellation.

[0028] In one embodiment, before the step of migrating the unfinished acquisition subtasks on the corresponding acquisition node, along with the intermediate data already acquired, to another acquisition node with sufficient resources for continued execution, the method further includes: After generating the migration instruction, the data processing of the acquisition subtask by the acquisition node is paused. When the acquisition node executes the acquisition subtask, the runtime state currently maintained in memory is persistently saved, and a state snapshot file is generated. The runtime state includes the internal data structure of the data processing logic and the progress indicator of the processed data. The state snapshot file is associated and encapsulated with the collected intermediate data to obtain a transferable task context package; Send the task context packet to the target acquisition node; The target acquisition node parses the task context packet, loads the state snapshot file to restore the runtime state, and continues to execute the acquisition subtask from the corresponding position based on the progress identifier.

[0029] As mentioned above, to address the shortcomings of traditional migration methods in performing stateful data acquisition tasks, which suffer from computational semantic errors, data duplication, or loss due to the loss of task runtime state caused by only migrating data, the following steps are taken: Step 1 ensures a synchronization point for state consistency. Once the migration is determined, a pause signal is first sent to the acquisition subtask, stopping it from pulling new data from the data source and pausing the internal processing logic of the received data. This ensures that at the moment the system state is captured, the task is in a static, deterministic instant, where all data and logical relationships in memory are frozen and self-consistent, with no ongoing intermediate operations. This avoids the uncertain state that may occur during state preservation due to tasks still executing concurrently, where half is written and half is not. Step 2 serializes all the memory of the task in memory and writes it to persistent storage. This includes: first, the internal data structures of the data processing logic, such as the time window queue content in stream processing, the bitmap of the deduplication filter, the intermediate accumulator of aggregation calculation, the current context of the data parser, etc.; second, the progress indicators of the processed data, such as the offset of the last successfully consumed message, the sequence number of the confirmed data block, or the pointer position of the file read. This snapshot file fully encapsulates all the state information of the task up to the current moment. It extracts the core of the task from volatile memory, making it independent of the physical existence of the source node, providing all the necessary information to revive the task on another node. This directly addresses the problem of losing intermediate processing logic states when only original data or intermediate results are migrated, ensuring the continuity of complex computational semantics. Step three associates and binds the state snapshot file representing the task's memory with the collected but unsent intermediate data representing the task's output through metadata, forming a logically self-contained migration unit. This task context packet clearly defines the correspondence between data and state, ensuring it is treated as an indivisible whole during migration. This prevents mismatch between state and data or loss of components during transmission, ensuring the integrity of the migration context. Step four uses a reliable network transmission mechanism to send the encapsulated task context packet completely from the source node to the pre-selected target acquisition node. This safely transfers the complete execution state of the task from a node about to fail or become overloaded to a new node. Step five is state reconstruction and seamless succession. After receiving the packet, the target node first parses and loads the state snapshot file, accurately reconstructing the runtime data structure in memory that is completely consistent with the source node's pause point. Then, based on the recovery progress indicator, it accurately locates the breakpoint in the data source or data stream. Finally, the target node issues a continuation instruction to the task, and the task will begin consuming new data from that breakpoint. For the data source and downstream system, this acquisition task is not interrupted or restarted, avoiding duplicate data acquisition or data gaps. For stateful computations such as aggregation and deduplication, it ensures strict continuity of computational logic before and after migration, avoiding errors in results due to state loss.

[0030] refer to Figure 5 In one feasible embodiment, for business scenarios that are highly dynamic and resource-sensitive, the method further includes: in the step of obtaining real-time status information of each data source in the cloud computing environment, the method includes: By analyzing the time interval and data volume of the data packets successfully collected by the acquisition node from the data source, the data output rate change trend of the data source is inferred in reverse, and used as part of the real-time status information; The step of processing the raw data locally at the acquisition node to generate reduced data includes: Real-time monitoring of the current utilization rate of the CPU and memory of the acquisition node itself; When the current utilization rate exceeds a preset first security threshold, the acquisition node is controlled to switch to a preset pass-through processing mode. In the pass-through processing mode, the acquisition node only performs format encapsulation on the raw data without performing aggregation or filtering calculations. The step of determining the transmission strategy for each acquisition node includes: Obtain all protocol data to be sent by the acquisition node, and sort them by sending priority according to the timeliness level of each protocol data; Monitor the network latency index of the transmission link between the acquisition node and the central node, and perform a moving average filter on the network latency index to obtain a smoothed latency value; Based on the smoothing delay value and the transmission priority ranking, the final transmission strategy of the acquisition node is determined comprehensively.

[0031] As mentioned above, in a cloud computing environment, real-time monitoring and data collection of the operational status of core business systems (such as large e-commerce trading platforms and financial service systems) is crucial for ensuring business stability and conducting immediate risk control and performance analysis. These systems are typically deployed in a microservice architecture, generating massive amounts of logs, metrics, and tracking data. Adopting a distributed collection architecture, deploying collection nodes near the business servers, is the mainstream solution for achieving low-latency, high-throughput data collection. Existing technologies can achieve basic distributed collection and transmission, but there are still shortcomings in optimizing for handling the inherent extreme dynamic loads and resource sensitivity of business systems. Specific challenges arise when performing real-time monitoring and data collection for core business systems (such as high-frequency trading and large online services). In these scenarios, data flows exhibit significant tidal effects and burstiness, and collection nodes often share resources with business processes. Traditional distributed collection methods, in pursuit of efficiency, often suffer from high state awareness overhead, high local processing resource consumption, and unstable transmission strategies, making it difficult to ensure both collection performance and the absolute stability and resource isolation of core business operations.

[0032] In this embodiment, three collaborative mechanisms are introduced to enhance adaptability and stability in highly dynamic and resource-sensitive scenarios. The principles are as follows: First, by analyzing the characteristics of collected data packets, the data source status is inferred in reverse, achieving trend perception with zero additional overhead and resolving the signaling burden and interference issues caused by active probing. Second, by monitoring the resource load of the collection node itself and triggering processing mode degradation, it automatically switches to a low-overhead direct forwarding mode when resources are scarce, mitigating the risk of local computing competing for resources with the host's core services. Finally, by globally prioritizing the transmission of multiple data streams within the node and applying a moving average filter to network metrics, a comprehensive strategy decision is made based on this, resolving the transmission contention problem under mixed data streams and the frequent policy oscillations caused by instantaneous network jitter.

[0033] Reference Figure 3 This application also provides a distributed architecture cloud computing data acquisition and transmission system, including: Module 1 is used to obtain real-time status information of various data sources in the cloud computing environment in response to the collection command; Parsing module 2 is used to parse the acquisition instruction into multiple acquisition sub-tasks based on the real-time status information, and distribute each acquisition sub-task to the corresponding acquisition node according to the real-time status information. The receiving module 3 is used to receive the acceptance response information returned by each acquisition node for the assigned acquisition sub-task, and mark the acquisition sub-task as assigned based on the acceptance response information. The generation module 4 is used to execute the acquisition sub-task through the acquisition node corresponding to the acquisition sub-task based on the labeling results to obtain the raw data, and process the raw data locally to generate the reduced data. The identification module 5 is used to identify the attribute information and real-time network status of the protocol data, and to determine the transmission strategy for each acquisition node based on the attributes of the protocol data and the real-time network status. The sending module 6 is used to send the protocol data to the central node through the acquisition node based on the transmission strategy.

[0034] As described above, it is understood that each component of the distributed architecture cloud computing data acquisition and transmission system proposed in this application can realize the function of any of the distributed architecture cloud computing data acquisition and transmission methods described above, and the specific structure will not be repeated.

[0035] Reference Figure 4 This application also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 4As shown, this computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores monitoring data and other data. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a distributed architecture cloud computing data acquisition and transmission method.

[0036] The processor described above executes the cloud computing data acquisition and transmission method with the distributed architecture, comprising: responding to an acquisition command, acquiring real-time status information of each data source in the cloud computing environment; based on the real-time status information, parsing the acquisition command into multiple acquisition sub-tasks, and distributing each acquisition sub-task to a corresponding acquisition node according to the real-time status information; receiving acceptance response information returned by each acquisition node for the distributed acquisition sub-task, and marking the acquisition sub-task as assigned based on the acceptance response information; based on the marking result, executing the acquisition sub-task through the acquisition node corresponding to the acquisition sub-task to obtain raw data, and processing the raw data locally to generate protocol data; identifying the attribute information and real-time network status of the protocol data, and determining a transmission strategy for each acquisition node based on the attributes of the protocol data and the real-time network status; and based on the transmission strategy, sending the protocol data to the central node through the acquisition node.

[0037] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a distributed cloud computing data acquisition and transmission method, comprising the steps of: responding to an acquisition command, acquiring real-time status information of each data source in the cloud computing environment; based on the real-time status information, parsing the acquisition command into multiple acquisition sub-tasks, and distributing each acquisition sub-task to a corresponding acquisition node according to the real-time status information; receiving acceptance response information returned by each acquisition node for the distributed acquisition sub-tasks, and marking the acquisition sub-tasks as assigned based on the acceptance response information; based on the marking result, executing the acquisition sub-tasks through the acquisition nodes corresponding to the acquisition sub-tasks to obtain raw data, and processing the raw data locally to generate protocol data; identifying the attribute information and real-time network status of the protocol data, and determining a transmission strategy for each acquisition node based on the attributes of the protocol data and the real-time network status; and based on the transmission strategy, sending the protocol data to a central node through the acquisition nodes.

[0038] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in this application and in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0039] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to the process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0040] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A distributed architecture cloud computing data acquisition and transmission method, characterized in that, The method includes: In response to the collection command, it obtains real-time status information of various data sources in the cloud computing environment; Based on the real-time status information, the acquisition command is parsed into multiple acquisition sub-tasks, and each acquisition sub-task is distributed to the corresponding acquisition node according to the real-time status information. Receive the acceptance response information returned by each acquisition node for the assigned acquisition subtask, and mark the acquisition subtask as assigned based on the acceptance response information; Based on the labeling results, the acquisition subtask is executed through the acquisition node corresponding to the acquisition subtask to obtain the raw data, and the raw data is processed locally to generate the reduced data. Identify the attribute information and real-time network status of the protocol data, and determine a transmission strategy for each acquisition node based on the attributes of the protocol data and the real-time network status. Based on the transmission strategy, the protocol data is sent to the central node through the acquisition node.

2. The cloud computing data acquisition and transmission method with a distributed architecture according to claim 1, characterized in that, The steps of parsing the acquisition command into multiple acquisition sub-tasks based on the real-time status information and distributing each acquisition sub-task to the corresponding acquisition node according to the real-time status information include: Analyze the operating characteristics of each data source reflected in the real-time status information, and based on the operating characteristics of each data source, parse the acquisition instruction into multiple acquisition sub-tasks, wherein the acquisition task of one data source is parsed into at least one acquisition sub-task. Obtain real-time load information for each data acquisition node; By combining the real-time status information and the real-time load information, a target acquisition node is determined for each acquisition subtask; When determining the target acquisition node, the acquisition subtasks corresponding to data sources with high data update frequency are preferentially assigned to acquisition nodes with lower real-time load.

3. The cloud computing data acquisition and transmission method with a distributed architecture according to claim 1, characterized in that, The steps of identifying the attribute information and real-time network status of the protocol data, and determining a transmission strategy for each acquisition node based on the attributes of the protocol data and the real-time network status, include: Analyze the data volume of the specification data and identify the preset timeliness level of the specification data; Monitor the network latency and packet loss rate of the transmission link between the acquisition node and the central node; The transmission strategy is determined based on the data volume, timeliness level, network latency, and packet loss rate. The transmission strategy includes: if the timeliness level of the protocol data is the first level, then determine to use the User Datagram Protocol for transmission; If the amount of the protocol data is greater than a preset size threshold, and the network latency is higher than a preset latency threshold or the packet loss rate is higher than a preset packet loss threshold, then the protocol data is determined to be fragmented and a breakpoint resume task is established for each fragment. If the timeliness level of the protocol data is lower than the first level (second level) and its data volume is not greater than the size threshold, then it is determined that the transmission control protocol will be used for transmission.

4. The cloud computing data acquisition and transmission method with a distributed architecture according to claim 1, characterized in that, The method further includes: Real-time monitoring of the resource consumption rate of each acquisition node when executing acquisition sub-tasks, and prediction of the time when the remaining available resources of each acquisition node are exhausted; Based on the prediction results, if it is detected that the remaining available resources of a certain acquisition node are exhausted earlier than the theoretical completion time of its current task, a migration instruction is generated. According to the migration instruction, the unfinished collection subtasks on the corresponding collection node, along with the intermediate data already collected, will be migrated to another collection node with sufficient resources to continue execution.

5. The cloud computing data acquisition and transmission method with a distributed architecture according to claim 4, characterized in that, The steps of real-time monitoring of the resource consumption rate of each acquisition node when executing acquisition sub-tasks and predicting the exhaustion time of the remaining available resources of each acquisition node include: Set the first and second time windows; The resource consumption rate of the acquisition node is monitored through a first time window and a second time window, wherein the length of the first time window is shorter than that of the second time window; Calculate the first average consumption rate of the acquisition node within the first time window and the second average consumption rate within the second time window, respectively. Calculate the difference between the first average consumption rate and the second average consumption rate; Based on the magnitude of the difference, the weights of the first average consumption rate and the second average consumption rate in the prediction model are dynamically adjusted, wherein the larger the difference, the higher the weight of the second average consumption rate used in the prediction model. Based on the adjusted prediction model, the predicted consumption rate of the acquisition node is calculated, and its exhaustion time is predicted according to the predicted consumption rate and the remaining available resources of the acquisition node.

6. The cloud computing data acquisition and transmission method with a distributed architecture according to claim 4, characterized in that, Based on the prediction results, if it is detected that the remaining available resources of a certain acquisition node are exhausted earlier than the theoretical completion time of its current task, a migration instruction is generated, specifically including: A first migration threshold and a second migration threshold are set, wherein the first migration threshold is greater than the second migration threshold; When the predicted exhaustion time is earlier than the theoretical completion time but later than the first migration threshold, the acquisition node is marked as an early warning state, and at least one candidate migration node is pre-selected for the task on the acquisition node. The migration instruction is generated when the predicted exhaustion time is earlier than the second migration threshold; If a data acquisition node is in an early warning state, and the predicted exhaustion time for several consecutive monitoring cycles is later than the first migration threshold, then the early warning state of that node is lifted.

7. The cloud computing data acquisition and transmission method with a distributed architecture according to claim 5, characterized in that, Before the step of migrating the unfinished collection subtasks on the corresponding collection node, along with the intermediate data already collected, to another collection node with sufficient resources for continued execution, the method further includes: After generating the migration instruction, the data processing of the acquisition subtask by the acquisition node is paused. When the acquisition node executes the acquisition subtask, the runtime state currently maintained in memory is persistently saved and a state snapshot file is generated. The runtime state includes the internal data structure of the data processing logic and the progress indicator of the processed data. The state snapshot file is associated and encapsulated with the collected intermediate data to obtain a transferable task context package; Send the task context packet to the target acquisition node; The target acquisition node parses the task context packet, loads the state snapshot file to restore the runtime state, and continues to execute the acquisition subtask from the corresponding position based on the progress identifier.

8. A distributed architecture cloud computing data acquisition and transmission system, characterized in that, include: The acquisition module is used to respond to acquisition commands and acquire real-time status information of various data sources in the cloud computing environment; The parsing module is used to parse the acquisition command into multiple acquisition sub-tasks based on the real-time status information, and distribute each acquisition sub-task to the corresponding acquisition node according to the real-time status information. The receiving module is used to receive the acceptance response information returned by each acquisition node for the assigned acquisition sub-task, and mark the acquisition sub-task as assigned based on the acceptance response information. The generation module is used to execute the acquisition subtask through the acquisition node corresponding to the acquisition subtask based on the labeling results to obtain the raw data, and process the raw data locally to generate the reduced data. The identification module is used to identify the attribute information and real-time network status of the protocol data, and determine the transmission strategy for each acquisition node based on the attributes of the protocol data and the real-time network status. The sending module is used to send the protocol data to the central node through the acquisition node based on the transmission strategy.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images