Data acquisition method and device, computer device and storage medium

By employing parallel data acquisition and a master-slave election mechanism, the interruption problem in data acquisition and push in large-scale data scenarios was solved, achieving zero-interruption and highly elastic data acquisition and push capabilities, optimizing resource utilization, and ensuring the high availability and stability of the system.

CN119603119BActive Publication Date: 2026-01-13CHINA TELECOM CLOUD TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411778775.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-12-05
Publication Date
2026-01-13
Estimated Expiration
2044-12-05

AI Technical Summary

Technical Problem

In large-scale data scenarios, existing data collection methods are subject to interruption risks and inflexible resource utilization, resulting in high data forwarding latency or resource waste, making it difficult to achieve zero-interruption and highly elastic data collection and push capabilities.

Method used

At least two data acquisition engines are used to collect data in parallel, and a master forwarding adapter working group and a slave forwarding adapter working group are formed through a master-slave election mechanism. The master forwarding adapter is responsible for data push, and the slave forwarding adapter serves as a hot standby. In case of failure, resource allocation is dynamically adjusted and master-slave election is carried out to ensure the continuity of data forwarding and system stability.

Benefits of technology

It achieves zero-interruption, highly elastic data collection and push capabilities in large-scale data scenarios, ensuring the system's elastic capacity and high availability, optimizing resource utilization, and reducing the risk of interruption and resource waste.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119603119B_ABST
    Figure CN119603119B_ABST
Patent Text Reader

Abstract

The application relates to a data collection method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises: collecting data in parallel by using at least two data collection engines, storing the collected data locally, and pushing the data to a data forwarding adapter corresponding to each data collection engine; performing master-slave election on the at least two data forwarding adapters; pushing the received data to be forwarded to a remote data storage through a master forwarding adapter workgroup, and discarding the received data to be forwarded through a slave forwarding adapter workgroup; and in the case of a fault of the data collection engine, re-performing master-slave election to obtain a new master forwarding adapter workgroup and slave forwarding adapter workgroup and continuing to execute. The method can realize zero-interruption and high-elasticity data collection and pushing capacity to ensure the elastic capacity and high availability in a large-scale data scene.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of data acquisition technology, and in particular to a data acquisition method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] In cloud computing, container virtualization, big data, and observability, data acquisition is a crucial step. In hybrid cloud or distributed container cloud scenarios, data acquisition typically involves collecting local data or metrics from various regional clusters, transferring them to local forwarding adapters, and finally pushing them to remote centralized high-capacity storage. This model performs well in small to medium-sized data scenarios, but it presents challenges in large-scale data scenarios: in clusters with thousands of nodes, the amount of data to be collected and stored locally is enormous, and restarting the data acquisition engine can cause interruptions in data acquisition and push. For example, in a 3000-node Kubernetes cluster, when the time-series database engine Prometheus restarts, it needs to load old local data; loading 7 days of full time-series data can take anywhere from tens of minutes to several minutes, during which time data acquisition and push services will be interrupted.

[0003] Current data forwarding adapters have limited forwarding capacity. During peak business periods, the large volume of data processed leads to high forwarding latency; during off-peak periods, the forwarding adapters' computing power is idle, resulting in wasted resources. To address these issues, common industry solutions include reducing the data storage cycle on the data acquisition engine side, reducing the number of data acquisition metrics, or deploying multiple data acquisition engines for data acquisition and push, with redundant data stored remotely and deduplicated during data computation. While these measures can improve the availability of data acquisition services, they either sacrifice the scale of usable business data or the computational efficiency of business applications.

[0004] Therefore, there is an urgent need for a highly available data acquisition method, device, computer equipment, computer-readable storage medium, and computer program product that can achieve zero-interruption and highly elastic data acquisition and push capabilities to ensure elastic capacity and high availability in large-scale data scenarios. Summary of the Invention

[0005] Therefore, it is necessary to provide a data acquisition method, device, computer equipment, computer-readable storage medium, and computer program product that can achieve zero-interruption and highly elastic data acquisition and push capabilities to ensure elastic capacity and high availability in large-scale data scenarios, in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides a data acquisition method, including:

[0007] Data is collected in parallel using at least two data acquisition engines. The collected data is stored locally and then pushed to the data forwarding adapter corresponding to each data acquisition engine.

[0008] Perform a master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group;

[0009] The primary forwarding adapter working group pushes the received data to be forwarded to the remote data storage, and the secondary forwarding adapter working group discards the received data to be forwarded.

[0010] In the event of a failure of the data acquisition engine corresponding to the primary forwarding adapter working group, each data forwarding adapter in the primary forwarding adapter working group will relinquish its primary forwarding identity to trigger a new primary-secondary election for at least two data forwarding adapters, resulting in a new primary forwarding adapter working group and a new secondary forwarding adapter working group.

[0011] In one embodiment, the step of performing master-slave election on at least two data forwarding adapters to determine the master forwarding adapter working group and the slave forwarding adapter working group includes:

[0012] Hybrid master-slave election is performed for at least two data forwarding adapters;

[0013] After any data forwarding adapter elects the primary forwarding identity, it forms a primary forwarding adapter working group and distributes the primary forwarding identity to all data forwarding adapters in the primary forwarding adapter working group;

[0014] Expand the capacity of the primary forwarding adapter workgroup and shrink the capacity of the secondary forwarding adapter workgroup.

[0015] In one embodiment, the CPU utilization and memory utilization of the primary forwarding adapter workgroup and the secondary forwarding adapter workgroup are monitored as monitoring results, based on the horizontal Pod auto-expander.

[0016] Based on the monitoring results, the number of nodes in the primary forwarding adapter working group and the secondary forwarding adapter working group will be increased during peak business hours, and the number of nodes in the primary forwarding adapter working group and the secondary forwarding adapter working group will be reduced during off-peak business hours.

[0017] In one embodiment, the sum of the maximum number of working replicas M obtained by expanding the primary forwarding adapter working group and the minimum number of hot standby replicas N obtained by shrinking the secondary forwarding adapter working group is kept to be an odd number.

[0018] In one embodiment, if there is no data forwarding in the primary forwarding adapter workgroup, the corresponding primary forwarding identity is cancelled;

[0019] If no master forwarding identity exists among the data forwarding adapters in the entire forwarding adapter virtual cluster, a master-slave election operation is initiated using any data forwarding adapter in the slave forwarding adapter workgroup. If any data forwarding adapter is elected as the master forwarding identity, the corresponding slave forwarding adapter workgroup is converted into the master forwarding adapter workgroup.

[0020] In one embodiment, before pushing the received data to be forwarded to the remote data storage via the main forwarding adapter working group, the method further includes:

[0021] The data to be forwarded is converted to a different format to fit the data storage format of the remote data storage device.

[0022] Secondly, this application also provides a data acquisition device, comprising:

[0023] The data acquisition module is used to acquire data in parallel using at least two data acquisition engines, store the acquired data locally, and push the data to the data forwarding adapter corresponding to each data acquisition engine.

[0024] The master-slave election module is used to elect a master-slave data forwarding adapter from at least two data forwarding adapters, resulting in a master forwarding adapter working group and a slave forwarding adapter working group.

[0025] The data forwarding control module is used to push the received data to be forwarded to the remote data storage through the main forwarding adapter working group, and to discard the received data to be forwarded through the slave forwarding adapter working group.

[0026] The master-slave election module is also used to trigger a new master-slave election for at least two data forwarding adapters in the master forwarding adapter working group when the data acquisition engine corresponding to the master forwarding adapter working group fails, so that each data forwarding adapter in the master forwarding adapter working group will give up its master forwarding identity and obtain a new master forwarding adapter working group and slave forwarding adapter working group.

[0027] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0028] Data is collected in parallel using at least two data acquisition engines. The collected data is stored locally and then pushed to the data forwarding adapter corresponding to each data acquisition engine.

[0029] Perform a master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group;

[0030] The primary forwarding adapter working group pushes the received data to be forwarded to the remote data storage, and the secondary forwarding adapter working group discards the received data to be forwarded.

[0031] In the event of a failure of the data acquisition engine corresponding to the primary forwarding adapter working group, each data forwarding adapter in the primary forwarding adapter working group will relinquish its primary forwarding identity to trigger a new primary-secondary election for at least two data forwarding adapters, resulting in a new primary forwarding adapter working group and a new secondary forwarding adapter working group.

[0032] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0033] Data is collected in parallel using at least two data acquisition engines. The collected data is stored locally and then pushed to the data forwarding adapter corresponding to each data acquisition engine.

[0034] Perform a master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group;

[0035] The primary forwarding adapter working group pushes the received data to be forwarded to the remote data storage, and the secondary forwarding adapter working group discards the received data to be forwarded.

[0036] In the event of a failure of the data acquisition engine corresponding to the primary forwarding adapter working group, each data forwarding adapter in the primary forwarding adapter working group will relinquish its primary forwarding identity to trigger a new primary-secondary election for at least two data forwarding adapters, resulting in a new primary forwarding adapter working group and a new secondary forwarding adapter working group.

[0037] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0038] Data is collected in parallel using at least two data acquisition engines. The collected data is stored locally and then pushed to the data forwarding adapter corresponding to each data acquisition engine.

[0039] Perform a master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group;

[0040] The primary forwarding adapter working group pushes the received data to be forwarded to the remote data storage, and the secondary forwarding adapter working group discards the received data to be forwarded.

[0041] In the event of a failure of the data acquisition engine corresponding to the primary forwarding adapter working group, each data forwarding adapter in the primary forwarding adapter working group will relinquish its primary forwarding identity to trigger a new primary-secondary election for at least two data forwarding adapters, resulting in a new primary forwarding adapter working group and a new secondary forwarding adapter working group.

[0042] The aforementioned data acquisition method, apparatus, computer equipment, computer-readable storage medium, and computer program product utilize at least two data acquisition engines to acquire data in parallel, store the acquired data locally, and then push the data to the data forwarding adapter corresponding to each data acquisition engine. Parallel data acquisition improves data acquisition efficiency and throughput, enabling rapid processing of large amounts of data in large-scale data scenarios. A master-slave election is performed on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group. This master-slave election mechanism ensures high availability of the data forwarding adapters, guaranteeing continuous data delivery and system stability. The master forwarding adapter working group pushes received data to be forwarded to a remote data storage device, while the slave forwarding adapter working group discards received data to be forwarded. This mechanism optimizes resource utilization; the master forwarding adapter working group is responsible for data forwarding, while the slave forwarding adapter working group does not consume resources, reducing resource waste. In the event of a failure of the data acquisition engine corresponding to the master forwarding adapter working group, a new master forwarding adapter working group and a slave forwarding adapter working group are elected again, and the data forwarding process continues. This approach enables rapid fault recovery and master-slave hot-switch, ensuring zero-interruption data acquisition and delivery, and improving system resilience and reliability. In summary, this application achieves zero-interruption, highly resilient data acquisition and delivery capabilities in large-scale data scenarios, ensuring the system's elastic capacity and high availability. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a diagram illustrating the application environment of a data acquisition method in one embodiment.

[0045] Figure 2 This is a flowchart illustrating a data acquisition method in one embodiment;

[0046] Figure 3This is a flowchart illustrating the data acquisition method in another embodiment;

[0047] Figure 4 This is a device architecture diagram of the two data acquisition engines in the most detailed embodiment;

[0048] Figure 5 This is a device architecture diagram of the three data acquisition engines in the most detailed embodiment;

[0049] Figure 6 This is a structural block diagram of a data acquisition device in one embodiment;

[0050] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] The data acquisition method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0053] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. The server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0054] In one exemplary embodiment, such as Figure 2 As shown, a data acquisition method is provided, which can be applied to... Figure 1 Taking the server in the example, the explanation includes the following steps S202 to S208. Wherein:

[0055] Step S202: Use at least two data acquisition engines to collect data in parallel, store the collected data locally, and push the data to the data forwarding adapter corresponding to each data acquisition engine.

[0056] Specifically, multiple data acquisition engines are deployed, which can work simultaneously to acquire data from different data sources or different parts of the same data source in a parallel manner.

[0057] Each data acquisition engine stores the collected data locally to ensure data persistence. Local storage can be any type of storage medium, such as hard drives or SSDs, providing an instant backup and ensuring data integrity in case of network issues or remote storage unavailability. It can also connect to the data acquisition engine via a "remote writewebhook". The data collected by the engine is not only stored locally but also pushed to the corresponding data forwarding adapter in real time. This design allows data to be forwarded immediately upon acquisition, reducing latency during transmission.

[0058] Step S204: Perform master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group.

[0059] Specifically, this embodiment employs a master-slave election mechanism, a common distributed system design pattern used to elect a "master" instance from a group of service instances, while the other instances act as "slave" instances. In the context of data forwarding adapters, this mechanism determines which adapter instances will be responsible for the actual data forwarding work and which will serve as backups.

[0060] Step S206: The received data to be forwarded is pushed to the remote data storage via the master forwarding adapter working group, and the received data to be forwarded is discarded via the slave forwarding adapter working group.

[0061] Specifically, once the primary forwarding adapter working group is elected, it will be responsible for receiving data to be forwarded from the data acquisition engine and then pushing it to a remote centralized large-capacity data storage facility. The primary forwarding adapter working group refers to the set of data forwarding adapters elected as the primary node during the election process; they will undertake the main work of data forwarding.

[0062] The secondary forwarding adapter working group refers to the set of data forwarding adapters that were not elected as the primary node during the election process. These adapters exist as hot backups; they do not participate in data forwarding under normal circumstances, but they take over the work when the primary forwarding adapter working group fails, ensuring the continuity of data forwarding and the high availability of the system.

[0063] The number of instances in the primary forwarding adapter working group and the secondary forwarding adapter working group can be dynamically adjusted according to actual business needs and load conditions.

[0064] In step S208, if the data acquisition engine corresponding to the master forwarding adapter working group fails, each data forwarding adapter in the master forwarding adapter working group will relinquish its master forwarding identity to trigger a new master-slave election for at least two data forwarding adapters, resulting in a new master forwarding adapter working group and a slave forwarding adapter working group.

[0065] Specifically, the system monitors the status of the data acquisition engine. Once a failure is detected in the data acquisition engine corresponding to the primary forwarding adapter workgroup (e.g., instance restart or process freeze), the system triggers a fault recovery process. This involves restarting the master-slave election mechanism, which involves all available data forwarding adapters, with the aim of selecting a new primary forwarding adapter workgroup from among them.

[0066] During the election process, a data forwarding adapter that was originally part of the forwarding adapter working group may be elected as the primary forwarder and will take on the task of data forwarding.

[0067] Once a data forwarding adapter with the primary forwarding role is elected, it will form a new primary forwarding adapter workgroup with its corresponding data acquisition engine (if the engine that did not previously participate in the primary task is still healthy). Meanwhile, other data forwarding adapter instances that were not elected as primary will automatically become secondary forwarding adapter workgroups. The new primary forwarding adapter workgroup will take over the data forwarding task, continuing to push received data to be forwarded to the remote data storage. The secondary forwarding adapter workgroups will continue to serve as backups, awaiting the next possible election.

[0068] This process ensures that even if the main data acquisition engine fails, the system can quickly recover and continue executing data acquisition and push tasks, achieving system resilience and zero-interruption service. After the new master forwarding adapter workgroup is formed, the system may adjust the size of the master-slave workgroup according to the current load to optimize resource utilization and meet business needs. Through this design, the system can respond quickly in the event of a failure, achieve master-slave hot switching, and ensure the continuity of data acquisition and forwarding services, thereby improving the reliability and stability of the entire data acquisition system.

[0069] In the aforementioned data acquisition method, at least two data acquisition engines are used to collect data in parallel. The collected data is stored locally and then pushed to the data forwarding adapter corresponding to each data acquisition engine. Parallel data acquisition improves efficiency and throughput, enabling rapid processing of large amounts of data in large-scale scenarios. A master-slave election is performed on the at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group. This master-slave election mechanism ensures high availability of the data forwarding adapters, guaranteeing continuous data delivery and system stability. The master forwarding adapter working group pushes received data to be forwarded to a remote data storage device, while the slave forwarding adapter working group discards received data to be forwarded. This mechanism optimizes resource utilization; the master forwarding adapter working group is responsible for data forwarding, while the slave forwarding adapter working group does not consume resources, reducing resource waste. In the event of a failure in the data acquisition engine corresponding to the master forwarding adapter working group, a master-slave election is performed again on the at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group, and subsequent data forwarding operations continue. This achieves rapid fault recovery and hot master-slave switching, ensuring zero interruption in data acquisition and delivery, and improving system resilience and reliability. In summary, this application achieves zero-interruption and highly elastic data collection and push capabilities in large-scale data scenarios, ensuring the system's elastic capacity and high availability.

[0070] In one exemplary embodiment, such as Figure 3 As shown, a master-slave election is performed for at least two data forwarding adapters to determine the master forwarding adapter working group and the slave forwarding adapter working group, including:

[0071] Step S302: Perform a hybrid master-slave election for at least two data forwarding adapters;

[0072] Step S304: After any data forwarding adapter elects the primary forwarding identity, a primary forwarding adapter working group is formed, and the primary forwarding identity is distributed to all data forwarding adapters in the primary forwarding adapter working group;

[0073] Step S306: Expand the capacity of the primary forwarding adapter working group and shrink the capacity of the secondary forwarding adapter working group.

[0074] Specifically, multiple data forwarding adapter instances participate in the election process to determine which instances will assume the primary forwarding role and which will serve as backups for the secondary forwarding role. This election process may involve complex algorithms to ensure fairness and high system availability.

[0075] During the election process, if any data forwarding adapter successfully wins the primary forwarding role, that adapter instance will be included in the primary forwarding adapter workgroup. This means it will be responsible for the actual data forwarding work, pushing data from the local data engine to the remote mass data storage. Data forwarding adapter instances that are not elected as primary are included in the secondary forwarding adapter workgroup. They do not normally participate in data forwarding but serve as backups to ensure they can quickly take over tasks in the event of a primary forwarding adapter failure.

[0076] It should be noted that data forwarding can be handled by the main forwarding adapter working group itself, or by any data forwarding adapter in the main forwarding adapter working group that has the status of main forwarding. Both of these implementation methods will fall within the protection scope of this application.

[0077] Simultaneously, the size of the master-slave forwarding adapter workgroup will be dynamically adjusted based on current business needs and load conditions. The master forwarding adapter workgroup will expand to the maximum number of working replicas M to ensure sufficient processing capacity to handle peak data volumes. Conversely, the slave forwarding adapter workgroup will shrink to the minimum number of hot standby replicas N to reduce resource waste during off-peak periods. This technique allows the system to flexibly adjust resource allocation under different business loads, achieving optimized resource utilization. During peak business periods, processing capacity is increased by expanding the master forwarding adapter workgroup; during off-peak periods, unnecessary resource consumption is reduced by shrinking the slave forwarding adapter workgroup.

[0078] In this embodiment, through this master-slave election and dynamic scaling mechanism, the system can ensure sufficient data forwarding capacity at all times, and can quickly switch to the slave forwarding adapter when the master forwarding adapter fails, thus ensuring the continuity of data forwarding and the high availability of the system.

[0079] In one exemplary embodiment, the CPU utilization and memory utilization of the primary forwarding adapter workgroup and the secondary forwarding adapter workgroup are monitored as monitoring results, based on the horizontal Pod auto-expander.

[0080] Based on monitoring results, the number of nodes in both the primary forwarding adapter working group and the secondary forwarding adapter working group will be increased during peak business hours, and the number of nodes in both groups will be reduced during off-peak business hours.

[0081] Specifically, Horizontal Pod Autoscaling (HPA) is a feature in Kubernetes clusters that automatically adjusts the number of Pods based on CPU utilization, memory utilization, or other selected metrics. It continuously monitors the CPU and memory utilization of the master forwarding adapter workgroup and the slave forwarding adapter workgroup. These metrics are key parameters for measuring system load and performance, helping the system understand its current resource usage and demands.

[0082] Based on HPA monitoring results, the system can automatically increase the number of nodes in the primary and secondary forwarding adapter workgroups during peak business hours (i.e., when CPU and memory utilization are high). This ensures sufficient resources to process data under high load, avoiding forwarding latency and system overload. Conversely, during off-peak business hours (i.e., when CPU and memory utilization are low), the system can automatically reduce the number of nodes in these workgroups. This helps save resources, reduce costs, and maintain system responsiveness.

[0083] This approach of dynamically adjusting resources based on actual load demonstrates the system's elastic scaling capability. It allows the system to flexibly adjust resource allocation under different business demands, operating in an optimal manner. By reducing the number of nodes during off-peak hours, the system can reduce unnecessary resource waste, thereby lowering operating costs. Simultaneously, this dynamic adjustment also helps improve resource utilization efficiency.

[0084] In this embodiment, HPA is used to monitor and automatically adjust the size of the data forwarding adapter workgroup to adapt to different business loads, thereby optimizing resource utilization and cost control while maintaining high system performance and high availability. By increasing the number of nodes during peak business periods, the system can ensure high performance and high availability even under high load conditions, meeting business needs.

[0085] In an exemplary embodiment, the sum of the maximum number of working replicas M obtained by expanding the primary forwarding adapter working group and the minimum number of hot standby replicas N obtained by shrinking the secondary forwarding adapter working group remains an odd number.

[0086] Specifically, keeping M+N an odd number ensures that there is always a clearly defined master forwarding adapter workgroup and slave forwarding adapter workgroup. This guarantees that a master node will always be elected during master-slave election, preventing two or more nodes from simultaneously believing themselves to be the master node, thus ensuring system consistency and stability.

[0087] In distributed systems, if the number of master nodes is even, a phenomenon known as "split-brain" may occur, where the system's nodes split into two independent parts, each with its own master node, leading to data inconsistency. Maintaining M+N as an odd number can reduce the risk of this split.

[0088] When M+N is odd, the election logic can be simplified because there is always a definite master node. This simplifies the election process and improves system responsiveness and reliability. By ensuring a definite master node, the system can quickly failover in the event of a master node failure, reducing system downtime and improving overall availability. Simultaneously, this technique also allows the system to optimize resources while maintaining high availability. During off-peak periods, resources can be saved by reducing the number of replicas M in the master forwarding adapter workgroup and increasing the number of replicas N in the slave forwarding adapter workgroup.

[0089] In this embodiment, the system can dynamically adjust the values ​​of M and N according to business needs and load conditions to keep M+N an odd number. In this way, the system can flexibly adjust resource allocation under different business scenarios, while ensuring that there is always a clearly defined master node to handle data forwarding tasks.

[0090] In one exemplary embodiment, if there is no data forwarding in the primary forwarding adapter workgroup, the corresponding primary forwarding identity is cancelled;

[0091] If no master forwarding identity exists among the data forwarding adapters in the entire forwarding adapter virtual cluster, a master-slave election operation is initiated using any data forwarding adapter in the slave forwarding adapter workgroup. If any data forwarding adapter is elected as the master forwarding identity, the corresponding slave forwarding adapter workgroup is converted into the master forwarding adapter workgroup.

[0092] Specifically, the primary responsibility of the main forwarding adapter working group is to receive data pushed by the data acquisition engine and forward it to a remote mass data storage facility. If the main forwarding adapter working group is unable to receive data for some reason (such as a data acquisition engine failure), then they are not actually performing the data forwarding task.

[0093] If there is no data forwarding in the primary forwarding adapter workgroup, the system will revoke the "primary forwarding identity" of these adapters. This is to ensure that resources are not wasted and to trigger a re-election so that data forwarding tasks can be transferred to adapters that are functioning properly.

[0094] When none of the data forwarding adapters in the entire forwarding adapter virtual cluster holds the primary forwarding identity, it means that the system currently has no active data forwarding path. In this case, the system needs to re-elect a primary forwarding adapter workgroup to restore data forwarding functionality.

[0095] The system will initiate a master-slave election operation using any data forwarding adapter in the slave forwarding adapter workgroup. Under normal circumstances, any data forwarding adapter in the slave forwarding adapter workgroup does not participate in data forwarding, but they are designed as hot backups ready to take over the master forwarding task at any time.

[0096] During the re-election process, if any data forwarding adapter in the forwarding adapter workgroup successfully obtains the primary forwarding identity, then this data forwarding adapter and its workgroup will be converted into the primary forwarding adapter workgroup and begin to undertake the task of data forwarding.

[0097] In this embodiment, the system's resilience and high availability are ensured in this way. Even if the primary forwarding adapter working group fails or is unable to receive data, the system can quickly restore data forwarding service with minimal delay by electing a new primary forwarding adapter from the working group. Seamless switching is achieved, ensuring the continuity of data forwarding and guaranteeing uninterrupted service even in the event of failures or other anomalies, thus ensuring the continuity of data forwarding service and the stability of the system.

[0098] In one exemplary embodiment, before pushing the received data to be forwarded to the remote data storage via the main forwarding adapter working group, the method further includes:

[0099] The data to be forwarded is converted to a different format to fit the data storage format of the remote data storage device.

[0100] Specifically, different remote data repositories may support different data formats and protocols. To ensure that data collected from a data acquisition engine can be correctly processed by the remote storage system, the data must be converted into a specific format supported by the remote system. Data collected by a data acquisition engine (such as Prometheus) may exist in a specific format, while a remote data repository (such as a time-series database TSDB) may require a different format. Therefore, data needs to be format-converted before being pushed to remote storage.

[0101] The data forwarding adapter is responsible not only for forwarding data from the data acquisition engine to remote storage, but also for performing necessary format conversions during the forwarding process. This conversion ensures data compatibility and consistency during transmission. Appropriate format conversion can improve data storage efficiency. For example, some data storage systems may be optimized for specific data formats, making storage and retrieval operations more efficient.

[0102] Format conversion includes data preprocessing steps, such as data integrity, data privacy, data compression, encryption, and adding metadata, to meet the requirements of remote storage systems.

[0103] In this embodiment, by converting the format of the data to be forwarded, it is ensured that the data can be correctly received and stored by the remote data storage device. This demonstrates the bridging role of the data forwarding adapter in the data acquisition and storage system, ensuring smooth data flow and compatibility.

[0104] The most detailed embodiment of this application is as follows:

[0105] like Figure 4 As shown, the specific operating mechanism of the above preferred embodiment is as follows:

[0106] 1. Deploy dual local data engines, Prometheus, within the container cluster to collect data concurrently.

[0107] 2. After the blue and green forwarding adapter instances in the container cluster start, they are added to the forwarding adapter virtual cluster. If there is no master node in the cluster, a master-slave election will be conducted. Once any instance wins the master-slave election, it will be granted the master forwarding identity.

[0108] 3. The primary forwarding adapter workgroup will automatically expand to the maximum number of working replicas M to meet the computing power requirements of the primary forwarding adapter workgroup, while the secondary forwarding adapter workgroup will automatically shrink to the minimum number of hot standby replicas N to reduce resource waste in the forwarding adapter cluster.

[0109] 4. After receiving the data submitted by the data acquisition engine, the main forwarding adapter working group performs data format conversion to match the remote data storage interface, and then pushes the data to be pushed to the remote centralized large-capacity data storage; the secondary forwarding adapter working group directly discards the received data to be pushed.

[0110] 5. Receive data submitted from the main forwarding adapter working group and further store it in a remote centralized large-capacity data storage, such as OSS object storage or a TSBDB storage with a larger storage capacity, such as VictoriaMetrics.

[0111] 6. If the data acquisition engine corresponding to the primary forwarding adapter workgroup malfunctions, such as when the instance restarts or the process freezes, all forwarding adapters will have no data to forward. The primary forwarding adapter workgroup will gradually relinquish its primary forwarding role until there are no available primary nodes in the entire cluster.

[0112] 7. At this point, the forwarding adapter working group will initiate a master-slave election, and the process will return to step 2 above, completing the dynamic switching of the master-slave forwarding identity of the blue-green forwarding adapter.

[0113] This process is repeated, and the above operating mechanism ensures high availability of data collection and delivery between the two data acquisition engines.

[0114] Furthermore, the data structure involved in the mixed primary election of the blue-green forwarding adapters is as follows:

[0115]

[0116] Furthermore, the expansion and contraction of the blue and green forwarding adapters are uniformly controlled by the forwarding adapter cluster controller. To ensure that the forwarding adapter cluster successfully elects a master, the sum of the maximum number of working replicas M and the minimum number of hot standby replicas N must be kept odd.

[0117] Furthermore, the forwarding adapter cluster controller can provide HPA capabilities based on the observed CPU and memory utilization of the blue-green forwarding adapters. In other words, based on the indicator thresholds set by HPA, the number of nodes in the master forwarding adapter working group and the slave forwarding adapter working group is increased during peak business periods to meet the forwarding computing power requirements, and the number of nodes in the master forwarding adapter working group and the slave forwarding adapter working group is reduced during off-peak business periods to reduce resource waste.

[0118] Furthermore, assuming the availability of a single data acquisition engine is 99.9%, the concurrent acquisition engine availability of two data acquisition engines (active-active setup) is: 1 - (1 - 99.9%) 2= 99.9999%, availability has been improved.

[0119] Furthermore, such as Figure 5 As shown, theoretically, the availability of the data acquisition system can be further improved by horizontally expanding the number of data acquisition engines. An extended embodiment based on the technical solution described in this invention is as follows:

[0120] In the extended embodiment, the original forwarding adapter cluster controller is upgraded to a data acquisition and forwarding controller, which simultaneously manages the cluster of the data acquisition engine and the cluster of the forwarding adapter.

[0121] The data acquisition engine is clustered and can be dynamically expanded to 2, 3, 5 or even more units depending on business needs.

[0122] The cluster availability calculation expression based on this preferred embodiment is as follows:

[0123] Multi-active data engine availability = 1 - (1 - single data engine availability) n Where n is the number of data acquisition engines in the cluster.

[0124] For example, assuming a single data acquisition engine has an availability of 99.9%, the concurrent acquisition engine availability of three data acquisition engines (multi-active) is: 1 - (1 - 99.9%). 3 =99.9999999%.

[0125] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0126] Based on the same inventive concept, this application also provides a data acquisition device for implementing the data acquisition method described above. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data acquisition device embodiments provided below can be found in the limitations of the data acquisition method described above, and will not be repeated here.

[0127] In one exemplary embodiment, such as Figure 6 As shown, a data acquisition device is provided, comprising:

[0128] The data acquisition module 602 is used to acquire data in parallel using at least two data acquisition engines, store the acquired data locally, and push the data to the data forwarding adapter corresponding to each data acquisition engine.

[0129] The master-slave election module 604 is used to perform master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group.

[0130] The data forwarding control module 606 is used to push the received data to be forwarded to a remote data storage device through the master forwarding adapter working group, and to discard the received data to be forwarded through the slave forwarding adapter working group.

[0131] The master-slave election module 604 is also used to trigger a new master-slave election for at least two data forwarding adapters in the event that the data acquisition engine corresponding to the master forwarding adapter working group fails, so that each data forwarding adapter in the master forwarding adapter working group will give up its master forwarding identity and obtain a new master forwarding adapter working group and slave forwarding adapter working group.

[0132] In an exemplary embodiment, the master-slave election module 604 is specifically used to perform a hybrid master-slave election for at least two data forwarding adapters; after any data forwarding adapter elects a master forwarding identity, it forms a master forwarding adapter working group and distributes the master forwarding identity to all data forwarding adapters in the master forwarding adapter working group; it expands the master forwarding adapter working group and shrinks the slave forwarding adapter working group.

[0133] In an exemplary embodiment, the data forwarding control module 606 is further configured to monitor the CPU utilization and memory utilization of the primary forwarding adapter workgroup and the secondary forwarding adapter workgroup respectively, based on the horizontal Pod auto-expander, as monitoring results; and based on the monitoring results, increase the number of nodes of the primary forwarding adapter workgroup and the secondary forwarding adapter workgroup respectively during peak business periods, and decrease the number of nodes of the primary forwarding adapter workgroup and the secondary forwarding adapter workgroup respectively during off-peak business periods.

[0134] In an exemplary embodiment, the sum of the maximum number of working replicas M obtained by expanding the primary forwarding adapter working group and the minimum number of hot standby replicas N obtained by shrinking the secondary forwarding adapter working group remains an odd number.

[0135] In an exemplary embodiment, the data forwarding control module 606 is used to cancel the corresponding primary forwarding identity when there is no data forwarding in the primary forwarding adapter workgroup; when there is no primary forwarding identity in any data forwarding adapter of the entire forwarding adapter virtual cluster, it initiates a primary-slave election operation using any data forwarding adapter in the secondary forwarding adapter workgroup, and when any data forwarding adapter is elected as the primary forwarding identity, it converts the corresponding secondary forwarding adapter workgroup into the primary forwarding adapter workgroup.

[0136] In one exemplary embodiment, the data forwarding control module 606 is used to convert the format of the data to be forwarded to adapt to the data storage format of the remote data storage device.

[0137] Each module in the aforementioned data acquisition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0138] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a data acquisition method.

[0139] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0140] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0141] Data is collected in parallel using at least two data acquisition engines. The collected data is stored locally and then pushed to the data forwarding adapter corresponding to each data acquisition engine.

[0142] Perform a master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group;

[0143] The primary forwarding adapter working group pushes the received data to be forwarded to the remote data storage, and the secondary forwarding adapter working group discards the received data to be forwarded.

[0144] In the event of a failure of the data acquisition engine corresponding to the primary forwarding adapter working group, each data forwarding adapter in the primary forwarding adapter working group will relinquish its primary forwarding identity to trigger a new primary-secondary election for at least two data forwarding adapters, resulting in a new primary forwarding adapter working group and a new secondary forwarding adapter working group.

[0145] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0146] Hybrid master-slave election is performed for at least two data forwarding adapters;

[0147] After any data forwarding adapter elects the primary forwarding identity, it forms a primary forwarding adapter working group and distributes the primary forwarding identity to all data forwarding adapters in the primary forwarding adapter working group;

[0148] Expand the capacity of the primary forwarding adapter workgroup and shrink the capacity of the secondary forwarding adapter workgroup.

[0149] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0150] Based on the horizontal Pod auto-expander, monitor the CPU utilization and memory utilization of the primary forwarding adapter workgroup and the secondary forwarding adapter workgroup respectively as monitoring results;

[0151] Based on monitoring results, the number of nodes in both the primary forwarding adapter working group and the secondary forwarding adapter working group will be increased during peak business hours, and the number of nodes in both groups will be reduced during off-peak business hours.

[0152] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0153] The sum of the maximum number of working replicas M obtained by expanding the primary forwarding adapter working group and the minimum number of hot standby replicas N obtained by shrinking the secondary forwarding adapter working group remains an odd number.

[0154] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0155] If there is no data forwarding in the primary forwarding adapter workgroup, cancel the corresponding primary forwarding identity;

[0156] If no master forwarding identity exists among the data forwarding adapters in the entire forwarding adapter virtual cluster, a master-slave election operation is initiated using any data forwarding adapter in the slave forwarding adapter workgroup. If any data forwarding adapter is elected as the master forwarding identity, the corresponding slave forwarding adapter workgroup is converted into the master forwarding adapter workgroup.

[0157] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0158] The data to be forwarded is converted to a different format to fit the data storage format of the remote data storage device.

[0159] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0160] Data is collected in parallel using at least two data acquisition engines. The collected data is stored locally and then pushed to the data forwarding adapter corresponding to each data acquisition engine.

[0161] Perform a master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group;

[0162] The primary forwarding adapter working group pushes the received data to be forwarded to the remote data storage, and the secondary forwarding adapter working group discards the received data to be forwarded.

[0163] In the event of a failure of the data acquisition engine corresponding to the primary forwarding adapter working group, each data forwarding adapter in the primary forwarding adapter working group will relinquish its primary forwarding identity to trigger a new primary-secondary election for at least two data forwarding adapters, resulting in a new primary forwarding adapter working group and a new secondary forwarding adapter working group.

[0164] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0165] Hybrid master-slave election is performed for at least two data forwarding adapters;

[0166] After any data forwarding adapter elects the primary forwarding identity, it forms a primary forwarding adapter working group and distributes the primary forwarding identity to all data forwarding adapters in the primary forwarding adapter working group;

[0167] Expand the capacity of the primary forwarding adapter workgroup and shrink the capacity of the secondary forwarding adapter workgroup.

[0168] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0169] Based on the horizontal Pod auto-expander, monitor the CPU utilization and memory utilization of the primary forwarding adapter workgroup and the secondary forwarding adapter workgroup respectively as monitoring results;

[0170] Based on monitoring results, the number of nodes in both the primary forwarding adapter working group and the secondary forwarding adapter working group will be increased during peak business hours, and the number of nodes in both groups will be reduced during off-peak business hours.

[0171] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0172] The sum of the maximum number of working replicas M obtained by expanding the primary forwarding adapter working group and the minimum number of hot standby replicas N obtained by shrinking the secondary forwarding adapter working group remains an odd number.

[0173] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0174] If there is no data forwarding in the primary forwarding adapter workgroup, cancel the corresponding primary forwarding identity;

[0175] If no master forwarding identity exists among the data forwarding adapters in the entire forwarding adapter virtual cluster, a master-slave election operation is initiated using any data forwarding adapter in the slave forwarding adapter workgroup. If any data forwarding adapter is elected as the master forwarding identity, the corresponding slave forwarding adapter workgroup is converted into the master forwarding adapter workgroup.

[0176] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0177] The data to be forwarded is converted to a different format to fit the data storage format of the remote data storage device.

[0178] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0179] Data is collected in parallel using at least two data acquisition engines. The collected data is stored locally and then pushed to the data forwarding adapter corresponding to each data acquisition engine.

[0180] Perform a master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group;

[0181] The primary forwarding adapter working group pushes the received data to be forwarded to the remote data storage, and the secondary forwarding adapter working group discards the received data to be forwarded.

[0182] In the event of a failure of the data acquisition engine corresponding to the primary forwarding adapter working group, each data forwarding adapter in the primary forwarding adapter working group will relinquish its primary forwarding identity to trigger a new primary-secondary election for at least two data forwarding adapters, resulting in a new primary forwarding adapter working group and a new secondary forwarding adapter working group.

[0183] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0184] Hybrid master-slave election is performed for at least two data forwarding adapters;

[0185] After any data forwarding adapter elects the primary forwarding identity, it forms a primary forwarding adapter working group and distributes the primary forwarding identity to all data forwarding adapters in the primary forwarding adapter working group;

[0186] Expand the capacity of the primary forwarding adapter workgroup and shrink the capacity of the secondary forwarding adapter workgroup.

[0187] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0188] Based on the horizontal Pod auto-expander, monitor the CPU utilization and memory utilization of the primary forwarding adapter workgroup and the secondary forwarding adapter workgroup respectively as monitoring results;

[0189] Based on monitoring results, the number of nodes in both the primary forwarding adapter working group and the secondary forwarding adapter working group will be increased during peak business hours, and the number of nodes in both groups will be reduced during off-peak business hours.

[0190] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0191] The sum of the maximum number of working replicas M obtained by expanding the primary forwarding adapter working group and the minimum number of hot standby replicas N obtained by shrinking the secondary forwarding adapter working group remains an odd number.

[0192] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0193] If there is no data forwarding in the primary forwarding adapter workgroup, cancel the corresponding primary forwarding identity;

[0194] If no master forwarding identity exists among the data forwarding adapters in the entire forwarding adapter virtual cluster, a master-slave election operation is initiated using any data forwarding adapter in the slave forwarding adapter workgroup. If any data forwarding adapter is elected as the master forwarding identity, the corresponding slave forwarding adapter workgroup is converted into the master forwarding adapter workgroup.

[0195] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0196] The data to be forwarded is converted to a different format to fit the data storage format of the remote data storage device.

[0197] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0198] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0199] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0200] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data acquisition method, characterized by, The method comprises: parallel data collection using at least two data collection engines, local storage of the collected data, and pushing the data to a data forwarding adapter corresponding to each data collection engine; master-slave election of the at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group; pushing, by the master forwarding adapter working group, received data to be forwarded to a remote data storage, and discarding, by the slave forwarding adapter working group, the received data to be forwarded; in the case of a failure of a data collection engine corresponding to the master forwarding adapter working group, each data forwarding adapter in the master forwarding adapter working group will give up the master forwarding identity to trigger a re-master-slave election of the at least two data forwarding adapters to obtain a new master forwarding adapter working group and a slave forwarding adapter working group.

2. The method of claim 1, wherein, The master-slave election of the at least two data forwarding adapters to determine the master forwarding adapter working group and the slave forwarding adapter working group comprises: hybrid master-slave election of the at least two data forwarding adapters; forming the master forwarding adapter working group after any data forwarding adapter elects a master forwarding identity, and distributing the master forwarding identity to all data forwarding adapters in the master forwarding adapter working group; expanding the master forwarding adapter working group and shrinking the slave forwarding adapter working group.

3. The method of claim 1, wherein, The method further comprises: monitoring, according to a horizontal pod autoscaler, CPU utilization and memory utilization of the master forwarding adapter working group and the slave forwarding adapter working group as a monitoring result; according to the monitoring result, increasing the number of nodes of the master forwarding adapter working group and the slave forwarding adapter working group during a business peak period, and reducing the number of nodes of the master forwarding adapter working group and the slave forwarding adapter working group during a business trough period.

4. The method of claim 2, wherein, The sum of the maximum working state copy number M obtained by expanding the master forwarding adapter working group and the minimum hot standby state copy number N obtained by shrinking the slave forwarding adapter working group is an odd number.

5. The method of claim 1, wherein, The method further comprises: canceling the corresponding master forwarding identity in the case of no data forwarding in the master forwarding adapter working group; in the case of no master forwarding identity in all data forwarding adapters of the entire forwarding adapter virtual cluster, initiating a master-slave election operation using any data forwarding adapter in the slave forwarding adapter working group, and in the case of any data forwarding adapter electing a master forwarding identity, converting the corresponding slave forwarding adapter working group into a master forwarding adapter working group.

6. The method of claim 1, wherein, Before pushing the received data to be forwarded to the remote data storage by the master forwarding adapter working group, the method further comprises: format conversion of the data to be forwarded to adapt to the data storage format of the remote data storage.

7. A data acquisition device, characterized by The apparatus comprises: a data collection module for parallel data collection using at least two data collection engines, local storage of the collected data, and pushing the data to a data forwarding adapter corresponding to each data collection engine; a master-slave election module for master-slave election of the at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group; The data forwarding control module is configured to push the received data to be forwarded to the remote data storage through the master forwarding adapter team, and discard the received data to be forwarded through the slave forwarding adapter team; The master-slave election module is further configured to, in a case where the data acquisition engine corresponding to the master forwarding adapter team fails, cause each data forwarding adapter in the master forwarding adapter team to give up the master forwarding identity, so as to trigger a re-election of the master-slave election of the at least two data forwarding adapters, and obtain a new master forwarding adapter team and a slave forwarding adapter team.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

Citation Information

Patent Citations

  • Elastic scaling method and system for power system

    CN107515809A

  • Data collector capacity expansion and contraction method and device, computer equipment, readable storage medium and computer program product

    CN118659974A