Data acquisition method and device, computer equipment and storage medium

Abstract

The invention relates to a data acquisition method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: collecting data in parallel by using at least two data collection engines, locally storing the collected data, and forwarding and pushing the data to a data forwarding adapter corresponding to each data collection engine; performing master-slave election on the at least two data forwarding adapters; pushing the received to-be-forwarded data to a far-end data memory through the master forwarding adapter working group, and discarding the received to-be-forwarded data through the slave forwarding adapter working group; and under the condition that the data acquisition engine has a fault, performing master-slave election again to obtain a new master forwarding adapter working group and a new slave forwarding adapter working group, and continuing to execute the steps of the master forwarding adapter working group and the slave forwarding adapter working group. By adopting the method, zero-interruption and high-elasticity data acquisition and pushing capability can be realized, so that the elastic capacity and high availability in a large-scale data scene are ensured.

Description

Technical Field

[0001] The present application relates to the technical field of data acquisition, and in particular to a data acquisition method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] In the fields of cloud computing, container virtualization, big data, and observability, data collection is a key link. In hybrid cloud or distributed container cloud scenarios, data collection usually involves collecting local data or indicators in clusters in various regions, then transferring them to the local domain forwarding adapter, and finally pushing them to remote centralized large-capacity storage. This model performs well in small and medium-scale data scenarios, but there are problems in large-scale data scenarios: in clusters with thousands of nodes, the amount of data to be collected and stored locally is huge, and restarting the data collection engine may cause data collection and push interruptions. For example, in a Kubernetes cluster with 3,000 nodes, the time series database engine Prometheus needs to load local old data when it is restarted. The loading time of 7 days of full time series data may range from more than ten minutes to dozens of minutes, during which time the data collection and push services will be interrupted.

[0003] However, the current data forwarding adapters have forwarding capacity limitations. During business peak periods, they process large amounts of data, resulting in high forwarding delays. During business troughs, the computing power of the forwarding adapters is idle, resulting in a waste of resources. To solve these problems, common avoidance measures in the industry include reducing the data storage cycle on the data collection engine side, reducing the number of data collection indicators, or deploying multiple data collection engines for data collection and push, storing redundant copies of data remotely, and performing data deduplication during data calculations in remote storage. Although these measures can improve the availability of data collection services, they either sacrifice the scale of available business data or the computing efficiency of business applications.

[0004] Therefore, there is an urgent need for a highly available data collection method, apparatus, computer equipment, computer-readable storage medium and computer program product that can achieve zero-interruption, highly elastic data collection and push capabilities to ensure elastic capacity and high availability in large-scale data scenarios. Summary of the invention

[0005] Based on this, it is necessary to provide a data collection method, device, computer equipment, computer-readable storage medium and computer program product that can achieve zero-interruption, high-elasticity data collection and push capabilities to ensure elastic capacity and high availability in large-scale data scenarios in response to the above-mentioned technical problems.

[0006] In a first aspect, the present application provides a data collection method, comprising:

[0007] Using at least two data collection engines to collect data in parallel, storing the collected data locally, and forwarding the data to a data forwarding adapter corresponding to each data collection engine;

[0008] Performing master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group;

[0009] Pushing the received data to be forwarded to the remote data storage through the master forwarding adapter working group, and discarding the received data to be forwarded through the slave forwarding adapter working group;

[0010] When the data collection engine corresponding to the master forwarding adapter working group fails, each data forwarding adapter in the master forwarding adapter working group will give up the master forwarding identity to trigger another master-slave election for at least two data forwarding adapters to obtain a new master forwarding adapter working group and slave forwarding adapter working group.

[0011] In one embodiment, performing master-slave election on at least two data forwarding adapters to determine a master forwarding adapter working group and a slave forwarding adapter working group includes:

[0012] Perform hybrid master-slave election on at least two data forwarding adapters;

[0013] After any data forwarding adapter is elected to obtain the master forwarding identity, a master forwarding adapter working group is formed, and the master forwarding identity is distributed to all data forwarding adapters in the master forwarding adapter working group;

[0014] The capacity of the master forwarding adapter working group is expanded, and the capacity of the slave forwarding adapter working group is reduced.

[0015] In one of the embodiments, according to the horizontal Pod autoscaler, the CPU utilization and memory utilization of the master forwarding adapter working group and the slave forwarding adapter working group are monitored as monitoring results;

[0016] According to the monitoring result, the number of nodes of the master forwarding adapter working group and the slave forwarding adapter working group is increased during the peak business period, and the number of nodes of the master forwarding adapter working group and the slave forwarding adapter working group is reduced during the low business period.

[0017] In one embodiment, the sum of the maximum number of working copies M obtained by expanding the capacity of the primary forwarding adapter workgroup and the minimum number of hot standby copies N obtained by shrinking the capacity of the secondary forwarding adapter workgroup is maintained as an odd number.

[0018] In one of the embodiments, when there is no data forwarding in the master forwarding adapter working group, the corresponding master forwarding identity is cancelled;

[0019] When there is no master forwarding identity in the data forwarding adapters of the entire forwarding adapter virtual cluster, a master-slave election operation is initiated using any data forwarding adapter in the slave forwarding adapter workgroup. When any data forwarding adapter obtains the master forwarding identity through election, the corresponding slave forwarding adapter workgroup is converted to the master forwarding adapter workgroup.

[0020] In one embodiment, before pushing the received data to be forwarded to the remote data storage through the main forwarding adapter working group, the method further includes:

[0021] The data to be forwarded is formatted to adapt to the data storage format of the remote data storage.

[0022] In a second aspect, the present application also provides a data acquisition device, comprising:

[0023] A data acquisition module, used to collect data in parallel using at least two data acquisition engines, store the collected data locally, and forward the data to a data forwarding adapter corresponding to each data acquisition engine;

[0024] A master-slave election module is used to perform master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group;

[0025] A data forwarding control module, used to push the received data to be forwarded to a remote data storage device through a master forwarding adapter working group, and to discard the received data to be forwarded through a slave forwarding adapter working group;

[0026] The master-slave election module is also used to trigger another master-slave election activity for at least two data forwarding adapters when a data acquisition engine corresponding to the master forwarding adapter working group fails. This will cause each data forwarding adapter in the master forwarding adapter working group to give up its master forwarding identity, thereby obtaining a new master forwarding adapter working group and a slave forwarding adapter working group.

[0027] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0028] Using at least two data collection engines to collect data in parallel, storing the collected data locally, and forwarding the data to a data forwarding adapter corresponding to each data collection engine;

[0029] Performing master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group;

[0030] Pushing the received data to be forwarded to the remote data storage through the master forwarding adapter working group, and discarding the received data to be forwarded through the slave forwarding adapter working group;

[0031] When the data collection engine corresponding to the master forwarding adapter working group fails, each data forwarding adapter in the master forwarding adapter working group will give up the master forwarding identity to trigger another master-slave election for at least two data forwarding adapters to obtain a new master forwarding adapter working group and slave forwarding adapter working group.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0033] Using at least two data collection engines to collect data in parallel, storing the collected data locally, and forwarding the data to a data forwarding adapter corresponding to each data collection engine;

[0034] Performing master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group;

[0035] Pushing the received data to be forwarded to the remote data storage through the master forwarding adapter working group, and discarding the received data to be forwarded through the slave forwarding adapter working group;

[0036] When the data collection engine corresponding to the master forwarding adapter working group fails, each data forwarding adapter in the master forwarding adapter working group will give up the master forwarding identity to trigger another master-slave election for at least two data forwarding adapters to obtain a new master forwarding adapter working group and slave forwarding adapter working group.

[0037] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0038] Using at least two data collection engines to collect data in parallel, storing the collected data locally, and forwarding the data to a data forwarding adapter corresponding to each data collection engine;

[0039] Performing master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group;

[0040] Pushing the received data to be forwarded to the remote data storage through the master forwarding adapter working group, and discarding the received data to be forwarded through the slave forwarding adapter working group;

[0041] When the data collection engine corresponding to the master forwarding adapter working group fails, each data forwarding adapter in the master forwarding adapter working group will give up the master forwarding identity to trigger another master-slave election for at least two data forwarding adapters to obtain a new master forwarding adapter working group and slave forwarding adapter working group.

[0042] The above-mentioned data acquisition method, device, computer equipment, computer-readable storage medium and computer program product use at least two data acquisition engines to collect data in parallel, store the collected data locally, and forward the data to the data forwarding adapter corresponding to each data acquisition engine. By collecting data in parallel, the efficiency and throughput of data acquisition are improved, and a large amount of data can be quickly processed in a large-scale data scenario. At least two data forwarding adapters are elected as masters and slaves to obtain a master forwarding adapter working group and a slave forwarding adapter working group. Through the master-slave election mechanism, the high availability of the data forwarding adapter is achieved, and the continuous push of data and the stability of the system are ensured. Through the master forwarding adapter working group, the received data to be forwarded is pushed to the remote data storage, and through the slave forwarding adapter working group, the received data to be forwarded is discarded. This mechanism optimizes resource utilization, the master forwarding adapter working group is responsible for data forwarding, and the slave forwarding adapter working group does not consume resources, reducing resource waste. In the case that the data acquisition engine corresponding to the master forwarding adapter working group fails, the master-slave election is performed on at least two data forwarding adapters again, and the steps of obtaining a new master forwarding adapter working group and a slave forwarding adapter working group are obtained and the step of data forwarding is continued. It realizes fast recovery of faults and hot switching between master and slave, ensures zero interruption of data collection and push, and improves the elasticity and reliability of the system. In short, this application can achieve the purpose of zero interruption and highly elastic data collection and push capabilities in large-scale data scenarios, ensuring the elastic capacity and high availability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 An application environment diagram of a data collection method in an embodiment;

[0045] Figure 2 A schematic diagram of a flow chart of a data collection method in one embodiment;

[0046] Figure 3A schematic diagram of a flow chart of a data collection method in another embodiment;

[0047] Figure 4 It is a device architecture diagram of two data acquisition engines in the most detailed embodiment;

[0048] Figure 5 It is a device architecture diagram of three data acquisition engines in the most detailed embodiment;

[0049] Figure 6 is a structural block diagram of a data acquisition device in one embodiment;

[0050] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] The data collection method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0053] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0054] In an exemplary embodiment, Figure 2 As shown, a data acquisition method is provided, which is applied to Figure 1 The server in the example is used to illustrate, including the following steps S202 to S208. Among them:

[0055] Step S202: using at least two data collection engines to collect data in parallel, storing the collected data locally, and forwarding the data to a data forwarding adapter corresponding to each data collection engine.

[0056] Specifically, multiple data collection engines are deployed, and these engines can work simultaneously to collect data from different data sources or different parts of the same data source in a parallel manner.

[0057] After collecting data, each data collection engine will store the data locally to ensure data persistence. Local storage can be any form of storage media, such as hard disk, SSD, etc., which provides an instant backup for the data and allows data not to be lost when there are network problems or remote storage is unavailable. At the same time, it can be connected to the data collection engine through "remote writewebhook". The data collected by the data collection engine is not only stored locally, but also pushed to the corresponding data forwarding adapter in real time. This design allows data to be forwarded instantly while being collected, reducing the delay in data transmission.

[0058] Step S204: performing master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group.

[0059] Specifically, this embodiment adopts a master-slave election mechanism, which is a common distributed system design pattern for electing a "master" instance from a group of service instances, while the other instances serve as "slave" instances. In the context of data forwarding adapters, this mechanism is used to determine which adapter instances will be responsible for actual data forwarding work and which ones will serve as backups.

[0060] Step S206: Push the received data to be forwarded to the remote data storage through the master forwarding adapter working group, and discard the received data to be forwarded through the slave forwarding adapter working group.

[0061] Specifically, once the master forwarding adapter working group is elected, it will be responsible for receiving the data to be forwarded from the data collection engine and then pushing it to the remote centralized large-capacity data storage. The master forwarding adapter working group refers to the set of data forwarding adapters that are selected as the master node during the election process, and they will undertake the main work of data forwarding.

[0062] The slave forwarding adapter workgroup refers to the data forwarding adapter set that was not selected as the master node during the election process. These adapters exist as hot backups. They will not participate in data forwarding under normal circumstances, but will take over when the master forwarding adapter workgroup fails to ensure the continuity of data forwarding and high availability of the system.

[0063] The master forwarding adapter working group and the slave forwarding adapter working group can dynamically adjust the number of instances according to actual business needs and load conditions.

[0064] Step S208, when the data collection engine corresponding to the master forwarding adapter working group fails, each data forwarding adapter in the master forwarding adapter working group will give up the master forwarding identity to trigger another master-slave election activity for at least two data forwarding adapters to obtain a new master forwarding adapter working group and slave forwarding adapter working group.

[0065] Specifically, the status of the data collection engine is monitored. Once a failure is detected in the data collection engine corresponding to the master forwarding adapter workgroup (for example, instance restart or process deadlock), the system triggers the fault recovery process and restarts the master-slave election mechanism. The election process involves all available data forwarding adapters, with the goal of selecting a new master forwarding adapter workgroup from them.

[0066] During the election process, a data forwarding adapter that was originally part of a slave forwarding adapter working group may be elected as a master forwarding adapter and will assume the task of data forwarding.

[0067] Once the data forwarding adapter with the master forwarding identity is elected, it will form a new master forwarding adapter working group with its corresponding data collection engine (if the engine that did not participate in the master task is still healthy). At the same time, other data forwarding adapter instances that have not been elected as the master will automatically become slave forwarding adapter working groups. The new master forwarding adapter working group will take over the data forwarding task and continue to push the received data to be forwarded to the remote data storage. The slave forwarding adapter working group will continue to serve as a backup, waiting for the next possible election.

[0068] This process ensures that even if the main data collection engine fails, the system can quickly recover and continue to perform data collection and push tasks, achieving system elasticity and zero-interruption service. After the new master-forwarding adapter workgroup is formed, the system may adjust the size of the master-slave workgroup according to the current load situation to optimize resource usage and meet business needs. Through this design, the system can respond quickly when a failure occurs, realize master-slave hot switching, ensure the continuity of data collection and forwarding services, and thus improve the reliability and stability of the entire data collection system.

[0069] In the above data acquisition method, at least two data acquisition engines are used to collect data in parallel, the collected data is stored locally, and the data is forwarded to the data forwarding adapter corresponding to each data acquisition engine. By collecting data in parallel, the efficiency and throughput of data acquisition are improved, and a large amount of data can be quickly processed in a large-scale data scenario. At least two data forwarding adapters are elected as masters and slaves to obtain a master forwarding adapter working group and a slave forwarding adapter working group. Through the master-slave election mechanism, the high availability of the data forwarding adapter is achieved, and the continuous push of data and the stability of the system are ensured. Through the master forwarding adapter working group, the received data to be forwarded is pushed to the remote data storage, and through the slave forwarding adapter working group, the received data to be forwarded is discarded. This mechanism optimizes resource utilization, the master forwarding adapter working group is responsible for data forwarding, and the slave forwarding adapter working group does not consume resources, reducing resource waste. In the case that the data acquisition engine corresponding to the master forwarding adapter working group fails, the master-slave election is performed again on at least two data forwarding adapters to obtain the steps of the master forwarding adapter working group and the slave forwarding adapter working group and continue to perform subsequent data forwarding work. Rapid recovery of faults and master-slave hot switching are achieved, ensuring zero interruption of data acquisition and push, and improving the elasticity and reliability of the system. In summary, this application can achieve the goal of zero-interruption, highly elastic data collection and push capabilities in large-scale data scenarios, ensuring the elastic capacity and high availability of the system.

[0070] In an exemplary embodiment, Figure 3 As shown, performing master-slave election on at least two data forwarding adapters to determine a master forwarding adapter working group and a slave forwarding adapter working group includes:

[0071] Step S302, performing a hybrid master-slave election on at least two data forwarding adapters;

[0072] Step S304, after any data forwarding adapter elects a master forwarding identity, a master forwarding adapter working group is formed, and the master forwarding identity is distributed to all data forwarding adapters in the master forwarding adapter working group;

[0073] Step S306: Expand the capacity of the master forwarding adapter working group and reduce the capacity of the slave forwarding adapter working group.

[0074] Specifically, multiple data forwarding adapter instances are used to participate in an election process to determine which instances will assume the role of the primary forwarding identity and which will serve as backups for the secondary forwarding identity. This election process may involve complex algorithms to ensure the fairness of the election and the high availability of the system.

[0075] During the election process, if any data forwarding adapter is successfully elected as the master forwarding adapter, then this adapter instance will be included in the master forwarding adapter working group. This means that it will be responsible for the actual data forwarding work, pushing data from the local data engine to the remote large-capacity data storage. Those data forwarding adapter instances that have not been elected as the master are included in the slave forwarding adapter working group. They will not participate in data forwarding under normal circumstances, but serve as backups to ensure that they can quickly take over the task when the master forwarding adapter fails.

[0076] It should be noted that the main forwarding adapter working group itself can be responsible for data forwarding, or any data forwarding adapter with a main forwarding identity in the main forwarding adapter working group can be responsible for data forwarding. Both implementations will fall within the protection scope of this application.

[0077] At the same time, the size of the master-slave forwarding adapter working group will be dynamically adjusted according to the current business needs and load conditions. The master forwarding adapter working group will be expanded to the maximum number of working copies M to ensure sufficient processing power to handle the peak data volume. On the contrary, the slave forwarding adapter working group will be reduced to the minimum number of hot standby copies N to reduce resource waste during business troughs. This technical approach allows the system to flexibly adjust resource allocation under different business loads and optimize resource use. During business peaks, processing power is increased by expanding the master forwarding adapter working group; during business troughs, unnecessary resource consumption is reduced by reducing the slave forwarding adapter working group.

[0078] In this embodiment, through this master-slave election and dynamic expansion and contraction mechanism, the system can ensure sufficient data forwarding capacity at all times, and can quickly switch to the slave forwarding adapter when the master forwarding adapter fails, ensuring the continuity of data forwarding and high availability of the system.

[0079] In an exemplary embodiment, according to the horizontal Pod autoscaler, the CPU utilization and memory utilization of each of the master forwarding adapter workgroup and the slave forwarding adapter workgroup are monitored as monitoring results;

[0080] According to the monitoring results, the number of nodes of the master forwarding adapter working group and the slave forwarding adapter working group is increased during business peak hours, and the number of nodes of the master forwarding adapter working group and the slave forwarding adapter working group is reduced during business trough hours.

[0081] Specifically, the Horizontal Pod Autoscaler (HPA) is a function in the Kubernetes cluster that automatically adjusts the number of Pods based on CPU utilization, memory utilization, or other selected metrics, and continuously monitors the CPU utilization and memory utilization of the master forwarding adapter workgroup and the slave forwarding adapter workgroup. These indicators are key parameters for measuring system load and performance, and can help the system understand current resource usage and demand.

[0082] Based on the monitoring results of HPA, the system can automatically increase the number of nodes in the master forwarding adapter workgroup and the slave forwarding adapter workgroup during business peak hours (i.e., when CPU and memory utilization are high). This ensures that there are enough resources to process data during high loads, avoiding forwarding delays and system overload. Conversely, during business trough hours (i.e., when CPU and memory utilization are low), the system can automatically reduce the number of nodes in these workgroups. This helps save resources and reduce costs while maintaining the responsiveness of the system.

[0083] This approach of dynamically adjusting resources based on actual load reflects the system's elastic expansion and contraction capabilities. It allows the system to flexibly adjust resource allocation under different business needs and operate in the most optimized way. By reducing the number of nodes when business is low, the system can reduce unnecessary resource waste and thus reduce operating costs. At the same time, this dynamic adjustment also helps to improve resource utilization efficiency.

[0084] In this embodiment, HPA is used to monitor and automatically adjust the size of the data forwarding adapter workgroup to adapt to different business loads, optimize resource usage and control costs, and maintain high performance and high availability of the system. By increasing the number of nodes during business peaks, the system can ensure that high performance and high availability can be maintained even under high load conditions to meet business needs.

[0085] In an exemplary embodiment, the sum of the maximum number of working copies M obtained by expanding the capacity of the primary forwarding adapter workgroup and the minimum number of hot standby copies N obtained by shrinking the capacity of the secondary forwarding adapter workgroup is maintained as an odd number.

[0086] Specifically, the purpose of keeping M+N as an odd number is to ensure that at any time, there is a clear master forwarding adapter working group and a slave forwarding adapter working group. In this way, when performing master-slave elections, a master node can always be elected to avoid the situation where two or more nodes simultaneously consider themselves to be the master node, ensuring the consistency and stability of the system.

[0087] In a distributed system, if the number of master nodes is even, the so-called "brain split" phenomenon may occur, that is, the nodes in the system are split into two independent parts, each with its own master node, resulting in inconsistent data. By keeping M+N an odd number, the risk of this split can be reduced.

[0088] When M+N is an odd number, the election logic can be simplified because there is always a clear master node. This simplifies the election process and improves the responsiveness and reliability of the system. By ensuring that there is a clear master node, the system can quickly failover when the master node fails, reducing the unavailability time of the system and improving overall availability. At the same time, this technical approach also allows the system to optimize resources while ensuring high availability. During business downturns, resources can be saved by reducing the number of copies M of the primary forwarding adapter workgroup and increasing the number of copies N of the slave forwarding adapter workgroup.

[0089] In this embodiment, according to business needs and load conditions, the system can dynamically adjust the values ​​of M and N to keep M + N an odd number. In this way, the system can flexibly adjust resource allocation in different business scenarios, while ensuring that there is a clear master node to handle data forwarding tasks at any time.

[0090] In an exemplary embodiment, when there is no data forwarding in the primary forwarding adapter working group, the corresponding primary forwarding identity is cancelled;

[0091] When there is no master forwarding identity in the data forwarding adapters of the entire forwarding adapter virtual cluster, a master-slave election operation is initiated using any data forwarding adapter in the slave forwarding adapter workgroup. When any data forwarding adapter obtains the master forwarding identity through election, the corresponding slave forwarding adapter workgroup is converted to the master forwarding adapter workgroup.

[0092] Specifically, the main responsibility of the main forwarding adapter working group is to receive the data pushed by the data collection engine and forward it to the remote large-capacity data storage. If the main forwarding adapter working group cannot receive the data for some reason (such as a data collection engine failure), then they do not actually perform the task of data forwarding.

[0093] When there is no data forwarding in the primary forwarding adapter workgroup, the system will cancel the "primary forwarding identity" of these adapters. This is to ensure that resources are not occupied ineffectively and to trigger a re-election so that the data forwarding task can be transferred to the adapter that can work normally.

[0094] When no data forwarding adapter in the entire forwarding adapter virtual cluster has the master forwarding identity, it means that the system currently has no active data forwarding path. In this case, the system needs to re-elect a master forwarding adapter workgroup to restore the data forwarding function.

[0095] The system will use any data forwarding adapter in the slave forwarding adapter working group to initiate the master-slave election operation. Any data forwarding adapter in the slave forwarding adapter working group does not participate in data forwarding under normal circumstances, but they are designed to be hot backups ready to take over the master forwarding task at any time.

[0096] During the re-election process, if any data forwarding adapter in a slave forwarding adapter working group successfully obtains the master forwarding identity, then this data forwarding adapter and its working group will be converted to a master forwarding adapter working group and begin to assume the task of data forwarding.

[0097] In this embodiment, the elasticity and high availability of the system are ensured in this way. Even if the primary forwarding adapter working group fails or cannot receive data, the system can quickly select a new primary forwarding adapter from the forwarding adapter working group to restore the data forwarding service with minimal delay. Seamless switching can be achieved to ensure the continuity of data forwarding, and even in the event of a failure or other abnormal situation, the service can be guaranteed to be uninterrupted, ensuring the continuity of data forwarding service and the stability of the system.

[0098] In an exemplary embodiment, before the received data to be forwarded is pushed to the remote data storage by the main forwarding adapter working group, the method further includes:

[0099] The data to be forwarded is formatted to adapt to the data storage format of the remote data storage.

[0100] Specifically, different remote data storages may support different data formats and protocols. In order to ensure that the data collected from the data collection engine can be correctly processed by the remote storage system, the data must be converted into a specific format supported by the remote system. The data collected by the data collection engine (such as Prometheus) may exist in a specific format, while the remote data storage (such as the time series database TSDB) may require another format. Therefore, the data needs to be format converted before being pushed to the remote storage.

[0101] The data forwarding adapter is not only responsible for forwarding data from the data acquisition engine to the remote storage, but also responsible for performing necessary format conversion on the data during the forwarding process. This conversion ensures the compatibility and consistency of data during transmission. Appropriate format conversion can improve the efficiency of data storage. For example, some data storage systems may optimize data in a specific format, making storage and query operations more efficient.

[0102] Format conversion includes data preprocessing steps, such as data integrity, data privacy, data compression, encryption, adding metadata, etc., to meet the requirements of remote storage systems.

[0103] In this embodiment, by converting the format of the data to be forwarded, it is ensured that the data can be correctly received and stored by the remote data storage device, which reflects the bridge role of the data forwarding adapter in the data acquisition and storage system and ensures the smooth flow and compatibility of data.

[0104] The most detailed embodiment of this application is:

[0105] like Figure 4 As shown, the specific operating mechanism of the above preferred embodiment is as follows:

[0106] 1. Deploy two local data engines, Prometheus, in the container cluster to collect data concurrently.

[0107] 2. After the blue and green forwarding adapter instances in the container cluster are started, they join the forwarding adapter virtual cluster. If there is no master node in the cluster, a master-slave election is performed. If any instance wins the master-slave election, it will be given the master forwarding identity.

[0108] 3. The main forwarding adapter working group will automatically expand to the maximum number of working copies M to meet the computing power requirements of the main forwarding adapter working group. The slave forwarding adapter working group will automatically shrink to the minimum number of hot standby copies N to reduce the waste of forwarding adapter cluster resources.

[0109] 4. After receiving the data submitted by the data collection engine, the main forwarding adapter working group converts the data format to match the remote data storage interface, and then pushes the data to be pushed to the remote centralized large-capacity data storage; the slave forwarding adapter working group directly discards the received data to be pushed.

[0110] 5. Receive data submitted by the main forwarding adapter working group and further store it in a remote centralized large-capacity data storage, such as OSS object storage or TSDBB storage with larger storage capacity such as VictoriaMetrics.

[0111] 6. If the data collection engine corresponding to the main forwarding adapter working group works abnormally, such as instance restart or process zombie, all forwarding adapters will have no data to forward, and the main forwarding adapter working group will gradually give up the main forwarding identity until there is no available master node in the entire cluster.

[0112] 7. At this point, the slave forwarding adapter working group will initiate a master-slave election, and the process returns to step 2 above to complete the dynamic switching of the master and slave forwarding identities of the blue-green forwarding adapter.

[0113] This process is repeated over and over again, and the above operating mechanism ensures high availability of data collection and push of the two data collection engines.

[0114] Furthermore, the data structure involved in the mixed master election of the blue and green forwarding adapters is as follows:

[0115]

[0116] Furthermore, the expansion and reduction of the blue and green forwarding adapters are uniformly controlled by the forwarding adapter cluster controller. To ensure the successful election of the master in the forwarding adapter cluster, the sum of the maximum number of working replicas M and the minimum number of hot standby replicas N must be maintained as an odd number.

[0117] Furthermore, the forwarding adapter cluster controller can provide HPA capabilities based on the observed CPU utilization and memory utilization of the blue and green forwarding adapters. In other words, based on the indicator thresholds set by the HPA, the number of nodes of the master forwarding adapter workgroup and the slave forwarding adapter workgroup of the forwarding adapter are expanded during the peak business period to meet the forwarding computing power requirements, and the number of nodes of the master forwarding adapter workgroup and the slave forwarding adapter workgroup are reduced during the low business period to reduce resource waste.

[0118] Furthermore, assuming that the availability of a single data collection engine is 99.9%, the concurrent collection engine availability of two data collection engines (active-active) is: 1-(1-99.9%) 2= 99.9999%, improved availability.

[0119] Further, if Figure 5 As shown, if the number of data acquisition engines is expanded horizontally, the availability of the data acquisition system can be further improved in theory. The extended embodiments based on the technical solution of the present invention are as follows:

[0120] In an extended embodiment, the original forwarding adapter cluster controller is upgraded to a data acquisition forwarding controller, which simultaneously performs cluster management of data acquisition engines and cluster management of forwarding adapters.

[0121] The data collection engine is in cluster form and can be dynamically expanded to 2, 3, 5 or even more according to business needs.

[0122] The cluster availability calculation expression based on this preferred embodiment is as follows:

[0123] Multi-active data engine availability = 1-(1-single data engine availability) n ; Where n is the number of data collection engines in the cluster.

[0124] For example, assuming that the availability of a single data collection engine is 99.9%, the concurrent collection engine availability of three data collection engines (multi-active) is: 1-(1-99.9%) 3 =99.99999999%.

[0125] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0126] Based on the same inventive concept, the embodiment of the present application also provides a data acquisition device for implementing the data acquisition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data acquisition device embodiments provided below can refer to the limitations on the data acquisition method above, and will not be repeated here.

[0127] In an exemplary embodiment, Figure 6 As shown, a data acquisition device is provided, comprising:

[0128] The data collection module 602 is used to collect data in parallel using at least two data collection engines, store the collected data locally, and forward the data to the data forwarding adapter corresponding to each data collection engine;

[0129] A master-slave election module 604 is used to perform a master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group;

[0130] The data forwarding control module 606 is used to push the received data to be forwarded to the remote data storage through the master forwarding adapter working group, and to discard the received data to be forwarded through the slave forwarding adapter working group;

[0131] The master-slave election module 604 is also used to trigger another master-slave election activity for at least two data forwarding adapters when a data collection engine corresponding to the master forwarding adapter working group fails, so that each data forwarding adapter in the master forwarding adapter working group will give up the master forwarding identity to obtain a new master forwarding adapter working group and a slave forwarding adapter working group.

[0132] In an exemplary embodiment, the master-slave election module 604 is specifically used to perform mixed master-slave election on at least two data forwarding adapters; after any data forwarding adapter elects a master forwarding identity, a master forwarding adapter working group is formed, and the master forwarding identity is distributed to all data forwarding adapters in the master forwarding adapter working group; the master forwarding adapter working group is expanded and the slave forwarding adapter working group is reduced.

[0133] In an exemplary embodiment, the data forwarding control module 606 is also used to monitor the CPU utilization and memory utilization of the master forwarding adapter workgroup and the slave forwarding adapter workgroup respectively according to the horizontal Pod auto-scaler as monitoring results; according to the monitoring results, the number of nodes of the master forwarding adapter workgroup and the slave forwarding adapter workgroup are increased during business peak periods, and the number of nodes of the master forwarding adapter workgroup and the slave forwarding adapter workgroup are reduced during business trough periods.

[0134] In an exemplary embodiment, the sum of the maximum number of working copies M obtained by expanding the capacity of the primary forwarding adapter workgroup and the minimum number of hot standby copies N obtained by shrinking the capacity of the secondary forwarding adapter workgroup is maintained as an odd number.

[0135] In an exemplary embodiment, the data forwarding control module 606 is used to cancel the corresponding master forwarding identity when there is no data forwarding in the master forwarding adapter workgroup; when there is no master forwarding identity in the data forwarding adapters of the entire forwarding adapter virtual cluster, use any data forwarding adapter in the slave forwarding adapter workgroup to initiate a master-slave election operation, and when any data forwarding adapter obtains the master forwarding identity through election, the corresponding slave forwarding adapter workgroup is converted into the master forwarding adapter workgroup.

[0136] In an exemplary embodiment, the data forwarding control module 606 is used to convert the format of the data to be forwarded to adapt to the data storage format of the remote data storage.

[0137] Each module in the above data acquisition device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0138] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data acquisition method is implemented.

[0139] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0140] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0141] Using at least two data collection engines to collect data in parallel, storing the collected data locally, and forwarding the data to a data forwarding adapter corresponding to each data collection engine;

[0142] Performing master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group;

[0143] Pushing the received data to be forwarded to the remote data storage through the master forwarding adapter working group, and discarding the received data to be forwarded through the slave forwarding adapter working group;

[0144] When the data collection engine corresponding to the master forwarding adapter working group fails, each data forwarding adapter in the master forwarding adapter working group will give up the master forwarding identity to trigger another master-slave election for at least two data forwarding adapters to obtain a new master forwarding adapter working group and slave forwarding adapter working group.

[0145] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0146] Perform hybrid master-slave election on at least two data forwarding adapters;

[0147] After any data forwarding adapter is elected to obtain the master forwarding identity, a master forwarding adapter working group is formed, and the master forwarding identity is distributed to all data forwarding adapters in the master forwarding adapter working group;

[0148] The capacity of the master forwarding adapter working group is expanded, and the capacity of the slave forwarding adapter working group is reduced.

[0149] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0150] According to the horizontal Pod autoscaler, the CPU utilization and memory utilization of the master forwarding adapter workgroup and the slave forwarding adapter workgroup are monitored as monitoring results;

[0151] According to the monitoring results, the number of nodes of the master forwarding adapter working group and the slave forwarding adapter working group is increased during business peak hours, and the number of nodes of the master forwarding adapter working group and the slave forwarding adapter working group is reduced during business trough hours.

[0152] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0153] The sum of the maximum number of working copies M obtained by expanding the capacity of the primary forwarding adapter workgroup and the minimum number of hot standby copies N obtained by shrinking the capacity of the secondary forwarding adapter workgroup is maintained to be an odd number.

[0154] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0155] In the case where there is no data forwarding in the primary forwarding adapter working group, the corresponding primary forwarding identity is cancelled;

[0156] When there is no master forwarding identity in the data forwarding adapters of the entire forwarding adapter virtual cluster, a master-slave election operation is initiated using any data forwarding adapter in the slave forwarding adapter workgroup. When any data forwarding adapter obtains the master forwarding identity through election, the corresponding slave forwarding adapter workgroup is converted to the master forwarding adapter workgroup.

[0157] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0158] The data to be forwarded is formatted to adapt to the data storage format of the remote data storage.

[0159] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0160] Using at least two data collection engines to collect data in parallel, storing the collected data locally, and forwarding the data to a data forwarding adapter corresponding to each data collection engine;

[0161] Performing master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group;

[0162] Pushing the received data to be forwarded to the remote data storage through the master forwarding adapter working group, and discarding the received data to be forwarded through the slave forwarding adapter working group;

[0163] When the data collection engine corresponding to the master forwarding adapter working group fails, each data forwarding adapter in the master forwarding adapter working group will give up the master forwarding identity to trigger another master-slave election for at least two data forwarding adapters to obtain a new master forwarding adapter working group and slave forwarding adapter working group.

[0164] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0165] Perform hybrid master-slave election on at least two data forwarding adapters;

[0166] After any data forwarding adapter is elected to obtain the master forwarding identity, a master forwarding adapter working group is formed, and the master forwarding identity is distributed to all data forwarding adapters in the master forwarding adapter working group;

[0167] The capacity of the master forwarding adapter working group is expanded, and the capacity of the slave forwarding adapter working group is reduced.

[0168] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0169] According to the horizontal Pod autoscaler, the CPU utilization and memory utilization of the master forwarding adapter workgroup and the slave forwarding adapter workgroup are monitored as monitoring results;

[0170] According to the monitoring results, the number of nodes of the master forwarding adapter working group and the slave forwarding adapter working group is increased during business peak hours, and the number of nodes of the master forwarding adapter working group and the slave forwarding adapter working group is reduced during business trough hours.

[0171] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0172] The sum of the maximum number of working copies M obtained by expanding the capacity of the primary forwarding adapter workgroup and the minimum number of hot standby copies N obtained by shrinking the capacity of the secondary forwarding adapter workgroup is maintained to be an odd number.

[0173] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0174] In the case where there is no data forwarding in the primary forwarding adapter working group, the corresponding primary forwarding identity is cancelled;

[0175] When there is no master forwarding identity in the data forwarding adapters of the entire forwarding adapter virtual cluster, a master-slave election operation is initiated using any data forwarding adapter in the slave forwarding adapter workgroup. When any data forwarding adapter obtains the master forwarding identity through election, the corresponding slave forwarding adapter workgroup is converted to the master forwarding adapter workgroup.

[0176] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0177] The data to be forwarded is formatted to adapt to the data storage format of the remote data storage.

[0178] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0179] Using at least two data collection engines to collect data in parallel, storing the collected data locally, and forwarding the data to a data forwarding adapter corresponding to each data collection engine;

[0180] Performing master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group;

[0181] Pushing the received data to be forwarded to the remote data storage through the master forwarding adapter working group, and discarding the received data to be forwarded through the slave forwarding adapter working group;

[0182] When the data collection engine corresponding to the master forwarding adapter working group fails, each data forwarding adapter in the master forwarding adapter working group will give up the master forwarding identity to trigger another master-slave election for at least two data forwarding adapters to obtain a new master forwarding adapter working group and slave forwarding adapter working group.

[0183] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0184] Perform hybrid master-slave election on at least two data forwarding adapters;

[0185] After any data forwarding adapter is elected to obtain the master forwarding identity, a master forwarding adapter working group is formed, and the master forwarding identity is distributed to all data forwarding adapters in the master forwarding adapter working group;

[0186] The capacity of the master forwarding adapter working group is expanded, and the capacity of the slave forwarding adapter working group is reduced.

[0187] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0188] According to the horizontal Pod autoscaler, the CPU utilization and memory utilization of the master forwarding adapter workgroup and the slave forwarding adapter workgroup are monitored as monitoring results;

[0189] According to the monitoring results, the number of nodes of the master forwarding adapter working group and the slave forwarding adapter working group is increased during business peak hours, and the number of nodes of the master forwarding adapter working group and the slave forwarding adapter working group is reduced during business trough hours.

[0190] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0191] The sum of the maximum number of working copies M obtained by expanding the capacity of the primary forwarding adapter workgroup and the minimum number of hot standby copies N obtained by shrinking the capacity of the secondary forwarding adapter workgroup is maintained to be an odd number.

[0192] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0193] In the case where there is no data forwarding in the primary forwarding adapter working group, the corresponding primary forwarding identity is cancelled;

[0194] When there is no master forwarding identity in the data forwarding adapters of the entire forwarding adapter virtual cluster, a master-slave election operation is initiated using any data forwarding adapter in the slave forwarding adapter workgroup. When any data forwarding adapter obtains the master forwarding identity through election, the corresponding slave forwarding adapter workgroup is converted to the master forwarding adapter workgroup.

[0195] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0196] The data to be forwarded is formatted to adapt to the data storage format of the remote data storage.

[0197] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0198] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0199] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0200] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data collection method, characterized in that: The method comprises: Using at least two data collection engines to collect data in parallel, storing the collected data locally, and forwarding the data to a data forwarding adapter corresponding to each data collection engine; Performing master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group; Pushing the received data to be forwarded to the remote data storage through the master forwarding adapter working group, and discarding the received data to be forwarded through the slave forwarding adapter working group; When the data collection engine corresponding to the master forwarding adapter working group fails, each data forwarding adapter in the master forwarding adapter working group will give up the master forwarding identity to trigger another master-slave election for at least two data forwarding adapters to obtain a new master forwarding adapter working group and slave forwarding adapter working group.

2. The method according to claim 1, characterized in that The step of performing master-slave election on at least two data forwarding adapters to determine a master forwarding adapter working group and a slave forwarding adapter working group includes: Perform hybrid master-slave election on at least two data forwarding adapters; After any data forwarding adapter is elected to obtain the master forwarding identity, a master forwarding adapter working group is formed, and the master forwarding identity is distributed to all data forwarding adapters in the master forwarding adapter working group; The capacity of the master forwarding adapter working group is expanded, and the capacity of the slave forwarding adapter working group is reduced.

3. The method according to claim 1, characterized in that The method further comprises: According to the horizontal Pod autoscaler, the CPU utilization and memory utilization of the master forwarding adapter workgroup and the slave forwarding adapter workgroup are monitored as monitoring results; According to the monitoring results, the number of nodes of the master forwarding adapter working group and the slave forwarding adapter working group is increased during the peak business period, and the number of nodes of the master forwarding adapter working group and the slave forwarding adapter working group is reduced during the low business period.

4. The method according to claim 2, characterized in that: The sum of the maximum number of working copies M obtained by expanding the capacity of the primary forwarding adapter workgroup and the minimum number of hot standby copies N obtained by shrinking the capacity of the secondary forwarding adapter workgroup is maintained to be an odd number.

5. The method according to claim 1, characterized in that The method further comprises: In the case where there is no data forwarding in the primary forwarding adapter working group, the corresponding primary forwarding identity is cancelled; When there is no master forwarding identity in the data forwarding adapters of the entire forwarding adapter virtual cluster, a master-slave election operation is initiated using any data forwarding adapter in the slave forwarding adapter workgroup. When any data forwarding adapter obtains the master forwarding identity through election, the corresponding slave forwarding adapter workgroup is converted to the master forwarding adapter workgroup.

6. The method according to claim 1, characterized in that Before the received data to be forwarded is pushed to the remote data storage device through the main forwarding adapter working group, the method further includes: The data to be forwarded is formatted to adapt to the data storage format of the remote data storage.

7. A data acquisition device, characterized in that: The device comprises: A data acquisition module, used to collect data in parallel using at least two data acquisition engines, store the collected data locally, and forward the data to a data forwarding adapter corresponding to each data acquisition engine; A master-slave election module is used to perform master-slave election on at least two data forwarding adapters to obtain a master forwarding adapter working group and a slave forwarding adapter working group; A data forwarding control module, used to push the received data to be forwarded to a remote data storage device through a master forwarding adapter working group, and to discard the received data to be forwarded through a slave forwarding adapter working group; The master-slave election module is also used to trigger another master-slave election activity for at least two data forwarding adapters when a data acquisition engine corresponding to the master forwarding adapter working group fails. This will cause each data forwarding adapter in the master forwarding adapter working group to give up its master forwarding identity, thereby obtaining a new master forwarding adapter working group and a slave forwarding adapter working group.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images