Distributed data acquisition method, device and system and scheduling platform

Abstract

The invention provides a distributed data acquisition method, device and system and a scheduling platform, and belongs to the technical field of data acquisition. The method is suitable for a scheduling platform, and the scheduling platform adopts a cluster architecture. The method comprises the following steps: acquiring monitoring data of each acquisition node; the monitoring data comprises a running state, a resource occupation condition, a channel number and a measuring point number of each channel; based on the monitoring data of each acquisition node, a preset scheduling algorithm is adopted to adjust the binding relationship between each acquisition node and the channel so as to realize load balancing of each acquisition node; according to the acquisition period and the data delay allowable time of each channel, determining the fragment number of each channel, and issuing the fragment number of each channel to a bound acquisition node; the number of the fragments of each channel is used for indicating the bound acquisition node to generate threads with the same number as the fragments, and the acquisition tasks of the corresponding channels are executed in parallel. According to the invention, high-frequency and low-delay data acquisition requirements can be met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data collection, and in particular to a distributed data collection method, device, system and scheduling platform. BACKGROUND

[0002] With the rapid development of the Internet of Things and the industrial Internet, in the data center, the data collection system needs to cope with massive, high-frequency measurement point data.

[0003] In the related art, the data collection architecture mostly adopts a centralized architecture, and its performance depends on the resources of a single node. However, the resources of a single node are limited, and as the number of collected measurement points continues to grow, the hardware needs to be upgraded frequently to meet the demand, but there is a physical upper limit to hardware upgrades, which cannot meet the high-frequency and low-latency data collection requirements. SUMMARY

[0004] The embodiments of the present application provide a distributed data collection method, device, system and scheduling platform to solve the problem that the prior art cannot meet the high-frequency and low-latency data collection requirements.

[0005] In a first aspect, the embodiments of the present application provide a distributed data collection method applicable to a scheduling platform, and the scheduling platform adopts a cluster architecture. The method comprises: obtaining monitoring data of each collection node; the monitoring data includes running status, resource occupation, channel number and measurement point number of each channel; based on the monitoring data of each collection node, using a preset scheduling algorithm to adjust the binding relationship between each collection node and the channel, so as to realize load balancing of each collection node; determining the number of shards of each channel according to the collection period and the data delay allowed time of each channel, and sending the number of shards of each channel to the bound collection node; the number of shards of each channel is used to instruct the bound collection node to generate the same number of threads as the number of shards, and execute the collection task of the corresponding channel in parallel.

[0006] In a possible implementation, the number of shards of each channel is determined according to the collection period and the data delay allowed time of each channel, comprising: for each channel, calculating the ratio of the collection period and the data delay allowed time of the channel, and rounding up the ratio to obtain a first number, and determining the number of shards of the channel according to the first number; the number of shards of the channel is greater than or equal to the first number; wherein the collection period of each channel is the time required for all measurement points in the channel to collect once in turn; the number of measurement points included in the collection task executed by each thread generated by the collection node bound to each channel is the ratio of the number of measurement points of the channel to the number of shards of the channel.

[0007] In a possible implementation, based on the monitoring data of each collection node, a preset scheduling algorithm is used to adjust the binding relationship between each collection node and the channel, including: Based on the number of channels of each collection node and the number of measuring points of each channel, the total number of measuring points of each collection node is determined. The total number of measuring points, the resource occupation and the running state of each collection node are weighted and summed to obtain the load score of each collection node; wherein the weight of the total number of measuring points is greater than the weight of the resource occupation, and the weight of the total number of measuring points is greater than the weight of the running state. The load of each collection node is normalized to obtain the load rate of each collection node. Based on the load rate of each collection node, the binding relationship between each collection node and the channel is adjusted.

[0008] In a possible implementation, the above method further includes: When receiving the registration request of a new collection node, the new collection node is added to the collection node list; The progressive task allocation mode is used to allocate channels for the new collection node until the new collection node and other collection nodes reach a load balancing state.

[0009] In a possible implementation, the above method further includes: If a heartbeat packet of a certain collection node is not received within a preset time length, the collection node is determined to be a faulty collection node; wherein each collection node sends a heartbeat packet to the scheduling platform at a regular time when it is not faulty; Based on the monitoring data of each non-faulty collection node, the binding relationship between each non-faulty collection node and the channel is adjusted to allocate the channels of the faulty node to the non-faulty node and ensure the load balancing of each non-faulty collection node.

[0010] In a possible implementation, the scheduling platform includes an extensible interface that supports integrating a preset scheduling algorithm in a plug-in manner.

[0011] In a possible implementation, the scheduling platform includes 2N+1 scheduling nodes; wherein N is the maximum number of scheduling nodes that the scheduling platform allows to fail at the same time, and N+1 is the minimum number of scheduling nodes required for normal operation of the scheduling platform; Each scheduling node synchronizes data through the Raft consensus protocol; Each collection node operates independently.

[0012] In a second aspect, an embodiment of the present application provides a distributed data collection device suitable for a scheduling platform, the scheduling platform adopts a cluster architecture, and the device includes: The acquisition module is configured to acquire monitoring data of each acquisition node, wherein the monitoring data comprises an operating state, resource occupation, a number of channels, and a number of measuring points of each channel. The balancing module is configured to adjust a binding relationship between each acquisition node and a channel based on the monitoring data of each acquisition node by using a preset scheduling algorithm, so as to realize load balancing of each acquisition node. The fragmentation module is configured to determine a number of fragments of each channel according to an acquisition period and a data delay allowable time of each channel, and to distribute the number of fragments of each channel to the bound acquisition node, wherein the number of fragments of each channel is used to instruct the bound acquisition node to generate the same number of threads as the number of fragments, and to execute the acquisition task of the corresponding channel in parallel.

[0013] In a third aspect, an embodiment of the present application provides a scheduling platform, comprising a processor and a memory, the memory is configured to store a computer program, and the processor is configured to call and run the computer program stored in the memory to execute the distributed data acquisition method as described in the first aspect or any possible implementation manner of the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a distributed data acquisition system, comprising the scheduling platform as described in the third aspect and a plurality of acquisition nodes, wherein the scheduling platform is in communication connection with each acquisition node, and the scheduling platform adopts a cluster architecture.

[0015] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the distributed data acquisition method as described in the first aspect or any possible implementation manner of the first aspect.

[0016] The embodiment of the present application provides a distributed data acquisition method, device, system and scheduling platform, the method adjusts a binding relationship between each acquisition node and a channel based on monitoring data of each acquisition node by using a preset scheduling algorithm, can realize load balancing of each acquisition node, avoids that a single acquisition node becomes a system bottleneck due to overloading, and simultaneously determines a number of fragments of each channel according to an acquisition period and a data delay allowable time of each channel, and distributes the number of fragments of each channel to the bound acquisition node, so that the bound acquisition node generates the same number of threads as the number of fragments, and executes the acquisition task of the corresponding channel in parallel, which can significantly reduce data acquisition delay, improve the acquisition efficiency and real-time performance of massive measurement data, and meet the data acquisition requirements of high frequency and low delay; the scheduling platform in the present application adopts a cluster architecture, which can eliminate the risk of being unable to work normally due to single point failure, and improve the high availability and high reliability of the scheduling platform. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0018] Figure 1 is a structural schematic diagram of a distributed data acquisition system provided by an embodiment of the present application; Figure 2 is an implementation flowchart of a distributed data acquisition method provided by an embodiment of the present application; Figure 3 is a schematic diagram of a distributed data acquisition device provided by an embodiment of the present application; Figure 4 is a schematic diagram of a scheduling platform provided by an embodiment of the present application. DETAILED DESCRIPTION

[0019] In the following description, specific details are set forth in order to provide a thorough understanding of embodiments of the present application. However, persons of ordinary skill in the art will readily recognize that embodiments of the present application can be practiced without these specific details. In other instances, well-known structures, devices, circuits, and methods have not been described in detail in order to avoid obscuring the present application.

[0020] In order to make the objects, technical solutions and advantages of the present application clearer, the following will be described with reference to the accompanying drawings through specific embodiments.

[0021] Referring to Figure 1 , a structural schematic diagram of a distributed data acquisition system provided by an embodiment of the present application is shown. As shown in Figure 1 , the distributed data acquisition system includes a scheduling platform 10 and a plurality of acquisition nodes 12, the scheduling platform 10 is in communication connection with each acquisition node 12; the scheduling platform 10 adopts a cluster architecture.

[0022] The cluster architecture refers to connecting a plurality of independent computers (referred to as nodes) through software and network to make them work cooperatively, and providing services as a unified whole to the outside. The core goal of the cluster architecture is to complete a task together, thereby obtaining higher availability, stronger performance, higher reliability and better scalability.

[0023] The scheduling platform 10 in the embodiments of the present application adopts a cluster architecture, which can improve the performance, availability, reliability and scalability of the scheduling platform 10, so that the scheduling platform 10 can work normally even if one node fails, and the performance of the scheduling platform 10 does not depend on the hardware performance of a single node.

[0024] The scheduling platform 10 is a core management platform, responsible for channel scheduling, node management, state monitoring and fault handling, etc.

[0025] The collection nodes 12 adopt a stateless design mode, which gives them better scalability and facilitates horizontal expansion to support massive data. The collection nodes 12 are responsible for actual data collection tasks, and register with and periodically report the state to the scheduling platform 10. The collection nodes 12 can also be referred to as collection service nodes.

[0026] Each collection node 12 can include multiple channels. The channel can be the smallest scheduling unit of the scheduling platform 10, and represents a data collection task, such as a collection stream of a specific data source. Each channel can include multiple measurement points. The measurement point can be understood as a data collection point or a data collection quantity. For example, each device has many physical quantities or state quantities that need to be monitored, collected or controlled, and each such quantity can be referred to as a measurement point. For example, for a photovoltaic inverter, current, voltage, power, temperature, working state, etc. can all be measurement points.

[0027] Different collection nodes 12 can include the same number of channels, or different numbers of channels; different channels can include the same number of measurement points, or different numbers of measurement points, which can all be set according to actual needs.

[0028] The embodiments of the present application realize easy maintenance, easy expansion, low delay and high reliability of massive data collection by building an independent scheduling platform 10, overall managing all collection nodes 12, and distributing collection data channels to appropriate collection nodes 12 according to a preset scheduling algorithm.

[0029] The scheduling platform 10 defines the channel as its smallest scheduling unit. The collection node 12 is responsible for maintaining the running data of each channel, and periodically sends a heartbeat packet and monitoring data to the scheduling platform 10, so that the scheduling platform 10 can obtain the running load of each node. For abnormal conditions, the scheduling platform 10 can trigger an event alarm, and notify relevant personnel through email or SMS, helping the operation and maintenance team to reduce workload and quickly locate problems.

[0030] The system maintenance personnel can view the distribution of each channel and the running state of the collection node 12 through the scheduling platform 10, manually adjust the scheduling algorithm according to their own management needs, and intervene in the balance mode of scheduling.

[0031] The scheduling platform 10 can automatically discover the newly added collection node 12, and according to the running state of the collection node 12 and the scheduling algorithm, part of the channels on other collection nodes 12 are transferred to the new collection node 12 to achieve the purpose of self-balancing. Through the automatic discovery mechanism, the elastic expansion of the collection node 12 can be realized.

[0032] This design not only improves the flexibility and response speed of the system, but also greatly enhances the stability of the system, especially in the face of large-scale data flow, which can effectively avoid the risk of system paralysis caused by single node overload. By introducing advanced load balancing algorithms and intelligent fault recovery mechanisms, the embodiments of the present application can significantly improve the overall performance of the data collection system and ensure the efficiency and reliability of data transmission.

[0033] In some possible implementations, the scheduling platform 10 can include a plurality of scheduling nodes 11. Specifically, the scheduling platform 10 can include 2N+1 scheduling nodes 11; wherein N is the maximum number of scheduling nodes 11 that the scheduling platform 10 allows to fail at the same time, N+1 is the minimum number of scheduling nodes 11 required for the normal operation of the scheduling platform 10; Each scheduling node 11 synchronizes data through the Raft consensus protocol; Each collection node 12 operates independently.

[0034] Among them, the scheduling node 11 can also be referred to as a scheduling service node.

[0035] In the embodiments of the present application, the scheduling platform 10 adopts a cluster architecture design to improve the overall reliability of the platform. The metadata is stored on each scheduling node 11 and synchronized through the Raft consensus protocol to ensure that the scheduling platform 10 can continue to operate even in the case of partial scheduling node 11 failure. The above-mentioned metadata can include cluster crew information, collection node information, task scheduling information, scheduling algorithm, and system configuration information, etc.

[0036] The number of scheduling nodes 11 is set to 2N+1, which means that up to N scheduling nodes 11 can be tolerated.

[0037] Exemplarily, when N=1, the scheduling platform 10 includes 3 scheduling nodes 11, and up to 1 scheduling node 11 is allowed to fail; when N=2, the scheduling platform 10 includes 5 scheduling nodes 11, and up to 2 scheduling nodes 11 are allowed to fail; when N=3, the scheduling platform 10 includes 7 scheduling nodes 11, and up to 3 scheduling nodes 11 are allowed to fail; when N=4, the scheduling platform 10 includes 9 scheduling nodes 11, and up to 4 scheduling nodes 11 are allowed to fail; and so on, which will not be repeated here.

[0038] Each acquisition node 12 can operate independently without being affected by other acquisition nodes 12, thus ensuring that even if some nodes fail, the entire system can still maintain normal operation.

[0039] It should be noted that, although Figure 1 Each scheduling node 11 is connected to each acquisition node 12, but in practical applications, the scheduling platform 10 can have a unified interface to connect to each acquisition node 12.

[0040] The following is based on Figure 1 , combined Figure 2 The distributed data acquisition method provided in the embodiments of this application will be described.

[0041] See Figure 2 This document illustrates a flowchart of the implementation of a distributed data acquisition method provided in an embodiment of the present invention. This method is applicable to a scheduling platform, which employs a cluster architecture. In other words, the executing entity of the distributed data acquisition method can be the scheduling platform. The scheduling platform can be a single terminal or multiple terminals; that is, the distributed data acquisition method provided in this embodiment can be executed by one terminal or multiple terminals, and can be configured according to actual needs. No specific limitations are imposed here.

[0042] See Figure 2 The method is described in detail below: In S201, monitoring data from each acquisition node is acquired; the monitoring data includes operating status, resource usage, number of channels, and number of measurement points for each channel.

[0043] In this embodiment of the application, each acquisition node can report its own monitoring data to the scheduling platform every first preset period. The scheduling platform can then receive the monitoring data reported by each acquisition node every first preset period, thereby determining the load status of each acquisition node and whether the channels of each acquisition node need to be adjusted.

[0044] The monitoring data of the acquisition node can include the node's operating status, resource usage, number of channels, and number of measurement points per channel. The operating status of the acquisition node can include the time taken to complete a single data acquisition task, i.e., the acquisition cycle of the acquisition node. The resource usage of the acquisition node can be understood as the hardware resource usage of the acquisition node, and can include at least one of the following: CPU usage, memory usage, network bandwidth usage, and disk usage. The number of channels and the number of measurement points per channel refer to the total number of channels contained in the acquisition node and the number of measurement points contained in each channel of the acquisition node.

[0045] The duration of the first preset period can be set according to actual needs, and is not specifically limited here.

[0046] In S202, based on the monitoring data of each collection node, a preset scheduling algorithm is used to adjust the binding relationship between each collection node and the channel, so as to realize the load balancing of each collection node.

[0047] In the embodiments of the present application, the load condition of each collection node can be determined based on the monitoring data of each collection node, so that it can be judged whether each collection node is load balanced. If the loads of each collection node are not balanced, a preset scheduling algorithm can be used to adjust the binding relationship between each collection node and the channel, so as to realize the load balancing of each collection node.

[0048] The above adjustment of the binding relationship between each collection node and the channel includes adjustment of the corresponding relationship between the collection node and the channel stored in the scheduling platform, and also includes adjustment of the actual binding relationship between the collection node and the channel. That is, it can be understood that after the corresponding relationship between the collection node and the channel stored in the scheduling platform is adjusted by using the preset scheduling algorithm, the position of the channel is adjusted based on the adjusted corresponding relationship between the collection node and the channel stored in the scheduling platform, so that the actual binding relationship between the collection node and the channel is the same as the adjusted corresponding relationship between the collection node and the channel.

[0049] The binding relationship between each collection node and the channel can be understood as which channels are included in each collection node, that is, the corresponding relationship or mapping relationship between each collection node and the channel. For example, collection node 1 includes channel 1 and channel 2, and collection node 2 includes channel 3 and channel 4. Then, collection node 1 and channel 1 and channel 2 have a binding relationship, and collection node 2 and channel 3 and channel 4 have a binding relationship.

[0050] The load balancing of each collection node can be understood as the absolute value of the difference between the load rates of each collection node being less than or equal to a preset load rate difference, or the difference between the maximum load rate and the minimum load rate in the load rates of each collection node being less than or equal to a preset load rate difference. The preset load rate difference is a small value, and its specific value can be set according to actual needs.

[0051] The embodiments of the present application can automatically balance the loads of each collection node without manual intervention, which can avoid channel tilting and reduce the work burden of operation and maintenance personnel.

[0052] In S203, the number of shards of each channel is determined according to the collection period and the data delay allowed time of each channel, and the number of shards of each channel is sent to the bound collection node; the number of shards of each channel is used to instruct the bound collection node to generate the same number of threads as the number of shards, and execute the collection task of the corresponding channel in parallel.

[0053] The collection period of a channel can be understood as the time length required for the channel to complete a data collection task, that is, the time length required for all the measuring points in the channel to collect data once. When a channel executes a data collection task using a single thread, the measuring points included in the channel need to collect data sequentially and cannot be executed in parallel, so when the number of measuring points is large, the collection period of the channel will be long, which will further result in a long data delay. That is, the collection period of the channel here is the time required for all the measuring points in the channel to collect data sequentially once.

[0054] The data delay allowance time can be understood as the maximum time allowed for data delay. For example, when a measuring point needs to be collected, the maximum time allowed from the time when the collection instruction of the measuring point is issued to the time when the collected data of the measuring point is received is the data delay allowance time.

[0055] When the collection period of a channel is greater than the data delay allowance time, the channel will inevitably have a data delay. To solve this problem, embodiments of the present application determine the number of fragments of each channel according to the collection period and the data delay allowance time of each channel, and issue the number of fragments of each channel to the bound collection node. After each collection node receives the number of fragments of the channel bound thereto, the measuring points of the channel can be evenly divided according to the number of fragments, and when executing the data collection task of the channel, the same number of threads as the number of fragments is created to execute the data collection task of the channel in parallel through multiple threads. Each thread executes the collection task of the measuring points after the division.

[0056] Exemplarily, assuming that the number of fragments of a channel is 3 and the number of measuring points of the channel is 30,000, the collection node corresponding to the channel can generate 3 threads, and divide the measuring points of the channel into 3 parts, each with 10,000 measuring points. Each thread executes the collection task of 10,000 measuring points, so that the collection period of the channel can be reduced to 1 / 3 of the original.

[0057] The method of the embodiments of the present application can take advantage of the multi-core CPU of the collection node, realize multi-measuring point concurrent collection, improve the real-time performance of measuring point data, and thus enable the operation and maintenance personnel to discover potential fault problems earlier and take measures to avoid risks in time. At the same time, through the data fragmentation and multi-thread concurrent mechanism, the data collection delay can be reduced, and the real-time performance and accuracy of data collection can be improved.

[0058] The embodiments of the present application can realize load balancing of each acquisition node, avoid a single acquisition node from becoming a system bottleneck due to overloading, and can significantly reduce data acquisition delay and improve the acquisition efficiency and real-time performance of massive measurement data to meet the high-frequency and low-delay data acquisition requirements by adjusting the binding relationship between each acquisition node and the channel based on the monitoring data of each acquisition node, using a preset scheduling algorithm, determining the number of shards of each channel according to the acquisition period and the data delay allowable time of each channel, and distributing the number of shards of each channel to the bound acquisition node to enable the bound acquisition node to generate the same number of threads as the number of shards and perform the acquisition task of the corresponding channel in parallel.

[0059] In some embodiments, the determination of the number of shards of each channel according to the acquisition period and the data delay allowable time of each channel includes: For each channel, the ratio of the acquisition period to the data delay allowable time of the channel is calculated, and the ratio is rounded up to obtain a first number, and the number of shards of the channel is determined according to the first number; the number of shards of the channel is greater than or equal to the first number. The acquisition period of each channel is the time required for all measurement points in the channel to be acquired once in turn; the number of measurement points included in the acquisition task performed by each thread generated by the acquisition node bound to the channel is the ratio of the number of measurement points of the channel to the number of shards of the channel.

[0060] In the embodiments of the present application, in order to avoid the acquisition period of each channel exceeding the data delay allowable time, for each channel, the ratio of the acquisition period to the data delay allowable time of the channel is calculated, and the ratio is rounded up to obtain a positive integer, i.e., a first number, so that the number of shards of the channel can be determined as a positive integer greater than or equal to the first number.

[0061] For example, assuming that the ratio of the acquisition period to the data delay allowable time of a certain channel is 2.6, then the first number obtained by rounding up is 3, and the number of shards of the channel is greater than or equal to 3. If the number of shards of the channel is 3 in order to save threads, and the number of shards of the channel is greater than 3 in order to improve the acquisition efficiency and shorten the acquisition time, the maximum value under the condition of allowing the resources of the corresponding acquisition node.

[0062] After determining the number of shards of the channel, the measurement points of the channel can be evenly divided according to the number of shards, and each thread performs a part of the measurement point acquisition task. For details, refer to the foregoing embodiments, which will not be described here.

[0063] The data delay allowable time of each channel can be the same or set to different times according to actual needs, which is not specifically limited here.

[0064] The embodiment of the application dynamically calculates the number of fragments of the channel required to ensure the timeliness by associating the collection period of the channel with the data delay allowable time, and guides the collection node to generate corresponding parallel threads, which can ensure that even when the data volume increases dramatically, the total collection time can be compressed within the allowable delay range by increasing parallel execution, can greatly reduce the end-to-end delay of data from the source end to the platform, and enables the operation and maintenance personnel to discover potential faults earlier, thereby providing a solid data foundation for real-time monitoring and rapid decision-making.

[0065] In some embodiments, the binding relationship between each collection node and the channel is adjusted based on the monitoring data of each collection node using a preset scheduling algorithm, including: The total number of measuring points of each collection node is determined based on the number of channels of each collection node and the number of measuring points of each channel. The load score of each collection node is obtained by weighted summation of the total number of measuring points, resource occupation and running state of each collection node; wherein the weight of the total number of measuring points is greater than the weight of the resource occupation, and the weight of the total number of measuring points is greater than the weight of the running state. The load rate of each collection node is obtained by normalizing the load of each collection node. The binding relationship between each collection node and the channel is adjusted based on the load rate of each collection node.

[0066] The total number of measuring points of each collection node is the sum of the number of measuring points of each channel included in the collection node. If the number of measuring points of each channel of the collection node is the same, the total number of measuring points of the collection node is the product of the number of channels of the collection node and the number of measuring points of the channel. If the number of measuring points of each channel of the collection node is not the same, the corresponding number of measuring points same as the number of channels is obtained according to the number of channels of the collection node, and the total number of measuring points is obtained by summing the number of measuring points.

[0067] The total number of measuring points, resource occupation and running state of the collection node have respective corresponding weights. However, the embodiment of the present application takes the total number of measuring points as the main basis, and takes the resource occupation and the running state as auxiliary conditions to determine the load condition of each collection node. Therefore, the weight of the total number of measuring points is greater than the weight of the resource occupation, and the weight of the total number of measuring points is greater than the weight of the running state, but the sum of the weights of the three is 1. Exemplarily, the weight of the total number of measuring points can be 0.4, and the weights of the resource occupation and the running state can be 0.3; or the weight of the total number of measuring points can be 0.5, the weight of the resource occupation can be 0.3, and the weight of the running state can be 0.2.

[0068] It should be noted that the information contained in the resource occupation and the running state can be more than one. When the information contained in the resource occupation is more than one, the sum of the weights of each information contained in the resource occupation is the weight of the resource occupation. The weight of each information contained in the resource occupation can be divided equally into the weight of the resource occupation, or can be set according to actual needs.

[0069] Exemplarily, it is assumed that the resource occupation includes CPU usage, memory usage, network bandwidth usage and disk usage, and the weight of the resource occupation is 0.3. Therefore, the weights of the CPU usage, the memory usage, the network bandwidth usage and the disk usage can be 0.075.

[0070] The running state is the same, and will not be repeated here.

[0071] The total number of measuring points, resource occupation and running state of each collection node are weighted and summed to obtain the load score of each collection node. Exemplarily, the load rate of the collection node = (the load score of the collection node - the minimum possible load score of the collection node) / (the maximum possible load score of the collection node - the minimum possible load score of the collection node). Wherein, the minimum possible load score of the collection node is the load score when the collection node is in idle state. At this time, each item of the resource occupation is 0, the total number of measuring points is 0, and the running state is 0. Therefore, the minimum possible load score of the collection node is 0. The maximum possible load score of the collection node is the load score when the collection node is in full load state. At this time, each item of the resource occupation is 100%, the total number of measuring points is the maximum number of measuring points that a single collection node can bear, and the running state is the running state corresponding to the maximum number of measuring points that a single collection node can bear. The maximum possible load score of the collection node is calculated based on these factors.

[0072] Based on the foregoing process, the load rates of the respective collection nodes can be calculated, and then it is determined whether the absolute values of the differences between the load rates of the respective collection nodes are less than or equal to a preset load rate difference, or whether the difference between the maximum load rate and the minimum load rate among the load rates of the respective collection nodes is less than or equal to the preset load rate difference. If the determination is satisfied, it indicates that the respective collection nodes are balanced in load, and there is no need to adjust the binding relationship between the respective collection nodes and the channels. If the determination is not satisfied, it indicates that the respective collection nodes are unbalanced in load, and the binding relationship between the respective collection nodes and the channels needs to be adjusted.

[0073] The above adjustment of the binding relationship between the respective collection nodes and the channels can include: pairing the collection node with the maximum load rate and the collection node with the minimum load rate; removing the paired collection nodes, and determining whether the difference between the maximum load rate and the minimum load rate among the load rates of the remaining collection nodes is less than or equal to the preset load rate difference. If the determination is satisfied, the pairing is stopped. If the determination is not satisfied, the collection node with the maximum load rate and the collection node with the minimum load rate among the remaining collection nodes are paired. The step of removing the paired collection nodes is repeatedly performed, and if the determination is not satisfied, the collection node with the maximum load rate and the collection node with the minimum load rate among the remaining collection nodes are paired until the difference between the maximum load rate and the minimum load rate among the load rates of the remaining collection nodes is less than or equal to the preset load rate difference. After the pairing is completed, for each pair of the two collection nodes, the channel of the collection node with the higher load rate is transferred to the collection node with the lower load rate, so that the two collection nodes are balanced in load.

[0074] The total number of measurement points is taken as the main evaluation basis of the load score in the embodiments of the present application, which can more truly reflect the actual business pressure of the collection nodes than the traditional method that only relies on the hardware resource condition. Because the total number of measurement points directly determines the data processing amount and the calculation amount, the load trend can be predicted in advance based on the total number of measurement points as the main basis, so as to avoid passive response when the hardware resource has appeared bottleneck, thereby realizing the change from passive remedy to active prevention, making the load balancing more accurate and forward-looking. The multi-dimensional load score is converted into a unified load rate through normalization processing, so that the nodes with different configurations and different tasks have comparability, providing a fair and unified decision basis for intelligent scheduling. The scheduling platform can accurately migrate the channel of the collection node with the high load rate to the collection node with the low load rate according to the unified load rate, so as to ensure that the business pressure of all collection nodes tends to be balanced, effectively avoid the overall system throughput bottleneck caused by unbalanced load of the collection nodes, and maximize the overall computing potential of the collection nodes, thereby supporting higher data collection throughput.

[0075] In some embodiments, the above method further includes: adding the new collection node to the collection node list when receiving the registration request of the new collection node; The channels are allocated to the new collection node in a progressive task allocation manner until the new collection node and other collection nodes reach a load balancing state.

[0076] In the embodiment of the present application, when a new collection node is started, it needs to be registered with the scheduling platform. The scheduling platform stores a list of registered collection nodes. When there is a registration request of a new collection node, the new collection node is added to the list of collection nodes, which is equivalent to adding the new collection node to the whole collection node.

[0077] When a new collection node is added, it does not have a measurement point and the load rate is 0. At this time, the channels of the collection node with a high load rate can be migrated to the new collection node. However, the new collection node may not be stable. If a large number of channels are migrated to the new collection node at this time, it may cause the interruption time of channel collection to be too long. Therefore, the embodiment of the present application allocates channels to the new collection node in a progressive task allocation manner, that is, in a slow heating mode, which helps the new collection node to smoothly transition to a normal working state and can reduce the risk caused by a sudden large load.

[0078] In some embodiments, the above method further comprises: If a heartbeat packet of a certain collection node is not received within a preset time length, the collection node is determined to be a faulty collection node; wherein each collection node sends a heartbeat packet to the scheduling platform at a regular time when it is not faulty; Based on the monitoring data of each non-faulty collection node, the binding relationship between each non-faulty collection node and the channel is adjusted to allocate the channels of the faulty node to the non-faulty node and ensure the load balancing of each non-faulty collection node.

[0079] In the embodiment of the present application, the scheduling platform automatically discovers the faulty collection node through a heartbeat detection mechanism and migrates the channels of the faulty collection node to the non-faulty collection node. Specifically, each collection node sends a heartbeat packet to the scheduling platform at a second preset period. If the scheduling platform does not receive a heartbeat packet of a certain collection node within a preset time length, it can be considered that the collection node has failed and is a faulty collection node. The preset time length is greater than the second preset period or greater than twice the second preset period.

[0080] The second preset period can be equal to or not equal to the first preset period, and the values of the two can be set according to actual needs, which are not limited here.

[0081] After discovering the faulty collection node, the channels of the faulty collection node need to be migrated to the non-faulty collection node, but the load balancing of each non-faulty collection node also needs to be ensured during the migration process.

[0082] In some possible implementations, if the faulty collection node recovers after a period of time, that is, the dispatch platform regularly receives the heartbeat packet of the faulty collection node again, it can be considered that the faulty collection node recovers, and the channel can be rescheduled to the collection node.

[0083] The embodiments of the present application can ensure that even when some collection nodes have problems, a quick response can be made to maintain the continuity of data collection, while greatly simplifying the operation of the operation and maintenance personnel, reducing human intervention, and reducing the operation and maintenance cost and the risk of human error.

[0084] The embodiments of the present application enable the system to have the elastic expansion capability, can smoothly increase or decrease the collection nodes according to the load condition, and does not need to stop for hardware upgrade. When the collection node fails, the dispatch platform can automatically and quickly migrate the affected channel to the healthy node to realize the fault self-healing of the business, thereby ensuring the continuity, stability and availability of the data collection service.

[0085] In some embodiments, the dispatch platform includes an extensible interface to support the integration of preset scheduling algorithms in a plug-in manner.

[0086] In the embodiments of the present application, the dispatch platform provides an extensible interface to support the integration of preset scheduling algorithms in a plug-in manner. That is, the preset scheduling algorithm can be a self-defined scheduling algorithm, a corresponding intelligent scheduling algorithm can be developed according to the actual needs, and the scheduling algorithm can be replaced through the interface provided by the dispatch platform, so that the user can flexibly adjust the scheduling logic according to the actual application scene to better adapt to different business needs. In addition, this mechanism can also ensure the best performance of the system in different environments.

[0087] By default, the scheduling and load balancing can be realized by the method in the foregoing embodiments.

[0088] In some embodiments, the dispatch platform includes 2N+1 scheduling nodes; wherein N is the maximum number of scheduling nodes that the dispatch platform allows to fail at the same time, and N+1 is the minimum number of scheduling nodes required for normal operation of the dispatch platform. Each scheduling node synchronizes data through the Raft consensus protocol. Each collection node is independently operated.

[0089] The embodiments of the present application can refer to the related description in the foregoing distributed data collection system embodiments, which will not be repeated here.

[0090] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0091] Figure 3 A structure diagram of a distributed data collection device provided by an embodiment of the present application is shown. For ease of illustration, only parts related to the embodiment of the present application are shown, and are described in detail as follows. The distributed data collection device is applicable to a scheduling platform, and the scheduling platform adopts a cluster architecture. As shown in the figure, the distributed data collection device 30 comprises an acquisition module 31, a balancing module 32 and a sharding module 33. Figure 3 The acquisition module 31 is configured to acquire monitoring data of each collection node; the monitoring data comprises a running state, a resource occupation condition, a channel quantity and a measuring point quantity of each channel. The balancing module 32 is configured to, based on the monitoring data of each collection node, adopt a preset scheduling algorithm to adjust a binding relationship between each collection node and a channel, so as to realize load balancing of each collection node.

[0092] The sharding module 33 is configured to determine a sharding quantity of each channel according to a collection period and a data delay allowable time of each channel, and to issue the sharding quantity of each channel to a bound collection node; the sharding quantity of each channel is configured to instruct the bound collection node to generate threads with the same quantity as the sharding quantity, and to execute a collection task of the corresponding channel in parallel. In a possible implementation, in the sharding module 33, the determination of the sharding quantity of each channel according to the collection period and the data delay allowable time of each channel comprises: For each channel, a ratio of the collection period of the channel to the data delay allowable time is calculated, and the ratio is rounded up to obtain a first quantity, and the sharding quantity of the channel is determined according to the first quantity; the sharding quantity of the channel is greater than or equal to the first quantity.

[0093] The collection period of each channel is the time required for all measuring points in the channel to be collected once in turn; the measuring point quantity included in the collection task executed by each thread generated by the bound collection node of each channel is the ratio of the measuring point quantity of the channel to the sharding quantity of the channel. In a possible implementation, in the balancing module 32, the adjustment of the binding relationship between each collection node and a channel based on the monitoring data of each collection node by adopting a preset scheduling algorithm comprises: The total measuring point quantity of each collection node is determined based on the channel quantity of each collection node and the measuring point quantity of each channel.

[0094] The total measuring point quantity, the resource occupation condition and the running state of each collection node are weighted and summed to obtain a load score of each collection node; the weight of the total measuring point quantity is greater than the weight of the resource occupation condition, and the weight of the total measuring point quantity is greater than the weight of the running state. ​The load of each acquisition node is normalized to obtain the load rate of each acquisition node. Based on the load rate of each acquisition node, adjust the binding relationship between each acquisition node and the channel.

[0095] In one possible implementation, the equalization module 32 can also be used for: When a registration request for a new data acquisition node is received, the new data acquisition node is added to the data acquisition node list; A gradual task allocation method is used to allocate channels to new acquisition nodes until the new acquisition nodes and other acquisition nodes reach a load balance state.

[0096] In one possible implementation, the equalization module 32 can also be used for: If no heartbeat packet is received from a certain data collection node within a preset time period, the data collection node is determined to be a faulty data collection node; when each data collection node is not faulty, it sends a heartbeat packet to the scheduling platform at regular intervals. Based on the monitoring data of each non-faulty acquisition node, the binding relationship between each non-faulty acquisition node and the channel is adjusted to allocate the channel of the faulty node to the non-faulty node and ensure the load balance of each non-faulty acquisition node.

[0097] In one possible implementation, the scheduling platform includes an extensible interface that supports the integration of preset scheduling algorithms in a plug-in manner.

[0098] In one possible implementation, the scheduling platform includes 2N+1 scheduling nodes; where N is the maximum number of scheduling nodes that the scheduling platform can tolerate simultaneous failures, and N+1 is the minimum number of scheduling nodes required for the scheduling platform to operate normally. Each scheduling node synchronizes data using the Raft consensus protocol; Each data acquisition node operates independently.

[0099] Figure 4 This is a schematic diagram of the scheduling platform provided in an embodiment of the present invention. Figure 4 As shown, the scheduling platform 4 in this embodiment includes a processor 40 and a memory 41. The memory 41 is used to store a computer program 42, and the processor 40 is used to call and run the computer program 42 stored in the memory 41 to execute the steps in the above-described distributed data acquisition method embodiments. Alternatively, the processor 40 is used to call and run the computer program 42 stored in the memory 41 to implement the functions of each module / unit in the above-described device embodiments.

[0100] For example, the computer program 42 can be divided into one or more modules / units, which are stored in the memory 41 and executed by the processor 40 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 42 in the scheduling platform 4.

[0101] The scheduling platform 4 can include, but is not limited to, the processor 40 and the memory 41. Those skilled in the art can understand that the scheduling platform 4 can include more or fewer components, or combine certain components, or include different components, such as the scheduling platform can also include an input / output device, a network access device, a bus, etc. Figure 4 The scheduling platform 4 shown in the figure is only an example and does not constitute a limitation on the scheduling platform 4, and can include more or fewer components than those shown in the figure, or combine certain components, or different components, for example, the scheduling platform can also include an input / output device, a network access device, a bus, etc.

[0102] The processor 40 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0103] The memory 41 can be an internal storage unit of the scheduling platform 4, such as a hard disk or a memory of the scheduling platform 4. The memory 41 can also be an external storage device of the scheduling platform 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 41 can include both the internal storage unit and the external storage device of the scheduling platform 4. The memory 41 is used to store the computer program and other programs and data required by the scheduling platform. The memory 41 can also be used to temporarily store data that has been output or will be output.

[0104] Corresponding to the above scheduling platform, the embodiment of the present application also provides a distributed data acquisition system, which includes the above scheduling platform and a plurality of acquisition nodes, and the scheduling platform is in communication connection with each acquisition node; the scheduling platform adopts a cluster architecture.

[0105] The description of the distributed data collection system can refer to the description in the foregoing embodiments, and will not be repeated.

[0106] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps of any one of the foregoing distributed data collection methods.

[0107] The embodiment of the present application further provides a computer program product, which comprises a computer program. The computer program is executed by a processor to implement any one of the foregoing distributed data collection methods.

[0108] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example for illustration. In actual applications, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0109] In the foregoing embodiments, the description of each embodiment has its own emphasis. The parts not described or recorded in detail in a certain embodiment can refer to the relevant description of other embodiments.

[0110] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0111] In the embodiments of the present application, it should be understood that the disclosed apparatus / scheduling platform and method can be implemented in other manners. For example, the described apparatus / scheduling platform embodiments are merely schematic. For example, the division of the modules or units is merely logical function division. There can be another division manner for the actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0112] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0113] In addition, each functional unit in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0114] The integrated module / unit, if implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, all or part of the flow of the above-mentioned embodiment methods can be implemented by a computer program instructing related hardware to complete, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of each distributed data acquisition method embodiment can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0115] The above-described embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A distributed data collection method, characterized by, The method is suitable for a scheduling platform adopting a cluster architecture, and comprises the following steps: Obtaining monitoring data of each collection node; the monitoring data comprises running state, resource occupation, channel number and measuring point number of each channel; Based on the monitoring data of each collection node, a preset scheduling algorithm is adopted to adjust the binding relationship between each collection node and channel, so as to realize load balancing of each collection node; According to the collection period and data delay allowable time of each channel, the number of fragments of each channel is determined, and the number of fragments of each channel is sent to the bound collection node; the number of fragments of each channel is used to instruct the bound collection node to generate the same number of threads as the number of fragments, and execute the collection task of the corresponding channel in parallel.

2. The distributed data collection method of claim 1, wherein, The number of fragments of each channel is determined according to the collection period and data delay allowable time of each channel, which comprises: For each channel, the ratio of the collection period to the data delay allowable time of the channel is calculated, and the ratio is rounded up to obtain a first number, and the number of fragments of the channel is determined according to the first number; the number of fragments of the channel is greater than or equal to the first number; Wherein, the collection period of each channel is the time required for all measuring points in the channel to collect once in turn; the number of measuring points included in the collection task executed by each thread generated by the collection node bound to each channel is the ratio of the number of measuring points of the channel to the number of fragments of the channel.

3. The distributed data collection method of claim 1, wherein, Based on the monitoring data of each collection node, a preset scheduling algorithm is adopted to adjust the binding relationship between each collection node and channel, which comprises: Based on the number of channels of each collection node and the number of measuring points of each channel, the total number of measuring points of each collection node is determined; The total number of measuring points, resource occupation and running state of each collection node are weighted and summed to obtain the load score of each collection node; wherein, the weight of the total number of measuring points is greater than the weight of the resource occupation, and the weight of the total number of measuring points is greater than the weight of the running state; The load of each collection node is normalized to obtain the load rate of each collection node; Based on the load rate of each collection node, the binding relationship between each collection node and channel is adjusted.

4. The distributed data collection method of claim 1, wherein, Further comprising: When receiving a registration request of a new collection node, the new collection node is added to the collection node list; An incremental task allocation mode is adopted to allocate channels to the new collection node until the new collection node and other collection nodes reach a load balancing state.

5. The distributed data collection method of claim 1, wherein, Further comprising: If a heartbeat packet of a certain collection node is not received within a preset time length, the collection node is determined to be a faulty collection node; wherein, each collection node sends a heartbeat packet to the scheduling platform at a regular time when it is not faulty; Based on the monitoring data of each non-faulty collection node, the binding relationship between each non-faulty collection node and channel is adjusted to allocate the channels of the faulty node to the non-faulty node, and to ensure the load balancing of each non-faulty collection node.

6. The distributed data collection method of any one of claims 1 to 5, wherein, The scheduling platform comprises an extensible interface, which supports integrating the preset scheduling algorithm in a plug-in manner.

7. The distributed data collection method according to any one of claims 1 to 5, wherein, The scheduling platform comprises 2N+1 scheduling nodes; wherein, N is the maximum number of scheduling nodes allowed to be simultaneously faulty of the scheduling platform, and N+1 is the minimum number of scheduling nodes required for normal operation of the scheduling platform; Each scheduling node synchronizes data through a Raft consensus protocol; Each acquisition node independently operates.

8. A distributed data acquisition device, characterized by The device is suitable for a scheduling platform adopting a cluster architecture, and the device comprises: An acquisition module, configured to acquire monitoring data of each acquisition node; the monitoring data comprises a running state, resource occupation, a number of channels, and a number of measuring points of each channel; A balancing module, configured to adjust a binding relationship between each acquisition node and a channel based on the monitoring data of each acquisition node by using a preset scheduling algorithm, so as to realize load balancing of each acquisition node; A sharding module, configured to determine a number of shards of each channel according to an acquisition period and a data delay allowable time of each channel, and to distribute the number of shards of each channel to a bound acquisition node; the number of shards of each channel is used to instruct the bound acquisition node to generate threads with the same number of shards as the number of shards, and to execute acquisition tasks of the corresponding channel in parallel.

9. A dispatch platform characterized by, The device comprises a memory and a processor, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the distributed data acquisition method in any one of claims 1 to 7.

10. A distributed data acquisition system, characterized by The device comprises the scheduling platform and a plurality of acquisition nodes in claim 9, the scheduling platform is in communication connection with each acquisition node; and the scheduling platform adopts a cluster architecture.

Images