Data processing method, data acquisition system, electronic device and storage medium

Abstract

The present application discloses a data processing method, a data processing system, an electronic device and a storage medium. The data processing method of the implementation mode of the present application includes: obtaining multiple target nodes and multiple crawling nodes; classifying multiple target nodes and multiple crawling nodes based on database type; sorting the load of multiple crawling nodes of the same category; and allocating target nodes of the same category to the crawling nodes according to the sorting results, so as to balance the load among multiple crawling nodes. The data processing method in the present application, for the multiple target nodes and crawling nodes obtained, according to the database type, classifies the target nodes that generate the index data of the same database into one category, classifies the crawling nodes that crawl the index data of the same database into one category, and sorts the load of multiple crawling nodes of the same category, and then assigns the target nodes of the same category to the crawling nodes according to the sorting results, so as to ensure the load balancing of the crawling nodes and improve the stability of the crawled data.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a data processing method, a data acquisition system, an electronic device and a storage medium. Background Art

[0002] Monitoring, as part of the underlying infrastructure, is an essential part of ensuring the normal operation of online business services. Through the monitoring system, service development and operation and maintenance personnel can timely and effectively discover service operation anomalies. From discovery to location to resolution of online problems, data collection is essential as one of the basic functions of monitoring. However, when there are monitoring nodes in the existing monitoring system that need to capture massive amounts of time series data, the task allocation of the capture nodes that perform data collection tasks will be unbalanced. Summary of the invention

[0003] Embodiments of the present application provide a data processing method, a data acquisition system, an electronic device, and a storage medium.

[0004] The data processing method of the embodiment of the present application is used in a data acquisition system, and the data processing method includes:

[0005] Get multiple target nodes and multiple grab nodes;

[0006] Based on the database type, the plurality of target nodes and the plurality of crawling nodes are classified, the target nodes can provide time series data corresponding to the database type, and the crawling nodes can crawl the time series data provided by the target nodes corresponding to the database type;

[0007] Performing load sorting on the plurality of said crawling nodes of the same category;

[0008] The target nodes of the same category are allocated to the grabbing nodes according to the sorting result, so as to balance the load among the plurality of the grabbing nodes.

[0009] The data processing method in the present application classifies the target nodes that generate indicator data of the same database into one category and the crawling nodes that crawl indicator data of the same database into one category according to the database type, and sorts the load of multiple crawling nodes of the same category, and then allocates target nodes of the same category to the crawling nodes according to the sorting results, so as to ensure the load balancing of the crawling nodes, thereby improving the durability and stability of the crawled data.

[0010] In an embodiment of the present application, a data acquisition system is provided, and the data acquisition system comprises:

[0011] Target node;

[0012] A capture node, used to capture the time series data provided by the target node;

[0013] The scheduling node is used to execute the data processing method in the above implementation mode.

[0014] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the data processing method described in any of the above embodiments.

[0015] An embodiment of the present application provides a non-volatile computer-readable storage medium of computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the processors execute the data processing method described in any of the above embodiments.

[0016] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0018] Figure 1 It is a flowchart of a data processing method in an embodiment of the present application;

[0019] Figure 2 It is a structural schematic diagram of a data acquisition system in an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of the structure of an electronic device in an embodiment of the present application;

[0021] Figure 4 It is a flowchart of a data processing method in an embodiment of the present application;

[0022] Figure 5 It is a flowchart of a data processing method in an embodiment of the present application;

[0023] Figure 6 It is a flowchart of a data processing method in an embodiment of the present application;

[0024] Figure 7 It is a schematic diagram of a scenario of calculating a first hash value and a second hash value in an implementation manner of the present application;

[0025] Figure 8 It is a flowchart of a data processing method in an embodiment of the present application;

[0026] Fig. 9is a schematic diagram of a scenario for calculating a third hash value and a fourth hash value in an implementation manner of the present application;

[0027] Fig.10 It is a flowchart of a data processing method in an embodiment of the present application;

[0028] Fig.11 It is a flowchart of a data processing method in an embodiment of the present application;

[0029] Fig.12 It is a schematic diagram of the deployment of the scheduling node in the data acquisition system in the implementation manner of the present application.

[0030] Description of main component symbols:

[0031] Data acquisition system 100 , target node 11 , capture node 12 , scheduling node 13 , registration center 14 , snapshot file 15 , electronic device 200 , memory 21 , processor 22 . DETAILED DESCRIPTION

[0032] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0033] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or reference letters in different examples, and this repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed. In addition, the various specific processes and material examples provided by the present application, but those of ordinary skill in the art can appreciate the application of other processes and / or the use of other materials.

[0034] See also Figure 1 The present application embodiment provides a data processing method, the data processing method comprising:

[0035] Step S10: Acquire multiple target nodes 11 and multiple captured nodes 12;

[0036] Step S20: Based on the database type, multiple target nodes 11 and multiple capture nodes 12 are classified, the target node 11 can provide time series data of the corresponding database type, and the capture node 12 can capture the time series data provided by the target node 11 of the corresponding database type;

[0037] Step S30: sorting the loads of multiple capture nodes 12 of the same category;

[0038] Step S40 : allocating target nodes 11 of the same category to the capture nodes 12 according to the sorting result, so as to balance the load among the multiple capture nodes 12 .

[0039] See also Figure 2 The embodiment of the present application provides a data collection system 100, which includes a target node 11, a capture node 12, a scheduling node 13 and a registration center 14. The capture node 12 can be used to capture the time series data provided by the target node 11, the scheduling node 13 can implement the data processing method in the embodiment of the present application, and the target node 11, the capture node 12 and the scheduling node 13 can exchange information with the registration center 14.

[0040] The data processing method in the embodiment of the present application can be implemented by the data acquisition system 100 in the embodiment of the present application. For example, step S10, step S20, step S30 and step S40 can be implemented by the scheduling node 13 (schduler) of the data acquisition system 100. In other words, the scheduling node 13 is used to obtain multiple target nodes 11 and multiple capture nodes 12, and to classify multiple target nodes 11 and multiple capture nodes 12 based on database type, the target node 11 can provide time series data of the corresponding database type, the capture node 12 can capture the time series data provided by the target node 11 of the corresponding database type, and to load sort multiple capture nodes 12 of the same category, and to allocate target nodes 11 of the same category to the capture node 12 according to the sorting result, so as to balance the load between multiple capture nodes 12.

[0041] See also Figure 3 The present application also provides an electronic device 200. The electronic device 200 includes a memory 21 and a processor 22. The memory 21 is used to store computer programs, and the processor 22 can execute computer programs to implement the data processing method in the present application. The processor 22 and the memory 21 can be connected via a bus or other means.

[0042] The electronic device 200 in the present application may be a physical device such as a desktop computer, etc. In particular, when deployed online on a large scale, the electronic device 200 may be a virtual machine.

[0043] The data processing method in the present application classifies the target nodes 11 that generate indicator data of the same database into one category and the grabbing nodes 12 that grab indicator data of the same database into another category for the acquired multiple target nodes 11 and grabbing nodes 12 according to the database type, and sorts the loads of multiple grabbing nodes 12 of the same category, and then allocates target nodes 11 of the same category to the grabbing nodes 12 according to the sorting results, so as to ensure the load balancing of the grabbing nodes 12, thereby improving the durability and stability of the grabbed data.

[0044] Specifically, monitoring, as part of the underlying infrastructure, is an essential part of ensuring the normal operation of online business services. Through the monitoring system, service development and operation and maintenance personnel can promptly and effectively discover service operation anomalies. From discovery to location to resolution of online problems, they all rely on the data collection process, which is one of the basic functions of monitoring.

[0045] Existing monitoring systems, such as Prometheus, collect massive amounts of time series data by either having the crawler nodes periodically pull data from statically configured hosts, or by using service discovery, where the target devices to be monitored can be registered with a service center, such as Kubernetes components, Consul components, etc. The module that schedules tasks can then perform service discovery from the service center to obtain a list of devices that need to be monitored, and then assign tasks to the crawler nodes to pull the time series data for the corresponding target devices.

[0046] Using the former method will require manual configuration or configuration files every time the number of cloud hosts to be monitored changes, which will cause the configuration files to be updated repeatedly and difficult to maintain; using the latter method, even if the number changes of target devices can be dynamically sensed, it is still impossible to ensure that the crawling tasks assigned to multiple crawling nodes are balanced. In this way, when there is a large amount of distributed time series data, such as a large number of business computer rooms to be monitored, which are scattered across the country / globe, if the crawling nodes cannot crawl monitoring data as load-balanced as possible, the persistence and stability of data crawling cannot be guaranteed.

[0047] In view of this, the present application provides a data processing method, which can be applied to the data acquisition system 100. By using this method, the capture node 12 in the data acquisition system 100 can achieve dynamic load balancing.

[0048] It should be noted that during the execution of the business service itself, data is always stored in the corresponding database that has been established, that is, multiple business services can correspond to multiple databases. Since the data acquisition system 100 in this solution can monitor the operating status of business services to ensure the normal operation of online business services, service development and operation and maintenance personnel can promptly and effectively discover abnormalities in service operation. It can be understood that when it is necessary to monitor the operating status of the service, it is necessary to obtain the operating indicators of the service, then the data acquisition system 100 can regularly collect some indicator data of the corresponding database, and then store the indicator data in TSDB (Time Series Database), and then obtain data from TSDB in the graphical interface to display the current operating status indicators of each service.

[0049] In addition, the index data collected is time series data, which is data indexed by time dimension. Time series data describes the measurement value of a measured subject at each time point within a time range. For operation and maintenance personnel, monitoring data is a type of time series data. For example, the monitoring data of the CPU of a machine actually represents the actual CPU consumption value of the machine at each time point.

[0050] In step S10 of the data processing method, the target node 11 is an exporter (ip:port), which is a service that can provide time series data. The service is a set of running programs that provide corresponding services. The target node 11 can provide monitoring sample data to the data acquisition system 100, such as providing indicator data to reflect the current service operation status. The indicator data is generated by the target node 11 after collecting relevant data from the database where it is located and calculated to reflect the current service operation status.

[0051] For example, when the target node 11 is node_exporter, the generated indicator data can reflect the CPU operating status, the number of CPU cores, and the memory usage rate; when the target node 11 is mysql_exporter, the generated indicator data can reflect how much data is stored per second, the number of times data storage fails, etc.

[0052] The scraping node 12 (scraper) can be used to complete the action of scraping indicator data. Among them, the target node 11 can expose the interface, and the scraping node 12 can scrape the time series data of the corresponding database generated by the target node 11 by accessing the interface. It can be understood that the scheduling node 13 can allocate the number of target nodes 11 that the scraping node 12 needs to scrape.

[0053] The number of target nodes 11 and capture nodes 12 can be multiple, for example Figure 2As shown, the target node 11 includes the target node 0, the target node 1, the target node 2 and the target node 3, and the grabbing node 12 includes the grabbing node 0 and the grabbing node 1.

[0054] Multiple target nodes 11 and capture nodes 12 can register services in the registration center 14 before use. Service registration means that the service process registers its own location in the registration center 14, usually registering its own host and port number, and can also register identity authentication information, detailed information of the operating environment, etc. according to actual needs.

[0055] The registration center 14 is equivalent to a configuration center, which is also a service in itself, and is essentially a set of running programs. The registration center 14 in this application can be a registration center cluster, such as Consul Cluster (an open source service discovery and configuration management center service developed in GO language). After the target node 11 and the capture node 12 register the service in the registration center 14, the target node 11 and the capture node 12 can be discovered by the caller in a timely manner. In addition, when the target node 11 and the capture node 12 are newly added, it is called service addition, and when there is a deletion, it is called service deletion. Regardless of whether the service is added or deleted, the target node 11 and the capture node 12 need to update the information in the registration center 14 in a timely manner so that it can be perceived in real time.

[0056] In particular, since the interfaces exposed by the target nodes 11 of different databases are inconsistent, and the indicator data exported by the target nodes 11 of different databases are also inconsistent, when registering, the database type to be captured by the capture node 12 can be marked in advance. In other words, the target node 11 and the capture node 12 can carry parameters for registration, and the parameters can be database types.

[0057] In this way, Figure 2 As shown, in step S10, the scheduling node 13 can obtain multiple target nodes 11 and multiple crawling nodes 12 from the registration center 14, and the behavior of obtaining registered information from the registration center 14 can be called service discovery. It should also be noted that the action of obtaining multiple target nodes 11 and multiple crawling nodes 12 can be obtained by a scheduled task in the scheduling node 13 (a module implemented in a software program that can execute a specific function at a fixed time, which is part of the main program of the scheduling node 13) from the registration center 14 at a fixed time, so that the real-time information of the target node 11 and the crawling node 12 can be dynamically updated.

[0058] In step S20, the scheduling node 13 may classify the multiple target nodes 11 and the multiple crawling nodes 12 based on the database type. As described above, the target nodes 11 and the crawling nodes 12 may carry parameters to register in the registration center 14, and the parameters may be database types. In this way, after the list of target nodes 11 and the list of crawling nodes 12 are pulled, they may be classified according to the database type.

[0059] At this time, the parameter of the database type carried by the target node 11 represents the corresponding database that the target node 11 can collect. The target node 11 can generate indicator data that can reflect the service operation status through some collected data, and can collect the indicator data of the database through the exposed interface of the target node 11. The parameter of the database type carried by the crawling node 12 represents the corresponding database that the crawling node 12 can crawl.

[0060] Classification based on database type means that the target nodes 11 that can collect data from the same database to calculate and generate indicator data are classified into one category, and the capture nodes 12 that can capture indicator data from the same database are classified into one category. At this time, if the indicator data provided by the target node 11 and the indicator data captured by the capture node 12 are both from the same database, the corresponding target node 11 and the capture node 12 are classified into the same category.

[0061] For example, in Figure 2 In the example, target node 0, target node 1 and grasped node 0 are classified into one category, target node 0, target node 1, target node 2 and target node 3 are classified into one category, and grasped node 0 and grasped node 1 are classified into one category.

[0062] In step S30, the scheduling node 13 may perform load sorting on multiple grab nodes 12 of the same category. The load refers to the list of target nodes 11 assigned to the grab nodes 12 by the scheduling node 13. It should be noted that after obtaining the grab nodes 12 from the registration center 14, the list of target nodes 11 of the grab nodes 12 may be obtained from the interface provided by the grab nodes 12, or the corresponding relationship between the grab nodes 12 and the target nodes 11 may be obtained directly from the registration center 14, thereby determining the list of target nodes 11 corresponding to each grab node 12.

[0063] As mentioned above, the same category may mean that the indicator data captured by the capture node 12 comes from the same database. The load sorting can be based on the number of target nodes 11 that the capture node 12 currently has allocated or bound to, to sort the capture nodes 12 of the same category. Of course, the sorting can also be based on the remaining service resources of the current capture node 12.

[0064] In particular, when the target node 11 list of the grabbing node 12 obtained in step S10 is empty, the multiple grabbing nodes 12 can be sorted according to the pulling order. Of course, the load of multiple grabbing nodes 12 of the same category can also be sorted according to other conditions.

[0065] In step S40, the scheduling node 13 can design certain conditions to allocate target nodes 11 of the same category to the grabbing nodes 12 according to the sorting results and actual needs, so as to balance the load among multiple grabbing nodes 12. Among them, the target nodes 11 can be allocated to the grabbing nodes 12 with more remaining service resources first, or the target nodes 11 can be allocated to the grabbing nodes 12 with lower load first. The specific allocation conditions are not inherently limited in this application.

[0066] For example, in Figure 2 In the example, target node 0 and target node 1 are assigned to grab node 0, and target node 2 and target node 3 are assigned to grab node 1, so as to achieve load balancing among multiple grab nodes 12.

[0067] See also Figure 4 In some embodiments, the data processing method may include:

[0068] Step S50: Sending the assigned correspondence between the capture node 12 and the target node 11 to the registration center 14, so that the capture node 12 can obtain the correspondence from the registration center 14 and capture the time series data of the target node 11 according to the correspondence.

[0069] In some embodiments, the scheduling node 13 is used to send the correspondence between the assigned capture node 12 and the target node 11 to the registration center 14, so that the capture node 12 can obtain the correspondence from the registration center 14 and capture the timing data of the target node 11 according to the correspondence.

[0070] In some embodiments, the processor 22 is used to send the assigned correspondence between the capture node 12 and the target node 11 to the registration center 14, so that the capture node 12 can obtain the correspondence from the registration center 14 and capture the time series data of the target node 11 according to the correspondence.

[0071] In this way, the message communication between different nodes can be reduced, thereby reducing the coupling between different nodes, so as to ensure the stability of data capture.

[0072] Specifically, in step S50, after performing load balancing in step S40, the scheduling node 13 may send the corresponding relationship between the assigned grabbing nodes 12 and the target nodes 11 to the registration center 14, wherein the corresponding relationship refers to the list of target nodes 11 assigned to each grabbing node 12. Then, the grabbing node 12 may exchange information with the registration center 14 to obtain the corresponding relationship, so that the grabbing node 12 may determine the list of target nodes 11 that need to be grabbed.

[0073] In this mode, the crawling node 12 only needs to communicate with the registration center 14 to obtain the corresponding target node 11 list, thereby reducing the message communication between different nodes, so as to reduce the coupling between different nodes and ensure the stability of data crawling.

[0074] See also Figure 5 In some embodiments, the data processing method may further include:

[0075] Step S60: reacquire multiple target nodes 11 and multiple captured nodes 12;

[0076] Step S70: when at least one of the target node 11 and the captured node 12 changes, updating the corresponding relationship between the captured node 12 and the target node 11;

[0077] Step S80: When neither the target node 11 nor the captured node 12 changes, the original corresponding relationship is maintained.

[0078] In some embodiments, the scheduling node 13 is used to reacquire multiple target nodes 11 and multiple grabbing nodes 12, and to update the correspondence between the grabbing node 12 and the target node 11 when at least one of the target node 11 and the grabbing node 12 changes, and to maintain the original correspondence when neither the target node 11 nor the grabbing node 12 changes.

[0079] In some embodiments, the processor 22 is used to reacquire multiple target nodes 11 and multiple grab nodes 12, and to update the correspondence between the grab node 12 and the target node 11 when at least one of the target node 11 and the grab node 12 changes, and to maintain the original correspondence when neither the target node 11 nor the grab node 12 changes.

[0080] In this way, by re-triggering the load balancing action every time a change is found between the target node 11 and the capture node 12 to reallocate the list of target nodes 11 that the capture node 12 needs to capture, the scalability of the data acquisition system 100 is enhanced.

[0081] Specifically, when there is massive distributed time series data, such as a large number of business computer rooms to be monitored, the number of cloud hosts is often in dynamic change due to elastic scaling, that is, the monitoring instances are in real-time change. Then the data acquisition system 100 must perceive this change in time to achieve dynamic load balancing.

[0082] In step S60 , multiple target nodes 11 and multiple capture nodes 12 can be retrieved through the scheduled tasks in the scheduling node 13 to perform subsequent steps.

[0083] In step S70, at least one of the target node 11 and the captured node 12 may be changed, the target node 11 may be changed, the captured node 12 may be changed, or both the target node 11 and the captured node 12 may be changed. The change of the node may refer to a change in the number of nodes, such as an increase or decrease; or a change in the IP address of the node, such as a change in the cloud host for execution; or a change in the parameters carried by the node, such as a change in the database type.

[0084] In one embodiment, mysql_exporter is executed on cloud host A, then 192.168.31.237:8080 is one of the target nodes 11, its IP address is 192.168.31.237, port is 8080, and the parameter exporter / mysql is carried during registration. When executed on another cloud host B, the target node 11 is 192.168.31.300:8080. It can be understood that since the parameter exporter / mysql is still carried during registration, the carried parameter remains unchanged and is still the target node 11 of the same database type, and its port remains the same, only the IP address has changed; or, mysql_exporter is abandoned and node_exporter is used instead, and the parameter type carried by the target node 11 has changed.

[0085] After at least one of the target node 11 and the capture node 12 changes, updating the correspondence between the capture node 12 and the target node 11 can be done by repeating steps S20, S30 and S40, re-triggering the load balancing action, and obtaining a new correspondence between the capture node 12 and the target node 11.

[0086] Then, the new correspondence relationship may be sent to the registration center 14 so that the registration center 14 synchronously updates the correspondence relationship, so that the crawling node 12 may obtain the newly allocated target node 11 list from the registration center 14 in a timely manner.

[0087] In step S80, when neither the target node 11 nor the capture node 12 has changed, and the number of nodes and the information of the nodes themselves remain unchanged, steps S20, S30 and S40 are repeated to re-trigger the load balancing action. It can be easily understood that the correspondence between the capture node 12 and the target node 11 obtained at this time is consistent with the previous one, that is, the correspondence transmitted back to the registration center 14 is consistent with the previous one.

[0088] In particular, it should be noted that in the above steps, regardless of whether the target node 11 and the capture node 12 change or not, the scheduling node 13 will re-trigger the load balancing action to update the list of target nodes 11 that the capture node 12 needs to capture, thereby causing the scheduling node 13 to frequently perform unnecessary load balancing calculations. Then, by adding a step of determining whether the target node 11 and the capture node 12 change, the scheduling node 13 can be prevented from frequently performing unnecessary load balancing calculations. The specific method steps will be described in detail below.

[0089] See also Figure 6 and Figure 7 In some embodiments, the data processing method may further include:

[0090] Step S21: Obtain the first hash value obtained by calculating the information of all target nodes 11 of the same category last time, where the information of the target node 11 includes the node address, node type and database type;

[0091] Step S22: Calculate the information of all target nodes 11 of the same category and obtain a second hash value;

[0092] Step S23: When the first hash value is different from the second hash value, confirm that the target node 11 has changed.

[0093] In some implementations, step S21 , step S22 , and step S23 may be implemented by the scheduling node 13 .

[0094] That is, the scheduling node 13 is used to obtain the first hash value obtained by calculating the information of all target nodes 11 of the same category last time, the information of the target node 11 includes the node address, node type and database type, and is used to calculate the information of all current target nodes 11 of the same category and obtain the second hash value, and is used to confirm that the target node 11 has changed when the first hash value is different from the second hash value.

[0095] In some embodiments, the processor 22 is used to obtain the first hash value obtained by calculating the information of all target nodes 11 of the same category last time, the information of the target node 11 includes the node address, node type and database type, and is used to calculate the information of all current target nodes 11 of the same category and obtain the second hash value, and is used to confirm that the target node 11 has changed when the first hash value is different from the second hash value.

[0096] In this way, by comparing the first and second hash values ​​to confirm whether the target node 11 has changed, it is possible to re-trigger the load balancing action to update the list of target nodes 11 that the capture node 12 needs to capture only when the first and second hash values ​​are different, thereby reducing the frequency of load balancing calculations performed by the scheduling node 13.

[0097] Specifically, Hash, generally translated as hash, hash, or transliterated as hash, is to transform an input of arbitrary length (also called pre-image) into an output of fixed length through a hash algorithm, and the output is the hash value. Simply put, it is a function that compresses a message of arbitrary length into a message digest of a fixed length. The hash algorithm is irreversible. Although two different inputs may get the same value, when using a highly secure hash algorithm such as MD5 and SHA, it is almost impossible for two different files to get the same hash result. Therefore, once a file is modified, it can be detected. In this way, the hash consistency algorithm can be used in this application to confirm whether the target node 11 has changed.

[0098] In step S21, the scheduling node 13 may obtain the first hash value obtained by calculating the information of all target nodes 11 of the same category last time. In step S22, the scheduling node 13 may calculate the information of all target nodes 11 of the same category currently and obtain the second hash value.

[0099] In this application, the scheduling node 13 will pull the information of the target node 11 and calculate the hash value of it every time it obtains all the target nodes 11 through the scheduled task. For example, the node address of the target node 11, that is, the IP address and port, the node type, that is, the registration parameters carried to indicate the identity of the target node 11, and the database type, that is, the time series data of which database can be exposed to provide the generated interface.

[0100] In the present application, the hash value calculated by the information of all target nodes 11 currently obtained each time is recorded as the second hash value, and the hash value calculated in the last scheduled task for comparison with the current second hash value is recorded as the first hash value.

[0101] That is to say, the current second hash value can be used as the first hash value for the next comparison, and the second hash value can be saved and overwrite the first hash value. It is easy to understand that each time the scheduling node 13 obtains the information of the target node 11, only one hash calculation is performed.

[0102] In addition, since there may be multiple categories of target nodes 11, the first and second are only descriptive terms and do not limit the specific number of hash values. In practice, there may be multiple first and second hash values ​​at the same time.

[0103] In step S23, when the first hash value is different from the second hash value, the scheduling node 13 can confirm that the target node 11 has changed. It can be understood that when a hash algorithm with higher security is adopted, when the input is not exactly the same, the hash value must be different, then it can be confirmed that the target node 11 has changed, which may be a change in quantity, IP, or database type.

[0104] In this way, it is possible to choose to select that when the first hash value is confirmed to be different from the second hash value, the scheduling node 13 performs load balancing calculation again, allocates the target node 11 list to be collected to the capture node 12, obtains a new correspondence and transmits it back to the registration center 14; and, when the first hash value is confirmed to be the same as the second hash value, no load balancing calculation is performed, and the last target node 11 list obtained from the capture node 12 interface, or the last target node 11 list that has been saved, or the last target node 11 list directly obtained from the registration center 14, obtains the last correspondence and transmits it back to the registration center 14. This avoids the scheduling node 13 from performing unnecessary and frequent load balancing calculations.

[0105] In one embodiment, Figure 7 As shown, target node 3 is the newly added target node 11. In step S21, the first hash value calculated for the information of target node 0, target node 1 and target node 2 is obtained. In step S22, the second hash value is calculated for the information of the current target node 0, target node 1, target node 2 and target node 3. Since target node 3 is newly added, the first hash value is different from the second hash value, and it is determined that target node 11 has changed, and the load balancing calculation can be performed again.

[0106] In particular, the information of the target node 11 may also include other information, for example, when the target node 11 is registered, it may also carry various information such as machine configuration information, machine location information, memory information, etc. The specific information carried can be set according to actual needs, and this application does not impose inherent restrictions on this, as long as it includes basic node address, node type and database type information. It can be understood that when the target node 11 carries parameters other than the node address, node type and database type when registering, all parameter information will be added for calculation when performing hash calculation on all target nodes 11.

[0107] See also Figure 8 and Fig. 9 In some embodiments, the data processing method further comprises:

[0108] Step S24: obtaining a third hash value obtained by calculating the information of all captured nodes 12 of the same category last time, where the information of the captured node 12 includes the node address, node type and database type;

[0109] Step S25: Calculate the information of all captured nodes 12 of the same category and obtain a fourth hash value;

[0110] Step S26: When the third hash value is different from the fourth hash value, it is confirmed that the crawling node 12 has changed.

[0111] In some implementations, step S24 , step S25 , and step S26 may be implemented by the scheduling node 13 .

[0112] That is, the scheduling node 13 is used to obtain the third hash value obtained by calculating the information of all the grabbing nodes 12 of the same category last time, the information of the grabbing node 12 includes the node address, node type and database type, and is used to calculate the information of all the current grabbing nodes 12 of the same category and obtain the fourth hash value, and is used to confirm that the grabbing node 12 has changed when the third hash value is different from the fourth hash value.

[0113] In some embodiments, the processor 22 is used to obtain the third hash value obtained by calculating the information of all captured nodes 12 of the same category last time, the information of the captured node 12 includes the node address, node type and database type, and is used to calculate the information of all captured nodes 12 of the same category currently and obtain the fourth hash value, and is used to confirm that the captured node 12 has changed when the third hash value is different from the fourth hash value.

[0114] In this way, by comparing the third and fourth hash values ​​to confirm whether the information of the capture node 12 has changed, it is possible to re-trigger the load balancing action to update the correspondence between the capture node 12 and the target node 11 only when the third and fourth hash values ​​are different, thereby reducing the frequency of load balancing calculations performed by the scheduling node 13.

[0115] Specifically, in step S21, the scheduling node 13 may obtain the third hash value obtained by calculating the information of all the same-category grabbing nodes 12 last time. In step S22, the scheduling node 13 may calculate the information of all the current same-category grabbing nodes 12 and obtain the fourth hash value.

[0116] In the present application, each time the scheduling node 13 obtains all the grabbing nodes 12 through the scheduled task, it will pull the information of all the grabbing nodes 12 and calculate the hash value for them. For example, the node address of the grabbing node 12, that is, the IP address and port, the node type, that is, the registration parameters carried to indicate the identity of the grabbing node 12, and the database type, that is, the timing data provided by the target node 11 of the corresponding database type that can be captured. In the present application, the hash value calculated by all the grabbing nodes 12 currently obtained each time is recorded as the fourth hash value, and the hash value calculated in the last scheduled task, which is compared with the current fourth hash value, is recorded as the third hash value.

[0117] That is to say, the current fourth hash value can be used as the third hash value for the next comparison, and the fourth hash value can be saved and overwrite the third hash value. It is easy to understand that each time the scheduling node 13 obtains the capture node 12, only one hash calculation is performed.

[0118] In addition, since there may be multiple categories of capture nodes 12, the third and fourth are only descriptive terms and do not limit the specific number of hash values. In fact, there may be multiple third and fourth hash values ​​at the same time.

[0119] In step S23, when the third hash value is different from the fourth hash value, the scheduling node 13 can confirm that the crawling node 12 has changed. It can be understood that when a hash algorithm with higher security is adopted, when the input is not exactly the same, the hash value must be different, then it can be confirmed that there is a change in the crawling node 12, which may be a change in quantity, IP, or database type.

[0120] In this way, it is possible to choose to select that when the third hash value is confirmed to be different from the fourth hash value, the scheduling node 13 performs load balancing calculation again, allocates the target node 11 list to be collected to the capture node 12, obtains a new correspondence and transmits it back to the registration center 14; and, when the third hash value is confirmed to be the same as the fourth hash value, no load balancing calculation is performed, and the last target node 11 list obtained from the capture node 12 interface, or the last target node 11 list that has been saved, or the last target node 11 list directly obtained from the registration center 14, obtains the last correspondence and transmits it back to the registration center 14. This avoids the scheduling node 13 from performing unnecessary and frequent load balancing calculations.

[0121] In one embodiment, Fig. 9 As shown, the grab node 3 is the newly added grab node 12. In step S21, the first hash value calculated for the information of the grab nodes 0, 1 and 2 is obtained. In step S22, the second hash value is calculated for the information of the current grab nodes 0, 1, 2 and 3. Since the grab node 3 is newly added, the third hash value is different from the fourth hash value, and it is determined that the grab node 12 has changed, and the load balancing calculation can be performed again.

[0122] In this way, the target node 11 and the capture node 12 are first registered in the registration center 14 with parameters, and then the scheduling node 13 pulls the target node 11 and the capture node 12 from the registration center 14 through the scheduled task. Then, the first hash value, the second hash value, the third hash value and the fourth hash value are compared. When at least one of the first hash value and the second hash value or the third hash value and the fourth hash value is not equal, the database type is classified, and after the load balancing algorithm is used to allocate the load to the capture node 12, the corresponding relationship between the capture node 12 and the target node 11 obtained after the allocation is completed is transmitted back to the registration center 14, so that the capture node 12 obtains the corresponding relationship from the registration center 14, thereby confirming and capturing the allocated target node 11.

[0123] That is, through the form of service registration and service discovery, the coupling between nodes is reduced to ensure the stability of data capture; and combined with the hash comparison algorithm, in a massive distributed system, when the monitoring node and the capture node 12 are in dynamic expansion, the scheduling node 13 can timely update the list information of the target node 11 and the capture node 12 according to whether the hash value changes, and determine whether the load balancing list needs to be refreshed, that is, re-do the load balancing calculation to allocate the target node 11 list to the capture node 12 and update the corresponding relationship. Thereby, the scalability of the data acquisition system 100 can be increased, and the update frequency of the scheduling node 13 can be effectively reduced.

[0124] In particular, the information of the crawling node 12 may also include other information, for example, when the crawling node 12 is registered, it may also carry various information such as machine configuration information, machine location information, memory information, etc. The specific information carried can be set according to actual needs, and this application does not impose inherent restrictions on this, as long as it includes basic node address, node type and database type information. It can be understood that when the crawling node 12 carries parameters other than the node address, node type and database type when registering, all parameter information will be added for calculation when performing hash calculation on all crawling nodes 12.

[0125] See also Figure 2 and Fig.10 In some embodiments, assigning a target node 11 of the same category to the crawling node 12 according to the sorting result (step S40) includes:

[0126] Step S41: preferentially assigning target nodes 11 of the same category to grab nodes 12 with low load.

[0127] In some embodiments, step S41 may be implemented by the scheduling node 13. That is, the scheduling node 13 is used to preferentially allocate target nodes 11 of the same category to the grabbing nodes 12 with low load.

[0128] In some embodiments, the processor 22 is configured to preferentially assign target nodes 11 of the same category to crawling nodes 12 with low load.

[0129] In this way, by preferentially allocating target nodes 11 of the same category to grabbing nodes 12 with low load, it can be ensured that the grabbing tasks allocated to multiple grabbing nodes 12 are balanced, avoiding the pressure caused by too heavy tasks on a certain grabbing node 12.

[0130] Specifically, in step S41, through step S30, multiple grabbing nodes 12 of the same category are load sorted to obtain a sorting result. It can be understood that the load sorting can sort the grabbing nodes 12 of the same category according to the number of target nodes 11 currently allocated or bound to the grabbing nodes 12 that need to be grabbed. Of course, the sorting can also be performed according to the remaining service resources of the current grabbing nodes 12.

[0131] When the former method is used for sorting, giving priority to allocating target nodes 11 of the same category to grabbing nodes 12 with low load can be to first allocate grabbing tasks to grabbing nodes 12 that are bound to a smaller number of target nodes 11; when the latter method is used for sorting, giving priority to allocating target nodes 11 of the same category to grabbing nodes 12 with low load can be to first allocate grabbing tasks to grabbing nodes 12 with more remaining service resources.

[0132] See also Fig.11In some embodiments, assigning a target node 11 of the same category to the crawling node 12 according to the sorting result (step S40) includes:

[0133] Step S42: when the load difference between the grabbing node 12 with the largest load and the grabbing node 12 with the smallest load is greater than a preset range, in response to the command, reallocate the target nodes 11 of the same category to the grabbing nodes 12 .

[0134] In some embodiments, step S42 may be implemented by the scheduling node 13. In other words, the scheduling node 13 is used to reallocate target nodes 11 of the same category to the grabbing nodes 12 in response to a command when the load difference between the grabbing node 12 with the largest load and the grabbing node 12 with the smallest load is greater than a preset range.

[0135] In some embodiments, the processor 22 is used to reallocate target nodes 11 of the same category to the grabbing nodes 12 in response to a command when the load difference between the grabbing node 12 with the largest load and the grabbing node 12 with the smallest load is greater than a preset range.

[0136] In this way, by issuing a command by the user, the scheduling node 13 responds to the command and reallocates the target node 11 of the same category to the capture node 12, so that the load between multiple capture nodes 12 can be balanced more efficiently to ensure the stability of data capture.

[0137] Specifically, in step S42, the preset range can be set according to the deployment method of the server, the service resources of the server and the actual needs. For example, it can be determined according to the load balancing algorithm. After the algorithm is determined, the error range that may be generated by the algorithm when the load balancing algorithm is specifically implemented can be calculated. Therefore, the approximate maximum and minimum load difference can be set as the preset range.

[0138] By setting a preset range and retaining the active provision of commands to reallocate target nodes 11 of the same category to the crawling nodes 12, load balancing between multiple crawling nodes 12 can be quickly restored in the face of extreme concurrent scenarios.

[0139] For example, in one scenario, a large number of crawling tasks pour in within a very short period of time. Due to concurrent processing, the target node 11 list calculated by crawling node 12 at the current moment may be wrong. At this time, crawling node 0 in crawling node 12 may have been assigned 100 tasks including target node 0-target node 99, while crawling node 1 in crawling node 12 has only been assigned 2 tasks including target node 100-target node 101.

[0140] At this time, the load difference between the grabbing node 0 with the largest load and the grabbing node 1 with the smallest load has exceeded the preset range. If the load balancing is automatically performed according to the load balancing algorithm, for example, the next time a target node 102 task arrives, it will be assigned to the grabbing node 1, which will cause the load difference between the grabbing nodes 12 to be unable to be reduced in a short period of time, which may easily cause the data acquisition system 100 to be unstable.

[0141] Then next time when a target node 102 task arrives, the user can actively issue a command to the scheduling node 13 to actively perform load balancing, that is, to reallocate the tasks of the crawling node 0 and the crawling node 1, so that the crawling node 0 is allocated target node 0-target node 55, and the crawling node 1 is allocated target node 56-target node 102, so that the load between multiple crawling nodes 12 can be balanced more efficiently to ensure the stability of data crawling.

[0142] See also Fig.12 In some implementations, the scheduling node 13 includes at least one master node and one slave node. When the master node fails, the slave node can take over the work of the master node. In this way, the operation of the scheduling node 13 in a disaster recovery scenario can be guaranteed.

[0143] Specifically, the scheduling node 13 mentioned in the present application refers to a node that can implement the scheduling function. The scheduling node 13 can be deployed in a form including but not limited to cluster deployment. The scheduling node 13 includes at least one master node and one slave node. In other words, the master node and the slave node can both be considered as scheduling nodes 13 to implement the scheduling function.

[0144] The scheduling node 13 can automatically elect a master node and a slave node, and data synchronization can be achieved between the master node and the slave node. Fig.12 As shown, the scheduling node 130 is the master node, and the scheduling nodes 131 and 132 are slave nodes. The master node is responsible for pulling data from the registration center 14, that is, pulling the crawling node 12 and the target node 11, and forwarding this batch of data to the slave nodes on average. The slave nodes calculate the load information and synchronize the calculation in the scheduling node 13 cluster. This can reduce the load balancing calculation amount of each slave node and improve the load balancing efficiency.

[0145] It can be understood that the master node and slave node in the scheduling node 13 are only for the scheduling node 13 cluster, and externally appear to be only one node. When the scheduling node 13 is deployed into multiple, if no processing is done, each scheduling node 13 will have the same corresponding relationship after load balancing because the captured node 12 and the target node 11 pulled are the same. This deployment does not play any effective positive role, so the present application adopts the form of deploying the scheduling node 13 as a master-slave node, so that the existence of the master node is to manage the slave node, the slave node processes specific business requests, and the master and slave nodes rely on heartbeat detection for status perception. When the master node cannot provide services due to a failure, the slave node will take over the master node and continue to provide services.

[0146] See also Fig.12 In some embodiments, the data acquisition system may further include a registration center 14, and the target node 11, the capture node 12 and the scheduling node 13 may exchange information with the registration center 14. The scheduling node 13 may generate a snapshot file 15 after acquiring the target node 11 and the capture node 12 each time, so that itself and the registration center 14 may restore data through the snapshot file 15.

[0147] In this way, when the scheduling node 13 encounters an abnormal reason and restarts, the data can be restored according to the snapshot file 15, and because the snapshot file 15 is generated each time the target node 11 and the capture node 12 are acquired, that is, the file data will be loaded into the memory each time the target node 11 and the capture node 12 are acquired, the timeliness of the restored file data can be guaranteed. In addition, after the registration center 14 crashes or data is lost, the data of the registration center 14 can also be restored through the above snapshot file 15.

[0148] Specifically, the Storage Networking Industry Association (SNIA) defines a snapshot as: a fully usable copy of a specified data set, which contains a static image of the source data at the time of the copy. A snapshot can be a copy or replication of data reproduction. For a file system, a file system snapshot is an instant copy of the file system, which contains all the information of the file system at the time the snapshot is generated, and is itself a fully usable copy.

[0149] The snapshot file 15 can be saved on the machine running the scheduling node 13, so the scheduling node 13 can directly load the snapshot file 15 for data recovery. In particular, in the data processing method of the present application, the information of the capture node 12 and the target node 11 will be generated into a local snapshot file 15 according to the configured time interval, that is, the snapshot file 15 is generated each time the target node 11 and the capture node 12 are acquired, so that the timeliness of the saved snapshot file 15 can be guaranteed, thereby ensuring the timeliness of the data of the scheduling node 13 and the data of the registration center 14 restored according to the snapshot file 15.

[0150] It should also be noted that the scheduling node 13 can register the node data back to the registration center 14 so that the registration center 14 can restore the data, or the registration center 14 can obtain the snapshot file 15 from the interface of the scheduling node 13 to restore the data.

[0151] The embodiments of the present application provide a non-volatile computer-readable storage medium storing a computer program. When the computer program is executed by one or more processors 22, the processor 22 executes the data processing method of any of the above embodiments.

[0152] For example, when the computer program is executed by one or more processors 22, the processor 22 is caused to perform the following steps:

[0153] Step S10: Acquire multiple target nodes 11 and multiple captured nodes 12;

[0154] Step S20: Classifying the plurality of target nodes 11 and the plurality of captured nodes 12 based on database type;

[0155] Step S30: sorting the loads of the plurality of grabbing nodes 12 of the same category;

[0156] Step S40 : allocating the target nodes 11 of the same category to the capture nodes 12 according to the sorting result, so as to balance the load among the plurality of capture nodes 12 .

[0157] Specifically, the processor 22 may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0158] The computer program may be stored in the memory 21. The memory 21, as a non-transitory computer-readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the methods in the above method embodiments. The processor 22 executes various functional applications and data processing of the processor 22 by running the non-transitory software programs, instructions and modules stored in the memory 21, that is, implements the methods in the above method embodiments.

[0159] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the implemented program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory 21.

[0160] In the description of this specification, the description with reference to the terms "one embodiment", "certain embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0161] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A data processing method, used in a data acquisition system, characterized in that: The data processing method comprises: Get multiple target nodes and multiple grab nodes; Based on the database type, the plurality of target nodes and the plurality of crawling nodes are classified, the target nodes can provide time series data corresponding to the database type, and the crawling nodes can crawl the time series data provided by the target nodes corresponding to the database type, wherein the time series data of the database type comes from the time series data of the online business service operation; Performing load sorting on the plurality of said crawling nodes of the same category; Allocating the target nodes of the same category to the grabbing nodes according to the sorting result, so as to balance the load among the plurality of the grabbing nodes; Sending the assigned correspondence between the capture node and the target node to a registration center, so that the capture node can obtain the correspondence from the registration center and capture the corresponding time series data of the target node according to the correspondence; Re-acquire multiple target nodes and multiple grab nodes; Obtaining a first hash value obtained by calculating information of all target nodes of the same category last time, wherein the information of the target node includes a node address, a node type, and the database type; Calculate the information of all the target nodes of the same category and obtain a second hash value; When the first hash value is different from the second hash value, confirming that the target node has changed; When the target node changes, the corresponding relationship between the captured node and the target node is updated.

2. The data processing method according to claim 1, characterized in that: The data processing method further includes: Re-acquire multiple target nodes and multiple grab nodes; When the grabbing node changes, or when both the target node and the grabbing node change, updating the corresponding relationship between the grabbing node and the target node; When neither the target node nor the captured node changes, the original corresponding relationship is maintained.

3. The data processing method according to claim 2, characterized in that: The data processing method comprises: Obtaining a third hash value obtained by calculating the information of all the captured nodes of the same category last time, wherein the information of the captured nodes includes a node address, a node type, and the database type; Calculate the information of all the captured nodes of the same category and obtain a fourth Hash value; When the third hash value is different from the fourth hash value, it is confirmed that the capture node has changed.

4. The data processing method according to claim 1, characterized in that: The allocating the target nodes of the same category to the captured nodes according to the sorting results includes: The target nodes of the same category are preferentially allocated to the grabbing nodes with low load.

5. The data processing method according to claim 1, characterized in that: The allocating the target nodes of the same category to the captured nodes according to the sorting results includes: When the load difference between the grabbing node with the largest load and the grabbing node with the smallest load is greater than a preset range, in response to a command, the target nodes of the same category are reallocated to the grabbing nodes.

6. A data acquisition system, characterized in that: include: Target node; A capture node, used to capture the time series data provided by the target node; A scheduling node, used to execute the data processing method described in any one of claims 1-5.

7. The data acquisition system according to claim 6, characterized in that: The scheduling node includes at least one master node and one slave node. When the master node fails, the slave node can take over the work of the master node.

8. The data acquisition system according to claim 6, characterized in that: The data acquisition system also includes a registration center, and the target node, the capture node and the scheduling node can exchange information with the registration center. The scheduling node can generate a snapshot file after acquiring the target node and the capture node each time, so that itself and the registration center can restore data through the snapshot file.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the data processing method according to any one of claims 1 to 5.

10. A non-volatile computer-readable storage medium of computer-executable instructions, characterized in that: When the computer executable instructions are executed by one or more processors, the processors are caused to execute the data processing method according to any one of claims 1 to 5.

Images