Data acquisition system and method

Abstract

The invention relates to the technical field of data processing, and provides a data acquisition system and method. The system comprises a plurality of acquisition modules which are respectively deployed in a plurality of acquired devices, and the plurality of acquired devices are distributed in at least one data center; the plurality of lower-layer monitoring modules are respectively deployed in a plurality of first servers, and the plurality of first servers are distributed in at least one data center; each lower-layer monitoring module is in communication connection with at least one acquisition module; the plurality of upper-layer monitoring modules are respectively deployed in the plurality of second servers; each upper-layer monitoring module is in communication connection with all the lower-layer monitoring modules; the plurality of first servers and the plurality of second servers form a federated cluster; the storage module is deployed in storage equipment; and the storage module is in communication connection with each upper-layer monitoring module. According to the method, the technical problem that the existing method has high requirements on data consistency, hardware and software resources and user programming ability can be avoided while the accuracy and reliability of data acquisition are ensured.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data acquisition system and method. Background Art

[0002] In the information and data era, whether it is business operations, scientific research or government management, it is increasingly dependent on data to make decisions. System data is the cornerstone to ensure the efficient operation of business processes and scientific decision-making. Therefore, it is particularly important to ensure the accuracy and reliability of system data collection.

[0003] Common data collection technologies include distributed data collection technology, real-time data collection technology, and custom data collection technology. Among them, distributed data collection technology has high requirements for data consistency; real-time data collection technology requires real-time data transmission and has high requirements for hardware and software resources; custom data collection technology requires users to have certain programming and customization capabilities. Summary of the invention

[0004] The embodiments of the present application provide a data acquisition system and method, which can ensure the accuracy and reliability of data acquisition while avoiding the technical problems of existing acquisition methods with high requirements on data consistency, hardware and software resources, and user programming capabilities.

[0005] In order to achieve the above objectives, in a first aspect, an embodiment of the present application provides a data acquisition system, the system comprising: A plurality of acquisition modules are respectively deployed in a plurality of collected devices, and the plurality of collected devices are distributed in at least one data center, and each acquisition module is configured to collect system resource data of the collected devices where it is deployed; a plurality of lower-level monitoring modules are respectively deployed in a plurality of first servers, and the plurality of first servers are distributed in at least one data center; each lower-level monitoring module is communicatively connected with at least one acquisition module, and is configured to obtain system resource data from the acquisition module communicated with it; a plurality of upper-level monitoring modules are respectively deployed in a plurality of second servers; each upper-level monitoring module is communicatively connected with all lower-level monitoring modules, and is configured to obtain system resource data from the lower-level monitoring modules communicated with it; the plurality of first servers and the plurality of second servers form a federal cluster; a storage module is deployed in a storage device; the storage module is communicatively connected with each upper-level monitoring module, and is configured to store the system resource data obtained by the upper-level monitoring module.

[0006] The data collection system shown in the embodiment of the present application collects data through the NodeExporter component of the Prometheus ecosystem, without the need for users to independently develop data collection plug-ins. By setting up two or more upper-level PrometheusServers, high availability is achieved, and data can still be collected normally when an upper-level Prometheus Server fails. In addition, the data collection system adopts the architecture of a federal cluster, and each lower-level Prometheus Server aggregates data to the upper-level Prometheus Server. In this way, when expansion is needed, only new NodeExporter and PrometheusServer need to be added, without adjusting the existing architecture. Therefore, the data collection system is easy to expand to adapt to different data collection scales.

[0007] In one implementation, the acquisition module and the lower-level monitoring module that are in communication connection belong to the same data center. With this implementation, the lower-level monitoring module of each data center is only responsible for the system resource data of the collected equipment in the data center, which can reduce network delay and bandwidth consumption, and reduce the complexity of the network architecture of the data acquisition system.

[0008] In one implementation, each acquisition module corresponds to a first number of acquisition tasks, and each lower-level monitoring module corresponds to a acquisition task; each acquisition module is communicatively connected to the first number of lower-level monitoring modules, and the acquisition modules and lower-level monitoring modules that are communicatively connected correspond to the same acquisition task; the lower-level monitoring module corresponding to the acquisition task is configured to obtain the system resource data obtained by the acquisition module when executing the acquisition task for the deployed acquired device. With this implementation, different types of acquisition tasks can be executed using the acquisition module, and different types of acquisition tasks can be divided into different lower-level monitoring modules. Each lower-level monitoring module is only responsible for collecting system resource data corresponding to one type of acquisition task, which is conducive to the targeted unification and archiving of system resource data corresponding to different acquisition tasks.

[0009] In one implementation, the data acquisition system also includes multiple middle-level monitoring modules, which are deployed in multiple third servers, and the multiple third servers are distributed in at least one data center; the middle-level monitoring modules are arranged between each upper-level monitoring module and all lower-level monitoring modules; wherein each middle-level monitoring module is communicatively connected to all upper-level monitoring modules, and each middle-level monitoring module is communicatively connected to at least one lower-level communication module. By adopting this implementation, the collected system resource data can be summarized layer by layer through the middle-level monitoring modules, thereby reducing the workload of a single monitoring module. In addition, by using this implementation, the data acquisition system architecture is divided into multiple independent layers, so that each layer can be expanded independently to cope with different load requirements.

[0010] In one implementation, the lower-layer monitoring module and the middle-layer monitoring module that are in communication connection belong to the same data center. With this implementation, the middle-layer monitoring module of each data center is only responsible for the system resource data collected by the lower-layer monitoring module in the data center, which can reduce network delay and bandwidth consumption, and reduce the complexity of the network architecture of the data collection system.

[0011] In one implementation, each middle-level monitoring module corresponds to a second number of collection tasks, each middle-level monitoring module is in communication connection with a second number of lower-level monitoring modules, and the middle-level monitoring modules and lower-level monitoring modules that are in communication connection correspond to the same collection task. With this implementation, the collection tasks can be divided into different middle-level monitoring modules based on the task type, and each middle-level monitoring module is responsible for collecting system resource data corresponding to several types of collection tasks, so as to facilitate archiving of the system resource data collected by the collection modules when executing different types of collection tasks.

[0012] In one implementation, the middle-level monitoring module includes at least one level, each level includes at least one sub-monitoring module, and each sub-monitoring module is deployed in a third server; in the highest level, each sub-monitoring module is connected to all upper-level monitoring modules; in the lowest level, each sub-monitoring module is connected to at least one lower-level monitoring module; in any other level, each sub-monitoring module is connected to a sub-monitoring module in an adjacent high-level level, and each sub-monitoring module is connected to at least one sub-monitoring module in an adjacent low-level level. By adopting this implementation, multiple levels can be set in the middle-level monitoring module to summarize the collected system resource data layer by layer, reduce the workload of a single level, and facilitate independent expansion of each level to cope with different load requirements.

[0013] In one implementation, the storage configuration information of the upper-layer monitoring module includes a first storage address, and the first storage address is configured to indicate a communication address of a storage device. With this implementation, the upper-layer monitoring module can determine the communication address of the storage device by reading the storage configuration information, and then send the system resource data to the storage device, thereby realizing remote storage of the system resource data.

[0014] In one implementation, the data acquisition system further includes a proxy module, which is deployed in a proxy server; the proxy module is in communication connection with at least one upper-layer monitoring module and a storage module, and is configured to forward the system resource data in the upper-layer monitoring module in communication connection with the proxy module to the storage module; the storage configuration information of the upper-layer monitoring module includes a second storage address, and the second storage address is configured to indicate the communication address of the proxy server. With this implementation, when the upper-layer monitoring module cannot communicate directly with the storage device, the system resource data can be indirectly sent to the storage device through the proxy module, thereby realizing remote storage of the system resource data.

[0015] In one implementation, the data collection system further includes a reverse proxy module, which is deployed in a reverse proxy server; the reverse proxy module is connected to all upper-layer monitoring modules and is configured to respond to request information input by a user, and obtain system resource data corresponding to the request information through at least one upper-layer monitoring module connected to the reverse proxy module. With this implementation, the reverse proxy server can feedback the system resource data corresponding to the request information to the user based on the request information, and then complete operations such as querying the system resource data, so that the user can use the system resource data to implement system monitoring.

[0016] In a second aspect, the present application further provides a data collection method, which is applied to the data collection system in the first aspect and any implementation thereof, and the method comprises: The acquisition module acquires the system resource data of the acquired device; the lower-level monitoring module acquires the system resource data from the acquisition module; the upper-level monitoring module acquires the system resource data from the lower-level monitoring module; the upper-level monitoring module sends the system resource data to the storage module; the storage module saves the system resource data.

[0017] In one implementation, the upper-level monitoring module obtains the system resource data from the lower-level monitoring module, including: the upper-level monitoring module sends a first data pull request to the middle-level monitoring module that is communicatively connected to the upper-level monitoring module; the middle-level monitoring module sends a second data pull request to the lower-level monitoring module that is communicatively connected to the middle-level monitoring module in response to the first data pull request; after receiving the system resource data corresponding to the second data pull request sent by the lower-level monitoring module, the middle-level monitoring module sends the system resource data to the upper-level monitoring module.

[0018] In one implementation, the upper-layer monitoring module obtains the system resource data from the lower-layer monitoring module, including: each of the upper-layer monitoring modules obtains the system resource data from at least one of the lower-layer monitoring modules respectively; or, the reverse proxy module adopts a polling method to determine a target upper-layer monitoring module among multiple upper-layer monitoring modules, and the target upper-layer monitoring module obtains the system resource data from at least one of the lower-layer monitoring modules.

[0019] In one implementation, the upper-layer monitoring module sends the system resource data to the storage module, including: The upper-layer monitoring module accesses the storage device corresponding to the first storage address based on the first storage address in the configuration information, and sends the system resource data to the storage module deployed in the storage device.

[0020] In one implementation, the upper-level monitoring module sends the system resource data to the storage module, including: the upper-level monitoring module accesses the proxy server corresponding to the second storage address based on the second storage address in the configuration information, and sends the system resource data to the proxy module deployed in the proxy server; the proxy module accesses the storage device corresponding to the first storage address based on the first storage address in the proxy configuration information, and sends the system resource data to the storage module deployed in the storage device.

[0021] In one implementation, the method further includes: The reverse proxy module responds to the request information input by the user and uses the upper-layer monitoring module to obtain the system resource data corresponding to the request information in the upper-layer monitoring module or the storage module.

[0022] It can be understood that the beneficial effects that can be achieved by the technical solution provided by the second aspect and any of its implementation methods provided above can refer to the beneficial effects in the first aspect and any of its optional implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 It is a simplified architecture diagram of the data acquisition system provided in an embodiment of the present application.

[0025] Figure 2 It is a system architecture diagram of the data acquisition system provided in an embodiment of the present application.

[0026] Figure 3 This is another deployment architecture diagram of the data acquisition system provided in an embodiment of the present application.

[0027] Figure 4 It is a structural diagram of a middle-level monitoring module provided in an embodiment of the present application.

[0028] Figure 5 This is a system architecture diagram of another data acquisition system provided in an embodiment of the present application.

[0029] Figure 6 It is a flow chart of the data collection method provided in the embodiment of the present application.

[0030] Figure 7 It is another flow chart of the data collection method provided in an embodiment of the present application.

[0031] Figure 8 This is another flow chart of the data collection method provided in the embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described clearly below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments of the present application, other embodiments obtained by ordinary technicians in this field without making creative work all belong to the protection scope of the present application.

[0033] In the following, the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0034] In addition, in the present application, directional terms such as "upper", "lower", "inner" and "outer" are defined relative to the orientation of the components schematically placed in the drawings. It should be understood that these directional terms are relative concepts. They are used for relative description and clarification, and they can change accordingly according to the changes in the orientation of the components placed in the drawings.

[0035] The following first describes the application scenarios of the embodiments of the present application.

[0036] In the information and data era, whether it is business operations, scientific research or government management, it is increasingly dependent on data to make decisions. System data is the cornerstone to ensure the efficient operation of business processes and scientific decision-making. Therefore, it is particularly important to ensure the accuracy and reliability of system data collection.

[0037] Common data collection technologies include distributed data collection technology, real-time data collection technology, and custom data collection technology. Among them, distributed data collection technology has high requirements for data consistency; real-time data collection technology requires real-time data transmission and has high requirements for hardware and software resources; custom data collection technology requires users to have certain programming and customization capabilities.

[0038] To solve this problem, there are currently some systems that can be used to implement data collection. Based on the idea of ​​federated layering, a hierarchical monitoring module that is compatible with the firewall network architecture is built. The hierarchical monitoring module includes a collection layer module, a federated layer module, and a global layer module. Each node at each level of the hierarchical monitoring module has two Prometheus instances, which collect data for the same monitoring target at the same time. The federated layer module accesses each collection layer module node to obtain and store the aggregated and reduced monitoring indicator data.

[0039] However, the above system configures two Prometheus instances for each node at the monitoring target level, which makes the network structure complex and requires more hardware resources. In addition, the hierarchical monitoring module is compatible with the multi-firewall network architecture, that is, the monitoring partition is implemented based on the firewall architecture, and it is impossible to partition based on the task type, which is not conducive to data archiving and analysis. In addition, the hierarchical monitoring module structure is limited by the firewall network architecture, so the scalability and portability are poor. In summary, the above system will cause several new problems when solving the problems of common data collection technologies.

[0040] Based on this, in order to ensure the accuracy and reliability of data collection while avoiding the problems of existing collection methods with high requirements on data consistency, hardware and software resources, and user programming capabilities, this application proposes a data collection system.

[0041] Combine the following Figures 1 to 5 , which illustrates the deployment architecture of the data acquisition system.

[0042] Figure 1 It is a simplified architecture diagram of the data acquisition system provided in an embodiment of the present application.

[0043] like Figure 1 As shown, the device to be collected is used as a collection node, on which a collection module is deployed. The collection module is in communication connection with the monitoring module. The monitoring module is in communication connection with the storage module deployed in the remote storage device.

[0044] Among them, the collection module may include the NodeExporter component of the Prometheus ecosystem; the monitoring module may include the server-side component of the Prometheus ecosystem, deployed in the Prometheus Server; the storage module may include a distributed file system, a database application, etc.

[0045] In this way, the NodeExporter component collects system resource data and can transmit it to the remote storage module through the monitoring module deployed on the Prometheus Server, and then save it in the storage device. Through this automated and centralized monitoring method, the data collection system can reduce manual intervention and reduce operation and maintenance costs.

[0046] Figure 2 It is a system architecture diagram of the data acquisition system provided in an embodiment of the present application.

[0047] like Figure 2 As shown, the data acquisition system may include multiple acquisition modules, multiple lower-level monitoring modules, multiple upper-level monitoring modules and a storage module.

[0048] Each acquisition module is deployed in a collected device and can collect system resource data of the collected device where it is deployed. Specifically, the collected device can be a PC, server, all-in-one machine, workstation or other electronic device, and multiple collected devices can be distributed in at least one data center. Figure 2 In the figure, multiple collected devices are represented by S1-S3 respectively, and are distributed in the same data center A. Figure 2 The different shapes of the collected devices can indicate that they are deployed with different operating systems, such as Windws, Linux, etc.

[0049] Each lower-level monitoring module is deployed in a first server (hereinafter referred to as the lower-level Prometheus Server), and each lower-level monitoring module is connected to at least one acquisition module in communication, and can obtain system resource data from the acquisition module connected to it in communication. Specifically, multiple lower-level Prometheus Servers can be distributed in at least one data center, so multiple lower-level monitoring modules can also be distributed in at least one data center. Figure 2 In the figure, the two lower-level monitoring modules are represented by P11 and P12 respectively, and are distributed in the same data center A.

[0050] Each upper-layer monitoring module is deployed in a second server (hereinafter referred to as the upper-layer Prometheus Server), and each upper-layer monitoring module is connected to all lower-layer monitoring modules and can obtain system resource data from the lower-layer monitoring modules to which it is connected. Figure 2 In the figure, two upper monitoring modules are represented by P21 and P22 respectively, and each upper monitoring module is connected to the lower monitoring modules P11-P12 in communication. In this way, each lower monitoring module is connected to two upper monitoring modules at the same time. When any upper monitoring module fails, the other upper monitoring module can still work normally to achieve high availability.

[0051] The storage module is deployed in a remote storage device, communicates with each upper-level monitoring module, and can store system resource data obtained by the upper-level monitoring module. Specifically, the storage device can adopt an object storage or a distributed storage system. Figure 2In the embodiment of the present application, a large amount of system resource data can be persistently stored by storing data in a remote storage device.

[0052] In some embodiments, the collection module may include the NodeExporter component of the Prometheus ecosystem; the lower-level monitoring module may include the server-side component of the Prometheus ecosystem; the upper-level monitoring module may include the server-side component of the Prometheus ecosystem; the storage module may include a distributed file system, a database application, etc.

[0053] At this time, the NodeExporter component can collect system resource data from the collected device. After that, the NodeExporter component transmits the collected system resource data to the lower-level monitoring module deployed on the lower-level Prometheus Server, and transmits the system resource data to the upper-level monitoring module deployed on the upper-level Prometheus Server through the lower-level monitoring module. After receiving the system data, the upper-level monitoring module can process the system resource data and store the system resource data in a local or remote storage device.

[0054] The data collection system provided in the embodiment of the present application collects data through the NodeExporter component of the Prometheus ecosystem, without the need for users to independently develop data collection plug-ins. By setting two or more upper-level monitoring modules, high availability is achieved, and data can still be collected normally when any upper-level monitoring module fails. In addition, the data collection system adopts the architecture of a federal cluster, and each lower-level monitoring module deployed by the lower-level Prometheus Server aggregates data to the upper-level monitoring module deployed by the upper-level Prometheus Server. In this way, when expansion is needed, only new NodeExporter and Prometheus Server need to be added, without adjusting the existing architecture. Therefore, the data collection system is easy to expand to adapt to different data collection scales.

[0055] Combine the following Figure 3 , introduces the specific connection methods between the acquisition module, lower-level monitoring module, and upper-level monitoring module.

[0056] Figure 3 This is another deployment architecture diagram of the data acquisition system provided in an embodiment of the present application.

[0057] First combine Figure 3 , introduces the specific connection method between the acquisition module and the lower-level monitoring module.

[0058] In scenarios with fewer collection tasks, such as Figure 3 In data center A, a lower-level monitoring module can communicate with the collection modules deployed on all collected devices to obtain system resource data of all collected devices.

[0059] In scenarios with many collection tasks, such as Figure 3 In the data center B, multiple lower-level monitoring modules can be set, and each lower-level monitoring module is connected to at least one acquisition module deployed by the collected device. In this way, each lower-level monitoring module obtains the system resource data collected by the acquisition module connected to it, and all lower-level monitoring modules can obtain the system resource data of all collected devices in the data center B.

[0060] Among them, in the scenario where the data acquisition system is used to implement multiple acquisition tasks, the communication connection relationship between the lower-level monitoring module and the acquisition module can be determined based on the acquisition task. Specifically, the acquisition task corresponding to each acquired device is determined, and the acquisition module deployed by the acquired device is connected to the lower-level monitoring module corresponding to the acquisition task. In this way, each lower-level monitoring module can be responsible for one acquisition task, thereby realizing the functional partitioning of the task level. Exemplarily, a lower-level monitoring module P12 is responsible for collecting infrastructure monitoring indicator data, and another lower-level monitoring module P13 is responsible for collecting application monitoring indicator data. In this way, if a acquired device S04 corresponds to an infrastructure monitoring indicator data acquisition task, a communication connection can be established between the acquisition module deployed by S04 and the lower-level monitoring module P12. At this time, the acquisition module deployed by S04 executes the infrastructure monitoring indicator data acquisition task, collects the infrastructure monitoring indicator data, and the lower-level monitoring module P12 obtains the infrastructure monitoring indicator data.

[0061] like Figure 3 In the data center C, each collected device can correspond to the first number of collection tasks. At this time, the collection module deployed by the collected device is simultaneously connected to the first number of lower-level monitoring modules, and each lower-level monitoring module is responsible for one collection task. Exemplarily, a lower-level monitoring module P14 is responsible for collecting infrastructure monitoring indicator data, and another lower-level monitoring module P15 is responsible for collecting application monitoring indicator data. In this way, if a collected device S09 corresponds to both the infrastructure monitoring indicator data collection task and the application monitoring indicator data collection task, a communication connection can be established between the collection module deployed by S09 and the lower-level monitoring modules P14 and P15, respectively.

[0062] The embodiment of the present application, through the connection method between the aforementioned data center B and data center C, can utilize the collection module to execute different types of collection tasks, and divide the different types of collection tasks into different lower-level monitoring modules. Each lower-level monitoring module is only responsible for collecting the system resource data corresponding to one type of collection task, which is conducive to the targeted unification and archiving of the system resource data corresponding to different collection tasks.

[0063] The following describes the connection method between the collection module and the lower-level monitoring module in the scenario of multiple data centers and multiple collection tasks. Specifically, the data center where each collected device is located and its corresponding collection task can be determined, and in at least one lower-level monitoring module of the data center, the lower-level monitoring module corresponding to the collection task is determined, and the collection module deployed by the collected device is connected to this lower-level monitoring module. In this way, each lower-level monitoring module can be responsible for a collection task in a data center.

[0064] In addition, it can be seen that in data center AC, each data center deploys a separate lower-level monitoring module, and the acquisition modules deployed by the collected devices in each data center can be communicatively connected to the lower-level monitoring modules of the data center. That is, the acquisition modules and lower-level monitoring modules with communication connections can belong to the same data center, and each data center forms a federated cluster. In this way, the lower-level monitoring modules of each data center are only responsible for the system resource data of the collected devices in the data center, which can reduce network latency and bandwidth consumption, and reduce the complexity of the network architecture of the data collection system.

[0065] In other embodiments, the acquisition module and the lower-level monitoring module that are in communication connection may also belong to different data centers. For example, when the acquisition task volume of a single data center is small, the acquisition modules deployed by the acquisition devices in multiple data centers may share a lower-level monitoring module. In this way, multiple data centers form a federated cluster, which can reduce the number of lower-level monitoring modules, reduce the complexity of the network architecture of the data acquisition system, and save hardware resources.

[0066] The data acquisition system provided in the embodiment of the present application can divide different types of acquisition tasks into different lower-level monitoring modules, so that each lower-level monitoring module is only responsible for collecting one type of data, which is more conducive to preprocessing and archiving data corresponding to different task types. In addition, for the case of multiple data centers, the data acquisition system can deploy a lower-level monitoring module for each data center separately. The lower-level monitoring module summarizes the data of the data center and transmits it to the upper-level monitoring module, which can realize the aggregation of data from multiple data centers.

[0067] Secondly, combine Figure 3 , introduces the specific connection method between the lower-level monitoring module and the upper-level monitoring module.

[0068] When the amount of collection tasks is not large, such as Figure 3 In data center A, the lower-level monitoring modules can be directly connected to each upper-level monitoring module.

[0069] Furthermore, it can be understood that in the case of multiple data centers and the amount of data collection tasks of each data center is not large, the lower-level monitoring module deployed separately in each data center can be directly connected to the upper-level monitoring module. In this way, the upper-level monitoring module can be responsible for the data of all data centers to realize the federated cluster.

[0070] In the case of large collection tasks, such as Figure 3 In data center B and data center C, a middle-level monitoring module is deployed on the third server of each data center, and the system resource data is summarized and forwarded through the middle-level monitoring module. Specifically, the middle-level monitoring module can be set between the lower-level monitoring module and the upper-level monitoring module, the lower-level monitoring module is connected to the middle-level monitoring module for communication, and the middle-level monitoring module is connected to all upper-level monitoring modules for communication. In this way, the middle-level monitoring module can summarize the system resource data transmitted by the lower-level monitoring module and forward it to the upper-level monitoring module.

[0071] In this way, the collected system resource data can be summarized layer by layer through the middle-level monitoring module, reducing the workload of a single monitoring module. In addition, this divides the data collection system architecture into multiple independent layers, making it easier for each layer to expand independently to cope with different load requirements.

[0072] In the scenario where the data acquisition system is used to implement multiple acquisition tasks, the communication connection relationship between the lower-level monitoring module and the middle-level monitoring module can be determined based on the acquisition task. Specifically, each middle-level monitoring module can correspond to a second number of acquisition tasks, and each lower-level monitoring module can correspond to one acquisition task. Based on this, the acquisition task corresponding to each lower-level monitoring module can be determined, and the lower-level monitoring module is communicatively connected to the middle-level monitoring module corresponding to the acquisition task. In this way, each middle-level monitoring module can be responsible for several acquisition tasks to achieve functional partitioning at the task level. Exemplarily, a lower-level monitoring module P12 is responsible for collecting infrastructure monitoring indicator data, and another lower-level monitoring module P13 is responsible for collecting application monitoring indicator data. In this way, if a middle-level monitoring module P31 corresponds to the above two data acquisition tasks at the same time, then the middle-level monitoring module P31 is communicatively connected to the lower-level monitoring modules P12 and P13 at the same time. That is, the middle-level monitoring module is communicatively connected to the second number of lower-level monitoring modules, and the middle-level monitoring module corresponds to the same acquisition task as the second number of lower-level monitoring modules.

[0073] In this way, the collection tasks are divided into different middle-level monitoring modules based on the task type. Each middle-level monitoring module is responsible for collecting system resource data corresponding to several types of collection tasks, which facilitates archiving of system resource data collected by the collection modules when performing different types of collection tasks.

[0074] Furthermore, the middle-level monitoring module includes one or more levels, each level includes at least one sub-monitoring module, and is deployed in a third server (hereinafter referred to as the middle-level Prometheus Server). The number of levels is positively correlated with the amount of collection tasks or the complexity of tasks.

[0075] Specifically, the middle-level monitoring module includes a top level, several middle levels and a bottom level. The sub-monitoring modules in the top level are communicatively connected to all upper-level monitoring modules; the sub-monitoring modules in the middle level are communicatively connected to a sub-monitoring module in the adjacent high level and several sub-monitoring modules in the adjacent low level; and the sub-monitoring modules in the bottom level are communicatively connected to at least one lower-level monitoring module.

[0076] Figure 4 It is a structural diagram of a middle-level monitoring module provided in an embodiment of the present application.

[0077] like Figure 4 The middle-level monitoring module P31 shown includes three levels, among which the highest level includes a sub-monitoring module, denoted as P331; the middle level includes two sub-monitoring modules, denoted as P321 and P322 respectively; the lowest level includes four sub-monitoring modules, denoted as P311-P314. Among them, the highest level P331 is communicatively connected to the upper-level monitoring module, namely P21. The middle-level P321 is communicatively connected to a sub-monitoring module of the adjacent high-level, namely P331; the middle-level P321 is also communicatively connected to two sub-monitoring modules of the adjacent low-level, namely P311 and P312. The connection method of the middle-level P322 is similar to that of P321, which will not be repeated here. The lowest level P311 is communicatively connected to two lower-level monitoring modules, namely P11 and P12. The connection method of the lowest level P312-P314 is similar to that of P311, which will not be repeated here.

[0078] In addition, it can be seen that in data center AC, each data center deploys a separate middle-level monitoring module, and the lower-level monitoring modules in each data center can be connected to the middle-level monitoring module of the data center. That is, the lower-level monitoring modules and the middle-level monitoring modules that have communication connections belong to the same data center, and each data center forms a federated cluster. In this way, the middle-level monitoring module of each data center is only responsible for the system resource data collected by the lower-level monitoring modules in the data center, which can reduce network latency and bandwidth consumption, and reduce the complexity of the network architecture.

[0079] In other embodiments, the lower-layer monitoring modules and the middle-layer monitoring modules that are in communication connection may also belong to different data centers. For example, when the amount of collection tasks in a single data center is small, the lower-layer monitoring modules in multiple data centers may share a middle-layer monitoring module. In this way, multiple data centers form a federated cluster, which can reduce the number of middle-layer monitoring modules, reduce the complexity of the network architecture of the data collection system, and save hardware resources.

[0080] In these embodiments, the communication connection relationship between the lower-level monitoring module and the middle-level monitoring module can also be determined based on the collection task to achieve task-level functional partitioning. It is understandable that lower-level monitoring modules corresponding to the same collection task can exist simultaneously in multiple data centers, so the number of lower-level monitoring modules that the middle-level monitoring module is communicatively connected to may not be the second number. Figure 3 In the data center B, there is a lower-level monitoring module P12 responsible for collecting infrastructure monitoring indicator data, and another lower-level monitoring module P13 responsible for collecting application monitoring indicator data; in the data center C, there is a lower-level monitoring module P14 responsible for collecting infrastructure monitoring indicator data. In this way, in the scenario where the lower-level monitoring module and the middle-level monitoring module are connected based on the collection task, if a middle-level monitoring module corresponds to the above two data collection tasks at the same time, then the middle-level monitoring module can communicate with P12, P13 and P14 at the same time ( Figure 3 not shown).

[0081] The embodiment of the present application can set up multiple levels in the middle-level monitoring module to summarize the collected system resource data layer by layer, thereby reducing the workload of a single level and facilitating independent expansion of each level to cope with different load requirements.

[0082] It should be noted that in the aforementioned Figure 3 and Figure 4 In order to highlight the connection between the acquisition module, the lower monitoring module and the upper monitoring module, multiple upper monitoring modules are simplified into one and the storage module is omitted.

[0083] The following describes the connection method between the upper-level monitoring module and the storage module.

[0084] In some embodiments, the configuration file of each upper-level monitoring module includes a first storage address, which can be used to indicate the communication address of the storage device deployed by the storage module, so that the upper-level monitoring module can perform read and write operations on the storage device. Specifically, the configuration file includes a remote_write field and a remote_read field, and the url values ​​corresponding to the remote_write field and the remote_read field are respectively used to specify the http service address of remote read / write. The configuration file may also include a tls_concig field for enabling https.

[0085] In these embodiments, the upper-level monitoring module may determine the communication address of the storage device by reading the storage configuration information, and then send the system resource data to the storage device to achieve remote storage of the system resource data.

[0086] In other embodiments, the configuration file of each upper-layer monitoring module includes a second storage address, which can be used to indicate the communication address of the proxy server. The proxy server is arranged between the upper-layer monitoring module and the storage module, so that the upper-layer monitoring module can upload the system resource data to the proxy server, and the proxy server forwards it to the storage module. Specifically, the configuration file includes a proxy_url field, and the url value corresponding to the proxy_url field is used to specify the communication address of the proxy server.

[0087] In these embodiments, the upper-level monitoring module may send the system resource data indirectly to the storage device via the proxy module, thereby realizing remote storage of the system resource data.

[0088] In other embodiments, the configuration file of each upper monitoring module includes both the first storage address and the second storage address. Thus, when the upper monitoring module and the storage module cannot communicate directly, the upper monitoring module can upload the system resource data to the storage module via the proxy server.

[0089] In these embodiments, when the upper-layer monitoring module communicates directly with the storage device, the system resource data can be sent directly to the storage device; when the upper-layer monitoring module cannot communicate directly with the storage device, the system resource data can be sent indirectly to the storage device through the proxy module. In this way, it is ensured that the system resource data can always be stored in the remote storage.

[0090] It can be understood that in actual application scenarios, different upper-level monitoring modules can use different connection methods to achieve communication connection with the storage module.

[0091] The following describes the connection method between the data acquisition system and the user end. In some embodiments, each upper-layer monitoring module can be connected to a reverse proxy server such as Nginx Server. The reverse proxy server is connected to the user end to receive data query and other requests input by the user. In this way, the reverse proxy server can feedback the system resource data corresponding to the request information to the user based on the request information, and then complete the query and analysis of the system resource data, so that the user can have a deep understanding of the operating status of each collected device.

[0092] In other embodiments, the data collection system can be integrated with other tools and systems, such as Kubernetes, Docker containers, etc. In this way, the data collection system can collect the CPU, memory, network and disk usage of the container, which is conducive to identifying resource bottlenecks or abnormal behaviors of the container; it can also monitor the node status of the Kubernetes cluster to ensure the stable operation of the Kubernetes cluster.

[0093] Figure 5 This is a system architecture diagram of another data acquisition system provided in an embodiment of the present application.

[0094] like Figure 5 As shown, Figure 5 Three federated clusters A, B and C (marked by dotted boxes) are formed in the system. Each federated cluster includes several lower-level monitoring modules (and middle-level monitoring modules) for collecting host system data collected by the collection modules in each host. In federated cluster A, collection modules S1 and S2 are connected to the lower-level monitoring module P11 in communication, collection module S3 is connected to the lower-level monitoring module P12 in communication, and lower-level monitoring modules P11 and P12 are connected to upper-level monitoring modules P21 and P22 in communication through middle-level monitoring module P31. In federated cluster B, collection modules S4 and S5 are connected to the lower-level monitoring module P13 in communication, collection module S6 is connected to the lower-level monitoring module P14 in communication, and lower-level monitoring modules P13 and P14 are connected to upper-level monitoring modules P21 and P22 in communication through middle-level monitoring module P32. In federated cluster C, collection modules S7, S8 and S9 are connected to the lower-level monitoring module P15 in communication, and lower-level monitoring module P15 is directly connected to upper-level monitoring modules P21 and P22 in communication. There are two upper-level monitoring modules to achieve high availability (HA). The database is connected to the upper-level monitoring modules P21 and P22 to save the data in the upper-level monitoring modules and realize remote storage of system resource data. The reverse proxy server Nginx Server is connected to the upper-level monitoring module to receive the request information entered by the user (such as Figure 5 The data is received by the user through the Web request in the middle) and forwarded to the upper-level monitoring module, and then data query and data analysis are realized through the upper-level monitoring module.

[0095] The embodiment of the present application also provides a data acquisition method, which is applied to the data acquisition system in the aforementioned embodiment.

[0096] Combine the following Figures 6 to 8 , which illustrates the process of the data collection method.

[0097] Figure 6 It is a flow chart of the data collection method provided in the embodiment of the present application.

[0098] like Figure 6As shown, this embodiment includes the following steps S11-S15: S11: The collection module collects system resource data of the collected device.

[0099] In step S11, the collection module deployed in each collected device collects the system resource data of the host. The collection module may include a NodeExporter component. The NodeExporter component is a key component in the Prometheus ecosystem, responsible for collecting system and hardware information from the running host (collected device) and exposing this information in a format that Prometheus can understand, so that the lower-level monitoring module can read the data collected by the NodeExporter component.

[0100] Specifically, the NodeExporter component collects various system resource data from the collected devices, such as CPU usage, memory usage, disk space, network statistics, etc., by polling and exposing HTTP endpoints, and exposes this data in text format.

[0101] Among them, the NodeExporter component can unify the collected system resource data by modifying labels and indicator names during the data collection stage.

[0102] S12: The lower-level monitoring module obtains system resource data from the acquisition module.

[0103] In step S12, each lower-level monitoring module obtains system resource data from the acquisition module to which it is connected. The lower-level monitoring module may include a server-side component of the Prometheus ecosystem. Prometheus is an open source monitoring and alarm system, in which the Prometheus Server can pull data from components such as NodeExporter for storage, query and display. In an embodiment of the present application, the lower-level monitoring module can pull data from the acquisition module.

[0104] Among them, each lower-level monitoring module can periodically pull the data collected by the NodeExporter component.

[0105] S13: The upper-layer monitoring module obtains system resource data from the lower-layer monitoring module.

[0106] In step S13, each monitoring module includes an interface for obtaining data in the monitoring module. The upper-level monitoring module that is in communication connection with the lower-level monitoring module can pull the data in the lower-level monitoring module through the interface of the lower-level monitoring module. In an embodiment of the present application, the upper-level monitoring module can pull the system resource data obtained by the lower-level monitoring module through the interface of the lower-level monitoring module.

[0107] S14: The upper-layer monitoring module sends the system resource data to the storage module.

[0108] S15: The storage module saves the system resource data.

[0109] In steps S14-S15, the upper-layer monitoring module sends the system resource data to the storage module deployed in the storage device, so that the storage module stores the system resource data in the storage device. The storage device may adopt an object storage or a distributed file system.

[0110] In some embodiments, after step S15, the following step S16 may be further included: S16: The reverse proxy module responds to the request information input by the user and uses the upper-layer monitoring module to obtain system resource data corresponding to the request information in the upper-layer monitoring module or the storage module.

[0111] In step S16, the reverse proxy module responds to the query instruction input by the user, such as a statement based on the PromQL language, and queries the system resource data corresponding to the query instruction. Specifically, the user terminal sends the query instruction input by the user to the reverse proxy module, and the reverse proxy module responds to the query instruction, determines the upper-level monitoring module to be queried among multiple upper-level monitoring modules, and forwards the query instruction to the upper-level monitoring module to be queried. The upper-level monitoring module to be queried queries the data in the upper-level monitoring module to be queried based on the query instruction, or queries the system resource data in the storage device through the upper-level monitoring module to be queried.

[0112] When querying system resource data in a storage device, the upper-level monitoring module to be queried determines the storage device communication address for remotely reading data based on the url value corresponding to the remote_read field in the configuration file, and reads the data stored in the storage device corresponding to the communication address.

[0113] The data acquisition method provided in the embodiment of the present application is applied to the data acquisition system shown in the aforementioned embodiment. Therefore, the implementation method and beneficial effects of each step in this embodiment can be found in the aforementioned embodiment and will not be repeated here.

[0114] Figure 7 It is another flow chart of the data collection method provided in an embodiment of the present application.

[0115] like Figure 7 As shown, this embodiment includes the following steps S21-S29: S21: The collection module collects system resource data of the collected device.

[0116] S22: The lower-level monitoring module obtains system resource data from the acquisition module.

[0117] For detailed description of steps S21-S22, please refer to steps S11-S12, which will not be repeated here.

[0118] S23: Determine each upper-layer monitoring module as a target upper-layer monitoring module; or, the reverse proxy module adopts a polling method to determine a target upper-layer monitoring module from a plurality of upper-layer monitoring modules.

[0119] In step S23, each upper-layer monitoring module can be used as a target upper-layer monitoring module to pull the system monitoring data in the lower-layer monitoring modules. In this way, each upper-layer monitoring module obtains the system monitoring data in all lower-layer monitoring modules.

[0120] Alternatively, the reverse proxy module determines the current target upper-layer monitoring module among multiple upper-layer monitoring modules by polling. The target upper-layer monitoring module pulls data from the lower-layer monitoring module. If the target upper-layer monitoring module fails, the target upper-layer monitoring module is re-determined.

[0121] S24: The target upper-level monitoring module sends a first data pull request to the middle-level monitoring module that is communicatively connected to the upper-level monitoring module.

[0122] S25: The middle-level monitoring module sends a second data pulling request to the lower-level monitoring module that is communicatively connected to the middle-level monitoring module in response to the first data pulling request.

[0123] S26: After receiving the system resource data corresponding to the second data pull request sent by the lower-layer monitoring module, the middle-layer monitoring module sends the system resource data to the target upper-layer monitoring module.

[0124] In steps S24-S26, if the lower-layer monitoring module is connected to the target upper-layer monitoring module through the middle-layer monitoring module, the target upper-layer monitoring module can pull the system resource data in the lower-layer monitoring module through the middle-layer monitoring module.

[0125] Specifically, the target upper-layer monitoring module sends a first data pull request to the middle-layer monitoring module to which it is communicatively connected, requesting to pull system resource data from the middle-layer monitoring module. In response to the first data pull request, the middle-layer monitoring module sends a second data pull request to the lower-layer monitoring module to which it is communicatively connected, requesting to pull system resource data from the lower-layer monitoring module. In response to the second data pull request, the lower-layer monitoring module sends the system resource data to the middle-layer monitoring module through the interface. After receiving the system resource data, the middle-layer monitoring module sends the system resource data to the target upper-layer monitoring module through the interface. In this way, layer-by-layer pulling of system resource data can be achieved.

[0126] S27: The upper-layer monitoring module accesses the storage device corresponding to the first storage address based on the first storage address in the configuration information, and sends the system resource data to the storage module deployed in the storage device.

[0127] In step S27, a first storage address is pre-configured in the configuration information of the upper-level monitoring module, and the upper-level monitoring module can send the system resource data to the storage device corresponding to the first storage address to implement storage of the system resource data.

[0128] Exemplarily, the upper-level monitoring module may determine a database address for remote storage based on a url value corresponding to a remote_write field in a configuration file, and write the system resource data into a remote database corresponding to the database address.

[0129] S28: The storage module saves the system resource data.

[0130] For a detailed description of step S28, please refer to step S15, which will not be repeated here.

[0131] In some embodiments, after step S28, the following step S29 may be further included: S29: The reverse proxy module responds to the request information input by the user and uses the upper-layer monitoring module to obtain system resource data corresponding to the request information in the upper-layer monitoring module or the storage module.

[0132] For a detailed description of step S29, please refer to step S16, which will not be repeated here.

[0133] The data acquisition method provided in the embodiment of the present application is applied to the data acquisition system shown in the aforementioned embodiment. Therefore, the implementation method and beneficial effects of each step in this embodiment can be found in the aforementioned embodiment and will not be repeated here.

[0134] Figure 8 This is another flow chart of the data collection method provided in the embodiment of the present application.

[0135] like Figure 8 As shown, this embodiment includes the following steps S31-S37: S31: The collection module collects system resource data of the collected device.

[0136] S32: The lower-level monitoring module obtains system resource data from the acquisition module.

[0137] S33: The upper-layer monitoring module obtains system resource data from the lower-layer monitoring module.

[0138] For detailed description of steps S31-S33, please refer to steps S11-S13, which will not be repeated here.

[0139] S34: The upper-layer monitoring module accesses the proxy server corresponding to the second storage address based on the second storage address in the configuration information, and sends the system resource data to the proxy module deployed in the proxy server.

[0140] S35: The proxy module accesses the storage device corresponding to the first storage address based on the first storage address in the proxy configuration information, and sends the system resource data to the storage module deployed in the storage device.

[0141] In steps S34-S35, if the upper-layer monitoring module cannot directly access the storage module, the forwarding of the system resource data can be achieved through the proxy server. Specifically, the configuration information of the upper-layer monitoring module is pre-configured with a second storage address, and the upper-layer monitoring module can send the system resource data to the proxy server corresponding to the second storage address. The proxy configuration information of the proxy server is pre-configured with a first storage address, so the proxy server can forward the system resource data to the storage module deployed by the storage device corresponding to the first storage address to achieve storage of the system resource data.

[0142] Exemplarily, if the upper monitoring module cannot access the database for remote storage, the proxy server is determined based on the url value corresponding to the proxy_url field in the configuration file, and the data is transmitted to the proxy server. The proxy server transmits the received data to the remote database.

[0143] S36: The storage module saves the system resource data.

[0144] For a detailed description of step S36, please refer to step S15, which will not be repeated here.

[0145] In some embodiments, after step S36, the following step S37 may be further included: S37: The reverse proxy module responds to the request information input by the user and uses the upper-layer monitoring module to obtain system resource data corresponding to the request information in the upper-layer monitoring module or the storage module.

[0146] For a detailed description of step S37, please refer to step S16, which will not be repeated here.

[0147] The data acquisition method provided in the embodiment of the present application is applied to the data acquisition system shown in the aforementioned embodiment. Therefore, the implementation method and beneficial effects of each step in this embodiment can be found in the aforementioned embodiment and will not be repeated here.

[0148] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0149] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0150] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0151] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0152] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0153] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A data acquisition system, characterized in that: The data acquisition system comprises: A plurality of collection modules are respectively deployed in a plurality of collected devices, the plurality of collected devices are distributed in at least one data center, and each collection module is configured to collect system resource data of the collected device where it is deployed; A plurality of lower-layer monitoring modules are respectively deployed in a plurality of first servers, and the plurality of first servers are distributed in the at least one data center; each of the lower-layer monitoring modules is communicatively connected to at least one of the acquisition modules, and is configured to obtain the system resource data from the acquisition module communicatively connected thereto; A plurality of upper-layer monitoring modules are respectively deployed in a plurality of second servers; each of the upper-layer monitoring modules is respectively connected to all the lower-layer monitoring modules in communication, and is configured to obtain the system resource data from the lower-layer monitoring modules in communication with it; the plurality of first servers and the plurality of second servers form a federated cluster; A storage module is deployed in a storage device; the storage module is communicatively connected with each of the upper-level monitoring modules and is configured to store the system resource data acquired by the upper-level monitoring modules.

2. The system according to claim 1, characterized in that The acquisition module and the lower-layer monitoring module that are in communication connection belong to the same data center.

3. The system according to claim 1, characterized in that Each of the acquisition modules corresponds to a first number of acquisition tasks, and each of the lower-layer monitoring modules corresponds to one of the acquisition tasks; Each of the acquisition modules is in communication connection with the first number of the lower-layer monitoring modules, and the acquisition modules and the lower-layer monitoring modules in communication connection correspond to the same acquisition task; The lower-level monitoring module corresponding to the collection task is configured to obtain the system resource data obtained by the collection module executing the collection task on the collected device deployed therein.

4. The system according to claim 3, characterized in that The data acquisition system further comprises a plurality of middle-level monitoring modules, which are deployed in a plurality of third servers, and the plurality of third servers are distributed in the at least one data center; The middle-level monitoring module is arranged between each of the upper-level monitoring modules and all the lower-level monitoring modules; Among them, each of the middle-level monitoring modules is communicatively connected to all of the upper-level monitoring modules, and each of the middle-level monitoring modules is communicatively connected to at least one of the lower-level communication modules.

5. The system according to claim 4, characterized in that The lower-layer monitoring module and the middle-layer monitoring module that are in communication connection belong to the same data center.

6. The system according to claim 4 or 5, characterized in that: Each of the middle-level monitoring modules corresponds to the second number of the acquisition tasks, each of the middle-level monitoring modules is communicatively connected with the second number of the lower-level monitoring modules, and the middle-level monitoring modules and the lower-level monitoring modules that are communicatively connected correspond to the same acquisition task.

7. The system according to claim 4, characterized in that The middle-level monitoring module includes at least one level, each level includes at least one sub-monitoring module, and each sub-monitoring module is deployed in one of the third servers; At the highest level, each of the sub-monitoring modules is communicatively connected with all the upper-level monitoring modules; In the lowest level, each of the sub-monitoring modules is communicatively connected with at least one of the lower-level monitoring modules; In any other of the levels, each of the sub-monitoring modules is communicatively connected to a sub-monitoring module in an adjacent higher level, and each of the sub-monitoring modules is communicatively connected to at least one sub-monitoring module in an adjacent lower level.

8. The system according to claim 1, characterized in that The storage configuration information of the upper-level monitoring module includes a first storage address, and the first storage address is configured to indicate a communication address of the storage device.

9. The system according to claim 1, characterized in that The data acquisition system also includes an agent module, which is deployed in the agent server; The proxy module is communicatively connected to at least one of the upper-layer monitoring modules and the storage module, and is configured to forward the system resource data in the upper-layer monitoring module communicatively connected thereto to the storage module; The storage configuration information of the upper-layer monitoring module includes a second storage address, and the second storage address is configured to indicate a communication address of the proxy server.

10. The system according to claim 1, characterized in that The data collection system also includes a reverse proxy module, which is deployed in the reverse proxy server; The reverse proxy module is connected to all the upper-level monitoring modules, and is configured to respond to request information input by a user and obtain the system resource data corresponding to the request information through at least one of the upper-level monitoring modules connected to the reverse proxy module.

11. A data collection method, characterized in that: The method is applied to the data acquisition system according to any one of claims 1 to 10, and the method comprises: The collection module collects system resource data of the collected equipment; The lower-layer monitoring module obtains the system resource data from the acquisition module; The upper layer monitoring module obtains the system resource data from the lower layer monitoring module; The upper layer monitoring module sends the system resource data to the storage module; The storage module stores the system resource data.

12. The method according to claim 11, characterized in that The upper-layer monitoring module obtains the system resource data from the lower-layer monitoring module, including: The upper-layer monitoring module sends a first data pull request to a middle-layer monitoring module that is communicatively connected to the upper-layer monitoring module; The middle-level monitoring module sends a second data pulling request to the lower-level monitoring module that is communicatively connected to the middle-level monitoring module in response to the first data pulling request; After receiving the system resource data corresponding to the second data pull request sent by the lower-layer monitoring module, the middle-layer monitoring module sends the system resource data to the upper-layer monitoring module.

13. The method according to claim 11 or 12, characterized in that: The upper-layer monitoring module obtains the system resource data from the lower-layer monitoring module, including: Each of the upper-layer monitoring modules obtains the system resource data from at least one of the lower-layer monitoring modules; or, The reverse proxy module adopts a polling method to determine a target upper-layer monitoring module among the plurality of upper-layer monitoring modules, and the target upper-layer monitoring module obtains the system resource data from at least one of the lower-layer monitoring modules.

14. The method according to claim 11, characterized in that The upper layer monitoring module sends the system resource data to the storage module, including: The upper-layer monitoring module accesses the storage device corresponding to the first storage address based on the first storage address in the configuration information, and sends the system resource data to the storage module deployed in the storage device.

15. The method according to claim 11, characterized in that The upper layer monitoring module sends the system resource data to the storage module, including: The upper-layer monitoring module accesses the proxy server corresponding to the second storage address based on the second storage address in the configuration information, and sends the system resource data to the proxy module deployed in the proxy server; The proxy module accesses a storage device corresponding to the first storage address based on the first storage address in the proxy configuration information, and sends the system resource data to the storage module deployed in the storage device.

16. The method according to claim 11, characterized in that The method further comprises: The reverse proxy module responds to the request information input by the user and uses the upper-layer monitoring module to obtain the system resource data corresponding to the request information in the upper-layer monitoring module or the storage module.

Images