Lightweight resource information acquisition system of cloud platform

Abstract

The invention belongs to the technical field of cloud computing, and particularly relates to a lightweight resource information acquisition system of a cloud platform, which comprises a resource acquisition module, a resource cache module and a network server module, the network server is connected with a working node and a client of a cloud platform through network interfaces, the resource acquisition module and the resource cache module are connected with the network server through function interfaces, and the resource cache module is connected with the resource acquisition module through a function interface; the invention provides an efficient and lightweight resource acquisition system based on a cloud platform, which is helpful for enterprises to master real-time states of cloud resources, helps operation and maintenance personnel to discover and solve problems in time, and improves the reliability of cloud services and user experience.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud computing, and in particular relates to a lightweight resource information collection system for a cloud platform. Background Art

[0002] With the rapid development and widespread application of cloud computing technology, enterprises and organizations have more and more cloud resources and services. How to efficiently collect, monitor and manage these resources has become a key issue. Cloud computing resource monitoring technology has emerged, aiming to collect information on various resources in the cloud environment through automated means, and to summarize and analyze them in a unified manner. The resource collection methods in general fields often have problems such as data redundancy, high collection delay, low efficiency and high resource consumption, which are difficult to meet the needs of enterprises for real-time and accurate monitoring of cloud resources. Summary of the invention

[0003] In view of this, the present invention provides a lightweight resource information collection system for a cloud platform, which is an efficient and lightweight resource collection system based on a cloud platform, which helps enterprises to grasp the real-time status of cloud resources, helps operation and maintenance personnel to discover and solve problems in time, and improves the reliability of cloud services and user experience.

[0004] The technical solution of the present invention is:

[0005] A lightweight resource information collection system for a cloud platform, comprising a resource collection module, a resource cache module and a network server module; the network server is connected to a working node and a client of the cloud platform via a network interface, the resource collection module and the resource cache module are connected to the network server via a function interface, and the resource cache module is connected to the resource collection module via a function interface;

[0006] The acquisition system is integrated in the management node of the cloud platform, and the acquisition system communicates with the management node of the cloud platform based on function calls;

[0007] The acquisition system uses a unified data analysis and data conversion method to process the multi-dimensional resource data of the cloud platform so that the resource data meets the cloud native resource data standard.

[0008] Furthermore, the network server module is used to initialize and start the resource collection module and the resource storage module, periodically perform collection and storage operations, and provide a network interface for monitoring and querying resource information.

[0009] Furthermore, the resource acquisition module includes a data capture submodule and a data analysis submodule. The data capture submodule calls the resource monitoring interface provided by the network server module to actively capture resource data from each node of the cloud platform through Http Pull; the data analysis submodule summarizes and analyzes all the captured data and eliminates invalid data.

[0010] Furthermore, the resource storage module includes a format conversion submodule and a data cache submodule. The format conversion submodule formats and converts the data summarized by the resource acquisition module according to the resource type; the data cache submodule stores the format-converted data in the memory in chronological order, and provides the network server with corresponding data from the data cache submodule through a function interface.

[0011] Furthermore, the steps of collecting monitoring data by the collection system are as follows:

[0012] Step 1: Initialize the global configuration parameters of the acquisition system;

[0013] Step 2: starting the acquisition system;

[0014] Step 3: The data capture submodule obtains information of all nodes on the cloud platform and performs data capture;

[0015] Step 4: The data parsing submodule parses the resource data of all nodes;

[0016] Step 5: The format conversion submodule converts the extracted resource data into structured data, and the data storage submodule stores the converted and parsed structured data into the cache according to the resource type;

[0017] Wherein: when a query request is sent to the acquisition system, the network server module returns the latest data in the cache to the client.

[0018] Furthermore, the acquisition system supports operation across Linux and real-time operating systems: the acquisition system is written in C++ language, and non-critical resource processing logic is deleted.

[0019] Furthermore, the acquisition system uses a memory pool strategy to manage the memory requirements of tasks.

[0020] Beneficial effects of the present invention:

[0021] The present invention integrates the system with the cloud computing platform, and uses the powerful computing power of the cloud platform to achieve real-time monitoring, rapid aggregation and analysis of resource data; uses containerized deployment to achieve flexible migration and scheduling; uniformly classifies different data according to resource types, and only monitors the key indicators of the cloud platform, reducing the processing flow of redundant data, thereby reducing the consumption of hardware resources; the system uses cloud-native standard resource definitions and network interfaces, and has better portability and availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 It is a system architecture diagram of the present invention.

[0024] Figure 2 The present invention is a system operation flow chart.

[0025] Figure 3 This is a data flow diagram between data attributes and modules involved in the present invention. DETAILED DESCRIPTION

[0026] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0027] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0028] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0029] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0030] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0031] In one embodiment of the present invention, a lightweight resource information collection system for a cloud platform is proposed, comprising a resource collection module, a resource cache module and a network server module; the network server is connected to the working node and the client of the cloud platform through a network interface, the resource collection module and the resource cache module are connected to the network server through a function interface, and the resource cache module is connected to the resource collection module through a function interface;

[0032] The acquisition system is integrated in the management node of the cloud platform, and the acquisition system communicates with the management node of the cloud platform based on function calls;

[0033] The acquisition system uses a unified data analysis and data conversion method to process the multi-dimensional resource data of the cloud platform so that the resource data meets the cloud native resource data standard.

[0034] In this embodiment, the network server module is used to initialize and start the resource collection module and the resource storage module, periodically perform collection and storage operations, and provide a network interface for monitoring and querying resource information.

[0035] In this embodiment, the resource acquisition module includes a data capture submodule and a data analysis submodule. The data capture submodule calls the resource monitoring interface provided by the network server module, and actively captures resource data from each node of the cloud platform through HttpPull; the data analysis submodule summarizes and analyzes all the captured data and eliminates invalid data.

[0036] In this embodiment, the resource storage module includes a format conversion submodule and a data cache submodule. The format conversion submodule formats and converts the data summarized by the resource acquisition module according to the resource type; the data cache submodule stores the format-converted data in the memory in chronological order, and provides the network server with corresponding data from the data cache submodule through a function interface.

[0037] In this embodiment, the steps of collecting monitoring data by the collection system are as follows:

[0038] Step 1: Initialize the global configuration parameters of the acquisition system;

[0039] Step 2: starting the acquisition system;

[0040] Step 3: The data capture submodule obtains information of all nodes on the cloud platform and performs data capture;

[0041] Step 4: The data parsing submodule parses the resource data of all nodes; Step 5: The format conversion submodule converts the extracted resource data into structured data, and the data storage submodule stores the converted and parsed structured data into the cache according to the resource type;

[0042] Wherein: when a query request is sent to the acquisition system, the network server module returns the latest data in the cache to the client.

[0043] In this embodiment, the acquisition system supports operation across Linux and real-time operating systems: the acquisition system is written in C++ language, and non-critical resource processing logic is deleted.

[0044] In this embodiment, the acquisition system uses a memory pool strategy to manage the memory requirements of tasks.

[0045] This embodiment is deployed on the cloud platform in the form of containers. The architecture of this embodiment is as follows Figure 1As shown in the figure, the lightweight resource information collection system of the cloud platform mainly includes a network server, a resource collection module and a resource cache module. In cloud computing, the network server module is used to initialize and start the resource collection and storage module, periodically perform collection and storage operations, and is responsible for providing a network interface for monitoring and querying resource information. The resource collection module is divided into data capture and data analysis submodules. The data capture module calls the resource monitoring interface provided by the network server to actively capture resource data from the port exposed by the proxy of each node; the data analysis module is responsible for summarizing and analyzing all captured data and eliminating invalid data. The resource storage module is divided into two submodules: format conversion and data cache. The format conversion module formats and stores the aggregated data according to the resource type, and the data cache stores the data in memory in chronological order, providing the network server with an interface for obtaining corresponding data from the memory by resource type.

[0046] Http Pull (active crawling)

[0047] The execution process of the present invention is as follows Figure 2 As shown, the specific steps are as follows:

[0048] Step 1: The network interface server first applies for a large continuous space as a memory pool, and then initializes the global configuration parameters of the resource information collection system. Users can customize the parameters through the command line when the system starts. If there is no configuration, the default value is used. The startup parameters include: IP address resolver, node collection port and, communication mode selection, collection operation time interval and running options;

[0049] Step 2: Start the system and run it as a thread, with one acquisition and storage operation as a cycle. During operation, the system finds a suitable size from the free blocks already allocated in the memory pool and allocates it to the program. When the program has finished using the memory block, it returns it to the memory pool for reuse. The system first performs the acquisition operation. If the acquisition operation fails, the system reports an error and discards the currently collected data. If the acquisition operation is successful, the storage operation is performed to store the data in the memory according to type. Each execution of the cycle is separated by a period of time, which can be entered by the user to prevent network congestion;

[0050] Step 3: Call the resource monitoring interface provided by the network server to obtain the information of all nodes, including address, port and status, and then traverse the monitored node information list to extract resource data one by one;

[0051] Step 4: When extracting resource data, if the running status of the node is abnormal, an empty data set is returned. If the running status is normal, the collected raw data is extracted into structured data. The data attributes are as follows: Figure 3 As shown in the resource collection section, the specific extraction process is as follows:

[0052] 1. Read the name of the node, store the structure of the node's resource information installation indicator data point, and store it in the map format as the node's data set;

[0053] 2. Read the name of the Pod and the namespace name to which it belongs, traverse all containers under the Pod, extract the container name and corresponding resource information, and verify the validity of the data in the container. If there is data with a startup time or total memory of zero, clear all container data under this Pod and store it in a map format as a container data set;

[0054] 3. Read the name and ID information of the component, extract the health status data returned by the component, and store it in a map format as the component data set.

[0055] Step 5: Store the converted structured data in the cache according to the resource type. If the data in the cache involves the calculation of accumulated resource data, the first collected data needs to be added as a calculation reference. The stored data attributes are as follows: Figure 3 As shown in the resource storage section, the specific storage process is as follows:

[0056] 1. Traverse all resource information collected this time. If resource information of the same node is collected multiple times due to network problems, skip the duplicate information and only use the earliest information obtained;

[0057] 2. Use this data to replace the latest data;

[0058] 3. If the timestamp of this collection operation is smaller than the first timestamp due to reasons such as machine restart, check the latest collection time in the cache. If the latest collection time is during the running of the container, use the latest data in the cache to replace the first data.

[0059] 4. If the timestamp of this collection operation is greater than the timestamp of the first collection operation, the data collected for the first time will continue to be retained;

[0060] 5. Write new data to the cache.

[0061] Step 6: When the network server receives a request to query a resource, it first parses and extracts the requested data, and the collection system returns the latest data in the cache to the client according to the resource type or name;

[0062] Step 7: The database and display tools obtain resource data with timestamps through the query interface provided by the network server, and store or display the resource data in time series, thereby helping cloud platform operation and maintenance personnel to better understand the real-time operation status of the platform and discover faults in a timely manner.

[0063] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A lightweight resource information collection system for a cloud platform, characterized in that: It includes a resource collection module, a resource cache module and a network server module; the network server is connected to the working node and the client of the cloud platform through a network interface, the resource collection module and the resource cache module are connected to the network server through a function interface, and the resource cache module is connected to the resource collection module through a function interface; The acquisition system is integrated in the management node of the cloud platform, and the acquisition system communicates with the management node of the cloud platform based on function calls; The acquisition system uses a unified data analysis and data conversion method to process the multi-dimensional resource data of the cloud platform so that the resource data meets the cloud native resource data standard.

2. The lightweight resource information collection system for a cloud platform according to claim 1, characterized in that: The network server module is used to initialize and start the resource acquisition module and the resource storage module, periodically perform acquisition and storage operations, and provide a network interface for monitoring and querying resource information.

3. The lightweight resource information collection system for cloud platform according to claim 2 is characterized in that: The resource acquisition module includes a data capture submodule and a data analysis submodule. The data capture submodule calls the resource monitoring interface provided by the network server module to actively capture resource data from each node of the cloud platform through Http Pull; the data analysis submodule aggregates and analyzes all captured data and eliminates invalid data.

4. The lightweight resource information collection system for cloud platform according to claim 3 is characterized in that: The resource storage module includes a format conversion submodule and a data cache submodule, and the format conversion submodule performs format conversion on the data collected by the resource acquisition module according to the resource type; The data cache submodule stores the format-converted data in a time sequence in the memory, and provides the network server with a function interface to obtain the corresponding data from the data cache submodule.

5. The lightweight resource information collection system for cloud platform according to claim 4 is characterized in that: The steps of collecting monitoring data by the collection system are as follows: Step 1: Initialize the global configuration parameters of the acquisition system; Step 2: starting the acquisition system; Step 3: The data capture submodule obtains information of all nodes on the cloud platform and performs data capture; Step 4: The data parsing submodule parses the resource data of all nodes; Step 5: The format conversion submodule converts the extracted resource data into structured data, and the data storage submodule stores the converted and parsed structured data into the cache according to the resource type; Wherein: when a query request is sent to the acquisition system, the network server module returns the latest data in the cache to the client.

6. The lightweight resource information collection system for cloud platform according to claim 5, characterized in that: The acquisition system supports operation across Linux and real-time operating systems: the acquisition system is written in C++ language, and non-critical resource processing logic is deleted.

7. The lightweight resource information collection system for a cloud platform according to claim 6, characterized in that: The acquisition system uses a memory pool strategy to manage the memory requirements of tasks.

Images