Method and equipment for realizing mixed multi-data-source monitoring data acquisition based on double queues and medium

Abstract

The invention discloses a method and equipment for realizing mixed multi-data source monitoring data acquisition based on double queues and a medium, a double-queue method is adopted when a multi-cloud management platform performs data acquisition on resources, the resources are always acquired by one queue, and the resources are transferred by the other queue, so that the resource acquisition efficiency is improved. The two queues form a pair of cloud resource storage queues, and alternately execute resource acquisition and resource transfer tasks; the multi-cloud management platform automatically initializes a pair of cloud resource storage queues for each data source according to the configuration of the data sources to be subjected to data acquisition, and configures a thread pool for each pair of cloud resource storage queues; a thread pool for data acquisition of each data source is independent, and each cloud resource storage queue is independent; and the multi-cloud management platform automatically adjusts the periodic acquisition time for each data source according to the current configuration parameters. According to the scheme of the invention, the multi-cloud management platform has high performance and convenient expansibility when carrying out data acquisition on resources.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing, and in particular to a method, device and medium for implementing hybrid multi-data source monitoring data collection based on dual queues. Background Art

[0002] As more and more business loads are migrated to the cloud, enterprises are using more and more public and private clouds. Since cloud platforms come from different vendors, they need to be managed separately, resulting in low delivery efficiency and high operation and maintenance costs. Enterprises urgently need a multi-cloud management platform (CMP for short) for unified management. Providing cloud resource management capabilities and operation and maintenance efficiency. Implementing a unified monitoring and alarm module is one of the core functional requirements of CMP. However, CMP implements a large number of managed resources, which is not only the IaaS resources such as hosts, virtual machines, and containers provided by various cloud platforms. It also includes PaaS resources deployed on the cloud. The data source for monitoring and alarm collection is not only from the cloud platform, but also from the installed open source basic components. The monitoring overview data collection is the first step in displaying the monitoring and alarm function module.

[0003] The principles of traditional cloud platform monitoring and alarm technology solutions are as follows: Figure 1 As shown, the technical principle implementation steps are as follows:

[0004] (1) After the service is started, build an initialization scheduling thread pool.

[0005] (2) Execute the collection tasks of each cloud platform at a certain periodic frequency. The collection tasks of different cloud platforms have different periods.

[0006] (3) The collection task processing logic of each cloud platform is to read the list data of resources (such as hosts or virtual machines) managed by each cloud platform, call the monitoring interface provided by the cloud vendor, and send a request.

[0007] (4) When sending monitoring data requests to a host or virtual machine, the collection task can send them one by one in a loop or in batches.

[0008] (5) After receiving the monitoring data response from the cloud vendor, each collection task converts it into a unified data storage format and performs persistence processing for display and use by the monitoring module.

[0009] The original technical principle still has these shortcomings: According to the user's operation and maintenance requirements, the CMP platform needs to monitor all resources under each cloud platform and unify the operation and maintenance, including IaaS layer resources and PaaS layer resources. These resources can only be processed by a single machine. The resource data is first loaded into the memory. The large amount of data is prone to content overflow. Even in the case of cluster deployment, data allocation cannot be performed, resulting in resource waste; resource collection tasks of different cloud platforms share the same thread pool. When there is a problem with the collection task of a certain cloud platform, it will affect the monitoring data collection of other cloud platforms; data collection of a single cloud platform, whether it is batch data collection in a cyclic manner or single data sending collection in a cyclic manner, is still relatively inefficient when a large number of resources need to be collected, which will cause the interval scheduling time to often time out. Summary of the invention

[0010] The purpose of the present invention is to provide a method, device and medium for realizing hybrid multi-data source monitoring data collection based on dual queues.

[0011] The hybrid multi-data source monitoring data collection method based on dual queues of the present invention can solve the problem that when there are a large number of cloud resources on the multi-cloud management platform for monitoring data collection, the data is first loaded into the memory, and the memory often overflows; solve the problem that when collection scheduling occurs, the system cannot fully utilize the existing machine resources, especially in the case of many data sources, the collection efficiency is not high, and the multi-cloud management platform cannot flexibly expand the platform as more and more data sources are connected; solve the problem that the data source monitoring and collection programs of each cloud platform share a thread pool, and when the interface collection performance of a certain cloud platform is relatively low, it will affect the data source monitoring and collection of other cloud platforms; and solve the problem that the data sources of each cloud platform cannot automatically adjust the monitoring and collection scheduling cycle according to the current configuration parameters.

[0012] To achieve the above purpose, the technical solution provided by the present invention is:

[0013] The first aspect of the present application provides a method for collecting monitoring data from hybrid multiple data sources based on dual queues, including:

[0014] The multi-cloud management platform uses a dual-queue method when collecting data on resources. One queue is always used to collect resources, and the other queue is used to transfer resources. The two queues form a pair of cloud resource storage queues, which perform resource collection and resource transfer tasks alternately.

[0015] The multi-cloud management platform automatically initializes a pair of cloud resource storage queues for each data source according to the configuration of the data source to be collected, and equips each pair of cloud resource storage queues with a thread pool; the thread pool for data collection of each data source is independent, and each cloud resource storage queue is independent;

[0016] The multi-cloud management platform automatically adjusts the periodic collection time for each data source based on the current configuration parameters.

[0017] The present invention provides a method for collecting monitoring data from multiple data sources based on dual queues, which specifically includes the following steps:

[0018] Step S1: Fill in the cloud platform collection environment configuration form items on the multi-cloud management platform and save them in the database;

[0019] Step S2: Perform initialization operations to load cloud resource data and basic configuration data into the cache in preparation for monitoring data collection;

[0020] Step S3: monitoring data collection and processing logic;

[0021] Step S4: The collected indicator data needs to be converted into a unified format of the database and stored in the database;

[0022] Step S5: A separate database and separate tables design is adopted for storage. Different cloud platforms establish different databases, and different cloud resource types establish different database tables.

[0023] To optimize the above technical solutions, the specific measures taken also include:

[0024] The cloud platform acquisition environment configuration form items are filled in on the multi-cloud management platform and saved in the database, where:

[0025] The cloud platform collection environment configuration form items include cloud platform configuration form items, monitoring data source configuration form items, cloud resource type configuration form items and cloud monitoring indicator configuration form items; the cloud platform collection environment configuration form items are monitored by monitoring the relevant parameters configured during the collection operation;

[0026] The cloud platform configuration form item is used to obtain the interface information of the cloud resources on each cloud platform, including: cloud platform name, cloud platform connection address, cloud platform code, interface user name, interface password;

[0027] The monitoring data source configuration form item is used to obtain cloud resource monitoring data, including: data source address, data source port, data source user name, and data source password;

[0028] The cloud resource type configuration form item is used to obtain cloud resource information, including: cloud resource name, cloud resource identifier, and cloud resource type;

[0029] The cloud monitoring indicator configuration form item is used to record indicator data of each cloud resource type, including: indicator code, indicator name, and cloud resource type code.

[0030] Furthermore, based on the collection monitoring of the cloud platform collection environment configuration form items, monitoring runtime configuration is performed, including cloud resource delivery parameter configuration, cloud monitoring thread pool configuration and cloud resource concurrent collection parameter configuration, wherein:

[0031] Obtain the number of cloud resources in the cloud resource queue by issuing parameter configurations from cloud resources;

[0032] Through the cloud monitoring thread pool configuration, configure the relevant parameters in each cloud thread pool, including: minimum thread pool, maximum thread pool, thread queue size and thread survival time;

[0033] Configure the number of concurrent threads for data collection by configuring the cloud resource concurrent collection parameters.

[0034] Furthermore, the initialization operation is performed to load the cloud resource data and basic configuration data into the cache in preparation for monitoring data collection, including:

[0035] After the service is started, it performs initialization operations and establishes a cloud resource storage queue with two queues for each data source on the distributed cache: one is an outgoing resource queue, which is used as a data outgoing resource queue; the other is an incoming resource queue, which is used as a data ingoing queue;

[0036] Read the cloud resource data of each cloud from the database and put it into one of the two queues of the data source corresponding to the cloud. The data information includes the following: cloud platform name, cloud platform code, cloud resource name, cloud resource identifier, cloud resource type, and cloud resource type code;

[0037] The basic configuration data is maintained by the monitoring basic configuration, using the ER relationship model; the monitoring basic configuration items are read from the database to load the cache, including data source data and indicator data, to provide data support for the collection of monitoring data;

[0038] Initialize the scheduling start time, and register the current time to the configuration center after initialization is completed;

[0039] Monitor the changes of scheduling time through the scheduling center, and initialize the scheduling when there is a change in scheduling time;

[0040] Initialize the thread pool size of each data source according to the configuration parameters of the thread pool of each data source.

[0041] Furthermore, the monitoring data collection and processing logic includes:

[0042] The scheduling system performs scheduling according to the written data source scheduling time;

[0043] Read the execution count value of a data source from the configuration center. The default initial value is 0. Divide this value by 2, and the remainder is 0 or 1. When the remainder is 0, it means that the cloud resource queue with sequence number 0 is defined as the outgoing resource queue, and the cloud resource queue with sequence number 1 is the incoming resource queue. When the remainder is 1, it means that the cloud resource queue with sequence number 1 is defined as the outgoing resource queue, and the cloud resource queue with sequence number 0 is defined as the incoming resource queue.

[0044] If the actual pull quantity is less than or equal to 0, the scheduling ends;

[0045] If the number of pulled data is not 0, it is grouped according to the number of concurrent threads that have been configured;

[0046] Then establish concurrent threads according to the number of concurrent threads;

[0047] Each thread is assigned one of a pair of cloud resource storage queues for execution. During execution, the thread obtains data source data and indicator data from the cache according to the cloud platform code and cloud resource type code, issues a request, obtains the indicator data value, and then stores the indicator.

[0048] After the execution is completed, the resource data of this group is stored in another queue of a pair of cloud resource storage queues;

[0049] When all threads are executed, determine whether the end mark is finished;

[0050] When the status of all services in the configuration center is completed through the listening program, it means that all resource monitoring indicator data has been obtained and the scheduling is completed; the number of the configuration center needs to be increased by 1, and the time at this time needs to be changed to the time of the next scheduling of the scheduler, and the above collection steps are repeated to collect data indicators again.

[0051] Furthermore, the storage adopts a sub-library and sub-table design, different cloud platforms establish different databases, and different cloud resource types establish different database tables, including:

[0052] Each time a data source is scheduled for collection, the scheduling number of the current data source is read to determine which of the two queues of the cloud resource storage queue under the data source is the outgoing resource queue and which is the incoming resource queue;

[0053] The configuration center records the status of each service under each data source; when the scheduling of each service is completed, the status of the service is changed to completed, and when each service starts scheduling, the status of the service is changed to executing; when all services under a data source are completed, it indicates that the scheduling and collection task of the current data source is completed.

[0054] Furthermore, the multi-cloud management platform automatically adjusts the periodic collection time for each data source based on the current configuration parameters, specifically: automatically adjusts the periodic collection time for each data source based on the resource machine configuration, the number of resources issued, the grouping parameters, and the data source interface collection performance; when the data source scheduling collection task ends, the collection time of this scheduling is recorded, compared with the last collection time, and a relatively larger collection task time is taken to calculate the next scheduling time, and written into the configuration center to schedule the next collection task of the data source.

[0055] The second aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the hybrid multi-data source monitoring data collection method based on dual queues as described above is implemented.

[0056] A third aspect of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the above-mentioned method for implementing hybrid multi-data source monitoring data collection based on dual queues.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] The hybrid multi-data source monitoring data collection method based on dual queues of the present invention adopts a dual queue method when collecting data on resources on a multi-cloud management platform. One queue is always kept to collect resources, and the other queue is kept to transfer resources. The two queues are executed alternately, and the operation is completely based on memory, which has high performance.

[0059] In the present invention, the multi-cloud management platform automatically initializes a pair of external storage queues for each data source according to the data source configuration to be collected, and equips each pair of external storage queues with a thread pool. When performing data collection, the resource performance of each machine is fully utilized, and more data sources can be targeted, with convenient scalability.

[0060] The thread pools for collecting data from various data sources of the present invention are independent, and the cloud resource queues are independent. When the interface collection performance of a data source is relatively low, it will not affect the monitoring and collection of other cloud platforms.

[0061] The present invention automatically adjusts the periodic collection time for each data source based on current configuration parameters such as machine configuration, number of resources sent, grouping parameters, and data source interface collection performance. It will not cause data to fail to be collected due to artificially setting the collection time too short. The periodic scheduling collection time of the data source can also be optimized by adjusting the parameter configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 : Schematic diagram of traditional cloud platform monitoring and alarm technology solution.

[0063] Figure 2 : The overall principle diagram of the hybrid multi-data source monitoring data acquisition method based on dual queues of the present invention.

[0064] Figure 3 : A specific flow chart of the hybrid multi-data source monitoring data acquisition method based on dual queues of the present invention.

[0065] Figure 4 : The monitoring basic configuration of the present invention is an ER relationship model.

[0066] Figure 5 : Data source thread pool initialization flow chart.

[0067] Figure 6 : Data acquisition and processing logic monitoring process diagram. DETAILED DESCRIPTION

[0068] The above contents of the present invention are further described in detail below in the form of embodiments, but this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the above contents of the present invention belong to the scope of the present invention.

[0069] In one embodiment, the present invention proposes a method for collecting monitoring data from a hybrid multi-data source based on a dual queue, comprising:

[0070] The multi-cloud management platform uses a dual-queue method when collecting data on resources. One queue is always used to collect resources, and the other queue is used to transfer resources. The two queues form a pair of cloud resource storage queues, which perform resource collection and resource transfer tasks alternately.

[0071] The multi-cloud management platform automatically initializes a pair of cloud resource storage queues for each data source according to the configuration of the data source to be collected, and equips each pair of cloud resource storage queues with a thread pool; the thread pool for data collection of each data source is independent, and each cloud resource storage queue is independent;

[0072] The multi-cloud management platform automatically adjusts the periodic collection time for each data source based on the current configuration parameters.

[0073] The technical solution of the present invention is further described in detail below in conjunction with specific embodiments:

[0074] After building the cloud monitoring multi-data source collection basic environment configuration through the multi-cloud monitoring collection environment configuration module, such as Figure 2 As shown, the following process is performed:

[0075] Step S1: Fill in the cloud platform acquisition environment configuration form items on the CMP platform and save them in the database.

[0076] The contents of the cloud platform configuration form items include: cloud platform name, cloud platform connection address, cloud platform code, interface user name, interface password, etc. It is mainly used to obtain the interface of cloud resources on each cloud platform.

[0077] The monitoring data source configuration form items include: data source address, data source port, data source user name, data source password, etc. The data source can be a cloud platform or a monitoring plug-in, mainly to obtain cloud resource monitoring data.

[0078] The cloud resource type configuration form items include: cloud resource name, cloud resource identifier, cloud resource type; such as host, virtual machine, database, etc.

[0079] The cloud monitoring indicator configuration form items include: indicator code, indicator name, cloud resource type code; it is mainly used to record which indicator data is collected for each resource type, such as: the monitoring indicators of the resource type host include CPU, memory and other data.

[0080] Monitoring collection configuration is the core configuration of monitoring collection, and is mainly used to establish management relationships with other configurations.

[0081] The monitoring basic configuration is mainly to maintain the basic data collected by monitoring. The basic configuration ER relationship model is as follows Figure 4 .

[0082] Monitoring runtime configuration mainly refers to the relevant parameters configured during the collection and runtime of monitoring indicators.

[0083] Cloud resource delivery parameter configuration: the number of cloud resources obtained in the cloud resource queue.

[0084] Cloud monitoring thread pool configuration: related parameters of each cloud thread pool configuration, including: minimum thread pool, maximum thread pool, thread queue size, thread survival time, etc.

[0085] Cloud resource concurrent collection parameter configuration: configure how many concurrent threads are used for data collection.

[0086] The cloud resource monitoring and collection process module is the core business logic of monitoring and collection.

[0087] Step S2: Initialization mainly loads cloud resource data and basic configuration data into the cache to prepare for monitoring data collection.

[0088] After the service is started, it performs initialization operations and creates two queues for each data source on the distributed cache. The naming rule of the queues is "data source code-serial number". One is used as a data outbound resource queue and the other is used as a data inbound resource queue, such as Ali-0 or Ali-1.

[0089] The cloud resource data of each cloud is read from the database and put into the queue with sequence number 0 of the two queues corresponding to the data source of the cloud. The data information includes the following: cloud platform name, cloud platform code, cloud resource name, cloud resource identifier, cloud resource type, cloud resource type code, etc.

[0090] Read the basic monitoring configuration items from the database to load the cache, including data sources and monitoring indicator items.

[0091] The key of the data source data structure is: "data source code-cloud resource type code-datasource", and the value is the data source object information, including: address, password, port number. The key of the monitoring item indicator data structure is: "data source code-cloud resource type code-item", and the value is a collection of indicator objects. The indicator object information includes: indicator code, indicator name, and these data provide data support for the collection of monitoring data.

[0092] Initialize the scheduling start time. After initialization is completed, register the current time to the configuration center.

[0093] The dispatch center monitors the changes in the dispatch time and initializes the dispatch when there is a change in the dispatch time.

[0094] Initialize the thread pool size of each data source according to the configuration parameters of each data source thread pool. The entire initialization process is as follows: Figure 5 shown.

[0095] Step S3: Taking a data source collection as an example, the monitoring data collection processing logic is as follows Figure 6 shown.

[0096] The scheduling system performs scheduling according to the scheduling time of a certain data source written in.

[0097] Read the value of excute_times of the execution times of a data source from the configuration center. The default initial value is 0. Divide this value by 2, and the remainder is 0 or 1. When the remainder is 0, it means that the cloud resource queue with sequence number 0 is defined as the outgoing resource queue, and the cloud resource queue with sequence number 1 is the incoming resource queue. When the remainder is 1, it means that the cloud resource queue with sequence number 1 is defined as the outgoing resource queue, and the sequence number 0 is defined as the incoming resource queue. Because it is a cluster deployment service, each service needs to read the data of the size of the configured resource parameters from the outgoing resource queue. For example, Alibaba Cloud's outgoing resource queue data is 10,000, which is collected by deploying 4 services. Each service pulls data for collection according to the configuration parameter 1000 issued. After a cycle is executed, there are still 6,000 cloud resource data left in the queue to be collected.

[0098] But sometimes when each service pulls data, the actual number of pulls may be less than the number of parameters configured or the number of pulls is 0. When the number is less than, it means that the resource queue is not enough to pull the number of parameters configured, and when it is 0, it means that the data in the resource queue has been pulled. This means that the current scheduling of the service is about to end, and the data should not be pulled in a loop. Therefore, if the actual number of pulls is less than or equal to 0, the end flag End_Tag = true.

[0099] If the number of pulled data is not 0, it will be grouped according to the configured number of concurrent threads. If the number of cloud resources pulled by each service is 1000 and the number of concurrent threads is 8, it will be divided into 8 groups, with 125 in each group.

[0100] Then create concurrent threads according to the number of concurrent threads. For example, if the number of concurrent threads is 8, then 8 threads need to be created.

[0101] Each thread is assigned a group to execute. When executing, the data source (address, port, user name, password) and indicator data are obtained from the cache according to the cloud platform code and cloud resource type code, and a request is issued to obtain the indicator data value, and then the indicator is stored.

[0102] After the execution is completed, the resource data of this group needs to be stored in another queue, namely the resource queue.

[0103] When all threads are executed, it is determined whether the end tag is ended, that is, whether End_Tag is true. If it is true, the end status is sent to the configuration center. If it is not true, the next cycle continues to pull data.

[0104] When the monitoring program monitors that the status of all services in the configuration center has been completed, it means that all resource monitoring indicator data has been obtained and the scheduling is completed. You need to increase the number of excute_times in the configuration center by 1, and change the time at this time to the time of the next scheduling of the scheduler, repeat the above collection steps, and then collect data indicators.

[0105] Step S4: The collected indicator data needs to be converted into a unified format of the database and stored in the database.

[0106] Step S5: A sub-library and sub-table design is adopted for storage. Different cloud platforms establish different databases, and different cloud resource types establish different database tables.

[0107] The configuration center module mainly records the number of scheduling of each data source, the collection completion status of each service under each data source, and the scheduling execution time of each data source.

[0108] Each time a data source is scheduled for collection, the scheduling number of the current data source must be read to determine which of the two queues under the data source is the outgoing resource queue and which is the incoming resource queue.

[0109] The configuration center records the status of each service under each data source; when the scheduling of each service is completed, the status of the service is modified to be completed, and when the scheduling of each service starts, the status of the service is modified to be in execution; when all services under the data source are completed, it proves that the scheduling and collection task of the current data source has ended.

[0110] When the data source's scheduled collection task has ended, record the collection time of this schedule, compare it with the last collection time, take the larger collection task time, and calculate the next scheduled time at the same time, write it into the configuration center, and schedule the next collection task of the data source.

[0111] In another embodiment of the present invention, an electronic device is proposed, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the hybrid multi-data source monitoring data collection method based on dual queues as described above is implemented.

[0112] In another embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program, and the computer program enables a computer to execute the above-mentioned method for implementing hybrid multi-data source monitoring data collection based on dual queues.

[0113] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0114] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession, without departing from the scope of the technical solution of the present invention, according to the technical essence of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for collecting monitoring data from mixed multiple data sources based on dual queues, characterized in that: include: The multi-cloud management platform uses a dual-queue method when collecting data on resources. One queue is always used to collect resources, and the other queue is used to transfer resources. The two queues form a pair of cloud resource storage queues, which perform resource collection and resource transfer tasks alternately. The multi-cloud management platform automatically initializes a pair of cloud resource storage queues for each data source according to the configuration of the data source for data collection, and configures a thread pool for each pair of cloud resource storage queues; Each data source has an independent thread pool for data collection, and each cloud resource storage queue is independent; The multi-cloud management platform automatically adjusts the periodic collection time for each data source based on the current configuration parameters.

2. The method for monitoring data collection based on dual queues and hybrid multi-data sources according to claim 1 is characterized in that: The following steps are involved: Step S1: Fill in the cloud platform collection environment configuration form items on the multi-cloud management platform and save them in the database; Step S2: Perform initialization operations to load cloud resource data and basic configuration data into the cache in preparation for monitoring data collection; Step S3: monitoring data collection and processing logic; Step S4: The collected indicator data needs to be converted into a unified format of the database and stored in the database; Step S5: A separate database and separate tables design is adopted for storage. Different cloud platforms establish different databases, and different cloud resource types establish different database tables.

3. The method for monitoring data collection based on dual queues and hybrid multi-data sources according to claim 2 is characterized in that: The cloud platform acquisition environment configuration form items are filled in on the multi-cloud management platform and saved in the database, where: The cloud platform collection environment configuration form items include cloud platform configuration form items, monitoring data source configuration form items, cloud resource type configuration form items and cloud monitoring indicator configuration form items; the cloud platform collection environment configuration form items are monitored by monitoring the relevant parameters configured during the collection operation; The cloud platform configuration form item is used to obtain the interface information of the cloud resources on each cloud platform, including: cloud platform name, cloud platform connection address, cloud platform code, interface user name, interface password; The monitoring data source configuration form item is used to obtain cloud resource monitoring data, including: data source address, data source port, data source user name, and data source password; The cloud resource type configuration form item is used to obtain cloud resource information, including: cloud resource name, cloud resource identifier, and cloud resource type; The cloud monitoring indicator configuration form item is used to record indicator data of each cloud resource type, including: indicator code, indicator name, and cloud resource type code.

4. The method for monitoring data collection based on dual queues and hybrid multi-data sources according to claim 3 is characterized in that: Based on the collection monitoring of the cloud platform collection environment configuration form items, the monitoring runtime configuration is performed, including cloud resource delivery parameter configuration, cloud monitoring thread pool configuration and cloud resource concurrent collection parameter configuration, where: Obtain the number of cloud resources in the cloud resource queue by issuing parameter configurations from cloud resources; Through the cloud monitoring thread pool configuration, configure the relevant parameters in each cloud thread pool, including: minimum thread pool, maximum thread pool, thread queue size and thread survival time; Configure the number of concurrent threads for data collection by configuring the cloud resource concurrent collection parameters.

5. The method for collecting monitoring data from hybrid multiple data sources based on dual queues according to claim 2 is characterized in that: The initialization operation described above loads the cloud resource data and basic configuration data into the cache in preparation for monitoring data collection, including: After the service is started, it performs initialization operations and establishes a cloud resource storage queue with two queues for each data source on the distributed cache: one is an outgoing resource queue, which is used as a data outgoing resource queue; the other is an incoming resource queue, which is used as a data ingoing queue; Read the cloud resource data of each cloud from the database and put it into one of the two queues of the data source corresponding to the cloud. The data information includes the following: cloud platform name, cloud platform code, cloud resource name, cloud resource identifier, cloud resource type, and cloud resource type code; The basic configuration data is maintained by the monitoring basic configuration, using the ER relationship model; the monitoring basic configuration items are read from the database to load the cache, including data source data and indicator data, to provide data support for the collection of monitoring data; Initialize the scheduling start time, and register the current time to the configuration center after initialization is completed; Monitor the changes of scheduling time through the scheduling center, and initialize the scheduling when there is a change in scheduling time; Initialize the thread pool size of each data source according to the configuration parameters of the thread pool of each data source.

6. The method for monitoring data collection based on dual queues and hybrid multi-data sources according to claim 5 is characterized in that: The monitoring data collection and processing logic includes: The scheduling system performs scheduling according to the written data source scheduling time; Read the execution count value of a data source from the configuration center. The default initial value is 0. Divide this value by 2, and the remainder is 0 or 1. When the remainder is 0, it means that the cloud resource queue with sequence number 0 is defined as the outgoing resource queue, and the cloud resource queue with sequence number 1 is the incoming resource queue. When the remainder is 1, it means that the cloud resource queue with sequence number 1 is defined as the outgoing resource queue, and the cloud resource queue with sequence number 0 is defined as the incoming resource queue. If the actual pull quantity is less than or equal to 0, the scheduling ends; If the number of pulled data is not 0, it is grouped according to the number of concurrent threads that have been configured; Then establish concurrent threads according to the number of concurrent threads; Each thread is assigned one of a pair of cloud resource storage queues for execution. During execution, the thread obtains data source data and indicator data from the cache according to the cloud platform code and cloud resource type code, issues a request, obtains the indicator data value, and then stores the indicator. After the execution is completed, the resource data of this group is stored in another queue of a pair of cloud resource storage queues; When all threads are executed, determine whether the end mark is finished; When the status of all services in the configuration center is completed through the listening program, it means that all resource monitoring indicator data has been obtained and the scheduling is completed; the number of the configuration center needs to be increased by 1, and the time at this time needs to be changed to the time of the next scheduling of the scheduler, and the above collection steps are repeated to collect data indicators again.

7. The method for collecting monitoring data from hybrid multiple data sources based on dual queues according to claim 6 is characterized in that: The storage is designed with separate databases and tables. Different cloud platforms establish different databases, and different cloud resource types establish different database tables, including: Each time a data source is scheduled for collection, the scheduling number of the current data source is read to determine which of the two queues of the cloud resource storage queue under the data source is the outgoing resource queue and which is the incoming resource queue; The configuration center records the status of each service under each data source; when the scheduling of each service is completed, the status of the service is changed to completed, and when each service starts scheduling, the status of the service is changed to executing; when all services under a data source are completed, it indicates that the scheduling and collection task of the current data source is completed.

8. The method for monitoring data collection based on dual queues and hybrid multi-data sources according to claim 7 is characterized in that: The multi-cloud management platform automatically adjusts the periodic collection time for each data source according to the current configuration parameters, specifically: automatically adjusts the periodic collection time for each data source according to the resource machine configuration, the number of resources issued, the grouping parameters, and the data source interface collection performance; When the data source schedules a collection task, the collection time of this schedule is recorded and compared with the last collection time. The relatively larger collection task time is used to calculate the next scheduling time and written into the configuration center to schedule the next collection task of the data source.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for collecting monitoring data from hybrid multiple data sources based on dual queues as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the method for collecting monitoring data based on hybrid multiple data sources based on dual queues according to any one of claims 1 to 8.

Images