Multi-resource data concurrent acquisition system and method on multi-cloud management platform, storage medium and equipment

Abstract

The invention discloses a multi-resource data concurrent acquisition system and method on a multi-cloud management platform, a storage medium and equipment, and the system comprises a task manager which is used for storing the dependency relationship among cloud resources provided by a cloud service provider, constructing a directed acyclic graph of a resource data acquisition task in combination with a timed task configured on the multi-cloud management platform, submitting to a synchronization manager; the synchronous manager is used for storing a load strategy, dynamically adjusting the capacity of a thread pool by using the load strategy according to load monitoring on the task executor and the data processor, putting a resource data acquisition task into the thread pool, and calling the task executor to issue the resource data acquisition task; the task executor is used for packaging an API (Application Program Interface) of a cloud service provider, executing a resource data acquisition task and acquiring multi-resource data; and the data processor converts the acquired multi-resource data into a data format required by the multi-cloud management platform, and stores the data format in a database, thereby realizing concurrent acquisition of the multi-resource data.

Description

Technical Field

[0001] The present invention relates to the field of multi-cloud management technology, and in particular to a system, method, storage medium and device for concurrently collecting multi-resource data on a multi-cloud management platform. Background Art

[0002] In the existing multi-cloud management platform, it is usually necessary to collect the resource data of users in the cloud service provider through API calls to the multi-cloud management platform for unified management. At present, the process of multi-resource data collection is as follows: enter the API access credentials of the cloud service provider in the multi-cloud management platform, write the relevant code of the cloud resource data synchronization task in the multi-cloud management platform, fix the synchronization logic of the cloud resources into the code coding logic, and form a calling method for data synchronization within the multi-cloud management platform; configure a fixed scheduled task trigger, and the triggered resource data collection task calls the data synchronization method within the multi-cloud management platform to collect multi-resource data.

[0003] The existing resource data collection method calls API access in sequence according to the pre-written cloud resource data synchronization code, which results in slow data collection and long time. When the load is low, it still uses fixed scheduled task triggers, resulting in low resource utilization and poor flexibility. Summary of the invention

[0004] In response to the problems existing in the prior art, the present invention provides a system, method, storage medium and device for concurrent collection of multi-resource data on a multi-cloud management platform, which realizes concurrent collection of multi-resource data according to the dependency relationship between cloud resources and improves data collection efficiency; at the same time, by configuring a dynamic load adjustment strategy, the resource utilization and flexibility of the multi-cloud management platform are improved.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solutions: a system for concurrently collecting multi-resource data on a multi-cloud management platform, wherein a task manager, a synchronization manager, a task executor and a data processor are arranged on the multi-cloud management platform;

[0006] The task manager is used to store the dependencies between cloud resources provided by the cloud service provider, construct a directed acyclic graph of resource data collection tasks in combination with the scheduled tasks configured on the multi-cloud management platform, and submit it to the synchronization manager;

[0007] The synchronization manager is used to store the load strategy, and according to the load monitoring of the task executor and the data processor, dynamically adjust the thread pool capacity using the load strategy, put the resource data collection task into the thread pool, and call the task executor to issue the resource data collection task;

[0008] The task executor is used to encapsulate the API of the cloud service provider, perform resource data collection tasks, and collect multi-resource data;

[0009] The data processor converts the collected multi-resource data into a data format required by the multi-cloud management platform and saves it to a database.

[0010] Furthermore, the process of obtaining the dependency relationship between the cloud resources is as follows: when providing services, the cloud service provider gives the types and information of the cloud resources and determines the dependency relationship between the cloud resources.

[0011] Furthermore, the load strategy is specifically as follows: the task executor and the data processor check the CPU utilization and memory utilization every 5 seconds. If the CPU utilization and memory utilization do not exceed the preset target values, the thread pool capacity is expanded based on the original thread pool capacity until either the CPU utilization or the memory utilization reaches the preset target value first; otherwise, the thread pool capacity is reduced based on the original thread pool capacity until both the CPU utilization and the memory utilization do not exceed the preset target values.

[0012] Furthermore, the present invention also provides a method for concurrently collecting multi-resource data on a multi-cloud management platform, which uses the system for concurrently collecting multi-resource data on the multi-cloud management platform to concurrently collect multi-resource data, and specifically includes the following steps:

[0013] Step S1: The user logs in to the multi-cloud management platform, selects a cloud service provider, enters access credentials, and configures a scheduled task;

[0014] Step S2: The scheduled task is initiated and executed. The task manager receives the request, constructs a directed acyclic graph of the resource data collection task based on the dependency relationship between the cloud resources provided by the cloud service provider, and submits it to the synchronization manager.

[0015] Step S3, the synchronization manager searches for the task node according to the constructed directed acyclic graph of the resource data collection task, puts it into the thread pool, and calls the task executor to issue the resource data collection task;

[0016] Step S4: The task executor searches the API of the cloud service provider according to the resource data collection task, collects multiple resource data, and sends the data to the data processor for data format conversion and saves it in the database.

[0017] Furthermore, step S2 includes the following sub-steps:

[0018] Step S2.1, the scheduled task is initiated and executed, the task manager receives the request and queries all access credentials in pages;

[0019] Step S2.2: for each access credential, determine the resource type of the cloud service provider according to the cloud service provider to which the access credential belongs, and construct a data collection task for each resource;

[0020] Step S2.3: Generate directed edges for each data collection task according to the dependency relationship between cloud resources provided by the cloud service provider, form a directed acyclic graph, and send it to the synchronization manager.

[0021] Furthermore, step S3 includes the following sub-steps:

[0022] Step S3.1, initialize the thread pool capacity of the synchronization manager;

[0023] Step S3.2, according to the constructed directed acyclic graph of resource data collection tasks, find the task nodes with in-degree 0. If the task nodes with in-degree 0 do not exceed the thread pool capacity, put the task nodes with in-degree 0 into the thread pool, and the thread pool calls the task executor to issue the resource data collection tasks; otherwise, put the task nodes with in-degree 0 into the thread pool in batches, and the thread pool calls the task executor to issue the resource data collection tasks;

[0024] Step S3.3: After the thread pool successfully sends all resource data collection tasks, the resource data collection tasks are deleted from the directed acyclic graph, and the task manager is called to change the status of the corresponding resource data collection tasks.

[0025] Furthermore, the CPU utilization and memory utilization in the task executor and the data processor are reported to the synchronization manager, and the synchronization manager dynamically adjusts the thread pool capacity according to the load strategy.

[0026] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the method for concurrently collecting multi-resource data on a multi-cloud management platform.

[0027] Furthermore, the present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for concurrently collecting multi-resource data on the multi-cloud management platform is implemented.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] (1) The system and method for concurrently collecting multi-resource data on a multi-cloud management platform of the present invention constructs a directed acyclic graph of resource data collection tasks, determines concurrent task nodes, and executes concurrently through a thread pool, thereby improving the concurrency of multi-resource task execution, shortening resource data collection time, and improving data collection efficiency;

[0030] (2) The system and method for concurrent collection of multi-resource data on a multi-cloud management platform of the present invention uses a load strategy to dynamically adjust the thread pool capacity, which can fully utilize server resources and prevent server overload and crash, thereby improving the overall stability of concurrent collection of multi-resource data on the multi-cloud management platform and reducing human intervention;

[0031] (3) The system and method for concurrently collecting multi-resource data on a multi-cloud management platform of the present invention sets a task manager, a synchronization manager, a task executor and a data processor on the multi-cloud management platform, reduces the code coupling between modules, facilitates maintenance, and has a simple structure and is easy to promote. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is an architecture diagram of a system for concurrently collecting multi-resource data on a multi-cloud management platform of the present invention;

[0033] Figure 2 The present invention is a flowchart of a method for concurrently collecting multi-resource data on a multi-cloud management platform. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is further explained below in conjunction with the accompanying drawings.

[0035] like Figure 1 The present invention provides a system for concurrent collection of multi-resource data on a multi-cloud management platform. A task manager, a synchronization manager, a task executor and a data processor are set on the multi-cloud management platform to reduce the code coupling between modules, facilitate maintenance, and have a simple structure and are easy to promote. The task manager is used to store the dependencies between cloud resources provided by the cloud service provider, and construct a directed acyclic graph of resource data collection tasks in combination with the scheduled tasks configured on the multi-cloud management platform, and submit it to the synchronization manager; the synchronization manager is used to store load strategies, and according to the load monitoring of the task executor and the data processor, the load strategy is used to dynamically adjust the thread pool capacity, and the resource data collection task is placed in the thread pool, and the task executor is called to issue the resource data collection task; the task executor is used to encapsulate the API of the cloud service provider, perform resource data collection tasks, and collect multi-resource data; the data processor converts the collected multi-resource data into the data format required by the multi-cloud management platform, and saves it to the database. The system for concurrent collection of multi-resource data on the multi-cloud management platform of the present invention improves the efficiency of data synchronization through concurrent resource data collection tasks, and reduces human intervention through dynamic load adjustment strategies, improves the resource utilization of the server, and can meet the needs of concurrent data collection in large-scale, multi-cloud environments.

[0036] The process of obtaining the dependency relationship between cloud resources in the present invention is: when providing services, the cloud service provider gives the types and information of cloud resources and determines the dependency relationship between cloud resources. For example, although Alibaba Cloud VPC and subnet are in a containment relationship, the VPC information of the subnet can be obtained by querying the subnet alone, and the obtained information is complete, so the two can exist independently in the dependency graph; Huawei Cloud virtual machines and network cards are in a binding relationship, and the ID of the virtual machine must be obtained to query the network card information, so at this time the two will form a dependency relationship in the dependency graph.

[0037] The load strategy in the present invention is specifically as follows: the task executor and the data processor check the CPU usage and memory usage every 5 seconds. If the CPU usage and memory usage do not exceed the preset target value, the thread pool capacity is expanded on the basis of the original thread pool capacity until any one of the CPU usage and memory usage reaches the preset target value first; otherwise, the thread pool capacity is reduced on the basis of the original thread pool capacity until the CPU usage and memory usage do not exceed the preset target value. The present invention can fully utilize server resources and prevent server overload and crash by dynamically adjusting the thread pool capacity using the load strategy, thereby improving the overall stability of concurrent data collection of multiple resources on the multi-cloud management platform and reducing human intervention.

[0038] like Figure 2 The present invention also provides a method for concurrently collecting multi-resource data on a multi-cloud management platform, which specifically includes the following steps:

[0039] Step S1: The user logs in to the multi-cloud management platform, selects a cloud service provider, enters access credentials, and configures a scheduled task;

[0040] Step S2: The scheduled task is initiated and executed. The task manager receives the request, constructs a directed acyclic graph of the resource data collection task based on the dependency relationship between the cloud resources provided by the cloud service provider, and submits it to the synchronization manager. This includes the following sub-steps:

[0041] Step S2.1, the scheduled task is initiated and executed, the task manager receives the request and queries all access credentials in pages;

[0042] Step S2.2: for each access credential, determine the resource type of the cloud service provider according to the cloud service provider to which the access credential belongs, and construct a data collection task for each resource;

[0043] Step S2.3: Generate directed edges for each data collection task according to the dependency relationship between cloud resources provided by the cloud service provider, form a directed acyclic graph, and send it to the synchronization manager.

[0044] Step S3, the synchronization manager searches for task nodes according to the constructed directed acyclic graph of resource data collection tasks, puts them into the thread pool, calls the task executor to issue the resource data collection tasks, determines the concurrent task nodes by constructing the directed acyclic graph of resource data collection tasks, and executes them concurrently through the thread pool, thereby improving the concurrency of multi-resource task execution, shortening the resource data collection time, and improving data collection efficiency; including the following sub-steps:

[0045] Step S3.1, initialize the thread pool capacity of the synchronization manager;

[0046] Step S3.2, according to the constructed directed acyclic graph of the resource data acquisition task, search for the task node with an in-degree of 0. If the task node with an in-degree of 0 does not exceed the thread pool capacity, put the task node with an in-degree of 0 into the thread pool, and the thread pool calls the task executor to issue the resource data acquisition task; otherwise, put the task nodes with an in-degree of 0 into the thread pool in batches, and the thread pool calls the task executor to issue the resource data acquisition task; the present invention regards the task node with an in-degree of 0 as a concurrent node, and maximizes the number of resources processed in parallel when the thread pool capacity allows, and calls the separate synchronization logic method of each cloud resource in this order to perform data synchronization, improve data acquisition efficiency, and save time;

[0047] Step S3.3: After the thread pool successfully sends all resource data collection tasks, the directed acyclic graph of the resource data collection tasks is deleted, and the task manager is called to change the status of the corresponding resource data collection tasks.

[0048] Step S4: The task executor searches the API of the cloud service provider according to the resource data collection task, collects multiple resource data, and sends the data to the data processor for data format conversion and saves it in the database.

[0049] In one technical solution of the present invention, it is necessary to report the CPU utilization and memory utilization in the task executor and the data processor to the synchronization manager. The synchronization manager dynamically adjusts the thread pool capacity according to the load strategy, fully utilizes the server resources, and improves the overall stability of concurrent multi-resource data collection on the multi-cloud management platform.

[0050] The system and method for concurrently collecting multi-resource data on a multi-cloud management platform of the present invention realizes concurrent collection of multi-resource data according to the dependency relationship between cloud resources, thereby improving data collection efficiency; at the same time, by configuring a dynamic load adjustment strategy, the resource utilization and flexibility of the multi-cloud management platform are improved.

[0051] In a technical solution of the present invention, a computer-readable storage medium is also provided, storing a computer program, wherein the computer program enables a computer to execute the method for concurrently collecting multi-resource data on a multi-cloud management platform.

[0052] In a technical solution of the present invention, an electronic device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for concurrently collecting multi-resource data on the multi-cloud management platform is implemented.

[0053] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0054] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0055] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A system for concurrently collecting multi-resource data on a multi-cloud management platform, characterized in that: Set up a task manager, synchronization manager, task executor, and data processor on the multi-cloud management platform; The task manager is used to store the dependencies between cloud resources provided by the cloud service provider, construct a directed acyclic graph of resource data collection tasks in combination with the scheduled tasks configured on the multi-cloud management platform, and submit it to the synchronization manager; The synchronization manager is used to store the load strategy, and according to the load monitoring of the task executor and the data processor, dynamically adjust the thread pool capacity using the load strategy, put the resource data collection task into the thread pool, and call the task executor to issue the resource data collection task; The task executor is used to encapsulate the API of the cloud service provider, perform resource data collection tasks, and collect multi-resource data; The data processor converts the collected multi-resource data into a data format required by the multi-cloud management platform and saves it to a database.

2. According to the system for concurrently collecting multi-resource data on a multi-cloud management platform according to claim 1, it is characterized in that: The process of obtaining the dependency relationship between the cloud resources is as follows: when providing services, the cloud service provider gives the types and information of the cloud resources and determines the dependency relationship between the cloud resources.

3. The system for concurrently collecting multi-resource data on a multi-cloud management platform according to claim 1, characterized in that: The load strategy is specifically as follows: the task executor and the data processor check the CPU usage and memory usage every 5 seconds. If the CPU usage and memory usage do not exceed the preset target values, the thread pool capacity is expanded based on the original thread pool capacity until either the CPU usage or the memory usage reaches the preset target value first; otherwise, the thread pool capacity is reduced based on the original thread pool capacity until both the CPU usage and the memory usage do not exceed the preset target values.

4. A method for concurrently collecting multi-resource data on a multi-cloud management platform, characterized in that: The system for concurrently collecting multi-resource data on a multi-cloud management platform according to any one of claims 1 to 3 is used to concurrently collect multi-resource data, specifically comprising the following steps: Step S1: The user logs in to the multi-cloud management platform, selects a cloud service provider, enters access credentials, and configures a scheduled task; Step S2: The scheduled task is initiated and executed. The task manager receives the request, constructs a directed acyclic graph of the resource data collection task based on the dependency relationship between the cloud resources provided by the cloud service provider, and submits it to the synchronization manager. Step S3, the synchronization manager searches for the task node according to the constructed directed acyclic graph of the resource data collection task, puts it into the thread pool, and calls the task executor to issue the resource data collection task; Step S4: The task executor searches the API of the cloud service provider according to the resource data collection task, collects multiple resource data, and sends the data to the data processor for data format conversion and saves it in the database.

5. The method for concurrently collecting multi-resource data on a multi-cloud management platform according to claim 4, characterized in that: Step S2 includes the following sub-steps: Step S2.1, the scheduled task is initiated and executed, the task manager receives the request and queries all access credentials in pages; Step S2.2: for each access credential, determine the resource type of the cloud service provider according to the cloud service provider to which the access credential belongs, and construct a data collection task for each resource; Step S2.3: Generate directed edges for each data collection task according to the dependency relationship between cloud resources provided by the cloud service provider, form a directed acyclic graph, and send it to the synchronization manager.

6. The method for concurrently collecting multi-resource data on a multi-cloud management platform according to claim 5, characterized in that: Step S3 includes the following sub-steps: Step S3.1, initialize the thread pool capacity of the synchronization manager; Step S3.2, according to the constructed directed acyclic graph of resource data collection tasks, find the task nodes with in-degree 0. If the task nodes with in-degree 0 do not exceed the thread pool capacity, put the task nodes with in-degree 0 into the thread pool, and the thread pool calls the task executor to issue the resource data collection tasks; otherwise, put the task nodes with in-degree 0 into the thread pool in batches, and the thread pool calls the task executor to issue the resource data collection tasks; Step S3.3: After the thread pool successfully sends all resource data collection tasks, the resource data collection tasks are deleted from the directed acyclic graph, and the task manager is called to change the status of the corresponding resource data collection tasks.

7. The method for concurrently collecting multi-resource data on a multi-cloud management platform according to claim 6, characterized in that: The CPU usage and memory usage in the task executor and data processor are reported to the synchronization manager, which dynamically adjusts the thread pool capacity based on the load strategy.

8. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables the computer to execute the method for concurrently collecting multi-resource data on a multi-cloud management platform as described in claim 6.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for concurrently collecting multi-resource data on a multi-cloud management platform as claimed in claim 6 is implemented.

Images