Cloud resource collection method and system based on cloud service provider

Abstract

The invention relates to a cloud resource collection method and system based on a cloud service provider. The method comprises the following steps: establishing an execution database; acquiring an acquisition rule through an acquisition task table in the execution database, and dividing an acquisition task into a plurality of acquisition nodes according to the acquisition rule; obtaining interface information corresponding to a collection node through a process node table in the execution database; and performing sequential execution on the plurality of acquisition nodes through the interface information to obtain an acquisition result. The maintainability and manageability of resource collection and data management are provided, and meanwhile the stability and reliability of cloud resource collection work are guaranteed.

Description

Technical Field

[0001] The present invention belongs to the field of cloud resource data management, and in particular relates to a cloud resource collection method and system based on a cloud service provider. Background Art

[0002] With the rapid development of cloud computing technology, more and more enterprises and organizations choose to deploy their businesses in the cloud to enjoy the advantages of cloud computing, such as elastic expansion, on-demand resource allocation, and reduced operation and maintenance costs. However, with the increase in cloud service providers and the complexity of cloud resources, the types of resources provided by various cloud service systems and the relationships between resources are different. In addition, due to the large differences in the amount of data in resources, large data volumes use SFTP file transfer methods, while small data volumes use API interfaces.

[0003] The interface methods supported by traditional cloud resource collection methods are relatively simple. With the continuous upgrading of cloud services and the updating of interfaces, they cannot provide effective collection for different data transmission interface methods, different data formats, different data levels, etc., and are prone to errors. They cannot meet the cloud resource collection needs in large-scale, multi-cloud service environments. Summary of the invention

[0004] In view of the above deficiencies in the prior art, the object of the invention is to provide a cloud resource collection method and system based on a cloud service provider.

[0005] The present invention provides a cloud resource collection method based on a cloud service provider, comprising:

[0006] S1: Establish execution database;

[0007] S2: acquiring a collection rule through the collection task table in the execution database, and dividing the collection task into a plurality of collection nodes according to the collection rule;

[0008] S3: Obtaining interface information corresponding to the collection node through the process node table in the execution database;

[0009] S4: sequentially executing multiple collection nodes through the interface information to obtain collection results.

[0010] According to a cloud resource collection method based on a cloud service provider provided by the present invention, the execution database in step S1 specifically includes:

[0011] Process instance table, process node table, collection task table, node task association table, process monitoring table;

[0012] The process instance table is used to record instance information of multiple instances of the acquisition task;

[0013] The node task association table is used to update the execution logic of the collection node in the collection task;

[0014] The process monitoring table is used to monitor the interface calling conditions of multiple collection nodes.

[0015] According to a cloud resource collection method based on a cloud service provider provided by the present invention, the instance information includes instance status, execution time, and end time.

[0016] According to a cloud resource acquisition method based on a cloud service provider provided by the present invention, the interface call situation includes interface call request frequency, interface response time, and interface call request success rate.

[0017] According to a cloud resource collection method based on a cloud service provider provided by the present invention, the interface information in step S3 includes interface ID, interface name, interface mode, cloud type, interface version, interface path, request method, HTTP content type, timeout, request header, request parameter body sample, return parameter body sample, and request header sample.

[0018] According to a cloud resource collection method based on a cloud service provider provided by the present invention, step S4 further includes:

[0019] By configuring the cron timer, the node subtask corresponding to each collection node is executed regularly;

[0020] Reduce the frequency of API calls to lazy data through caching strategies;

[0021] Multiple collection nodes are executed sequentially through asynchronous processing and concurrent processing.

[0022] According to a cloud resource collection method based on a cloud service provider provided by the present invention, in step S4, when an interface call fails during the execution of a collection node, the interface call is retried using an exponential backoff retry algorithm.

[0023] According to a cloud resource collection method based on a cloud service provider provided by the present invention, step S4 further includes:

[0024] When executing the collection node, the real-time collection results are detected through the Bloom filter.

[0025] According to a cloud resource collection method based on a cloud service provider provided by the present invention, the detection of real-time collection results by using a Bloom filter includes:

[0026] Discard duplicate values ​​in the collected results;

[0027] Identify outliers in your collection results

[0028] Check whether the required fields in the collection results are filled;

[0029] Check whether the data type in the collected results is correct.

[0030] The present invention also provides a cloud resource collection system based on a cloud service provider, which is used to execute a cloud resource collection method based on a cloud service provider as described in any one of the above items, including:

[0031] Receiving module: used to receive collection tasks;

[0032] Construction module: used to build the execution database;

[0033] Execution module: used to obtain collection rules from the execution database, divide the collection task into multiple collection nodes according to the collection rules; also used to obtain interface information corresponding to the collection nodes through the execution database; and also used to sequentially execute multiple collection nodes through the interface information to obtain collection results;

[0034] Monitoring module: used to monitor the execution status of the execution module executing multiple collection nodes.

[0035] The present invention provides a cloud resource collection method and system based on a cloud service provider. By establishing an execution database, including a process instance table, a process node table, a collection task table, a node task association table and a process monitoring table, the present invention can systematically manage cloud resource collection tasks, effectively track task status, execution time and end time, thereby ensuring efficient execution and monitoring of tasks; the present invention decomposes the collection task into multiple collection nodes according to the collection rules corresponding to the cloud service provider, and obtains corresponding interface information through the process node table, so that the collection process can be flexibly adjusted according to actual needs, thereby improving the flexibility and scalability of collection; in addition, the interface information provided according to the established database includes detailed interface attributes, which not only ensures the accuracy of interface calls, but also provides convenience for subsequent interface maintenance and optimization; by configuring a cron timer to realize timed execution, using a cache strategy to reduce the frequency of interface calls to inert data, and adopting asynchronous processing and concurrent processing technology, the present invention significantly improves resource utilization efficiency and reduces costs; in addition, the present invention effectively responds to the problems of network fluctuations or unstable cloud service provider interfaces through a monitoring and management strategy for the collection process, improves the reliability of collection, and ensures the high quality and accuracy of the collection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are only used to illustrate specific embodiments and are not considered to limit the present invention. In the entire drawings, the same reference symbols represent the same components. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0037] Figure 1 A schematic diagram of a cloud resource collection method based on a cloud service provider provided in an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of the structure of a cloud resource acquisition system based on a cloud service provider provided in an embodiment of the present invention.

[0039] Reference numerals:

[0040] 100, receiving module; 200, building module; 300, executing module; 400, monitoring module. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0042] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0043] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. The terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0044] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0045] Combine the following Figure 1 to Figure 2 Embodiments of the present invention are described.

[0046] like Figure 1 As shown, the present invention provides a cloud resource collection method based on a cloud service provider, comprising:

[0047] S1: Establish execution database.

[0048] The execution database in step S1 specifically includes:

[0049] Process instance table, process node table, collection task table, node task association table, process monitoring table;

[0050] The process instance table is used to record instance information of multiple instances of the acquisition task.

[0051] The instance information includes instance status, execution time, and end time.

[0052] Furthermore, the same collection task can have multiple instances, so the present invention records the specific execution process of each task through a process instance table, including the information of the currently executing and historical process instances, and records the instance status, execution time, end time and other related variables.

[0053] The node task association table is used to update the execution logic of the collection nodes in the collection task.

[0054] Furthermore, within a task, there are multiple nodes, and the order of the nodes is unique, that is, no two nodes have the same sequence number in the same task. When you want to add, delete, or reorder nodes in a task, you only need to update the node-task association table, which provides great flexibility and allows you to easily adjust the execution logic of the task.

[0055] The process monitoring table is used to monitor the interface calling conditions of multiple collection nodes.

[0056] The interface call status includes the interface call request frequency, interface response time, and interface call request success rate.

[0057] Furthermore, in order to ensure efficient and stable collection, the call records of each node's interface call request frequency, response time, and request failure are saved in detail in the process monitoring table to facilitate rapid problem location and repair.

[0058] S2: Acquire collection rules through the collection task table in the execution database, and divide the collection task into multiple collection nodes according to the collection rules.

[0059] Furthermore, each cloud service provider designs corresponding collection rules. Each task contains multiple collection nodes, and each node represents a cloud resource collection subtask. There is a sequence between resources. The later nodes strongly rely on the data of the previous nodes as input parameters. By adjusting the node sequence, the data collection rules between cloud resources can be flexibly changed. When a resource is added or deleted, the collection rules can also be adjusted by changing the workflow, avoiding a large amount of code modification.

[0060] S3: Obtaining interface information corresponding to the collection node through the process node table in the execution database.

[0061] Among them, the interface information in step S3 includes interface ID, interface name, interface mode, cloud type, interface version, interface path, request method, HTTP content type, timeout, request header, request parameter body sample, return parameter body sample, and request header sample.

[0062] Furthermore, when a certain node is executed, the node is configured with a unique interface ID, and the specific information of the API is queried through this ID to trigger the call. The configuration table should clearly and comprehensively display the detailed information of the interface. The table of the present invention contains ID, name, interface method (HTTP, SFTP), cloud type (IT cloud, CT cloud, Tianyi cloud), version, path (URL, file path), request method (GET, POST, PUT, etc.), HTTP content type (CONTENT_TYPE), timeout, request header, request parameter body sample, return parameter body sample, request header sample, and remarks. In addition, the process node table can also help developers or maintenance personnel understand, track, and maintain interface information.

[0063] S4: sequentially executing multiple collection nodes through the interface information to obtain collection results.

[0064] Wherein, step S4 also includes:

[0065] By configuring the cron timer, the node subtask corresponding to each collection node is executed regularly.

[0066] Furthermore, when executing the collection task, the present invention also provides an automated collection engine for scheduled tasks. The principle is to first configure a cron timer for all task flows, automatically load all scheduled tasks when the service starts, and then execute them according to their respective cron times. Each node subtask is executed, including interface information search, interface call, data analysis, data preprocessing, and database storage.

[0067] Reduce the frequency of interface calls to lazy data through caching strategies.

[0068] Furthermore, the present invention adopts a cache strategy to reduce the number of interface calls for data that does not change frequently, such as authentication information.

[0069] Multiple collection nodes are executed sequentially through asynchronous processing and concurrent processing.

[0070] Furthermore, for operations that take a long time, the present invention adopts an asynchronous processing method to avoid blocking the main thread, and uses multi-threading technology to process interface requests and data analysis tasks concurrently or in parallel to improve collection efficiency.

[0071] In the present invention, when encountering time-consuming tasks, such as tasks with large data volumes that require cyclic paging to obtain data, such as tasks that preprocess the current node data in parallel with the next acquisition node, the thread pool can be used for parallel processing. Specifically, the number of core threads is first set to 10, the maximum number of threads is set to 20, and then the ExecutorService is used to manage the thread pool, and CompletableFuture is used to combine multiple asynchronous tasks, set the task timeout, and appropriately handle or throw exceptions, so as to improve the efficiency of the execution of the acquisition task.

[0072] Among them, in step S4, when an interface call fails during the execution of the collection node, the interface call is retried through an exponential backoff retry algorithm.

[0073] Furthermore, the exponential backoff retry algorithm described above does not retry immediately when a request fails, but waits for a period of time before retrying, and the waiting time between each retry gradually increases to reduce continuous failures caused by network fluctuations or temporary server overload.

[0074] Specifically, each collection node first sends a request and checks the response status code. If the request fails, it enters the retry logic. The waiting time for the first retry is 3 seconds, the maximum number of retries is 5, and the backoff factor is set to 2, that is, the new waiting time is twice the previous waiting time. Each retry is recorded in the process monitoring table to facilitate troubleshooting and performance analysis when problems occur.

[0075] Wherein, step S4 also includes:

[0076] When executing the collection node, the real-time collection results are detected through the Bloom filter.

[0077] The detection of real-time collection results through Bloom filters includes:

[0078] Discard duplicate values ​​in the collected results; identify abnormal values ​​in the collected results; check whether the required fields in the collected results are filled in; check whether the data type in the collected results is correct.

[0079] Furthermore, during the execution of step S4, the present invention also uses a Bloom filter to quickly detect and remove duplicate data, identify and process abnormal values ​​in the data, check whether required fields have been filled in, whether the data type is correct, and other data preprocessing tasks.

[0080] Specifically, the BloomFilter class provided by Guava is used to create and initialize Bloom Filter, and the estimated capacity is set to 10,000 and the error rate is set to 0.01. Whenever the collection node task collects data, the data is added to the Bloom filter. Bloom Filter only needs to store bitmaps instead of specific elements, so the query speed is very fast and can effectively remove duplicates. Subsequently, the validator class DataValidator is created to define various methods for validating data models. The OutlierDetector anomaly detection algorithm class is also created, and IQR (interquartile range) is used to identify outliers. Finally, it is integrated into a data processing pipeline to preprocess the collected data.

[0081] In addition, during the execution process, if a node reports an error, first record the complete log and then retry. If the execution fails in the end, determine whether it has an impact on the subsequent process. If it has an impact, terminate the task execution, otherwise jump directly to the next node collection.

[0082] like Figure 2 As shown, the present invention also provides a cloud resource collection system based on a cloud service provider, comprising:

[0083] Receiving module 100: used to receive collection tasks;

[0084] Construction module 200: used to establish an execution database;

[0085] Execution module 300: used to obtain collection rules from the execution database, divide the collection task into multiple collection nodes according to the collection rules; also used to obtain interface information corresponding to the collection nodes through the execution database; and also used to sequentially execute multiple collection nodes through the interface information to obtain collection results;

[0086] The monitoring module 400 is used to monitor the execution status of the execution module 300 on multiple collection nodes.

[0087] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0088] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0089] The present invention provides a cloud resource collection method and system based on cloud service providers, aiming to solve the problem that the increase in cloud service providers and the complexity of cloud resources make it difficult to manage cloud resource data. Traditional cloud resource collection methods have a single collection mode, and with the continuous upgrading of cloud services and the updating of interfaces, traditional methods have become difficult to maintain and expand.

[0090] The present invention needs to integrate the interfaces provided by various cloud service providers. On the basis of clarifying the collection objectives and scope, including the types of cloud resources that need to be collected, the frequency and real-time requirements of data collection, and the requirements of data security and reliability, in order to ensure efficient and sequential collection between different cloud resources, a lightweight custom workflow engine is invented, which can support the design of collection rules and collection processes for each task to meet the collection requirements of different cloud resources. The engine can automatically call the corresponding collection tasks and can efficiently and accurately obtain cloud resource information according to the preset collection rules. After the collected cloud resource information is parsed and processed, it can be stored in the local database or displayed to the user to meet the needs of different application scenarios.

[0091] In addition, in order to ensure the reliability and stability of cloud resource acquisition, the present invention also sets up a monitoring and optimization mechanism, which can regularly check the operating status of the cloud resource acquisition engine, including the response of the interface, the integrity and accuracy of data acquisition, etc. Once an abnormal situation is found, the system will automatically track and record it, and optimize and adjust it according to the quality of the collected data to ensure that the cloud resource acquisition work is carried out efficiently and accurately.

[0092] The cloud resource collection method and system based on a cloud service provider of the present invention provide a lightweight workflow engine, which supports dynamic adjustment of the order and scope of collection by designing flexible collection rules, reduces the workload of manual operation and maintenance, and provides system maintainability and manageability. At the same time, a monitoring and optimization mechanism is set up to timely discover and handle system anomalies, thereby ensuring the stability and reliability of cloud resource collection work.

[0093] In summary, the cloud resource collection method based on multiple service providers proposed in the present invention effectively solves the problems of low efficiency, easy errors, difficult maintenance and expansion in traditional cloud resource collection methods through automation and intelligent means, and provides strong support for enterprises to efficiently and accurately collect and manage cloud resource information in a multi-cloud environment.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. A cloud resource collection method based on a cloud service provider, characterized in that: include: S1: Establish execution database; S2: acquiring a collection rule through the collection task table in the execution database, and dividing the collection task into a plurality of collection nodes according to the collection rule; S3: Obtaining interface information corresponding to the collection node through the process node table in the execution database; S4: sequentially executing multiple collection nodes through the interface information to obtain collection results.

2. A cloud resource collection method based on a cloud service provider according to claim 1, characterized in that: The execution database in step S1 specifically includes: Process instance table, process node table, collection task table, node task association table, process monitoring table; The process instance table is used to record instance information of multiple instances of the acquisition task; The node task association table is used to update the execution logic of the collection node in the collection task; The process monitoring table is used to monitor the interface calling conditions of multiple collection nodes.

3. A cloud resource collection method based on a cloud service provider according to claim 2, characterized in that: The instance information includes instance status, execution time, and end time.

4. The cloud resource collection method based on a cloud service provider according to claim 2, characterized in that: The interface call status includes interface call request frequency, interface response time, and interface call request success rate.

5. The cloud resource collection method based on a cloud service provider according to claim 1, characterized in that: The interface information in step S3 includes interface ID, interface name, interface mode, cloud type, interface version, interface path, request method, HTTP content type, timeout, request header, request parameter body sample, return parameter body sample, and request header sample.

6. The cloud resource collection method based on a cloud service provider according to claim 1, characterized in that: Step S4 also includes: By configuring the cron timer, the node subtask corresponding to each collection node is executed regularly; Reduce the frequency of API calls to lazy data through caching strategies; Multiple collection nodes are executed sequentially through asynchronous processing and concurrent processing.

7. The cloud resource collection method based on a cloud service provider according to claim 1, characterized in that: In step S4, when an interface call fails during the execution of the collection node, the interface call is retried using an exponential backoff retry algorithm.

8. The cloud resource collection method based on a cloud service provider according to claim 1, characterized in that: Step S4 also includes: When executing the collection node, the real-time collection results are detected through the Bloom filter.

9. A cloud resource collection method based on a cloud service provider according to claim 8, characterized in that: The detection of real-time collection results through Bloom filters includes: Discard duplicate values ​​in the collected results; Identify outliers in your collection results Check whether the required fields in the collection results are filled; Check whether the data type in the collected results is correct.

10. A cloud resource collection system based on a cloud service provider, used to execute a cloud resource collection method based on a cloud service provider as claimed in any one of claims 1 to 9, characterized in that: include: Receiving module: used to receive collection tasks; Construction module: used to build the execution database; Execution module: used to obtain collection rules from the execution database, divide the collection task into multiple collection nodes according to the collection rules; and also used to obtain interface information corresponding to the collection nodes through the execution database; It is also used to sequentially execute multiple collection nodes through the interface information to obtain collection results; Monitoring module: used to monitor the execution status of the execution module executing multiple collection nodes.

Images