Cloud resource cost reconciliation method and device, equipment and medium
By acquiring and aggregating cloud resource billing data, using Flink and Aviator rule engines for cleaning and verification, the problem of low efficiency in cloud resource cost reconciliation is solved, automated reconciliation and refined analysis are realized, and reliable data foundation and finance are provided. monitor.
Patent Information
- Application Number
- CN202411151857.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-19
AI Technical Summary
In the prior art, cloud resource cost reconciliation is only a general ledger comparison, with low manual efficiency and error-prone, making it difficult to achieve refined analysis of bill data, and the amount of cloud resource bill data is huge, making manual processing difficult.
Obtain cloud resource billing data through access keys, use Flink for aggregation statistics and matching verification, introduce the Aviator rule engine for data cleaning and labeling, realize automated collection and verification, and provide bill data display and monitoring.
It realizes automated reconciliation and refined analysis of cloud resource costs, reduces manual errors, provides a reliable data foundation, promptly discovers financial risks, and supports automatic updates and early warnings of financial indicators.
Smart Images

Figure CN118982433A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cloud data processing technology, and in particular to a method, device, equipment and medium for reconciling cloud resource costs. Background Art
[0002] In the context of the development of the cloud computing industry and enterprise digitalization, enterprises usually use cloud resources from multiple suppliers to support business operations. As the proportion of cloud resource costs in enterprise operating costs gradually increases, effective cloud resource cost analysis and management has become crucial.
[0003] However, different suppliers have different cloud resource bill fields and data formats, and the amount of cloud resource bill data increases with the increase in resource usage. When performing cost verification, it is necessary to manually export data from each supplier's console and verify it, which is only a general ledger reconciliation. When performing bill analysis, it is also necessary to manually clean and integrate cloud resource bill data in order to comprehensively analyze the cost of each supplier. The existing cloud resource bill data of some suppliers is generated in the following way: as long as there is a fee, it will be charged every half an hour, which results in a monthly bill of hundreds of millions of data. If it is processed manually, it is difficult to achieve refined analysis and application; and manual efficiency is low and prone to errors. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a method, device, equipment and medium for cloud resource cost reconciliation, which can significantly improve efficiency, reduce manual errors, and realize the online and refined analysis application of billing data.
[0005] In a first aspect, the present invention provides a method for reconciling cloud resource costs, comprising the following steps: Step 1: Use the access key to obtain the original cloud resource billing data at the set time according to the API billing interface or query interface of major cloud vendors, and store the cloud resource billing data; Step 2: Read cloud resource bill data through Flink, aggregate and count according to the supplier account dimension, obtain the cost amount of each account, and match and check the cost amount with the cost amount on the crawled supplier console; Step 3: If the cost amount of the supplier account is inconsistent with the cost amount on the crawled supplier console, an alarm is triggered to remind the setting personnel to handle it and end the step; If the cost amount of the supplier account is consistent with the cost amount on the crawled supplier console, the project, cost, and product tags of the cloud resource bill data are cleaned and tagged based on the Aviator rule engine; Step 4: Display the cleaned and labeled cloud resource billing data.
[0006] In a second aspect, the present invention provides a cloud resource cost reconciliation device, comprising the following modules: The data acquisition module is used to obtain the original cloud resource billing data through the API billing interface or query interface of major cloud vendors at a set time through the access key, and store the cloud resource billing data; The bill acquisition module is used to read cloud resource bill data through Flink, aggregate and count according to the supplier account dimension, obtain the cost amount of each account, and match and verify the cost amount with the cost amount on the crawled supplier console; The verification module is used to trigger an alarm if the cost amount of the supplier account is inconsistent with the cost amount on the crawled supplier console, reminding the setting personnel to handle it and end the step; If the cost amount of the supplier account is consistent with the cost amount on the crawled supplier console, the project, cost, and product tags of the cloud resource bill data are cleaned and tagged based on the Aviator rule engine; The display module is used to display the cleaned and labeled cloud resource bill data.
[0007] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.
[0008] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.
[0009] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages: The methods, devices, equipment and media provided in the embodiments of the present application realize the automatic collection and verification of cloud resource billing data of each supplier; provide billing data cleaning and labeling functions to make billing data of different suppliers online, and provide a reliable data basis for subsequent analysis applications; display data to facilitate users to conduct intuitive data analysis and monitoring, and timely discover potential financial risks; realize automatic updating and early warning of financial indicators, and help users to grasp the financial status in a timely manner and make corresponding decisions.
[0010] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.
[0012] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 is a flow chart of the method in Embodiment 1 of the present invention; Figure 3 It is a schematic diagram of the structure of the device in the second embodiment of the present invention. DETAILED DESCRIPTION
[0013] The embodiments of the present application provide a method, device, equipment and medium for reconciliation of cloud resource costs, thereby solving the problem that reconciliation in the prior art is only a comparison of general ledgers; and also solves the technical problems of low manual efficiency and easy errors.
[0014] The technical solution in the embodiment of the present application has the following general idea: Before introducing the specific embodiments, the system framework corresponding to the method of the embodiment of the present application is first introduced. Figure 1 As shown, the system is roughly divided into three parts: (1) Obtain and verify bills from various cloud vendors Step 1: Use the access key to obtain the original detailed cloud resource bill data from the bill API or query interface of major domestic and foreign public cloud vendors at the beginning of each month. The cloud resource bill data is first stored in the storage bucket S3. The specific crawling method is configured and integrated according to the platform support method, including: a. Bigquery query data and then export and download to the storage bucket S3; b. Call the API bill interface to generate a csv file and then summarize it in the cloud resource middle platform; c. Object storage is summarized and forwarded to the storage bucket S3; Step 2: Push the data in the storage bucket to Kafka on the data center. Initial storage in the storage bucket can prevent file loss and reduce the file loss rate; Step 3: The data center reads the cloud resource billing data through Flink. After reading the data, it aggregates and counts according to the supplier account dimension (supplier account: refers to the main account and sub-account registered with the cloud vendor. To achieve resource isolation, users are generally added under the main account, and the users correspond to sub-accounts. Both the main account and the sub-account can purchase resources and incur costs) to obtain the cost amount of each account for the current month. The calculation method of the cost amount is consistent with that of each cloud vendor's console, that is, the original currency amount is retained. In particular, the cost amount of some cloud vendors needs to deduct the coupon amount to keep it consistent with the console. Subsequently, the cost amount is matched and checked with the cost amount data on the crawled supplier console to ensure that there is no missing cost data; Step 4: If the amounts on both sides of a supplier's account are inconsistent, the system will automatically trigger an alarm, and the DingTalk robot will remind the relevant parties to handle the issue to ensure that the obtained cloud resource bill data is consistent with the platform and improve the efficiency of reconciliation. After that, the bill data corresponding to the account with inconsistent amounts will be re-obtained for reconciliation. Step 5: If there is a new account application, the account approval process will be initiated. After the account application is completed, the system will subscribe to the bill of the new account to avoid missing cloud resource bill data for the new account.
[0015] (2) The system supports the standardization of cloud resource billing data from different suppliers The Aviator rule engine is introduced. Based on this engine, flexible rule configuration and management can be achieved. Different condition parameters and corresponding result configurations are supported. The project, cost, and product tags of cloud resource bill data are cleaned and tagged to increase the value of cloud resource bill data and provide a reliable data foundation for refined analysis applications. Each cloud vendor's bill will be tagged according to its settings. Therefore, when applying for resources, the subproject and env tags will be added. These two tags can correspond to the project tag and cost tag respectively. Product labeling: Most bills contain product information, but the product information specifications of each supplier are different. For example, a server may be "Cloud Server ECS" in Alibaba Cloud and "Amazon EC2" in AWS. The present invention establishes a unified standard to label the bills of all suppliers with product labels. The cleaning is to confirm the project tags and cost tags with the person in charge of the account, confirm the project ownership and cost type corresponding to the resources used in the bill, and then set them up for automatic confirmation; The labeling includes project labels, cost type labels, product information labels, etc.; labeling can be used to split costs by project and expenses, and analyze the distribution of cost usage based on unified standard product information; The specific steps are as follows: Step 1: Specify unified storage specifications, design and develop the company's billing table, and implement cost calculation in accordance with specific logic to meet financial standards, such as converting the cost amount of overseas supplier bills into RMB, deducting the bill coupon amount, etc.; Step 2: Determine the parameters involved in the rule configuration and the parameters that need to be labeled, and design and develop the corresponding rule table, which is used to store rule expressions. Use strings to concatenate the rules into the rule expressions that need to be executed; Step 3: Determine the rule expressions and execution order corresponding to the condition information set and result information set; for example: "under XX condition, then put XX label"; the condition information set is XX condition, which corresponds to the condition information set; the result information set is the result corresponding to XX condition; The acquisition of condition information and result information is confirmed with relevant parties. For example, when O&M allocates resources to a department, it will specify the project tag and cost tag of the resource. Step 4: Write the determined rule expressions and execution order into the rule table, such as condition parameters (account_id=='14xxxxxxxxxxxx'&&product_category_name=='Cloud Monitoring') and tag parameters (subproject='op'); Step 5: At the same time, the Aviator rule engine is introduced to execute rule expressions, so that data can be labeled according to rule expressions. At the same time, the data is executed according to the written calculation logic. The calculation logic is the execution calculation method. For example, some overseas bills, such as AWS and Azure, are in US dollars and need to be multiplied by the exchange rate to convert them into RMB. Some bills are tax-inclusive or tax-exclusive, so the tax-inclusive cost must be divided by the tax rate. However, some suppliers' bills only have tax-inclusive or tax-exclusive costs. Therefore, according to the scenario requirements, the tax rate will be associated to calculate the corresponding tax-exclusive or tax-inclusive costs. The calculation logic needs to be edited in advance and then executed; Step 6: If the tag has an empty value, execute steps 3, 4, and 5 to complete labeling.
[0016] (3) Application and analysis of cloud resource billing data After obtaining standardized and highly accurate cloud resource bill data through the above steps, the system's visual dashboard is then used to display and export data; visualize and apply cloud resource bill data, and implement automatic updates and early warning functions for financial indicators. This can help users intuitively analyze and monitor data and promptly identify potential financial risks; The analysis data includes: the company's monthly cloud resource costs, monthly supplier costs, monthly main product costs, project costs, revenue, gross profit margin, supplier gross profit margin, etc. By making cost data online, business lines can improve their cost awareness and pay attention to cost changes in their business lines. Discovering potential financial risks includes: checking bills and promptly discovering bill problems; for example, the number of servers / IPs recorded in the bill is too different from that recorded in the business database; Based on the cost data obtained from the bill, combined with gross profit, etc., the gross profit margin of each supplier can be analyzed. If the gross profit margin change of a supplier exceeds the threshold, further drill-down analysis is required to determine whether it is caused by cost changes or revenue changes. The analysis of gross profit margin can help companies formulate business strategies; the analysis of supplier gross profit margin can also help with supplier selection or the configuration of business product listings.
[0017] Embodiment 1
[0018] like Figure 2 As shown, this embodiment provides a method for reconciling cloud resource costs, including the following steps: Step 1: Use the access key to query the cloud resource billing data of the corresponding supplier through BigQuery, and then export and download it to the storage bucket S3; generate a CSV file by calling the API billing interface of major cloud vendors, and then summarize it to the cloud resource middle platform, and then forward it to the storage bucket S3; Step 2: Read cloud resource bill data through Flink, aggregate and count according to the supplier account dimension, obtain the cost amount of each account, and match and check the cost amount with the cost amount on the crawled supplier console; Step 3: If the cost amount of the supplier account is inconsistent with the cost amount on the crawled supplier console, an alarm is triggered to remind the setting personnel to handle it and end the step; If the cost amount of the supplier account is consistent with the cost amount on the crawled supplier console, specify a unified storage specification, set the bill table, and perform cost calculation according to the set calculation logic; determine the parameters involved, the parameters that need to be tagged, the condition information set, and the result information set; configure the corresponding rule expressions and execution logic; based on the Aviator rule engine, clean and tag the cloud resource bill data according to the rule expressions and execution logic; Step 4: Display the cleaned and labeled cloud resource billing data.
[0019] Based on the same inventive concept, the present application also provides a device corresponding to the method in Example 1, see Example 2 for details.
[0020] Embodiment 2
[0021] like Figure 3 As shown, in this embodiment, a device for reconciliation of cloud resource costs is provided, including the following modules: The data acquisition module is used to query the cloud resource billing data of the corresponding supplier through BigQuery through the access key, and then export and download it to the storage bucket S3; generate a CSV file by calling the API billing interface of major cloud vendors, and then summarize it to the cloud resource middle platform, and then forward it to the storage bucket S3; The bill acquisition module is used to read cloud resource bill data through Flink, aggregate and count according to the supplier account dimension, obtain the cost amount of each account, and match and verify the cost amount with the cost amount on the crawled supplier console; The verification module is used to trigger an alarm if the cost amount of the supplier account is inconsistent with the cost amount on the crawled supplier console, reminding the setting personnel to handle it and end the step; If the cost amount of the supplier account is consistent with the cost amount on the crawled supplier console, specify a unified storage specification, set the bill table, and perform cost calculation according to the set calculation logic; determine the parameters involved, the parameters that need to be tagged, the condition information set, and the result information set; configure the corresponding rule expressions and execution logic; based on the Aviator rule engine, clean and tag the cloud resource bill data according to the rule expressions and execution logic; The display module is used to display the cleaned and labeled cloud resource bill data.
[0022] Since the device introduced in the second embodiment of the present invention is a device used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.
[0023] Based on the same inventive concept, the present application provides an electronic device embodiment corresponding to the first embodiment, see the third embodiment for details.
[0024] Embodiment 3
[0025] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the implementation method in the first embodiment can be implemented.
[0026] Since the electronic device introduced in this embodiment is a device used to implement the method in the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, a person skilled in the art can understand the specific implementation of the electronic device of the present embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present application is not described in detail here. As long as a person skilled in the art implements the device used by the method in the embodiment of the present application, it belongs to the scope of protection of the present application.
[0027] Based on the same inventive concept, the present application provides a storage medium corresponding to the first embodiment, see the fourth embodiment for details.
[0028] Embodiment 4
[0029] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the implementation method in the first embodiment can be implemented.
[0030] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages: the method, device, equipment and medium provided in the embodiments of the present application realize the automatic collection and verification of cloud resource billing data of each supplier; provide billing data cleaning and labeling functions to make billing data of different suppliers online, providing a reliable data basis for subsequent analysis applications; display data to facilitate users to conduct intuitive data analysis and monitoring, and timely discover potential financial risks; realize automatic updating and early warning of financial indicators, helping users to grasp the financial status in a timely manner and make corresponding decisions.
[0031] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0032] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0033] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0034] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.
[0035] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.
Claims
1. A method for reconciling cloud resource costs, characterized in that: The steps include: Step 1: Use the access key to obtain the original cloud resource billing data at the set time according to the API billing interface or query interface of major cloud vendors, and store the cloud resource billing data; Step 2: Read cloud resource bill data through Flink, aggregate and count according to the supplier account dimension, obtain the cost amount of each account, and match and check the cost amount with the cost amount on the crawled supplier console; Step 3: If the cost amount of the supplier account is inconsistent with the cost amount on the crawled supplier console, an alarm is triggered to remind the setting personnel to handle it and end the step; If the cost amount of the supplier account is consistent with the cost amount on the crawled supplier console, the project, cost, and product tags of the cloud resource bill data are cleaned and tagged based on the Aviator rule engine; Step 4: Display the cleaned and labeled cloud resource billing data.
2. A method for reconciling cloud resource costs according to claim 1, characterized in that: The step 1 is specifically as follows: Through the access key, query the cloud resource billing data of the corresponding supplier through BigQuery, and then export and download it to the storage bucket S3; generate a CSV file by calling the API billing interface of major cloud vendors, and then summarize it to the cloud resource middle platform, and then forward it to the storage bucket S3.
3. A method for reconciling cloud resource costs according to claim 1, characterized in that: If the cost amount of the supplier account is consistent with the cost amount on the crawled supplier console, the project, cost and product tags of the cloud resource bill data are cleaned and labeled based on the Aviator rule engine as follows: If the cost amount of the supplier account is consistent with the cost amount on the crawled supplier console, specify a unified storage specification, set the bill table, and perform cost calculation according to the set calculation logic; determine the parameters involved, the parameters that need to be labeled, the condition information set, and the result information set; configure the corresponding rule expression and execution logic; Based on the Aviator rule engine, cloud resource billing data is cleaned and labeled according to rule expressions and execution logic.
4. A device for reconciling cloud resource costs, characterized in that: Includes the following modules: The data acquisition module is used to obtain the original cloud resource billing data through the API billing interface or query interface of major cloud vendors at a set time through the access key, and store the cloud resource billing data; The bill acquisition module is used to read cloud resource bill data through Flink, aggregate and count according to the supplier account dimension, obtain the cost amount of each account, and match and verify the cost amount with the cost amount on the crawled supplier console; The verification module is used to trigger an alarm if the cost amount of the supplier account is inconsistent with the cost amount on the crawled supplier console, reminding the setting personnel to handle it and end the step; If the cost amount of the supplier account is consistent with the cost amount on the crawled supplier console, the project, cost, and product tags of the cloud resource bill data are cleaned and tagged based on the Aviator rule engine; The display module is used to display the cleaned and labeled cloud resource bill data.
5. The cloud resource cost reconciliation device according to claim 4, characterized in that: The data acquisition module is specifically: Through the access key, query the cloud resource billing data of the corresponding supplier through BigQuery, and then export and download it to the storage bucket S3; generate a CSV file by calling the API billing interface of major cloud vendors, and then summarize it to the cloud resource middle platform, and then forward it to the storage bucket S3.
6. The cloud resource cost reconciliation device according to claim 4, characterized in that: If the cost amount of the supplier account is consistent with the cost amount on the crawled supplier console, the project, cost and product tags of the cloud resource bill data are cleaned and labeled based on the Aviator rule engine. Specifically: if the cost amount of the supplier account is consistent with the cost amount on the crawled supplier console, a unified storage specification is specified, a bill table is set, and cost calculation is performed according to the set calculation logic; the parameters involved, the parameters that need to be labeled, the condition information set and the result information set are determined; the corresponding rule expressions and execution logic are configured; Based on the Aviator rule engine, cloud resource billing data is cleaned and labeled according to rule expressions and execution logic.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 3 is implemented.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.
Citation Information
Patent Citations
Public cloud customer cost anomaly detection system
CN110930183A
Resource consumption statistical method, system and device and readable storage medium
CN113283856A
Cloud resource bill analysis method and platform
CN114648399A
Cloud resource cost analysis method and device applied to multi-cloud management system
CN114756574A
Reconciliation method and device, terminal and storage medium
CN115392892A
Cited By
Bill sharing system, method and equipment
CN119850351A
