Resource processing method and device in target cluster, equipment and storage medium

Abstract

The invention discloses a resource processing method and device in a target cluster, equipment and a storage medium, and the method comprises the steps: obtaining resource data consumed by the target cluster in a specified time period, and determining at least one business volume associated with the resource data; generating unit consumption resources according to the resource data and the business volume; for any data table in the target cluster, obtaining service data generated by the data table in the specified time period, and determining table granularity resources corresponding to the data table based on the service data and the unit consumption resources; and identifying each data table contained in the target architecture, and determining the architecture granularity resource of the target architecture based on the table granularity resource of each data table. According to the technical scheme provided by one or more embodiments, the use efficiency of the resources in the cluster can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a resource processing method, device, equipment and storage medium in a target cluster. Background Art

[0002] At present, in order to realize application scenarios such as multi-dimensional data analysis, machine learning model evaluation, microservice monitoring and statistics, data clusters can be built based on database management systems.

[0003] In actual applications, business parties can share various resources in the data cluster. Generally speaking, in order to meet business needs, business parties tend to use as many resources in the data cluster as possible. When there are many business parties or the data cluster resources are limited, it will cause the problem of resource shortage in the data cluster.

[0004] In view of this, a method for processing resources within a data cluster is currently needed to facilitate planning of resource usage ratios within the data cluster, thereby improving resource usage efficiency. Summary of the invention

[0005] In view of this, one or more embodiments of the present disclosure provide a method, apparatus, device, and storage medium for processing resources in a target cluster, which can improve the resource utilization efficiency within the cluster.

[0006] On the one hand, the present disclosure provides a method for resource processing in a target cluster, the method comprising: obtaining resource data consumed by the target cluster within a specified time period, and determining at least one business volume associated with the resource data; generating unit consumption resources based on the resource data and the business volume; for any data table in the target cluster, obtaining the business data generated by the data table within the specified time period, and determining the table granularity resources corresponding to the data table based on the business data and the unit consumption resources; identifying each data table included in a target architecture, and determining the architecture granularity resources of the target architecture based on the table granularity resources of each data table.

[0007] On the other hand, the present disclosure also provides a resource processing device in a target cluster, the device comprising: a data processing unit, used to obtain resource data consumed by the target cluster within a specified time period, and determine at least one business volume associated with the resource data; a unit resource generation unit, used to generate unit consumption resources based on the resource data and the business volume; a granular resource determination unit, used to obtain, for any data table in the target cluster, the business data generated by the data table within the specified time period, and determine the table granularity resources corresponding to the data table based on the business data and the unit consumption resources; a resource determination unit, used to identify each data table included in the target architecture, and determine the architecture granularity resources of the target architecture based on the table granularity resources of each data table.

[0008] On the other hand, the present disclosure further provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the resource processing method in the target cluster is implemented.

[0009] On the other hand, the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the resource processing method in the target cluster is implemented.

[0010] The technical solution provided by one or more embodiments of the present disclosure can determine the unit consumption resources related to the business volume based on the resource data consumed by the target cluster within a specified time period. In order to measure the architecture granularity resources used by different architectures in the target cluster, the table granularity resources of each data table in the target cluster can be determined starting from the dimension of the data table. Subsequently, by aggregating the table granularity resources, the architecture granularity resources can be formed. By analyzing the architecture granularity resources, the resources actually used by different architectures can be determined, providing an accurate data basis for subsequent resource planning, thereby improving the resource utilization rate in the cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0012] Figure 1 A schematic diagram of the steps of a method for processing resources in a target cluster in one embodiment of the present disclosure is shown;

[0013] Figure 2 A flow chart of a resource processing method in a specific application example of the present disclosure is shown;

[0014] Figure 3A schematic diagram of functional modules of a resource processing device in a target cluster in one embodiment of the present disclosure is shown;

[0015] Figure 4 A schematic structural diagram of an electronic device in one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0017] An embodiment of the present disclosure provides a method for processing resources in a target cluster. Figure 1 , the method may include the following steps.

[0018] S1: Obtain resource data consumed by a target cluster within a specified period of time, and determine at least one service volume associated with the resource data.

[0019] In this embodiment, the target cluster can be any cluster that requires resource planning. The specified time period can limit the time period in which the data to be analyzed is located. In practical applications, the specified time period can be flexibly determined according to the needs of data analysis. For example, the specified time period can be 24 hours, a week, a month, and so on. Generally speaking, data analysis of the target cluster is periodic. After setting the specified time period, the duration corresponding to the specified time period can be used as the data analysis cycle. At the beginning of each data analysis cycle, the data analysis process of the target cluster can be started. Of course, periodic analysis of the target cluster is only one possible implementation method. In some application scenarios, the timing of data analysis can also be started according to actual needs. The present disclosure does not limit the timing of data analysis.

[0020] In this embodiment, the resource data consumed by the target cluster in a specified period of time can be represented by a structured data file. After the resource data is generated, the data warehouse tool in the target cluster can map the structured data file to a database table. By reading the information in the database table, the resource data consumed by the target cluster in the specified period of time can be obtained. For example, the data warehouse tool hive can be deployed in the target cluster. After hive maps the structured data file, a hive table can be obtained. By reading the content in the field representing the resource data in the hive table, the resource data consumed by the target cluster in the specified period of time can be obtained.

[0021] In this embodiment, resource data is often caused by the business in the target cluster. For example, storage business and query business can be generated in the target cluster, then along with the storage business and query business, consumed resource data will be generated. In view of this, after obtaining the resource data consumed by the target cluster in a specified period of time, at least one business volume associated with the resource data can also be determined. Among them, the correlation between business volume and resource data can be reflected in: when the business volume changes, it can cause changes in resource data to a certain extent. The amount of business volume associated with resource data in the target cluster often depends on the actual number of businesses in the target cluster. For example, the target cluster mainly provides two businesses, data storage and data query, then the two business volumes associated with the resource data can be storage volume and query volume. By identifying the actual business of the target cluster, the amount of business volume can be determined.

[0022] In practical applications, after determining the amount of business volume, the actual value of each business volume can usually be determined at the same time. For example, assuming that the business volume includes storage volume and query volume, the actual value of storage volume can be expressed by Gb (Gigabyte), and the actual value of query volume can be expressed by thousands of queries. It should be noted that the actual value of each business volume is also counted within a specified period of time, so that it can match the resource data consumed by the target cluster within the specified period of time.

[0023] S3: Generate unit consumption resources according to the resource data and the business volume.

[0024] In this implementation, after determining the resource data consumed by the target cluster within a specified period and acquiring various traffic volumes associated with the resource data, unit consumption resources corresponding to different traffic volumes may be generated.

[0025] Specifically, taking two business volumes as an example, assuming that at least one business volume obtained in step S1 includes a first business volume and a second business volume, then when generating unit consumption resources, the first business weight of the first business volume and the second business weight of the second business volume can be determined respectively. Generally speaking, the sum of the weights of the first business weight and the second business weight can be 1, and the specific values ​​of the first business weight and the second business weight can be preset by the administrator of the target cluster. For example, in the scenario of storage and query, the first business volume can be the data storage volume, and the second business volume can be the data query volume, and the business weights of each of these two business volumes can be 0.5. Of course, with different business scenarios, the type of business volume may also be different, and the business weights corresponding to different business volumes will also be different. The present disclosure does not limit the type of business volume and the setting of the corresponding business weight.

[0026] In this implementation, the resource data can be divided into the first business resource and the second business resource according to the first business weight and the second business weight. In practical applications, the product of the resource data and the first business weight can be used as the first business resource; similarly, the product of the resource data and the second business weight can be used as the second business resource.

[0027] In this implementation, after determining the first business resource and the second business resource, a first unit consumption resource may be generated based on the first business resource and the first business volume, and a second unit consumption resource may be generated based on the second business resource and the second business volume. Specifically, the ratio of the first business resource to the first business volume may be used as the first unit consumption resource. Similarly, the ratio of the second business resource to the second business volume may be used as the second unit consumption resource.

[0028] It should be noted that the above implementation method only takes two business volumes as an example to explain the technical solution of the present disclosure. This does not mean that the technical solution of the present disclosure can only be applied in the scenario of two business volumes. It can be understood by those skilled in the art that no matter how many business volumes there are, the technical solution of the present disclosure can be implemented by setting the business weights of each business volume, and these extended application scenarios should also fall within the protection scope of the present disclosure.

[0029] It can be seen from the above implementation that corresponding unit consumption resources can be generated for different business volumes. For example, for storage volume, resource data consumed per 1Gb of storage volume can be generated; for query volume, resource data consumed per 1,000 queries can be generated.

[0030] S5: For any data table in the target cluster, obtain the business data generated by the data table within the specified time period, and determine the table granularity resources corresponding to the data table based on the business data and the unit consumption resources.

[0031] In this embodiment, in order to analyze the resource data generated by different architectures in the target cluster, the data table in the target cluster can be used as the minimum measurement unit to calculate the table granularity resources of each data table.

[0032] Specifically, for any data table in the target cluster, the business data generated by the data table within a specified period of time can be obtained. The business data can correspond to the type of business volume in steps S1 and S3. For example, assuming that the business volume includes storage volume and query volume, the business data generated by the data table can be storage data and query data.

[0033] In this embodiment, the business data generated by the data table within a specified period of time can characterize the resource consumption degree of the data table within the specified period of time. Generally speaking, the specific data volume of the business data can be proportional to the resource consumption degree. By calculating the resource consumption of the data table within the specified period of time, the table granularity resource of the data table can be obtained. Specifically, if there is only one business data generated by the data table, the product of the specific data volume of the business data and the unit consumption resource corresponding to the business data can be used as the table granularity resource corresponding to the data table. If there are multiple business data generated by the data table, the specific data volume of each business data can be multiplied by the unit consumption resource corresponding to the business data, and the sum of each product can be used as the table granularity resource corresponding to the data table.

[0034] For example, in one embodiment, assuming that the business data generated by the data table includes first business data and second business data, then when generating the table granularity resources of the data table, the first business data can be identified from the business data, and the first unit consumption resources matching the first business data can be identified from the unit consumption resources. For example, if the first business data is storage data, then the first unit consumption resources can be the unit consumption resources corresponding to each 1Gb of storage data. Based on the first business data and the first unit consumption resources, the first granularity resources of the data table can be determined. The first granularity resources can be the product of the specific data volume of the first business data and the first unit consumption resources. Similarly, the second business data can be identified from the business data, and the second unit consumption resources matching the second business data can be identified from the unit consumption resources. Then, the second granularity resources of the data table can be determined based on the second business data and the second unit consumption resources.

[0035] After obtaining the first granularity resource and the second granularity resource, the first granularity resource and the second granularity resource may be aggregated to generate a table granularity resource corresponding to the data table, wherein the aggregation method may be to add the first granularity resource to the second granularity resource.

[0036] S7: Identify each data table included in the target architecture, and determine the architecture granularity resources of the target architecture based on the table granularity resources of each data table.

[0037] In this embodiment, after determining the table granularity resources of each data table in the target cluster, the table granularity resources of these data tables can be aggregated according to the data tables included in the target architecture, so as to calculate the architecture granularity resources of the target architecture, which can characterize the resource consumption generated by the target architecture in a specified period of time. By accurately calculating the resource consumption of different architectures, a data basis can be provided for subsequent resource planning, thereby improving the utilization rate of resources.

[0038] In this embodiment, the data tables in the target cluster often have ownership information, which can represent the owner of the data table. The target architecture also often covers a part of the ownership information. The ownership information covered in the target architecture can be divided based on the actual business or flexibly divided based on the preset strategy. For example, if the target architecture represents the Internet of Things business, then the ownership information associated with the Internet of Things business can be aggregated under the target architecture.

[0039] In order to count the architecture granularity resources of the target architecture, one or more target attribution information covered in the target architecture can be identified first, and then starting from the dimension of the data table, the data tables with the target attribution information are counted, and the statistically obtained data tables are used as the data tables included in the target architecture. For example, the target architecture contains attribution information A, attribution information B, and attribution information C, and the attribution information A is included in data table 1 and data table 2, the attribution information B is included in data table 3, and the attribution information C is included in data table 4 and data table 5, then data tables 1 to 5 can be used as the data tables included in the target architecture.

[0040] After determining the data tables included in the target architecture, the table granularity resources of each data table can be aggregated by resource aggregation to obtain the architecture granularity resources of the target architecture. In practical applications, the aggregation method can be a method of accumulating table granularity resources.

[0041] From the above methods, it can be seen that the target architecture can be flexibly set according to actual needs. Although the target architecture is relatively complex in actual applications, it can ultimately be implemented in the dimensions of the data table for resource statistics, thereby obtaining relatively accurate architecture granularity resources and providing an accurate data basis for subsequent resource planning.

[0042] In one embodiment, the target cluster is usually constructed according to a certain hierarchy. For example, the hierarchy can be represented from top to bottom as: cluster, database, data table. Correspondingly, the ownership information of the data table can also be represented by the hierarchical ownership information, and the hierarchical ownership information can include at least one of the table ownership information, the library ownership information and the cluster ownership information. Generally speaking, when obtaining the ownership information of the data table, the table ownership information can be directly read by reading the information export table in the cluster. But sometimes the table ownership information may not exist, then you can continue to try to read the library ownership information, and if the library ownership information still does not exist, you can read the final cluster ownership information. It can be seen that different hierarchical ownership information can have different priorities for the data table, among which the table ownership information has the highest priority and the cluster ownership information has the lowest priority. If the table ownership information can be read, the library ownership information or the cluster ownership information will often not be used as the ownership information of the data table.

[0043] That is to say, for any data table in the target cluster, one or more hierarchical belonging information of the data table can be read, and the hierarchical belonging information includes at least one of table belonging information, library belonging information and cluster belonging information. Then, according to the priority of each hierarchical belonging information, the belonging information of the data table can be determined in the one or more hierarchical belonging information. The purpose of such processing is to ensure that each data table will have its own belonging information, so that accurate architecture granularity resources can be statistically obtained later.

[0044] In a specific application example, the target cluster can be a clickhouse cluster, and the resources that need to be planned can be the cost resources in the clickhouse cluster. Generally speaking, in order to meet actual R&D needs, R&D personnel tend to use as many cluster resources as possible, which sometimes leads to unreasonable use of costs. In view of this, the cost usage of the clickhouse cluster can be analyzed and counted, and then reasonable cost planning can be completed.

[0045] In the clickhouse cluster, the cost consumption is mainly caused by storage business and query business, so the business volume associated with the cost resource can be storage volume and query volume. The storage volume is counted in Gb, and the query volume is counted in thousands of queries. The specified period can be 24 hours, and the data analysis cycle is daily. Please refer to Figure 2 By reading the hive table in the clickhouse cluster, you can get the cost data with daily granularity. At the same time, by reading the hive table, you can also know the query volume and storage volume of clickhouse on that day. In actual applications, the business weights of storage business and query business can be 0.5 respectively. Then, according to the business weights, the cost consumption of storage business and query business can be obtained respectively. By calculating the ratio of cost consumption to actual business volume, the storage unit cost and query unit cost can be obtained.

[0046] In practical applications, in order to count the actual business data of the data table, the duration of the specified time period can be used as a cycle to create a regularly executed task (cronjob). When the regularly executed task is executed, the information access interface of the target cluster can be called to obtain the first business data generated by the data table within the specified time period.

[0047] The information access interface may be a metadata interface provided by the clickhouse cluster, through which the data storage volume of the data table on the day can be obtained. At the same time, the attribution information corresponding to the data table can also be obtained.

[0048] In addition, you can also read the query log of the target cluster and read the second business data generated by the data table in the specified period from the query log. The query log can be recorded in the hive table, and by reading the query log, you can get the data query volume of the data table on that day.

[0049] like Figure 2 As shown, in actual applications, when obtaining the first business data of a data table, the library information in the target cluster can be read first according to the hierarchical structure in the target cluster, and the library information can represent the information of each database in the target cluster. For any database recorded in the library information, the table information of the database can be read. The table information can represent the information of each data table stored in the database. Through the data table recorded in the table information, the data storage capacity of the data table in the current cycle can be read, and the data storage capacity can be used as the first business data.

[0050] In this application example, after determining the storage unit cost and query unit cost of the clickhouse cluster, and obtaining the actual data storage volume and data query volume of the data table in clickhouse on that day, the product of the data storage volume and the storage unit cost can be used as the first granularity resource of the data table, and the product of the data query volume and the query unit cost can be used as the second granularity resource of the data table. The sum of the first granularity resource and the second granularity resource can be used as the table granularity resource of the data table.

[0051] Subsequently, based on the attribution information contained in the business line or team using the ClickHouse cluster, the actual cost consumption of the business line or team can be calculated. Based on the calculated cost consumption, the cost of the ClickHouse cluster can be reasonably planned to improve the cost conversion rate.

[0052] The technical solution provided by one or more embodiments of the present disclosure can determine the unit consumption resources related to the business volume based on the resource data consumed by the target cluster within a specified time period. In order to measure the architecture granularity resources used by different architectures in the target cluster, the table granularity resources of each data table in the target cluster can be determined starting from the dimension of the data table. Subsequently, by aggregating the table granularity resources, the architecture granularity resources can be formed. By analyzing the architecture granularity resources, the resources actually used by different architectures can be determined, providing an accurate data basis for subsequent resource planning, thereby improving the resource utilization rate in the cluster.

[0053] See also Figure 3 An embodiment of the present disclosure further provides a resource processing device in a target cluster, the device comprising:

[0054] The data processing unit 100 is configured to obtain resource data consumed by the target cluster within a specified period of time, and determine at least one service volume associated with the resource data;

[0055] The unit resource generation unit 200 is used to generate unit consumption resources according to the resource data and the business volume;

[0056] The granular resource determination unit 300 is used to obtain, for any data table in the target cluster, the business data generated by the data table within the specified time period, and determine the table granular resource corresponding to the data table based on the business data and the unit consumption resource;

[0057] The resource determination unit 400 is used to identify each data table included in the target architecture, and determine the architecture granularity resources of the target architecture based on the table granularity resources of each data table.

[0058] In one embodiment, the at least one business volume includes a first business volume and a second business volume; the unit resource generation unit 200 is specifically used to determine a first business weight of the first business volume and a second business weight of the second business volume, respectively; according to the first business weight and the second business weight, the resource data is divided into a first business resource and a second business resource; based on the first business resource and the first business volume, a first unit consumption resource is generated, and based on the second business resource and the second business volume, a second unit consumption resource is generated.

[0059] In one embodiment, the granular resource determination unit 300 is specifically used to create a periodic execution task with the duration of the specified time period as a period. When the periodic execution task is executed, the information access interface of the target cluster is called to obtain the first business data generated by the data table within the specified time period; the query log of the target cluster is read, and the second business data generated by the data table within the specified time period is read from the query log.

[0060] In one embodiment, the granular resource determination unit 300 is specifically used to read the library information in the target cluster; for any database recorded in the library information, read the table information of the database; for the data table recorded in the table information, read the data storage capacity of the data table in the current cycle, and use the data storage capacity as the first business data.

[0061] In one embodiment, the granularity resource determination unit 300 is specifically used to identify first business data from the business data, and identify first unit consumption resources matching the first business data from the unit consumption resources; determine the first granularity resources of the data table based on the first business data and the first unit consumption resources; identify second business data from the business data, and identify second unit consumption resources matching the second business data from the unit consumption resources; determine the second granularity resources of the data table based on the second business data and the second unit consumption resources; aggregate the first granularity resources and the second granularity resources to generate a table granularity resource corresponding to the data table.

[0062] In one embodiment, the resource determination unit 400 is specifically used to determine the ownership information of each data table in the target cluster and identify the target ownership information covered in the target architecture; count the data tables with the target ownership information, and use the data tables obtained by counting as the data tables included in the target architecture.

[0063] In one embodiment, the resource determination unit 400 is further specifically used to read one or more hierarchical affiliation information of any data table in the target cluster, wherein the hierarchical affiliation information includes at least one of table affiliation information, library affiliation information and cluster affiliation information; and determine the affiliation information of the data table in the one or more hierarchical affiliation information according to the priority of each hierarchical affiliation information.

[0064] Each unit described in the above embodiments may be implemented by a computer chip or a product having a certain function. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0065] For the convenience of description, the above devices are described in terms of functions and are described separately in various units. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0066] See also Figure 4 The present disclosure also provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the resource processing method in the target cluster is implemented.

[0067] The present disclosure also provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the resource processing method in the target cluster is implemented.

[0068] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0069] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present invention. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiment.

[0070] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0071] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0072] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, equipment, and storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0073] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

[0074] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for processing resources in a target cluster, It is characterized in that The method comprises: Obtain resource data consumed by the target cluster within a specified period of time, and determine at least one business volume associated with the resource data; Generate unit consumption resources according to the resource data and the business volume; For any data table in the target cluster, obtain the business data generated by the data table within the specified time period, and determine the table granularity resources corresponding to the data table based on the business data and the unit consumption resources; Identify each data table included in the target architecture, and determine the architecture granularity resources of the target architecture based on the table granularity resources of each data table.

2. The method according to claim 1, It is characterized in that The at least one traffic volume includes a first traffic volume and a second traffic volume; Generating unit consumption resources according to the resource data and the business volume includes: respectively determining a first service weight of the first service volume and a second service weight of the second service volume; Dividing the resource data into first business resources and second business resources according to the first business weight and the second business weight; A first unit consumption resource is generated based on the first service resource and the first service volume, and a second unit consumption resource is generated based on the second service resource and the second service volume.

3. The method according to claim 1 or 2, It is characterized in that Acquiring the business data generated by the data table within the specified time period includes: Creating a periodic execution task with the duration of the specified time period as a period, and when the periodic execution task is executed, calling the information access interface of the target cluster to obtain the first business data generated by the data table within the specified time period; The query log of the target cluster is read, and the second business data generated by the data table within the specified time period is read from the query log.

4. The method according to claim 3, It is characterized in that Acquiring the first business data generated by the data table within the specified time period includes: Reading library information in the target cluster; For any database recorded in the library information, read the table information of the database; For the data table recorded in the table information, the data storage volume of the data table in the current cycle is read, and the data storage volume is used as the first business data.

5. The method according to claim 1 or 2, It is characterized in that Determining the table granularity resource corresponding to the data table includes: Identifying first business data from the business data, and identifying first unit consumption resources matching the first business data from the unit consumption resources; Determine a first granularity resource of the data table according to the first business data and the first unit consumption resource; Identifying second business data from the business data, and identifying second unit consumption resources matching the second business data from the unit consumption resources; Determine a second granularity resource of the data table according to the second business data and the second unit consumption resource; The first granularity resource and the second granularity resource are aggregated to generate a table granularity resource corresponding to the data table.

6. The method according to claim 1 or 2, It is characterized in that Identify the various data tables contained in the target schema including: Determine the ownership information of each data table in the target cluster, and identify the target ownership information covered in the target architecture; Count the data tables having the target attribution information, and use the data tables obtained by counting as the data tables included in the target architecture.

7. The method according to claim 6, It is characterized in that Determining the ownership information of each data table in the target cluster includes: For any data table in the target cluster, read one or more hierarchical belonging information of the data table, where the hierarchical belonging information includes at least one of table belonging information, library belonging information and cluster belonging information; According to the priorities of the respective hierarchical attribution information, the attribution information of the data table is determined in the one or more hierarchical attribution information.

8. A resource processing device in a target cluster, It is characterized in that The device comprises: a data processing unit, configured to obtain resource data consumed by the target cluster within a specified period of time, and determine at least one business volume associated with the resource data; A unit resource generation unit, configured to generate unit consumption resources according to the resource data and the business volume; a granular resource determination unit, configured to obtain, for any data table in the target cluster, business data generated by the data table within the specified time period, and determine, based on the business data and the unit consumption resources, a table granular resource corresponding to the data table; The resource determination unit is used to identify each data table included in the target architecture, and determine the architecture granularity resources of the target architecture based on the table granularity resources of each data table.

9. An electronic device, It is characterized in that The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer storage medium, It is characterized in that The computer storage medium is used to store a computer program, and when the computer program is executed by a processor, it implements the method according to any one of claims 1 to 7.

Images