A resource management method, electronic device and computer-readable storage medium
By dynamically adjusting the number of containers, the blocking and idle problems in resource management are solved based on the rewrite cost information of the target table data in the data lake and the files to be rewritten, and the resource utilization rate and the stability of the data lake are improved.
Patent Information
- Application Number
- CN202411092063.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-09
AI Technical Summary
In the resource management of data lakes, due to improper resource allocation, resources are easily blocked or idle, affecting the normal operation of the platform.
By obtaining the rewriting cost information of the target table data in the data lake and the currently rewritten file, and combining the number of optimized containers and rewriting time of the preset resource group, the number of optimized containers is dynamically adjusted to ensure the reasonable allocation and use of resources.
It effectively avoids resource blockage and idleness, improves the efficiency of file rewriting, and realizes resource isolation management, ensuring the stable operation of the data lake.
Smart Images

Figure CN118626211B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of resource scheduling, and in particular to a resource management method, electronic equipment and computer-readable storage medium. Background Art
[0002] A data lake is a centralized repository for storing large amounts of raw data. It allows data to be stored in its original format without prior structuring, including structured data, semi-structured data, unstructured data, and binary data, etc., which places high demands on the management of the data lake. Due to the dynamic nature of data, the resources required in the process of maintaining data are also different.
[0003] At present, when managing a large amount of data of multiple types, since the current platform executes data management tasks based on the resources allocated by the current system, when there are a large number of data management tasks to be executed, it is easy to cause blockage in the application of resources. When there are fewer data management tasks to be executed, there will be a large amount of idle resources, which will affect the overall operation of the platform and is not conducive to the normal use needs of the platform. Summary of the invention
[0004] The main technical problem solved by the present application is to provide a resource management method, an electronic device and a computer-readable storage medium, which can ensure the appropriateness of resource allocation.
[0005] In order to solve the above technical problems, a technical solution adopted by the present application is: a resource management method is provided, and the resource management method includes: obtaining target table data corresponding to the same preset resource group in the data lake and the current file to be rewritten in the target table data; determining the current rewrite time of the current file to be rewritten in the target table data according to the current rewrite cost of the current file to be rewritten and the historical rewrite cost information of the historical rewrite file in the target table data, and the rewrite cost represents the resource data required to rewrite the corresponding file to be rewritten; determining the target number of optimization containers to be called from the preset resource group according to the preset optimization time of the optimization container in the preset resource group and the current rewrite time; and rewriting the current file to be rewritten in the target table data with the target number.
[0006] In some embodiments, the step of determining a target number of optimization containers to be called from the preset resource group according to the preset optimization time of the optimization containers in the preset resource group and the current rewrite time includes: determining an initial number of optimization containers to be called according to the preset optimization time of the optimization containers in the preset resource group and the current rewrite time; and determining the target number of optimization containers to be called according to the preset number of optimization containers in the preset resource group and the initial number of optimization containers to be called.
[0007] In some embodiments, the step of determining the target number of the optimization containers to be called according to the preset number of optimization containers in the preset resource group and the initial number of the optimization containers to be called includes: in response to the initial number of the optimization containers to be called being less than or equal to the preset number of optimization containers in the preset resource group, determining the initial number of the optimization containers to be called as the target number of the optimization containers to be called; in response to the initial number of the optimization containers to be called being greater than the preset number of optimization containers in the preset resource group, determining the preset number of optimization containers in the preset resource group as the target number of the optimization containers to be called.
[0008] In some embodiments, after determining the target number of optimized containers to be called from the preset resource group according to the preset optimization time of the optimized containers in the preset resource group and the current rewrite time, the resource management method further includes: using the historical number of optimized containers called when rewriting the historical rewrite file as the current number of optimized container configurations in the preset resource group; in response to the current number of optimized container configurations being greater than the target number of optimized containers to be called, reducing the current number of optimized container configurations until the number of optimized containers in the preset resource group is reduced from the current number of optimized container configurations to the target number; in response to the current number of optimized container configurations being less than the target number of optimized containers to be called, expanding the current number of optimized container configurations until the number of optimized containers in the preset resource group is expanded from the current number of optimized container configurations to the target number.
[0009] In some embodiments, the step of obtaining target table data corresponding to the same preset resource group in the data lake and the current file to be rewritten in the target table data includes: determining the target table data according to the rewrite time interval of each table data corresponding to the same preset resource group in the data lake; determining the current file to be rewritten of the target table data according to the file attribute information of each file in the target table data.
[0010] In some embodiments, the step of determining the target table data based on the rewrite time interval of each table data corresponding to the same preset resource group in the data lake includes: obtaining the historical rewrite time point at which the historical files to be rewritten corresponding to each table data in the preset resource group are started to be rewritten; determining the time difference between the historical rewrite time point corresponding to each table data and the current time point as the rewrite time interval of the corresponding table data, the current time point representing the time point at which the current file to be rewritten in the corresponding table data is started to be rewritten; in response to the rewrite time interval of each table data being greater than the preset rewrite time interval of the corresponding table data, determining the corresponding table data as the target table data in the preset resource group.
[0011] In some embodiments, the file attribute information includes the occupied space of the file, and the step of determining the current file to be rewritten in the target table data according to the file attribute information of each file in the target table data includes: in response to the occupied space of each file in the target table data being less than the preset file occupied space, determining the file whose occupied space is less than the preset file occupied space as the current file to be rewritten in the target table data; in response to the occupied space of each file in the target table data being greater than or equal to the preset file occupied space, obtaining the historical data deletion amount of the file whose occupied space is greater than or equal to the preset file occupied space; in response to the historical data deletion amount of the file whose occupied space is greater than or equal to the preset file occupied space is greater than a preset data deletion threshold, determining the file whose historical data deletion amount is greater than the preset data deletion threshold as the file to be rewritten in the target table data.
[0012] In some embodiments, the historical rewrite cost information includes historical rewrite time and historical rewrite cost, and the step of determining the current rewrite time of the current file to be rewritten in the target table data based on the current rewrite cost of the current file to be rewritten and the historical rewrite cost information of the historical rewrite file in the target table data includes: determining the ratio between the historical rewrite time and the historical rewrite cost as the unit rewrite time; and determining the product between the unit rewrite time and the current rewrite cost of the current file to be rewritten as the current rewrite time of the current file to be rewritten.
[0013] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, including a memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the above resource management method.
[0014] In order to solve the above technical problem, another technical solution adopted by the present application is: providing a computer-readable storage medium including program data stored therein, wherein the program data is used to implement the above resource management method when executed by a processor.
[0015] Compared with the current method of managing data by using a fixed number of optimized containers, which is prone to causing resource congestion or resource idleness, the above scheme proposes a resource management method in an embodiment of the present application, which includes: obtaining target table data corresponding to the same preset resource group in the data lake and the current file to be rewritten in the target table data; determining the current rewrite time of the current file to be rewritten in the target table data according to the current rewrite cost of the current file to be rewritten and the historical rewrite cost information of the historical rewrite file in the target table data, wherein the rewrite cost represents the resource data required to rewrite the corresponding file to be rewritten; determining the target number of optimized containers to be called from the preset resource group according to the preset optimization time and the current rewrite time of the optimized container in the preset resource group; and rewriting the current file to be rewritten in the target table data with the target number. Therefore, the target number of optimization containers required to rewrite the current file to be rewritten can be determined through the current rewrite time of the current file to be rewritten and the preset optimization time of the optimization container, and the current file to be rewritten is rewritten by the target number of optimization containers, thereby ensuring the efficiency of file rewriting; at the same time, the optimization container in the same preset resource group is called to rewrite the file to be rewritten under the preset resource group, thereby realizing resource isolation. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:
[0017] Figure 1 It is a flowchart of an exemplary embodiment of a resource management method shown in the present application;
[0018] Figure 2 yes Figure 1 A flowchart of an exemplary embodiment of step S130 in the resource management method is shown;
[0019] Figure 3 yes Figure 1 A schematic flow chart of an exemplary embodiment of the resource management method after step S130 is shown;
[0020] Figure 4 yes Figure 1 A flow chart of an exemplary embodiment of step S110 in the resource management method is shown;
[0021] Figure 5 is a flowchart of an exemplary embodiment of obtaining the current file to be rewritten in the target table data in the embodiment of the present application;
[0022] Figure 6 yes Figure 1 A flow chart of an exemplary embodiment of step S120 in the resource management method is shown;
[0023] Figure 7 is a flow chart of an exemplary embodiment of optimizing the determination of the number of containers in the embodiment of the present application;
[0024] Figure 8 It is a schematic diagram of a framework of an exemplary embodiment of rewriting tasks and optimizing containers in an optimization server in an embodiment of the present application;
[0025] Fig. 9 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application;
[0026] Fig.10 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION
[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be appreciated that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the art without making creative work are within the scope of protection of the present application.
[0028] First of all, it should be noted that the mainstream data lake frameworks include Hudi (Hadoop Upsert DeleteIncremental), Delta Lake, and Iceberg (Apache Iceberg). Their data management functions are all based on a series of meta files (metadata files). The framework itself provides some operation and maintenance functions that need to be triggered manually. For example, Iceberg supports the use of Java ap and Spark to delete expired snapshots, delete old metadata files, delete orphan files, and rewrite data files and rewrite manifest files. In actual application scenarios, the table data in the data lake is hundreds, thousands, or even tens of thousands, which will inevitably bring huge operation and maintenance costs. In addition, when there are more table data to be optimized, more maintenance resources may be required to ensure maintenance performance. On the contrary, it is necessary to save costs by reducing the maintenance resource usage.
[0029] Based on this, the present application proposes a resource management method, an electronic device and a computer-readable storage medium. Figure 1 , Figure 1It is a flowchart of an exemplary embodiment of the resource management method shown in the present application.
[0030] The execution subject of the resource management method may be a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The execution subject of the resource management method may also be a resource management device. In some possible implementations, the resource management method may be implemented by a processor calling a computer-readable instruction stored in a memory.
[0031] Specifically, the resource management method of this embodiment includes the following steps:
[0032] S110: Obtain target table data corresponding to the same preset resource group in the data lake and the current file to be rewritten in the target table data.
[0033] A data lake is a centralized repository for storing large amounts of raw data, allowing data to be stored in its original format. Exemplarily, a data lake can store structured data (such as tables in relational databases), semi-structured data (such as CSV, logs, XML, JSON), unstructured data (such as emails, documents, PDFs), and binary data (such as graphics, audio, and video). The table data created by users in the front-end Web UI, as well as the new data generated by optimizing the server and optimizing the container to manage the table data, will be stored in the data lake. The data lake can not only bring together different types of data, but also perform data analysis without the need for predefined models. At the same time, the data lake also ensures data persistence and security, allowing enterprises to save original data, avoid data silos, support real-time data, and improve operational efficiency.
[0034] A preset resource group is a logical container used to organize and manage associated resources. Declarative configuration files can be used to automatically create, update, and delete resources within the entire resource group. For example, users can configure preset resource groups in the front-end Web UI, and can specify the upper and lower limits of available resources in the preset resource group and the preset scheduling policy of the preset resource group. The resources include hardware resources, central processing unit (CPU) resources, and memory, etc. The preset scheduling policy refers to the scheduling sorting order of table data in the preset resource group. In addition, when creating table data, the user will specify the preset resource group to which it belongs. The default is default. When running optimization on table data, only the optimization container in the same preset resource group can be called to achieve resource isolation of each preset resource group.
[0035] The target table data is the table data that needs to be rewritten in the preset resource group. Exemplarily, the user can create table data in the front-end WebUI and specify the preset resource group to which the table data belongs, and determine the target table data based on the data management information of the table data. In addition, the user can also configure the data source information (catalog) in the front-end Web UI. As an example, the HMS Catalog (Hive metadata storage) can be configured, and the hive-site.xml, hdfs-site.xml and core-site.xml files need to be imported; if kerberos (Kerberos Authentication Protocol) and other authentication are enabled, then the authentication-related files need to be imported. In kerberos, the communication between users and services needs to go through a trusted third party, namely the kerberos authentication server (KDC, Key Distribution Center).
[0036] Files are various files generated in the process of table data management. Exemplarily, files in table data include metadata files, orphan files, etc. Metadata files are used to store information such as the structure, attributes, context, and management information of table data. Orphan files refer to any data left by programs that have been uninstalled or no longer run in table data. The current file to be rewritten refers to the file that needs to be rewritten in the target table data. Exemplarily, a file in the target table data that occupies a smaller space or has more data deleted in the file can be determined as the current file to be rewritten. Exemplarily, the current file to be rewritten can be one or more.
[0037] The resource management device creates at least one preset resource group, then creates table data and specifies the preset resource group to which the table data belongs, obtains target table data in the same preset resource group according to data management information of the table data, and then obtains the current file to be rewritten that the target table data needs to rewrite.
[0038] S120: Determine the current rewrite time of the current file to be rewritten in the target table data according to the current rewrite cost of the current file to be rewritten and the historical rewrite cost information of the historically rewritten files in the target table data, where the rewrite cost represents the resource data required to rewrite the corresponding file to be rewritten.
[0039] The rewriting process can be to merge multiple files to be rewritten into one file. The rewriting cost represents the resource data required to rewrite the corresponding file to be rewritten, that is, the cost or price required to rewrite the file to be rewritten. Among them, the resource data mainly includes computing resource costs, storage costs, etc. The current rewriting cost refers to the resource data required to rewrite the current file to be rewritten in the data lake.
[0040] The historical rewrite file refers to the file rewritten each time in the historical rewrite task. For example, the file rewritten in the last rewrite process can be determined as the historical rewrite file, or the file rewritten in the last rewrite process can be determined as the historical rewrite file, which is not specifically limited here, as long as it is the rewrite file before the current rewrite process.
[0041] The historical rewrite cost information includes various information of rewriting the historical rewrite file. For example, the rewrite cost information generated by the preset resource group in the last rewrite process can be determined as the historical rewrite cost information; the rewrite cost and rewrite time in the rewrite cost information generated in the previous N rewrite processes can also be averaged, and the historical rewrite cost information can be determined based on the average value obtained by calculation.
[0042] The current rewrite time refers to the time cost required to rewrite the current file to be rewritten in the preset resource group. Exemplarily, the current rewrite time can be determined based on the current rewrite cost and the preset unit time consumed by each cost, and the preset consumption time can be determined based on experience. In other embodiments, the current rewrite time can also be determined based on the current rewrite cost and the historical rewrite cost information of the historical rewrite file.
[0043] The resource management device obtains the historical rewrite cost information of the historical rewrite file, and determines the current rewrite time of the current file to be rewritten according to the current rewrite cost of the current file to be rewritten and the historical rewrite cost information of the historical rewrite file.
[0044] S130: Determine a target number of optimization containers to be retrieved from the preset resource group according to the preset optimization time and the current rewriting time of the optimization containers in the preset resource group.
[0045] The optimization container is used to rewrite the files to be rewritten. For example, the optimization container can be deployed on kubernetes (K8s for short) or yarn (Yet Another Resource Negotiator, Yet Another Resource Negotiator), and the container orchestration tool is used to manage the optimization container and allocate resources to ensure that the entire application system can run efficiently and stably. An optimization container can execute multiple tasks to be rewritten at the same time. A rewriting task refers to rewriting multiple files to be rewritten into one file.
[0046] The preset optimization time refers to the optimization time of the optimization container. For example, the optimization time of each optimization container can be set according to actual needs to determine the preset optimization time. Usually, the preset optimization time of the optimization container defaults to 1 hour, which means that when all the optimization containers in the preset resource group cannot complete all rewrite tasks within 1 hour, they need to be expanded. Automatically expand or delete idle optimization containers based on the current number of optimization containers in the preset resource group and the required number of optimization containers.
[0047] The target number refers to the number of optimization containers required to rewrite the current file to be rewritten corresponding to the same preset resource group in the data lake. Exemplarily, the target number of optimization containers can be determined according to the preset optimization time of the optimization container in the preset resource group and the current rewriting time of the current file to be rewritten.
[0048] The resource management device determines the target number of optimization containers required to be called for rewriting the current file to be rewritten in the preset resource group according to the preset optimization time of the optimization container in the preset resource group and the current rewriting time of rewriting the current file to be rewritten.
[0049] S140: rewriting the current to-be-rewritten files in the target table data with the target number.
[0050] The resource management device first sorts the target table data according to the preset scheduling strategy of the preset resource group to obtain the sorted target table data; groups the current files to be rewritten in the same target table data according to the preset rules to obtain the file groups composed of the current files to be rewritten; sorts the current files to be rewritten in the same target table data according to the current rewrite cost of each file group to obtain the sorted file groups; determines the execution order of the file groups according to the sorted target table data and the sorted file groups; and rewrites each file group according to the execution order of each file group by the target number of optimized containers.
[0051] Among them, the preset scheduling strategies include priority, balance, and quota. Priority is directly scheduled according to the priority of the table data, and users can pre-define it in the table properties of the table data. Exemplarily, when the preset scheduling strategy of the preset resource group is priority, the target table data can be directly sorted from small to large according to its priority; balance means that table data that has not been self-optimized for a long time has a higher scheduling priority. Exemplarily, when the preset scheduling strategy of the preset resource group is balance, the last optimization time point of the target table data is obtained, and the target table data is sorted from large to small according to the time difference between the last optimization time point and the current time point; quota is a scheduling strategy based on actual resource usage, which users can specify in the table properties of the table data, ranging from 0-1, and the lower the value, the fewer resources the table data can use. Exemplarily, when the preset scheduling policy of the preset resource group is quota, the resource management device first obtains the optimization time quotaTime that the target table data has occupied in the last hour, and then divides it by the optimization time quota self-optimizing.quota specified for each table data, and calculates the optimization time occupied by the target table data quotaOccupy = quotaTime / self-optimizing.quota, and sorts them from small to large according to quotaOccupy, that is, the smaller the allocated self-optimizing.quota, the larger the quotaOccupy, and the lower the scheduling priority.
[0052] Then, the file groups in the target table data are sorted. The file group consists of multiple files to be rewritten. Specifically, in the same target table data, the target table data is partitioned according to preset rules. For example, the target table data can be partitioned according to the date in the target table data, and the data of the same year can be divided into the same partition; according to the preset partition rules of the target table data, a corresponding file group is established for each partition, and then the current file to be rewritten is saved to the corresponding file group according to the file attribute information of the current file to be rewritten, so as to obtain the file group in the target table data. Calculate the current rewrite cost of each file group in the same target table data; sort each file group from large to small according to the current rewrite cost of each file group in the same target table data, and obtain each file group after sorting in the same target table data. Then, the sorted file groups in each target table data are uniformly sorted according to the sorting of the target table data to obtain the target sorting of each file group. As an example, if the preset resource group includes three target table data A, B and C, the three are sorted as A, B and C according to the preset scheduling strategy; the target table data A has two file groups a and aa, sorted as aa and a; the target table data B has three file groups b, bb and bbb, sorted as bb, b and bbb; the target table data C has two file groups c and cc, sorted as cc and c; then the final sorting of the file groups is aa, a, bb, b, bbb, cc and c. In this way, the file groups are processed from large to small according to their current rewrite costs, avoiding the problem that the file groups with large current rewrite costs cannot be scheduled for a long time.
[0053] The resource management device first sorts the target table data according to the preset scheduling strategy of the corresponding preset resource group; then sorts the file group composed of the current files to be rewritten in the target table data; finally, starting from the target table data that is sorted first, the target table data are uniformly sorted according to the sorting of the file groups in the target table data to obtain the target sorting of each file group; according to the target sorting of each file group, each file group is converted into an optimization task and saved on the optimization server waiting to be pulled by the optimization container. At this time, the table status to which the file group belongs is updated from idle (idle, no optimization required) to pending (waiting for the optimization container to pull and execute).
[0054] It can be seen that the resource management method of the embodiment of the present application obtains the target table data corresponding to the same preset resource group in the data lake and the current file to be rewritten in the target table data; determines the current rewrite time of the current file to be rewritten in the target table data according to the current rewrite cost of the current file to be rewritten and the historical rewrite cost information of the historical rewrite file in the target table data, and the rewrite cost represents the resource data required to rewrite the corresponding file to be rewritten; determines the target number of optimization containers to be called from the preset resource group according to the preset optimization time and the current rewrite time of the optimization container in the preset resource group; rewrites the current file to be rewritten in the target table data with the target number. Therefore, the target number of optimization containers required to rewrite the current file to be rewritten can be determined by the current rewrite time of the current file to be rewritten and the preset optimization time of the optimization container, and the current file to be rewritten is rewritten by the optimization containers of the target number, thereby ensuring the efficiency of rewriting; at the same time, the optimization container in the same preset resource group is called to rewrite the file to be rewritten under the preset resource group, thereby realizing resource isolation.
[0055] Based on the above embodiments, the embodiments of the present application adopt Figure 2 The flowchart details how to determine the target number of optimized containers to be called in a preset resource group. Figure 2 , Figure 2 yes Figure 1 The flowchart of an exemplary embodiment of step S130 in the resource management method is shown. Specifically, in step S130, the process of determining the target number of optimized containers to be retrieved from the preset resource group according to the preset optimization time and the current rewrite time of the optimized containers in the preset resource group includes the following steps:
[0056] S210: Determine the initial number of optimization containers to be called according to the preset optimization time and the current rewriting time of the optimization containers in the preset resource group.
[0057] The initial number is determined by the preset optimization time of the optimization container in the preset resource group and the current rewrite time of the current file to be rewritten. Exemplarily, the ratio between the current rewrite time of the current file to be rewritten and the preset optimization time of the optimization container in the preset resource group is determined as the initial number of the optimization containers to be called.
[0058] Exemplarily, the calculation of the initial number of optimized containers to be retrieved satisfies the following formula:
[0059] continer_count=rewriteTime / execute.time.per-container
[0060] Among them, continer_count indicates the initial number of containers to be optimized, rewriteTime indicates the current rewrite time of the file to be rewritten, and execute.time.per-container indicates the preset optimization time of each optimized container.
[0061] The resource management device calculates the current rewrite cost of the current file to be rewritten in the preset resource group. It can be understood that the current file to be rewritten in the preset resource group can include all files to be rewritten in the preset resource group, and the current rewrite cost refers to the total rewrite cost of all files to be rewritten in the preset resource group; the preset optimization time of the optimization container refers to the optimization time of each optimization container; therefore, by calculating the ratio between the current rewrite time of the current file to be rewritten and the preset optimization time of the optimization container, the initial number of optimization containers to be called can be obtained.
[0062] S220: Determine a target number of optimized containers to be retrieved according to a preset number of optimized containers in a preset resource group and an initial number of optimized containers to be retrieved.
[0063] The preset number refers to the maximum number of configurable optimization containers in the preset resource group. Exemplarily, the resource upper limit of the preset resource group, that is, the maximum resource value, is obtained, and the ratio between the resource upper limit and the preset resource usage rate of each optimization container is calculated to obtain the preset number of optimization containers in the preset resource group. As an example, if the resource upper limit in the preset resource group is 5G and the preset resource usage rate of each optimization container is 1G, then a maximum of 5 optimization containers can be configured in the preset resource group.
[0064] The resource management device determines the preset number of optimized containers in the preset resource group according to the resource upper limit in the preset resource group and the preset resource usage rate of each optimized container, and determines the target number of optimized containers to be called according to the preset number of optimized containers in the preset resource group and the initial number of optimized containers to be called. In other embodiments, in order to further save resources, when determining the target number of optimized containers to be called, the resource management device may also determine the initial number of optimized containers according to the preset multiples of the preset optimization time of the optimized containers in the preset resource group and the current rewrite time; and then determine the target number of optimized containers to be called according to the preset number of optimized containers in the preset resource group and the initial number of optimized containers to be called. The preset multiple may be 1.1 times, 1.2 times, etc.
[0065] Specifically, the process of determining the target number of optimization containers to be called according to the preset number of optimization containers in the preset resource group and the initial number of optimization containers to be called includes the following steps: in response to the initial number of optimization containers to be called being less than or equal to the preset number of optimization containers in the preset resource group, the initial number of optimization containers to be called is determined as the target number of optimization containers to be called; in response to the initial number of optimization containers to be called being greater than the preset number of optimization containers in the preset resource group, the preset number of optimization containers in the preset resource group is determined as the target number of optimization containers to be called. This ensures that the resources of the optimization containers are within the resource upper limit of the preset resource group.
[0066] Since the resource upper limit of the preset resource group has been set when configuring the preset resource group, the resources occupied by the optimized container in the preset resource group cannot exceed the resource upper limit of the preset resource group. Exemplarily, the calculation of the target number of optimized containers can satisfy the following formula:
[0067] need_continer_count=Math.min(container_limit,rewriteTime / execute.time.per-container)
[0068] Among them, need_continer_count indicates the target number of optimized containers, Math.min indicates the minimum value in the returned parameters, container_limit indicates the preset number of optimized containers, and rewriteTime / execute.time.per-container indicates the initial number of optimized containers.
[0069] In other embodiments, the process of determining the target number of optimization containers to be called according to the preset number of optimization containers in the preset resource group and the initial number of optimization containers to be called may also include the following steps: when the initial number of optimization containers to be called is greater than the preset number of optimization containers in the preset resource group, idle optimization containers may be called from other preset resource groups for operation optimization.
[0070] It can be seen that the resource management method of the embodiment of the present application determines the initial number of optimization containers to be called according to the preset optimization time and the current rewrite time of the optimization containers in the preset resource group; and determines the target number of optimization containers to be called according to the preset number of optimization containers in the preset resource group and the initial number of optimization containers to be called. This simplifies the calculation process of the initial number of optimization containers and prevents the resource proportion of the optimization containers from exceeding the resource upper limit of the preset resource group.
[0071] Based on the above embodiments, the embodiments of the present application adopt Figure 3The flowchart details how to scale up or scale down the number of optimization containers based on the target number of optimization containers. Figure 3 , Figure 3 yes Figure 1 The flowchart of an exemplary embodiment after step S130 in the resource management method is shown. Specifically, after the process of determining the target number of optimized containers to be retrieved from the preset resource group according to the preset optimization time and the current rewrite time of the optimized containers in the preset resource group in step S130, the following steps are also included:
[0072] S310: Using the historical number of optimized containers retrieved when rewriting the file obtained in the rewriting history as the current number of optimized container configurations in the preset resource group.
[0073] The historical number refers to the number of optimized containers called when rewriting the historical rewrite file in the preset resource group. For example, the rewrite file in the last rewrite process of the preset resource group can be used as the historical rewrite file, and the number of optimized containers that rewrite the historical rewrite file is used as the historical number of optimized containers.
[0074] The current number of optimized container configurations refers to the current number of optimized container configurations of the preset resource group. Generally speaking, the configuration of the optimized container in the last rewrite process of the preset resource group will be directly used as the current number of optimized container configurations in the corresponding preset resource group.
[0075] The resource management device obtains the historical number of optimized containers called in the last rewriting process of the preset resource group, and uses it as the current optimized container configuration number in the preset resource group.
[0076] S320: In response to the current number of optimized container configurations being greater than the target number of optimized containers to be retrieved, the current number of optimized container configurations is reduced until the number of optimized containers in the preset resource group is reduced from the current number of optimized container configurations to the target number.
[0077] The resource management device determines whether the current number of optimized container configurations is greater than the target number of optimized containers to be called. If so, it means that the current number of optimized container configurations is too large and there are surplus resources. Therefore, the current number of optimized container configurations is reduced until the number of optimized containers in the preset resource group is reduced from the current number of optimized container configurations to the target number. This not only ensures that the rewriting task has sufficient resources to run, but also reduces the problem of idle resources due to redundant occupation of resources, which affects system operation.
[0078] S330: In response to the current number of optimized container configurations being less than the target number of optimized containers to be retrieved, the current number of optimized container configurations is expanded until the number of optimized containers in the preset resource group is expanded from the current number of optimized container configurations to the target number.
[0079] The resource management device determines whether the current number of optimized containers is less than the target number of optimized containers to be called. If it is less, it means that the current number of optimized containers is too small and there is a lack of resources. Therefore, the current number of optimized containers is expanded until the number of optimized containers in the preset resource group is expanded from the current number of optimized containers to the target number. This ensures that there are enough optimized containers to perform optimization tasks under high load conditions and prevents resource congestion.
[0080] It can be seen that the resource management method of the embodiment of the present application automatically expands or contracts the current number of optimized container configurations according to the size relationship between the current number of optimized container configurations and the target number, ensuring that there are sufficient resources to perform optimization tasks under high load conditions, and reducing resource usage to save costs under low load conditions.
[0081] Based on the above embodiments, the embodiments of the present application adopt Figure 4 The flowchart details how to obtain the target table data in the preset resource group and the current file to be rewritten in the target table data. Figure 4 , Figure 4 yes Figure 1 The flowchart of an exemplary embodiment of step S110 in the resource management method is shown. Specifically, after obtaining the target table data corresponding to the same preset resource group in the data lake and the current file to be rewritten in the target table data in step S110, the following steps are included:
[0082] S410: Determine the target table data according to the rewriting time interval of each table data corresponding to the same preset resource group in the data lake.
[0083] The rewrite time interval refers to the time interval for rewriting table data. In some embodiments, the rewrite time interval may be the time difference between the time point at which the table data is historically rewritten and the current time point. The step of determining the target table data according to the rewrite time interval specifically includes: obtaining the historical rewrite time point at which the historical files to be rewritten corresponding to each table data in the preset resource group are started to be rewritten; determining the time difference between the historical rewrite time point corresponding to each table data and the current time point as the rewrite time interval of the corresponding table data, and the current time point represents the time point at which the current files to be rewritten in the corresponding table data are started to be rewritten; in response to the rewrite time interval of each table data being greater than the preset rewrite time interval of the corresponding table data, the corresponding table data is determined as the target table data in the preset resource group. Thus, the rewrite time interval is determined according to the time difference between the historical rewrite time point and the current time point of each table data, and compared with the preset rewrite time interval, so as to accurately locate the target table data that needs to be rewritten.
[0084] Among them, the historical rewrite time point can be the corresponding time point when the preset resource group last started to rewrite the data of each table, and of course it can also be the corresponding time point when the preset resource group last started to rewrite the data of each table. For example, the preset resource group includes table data A, and the time point when the table data A was last rewritten is 15:00, then 15:00 is used as the historical rewrite time point of table data A, and the current time point for rewriting table data A is 16:40. The difference between the current time point and the historical time point is 1.5 hours. If the preset rewrite time interval is 1 hour, it means that table data A needs to be rewritten, and table data A is used as the target table data in the corresponding preset resource group. In addition, the preset rewrite time interval indicates the time interval between two adjacent trigger file rewrites of each table data, and the default is 3600000 (1 hour). Only when the preset rewrite time interval is greater than 0, and the rewrite time interval of each table data is greater than the preset rewrite time interval of the corresponding table data, the corresponding table data will be determined as the target table data in the preset resource group.
[0085] In some other embodiments, the step of determining the target table data according to the rewrite time interval of each table data in the same preset resource group may also include: obtaining the historical rewrite time point of the historical rewrite file corresponding to each table data in the preset resource group; determining the standard rewrite time interval of each table data according to the historical rewrite time point of the historical rewrite file; determining the time difference between the last rewrite time point corresponding to each table data and the current time point as the rewrite time interval of the corresponding table data; in response to the rewrite time interval of each table data being greater than the standard rewrite time interval of the corresponding table data, determining the corresponding table data as the target table data in the preset resource group. The standard rewrite time interval may be the average value of the time intervals of multiple historical rewrite processes. For example, the average value of the interval time between the Nth rewrite and the N-1th rewrite of the table data and the interval time between the N-1th rewrite and the N-2th rewrite is calculated, and the average value is used as the standard rewrite time interval of the corresponding table data.
[0086] The resource management device determines the time difference between the historical rewrite time point when each table data corresponding to the same preset resource group in the data lake last started to be rewritten and the current time point as the rewrite time interval of the corresponding table data, and determines the table data whose rewrite time interval exceeds the preset rewrite time interval as the target table data.
[0087] S420: Determine the current file to be rewritten in the target table data according to the file attribute information of each file in the target table data.
[0088] The file attribute information includes the file occupied space, file modification information, etc. of each file in the target table data.
[0089] Specifically, in response to the occupied space of each file in the target table data being less than the preset file occupied space, the file occupied space less than the preset file occupied space is determined as the current file to be rewritten in the target table data; in response to the occupied space of each file in the target table data being greater than or equal to the preset file occupied space, the historical data deletion amount of the file occupied space greater than or equal to the preset file occupied space is obtained; in response to the historical data deletion amount of the file occupied space greater than or equal to the preset file occupied space being greater than the preset data deletion threshold, the file historical data deletion amount greater than the preset data deletion threshold is determined as the file to be rewritten in the target table data. In this way, whether it is the current file to be rewritten is determined based on the occupied space of each file in the target table data and the historical data deletion amount of the file, which not only supports rewriting small files whose occupied space is less than the preset file occupied space, but also supports rewriting large files whose historical data deletion amount is greater than the preset data deletion threshold. Files with less occupied space and larger deleted data amount can be rewritten to ensure the appropriateness of data management.
[0090] The space occupied by a file refers to the system storage space occupied by the file. The preset space occupied is usually 1 / 8 of the target file's space occupied, or it can be 1 / 7 or 1 / 6, etc. The target file's space occupied can be 64M, 128M, etc. When the space occupied by a file is less than the preset space occupied, it means that the file is a small file and needs to be rewritten.
[0091] The historical data deletion amount refers to the data deletion amount of each file in the past period of time. For example, the data amount of the file after the last optimization and the data amount of the file before the current optimization can be obtained, and the amount of data reduction of the file before the current optimization compared with the data amount of the file after the last optimization is calculated to obtain the historical data deletion amount of the corresponding file.
[0092] The preset data deletion threshold may be a preset maximum number of deletions for each file. Exemplarily, the product of the number of file lines and the overwrite data deletion threshold may be determined as the preset data deletion threshold. The overwrite data deletion threshold may be determined by a preset multiple of the total data volume in each file. The preset multiple may be 0.1, 0.2, etc., that is, 0.1 times the total file data volume is used as the overwrite data deletion threshold.
[0093] In some other embodiments, the resource management device may also determine only the files in the target table data whose occupied space is smaller than the preset file occupied space as the files currently to be rewritten in the target table data.
[0094] In combination with the above embodiments, the acquisition of the target table data in the same preset resource group in the data lake and the current files to be rewritten in the target table data are described in detail. Figure 5 , Figure 5It is a flowchart of an exemplary embodiment of obtaining the current file to be rewritten in the target table data in the embodiment of the present application.
[0095] The resource management device scans each table data in a preset resource group; obtains the rewrite time interval diffTime between the current time point currentTime and the last rewrite time lastRewriteTime of each table data; determines whether the preset rewrite time interval rewrite.trigger.interval is greater than 0 and whether the rewrite time interval diffTime is greater than the preset rewrite time interval rewrite.trigger.interval; if not, it is considered that the corresponding table data does not need to be rewritten; if so, the corresponding table data is determined as the target table data, and each file in the target table data is scanned; it is determined whether the occupied space (fileSize) of each file in the target table data is less than the preset file occupied space (min-file-size-bytes); if so, the corresponding file is saved to the corresponding file group according to the file partition; if not, it is further determined whether the historical data deletion amount (deletesCount) of the file is greater than the preset data deletion threshold, the preset data deletion threshold is obtained by the product of the number of file rows (Record) and the rewrite data deletion threshold (rewrite.delete.ratio), if so, the corresponding file is saved to the corresponding file group according to the file partition, if not, the file is skipped.
[0096] Based on the above embodiments, the embodiments of the present application adopt Figure 6 The flowchart details how to determine the current rewrite time of the file to be rewritten. Figure 6 , Figure 6 yes Figure 1 The flowchart of an exemplary embodiment of step S120 in the resource management method is shown. Specifically, the process of determining the current rewrite time of the current file to be rewritten in the target table data according to the current rewrite cost of the current file to be rewritten and the historical rewrite cost information of the historical rewrite files in the target table data in step S120 specifically includes the following steps:
[0097] S610: Determine the ratio between the historical rewriting time and the historical rewriting cost as the unit rewriting time.
[0098] The unit rewrite time refers to the time corresponding to each cost.
[0099] The resource management device obtains the historical rewriting time and the historical rewriting cost, and determines the ratio between the historical rewriting time and the historical rewriting cost as the unit rewriting time corresponding to each cost.
[0100] S620: Determine the product of the unit rewrite time and the current rewrite cost of the current file to be rewritten as the current rewrite time of the current file to be rewritten.
[0101] When calculating the current rewrite cost of the file to be rewritten, the resource management device first calculates the current rewrite cost of each file group in the target table data, and the calculation satisfies the following formula:
[0102] cost = cost of opening file + cost of reading data + cost of writing data
[0103] Among them, cost represents the current rewrite cost of each file group; the calculation formula for opening file cost is fileCnt*openFileCost, fileCnt represents the number of all files in the file group; openFileCost is a fixed value, the default is 2M or 4M; the data reading cost represents the sum of the occupied space of all files in the file group; the data writing cost is 3 or 4 times the data reading cost.
[0104] After the current rewrite cost of each file group in the target table data is obtained, the current rewrite costs of all file groups in the target table data are summed to obtain the current rewrite cost of the file to be rewritten.
[0105] After obtaining the current rewrite cost of the current file to be rewritten, the product between the unit rewrite time and the current rewrite cost of the current file to be rewritten is calculated to obtain the current rewrite time of the current file to be rewritten. Exemplarily, the calculation of the current rewrite time of the current file to be rewritten satisfies the following formula:
[0106] rewriteTime=cur_cost_sum * previous_actual_sum / previous_cost_sum
[0107] Among them, rewriteTime represents the current rewrite time, cur_cost_sum represents the current rewrite cost of the file to be rewritten, previous_actual_sum represents the historical rewrite time of the historical rewrite file, and previous_cost_sum represents the historical rewrite cost of the historical rewrite file.
[0108] In some other embodiments, the current rewrite time of the file to be rewritten may be determined as the product of the current rewrite cost of the file to be rewritten and the preset unit rewrite time.
[0109] In conjunction with the above embodiments, how to determine the target number of optimized containers in a preset resource group is described in detail. Figure 7 , Figure 7It is a flow chart of an exemplary embodiment of optimizing the determination of the number of containers in the embodiments of the present application.
[0110] First, the resource management device obtains the historical rewrite cost previous_cost_sum and the historical rewrite time previous_actual_sum; calculates the current rewrite cost cur_cost_sum of the current file to be rewritten, and the current rewrite cost of the current file to be rewritten can be calculated according to the file opening cost, the file reading cost and the data writing cost; determines the current rewrite time rewriteTime according to the historical rewrite cost previous_cost_sum, the historical rewrite time previous_actual_sum and the current rewrite cost cur_cost_sum, and its calculation formula is rewriteTime=cur_cost_sum *previous_actual_sum / previous_cost_sum; determines the ratio between the current rewrite time rewriteTime and the preset optimization time execute.time.per-container of the optimization container as the initial number of optimization containers, and determines the ratio between the resource upper limit of the preset resource group and the preset usage resources of the optimization container as the preset number of optimization containers container_limit, and selects the minimum value between the initial number and the preset number as the target number of optimization containers need_continer_count, and its calculation formula is need_continer_count= Math.min(container_limit, rewriteTime / execute.time.per-container); Finally, compare with the current number of optimized containers configured, and expand the number of optimized containers or reduce idle optimized containers.
[0111] It can be seen that the resource management method of the embodiment of the present application further determines the current rewrite time through the historical rewrite time and the historical rewrite cost. The historical rewrite cost information, as an objective real-time record, is conducive to reducing the subjectivity of the current rewrite time and improving the accuracy of the current rewrite time.
[0112] The resource management device includes a front-end Web UI, an optimization server, and an optimization container. Among them, the user configures the data source information and the preset resource group information in the front-end WebUI. The preset resource group information includes the preset resource group name, the upper and lower limits of the resources of the preset resource group, and the preset scheduling strategy of the preset resource group; the optimization server scans all table data in the data source and generates optimization tasks according to the rules. The rules are specified by the table attributes of the table data, including the preset resource group to which the table data belongs, the preset file occupied space, life cycle, rewrite time interval, etc. The optimization container can be deployed locally, on kubernetes or yarn. Each optimization container pulls the optimization task from the optimization server and executes it. The number of optimization containers is dynamically adjusted by the optimization server according to the number of optimization tasks and the upper and lower limits of the resources of the preset resource group. This solution realizes automatic management and optimization of table data through the front-end Web UI, the optimization server, and the optimization container. After the user configures the optimization rules, he does not need to care about the life cycle of the table data, small files, orphan files, etc., which improves the management and operation efficiency. During the whole process, the user only needs to configure the upper and lower limits of the use of the preset resource group, and the resource management device can automatically scale resources according to the resources required by the optimization task.
[0113] Specifically, the user first specifies various parameters of the table data, including:
[0114] (1) Whether to enable automatic optimization (self-optimizing.enabled). By default, it is not enabled.
[0115] (2) The default resource group self-optimizing.group to which the table data belongs. Specify the name of the default resource group. The default value is default.
[0116] (3) The life cycle of table data, table.retention-time, that is, table data will be deleted if it has not been updated for a specified period of time. The default value is -1, which means it is not enabled.
[0117] (4) Partition life cycle partition.retention-time in table data. A partition will be deleted if it has not been updated for a specified period of time. The default value is -1, which means it is not enabled.
[0118] (5) The snapshot life cycle snapshot.retention-time means that the snapshot will be cleared after being retained for a specified period of time. When the table data is updated, a corresponding snapshot will be generated, which includes both the data before and after the update;
[0119] (6) Snapshot life cycle snapshot.min-to-keep, which indicates the minimum number of snapshots to be retained. The default value is 1.
[0120] (7) The life cycle of the metadata file, metadata.retention-time, indicates that the metadata file will be cleaned up after being retained for a specified period of time;
[0121] (8) Orphan file cleaning interval clean-orphan-file.interval, the default value is -1 and is not enabled;
[0122] (9) File rewrite time interval rewrite.trigger.interval, the default value is 3600000 (1 hour), which means the time interval between two triggers of file rewrite for a table data. The file rewrite task runs in the optimized container and can only be pulled through the optimized container that belongs to the same preset resource group as the corresponding file;
[0123] (10) Small file threshold min-file-size-bytes. By default, the target file occupies 1 / 8 of the space. If the space occupied by a file is less than this value, it is considered a small file and needs to be rewritten.
[0124] (11) File rewrite deletion data threshold rewrite.delete.ratio, the default value is 0.1, which means that when the deleted data in a file reaches 0.1 times the total file data, the file will also be rewritten to improve the reading performance;
[0125] (12) The scheduling priority of table data is self-optimizing.priority, with a default value of 1 and a range of 1-10. It is valid when the scheduling strategy of the preset resource group to which the table data belongs is priority. The lower the value, the higher the priority. The scheduling priority of table data is self-optimizing. quota, with a default value of 0.1 and a range of 0-1. It is valid when the scheduling strategy of the preset resource group to which the table data belongs is quota. The lower the value, the fewer resources the table can use.
[0126] The front-end Web UI can display the data source, preset resource group, table data status, and historical rewrite tasks. The status of table data includes the preset resource group to which the table data belongs, the optimization status, the number of files, and the number of small files. The optimization status includes pending, which means waiting for the optimization container to pull and execute, idle, which means no optimization is required, and optimizing, which means optimization is being executed. Historical rewrite tasks include execution time, number of rewritten files, space occupied by rewritten files, etc. for user analysis. This supports the front-end Web UI page to configure data source information and preset resource groups, intuitively display the current status of each table data, and display the indicators of the rewrite task in the form of charts for user analysis.
[0127] In data management, not only files need to be rewritten and optimized, but also expired files need to be cleaned up regularly. These functions can be performed by the optimization server, which can support multiple life cycles and file management. The optimization server can also register various optimizers when it starts, and schedule each optimizer with the table.refresh.interval (default 10s). The optimization tasks of the optimization server include:
[0128] (1) Table data refreshing optimizer TableRefreshingOptimizer refreshes table data from the data source to the optimization server;
[0129] (2) Table data retention optimizer TableRetentionOptimizer calculates the difference diffTime between the current time point currentTime and the last update time point lastUpdateTime of the table data. If table.retention-time (the life cycle of the table data) is greater than 0 and diffTime is greater than table.retention-time, the table is deleted;
[0130] (3) PartitionRetentionOptimizer calculates the difference between the current time point currentTime and the last update time point lastUpdateTime of the partition of the table data. If partition.retention-time (partition lifecycle) is greater than 0 and diffTime is greater than partition.retention-time, the partition is deleted.
[0131] (4) SnapshotRetentionOptimizer and MetadataRetentionOptimizer delete expired snapshot and metadata files directly based on the functions provided by the data lake framework;
[0132] (5) OrphanCleanOptimizer, the orphan file retention optimizer, calculates the difference between the current time point currentTime and the last time the orphan files in the table data were cleaned up, lastCleanTime. If diffTime is greater than clean-orphan-file.interval (the life cycle of the orphan file), the orphan files in the table are cleaned up directly based on the functions provided by the data lake framework.
[0133] See also Figure 8 , Figure 8It is a framework diagram of an exemplary embodiment of rewriting tasks and optimization containers in the optimization server of the embodiment of the present application. The optimization server (Optimizer Server) includes preset resource group 1 (ResourceGroup1) and preset resource group 2 (Resource Group2), and each preset resource group includes at least one rewriting task (RewriteTask); the optimization container (Optimizer container) in the preset resource group 1 pulls the rewriting task (send_pollTask) of the preset resource group 1 in the optimization server, and the optimization server receives the pulled rewriting task (recv_pollTask) of the optimization container. After the optimization container pulls the rewriting task, it performs optimization processing through its optimization executor (Optimizer Executor); similarly, the optimization container in the preset resource group 2 pulls the rewriting task of the preset resource group 2 in the optimization server, and performs optimization processing through the optimization executor.
[0134] In order to implement the resource management method of the above embodiment, the present application proposes another electronic device, for details, please refer to Fig. 9 , Fig. 9 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application.
[0135] The electronic device 900 includes a memory 910 and a processor 920 , wherein the memory 910 and the processor 920 are coupled.
[0136] The memory 910 is used to store program data, and the processor 920 is used to execute the program data to implement the resource management method of the above embodiment.
[0137] In this embodiment, the processor 920 may also be referred to as a CPU (Central Processing Unit). The processor 920 may be an integrated circuit chip having signal processing capabilities. The processor 920 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 920 may also be any conventional processor, etc.
[0138] The present application also provides a computer-readable storage medium, such as Fig.10 As shown, the computer-readable storage medium 1000 is used to store program data 1001. When the program data 1001 is executed by the processor, it is used to implement the resource management method in the method embodiment of the present application.
[0139] The method involved in the resource management method embodiment of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.
[0140] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A resource management method, characterized in that: The resource management method comprises: Obtain target table data corresponding to the same preset resource group in the data lake and the current file to be rewritten in the target table data; Determine the current rewrite time of the current file to be rewritten in the target table data according to the current rewrite cost of the current file to be rewritten and the historical rewrite cost information of the historically rewritten files in the target table data, the rewrite cost representing the resource data required to rewrite the corresponding file to be rewritten; Determining a target number of optimization containers to be retrieved from the preset resource group according to the preset optimization time of the optimization containers in the preset resource group and the current rewriting time; The current to-be-rewritten file in the target table data is rewritten using the target number of optimized containers, and the preset resource group of the target number of optimized containers is the same as the preset resource group of the current to-be-rewritten file.
2. The resource management method according to claim 1, characterized in that: The step of determining the target number of optimized containers to be retrieved from the preset resource group according to the preset optimization time of the optimized containers in the preset resource group and the current rewrite time includes: Determine the initial number of optimization containers to be called according to the preset optimization time of the optimization containers in the preset resource group and the current rewrite time; The target number of the to-be-called optimized containers is determined according to the preset number of optimized containers in the preset resource group and the initial number of the to-be-called optimized containers.
3. The resource management method according to claim 2, characterized in that: The step of determining the target number of the to-be-called optimized containers according to the preset number of optimized containers in the preset resource group and the initial number of the to-be-called optimized containers comprises: In response to the initial number of the to-be-called optimized containers being less than or equal to the preset number of optimized containers in the preset resource group, determining the initial number of the to-be-called optimized containers as the target number of the to-be-called optimized containers; In response to the initial number of the to-be-called optimized containers being greater than the preset number of the optimized containers in the preset resource group, the preset number of the optimized containers in the preset resource group is determined as the target number of the to-be-called optimized containers.
4. The resource management method according to claim 1, characterized in that: After determining the target number of optimized containers to be retrieved from the preset resource group according to the preset optimization time of the optimized containers in the preset resource group and the current rewrite time, the resource management method further includes: The obtained historical number of optimized containers retrieved when rewriting the historical rewriting file is used as the current number of optimized container configurations in the preset resource group; In response to the current number of optimized container configurations being greater than the target number of optimized containers to be retrieved, reducing the current number of optimized container configurations until the number of optimized containers in the preset resource group is reduced from the current number of optimized container configurations to the target number; In response to the current number of optimized container configurations being less than the target number of optimized containers to be retrieved, the current number of optimized container configurations is expanded until the number of optimized containers in the preset resource group is expanded from the current number of optimized container configurations to the target number.
5. The resource management method according to claim 1, characterized in that: The step of obtaining target table data corresponding to the same preset resource group in the data lake and the current file to be rewritten in the target table data includes: Determine the target table data according to a rewriting time interval of each table data corresponding to the same preset resource group in the data lake; The current file to be rewritten in the target table data is determined according to the file attribute information of each file in the target table data.
6. The resource management method according to claim 5, characterized in that: The step of determining the target table data according to the rewriting time interval of each table data corresponding to the same preset resource group in the data lake includes: Obtaining a historical rewriting time point at which to start rewriting the historical files to be rewritten corresponding to the data of each table in the preset resource group; Determine the time difference between the historical rewriting time point corresponding to each table data and the current time point as the rewriting time interval of the corresponding table data, wherein the current time point represents the time point for starting to rewrite the current file to be rewritten in the corresponding table data; In response to the rewriting time interval of each table data being greater than the preset rewriting time interval of the corresponding table data, the corresponding table data is determined as the target table data in the preset resource group.
7. The resource management method according to claim 5, characterized in that: The file attribute information includes the occupied space of the file, and the step of determining the current file to be rewritten in the target table data according to the file attribute information of each file in the target table data includes: In response to the occupied space of each file in the target table data being smaller than the preset file occupied space, determining the file whose occupied space is smaller than the preset file occupied space as the current file to be rewritten in the target table data; In response to the occupied space of each file in the target table data being greater than or equal to the preset file occupied space, obtaining the historical data deletion amount of the file whose occupied space is greater than or equal to the preset file occupied space; In response to the historical data deletion amount of the file whose occupied space is greater than or equal to the preset file occupied space being greater than a preset data deletion threshold, the file whose historical data deletion amount is greater than the preset data deletion threshold is determined as a file to be rewritten in the target table data.
8. The resource management method according to claim 1, characterized in that: The historical rewrite cost information includes the historical rewrite time and the historical rewrite cost. The step of determining the current rewrite time of the current file to be rewritten in the target table data according to the current rewrite cost of the current file to be rewritten and the historical rewrite cost information of the historical rewrite files in the target table data includes: determining a ratio between the historical rewriting time and the historical rewriting cost as a unit rewriting time; The product of the unit rewrite time and the current rewrite cost of the current file to be rewritten is determined as the current rewrite time of the current file to be rewritten.
9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the resource management method according to any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that: include: Program data is stored, and when the program data is executed by a processor, it is used to implement the resource management method according to any one of claims 1 to 8.
