Data warehouse resource quota adjustment method and device, equipment and medium

Abstract

The application relates to a resource quota adjustment method and device of a data warehouse, equipment and a medium, wherein the method comprises the following steps: monitoring an actual usage quota of an expected quota of a target data warehouse, wherein the expected quota is a resource quota of a server cluster resource initialized in advance for the target data warehouse; in the case that the actual usage quota reaches a target load threshold of the expected quota, determining a target adjustment strategy matched with the target load threshold; and adjusting the expected quota of the target data warehouse according to the target adjustment strategy. The actual usage quota and the target load threshold are monitored to timely adjust the expected quota, thereby solving the problem that the expected quota is not adjusted in time.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, equipment and medium for adjusting resource quotas in a data warehouse. Background Technology

[0002] Hive is an offline data warehouse built on Hadoop. As a widely used component in the big data field, its importance is self-evident. During use, Hive data warehouses consume server resources, requiring frequent adjustments to file quantity and space quotas to improve server cluster resource utilization. Failure to make timely quota adjustments can easily reduce the efficiency of storing and processing massive amounts of data, increasing the operational costs of enterprise server resources.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method, apparatus, equipment, and medium for adjusting resource quotas in a data warehouse to solve the aforementioned technical problem that "failure to make timely quota adjustments can easily reduce the efficiency of storing and processing massive amounts of data."

[0005] According to one aspect of the embodiments of this application, this application provides a method for adjusting resource quotas in a data warehouse, comprising: monitoring the actual usage quota of a target data warehouse for a target quota, wherein the target quota is the resource quota of server cluster resources pre-initialized for the target data warehouse; determining a target adjustment strategy that matches the target load threshold when the actual usage quota reaches the target load threshold; and adjusting the target quota of the target data warehouse according to the target adjustment strategy.

[0006] Optionally, the target load threshold includes a high load threshold and a low load threshold. When the actual usage quota reaches the target load threshold of the expected quota, the target adjustment strategy that matches the target load threshold is determined as follows: when the actual usage quota is greater than or equal to the high load threshold, the expansion strategy is determined as the target adjustment strategy; when the actual usage quota is less than or equal to the low load threshold, the reduction strategy is determined as the target adjustment strategy.

[0007] Optionally, when the target adjustment strategy is an expansion strategy, adjusting the expected quota of the target data warehouse according to the target adjustment strategy includes: obtaining the value of a counter, and performing a merge operation on the files in the target data warehouse when the value of the counter is zero, wherein the counter is preset and is used to accumulate the number of merge operations; obtaining the new actual usage quota, and expanding the target data warehouse using an expansion trigger when the new actual usage quota is still greater than or equal to the high load threshold.

[0008] Optionally, scaling up the target data warehouse using a scaling trigger includes: activating the scaling trigger; using the scaling trigger to increase the capacity of the target data warehouse; and adjusting the first expected proportion of the capacity occupied by internal and external tables in the expected quota based on the first actual proportion of the capacity occupied by internal and external tables in the actual usage quota.

[0009] Optionally, when the target adjustment strategy is a scaling-down strategy, adjusting the expected quota of the target data warehouse according to the target adjustment strategy includes: activating a scaling-down trigger; using the scaling-down trigger to reduce the capacity of the target data warehouse, and adjusting the second expected ratio of the capacity occupied by internal tables and external tables in the expected quota based on the second actual ratio of the capacity occupied by internal tables and external tables in the actual usage quota.

[0010] Optionally, before monitoring the actual usage of the target data warehouse's expected quota, the method further includes: creating a target data warehouse, wherein the target data warehouse is used to store data for the target business; and initializing the expected quota of the target data warehouse based on the target business, wherein the expected quota includes file expected quota and space expected quota.

[0011] Optionally, after adjusting the expected quota of the target data warehouse according to the target adjustment strategy, the method further includes: obtaining adjustment data, wherein the adjustment data includes record data of the adjustment of the expected quota and the quota usage data before and after the adjustment; and uploading the adjustment data to the display module so that the display module displays the adjustment data to the administrator.

[0012] According to another aspect of the embodiments of this application, this application provides a resource quota adjustment device for a data warehouse, including: a quota monitoring module, used to monitor the actual usage quota of a target data warehouse for a target quota, wherein the target quota is the resource quota of server cluster resources pre-initialized for the target data warehouse; a strategy determination module, used to determine a target adjustment strategy matching the target load threshold when the actual usage quota reaches the target load threshold of the target quota; and a quota adjustment module, used to adjust the target quota of the target data warehouse according to the target adjustment strategy.

[0013] According to another aspect of the embodiments of this application, this application provides an electronic device, including a memory, a processor, a communication interface and a communication bus. The memory stores a computer program that can run on the processor. The memory and the processor communicate with each other through the communication bus and the communication interface. When the processor executes the computer program, it implements the steps of the above method.

[0014] According to another aspect of the embodiments of this application, this application also provides a computer-readable medium having processor-executable non-volatile program code that causes the processor to perform the above-described method.

[0015] Compared with related technologies, the technical solutions provided in this application have the following advantages:

[0016] This application discloses a method for adjusting resource quotas in a data warehouse, comprising: monitoring the actual usage quota of a target data warehouse for a predetermined quota, wherein the predetermined quota is the resource quota of a server cluster pre-initialized for the target data warehouse; determining a target adjustment strategy matching the predetermined load threshold when the actual usage quota reaches the target load threshold; and adjusting the predetermined quota of the target data warehouse according to the target adjustment strategy. By monitoring the relationship between the actual usage quota and the target load threshold to adjust the predetermined quota in a timely manner, the problem of untimely adjustment of the predetermined quota is solved. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the hardware environment for an optional data warehouse resource quota adjustment method provided in an embodiment of this application;

[0020] Figure 2 This is a flowchart illustrating an optional data warehouse resource quota adjustment method according to an embodiment of this application;

[0021] Figure 3 This is a flowchart of another optional data warehouse resource quota adjustment method provided according to an embodiment of this application;

[0022] Figure 4 This is a block diagram of an optional data warehouse resource quota adjustment device according to an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of an optional electronic device structure provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustration and has no specific meaning in itself. Therefore, "module" and "part" may be used interchangeably.

[0026] Hive is an offline data warehouse built on Hadoop, and its importance as a widely used component in the big data field is self-evident. During operation, Hive data warehouses consume server resources, requiring frequent adjustments to file quantity and space quotas to improve server cluster resource utilization. Failure to adjust quotas promptly can reduce the efficiency of storing and processing massive amounts of data, increasing the operational costs of enterprise server resources. Therefore, it is necessary to adjust the number of files and disk space used by the Hive database in a timely manner, and to automate the control and adjustment of Hive resource quota allocation through effective measures.

[0027] There is currently no effective solution to the above problems.

[0028] To address the problems mentioned in the background art, according to one aspect of the embodiments of this application, an embodiment of a data warehouse resource quota adjustment method is provided.

[0029] Optionally, in the embodiments of this application, the above-described data warehouse resource quota adjustment method can be applied to, for example... Figure 1 The hardware environment shown consists of terminal 101 and server 103. Figure 1 As shown, server 103 is connected to terminal 101 via a network and can be used to provide services to the terminal or clients installed on the terminal. Database 105 can be set up on the server or independently of the server to provide data storage services for server 103. The network mentioned above includes, but is not limited to, wide area network, metropolitan area network or local area network. Terminal 101 includes, but is not limited to, PC, mobile phone, tablet computer, etc.

[0030] The resource quota adjustment method for a data warehouse in this embodiment can be executed by server 103, or it can be jointly executed by server 103 and terminal 101, such as... Figure 2 As shown, it includes:

[0031] Step 201: Monitor the actual usage of the target data warehouse's expected quota, where the expected quota is the resource quota of the server cluster resources pre-initialized for the target data warehouse;

[0032] Step 203: If the actual usage quota reaches the target load threshold of the expected quota, determine the target adjustment strategy that matches the target load threshold.

[0033] Step 205: Adjust the expected quota of the target data warehouse according to the target adjustment strategy.

[0034] This application uses Hive data warehouse as the target data warehouse for description.

[0035] Monitor Hive resource usage quotas and set automated control triggers for expected quotas. Hive databases cannot expand in an unordered manner, so it's necessary to monitor the actual quota usage of the Hive database. Simultaneously, set quota adjustment triggers for the current Hive database. When the monitored file quantity quota or space quota reaches the threshold of the expected quota load, the corresponding task of adjusting the file quantity quota or disk space quota of the Hive database will be automatically executed.

[0036] When using Hive big data component resources, it is necessary to constrain file count quotas and disk space quotas. In the past, when using Hive resources, either no restrictions were placed on Hive resource quota usage, wasting server cluster resources; or Hive resource quota usage was only adjusted when it reached full load, affecting the operation of business applications. Even when timely notification was given that the expected quota of Hive resources needed to be adjusted, it still had to be done manually by developers, who could not focus on the development of business applications and had to consider the environmental constraints of cluster resources.

[0037] As an optional embodiment, the target load threshold includes a high load threshold and a low load threshold. When the actual usage quota reaches the target load threshold of the expected quota, determining the target adjustment strategy that matches the target load threshold includes: when the actual usage quota is greater than or equal to the high load threshold, determining the expansion strategy as the target adjustment strategy; when the actual usage quota is less than or equal to the low load threshold, determining the reduction strategy as the target adjustment strategy.

[0038] Optionally, the load threshold represents the proportion of the actual quota used to the expected quota, with the high load threshold being greater than the low load threshold. For example, if full load is represented by 1, then the high load threshold could be 0.8, and the low load threshold could be 0.2.

[0039] Optionally, when the actual quota used (including the quota used for the number of files or the quota used for disk space) is greater than or equal to the high load threshold of the expected quota for Hive resources, an expansion operation will be performed on the file quota or the expected quota for disk space.

[0040] Optionally, when the actual quota used (including the quota used for the number of files or the quota used for disk space) is less than or equal to the low load threshold of the expected quota for Hive resources, a reduction operation will be performed on the file relative to the expected quota for the number of files or the expected quota for disk space.

[0041] Based on the relationship between the actual usage quota and the target load threshold, it can be determined whether the current expected quota can meet the actual usage quota. If not, corresponding adjustments are made, and the usage of Hive database resource quotas is monitored in a timely manner. File quantity quotas are automatically adjusted to control the utilization of Hive cluster resources, and disk space quotas are automatically adjusted to efficiently utilize Hive cluster resources.

[0042] As an optional embodiment, when the target adjustment strategy is an expansion strategy, adjusting the expected quota of the target data warehouse according to the target adjustment strategy includes: obtaining the value of a counter, and performing a merge operation on the files in the target data warehouse when the value of the counter is zero, wherein the counter is preset and is used to accumulate the number of merge operations; obtaining the new actual usage quota, and expanding the target data warehouse using an expansion trigger when the new actual usage quota is still greater than or equal to the high load threshold.

[0043] Merge operations typically prioritize merging smaller files. By merging smaller files in the current Hive database, the actual number of files used and the disk space used are reduced.

[0044] Optionally, a counter is used to monitor the number of merge operations. A counter value of zero indicates that no merge operation has been performed, in which case a merge operation is performed on the files in the target data warehouse; if the counter value is not zero, it indicates that a merge operation has been performed, in which case no more file merges are performed.

[0045] As an optional embodiment, expanding the target data warehouse using a scaling trigger includes: activating the scaling trigger; expanding the capacity of the target data warehouse using the scaling trigger; and adjusting the first expected proportion of the capacity occupied by internal tables and external tables in the expected quota based on the first actual proportion of the capacity occupied by internal tables and external tables in the actual usage quota.

[0046] When Hive creates an internal table, it moves the data to the path pointed to by the data warehouse; when creating an external table, it only records the path where the data is located, without changing the location of the data itself. When deleting a table, the metadata and data of an internal table are deleted together, while only the metadata of an external table is deleted, not the data.

[0047] Optionally, if the number of files or the actual disk space usage quota still reaches the high load threshold of the expected quota of Hive resources after merging small files, the preset settings of cluster resources are executed to carry out the expansion process. At the same time, the expected number of files or the ratio of disk space for internal and external tables are set according to the current number of files or disk space usage of internal and external tables in the Hive database, so as to make reasonable use of Hive cluster resources.

[0048] As an optional embodiment, when the target adjustment strategy is a scaling-down strategy, adjusting the expected quota of the target data warehouse according to the target adjustment strategy includes: activating a scaling-down trigger; using the scaling-down trigger to reduce the capacity of the target data warehouse, and adjusting the second expected ratio of the capacity occupied by internal tables and external tables in the expected quota according to the second actual ratio of the capacity occupied by internal tables and external tables in the actual usage quota.

[0049] Optionally, if the actual number of files or disk space usage quota reaches the low load threshold of the expected quota of Hive resources, the preset settings of cluster resources are executed to perform a scaling-down process. At the same time, the expected number of files or disk space ratio of internal and external tables is set according to the current number of files or disk space usage of internal and external tables in the Hive database to reduce the waste of Hive cluster resources.

[0050] This application monitors the high load threshold where the actual usage quota of Hive resources reaches the expected quota. First, it reduces the actual usage of file quantity quota and disk space quota by merging small files. Then, it sets up expansion triggers to automatically increase the expected quota of Hive resources to meet the needs of business data volume.

[0051] If the actual usage quota of Hive resources reaches the low load threshold of the expected quota, a scaling-down trigger can be set to automatically reduce the expected quota of Hive resources, thereby reducing the waste of server cluster resources.

[0052] As an optional embodiment, before monitoring the actual usage of the target data warehouse on the expected quota, the method further includes: creating a target data warehouse, wherein the target data warehouse is used to store data of the target business; and initializing the expected quota of the target data warehouse based on the target business, wherein the expected quota includes file expected quota and space expected quota.

[0053] Optionally, the target data warehouse can be created based on the project's business needs, by creating a Hive data warehouse on a big data platform cluster.

[0054] Optionally, the expected quota for initializing the target data warehouse based on the target business includes: creating a data warehouse with a specified name, and initializing the total number of files and total disk space of the current database based on the company's existing server resources and the expected data volume of the project plan.

[0055] Because Hive cluster resources dynamically partition and insert data during tasks such as massive data storage and computation, a large number of small files are generated, resulting in a surge in the number of mappings. Therefore, it is necessary to initialize and set the file quantity ratio and disk space ratio of internal and external tables in the Hive database.

[0056] As an optional embodiment, after adjusting the expected quota of the target data warehouse according to the target adjustment strategy, the method further includes: acquiring adjustment data, wherein the adjustment data includes record data of adjusting the expected quota and quota usage data before and after the adjustment; and uploading the adjustment data to the display module so that the display module displays the adjustment data to the administrator.

[0057] Specifically, once the Hive resource expected quota automated control adjustment task is completed, the execution results will be uploaded to the monitoring system so that the monitoring system's display module can show the adjustment data to the administrators.

[0058] Optionally, the adjustment data includes recorded data on adjustments to the expected quota and quota usage data before and after the adjustment. The recorded data includes the number of adjustments and the adjustment strategy, etc. The quota usage data before and after the adjustment is a comparison of the data before and after the adjustment, such as a comparison of the actual quota usage before the adjustment and the actual quota usage after the adjustment.

[0059] Figure 2 The flowchart of an optional data warehouse resource quota adjustment method according to an embodiment of this application is shown in the figure:

[0060] 2-1. Monitor Hive data warehouse quota usage. The Hive resource expected quota initialization process mentions monitoring Hive resource quota usage, including the number of files and disk space quotas in the repository.

[0061] 2-2, Initialize the counter to zero. A counter is defined in the program to serve an auxiliary function;

[0062] 2-3, If the resource usage quota reaches the high load factor, execute 2-4;

[0063] 2-4 Check if the counter is 0. Checking the counter here is to determine if merging small files can be performed to reduce the actual Hive resource quota usage.

[0064] 2-5 Merge files, with a counter incrementing by 1 for each merge. If this is the first time merging small files, the small file merge operation will be performed. By merging small files in the current Hive database, the actual number of files used and disk space used are reduced.

[0065] Steps 2-6 initiate expansion triggers to automatically expand the expected quota. After merging files in steps 2-5, if the resource usage quota still reaches the high load factor of the expected Hive resource quota, expansion triggers for the number of files and the expected disk space quota will be initiated.

[0066] 2-7 If resource usage quotas reach a low load factor, proceed to step 2-8;

[0067] 2-8 Activate the shrinkage trigger to achieve automatic shrinkage of the expected quota.

[0068] The quota adjustment results for 2-9 were uploaded to the monitoring system.

[0069] This application discloses a method for adjusting resource quotas in a data warehouse, comprising: monitoring the actual usage quota of a target data warehouse for a predetermined quota, wherein the predetermined quota is the resource quota of a server cluster pre-initialized for the target data warehouse; determining a target adjustment strategy matching the predetermined load threshold when the actual usage quota reaches the target load threshold; and adjusting the predetermined quota of the target data warehouse according to the target adjustment strategy. By monitoring the relationship between the actual usage quota and the target load threshold to adjust the predetermined quota in a timely manner, the problem of untimely adjustment of the predetermined quota is solved.

[0070] According to another aspect of the embodiments of this application, this application provides a resource quota adjustment device for a data warehouse, such as... Figure 4 As shown, it includes:

[0071] The quota monitoring module 402 is used to monitor the actual usage quota of the target data warehouse for the expected quota, wherein the expected quota is the resource quota of the server cluster resources that are initialized in advance for the target data warehouse;

[0072] The strategy determination module 404 is used to determine a target adjustment strategy that matches the target load threshold when the actual usage quota reaches the target load threshold of the expected quota.

[0073] The quota adjustment module 406 is used to adjust the expected quota of the target data warehouse according to the target adjustment strategy.

[0074] It should be noted that the quota monitoring module 402 in this embodiment can be used to execute step 201 in this application embodiment, the strategy determination module 404 in this embodiment can be used to execute step 203 in this application embodiment, and the quota adjustment module 406 in this embodiment can be used to execute step 205 in this application embodiment.

[0075] Optionally, the strategy determination module 404 is further configured to determine the expansion strategy as the target adjustment strategy when the actual usage quota is greater than or equal to the high load threshold; and to determine the reduction strategy as the target adjustment strategy when the actual usage quota is less than or equal to the low load threshold.

[0076] Optionally, the quota adjustment module 406 also includes:

[0077] The merge submodule is used to obtain the value of the counter when the target adjustment strategy is the expansion strategy, and to perform a merge operation on the files in the target data warehouse when the value of the counter is zero. The counter is preset and is used to accumulate the number of merge operations.

[0078] The expansion submodule is used to obtain new actual usage quotas and, if the new actual usage quotas are still greater than or equal to the high load threshold, to expand the target data warehouse using expansion triggers.

[0079] Optionally, the expansion submodule is also used to initiate an expansion trigger; to expand the capacity of the target data warehouse using the expansion trigger, and to adjust the first expected proportion of the capacity occupied by internal and external tables in the expected quota based on the first actual proportion of the capacity occupied by internal and external tables in the actual usage quota.

[0080] Optionally, the quota adjustment module 406 is also used to activate a shrinkage trigger when the target adjustment strategy is a shrinkage strategy; to reduce the capacity of the target data warehouse using the shrinkage trigger, and to adjust the second expected ratio of the capacity of the internal and external tables in the expected quota according to the second actual ratio of the capacity occupied by the internal and external tables in the actual quota.

[0081] Optionally, the apparatus further includes a creation module for creating a target data warehouse before monitoring the actual usage of the target data warehouse on the expected quota, wherein the target data warehouse is used to store data for the target business; and initializing the expected quota of the target data warehouse based on the target business, wherein the expected quota includes expected file quota and expected space quota.

[0082] Optionally, the device further includes a sending module for acquiring adjustment data after adjusting the expected quota of the target data warehouse according to the target adjustment strategy, wherein the adjustment data includes record data of the adjustment of the expected quota and quota usage data before and after the adjustment; and uploading the adjustment data to the display module so that the display module displays the adjustment data to the administrator.

[0083] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of a device, can operate in environments such as... Figure 1 The hardware environment shown.

[0084] According to another aspect of the embodiments of this application, this application provides an electronic device, such as... Figure 5 As shown, the system includes a memory 501, a processor 503, a communication interface 505, and a communication bus 507. The memory 501 stores a computer program that can run on the processor 503. The memory 501 and the processor 503 communicate through the communication interface 505 and the communication bus 507. When the processor 503 executes the computer program, it implements the steps of the above method.

[0085] The memory and processor in the aforementioned electronic devices communicate with each other via a communication bus and a communication interface. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0086] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0087] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0088] According to another aspect of the embodiments of this application, a computer-readable medium having processor-executable non-volatile program code is also provided.

[0089] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0090] In specific implementation, the embodiments of this application can be referred to the above embodiments and have corresponding technical effects.

[0091] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0092] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0093] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0094] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0095] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0096] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0097] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0098] If the aforementioned function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks. It should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0099] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for adjusting resource quotas in a data warehouse, characterized in that, include: Monitor the actual usage of the target data warehouse's expected quota, wherein the expected quota is the resource quota of the server cluster resources pre-initialized for the target data warehouse; If the actual usage quota reaches the target load threshold of the expected quota, a target adjustment strategy matching the target load threshold is determined; The expected quota for the target data warehouse is adjusted according to the target adjustment strategy. The target load threshold includes a high load threshold and a low load threshold. When the actual usage quota reaches the target load threshold of the expected quota, determining the target adjustment strategy matching the target load threshold includes: if the number of files or disk space used in the actual usage quota is greater than or equal to the high load threshold, determining the expansion strategy as the target adjustment strategy; if the number of files or disk space used in the actual usage quota is less than or equal to the low load threshold, determining the reduction strategy as the target adjustment strategy. Specifically, during expansion, an expansion trigger is activated; the expansion trigger is used to increase the capacity of the target data warehouse, increasing the expected quota for the number of files or the expected quota for disk space, and adjusting the first expected ratio of the capacity occupied by the internal tables and the external tables in the expected quota based on the first actual ratio of the capacity occupied by the internal tables and the external tables in the actual usage quota; or, during shrinkage, a shrinkage trigger is activated; the shrinkage trigger is used to reduce the capacity of the target data warehouse, decreasing the expected quota for the number of files or the expected quota for disk space, and adjusting the second expected ratio of the capacity occupied by the internal tables and the external tables in the expected quota based on the second actual ratio of the capacity occupied by the internal tables and the external tables in the actual usage quota. Wherein, when the target adjustment strategy is the expansion strategy, adjusting the expected quota of the target data warehouse according to the target adjustment strategy includes: obtaining the value of a counter, and when the value of the counter is zero, performing a merge operation on the files in the target data warehouse, wherein the counter is preset and is used to accumulate the number of merge operations; obtaining a new actual usage quota, and when the new actual usage quota is still greater than or equal to the high load threshold, expanding the target data warehouse using an expansion trigger; Before monitoring the actual usage of the expected quota by the target data warehouse, the method further includes: creating the target data warehouse, wherein the target data warehouse is used to store data of the target business; and initializing the expected quota of the target data warehouse based on the target business, wherein the expected quota includes file expected quota and space expected quota.

2. The method according to claim 1, characterized in that, After adjusting the expected quota of the target data warehouse according to the target adjustment strategy, the method further includes: Obtain adjustment data, wherein the adjustment data includes record data of adjusting the expected quota and quota usage data before and after the adjustment; The adjustment data is uploaded to the display module so that the display module can show the adjustment data to the administrator.

3. A resource quota adjustment device for a data warehouse, characterized in that, include: The quota monitoring module is used to monitor the actual usage of the target data warehouse's expected quota, wherein the expected quota is the resource quota of the server cluster resources that have been pre-initialized for the target data warehouse; The strategy determination module is used to determine a target adjustment strategy that matches the target load threshold when the actual usage quota reaches the target load threshold of the expected quota. The quota adjustment module is used to adjust the expected quota of the target data warehouse according to the target adjustment strategy; The target load threshold includes a high load threshold and a low load threshold. When the actual usage quota reaches the target load threshold of the expected quota, determining the target adjustment strategy matching the target load threshold includes: if the number of files or disk space used in the actual usage quota is greater than or equal to the high load threshold, determining the expansion strategy as the target adjustment strategy; if the number of files or disk space used in the actual usage quota is less than or equal to the low load threshold, determining the reduction strategy as the target adjustment strategy. Specifically, during expansion, an expansion trigger is activated; the expansion trigger is used to increase the capacity of the target data warehouse, increasing the expected quota for the number of files or the expected quota for disk space, and adjusting the first expected ratio of the capacity occupied by the internal tables and the external tables in the expected quota based on the first actual ratio of the capacity occupied by the internal tables and the external tables in the actual usage quota; or, during shrinkage, a shrinkage trigger is activated; the shrinkage trigger is used to reduce the capacity of the target data warehouse, decreasing the expected quota for the number of files or the expected quota for disk space, and adjusting the second expected ratio of the capacity occupied by the internal tables and the external tables in the expected quota based on the second actual ratio of the capacity occupied by the internal tables and the external tables in the actual usage quota. Wherein, when the target adjustment strategy is the expansion strategy, adjusting the expected quota of the target data warehouse according to the target adjustment strategy includes: obtaining the value of a counter, and when the value of the counter is zero, performing a merge operation on the files in the target data warehouse, wherein the counter is preset and is used to accumulate the number of merge operations; obtaining a new actual usage quota, and when the new actual usage quota is still greater than or equal to the high load threshold, expanding the target data warehouse using an expansion trigger; Before monitoring the actual usage of the expected quota by the target data warehouse, the apparatus further includes: creating the target data warehouse, wherein the target data warehouse is used to store data of the target business; and initializing the expected quota of the target data warehouse based on the target business, wherein the expected quota includes file expected quota and space expected quota.

4. An electronic device comprising a memory, a processor, a communication interface, and a communication bus, wherein the memory stores a computer program executable on the processor, and the memory and the processor communicate via the communication bus and the communication interface, characterized in that... When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 2.

5. A computer-readable medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute the method of any one of claims 1 to 2.

Images