Hive library expansion method and device, processor and electronic equipment

Abstract

The application provides a method and device for expanding a HIVE library, a processor and an electronic device. The method comprises the following steps: obtaining an SQL file, and parsing the SQL file to obtain a current job parameter group corresponding to a HIVE SQL job contained in the SQL file, wherein the current job parameter group at least comprises the following parameters: a current SQL operation corresponding to the HIVE SQL job, a current job name corresponding to the HIVE SQL job, a table name of a HIVE library table currently corresponding to the HIVE SQL job, a current capacity size of the HIVE library table, and a current source data volume corresponding to the HIVE SQL job; inputting the job parameter group corresponding to each HIVE SQL job into a preset machine learning model to obtain a space increment of each HIVE library table; and expanding the HIVE library table corresponding to the space increment according to the space increment, thereby solving the problems that the expansion scheme in the existing art has poor timeliness, the expansion result has a long waiting time, and the space increment is difficult to estimate scientifically.

Description

Technical Field

[0001] This application relates to the field of big data processing, and more specifically, to a method for expanding a HIVE library, as well as an apparatus, processor, and electronic device. Background Technology

[0002] In existing technologies, enterprise big data platforms involve different tenants performing HIVE SQL jobs. Each tenant typically owns multiple HIVE databases. Each HIVE database involves multiple HIVE SQL jobs related to the same business. Therefore, the space of each HIVE database is independent and changes over time. Although the big data platform continuously expands as the volume of business data grows, the total capacity of the big data platform remains constant at any given time. During big data platform management, it is necessary to limit the space of each HIVE database. Each HIVE database has a maximum usable space, i.e., a quota. Each tenant needs to manually submit a "HIVE Database Quota Expansion Request Form" to the cluster administrator to apply for expansion. When approving expansions, the administrator finds it difficult to predict the size of the HIVE database capacity increase, especially since HIVE SQL execution requires a large short-term capacity increase, which is no longer needed after the job is completed.

[0003] The current manual application and approval system for capacity expansion is inefficient, with long waiting times and difficulty in scientifically estimating capacity increases. To avoid operations failing due to insufficient capacity, large capacity increments are often used for expansion, making it difficult to utilize resources effectively.

[0004] There is no effective solution to the aforementioned problems in the existing technology. Summary of the Invention

[0005] The main purpose of this application is to provide a method, apparatus, processor and electronic device for expanding the HIVE library, so as to at least solve the problems of poor timeliness of expansion schemes, long waiting time for expansion results and difficulty in scientifically estimating space increments in the manual application and approval method used in big data platforms in the prior art.

[0006] To achieve the above objectives, according to one aspect of this application, a method for expanding a HIVE database is provided, specifically including: obtaining an SQL file and parsing the SQL file to obtain the current job parameter group corresponding to the HIVE SQL job contained in the SQL file. The current job parameter group includes at least the following parameters: the current SQL operation corresponding to the HIVE SQL job, the current job name corresponding to the HIVE SQL job, the table name of the HIVE database table corresponding to the current HIVE SQL job, the current capacity of the HIVE database table, and the current source data volume corresponding to the HIVE SQL job. The SQL file contains information about at least one HIVE SQL job, and there is a one-to-one mapping relationship between HIVE database tables and HIVE SQL jobs. The HIVE database contains multiple HIVE database tables. The job parameter group corresponding to each HIVE SQL job is input into a preset machine learning model to obtain the space increment corresponding to each HIVE database table. The space increment of the HIVE database table is used to provide capacity support for executing the HIVE SQL job. Based on the space increment, an expansion operation is performed on the HIVE database table corresponding to the space increment.

[0007] Furthermore, before inputting the job parameter group corresponding to each HIVE SQL job into the preset model, the method includes: obtaining multiple sets of historical job parameter groups corresponding to multiple HIVE SQL jobs, each set of historical job parameter groups including at least the following parameters: the SQL operation corresponding to the current HIVE SQL job, the current job name corresponding to the current HIVE SQL job, the table name of the HIVE database table corresponding to the current HIVE SQL job, the current capacity of the HIVE database table, the peak space usage of the HIVE database table corresponding to the execution of the HIVE SQL job, and the source data volume corresponding to the current HIVE SQL job, indicating a one-to-one mapping relationship between the HIVE database table and the HIVE SQL job, with each HIVE SQL job corresponding to multiple sets of historical job parameter groups; obtaining multiple current source data volumes corresponding to multiple HIVE SQL jobs, with a one-to-one correspondence between the HIVE SQL jobs and the current source data volumes; and constructing a preset machine learning model based on the multiple sets of historical job parameter groups and the multiple current source data volumes using a preset machine learning algorithm.

[0008] Furthermore, after expanding the HIVE database table corresponding to the space increment based on the space increment, the method also includes: executing the HIVE SQL job corresponding to the HIVE database table after expanding it based on the space increment; monitoring the usage of the HIVE database table in real time, determining the peak usage of the HIVE database table during the execution of the HIVE SQL job, and updating the peak usage to the historical job HIVE database.

[0009] Furthermore, after controlling the input of the job parameter group corresponding to each HIVE SQL job into the preset machine learning model to obtain the space increment corresponding to each HIVE database table, the method also includes: determining at least one space increment corresponding to at least one HIVE database table; calculating the total increment corresponding to at least one space increment, and determining the total increment as the database space increment of the HIVE database.

[0010] Furthermore, obtaining the SQL file includes: real-time monitoring of the scheduling system's operating status, and obtaining the SQL file when the scheduling system is detected to have started a HIVE SQL job.

[0011] Furthermore, after expanding the HIVE database table corresponding to the space increment based on the space increment, the method includes: monitoring the execution status of each HIVE SQL job; after monitoring that the HIVE SQL job has been completed, determining the current data volume corresponding to the HIVE SQL job; and adjusting the capacity of the HIVE database table corresponding to the HIVE SQL job based on the current data volume.

[0012] According to another aspect of this application, a device for expanding a HIVE database is provided. A first parsing unit is used to acquire an SQL file and parse the SQL file to obtain the current job parameter group corresponding to the HIVE SQL job contained in the SQL file. The current job parameter group includes at least the following parameters: the current SQL operation corresponding to the HIVE SQL job, the current job name corresponding to the HIVE SQL job, the table name of the HIVE database table corresponding to the current HIVE SQL job, the current capacity of the HIVE database table, and the current source data volume corresponding to the HIVE SQL job. The SQL file contains information about at least one HIVE SQL job, and there is a one-to-one mapping relationship between HIVE database tables and HIVE SQL jobs. The HIVE database contains multiple HIVE database tables. A first control unit is used to control the input of the job parameter group corresponding to each HIVE SQL job into a preset machine learning model to obtain the space increment corresponding to each HIVE database table. The space increment of the HIVE database table is used to provide capacity support for executing the HIVE SQL job. An expansion unit is used to perform an expansion operation on the HIVE database table corresponding to the space increment based on the space increment.

[0013] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is running, it controls the device where the computer-readable storage medium is located to execute a method for expanding a HIVE library.

[0014] According to another aspect of this application, a processor is provided for running a program, wherein the program executes a method for expanding a HIVE library during runtime.

[0015] According to another aspect of this application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for performing an expansion of a HIVE library.

[0016] By applying the technical solution of this application, an SQL file is obtained and parsed to obtain the current job parameter group corresponding to the HIVE SQL job contained in the SQL file. The current job parameter group includes at least the following parameters: the current SQL operation corresponding to the HIVE SQL job, the current job name corresponding to the HIVE SQL job, the table name of the HIVE database table corresponding to the current HIVE SQL job, the current capacity of the HIVE database table, and the current source data volume corresponding to the HIVE SQL job. The SQL file contains information about at least one HIVE SQL job, and there is a one-to-one mapping relationship between HIVE database tables and HIVE SQL jobs. The HIVE database contains multiple HIVE database tables. The job parameter group corresponding to each HIVE SQL job is controlled to be input into a preset machine learning model to obtain the space increment corresponding to each HIVE database table. The space increment of the HIVE database table is used to generate space for the execution of HIVE SQL jobs. SQL jobs provide capacity support; based on the space increment, the corresponding HIVE database table is expanded. A machine learning model is used to predict the space increment of the ETL database table, solving the problems of poor timeliness, long waiting times for expansion results, and difficulty in scientifically estimating space increments in existing big data platforms that rely on manual application and approval for expansion. By obtaining the space increment of the HIVE database table through machine learning, the drawbacks of manual estimation are avoided. Furthermore, the new system scientifically predicts the actual capacity required by the HIVE database in a production environment, providing strong technical support for the effective utilization of hard drive capacity resources on big data platforms. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of a mobile terminal for a method of expanding the HIVE library according to an embodiment of the present invention.

[0019] Figure 2This is a flowchart of a method for expanding a HIVE library according to an embodiment of this application;

[0020] Figure 3 A schematic diagram illustrating the relationship between various subsystems in a system provided according to an embodiment of this application is shown;

[0021] Figure 4 A schematic diagram of the corresponding job monitoring system is shown;

[0022] Figure 5 This diagram illustrates how a machine learning model was constructed using multiple sets of historical job parameters corresponding to HIVE SQL jobs.

[0023] Figure 6 A schematic diagram is shown illustrating the determination of the total increment of the HIVE library space;

[0024] Figure 7 This is a schematic diagram of a HIVE library expansion device provided according to an embodiment of this application. Detailed Implementation

[0025] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0029] HIVE is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. It's a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop. The HIVE data warehouse tool can map structured data files to a database table and provide SQL query functionality, transforming SQL statements into MapReduce tasks for execution.

[0030] Big data platform: A data processing platform that processes data of various forms through a series of operations to ultimately form a structured and standardized data throughput for other users and support other systems.

[0031] ETL (Extract-Transform-Load) jobs: ETL is the process of extracting, cleaning, and transforming data from business systems and then loading it into a data warehouse. Its purpose is to integrate scattered, disorganized, and inconsistent data within an enterprise, providing analytical support for decision-making. ETL is a crucial component of BI (Business Intelligence) projects. Typically, HIVE SQL jobs for the same type of business are performed within a single HIVE instance. Each HIVE SQL job corresponds to its own table, and an ETL job comprises multiple HIVE SQL jobs. Generally, an ETL job contains one HIVE SQL file.

[0032] As described in the background section, the manual application and approval method for capacity expansion used in existing big data platforms has problems such as poor timeliness of expansion plans, long waiting time for expansion results, and difficulty in scientifically estimating space increments. In order to solve the above-mentioned problems in the existing technology, the embodiments of this application provide a method, apparatus, processor, and electronic device for expanding the HIVE library.

[0033] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0034] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a method of expanding the HIVE library according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0035] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the device information display method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0036] This embodiment provides a method for expanding a HIVE library that runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0037] This embodiment provides a method that runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0038] Figure 2 This is a flowchart illustrating a method for expanding a HIVE library according to an embodiment of this application. Figure 2As shown, the method includes the following steps:

[0039] Step S201: Obtain the SQL file and parse it to obtain the current job parameter group corresponding to the HIVE SQL job contained in the SQL file. The current job parameter group includes at least the following parameters: the current SQL operation corresponding to the HIVE SQL job, the current job name corresponding to the HIVE SQL job, the table name of the HIVE database table corresponding to the current HIVE SQL job, the current capacity of the HIVE database table, and the current source data volume corresponding to the HIVE SQL job. The SQL file contains information about at least one HIVE SQL job. There is a one-to-one mapping relationship between HIVE database tables and HIVE SQL jobs. The HIVE database contains multiple HIVE database tables. One SQL file contains the current job parameter group corresponding to one HIVE SQL job.

[0040] Specifically, this application provides a system comprising a scheduling subsystem, a job monitoring system, and a capacity estimation system, wherein the relationships between the subsystems are as follows: Figure 3 As shown, the job monitoring system monitors the running status of the scheduling system in real time, and when it detects that the scheduling system has started a HIVE SQL job, it obtains the SQL file sent by the scheduling system.

[0041] The schematic diagram corresponding to the above-mentioned ground operation monitoring system is as follows: Figure 4 As shown.

[0042] As mentioned above, HIVE data warehouse tools can map structured data files to a database table and provide SQL query functionality. They can transform SQL statements into MapReduce tasks for execution. HIVE's advantages include a low learning curve; it allows for rapid MapReduce statistics through SQL-like statements, simplifying MapReduce without requiring the development of dedicated MapReduce applications. HIVE SQL is well-suited for statistical analysis of data warehouses.

[0043] It should be noted that an ETL job contains multiple sub-jobs, and the HIVE SQL job is one of these sub-jobs.

[0044] Step S202: Control the input of the job parameter group corresponding to each HIVE SQL job into the preset machine learning model to obtain the space increment corresponding to each HIVE database table. The space increment of the HIVE database table is used to provide capacity support for executing HIVE SQL jobs.

[0045] Specifically, this application provides a capacity estimation system that uses a preset machine learning model to estimate the space increment of the HIVE database table used to execute HIVE SQL jobs, and expands the HIVE database table according to the estimated space increment to provide capacity support for the execution of HIVE SQL jobs.

[0046] Step S203: Based on the space increment, expand the HIVE database table corresponding to the space increment.

[0047] This embodiment uses machine learning to obtain the space increment of the HIVE database table, thus avoiding the drawbacks of manual intuitive estimation. At the same time, the new system scientifically predicts the actual size of the HIVE database required in the production environment, which can provide a strong guarantee for the effective utilization of hard disk capacity resources of big data platforms.

[0048] In the specific implementation process, before step S202 above, the following steps are also included:

[0049] Retrieve multiple sets of historical job parameter groups corresponding to multiple HIVE SQL jobs. Each set of historical job parameter groups must include at least the following parameters: the SQL operation corresponding to the current HIVE SQL job, the job name corresponding to the current HIVE SQL job, the table name of the HIVE database table corresponding to the current HIVE SQL job, the current capacity of the HIVE database table, the peak space usage of the HIVE database table during the execution of the HIVE SQL job, and the amount of source data corresponding to the current HIVE SQL job. There is a one-to-one mapping relationship between HIVE database tables and HIVE SQL jobs. Each HIVE SQL job corresponds to multiple sets of historical job parameter groups, as detailed below. Figure 5 As shown, Figure 5 The image shows a machine learning model constructed using multiple sets of historical job parameters corresponding to HIVE SQL jobs. Each set of historical job parameters includes the corresponding HIVE SQL operation during the historical execution process, the amount of source data, the corresponding HIVE database table name, the HIVE SQL job name, and the peak value of the spatial increment of the corresponding HIVE database table during the historical execution process. Figure 5 As shown, by inputting multiple sets of historical job parameters into the machine learning model, the data corresponding to the current HIVE SQL job execution can be obtained. Figure 5 The paper also shows how to obtain the predicted spatial increment value of the HIVE library table by inputting the current data into a machine learning model.

[0050] Retrieves multiple current source data volumes corresponding to multiple HIVE SQL jobs. There is a one-to-one correspondence between HIVE SQL jobs and current source data volumes. The current source data volume is as follows: Figure 5The image shows the SQL operations, source data volume, HIVE table name, and HIVE SQL job name corresponding to the current HIVE SQL job. The capacity estimation system uses historical data of the job obtained from the monitoring system and combines it with parameters such as the source data volume of the current job. It then uses machine learning algorithms (such as Least Square, Linear Regression, LSTM, etc.) to build a machine learning model and predicts the capacity of the database tables corresponding to the current job based on the job parameters.

[0051] Based on multiple sets of historical job parameters and multiple current source data volumes, a pre-defined machine learning model is constructed using a pre-defined machine learning algorithm. Since there is a one-to-one correspondence between table names in the HIVE database and ETL jobs, the corresponding job names can be determined after obtaining all tables in the HIVE database. Based on the job names, a table capacity estimation system (such as...) is used... Figure 3 As shown, the increment of the tablespace can be obtained.

[0052] As described above, in the specific implementation process, the space increment of each database table is estimated through the capacity estimation system, and the sum of the space increments of all database tables is calculated to determine the total space increment of the HIVE database. Specifically, as follows... Figure 6 As shown.

[0053] Since there is a one-to-one correspondence between table names in the HIVE database and ETL jobs, once all tables in the HIVE database are obtained, their corresponding job names can be determined. Based on the job names, the "table capacity estimation system" (such as...) can be used to... Figure 6 As shown, the tablespace increment can be obtained. After obtaining all tablespace increments, summing these increments will give the HIVE library's space increment. Then, the system automatically expands the HIVE library based on the increment value. Note: When a job is not started, its corresponding tablespace increment is 0. After the job is completed, the tablespace capacity increment should be the difference between the current tablespace usage and the tablespace usage before the job.

[0054] It should be noted that when a job is not started, the corresponding tablespace increment is 0.

[0055] As described above, after extracting data from the database through the capacity estimation system, a machine learning model is used to fit the data and estimate the capacity increment required for the current job. After the prediction is completed, the predicted value is applied to the HIVE library to ensure that the space required for the job meets the needs.

[0056] After the job is completed, the tablespace increment should be the difference between the current tablespace usage and the tablespace usage before the job.

[0057] In the specific implementation process, after step S203 above, the following steps are also included: executing the HIVE SQL job corresponding to the HIVE database table based on the expanded space increment; monitoring the usage of the HIVE database table in real time, determining the peak usage of the HIVE database table during the execution of the HIVE SQL job, and updating the peak usage to the historical job HIVE database. After the database table is expanded, the HIVE SQL job is executed based on the expanded database table, and the job execution process is monitored in real time. If no HIVE SQL job is executed, the capacity of the corresponding database table is A. During the execution process, an instantaneous space increment is required. Therefore, the instantaneous capacity requirement during job execution may be 3A or 4A. If the limit in the database table is insufficient, an error will be triggered. Therefore, it is necessary to record the peak capacity of the database table during job execution. Reaching the peak capacity required for job execution through expansion will avoid or reduce the probability of errors. At the same time, by recording the peak capacity of the database table during job execution, the peak capacity can be used as historical data to build or correct machine learning models.

[0058] In some embodiments, after step S203, the following step is further included:

[0059] The execution status of each HIVE SQL job is monitored. Upon completion of a HIVE SQL job, the current data volume corresponding to that job is determined. Based on this data volume, the capacity of the corresponding HIVE database tables is adjusted. Since jobs require significant instantaneous space during execution, the space increment of the database tables needs to be predicted and expanded to the peak capacity required during that instantaneous period. After the job completes, the instantaneous usage is released, and the database table limit is restored to C, where C is greater than A but less than the peak capacity of the database table during job execution. By predicting the capacity required during job execution and expanding the database tables accordingly, the database table capacity is restored to the capacity corresponding to the source data of the current job after the job completes. This approach supports the job while also saving database table space.

[0060] This application provides a method for expanding the HIVE library, which solves the problems of poor timeliness of expansion plans, long waiting time for expansion results, and difficulty in scientifically estimating space increments in the manual application and approval method used in existing big data platforms. By obtaining the space increment of the HIVE library table through machine learning, the drawbacks of manual intuitive estimation are avoided. At the same time, the new system scientifically predicts the actual size of the HIVE library required in the production environment, which can provide a strong guarantee for the effective utilization of hard disk capacity resources of big data platforms.

[0061] This application also provides a device for expanding a HIVE library. It should be noted that this device can be used to execute the method for expanding a HIVE library provided in this application. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0062] The following describes a device for expanding a HIVE library according to an embodiment of this application.

[0063] Figure 7 This is a schematic diagram of a HIVE library expansion device according to an embodiment of this application. Figure 7 As shown, the device includes: a first parsing unit 301, used to acquire an SQL file and parse the SQL file to obtain the current job parameter group corresponding to the HIVE SQL job contained in the SQL file. The current job parameter group includes at least the following parameters: the current SQL operation corresponding to the HIVE SQL job, the current job name corresponding to the HIVE SQL job, the table name of the HIVE database table corresponding to the current HIVE SQL job, the current capacity of the HIVE database table, and the current source data volume corresponding to the HIVE SQL job. The SQL file contains information about at least one HIVE SQL job, and there is a one-to-one mapping relationship between the HIVE database table and the HIVE SQL job. The HIVE database contains multiple HIVE database tables. A first control unit 702 is used to control the input of the job parameter group corresponding to each HIVE SQL job into a preset machine learning model to obtain the space increment corresponding to each HIVE database table. The space increment of the HIVE database table is used to provide capacity support for executing the HIVE SQL job. An expansion unit 707 is used to perform an expansion operation on the HIVE database table corresponding to the space increment according to the space increment.

[0064] As an optional solution, the device includes: a first acquisition unit, used to acquire multiple sets of historical job parameter groups corresponding to multiple HIVE SQL jobs before inputting the job parameter group corresponding to each HIVE SQL job into the preset model. Each set of historical job parameter groups includes at least the following parameters: the SQL operation corresponding to the current HIVE SQL job, the current job name corresponding to the current HIVE SQL job, the table name of the HIVE database table corresponding to the current HIVE SQL job, the current capacity of the HIVE database table, the peak space usage of the HIVE database table corresponding to the execution of the HIVE SQL job, and the source data volume corresponding to the current HIVE SQL job. There is a one-to-one mapping relationship between the HIVE database table and the HIVE SQL job, and each HIVE SQL job corresponds to multiple sets of historical job parameter groups; a second acquisition unit, used to acquire multiple current source data volumes corresponding to multiple HIVE SQL jobs, and there is a one-to-one correspondence between the HIVE SQL jobs and the current source data volumes; and a construction unit, used to construct a preset machine learning model based on the multiple sets of historical job parameter groups and the multiple current source data volumes using a preset machine learning algorithm.

[0065] In an optional embodiment, the apparatus further includes: an execution unit, configured to perform a HIVE SQL job corresponding to the HIVE database table after expanding the HIVE database table according to the space increment; and a first determining unit, configured to monitor the usage of the HIVE database table in real time, determine the peak usage of the HIVE database table during the execution of the HIVE SQL job, and update the peak usage to the historical job HIVE database.

[0066] In this embodiment, the device further includes: a second determining unit, configured to determine at least one space increment corresponding to at least one HIVE database table after controlling the input of the job parameter group corresponding to each HIVE SQL job into a preset machine learning model to obtain the space increment corresponding to each HIVE database table; and a third determining unit, configured to calculate the total increment corresponding to at least one space increment and determine the total increment as the database space increment of the HIVE database.

[0067] As an optional solution, the first parsing unit 701 includes: a monitoring subunit, used to monitor the running status of the scheduling system in real time, and to obtain the SQL file when the scheduling system starts a HIVE SQL job.

[0068] An optional solution includes: a monitoring unit, configured to monitor the execution status of each HIVE SQL job after expanding the HIVE database table corresponding to the space increment based on the space increment, and determine the current data volume corresponding to the HIVE SQL job after detecting that the HIVE SQL job has been completed; and an adjustment unit, configured to adjust the capacity of the HIVE database table corresponding to the HIVE SQL job based on the current data volume.

[0069] An expansion device for a HIVE library includes a processor and a memory. The aforementioned first parsing unit 701 and others are stored as program units in the memory, and the processor executes these program units stored in the memory to implement corresponding functions. All of the above modules reside in the same processor; alternatively, the modules may be located in different processors in any combination.

[0070] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting kernel parameters, the problems of poor timeliness, long waiting times for expansion results, and difficulty in scientifically estimating space increments in existing big data platforms using manual application and approval methods can be addressed.

[0071] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0072] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is running, it controls the device where the computer-readable storage medium is located to execute a method for expanding a HIVE library.

[0073] Specifically, one method for expanding a HIVE library includes:

[0074] Step S201: Obtain the SQL file and parse the SQL file to obtain the current job parameter group corresponding to the HIVE SQL job contained in the SQL file. The current job parameter group includes at least the following parameters: the current SQL operation corresponding to the HIVE SQL job, the current job name corresponding to the HIVE SQL job, the table name of the HIVE database table corresponding to the current HIVE SQL job, the current capacity of the HIVE database table, and the current source data volume corresponding to the HIVE SQL job. The SQL file contains information about at least one HIVE SQL job. There is a one-to-one mapping relationship between the HIVE database table and the HIVE SQL job. The HIVE database contains multiple HIVE database tables.

[0075] Specifically, the system monitors the operational status of the scheduling system in real time and retrieves the SQL file when the scheduling system starts a HIVE SQL job. This application provides a system comprising a scheduling subsystem, a job monitoring system, and a capacity estimation system, wherein the relationships between the subsystems are as follows: Figure 3 As shown, when the scheduling system starts a HIVE SQL job, it sends the SQL file to the job monitoring system.

[0076] The schematic diagram corresponding to the above-mentioned ground operation monitoring system is as follows: Figure 4 As shown.

[0077] As mentioned above, HIVE data warehouse tools can map structured data files to a database table and provide SQL query functionality. They can transform SQL statements into MapReduce tasks for execution. HIVE's advantages include a low learning curve; it allows for rapid MapReduce statistics through SQL-like statements, simplifying MapReduce without requiring the development of dedicated MapReduce applications. HIVE SQL is well-suited for statistical analysis of data warehouses.

[0078] It should be noted that an ETL job contains multiple sub-jobs, and the HIVE SQL job is one of these sub-jobs.

[0079] Step S202: Control the input of the job parameter group corresponding to each HIVE SQL job into the preset machine learning model to obtain the space increment corresponding to each HIVE database table. The space increment of the HIVE database table is used to provide capacity support for executing HIVE SQL jobs.

[0080] Specifically, this application provides a capacity estimation system that uses a preset machine learning model to estimate the space increment of the HIVE database table used to execute HIVE SQL jobs, and expands the HIVE database table according to the estimated space increment to provide capacity support for the execution of HIVE SQL jobs.

[0081] Step S203: Based on the space increment, expand the HIVE database table corresponding to the space increment.

[0082] Optionally, before inputting the job parameter group corresponding to each HIVE SQL job into the preset model, the method includes: obtaining multiple sets of historical job parameter groups corresponding to multiple HIVE SQL jobs, each set of historical job parameter groups including at least the following parameters: the SQL operation corresponding to the current HIVE SQL job, the job name corresponding to the current HIVE SQL job, the table name of the HIVE database table corresponding to the current HIVE SQL job, the current capacity of the HIVE database table, the peak space usage of the HIVE database table corresponding to the execution of the HIVE SQL job, the source data volume corresponding to the current HIVE SQL job, and a one-to-one mapping relationship between the HIVE database table and the HIVE SQL job, with each HIVE SQL job corresponding to multiple sets of historical job parameter groups; obtaining multiple current source data volumes corresponding to multiple HIVE SQL jobs, with a one-to-one correspondence between the HIVE SQL jobs and the current source data volumes; and constructing a preset machine learning model based on the multiple sets of historical job parameter groups and the multiple current source data volumes using a preset machine learning algorithm.

[0083] Optionally, after expanding the HIVE database table corresponding to the space increment based on the space increment, the method further includes: executing the HIVE SQL job corresponding to the HIVE database table after the space increment has been expanded; monitoring the usage of the HIVE database table in real time, determining the peak usage of the HIVE database table during the execution of the HIVE SQL job, and updating the peak usage to the historical job HIVE database.

[0084] Optionally, after controlling the input of the job parameter group corresponding to each HIVE SQL job into the preset machine learning model to obtain the space increment corresponding to each HIVE database table, the method further includes: determining at least one space increment corresponding to at least one HIVE database table; calculating the total increment corresponding to at least one space increment, and determining the total increment as the database space increment of the HIVE database.

[0085] Optionally, obtaining the SQL file includes: real-time monitoring of the scheduling system's operating status, and obtaining the SQL file when the scheduling system is detected to have started a HIVE SQL job.

[0086] Optionally, after expanding the HIVE database table corresponding to the space increment based on the space increment, the method includes: monitoring the execution status of each HIVE SQL job; after monitoring that the HIVE SQL job has been completed, determining the current data volume corresponding to the HIVE SQL job; and adjusting the capacity of the HIVE database table corresponding to the HIVE SQL job based on the current data volume.

[0087] This invention provides a processor for running a program, wherein the program executes a method for expanding a HIVE library during runtime.

[0088] Specifically, one method for expanding the HIVE library includes:

[0089] Step S201: Obtain the SQL file and parse the SQL file to obtain the current job parameter group corresponding to the HIVE SQL job contained in the SQL file. The current job parameter group includes at least the following parameters: the current SQL operation corresponding to the HIVE SQL job, the current job name corresponding to the HIVE SQL job, the table name of the HIVE database table corresponding to the current HIVE SQL job, the current capacity of the HIVE database table, and the current source data volume corresponding to the HIVE SQL job. The SQL file contains information about at least one HIVE SQL job. There is a one-to-one mapping relationship between the HIVE database table and the HIVE SQL job. The HIVE database contains multiple HIVE database tables.

[0090] Specifically, the system monitors the operational status of the scheduling system in real time and retrieves the SQL file when the scheduling system starts a HIVE SQL job. This application provides a system comprising a scheduling subsystem, a job monitoring system, and a capacity estimation system, wherein the relationships between the subsystems are as follows: Figure 3 As shown, when the scheduling system starts a HIVE SQL job, it sends the SQL file to the job monitoring system.

[0091] The schematic diagram corresponding to the above-mentioned ground operation monitoring system is as follows: Figure 4 As shown.

[0092] As mentioned above, HIVE data warehouse tools can map structured data files to a database table and provide SQL query functionality. They can transform SQL statements into MapReduce tasks for execution. HIVE's advantages include a low learning curve; it allows for rapid MapReduce statistics through SQL-like statements, simplifying MapReduce without requiring the development of dedicated MapReduce applications. HIVE SQL is well-suited for statistical analysis of data warehouses.

[0093] It should be noted that an ETL job contains multiple sub-jobs, and the HIVE SQL job is one of these sub-jobs.

[0094] Step S202: Control the input of the job parameter group corresponding to each HIVE SQL job into the preset machine learning model to obtain the space increment corresponding to each HIVE database table. The space increment of the HIVE database table is used to provide capacity support for executing HIVE SQL jobs.

[0095] Specifically, this application provides a capacity estimation system that uses a preset machine learning model to estimate the space increment of the HIVE database table used to execute HIVE SQL jobs, and expands the HIVE database table according to the estimated space increment to provide capacity support for the execution of HIVE SQL jobs.

[0096] Step S203: Based on the space increment, expand the HIVE database table corresponding to the space increment.

[0097] Optionally, before inputting the job parameter group corresponding to each HIVE SQL job into the preset model, the method includes: obtaining multiple sets of historical job parameter groups corresponding to multiple HIVE SQL jobs, each set of historical job parameter groups including at least the following parameters: the SQL operation corresponding to the current HIVE SQL job, the job name corresponding to the current HIVE SQL job, the table name of the HIVE database table corresponding to the current HIVE SQL job, the current capacity of the HIVE database table, the peak space usage of the HIVE database table corresponding to the execution of the HIVE SQL job, the source data volume corresponding to the current HIVE SQL job, and a one-to-one mapping relationship between the HIVE database table and the HIVE SQL job, with each HIVE SQL job corresponding to multiple sets of historical job parameter groups; obtaining multiple current source data volumes corresponding to multiple HIVE SQL jobs, with a one-to-one correspondence between the HIVE SQL jobs and the current source data volumes; and constructing a preset machine learning model based on the multiple sets of historical job parameter groups and the multiple current source data volumes using a preset machine learning algorithm.

[0098] Optionally, after expanding the HIVE database table corresponding to the space increment based on the space increment, the method further includes: executing the HIVE SQL job corresponding to the HIVE database table after the space increment has been expanded; monitoring the usage of the HIVE database table in real time, determining the peak usage of the HIVE database table during the execution of the HIVE SQL job, and updating the peak usage to the historical job HIVE database.

[0099] Optionally, after controlling the input of the job parameter group corresponding to each HIVE SQL job into the preset machine learning model to obtain the space increment corresponding to each HIVE database table, the method further includes: determining at least one space increment corresponding to at least one HIVE database table; calculating the total increment corresponding to at least one space increment, and determining the total increment as the database space increment of the HIVE database.

[0100] Optionally, obtaining the SQL file includes: real-time monitoring of the scheduling system's operating status, and obtaining the SQL file when the scheduling system is detected to have started a HIVE SQL job.

[0101] Optionally, after expanding the HIVE database table corresponding to the space increment based on the space increment, the method includes: monitoring the execution status of each HIVE SQL job; after monitoring that the HIVE SQL job has been completed, determining the current data volume corresponding to the HIVE SQL job; and adjusting the capacity of the HIVE database table corresponding to the HIVE SQL job based on the current data volume.

[0102] This invention provides an apparatus, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps: Step S201, obtaining an SQL file and parsing the SQL file to obtain the current job parameter group corresponding to the HIVE SQL job contained in the SQL file. The current job parameter group includes at least the following parameters: the current SQL operation corresponding to the HIVE SQL job, the current job name corresponding to the HIVE SQL job, the table name of the HIVE database table corresponding to the current HIVE SQL job, the current capacity of the HIVE database table, and the current source data volume corresponding to the HIVE SQL job. The SQL file contains information about at least one HIVE SQL job, and there is a one-to-one mapping relationship between the HIVE database table and the HIVE SQL job. The HIVE database contains multiple HIVE database tables.

[0103] Specifically, the system monitors the operational status of the scheduling system in real time and retrieves the SQL file when the scheduling system starts a HIVE SQL job. This application provides a system comprising a scheduling subsystem, a job monitoring system, and a capacity estimation system, wherein the relationships between the subsystems are as follows: Figure 3 As shown, when the scheduling system starts a HIVE SQL job, it sends the SQL file to the job monitoring system.

[0104] The schematic diagram corresponding to the above-mentioned ground operation monitoring system is as follows: Figure 4 As shown.

[0105] As mentioned above, HIVE data warehouse tools can map structured data files to a database table and provide SQL query functionality. They can transform SQL statements into MapReduce tasks for execution. HIVE's advantages include a low learning curve; it allows for rapid MapReduce statistics through SQL-like statements, simplifying MapReduce without requiring the development of dedicated MapReduce applications. HIVE SQL is well-suited for statistical analysis of data warehouses.

[0106] It should be noted that an ETL job contains multiple sub-jobs, and the HIVE SQL job is one of these sub-jobs.

[0107] Step S202: Control the input of the job parameter group corresponding to each HIVE SQL job into the preset machine learning model to obtain the space increment corresponding to each HIVE database table. The space increment of the HIVE database table is used to provide capacity support for executing HIVE SQL jobs.

[0108] Specifically, this application provides a capacity estimation system that uses a preset machine learning model to estimate the space increment of the HIVE database table used to execute HIVE SQL jobs, and expands the HIVE database table according to the estimated space increment to provide capacity support for the execution of HIVE SQL jobs.

[0109] Step S203: Based on the space increment, perform an expansion operation on the HIVE database table corresponding to the space increment. The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0110] Optionally, before inputting the job parameter group corresponding to each HIVE SQL job into the preset model, the method includes: obtaining multiple sets of historical job parameter groups corresponding to multiple HIVE SQL jobs, each set of historical job parameter groups including at least the following parameters: the SQL operation corresponding to the current HIVE SQL job, the job name corresponding to the current HIVE SQL job, the table name of the HIVE database table corresponding to the current HIVE SQL job, the current capacity of the HIVE database table, the peak space usage of the HIVE database table corresponding to the execution of the HIVE SQL job, the source data volume corresponding to the current HIVE SQL job, and a one-to-one mapping relationship between the HIVE database table and the HIVE SQL job, with each HIVE SQL job corresponding to multiple sets of historical job parameter groups; obtaining multiple current source data volumes corresponding to multiple HIVE SQL jobs, with a one-to-one correspondence between the HIVE SQL jobs and the current source data volumes; and constructing a preset machine learning model based on the multiple sets of historical job parameter groups and the multiple current source data volumes using a preset machine learning algorithm.

[0111] Optionally, after expanding the HIVE database table corresponding to the space increment based on the space increment, the method further includes: executing the HIVE SQL job corresponding to the HIVE database table after the space increment has been expanded; monitoring the usage of the HIVE database table in real time, determining the peak usage of the HIVE database table during the execution of the HIVE SQL job, and updating the peak usage to the job history database.

[0112] Optionally, after controlling the input of the job parameter group corresponding to each HIVE SQL job into the preset machine learning model to obtain the space increment corresponding to each HIVE database table, the method further includes: determining at least one space increment corresponding to at least one HIVE database table; calculating the total increment corresponding to at least one space increment, and determining the total increment as the database space increment of the HIVE database.

[0113] Optionally, obtaining the SQL file includes: real-time monitoring of the scheduling system's operating status, and obtaining the SQL file when the scheduling system is detected to have started a HIVE SQL job.

[0114] Optionally, after expanding the HIVE database table corresponding to the space increment based on the space increment, the method includes: monitoring the execution status of each HIVE SQL job; after monitoring that the HIVE SQL job has been completed, determining the current data volume corresponding to the HIVE SQL job; and adjusting the capacity of the HIVE database table corresponding to the HIVE SQL job based on the current data volume.

[0115] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps: Step S201, obtaining an SQL file and parsing the SQL file to obtain the current job parameter group corresponding to the HIVE SQL job contained in the SQL file. The current job parameter group includes at least the following parameters: the current SQL operation corresponding to the HIVE SQL job, the current job name corresponding to the HIVE SQL job, the table name of the HIVE database table currently corresponding to the HIVE SQL job, the current capacity of the HIVE database table, and the current source data volume corresponding to the HIVE SQL job. The SQL file contains information about at least one HIVE SQL job, there is a one-to-one mapping relationship between the HIVE database table and the HIVE SQL job, and the HIVE database contains multiple HIVE database tables.

[0116] Specifically, the system monitors the operational status of the scheduling system in real time and retrieves the SQL file when the scheduling system starts a HIVE SQL job. This application provides a system comprising a scheduling subsystem, a job monitoring system, and a capacity estimation system, wherein the relationships between the subsystems are as follows: Figure 3 As shown, when the scheduling system starts a HIVE SQL job, it sends the SQL file to the job monitoring system.

[0117] The schematic diagram corresponding to the above-mentioned ground operation monitoring system is as follows: Figure 4 As shown.

[0118] As mentioned above, HIVE data warehouse tools can map structured data files to a database table and provide SQL query functionality. They can transform SQL statements into MapReduce tasks for execution. HIVE's advantages include a low learning curve; it allows for rapid MapReduce statistics through SQL-like statements, simplifying MapReduce without requiring the development of dedicated MapReduce applications. HIVE SQL is well-suited for statistical analysis of data warehouses.

[0119] It should be noted that an ETL job contains multiple sub-jobs, and the HIVE SQL job is one of these sub-jobs.

[0120] Step S202: Control the input of the job parameter group corresponding to each HIVE SQL job into the preset machine learning model to obtain the space increment corresponding to each HIVE database table. The space increment of the HIVE database table is used to provide capacity support for executing HIVE SQL jobs.

[0121] Specifically, this application provides a capacity estimation system that uses a preset machine learning model to estimate the space increment of the HIVE database table used to execute HIVE SQL jobs, and expands the HIVE database table according to the estimated space increment to provide capacity support for the execution of HIVE SQL jobs.

[0122] Step S203: Based on the space increment, expand the HIVE database table corresponding to the space increment.

[0123] Optionally, before inputting the job parameter group corresponding to each HIVE SQL job into the preset model, the method includes: obtaining multiple sets of historical job parameter groups corresponding to multiple HIVE SQL jobs, each set of historical job parameter groups including at least the following parameters: the SQL operation corresponding to the current HIVE SQL job, the job name corresponding to the current HIVE SQL job, the table name of the HIVE database table corresponding to the current HIVE SQL job, the current capacity of the HIVE database table, the peak space usage of the HIVE database table corresponding to the execution of the HIVE SQL job, the source data volume corresponding to the current HIVE SQL job, and a one-to-one mapping relationship between the HIVE database table and the HIVE SQL job, with each HIVE SQL job corresponding to multiple sets of historical job parameter groups; obtaining multiple current source data volumes corresponding to multiple HIVE SQL jobs, with a one-to-one correspondence between the HIVE SQL jobs and the current source data volumes; and constructing a preset machine learning model based on the multiple sets of historical job parameter groups and the multiple current source data volumes using a preset machine learning algorithm.

[0124] Optionally, after expanding the HIVE database table corresponding to the space increment based on the space increment, the method further includes: executing the HIVE SQL job corresponding to the HIVE database table after the space increment has been expanded; monitoring the usage of the HIVE database table in real time, determining the peak usage of the HIVE database table during the execution of the HIVE SQL job, and updating the peak usage to the historical job HIVE database.

[0125] Optionally, after controlling the input of the job parameter group corresponding to each HIVE SQL job into the preset machine learning model to obtain the space increment corresponding to each HIVE database table, the method further includes: determining at least one space increment corresponding to at least one HIVE database table; calculating the total increment corresponding to at least one space increment, and determining the total increment as the database space increment of the HIVE database.

[0126] Optionally, obtaining the SQL file includes: real-time monitoring of the scheduling system's operating status, and obtaining the SQL file when the scheduling system is detected to have started a HIVE SQL job.

[0127] Optionally, after expanding the HIVE database table corresponding to the space increment based on the space increment, the method includes: monitoring the execution status of each HIVE SQL job; after monitoring that the HIVE SQL job has been completed, determining the current data volume corresponding to the HIVE SQL job; and adjusting the capacity of the HIVE database table corresponding to the HIVE SQL job based on the current data volume.

[0128] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0129] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0131] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0133] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0134] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0135] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0137] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0138] 1) This application adopts a data-driven approach to monitor the execution of HIVE SQL in ETL jobs and store it in the database. The information in the database is used to scientifically and automatically estimate and arrange the space required by the job, so as to ensure that the limited HIVE library space in the cluster is used efficiently.

[0139] 2) The HIVE library expansion method provided in this application avoids manual and intuitive estimation and instead uses a new system to scientifically predict the actual size of the HIVE library required in the production environment, which can provide a strong guarantee for the effective utilization of hard disk capacity resources of big data platforms.

[0140] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for expanding a HIVE library, characterized in that, include: Obtain the SQL file and parse the SQL file to obtain the current job parameter group corresponding to the HIVE SQL job contained in the SQL file. The current job parameter group includes at least the following parameters: the current SQL operation corresponding to the HIVE SQL job, the current job name corresponding to the HIVE SQL job, the table name of the HIVE database table currently corresponding to the HIVE SQL job, the current capacity of the HIVE database table, and the current source data volume corresponding to the HIVE SQL job. The SQL file contains information about at least one HIVE SQL job, and there is a one-to-one mapping relationship between the HIVE database table and the HIVE SQL job. The HIVE database contains multiple HIVE database tables. The job parameter group corresponding to each HIVE SQL job is controlled to be input into a preset machine learning model to obtain the space increment corresponding to each HIVE database table, wherein the space increment of the HIVE database table is used to provide capacity support for executing the HIVE SQL job; Based on the space increment, the HIVE database table corresponding to the space increment is expanded. Before controlling the input of the job parameter group corresponding to each HIVE SQL job into the preset model, the method includes: obtaining multiple sets of historical job parameter groups corresponding to multiple HIVE SQL jobs, each set of historical job parameter groups including at least the following parameters: the SQL operation corresponding to the current HIVE SQL job, the job name corresponding to the current HIVE SQL job, the table name of the HIVE database table corresponding to the current HIVE SQL job, the current capacity of the HIVE database table, the peak space usage of the HIVE database table corresponding to the execution of the HIVE SQL job, the amount of source data corresponding to the current HIVE SQL job, and there is a one-to-one mapping relationship between the HIVE database table and the HIVE SQL job, with each HIVE SQL job corresponding to multiple sets of historical job parameter groups; obtaining multiple current source data volumes corresponding to multiple HIVE SQL jobs, with a one-to-one correspondence between the HIVE SQL jobs and the current source data volumes; and constructing the preset machine learning model based on the multiple sets of historical job parameter groups and the multiple current source data volumes using a preset machine learning algorithm.

2. The method according to claim 1, characterized in that, After expanding the HIVE database table corresponding to the space increment based on the space increment, the method further includes: Based on the HIVE database tables after the space incremental expansion, execute the HIVE SQL jobs corresponding to the HIVE database tables; The usage of the HIVE database tables is monitored in real time, and the peak usage of the HIVE database tables during the execution of the HIVE SQL job is determined. The peak usage is then updated in the job history HIVE database.

3. The method according to claim 1, characterized in that, After controlling the input of the job parameter group corresponding to each HIVE SQL job into a preset machine learning model to obtain the space increment corresponding to each HIVE database table, the method further includes: Determine at least one of the space increments corresponding to at least one of the aforementioned HIVE library tables; Calculate the total increment corresponding to at least one of the space increments, and determine the total increment as the database space increment of the HIVE database.

4. The method according to claim 1, characterized in that, Obtain the SQL file, including: The system monitors the operation status of the scheduling system in real time and obtains the SQL file when it detects that the scheduling system has started a HIVE SQL job.

5. The method according to claim 1, characterized in that, After expanding the HIVE database table corresponding to the space increment based on the space increment, the method includes: Monitor the execution status of each HIVE SQL job, and determine the current data volume corresponding to the HIVE SQL job after the HIVE SQL job is detected to have been completed; Based on the current data volume, adjust the capacity of the HIVE database table corresponding to the HIVE SQL job.

6. A device for expanding a HIVE library, characterized in that, include: The first parsing unit is used to obtain an SQL file and parse the SQL file to obtain the current job parameter group corresponding to the HIVE SQL job contained in the SQL file. The current job parameter group includes at least the following parameters: the current SQL operation corresponding to the HIVE SQL job, the current job name corresponding to the HIVE SQL job, the table name of the HIVE database table currently corresponding to the HIVE SQL job, the current capacity of the HIVE database table, and the current source data volume corresponding to the HIVE SQL job. The SQL file contains information about at least one HIVE SQL job, and there is a one-to-one mapping relationship between the HIVE database table and the HIVE SQL job. The HIVE database contains multiple HIVE database tables. The first control unit is used to control the input of the job parameter group corresponding to each HIVE SQL job into a preset machine learning model to obtain the space increment corresponding to each HIVE database table, wherein the space increment of the HIVE database table is used to provide capacity support for executing the HIVE SQL job; The expansion unit is used to expand the HIVE database table corresponding to the space increment based on the space increment. The device includes: a first acquisition unit, configured to acquire multiple sets of historical job parameter groups corresponding to multiple HIVE SQL jobs, each set of historical job parameter groups including at least the following parameters: the SQL operation corresponding to the current HIVE SQL job, the current job name corresponding to the current HIVE SQL job, the table name of the HIVE database table corresponding to the current HIVE SQL job, the current capacity of the HIVE database table, the peak space usage of the HIVE database table corresponding to the execution of the HIVE SQL job, the amount of source data corresponding to the current HIVE SQL job, and a one-to-one mapping relationship between the HIVE database table and the HIVE SQL job, with each HIVE SQL job corresponding to multiple sets of historical job parameter groups; a second acquisition unit, configured to acquire multiple current source data volumes corresponding to multiple HIVE SQL jobs, with a one-to-one correspondence between the HIVE SQL jobs and the current source data volumes; and a construction unit, configured to construct the preset machine learning model based on the multiple sets of historical job parameter groups and the multiple current source data volumes using a preset machine learning algorithm.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform a method for expanding a HIVE library as described in any one of claims 1 to 5.

8. A processor, characterized in that, The processor is used to run a program, wherein the program executes a method for expanding a HIVE library according to any one of claims 1 to 5.

9. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a method for performing an expansion of a HIVE library as described in any one of claims 1 to 5.

Images