A multi-version Hive support method, device, equipment and medium based on Sqoop

Abstract

The present application discloses a method, apparatus, device and medium for supporting multi-version Hive based on Sqoop, including: extracting Hive command parameters from a first Sqoop command; wherein the first Sqoop command is a Sqoop command for importing data into Hive; determining Hadoop parameters corresponding to the Hadoop version on which the Hive version to be connected depends based on the Hive command parameters and the Sqoop configuration file; determining the first MapReduce parameters and the first MapReduce method corresponding to the Hive version to be connected based on the Hive command parameters and the Hadoop parameters; generating a MapReduce job based on the first MapReduce parameters and the first MapReduce method. In this way, Sqoop can support multi-version Hive.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a Sqoop-based multi-version Hive support method, device, equipment and medium. Background Art

[0002] Sqoop is a tool for transferring data between Hadoop and relational databases, using Hadoop MapReduce to extract and load data. Hive is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. It is a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop. Currently, the Sqoop component cannot support multiple versions of Hive. Summary of the invention

[0003] In view of this, the purpose of this application is to provide a multi-version Hive support method, device, equipment and medium based on Sqoop, which can enable Sqoop to support multi-version Hive. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a multi-version Hive support method based on Sqoop, comprising:

[0005] Extracting Hive command parameters from a first Sqoop command; wherein the first Sqoop command is a Sqoop command for importing data into Hive;

[0006] Based on the Hive command parameters and the Sqoop configuration file, determine the Hadoop parameters corresponding to the Hadoop version that the Hive version to be connected depends on;

[0007] Determine, based on the Hive command parameters and the Hadoop parameters, a first MapReduce parameter and a first MapReduce method corresponding to the Hive version to be connected;

[0008] A MapReduce job is generated based on the first MapReduce parameter and the first MapReduce method.

[0009] Optionally, the determining, based on the Hive command parameters and the Sqoop configuration file, Hadoop parameters corresponding to the Hadoop version on which the Hive version to be connected depends includes:

[0010] Extract Hive version configuration information from the configuration file;

[0011] Based on the Hive command parameters and the Hive version configuration information, a Hadoop parameter corresponding to the Hadoop version on which the Hive version to be connected depends is determined.

[0012] Optionally, extracting a Hive command parameter from the first Sqoop command includes:

[0013] When the preset shell script file is executed, the Hive command parameter is extracted from the first Sqoop command in the shell script file.

[0014] Optionally, the method further includes:

[0015] Determine the second MapReduce parameter and the second MapReduce method based on the second Sqoop command; wherein the second Sqoop command is a Sqoop command unrelated to Hive import.

[0016] Optionally, the generating a MapReduce job based on the first MapReduce parameter and the first MapReduce method includes:

[0017] A MapReduce job file is generated based on the first MapReduce parameter, the first MapReduce method, the second MapReduce parameter, and the second MapReduce method.

[0018] Optionally, the method further includes:

[0019] Packaging the MapReduce job file into a job jar package;

[0020] Send the job jar package to the MapReduce cluster for operation and processing.

[0021] In a second aspect, the present application discloses a multi-version Hive support device based on Sqoop, comprising:

[0022] A Hive command parameter extraction module, used to extract Hive command parameters from a first Sqoop command; wherein the first Sqoop command is a Sqoop command for importing data into Hive;

[0023] A Hadoop parameter determination module, used to determine the Hadoop parameters corresponding to the Hadoop version on which the Hive version to be connected depends based on the Hive command parameters and the Sqoop configuration file;

[0024] A first MapReduce parameter and method determination module, configured to determine a first MapReduce parameter and a first MapReduce method corresponding to the Hive version to be connected based on the Hive command parameters and the Hadoop parameters;

[0025] A MapReduce job generation module is used to generate a MapReduce job based on the first MapReduce parameter and the first MapReduce method.

[0026] Optionally, the Hadoop parameter determination module is specifically used to:

[0027] Extract Hive version configuration information from the configuration file;

[0028] Based on the Hive command parameters and the Hive version configuration information, a Hadoop parameter corresponding to the Hadoop version on which the Hive version to be connected depends is determined.

[0029] In a third aspect, the present application discloses an electronic device, including a processor and a memory; wherein:

[0030] The memory is used to store the computer program;

[0031] The processor is used to execute the computer program to implement the aforementioned Sqoop-based multi-version Hive support method.

[0032] In a fourth aspect, the present application discloses a computer-readable storage medium, characterized in that it is used to store a computer program, wherein the computer program implements the aforementioned Sqoop-based multi-version Hive support method when executed by a processor.

[0033] It can be seen that the present application first extracts Hive command parameters from the first Sqoop command; wherein the first Sqoop command is a Sqoop command for importing data into Hive, and then based on the Hive command parameters and the Sqoop configuration file, determines the Hadoop parameters corresponding to the Hadoop version on which the Hive version to be connected depends, and then based on the Hive command parameters and the Hadoop parameters, determines the first MapReduce parameters and the first MapReduce method corresponding to the Hive version to be connected, and finally generates a MapReduce job based on the first MapReduce parameters and the first MapReduce method. That is, the present application first extracts Hive command parameters from the Sqoop command for importing data into Hive, and then based on the Hive command parameters and the Sqoop configuration file, determines the Hadoop parameters corresponding to the Hadoop version on which the Hive version to be connected depends, and for any Hive version, the Hadoop parameters corresponding to the Hadoop version it depends on can be determined in this way, and then the corresponding MapReduce parameters and methods can be determined, and finally the MapReduce job is performed based on the MapReduce parameters and methods, so that Sqoop can support multiple versions of Hive. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0035] Figure 1 A flow chart of a multi-version Hive support method based on Sqoop disclosed in this application;

[0036] Figure 2 A flowchart of a specific Sqoop-based multi-version Hive support method disclosed in this application;

[0037] Figure 3 A specific schematic diagram of multi-version Hive support based on Sqoop disclosed in this application;

[0038] Figure 4 A schematic diagram of a specific Sqoop-based multi-version Hive support device structure disclosed in this application;

[0039] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0041] Sqoop is a tool for transferring data between Hadoop and relational databases, using Hadoop MapReduce to extract and load data. Hive is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. It is a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop. At present, the Sqoop component cannot support multiple versions of Hive. To this end, the present application provides a multi-version Hive support solution based on Sqoop, which enables Sqoop to support multiple versions of Hive.

[0042] See also Figure 1 As shown, the embodiment of the present application discloses a multi-version Hive support method based on Sqoop, including:

[0043] Step S11: extracting Hive command parameters from the first Sqoop command; wherein the first Sqoop command is a Sqoop command for importing data into Hive.

[0044] In a specific implementation manner, this embodiment can extract the Hive command parameter from the first Sqoop command in the shell script file when the preset shell script file is executed.

[0045] Specifically, you can write a Sqoop command for importing data into Hive in advance according to the Sqoop command rules to obtain a Sqoop command script, save the Sqoop command script into a shell file, execute the shell script file, and when Sqoop reads the command parameters that Sqoop can recognize, it can read all the command parameters and extract the command parameters related to Hive, such as hive-table, hive-create-table and other parameters in the Sqoop command parameters;

[0046] Step S12: Based on the Hive command parameters and the Sqoop configuration file, determine the Hadoop parameters corresponding to the Hadoop version that the Hive version to be connected depends on.

[0047] In a specific implementation manner, Hive version configuration information may be extracted from the configuration file; and Hadoop parameters corresponding to the Hadoop version on which the Hive version to be connected depends are determined based on the Hive command parameters and the Hive version configuration information.

[0048] That is, in this embodiment, the Hive version to be connected can be configured in the Sqoop configuration file, and then the Hadoop version and corresponding Hadoop parameters of the Hive version to be connected to be dependent on can be determined.

[0049] It should be noted that different versions of Hive need to rely on different Hadoop versions. If the corresponding error occurs, an "Unrecognized Hadoop major version number" error will be generated. Therefore, determine the Hadoop version that the Hive version to be connected depends on, and then determine the corresponding Hadoop parameters;

[0050] Step S13: Based on the Hive command parameters and the Hadoop parameters, determine the first MapReduce parameters and the first MapReduce method corresponding to the Hive version to be connected.

[0051] Step S14: Generate a MapReduce job based on the first MapReduce parameter and the first MapReduce method.

[0052] It can be seen that the embodiment of the present application first extracts the Hive command parameters from the first Sqoop command; wherein the first Sqoop command is a Sqoop command for importing data into Hive, and then based on the Hive command parameters and the Sqoop configuration file, determines the Hadoop parameters corresponding to the Hadoop version on which the Hive version to be connected depends, and then based on the Hive command parameters and the Hadoop parameters, determines the first MapReduce parameters and the first MapReduce method corresponding to the Hive version to be connected, and finally generates a MapReduce job based on the first MapReduce parameters and the first MapReduce method. That is, the embodiment of the present application first extracts the Hive command parameters from the Sqoop command for importing data into Hive, and then based on the Hive command parameters and the Sqoop configuration file, determines the Hadoop parameters corresponding to the Hadoop version on which the Hive version to be connected depends, and for any Hive version, the Hadoop parameters corresponding to the Hadoop version it depends on can be determined in this way, and then the corresponding MapReduce parameters and methods can be determined, and finally the MapReduce job is performed based on the MapReduce parameters and methods, so that Sqoop can support multiple versions of Hive.

[0053] See also Figure 2 As shown, the embodiment of the present application discloses a multi-version Hive support method based on Sqoop, including:

[0054] Step S21: extracting Hive command parameters from the first Sqoop command; wherein the first Sqoop command is a Sqoop command for importing data into Hive.

[0055] Step S22: Based on the Hive command parameters and the Sqoop configuration file, determine the Hadoop parameters corresponding to the Hadoop version that the Hive version to be connected depends on.

[0056] Step S23: Based on the Hive command parameters and the Hadoop parameters, determine the first MapReduce parameters and the first MapReduce method corresponding to the Hive version to be connected.

[0057] Step S24: determining a second MapReduce parameter and a second MapReduce method based on a second Sqoop command; wherein the second Sqoop command is a Sqoop command unrelated to Hive import.

[0058] Step S25: Generate a MapReduce job file based on the first MapReduce parameter, the first MapReduce method, the second MapReduce parameter, and the second MapReduce method.

[0059] Step S26: Pack the MapReduce job file into a job jar package.

[0060] Step S27: Send the job jar package to the MapReduce cluster for operation processing.

[0061] That is, in a specific implementation manner, this embodiment can extract Hive-related parameters in the Sqoop command that imports data into Hive, determine the Hadoop version and corresponding Hadoop parameters on which the Hive version to be connected depends based on the Hive-related parameters and the configuration file, and then determine the MapReduce parameters and MapReduce methods related to the Hive import, and determine the corresponding MapReduce parameters and MapReduce methods based on the Sqoop command that is not related to the Hive import, generate a MapReduce job file with the MapReduce parameters and MapReduce methods determined by the above two parts, package the MapReduce job file into a job jar package, and then send the job jar package to the MapReduce cluster for operation and processing.

[0062] For example, see Figure 3 As shown, Figure 3 This is a specific schematic diagram of multi-version Hive support based on Sqoop disclosed in an embodiment of the present application. In a specific implementation, corresponding Hive command identifiers, Hadoop version selectors, Hive version selectors, MR job converters, and MR jar package uploaders can be written. In the specific implementation, Hive versions include Hive3 and Hive2, etc. Specifically:

[0063] The Hive command identifier is used to parse the Sqoop command and identify the Hive-related command parameters. The Hive version to be connected is configured in the Sqoop configuration file and sent to the Hadoop version selector.

[0064] After obtaining the Hive-related command parameters extracted from the Sqoop command through the Hadoop version selector, the Hadoop version that Hive depends on and the corresponding Hadoop parameters are determined according to the parameters and the Hive version to be connected configured in the Sqoop configuration file, and then sent to the Hive version selector.

[0065] After obtaining the Hive-related command parameters extracted from the Sqoop command and the Hadoop-related parameters relied on in the previous step through the Hive version selector, generate the parameters and methods corresponding to the Hive version according to the parameters and configuration to connect to, and send them to the MR job converter.

[0066] After receiving the MapReduce parameters and methods corresponding to the Hive version generated by the Hive version selector through the MR job converter, the parameters and methods of the MapReduce job generated by the Sqoop command that are not related to the Hive import command are integrated and generated into a complete MapReduce job file, and the generated complete MapReduce job file is sent to the MR jar package uploader.

[0067] The MapReduce job file generated by the MR job converter is received through the MR jar package uploader, and then packaged to generate a job jar package that can be recognized by the MapReduce cluster, and uploaded to the MapReduce cluster for running and processing.

[0068] In this way, the functionality of Sqoop is improved, and the two problems that Sqoop currently has, such as the inability to fully support multiple versions at the same time, such as Hive2 and Hive, and the need to change the dependent Hadoop version when switching Hive versions, are solved. This expands the functionality of Sqoop and improves the usability of Sqoop components.

[0069] See also Figure 4 As shown, the present application discloses a multi-version Hive support device based on Sqoop, including:

[0070] A Hive command parameter extraction module 11 is used to extract Hive command parameters from a first Sqoop command; wherein the first Sqoop command is a Sqoop command for importing data into Hive;

[0071] A Hadoop parameter determination module 12 is used to determine the Hadoop parameters corresponding to the Hadoop version that the Hive version to be connected depends on based on the Hive command parameters and the Sqoop configuration file;

[0072] A first MapReduce parameter and method determination module 13, configured to determine a first MapReduce parameter and a first MapReduce method corresponding to the Hive version to be connected based on the Hive command parameters and the Hadoop parameters;

[0073] The MapReduce job generating module 14 is configured to generate a MapReduce job based on the first MapReduce parameter and the first MapReduce method.

[0074] It can be seen that the embodiment of the present application first extracts Hive command parameters from the first Sqoop command; wherein the first Sqoop command is a Sqoop command for importing data into Hive, and then based on the Hive command parameters and the Sqoop configuration file, determines the Hadoop parameters corresponding to the Hadoop version on which the Hive version to be connected depends, and then based on the Hive command parameters and the Hadoop parameters, determines the first MapReduce parameters and the first MapReduce method corresponding to the Hive version to be connected, and finally generates a MapReduce job based on the first MapReduce parameters and the first MapReduce method. That is, the present application first extracts Hive command parameters from the Sqoop command for importing data into Hive, and then based on the Hive command parameters and the Sqoop configuration file, determines the Hadoop parameters corresponding to the Hadoop version on which the Hive version to be connected depends, and for any Hive version, the Hadoop parameters corresponding to the Hadoop version it depends on can be determined in this way, and then the corresponding MapReduce parameters and methods can be determined, and finally the MapReduce job is performed based on the MapReduce parameters and methods, so that Sqoop can support multiple versions of Hive.

[0075] The Hadoop parameter determination module 12 specifically includes:

[0076] A Hive version configuration information extraction unit, used to extract Hive version configuration information from the configuration file;

[0077] The Hadoop parameter determination unit is used to determine the Hadoop parameters corresponding to the Hadoop version on which the Hive version to be connected depends based on the Hive command parameters and the Hive version configuration information.

[0078] The Hive command parameter extraction module 11 is specifically used to extract the Hive command parameter from the first Sqoop command in the shell script file when a preset shell script file is executed.

[0079] The device also includes a second MapReduce parameter and method determination module, which is used to determine second MapReduce parameters and a second MapReduce method based on a second Sqoop command; wherein the second Sqoop command is a Sqoop command unrelated to Hive import.

[0080] Accordingly, the MapReduce job generation module 14 specifically includes:

[0081] A MapReduce job file generating unit is used to generate a MapReduce job file based on the first MapReduce parameter, the first MapReduce method, the second MapReduce parameter and the second MapReduce method.

[0082] The device also includes:

[0083] A job jar package packaging module is used to package the MapReduce job file into a job jar package;

[0084] The job jar package sending module is used to send the job jar package to the MapReduce cluster for running and processing.

[0085] See also Figure 5 As shown, the embodiment of the present application discloses an electronic device 20, which includes a processor 21 and a memory 22; wherein the memory 22 is used to store a computer program; and the processor 21 is used to execute the computer program to implement the following steps:

[0086] Extracting Hive command parameters from a first Sqoop command; wherein the first Sqoop command is a Sqoop command for importing data into Hive; determining Hadoop parameters corresponding to the Hadoop version on which the Hive version to be connected depends based on the Hive command parameters and a Sqoop configuration file; determining first MapReduce parameters and a first MapReduce method corresponding to the Hive version to be connected based on the Hive command parameters and the Hadoop parameters; and generating a MapReduce job based on the first MapReduce parameters and the first MapReduce method.

[0087] It can be seen that the embodiment of the present application first extracts Hive command parameters from the first Sqoop command; wherein the first Sqoop command is a Sqoop command for importing data into Hive, and then based on the Hive command parameters and the Sqoop configuration file, determines the Hadoop parameters corresponding to the Hadoop version on which the Hive version to be connected depends, and then based on the Hive command parameters and the Hadoop parameters, determines the first MapReduce parameters and the first MapReduce method corresponding to the Hive version to be connected, and finally generates a MapReduce job based on the first MapReduce parameters and the first MapReduce method. That is, the present application first extracts Hive command parameters from the Sqoop command for importing data into Hive, and then based on the Hive command parameters and the Sqoop configuration file, determines the Hadoop parameters corresponding to the Hadoop version on which the Hive version to be connected depends, and for any Hive version, the Hadoop parameters corresponding to the Hadoop version it depends on can be determined in this way, and then the corresponding MapReduce parameters and methods can be determined, and finally the MapReduce job is performed based on the MapReduce parameters and methods, so that Sqoop can support multiple versions of Hive.

[0088] In this embodiment, when the processor 21 executes the computer subroutine stored in the memory 22, the following steps may be specifically implemented: extracting Hive version configuration information from the configuration file; and determining, based on the Hive command parameters and the Hive version configuration information, Hadoop parameters corresponding to the Hadoop version on which the Hive version to be connected depends.

[0089] In this embodiment, when the processor 21 executes the computer subroutine stored in the memory 22, the following steps may be specifically implemented: when a preset shell script file is executed, the Hive command parameter is extracted from the first Sqoop command in the shell script file.

[0090] In this embodiment, when the processor 21 executes the computer subroutine stored in the memory 22, the following steps can be specifically implemented: determining a second MapReduce parameter and a second MapReduce method based on a second Sqoop command; wherein the second Sqoop command is a Sqoop command unrelated to Hive import.

[0091] In this embodiment, when the processor 21 executes the computer subroutine stored in the memory 22, the following steps may be specifically implemented: generating a MapReduce job file based on the first MapReduce parameter, the first MapReduce method, the second MapReduce parameter, and the second MapReduce method.

[0092] In this embodiment, when the processor 21 executes the computer subroutine stored in the memory 22, the following steps may be specifically implemented: packaging the MapReduce job file into a job jar package; and sending the job jar package to a MapReduce cluster for operation and processing.

[0093] Furthermore, the memory 22 as a carrier for storing resources may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.

[0094] In addition, the electronic device 20 also includes a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26; wherein the power supply 23 is used to provide working voltage for each hardware device on the server 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input / output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0095] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium for storing a computer program, wherein when the computer program is executed by a processor, the multi-version Hive support method based on Sqoop disclosed in the aforementioned embodiment is implemented.

[0096] For the specific process of the above-mentioned Sqoop-based multi-version Hive support method, reference may be made to the corresponding content disclosed in the aforementioned embodiments, which will not be repeated here.

[0097] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0098] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0099] The above is a detailed introduction to a multi-version Hive support method, device, equipment and medium based on Sqoop provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A multi-version Hive support method based on Sqoop, characterized in that: include: Extracting Hive command parameters from a first Sqoop command; wherein the first Sqoop command is a Sqoop command for importing data into Hive; Based on the Hive command parameters and the Sqoop configuration file, determine the Hadoop parameters corresponding to the Hadoop version that the Hive version to be connected depends on; Determine, based on the Hive command parameters and the Hadoop parameters, a first MapReduce parameter and a first MapReduce method corresponding to the Hive version to be connected; generating a MapReduce job based on the first MapReduce parameter and the first MapReduce method; The determining, based on the Hive command parameters and the Sqoop configuration file, Hadoop parameters corresponding to the Hadoop version on which the Hive version to be connected depends includes: Extract Hive version configuration information from the configuration file; Based on the Hive command parameters and the Hive version configuration information, a Hadoop parameter corresponding to the Hadoop version on which the Hive version to be connected depends is determined.

2. The multi-version Hive support method based on Sqoop according to claim 1, characterized in that: The extracting of Hive command parameters from the first Sqoop command includes: When the preset shell script file is executed, the Hive command parameter is extracted from the first Sqoop command in the shell script file.

3. The multi-version Hive support method based on Sqoop according to claim 1, characterized in that: Also includes: Determine the second MapReduce parameter and the second MapReduce method based on the second Sqoop command; wherein the second Sqoop command is a Sqoop command unrelated to Hive import.

4. The multi-version Hive support method based on Sqoop according to claim 3 is characterized in that: The generating a MapReduce job based on the first MapReduce parameter and the first MapReduce method includes: A MapReduce job file is generated based on the first MapReduce parameter, the first MapReduce method, the second MapReduce parameter, and the second MapReduce method.

5. The multi-version Hive support method based on Sqoop according to claim 4 is characterized in that: After the MapReduce job file is generated based on the first MapReduce parameter, the first MapReduce method, the second MapReduce parameter, and the second MapReduce method, the method further includes: Packaging the MapReduce job file into a job jar package; Send the job jar package to the MapReduce cluster for operation and processing.

6. A multi-version Hive support device based on Sqoop, characterized in that: include: A Hive command parameter extraction module, used to extract Hive command parameters from a first Sqoop command; wherein the first Sqoop command is a Sqoop command for importing data into Hive; A Hadoop parameter determination module, used to determine the Hadoop parameters corresponding to the Hadoop version on which the Hive version to be connected depends based on the Hive command parameters and the Sqoop configuration file; A first MapReduce parameter and method determination module, configured to determine a first MapReduce parameter and a first MapReduce method corresponding to the Hive version to be connected based on the Hive command parameters and the Hadoop parameters; A MapReduce job generation module, configured to generate a MapReduce job based on the first MapReduce parameter and the first MapReduce method; The Hadoop parameter determination module is specifically used for: Extract Hive version configuration information from the configuration file; Based on the Hive command parameters and the Hive version configuration information, a Hadoop parameter corresponding to the Hadoop version on which the Hive version to be connected depends is determined.

7. An electronic device, characterized in that: comprising a processor and a memory; wherein, The memory is used to store the computer program; The processor is used to execute the computer program to implement the Sqoop-based multi-version Hive support method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the Sqoop-based multi-version Hive support method according to any one of claims 1 to 5 is implemented.

Images