HIVE data exporting method, electronic equipment and storage medium
By generating statement identifiers and export rule tables, and utilizing parallel queries and data interface addresses, the problems of low efficiency and poor security in exporting massive amounts of data in HIVE are solved, achieving efficient and secure data export.
Patent Information
- Application Number
- CN202410988980.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2026-01-23
AI Technical Summary
In existing technologies, the methods for exporting massive amounts of data from HIVE are inefficient and insecure, and are prone to failure midway, failing to meet the requirements for efficient and secure data export.
By generating statement identifiers and parsing query statements to determine the export rule table, the target data is searched in the target data cluster according to preset rules, and the data is stored in the corresponding file in the storage directory. The file is exported using parallel queries and data interface addresses, and the file lifecycle and updates are managed to reduce the number of network transmissions.
It improves the efficiency and security of HIVE data export, reduces the high time consumption caused by network communication, and enables rapid data acquisition.
Smart Images
Figure CN121387964A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data export, in particular to a HIVE data export method, electronic equipment, storage medium and computer program equipment. BACKGROUND
[0002] Hive is a database for mass data storage, and is a core component of a big data offline data warehouse, and plays a very important role. In most scenarios, an algorithm or a model often needs to be trained with massive data to improve its accuracy. These data are not one-time, so generally some official websites will be downloaded in advance to prepare data in Hive, and then downloaded when needed for calculation or training. For such scenarios, conditional export of Hive data will occur frequently. On the one hand, for the scenario of HIVE mass data, the efficiency of the general JDBC export method is very low and unsafe, and is prone to failure. SUMMARY
[0003] The purpose of the embodiments of the present application is to provide a HIVE data export method that can be efficient and safe.
[0004] In order to achieve the above purpose, the first aspect of the present application provides a HIVE data export method, the method comprising:
[0005] obtaining a target query statement for a target data cluster;
[0006] generating a statement identifier corresponding to the target query statement;
[0007] generating a target export file name for storing an export file in a storage directory according to the statement identifier, wherein the storage directory refers to a storage directory of the export file created on a distributed file system corresponding to the target data cluster;
[0008] parsing the target query statement to determine a data export rule table corresponding to the target query statement, wherein the data export rule table defines a data type name and a field name of the exported data;
[0009] According to a preset rule, finding out the corresponding target data in the target data cluster according to the target query statement;
[0010] defining the target data according to the data export rule table, and storing the target data in a file corresponding to the target export file name in the storage directory.
[0011] In the embodiment of the present application, the target data is found according to the target query statement in the target data cluster according to the preset rule, which comprises: dividing the data in the target data cluster equally to obtain a preset number of data sets; copying the target query statement to obtain a preset number of target statements; querying all data sets in the target data cluster simultaneously by all target query statements to obtain sub-target data in each data set; and determining the union of all sub-target data as the target data.
[0012] In the embodiment of the present application, the method further comprises: in the case that the data export rule table corresponding to the target query statement cannot be determined by analyzing the target query statement, creating a preset condition for the target query statement, so that the target query statement queries an empty data set under the preset condition, wherein the empty data set comprises a data type name and a field name; and establishing the data export rule table corresponding to the target query statement according to the data type name and the field name included in the empty data set.
[0013] In the embodiment of the present application, the method further comprises: after generating the target export file name for storing the export file in the storage directory according to the statement identifier, judging whether the target export file name already exists; and in the case that it is determined that the target export file name already exists, determining the file corresponding to the target export file name in the storage directory as the target export file corresponding to the target query statement.
[0014] In the embodiment of the present application, the method further comprises: obtaining the data interface address of the target data cluster; and after storing the target data in the file corresponding to the target export file name in the storage directory, exporting the file corresponding to the target export file name through the data interface address.
[0015] In the embodiment of the present application, the method further comprises: obtaining the update frequency of the cached data in each file in the storage directory; determining the life cycle of each file according to the update frequency; and re-determining the target data corresponding to the file according to the life cycle of the file, and replacing the stored data with the re-determined target data.
[0016] In the embodiment of the present application, the method further comprises: in the case that it is determined that there is an empty file in the storage directory, updating the storage directory.
[0017] The second aspect of the present application provides an electronic device comprising: a processor, and a memory connected with the processor in communication;
[0018] The memory stores computer execution instructions;
[0019] The processor executes the computer execution instructions stored in the memory to implement the method of HIVE data export in any one of the above embodiments.
[0020] The third aspect of the present application provides a machine readable storage medium, the machine readable storage medium stores instructions for causing a machine to execute the method of HIVE data export according to any one of the above embodiments.
[0021] The fourth aspect of the present application provides a computer program product, comprising a computer program, the computer program implements the method of HIVE data export according to any one of the above embodiments when executed by a processor.
[0022] Through the above technical solution, the export file corresponding to the target query statement is established, the target data cluster is queried according to the target query statement according to the preset rule, the target data obtained by the query is defined through the export rule table corresponding to the target data, and the target data obtained by the query is stored in the export file corresponding to the target query statement, so that the data corresponding to the target query statement is quickly obtained through the export file. Reduce the number of network transmission and the high time consumption caused by network communication, greatly improve the data transmission efficiency.
[0023] Other features and advantages of the embodiments of the present application will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF DRAWINGS
[0024] The accompanying drawings are included to provide a further understanding of the embodiments of the present application, and constitute a part of the specification, and are used together with the following specific embodiments to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the drawings:
[0025] Figure 1 The flowchart of the method of HIVE data export according to the embodiments of the present application is schematically shown;
[0026] Figure 2 The structural block diagram of an electronic device according to the embodiments of the present application is schematically shown;
[0027] Figure 3 The internal structure diagram of a computer device according to the embodiments of the present application is schematically shown. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific embodiments described here are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.
[0029] It should be noted that if the embodiments of the present application involve directionality indication (such as up, down, left, right, front, back, etc.), the directionality indication is only used to explain the relative position relationship, motion condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directionality indication also changes accordingly.
[0030] In addition, if the embodiments of the present application involve descriptions such as "first", "second", etc., the descriptions of "first", "second", etc. are only for description purposes, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of a person skilled in the art, and when the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the protection scope required by the present application.
[0031] Figure 1 The flowchart of the method for HIVE data export according to the embodiments of the present application is schematically shown. As shown in Figure 1 The embodiments of the present application provide a method for HIVE data export, which can include the following steps.
[0032] Step 101, obtaining a target query statement for a target data cluster;
[0033] Step 102, generating a statement identifier corresponding to the target query statement;
[0034] Step 103, generating a target export file name for storing the export file in the storage directory according to the statement identifier, wherein the storage directory refers to the storage directory of the export file created on the distributed file system corresponding to the target data cluster;
[0035] Step 104, parsing the target query statement to determine a data export rule table corresponding to the target query statement, wherein the data export rule table defines the data type name and field name of the exported data;
[0036] Step 105, according to a preset rule, finding out the corresponding target data in the target data cluster according to the target query statement;
[0037] Step 106, defining the target data according to the data export rule table, and storing the target data into the file corresponding to the target export file name in the storage directory.
[0038] The processor can acquire a target query statement for a target data cluster to be queried, the target data cluster can refer to a data cluster stored in HIVE, HIVE is a database that can support storage of massive data, and the data is stored on a distributed file system. And generate a statement identifier corresponding to the target query statement according to the target query statement. For example, the processor can acquire a select statement for a target data cluster to be queried, and generate a "unique identifier ID" as the statement identifier of the query statement by performing md5 calculation on the select statement. The processor can generate a target export file name for storing the export file in the storage directory according to the statement identifier corresponding to the target query statement, wherein the storage directory refers to a storage directory of the export file created on the distributed file system corresponding to the target data cluster. That is, the processor can create a storage directory of the export file on the distributed file system corresponding to the target data cluster, and generate a target export file name for storing the export file in the storage directory according to the statement identifier corresponding to the query statement. The processor can parse the target query statement to determine the export rule table corresponding to the target query statement, wherein the data export rule table can define the data type name and field name of the exported data, that is, the export rule table can define the data type name and field name corresponding to the data queried by the query statement. The processor can perform data query in the target data cluster to be queried according to the preset rule through the target query statement, so as to obtain target data. After the processor obtains the corresponding target data in the target data cluster through the target query statement, the processor can define the target data according to the data export rule table corresponding to the query statement, and store the target data in the file corresponding to the target export file name in the storage directory. The processor can export the file corresponding to the target export file name to obtain the data corresponding to the target query statement.
[0039] In one embodiment, according to the preset rule, the target data corresponding to the target query statement is found in the target data cluster, including: dividing the data in the target data cluster to obtain a predetermined number of data sets; copying the target query statement to obtain a predetermined number of target statements; querying all data sets in the target data cluster simultaneously by all target query statements to obtain sub-target data in each data set; and determining the union of all sub-target data as the target data.
[0040] The processor can search the target data in the target data cluster according to the target query statement according to a preset rule. The processor can divide the data in the target data cluster according to a preset number to obtain a preset number of data sets, wherein the preset number can be set according to the data input by the user. The processor can also copy the target query statement according to the preset number to obtain a preset number of target statements. After obtaining the same preset number of target statements and data sets, the processor can simultaneously query all data sets in the target data cluster through all target query statements, thereby obtaining sub-target data in each data set, that is, all data sets are queried in parallel through all target query statements, thereby obtaining sub-target data corresponding to the target query statement in each data set, and then determining the union of all sub-target data as the target data corresponding to the target query statement in the target data cluster. For example, assuming that the processor sets the preset number to 5 according to the data input by the user, the processor can divide the data in the target data cluster into 5 equal parts to obtain 5 data sets, and copy the target query statement to obtain 5 target query statements. The processor can simultaneously query the 5 data sets through the 5 target query statements, thereby obtaining sub-target data corresponding to the target query statement in each data set, and then determining the union of the 5 sub-target data as the target data corresponding to the target query statement in the target data cluster.
[0041] In one embodiment, the method further comprises: in the case that the data export rule table corresponding to the target query statement cannot be determined by parsing the target query statement, creating a preset condition for the target query statement, so that the target query statement queries an empty data set under the preset condition, wherein the empty data set includes a data type name and a field name; and establishing the data export rule table corresponding to the target query statement according to the data type name and the field name included in the empty data set.
[0042] The processor can determine the data export rule table corresponding to the target query statement by parsing the target query statement, but when the processor cannot obtain the data export rule table corresponding to the target query statement by parsing the target query statement, the processor can create a preset condition for the target query statement, so that the target query statement queries an empty data set under the limitation of the preset condition, wherein the empty data set can include a data type name and a field name, but the corresponding data under each data type name and field name is an empty set. The processor can establish the data export rule table corresponding to the target query statement according to the data type name and the field name included in the empty data set.
[0043] In one embodiment, the method further comprises: after generating the target export file name for storing the export file in the storage directory according to the statement identifier, determining whether the target export file name already exists; and in the case where it is determined that the target export file name already exists, determining the file corresponding to the target export file name in the storage directory as the target export file corresponding to the target query statement.
[0044] After generating the target export file name for storing the export file in the storage directory according to the statement identifier, the processor can determine whether the target export file name already exists, and if the target export file name already exists, the processor can directly determine the file corresponding to the target export file name in the storage directory as the target export file corresponding to the target query statement.
[0045] In one embodiment, the method further comprises: obtaining a data interface address of the target data cluster; and after storing the target data in the file corresponding to the target export file name in the storage directory, exporting the file corresponding to the target export file name through the data interface address.
[0046] The processor can obtain a data interface address of the target data cluster, and after storing the target data corresponding to the target query statement in the file corresponding to the target export file name in the storage directory, the processor can export the file corresponding to the target export file name through the data interface address.
[0047] In one embodiment, the method further comprises: obtaining the update frequency of the data cached in each file in the storage directory; determining the life cycle of each file according to the update frequency; and re-determining the target data corresponding to the file according to the life cycle of the file and replacing the stored data with the re-determined target data.
[0048] The processor can obtain the update frequency of the data cached in each file in the storage directory, determine the life cycle of each file according to the update frequency, re-determine the target data corresponding to the file according to the life cycle of the file, and replace the stored data with the re-determined target data. For example, assuming that the data attribute of the data cached in the file is never-updated data, the data cached in the file is permanently trusted, and if the data cached in the file is not referenceable after a certain period of time, the data may be outdated. The processor can determine the update frequency of the data, determine the life cycle of the file according to the update frequency of the data, the life cycle of the file is short if the data is updated frequently, and the life cycle of the file is long if the data is not updated frequently. The processor can re-determine the target data corresponding to the file according to the life cycle of the file, re-determine the data in the file according to the above embodiment, and replace the originally stored data in the file with the newly confirmed data, thereby updating the data in the file.
[0049] In one embodiment, the method further comprises: updating the storage directory in a case where it is determined that there is an empty file in the storage directory.
[0050] The processor can scan and detect the storage directory at a preset frequency. In a case where it is determined that there is an empty file in the storage directory, the processor can update the storage directory, for example, the processor can delete the empty file from the storage directory.
[0051] In one embodiment, as shown in Figure 2 An electronic device 200 is provided, which includes a processor 201 and a memory 202 connected with the processor 201 in communication; the memory 202 stores computer execution instructions; the processor 201 executes the computer execution instructions stored in the memory 202 to implement the method of HIVE data export according to any one of the above embodiments.
[0052] The processor can obtain a target query statement for a target data cluster stored in HIVE to be queried, and generate a statement identifier corresponding to the target query statement according to the target query statement. The processor can generate a target export file name for storing an export file in the storage directory according to the statement identifier corresponding to the target query statement, wherein the storage directory refers to a storage directory of a universal file created on a distributed file system corresponding to the target data cluster. After generating the target export file name for storing the export file in the storage directory according to the statement identifier, the processor can determine whether the target export file name already exists. If the target export file name already exists, the processor can directly determine the file corresponding to the target export file name in the storage directory as the target export file corresponding to the target query statement. The processor can obtain a data interface address of the target data cluster, and export the file corresponding to the target export file name through the data interface address. The data stored in the file is the target data corresponding to the target query statement in the target data cluster.
[0053] If the target export file name does not exist in the storage directory, the processor can parse the target query statement to determine the export rule table corresponding to the target query statement. The data export rule table can define the data type name and field name of the exported data, that is, the export rule table can define the data type name and field name corresponding to the data queried by the query statement. However, when the processor cannot obtain the data export rule table corresponding to the target query statement by parsing the target query statement, the processor can create a preset condition for the target query statement, so that the target query statement queries an empty data set under the limitation of the preset condition. The empty data set can include data type names and field names, but the corresponding data under each data type name and field name is an empty set. The processor can establish the data export rule table corresponding to the target query statement according to the data type name and field name included in the empty data set. The processor can search for target data in the target data cluster according to the target query statement and a preset rule. The processor can divide the data in the target data cluster according to a preset number to obtain a plurality of data sets, wherein the preset number can be set according to user input. The processor can also copy the target query statement according to the preset number to obtain a plurality of target statements. After obtaining the same preset number of target statements and data sets, the processor can simultaneously query all data sets in the target data cluster through all target query statements to obtain sub-target data in each data set, that is, all data sets are queried in parallel through all target query statements to obtain sub-target data corresponding to the target query statement in each data set. The union of all sub-target data is determined as the target data corresponding to the target query statement in the target data cluster. After the processor queries the corresponding target data in the target data set according to the target query statement, the processor can define the target data according to the data export rule table corresponding to the query statement, and store the target data in the file corresponding to the target export file name in the storage directory. The file corresponding to the target export file name is exported through the data interface address. The data stored in the file is the target data corresponding to the target query statement in the target data cluster.
[0054] The processor can also scan and detect the storage directory at a preset frequency. When it is determined that there is an empty file in the storage directory, the processor can update the storage directory. For example, the processor can delete the empty file from the storage directory. The processor can obtain the update frequency of the cached data in each file in the storage directory, determine the life cycle of each file according to the update frequency, and re-determine the target data corresponding to the file according to the life cycle of the file, and replace the stored data with the re-determined target data. For example, if the data attribute of the cached data in the file is never-updated data, the data cached in the file is permanently trusted. If the cached data in the file is not referenceable after a certain period of time, the data may become outdated. The processor can determine the update frequency of the data, and determine the life cycle of the file according to the update frequency of the data. If the data is updated frequently, the life cycle of the file is short. If the data is not updated frequently, the life cycle of the file is long. The processor can re-determine the target data corresponding to the file according to the life cycle of the file. The processor can re-determine the data in the file according to the above embodiment, and replace the originally stored data in the file with the newly determined data, thereby updating the data in the file.
[0055] Through the above scheme, the export file corresponding to the target query statement is established, the target data cluster is queried in parallel according to the target query statement, the target data obtained by the query is defined according to the export rule table corresponding to the target data, and the target data obtained by the query is stored in the export file corresponding to the target query statement, so that the data corresponding to the target query statement is quickly obtained through the export file. Reduce the number of network transmissions and the high time consumption caused by network communication, greatly improve the data transmission efficiency. For the same query, only one calculation is needed, and subsequent data can be directly downloaded.
[0056] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in FIG. 1. Figure 3As shown. The computer device includes a processor A01, a network interface A02, memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The database stores relevant data within a distributed file system. The network interface A02 communicates with external terminals via a network connection. When executed by the processor A01, the computer program B02 implements a method for exporting HIVE data.
[0057] Figure 1 This is a flowchart illustrating a method for exporting HIVE data in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Furthermore, Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0058] This application provides an apparatus including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: obtaining a target query statement for a target data cluster; generating a statement identifier corresponding to the target query statement; generating a target exported file name in a storage directory for storing exported files based on the statement identifier, wherein the storage directory refers to a directory created on the distributed file system corresponding to the target data cluster; parsing the target query statement to determine a data export rule table corresponding to the target query statement, wherein the data export rule table defines the data type name and field name of the exported data; searching for the corresponding target data in the target data cluster according to preset rules based on the target query statement; defining the target data according to the data export rule table and storing the target data in a file in the storage directory corresponding to the target exported file name.
[0059] In one embodiment, the searching the target data set according to the target query statement to obtain the target data comprises: dividing the data in the target data set into a preset number of data sets; copying the target query statement to obtain a preset number of target statements; simultaneously querying all the data sets in the target data set by using all the target statements to obtain sub-target data in each data set; and determining the union of all the sub-target data as the target data.
[0060] In one embodiment, the method further comprises: in a case where the data export rule table corresponding to the target query statement cannot be determined by parsing the target query statement, creating a preset condition for the target query statement, so that the target query statement queries an empty data set under the preset condition, wherein the empty data set comprises a data type name and a field name; and establishing the data export rule table corresponding to the target query statement according to the data type name and the field name included in the empty data set.
[0061] In one embodiment, the method further comprises: after generating the target export file name for storing the export file in the storage directory according to the statement identifier, determining whether the target export file name already exists; and in a case where it is determined that the target export file name already exists, determining the file corresponding to the target export file name in the storage directory as the target export file corresponding to the target query statement.
[0062] In one embodiment, the method further comprises: obtaining the data interface address of the target data set; and after storing the target data in the file corresponding to the target export file name in the storage directory, exporting the file corresponding to the target export file name through the data interface address.
[0063] In one embodiment, the method further comprises: obtaining the update frequency of the cached data in each file in the storage directory; determining the life cycle of each file according to the update frequency; and re-determining the target data corresponding to the file according to the life cycle of the file, and replacing the stored data with the re-determined target data.
[0064] In one embodiment, the method further comprises: in a case where it is determined that there is an empty file in the storage directory, updating the storage directory.
[0065] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0066] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.
[0067] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.
[0068] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.
[0069] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.
[0070] The memory can include non-persistent memory and / or volatile memory, such as a random access memory (RAM) including a cache area for the temporary storage of data. The memory can also include non-volatile memory, such as read only memory (ROM) for storing structural information and / or instruction code. Both volatile and non-volatile memory can be implemented as a flash memory, a magnetic memory, an optical memory, and / or any non-transitory computer readable storage medium.
[0071] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.
[0072] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or apparatus that includes a list of elements does not only include those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.
[0073] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.
Claims
1. A method for HIVE data export, the method comprising: The method comprises: acquiring a target query statement for a target data cluster; generating a statement identifier corresponding to the target query statement; generating a target export file name for storing an export file in a storage directory according to the statement identifier, wherein the storage directory refers to a storage directory of the export file created on a distributed file system corresponding to the target data cluster; parsing the target query statement to determine a data export rule table corresponding to the target query statement, wherein the data export rule table defines a data type name and a field name of the exported data; according to a preset rule, finding corresponding target data in the target data cluster according to the target query statement; defining the target data according to the data export rule table, and storing the target data into a file corresponding to the target export file name in the storage directory.
2. The method of HIVE data export of claim 1, wherein, According to a preset rule, the target data in the target data cluster is found according to the target query statement, which comprises: evenly dividing the data in the target data cluster to obtain a preset number of data sets; copying the target query statement to obtain a preset number of target statements; querying all data sets in the target data cluster simultaneously by all target query statements to obtain sub-target data in each data set; determining the union of all sub-target data as the target data.
3. The method of HIVE data export of claim 1, wherein, The method further comprises: in the case that the data export rule table corresponding to the target query statement cannot be determined by parsing the target query statement, creating a preset condition for the target query statement, so that the target query statement queries an empty data set under the preset condition, wherein the empty data set includes a data type name and a field name; establishing a data export rule table corresponding to the target query statement according to the data type name and the field name included in the empty data set.
4. The method of HIVE data export of claim 1, wherein, The method further comprises: after generating the target export file name for storing the export file in the storage directory according to the statement identifier, determining whether the target export file name already exists; in the case that it is determined that the target export file name already exists, determining the file corresponding to the target export file name in the storage directory as a target export file corresponding to the target query statement.
5. The method of HIVE data export of claim 1 or 4, wherein, The method further comprises: acquiring a data interface address of the target data cluster; after storing the target data into the file corresponding to the target export file name in the storage directory, exporting the file corresponding to the target export file name through the data interface address.
6. The method of HIVE data export of claim 1, wherein, The method further comprises: acquiring the update frequency of the cached data in each file in the storage directory; determining the life cycle of each file according to the update frequency; re-determining the target data corresponding to the file according to the life cycle of the file, and replacing the stored data with the re-determined target data.
7. The method of HIVE data export of claim 6, wherein, The method further comprises: in the case that it is determined that there is an empty file in the storage directory, updating the storage directory.
8. An electronic device, comprising: It comprises: a processor, and a memory connected to the processor in communication; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method of HIVE data export according to any one of claims 1 to 7.
9. A machine-readable storage medium having stored thereon instructions, the instructions being executable by a machine to cause the machine to: The instructions, when executed by a processor, cause the processor to be configured to perform the method of HIVE data export according to any one of claims 1 to 7.
10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method of HIVE data export according to any one of claims 1 to 7.