Method and device for executing data query task

Abstract

The invention discloses an execution method and device for a data query task, and the method comprises the steps: obtaining a plurality of local resource data corresponding to the data query task, and dividing the plurality of local resource data into a dependency file and a calculation program file when determining that the dependency file exists in the plurality of local resource data; obtaining a file fixed position corresponding to the dependent file, and uploading the calculation program file to a pre-created temporary working directory; and submitting a data query task and a file fixed position to the cluster resource management system, so that the cluster resource management system obtains a dependent file based on the received file fixed position, downloads a calculation program file from the temporary working directory, starts an engine task based on the calculation program file and the dependent file, and sends the engine task to the cluster resource management system. Obtaining task processing data corresponding to the data query task; compared with the prior art, the technical scheme of the invention can avoid repeated uploading and downloading of the dependent file and improve the task execution speed.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and particularly to a method and device for executing a data query task. Background Art

[0002] In the era of big data, as the amount of data continues to increase, the execution efficiency of the cluster resource management system YARN becomes increasingly crucial.

[0003] In the execution process of a traditional Hive query data submission task, it mainly includes: creating a working folder in the HDFS cluster, uploading local resources to the HDFS working folder, starting the task, downloading resources, and executing the task. Among them, in the local resource upload stage and the resource download stage, it is necessary to complete the upload or download of all local resources. Otherwise, when executing the task, the task execution will fail because the set engine cannot be started.

[0004] Since local resources involve multiple files, the data packet volume of local resources is generally large, and it takes a long time to upload or download local resources; during the peak of the big data cluster, due to a large number of local resources involved, it will seriously affect the efficiency of the cluster resource management system YARN to execute tasks, thereby prolonging the running time of Hive queries. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: to provide a method and device for executing a data query task, and improve the task execution speed.

[0006] To solve the above technical problem, the present invention provides a method for executing a data query task, including: obtaining a plurality of local resource data corresponding to a data query task, and when it is determined that there are dependent files among the plurality of local resource data, dividing the plurality of local resource data into dependent files and calculation program files; obtaining the fixed file positions corresponding to the dependent files, and uploading the calculation program files to a pre-created temporary working directory; submitting the data query task and the file fixed positions to the cluster resource management system, so that the cluster resource management system obtains the dependent files based on the received file fixed positions, downloads the calculation program files from the temporary working directory, and starts an engine task based on the calculation program files and the dependent files to obtain task processing data corresponding to the data query task.

[0007] In some embodiments, before the cluster resource management system obtains the dependent files based on the received file fixed positions, it further includes: the cluster resource management system determines whether there are dependent files in the local cache directory of the cluster resource management system based on the received file fixed positions. If so, it calls the dependent files in the local cache directory; otherwise, it obtains the dependent files based on the received file fixed positions.

[0008] In some embodiments, the cluster resource management system determines whether there is a dependent file in the local cache directory of the cluster resource management system based on the received fixed file location, including: the cluster resource management system determines the source file path based on the fixed file location, and determines the source file system based on the source file path; obtains the target file path, and determines the target file system based on the target file path; determines whether the source file system and the target file system are the same. If so, it is determined that there is a dependent file in the local cache directory of the cluster resource management system; otherwise, it continues to determine whether there is a dependent file in the target file path. If so, it is determined that there is a dependent file in the local cache directory of the cluster resource management system.

[0009] In some embodiments, the cluster resource management system obtains a dependent file based on the received fixed file location, including: the cluster resource management system determines whether there is a dependent file in the fixed file location. If so, it performs a file copy operation, copies the dependent file to a pre-created local resource object, generates a dependent file object, and stores the dependent file object in the local cache directory.

[0010] In some embodiments, before determining that there is a dependent file in multiple local resource data, it further includes: respectively obtaining the mode information corresponding to the multiple local resource data, and determining whether there is a distributed file system file in the multiple local resource data based on the mode information. If so, it is determined that there is a dependent file in the multiple local resource data; otherwise, it is determined that there is no dependent file in the multiple local resource data.

[0011] In some embodiments, obtaining multiple local resource data corresponding to a data query task includes: obtaining a first target path under an environment variable configuration, and obtaining multiple first archive files based on the first target path; obtaining a second target path under a configuration file, and obtaining multiple second archive files based on the second target path; using the multiple first archive files and the multiple second archive files as the multiple local resource data corresponding to the data query task.

[0012] The present invention also provides an execution device for a data query task, including: a local resource data acquisition module, a local resource data upload module, and a local resource data download module; wherein, the local resource data acquisition module is configured to acquire a plurality of local resource data corresponding to the data query task, and when it is determined that there are dependent files among the plurality of local resource data, divide the plurality of local resource data into dependent files and calculation program files; the local resource data upload module is configured to acquire the fixed file location corresponding to the dependent files and upload the calculation program files to a pre-created temporary working directory; the local resource data download module is configured to submit the data query task and the file fixed location to the cluster resource management system, so that the cluster resource management system acquires the dependent files based on the received file fixed location, downloads the calculation program files from the temporary working directory, and starts an engine task based on the calculation program files and the dependent files to obtain the task processing data corresponding to the data query task.

[0013] In some embodiments, before the cluster resource management system in the local resource data download module acquires the dependent files based on the received file fixed location, it further includes: the cluster resource management system determines whether there are dependent files in the local cache directory of the cluster resource management system based on the received file fixed location. If so, it calls the dependent files in the local cache directory; otherwise, it acquires the dependent files based on the received file fixed location.

[0014] In some embodiments, the cluster resource management system in the local resource data download module determines whether there are dependent files in the local cache directory of the cluster resource management system based on the received file fixed location, including: the cluster resource management system determines the source file path based on the file fixed location, and determines the source file system based on the source file path; acquires the target file path, and determines the target file system based on the target file path; determines whether the source file system and the target file system are the same. If so, it determines that there are dependent files in the local cache directory of the cluster resource management system; otherwise, it continues to determine whether there are dependent files in the target file path. If so, it determines that there are dependent files in the local cache directory of the cluster resource management system.

[0015] In some embodiments, the cluster resource management system in the local resource data download module acquires the dependent files based on the received file fixed location, including: the cluster resource management system determines whether there are dependent files in the file fixed location based on the received file fixed location. If so, it performs a file copy operation, copies the dependent files to a pre-created local resource object to generate a dependent file object, and stores the dependent file object in the local cache directory.

[0016] In some embodiments, before determining that there are dependent files among multiple local resource data, the local resource data acquisition module further includes: respectively obtaining the mode information corresponding to the multiple local resource data, and based on the mode information, determining whether there are distributed file system files among the multiple local resource data. If so, it is determined that there are dependent files among the multiple local resource data; otherwise, it is determined that there are no dependent files among the multiple local resource data.

[0017] In some embodiments, the local resource data acquisition module is used to obtain multiple local resource data corresponding to a data query task, including: obtaining a first target path under the environment variable configuration, and based on the first target path, obtaining multiple first archive files; obtaining a second target path under the configuration file, and based on the second target path, obtaining multiple second archive files; and using the multiple first archive files and the multiple second archive files as the multiple local resource data corresponding to the data query task.

[0018] The present invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the execution method of the data query task as described in any one of the above is implemented.

[0019] The present invention also provides a computer-readable storage medium. The computer-readable storage medium includes a stored computer program. When the computer program runs, it controls the device where the computer-readable storage medium is located to execute the execution method of the data query task as described in any one of the above.

[0020] Compared with the prior art, the execution method and device for a data query task in an embodiment of the present invention have the following beneficial effects:

[0021] By dividing the multiple local resource data corresponding to the obtained data query task into dependent files and computing program files, for the dependent files, obtaining their corresponding fixed file positions, and for the computing program files, uploading them to a pre-created temporary working directory; different from the prior art where it is necessary to upload both the dependent files and the computing program files simultaneously, avoiding the upload of the dependent files can improve the speed of submitting tasks to Hive; at the same time, after submitting the data query task and the fixed file positions to the cluster resource management system, the cluster resource management system only needs to download the computing program files from the temporary working directory, and for the dependent files, it obtains them based on the received fixed file positions, avoiding the download of the dependent files, which can improve the speed of the cluster resource management system to start tasks, and thus improve the task execution speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a schematic diagram of the execution process of submitting a traditional Hive query data task;

[0023] Figure 2 It is a schematic flowchart of an embodiment of a method for executing a data query task provided by the present invention;

[0024] Figure 3 It is a schematic flowchart of another embodiment of a method for executing a data query task provided by the present invention;

[0025] Figure 4 It is a schematic structural diagram of an embodiment of a device for executing a data query task provided by the present invention;

[0026] Figure 5 It is a schematic structural diagram of a terminal device provided by the present invention. Detailed implementation manners

[0027] Hive is a data warehouse tool running on Hadoop, which performs data queries through a third-party engine, such as Tez or MapReduce (MR). As shown in Figure 1 below, Figure 1 It is a schematic diagram of the execution process of a traditional Hive query data submission task; these queries usually go through multiple stages, and each stage requires the following steps: first, create a temporary working directory on the HDFS cluster to store relevant file data; through the HDFS client (hdfs-Client) of the Hadoop Distributed File System, upload the local resource data required for running the query task, including dependency files and calculation program files, etc., to the temporary working directory in the HDFS cluster; Hive usually performs task scheduling and resource management through the cluster resource management system YARN. Therefore, the YARN client (yarn-Client) that submits and executes Hive jobs on the cluster resource management system YARN submits the query task to the cluster resource management system YARN. Before starting the query task, the cluster resource management system YARN will download the resource files required for task execution, including dependency files and calculation program files, etc., from the HDFS cluster according to the task configuration. After the relevant resources are downloaded, it will start the corresponding engine task, which will process the data according to the query requirements and write the processed results to the specified directory.

[0028] Since Hive queries usually need to execute the above steps multiple times, among which, the steps of uploading dependency files and calculation program files to the temporary working directory in the HDFS cluster and downloading the dependency files and calculation program files required for task execution from the HDFS cluster are repetitive functions. At the same time, since resources such as dependency packages may be very large, this will seriously affect the efficiency of the cluster resource management system YARN in executing tasks, thereby prolonging the running time of Hive queries.

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts belong to the scope of protection of the present invention.

[0030] Embodiment 1, refer to Figure 2 , Figure 2 which is a schematic flowchart of an embodiment of a method for executing a data query task provided by the present invention. As Figure 2 shown, the method includes steps 101 - 103, specifically as follows:

[0031] Step 101: Obtain multiple local resource data corresponding to the data query task. When it is determined that there are dependent files among the multiple local resource data, divide the multiple local resource data into dependent files and calculation program files.

[0032] In one embodiment, Hive is a data warehouse based on Hadoop. It provides the ability to perform SQL queries and can be used for data analysis, data mining, etc. In Hive, by using a third - party engine to accelerate queries, query parallelization can be achieved, thereby improving query efficiency. However, when using a third - party engine for queries, Hive first converts the query statement into some MapReduce jobs or Tez tasks, and these jobs or tasks are submitted to the Hadoop cluster for execution. MapReduce or Tez divides the input data into several data blocks and performs parallel computing on different nodes. Therefore, in order to enable the MapReduce or Tez tasks on each node to proceed smoothly, it is necessary to first obtain multiple local resource data corresponding to the data query task and allocate local resource data such as files, libraries, and cached data required for calculation to the corresponding nodes in advance. These files can be directly accessed by MapReduce or Tez, thereby accelerating the query.

[0033] In one embodiment, since Hive will load the files of the environment variable $HIVE_AUX_JARS_PATH and the property hive.aux.jars.path in the configuration file hive-site.xml during execution as local resource data, if not configured, setting the corresponding engine during task execution will cause the execution to fail. Therefore, in this embodiment, when obtaining multiple local resource data corresponding to a data query task, by obtaining the first target path under the environment variable configuration, based on the first target path, obtaining multiple first archive files; obtaining the second target path under the configuration file, based on the second target path, obtaining multiple second archive files; and using the multiple first archive files and the multiple second archive files as the multiple local resource data corresponding to the data query task.

[0034] Preferably, the environment variable $HIVE_AUX_JARS_PATH is an environment variable used to specify the path of additional JAR files, which points to a file path that contains the class libraries or dependent JAR files required by Hive; the JAR file is a compressed file format used to store and distribute Java programs, which can contain Java classes, resource files, and other related files, and these JAR files are usually used for the extension of functions such as custom functions and user-defined functions (UDFs).

[0035] Preferably, the first target path refers to a file path in the environment variable that points to the class libraries or dependent JAR files required by Hive.

[0036] Preferably, the first archive file refers to the class libraries or dependent JAR files required by Hive pointed to by the first target path.

[0037] Preferably, the configuration file hive-site.xml is the configuration file of Hive, which is used to configure various parameters during the operation of Hive.

[0038] Preferably, the second target path refers to the path of the additional JAR files specified by the hive.aux.jars.path property in the configuration file.

[0039] Preferably, the second archive file refers to the additional JAR files pointed to by the first target path.

[0040] In one embodiment, before determining that there are dependent files among the multiple local resource data, it further includes: respectively obtaining the schema information corresponding to the multiple local resource data, and based on the schema information, determining whether there are distributed file system files among the multiple local resource data. If so, it is determined that there are dependent files among the multiple local resource data; otherwise, it is determined that there are no dependent files among the multiple local resource data.

[0041] Preferably, the mode information value refers to the structure information defined for local resource data in Hive, which can also be called Schema information.

[0042] Preferably, the distributed file system file refers to the Hadoop Distributed File System file, i.e., the HDFS file. Among them, the Hadoop Distributed File System file includes, but is not limited to, Parquet files, Avro files, and ORC files.

[0043] Specifically, when determining whether there is a distributed file system file in multiple local resource data based on the mode information, the storage path in the mode information is obtained in sequence, and it is judged whether the prefix of the storage path is hdfs. If so, it is determined that the local resource data corresponding to the mode information is a distributed file system file, and there is a distributed file system file in the multiple local resource data; otherwise, it is determined that there is no distributed file system file in the multiple local resource data.

[0044] In one embodiment, when dividing multiple local resource data into dependent files and calculation program files, it is respectively judged whether each local resource data is a distributed file system file. If so, the current local resource data is classified as a dependent file; otherwise, the current local resource data is classified as a calculation program file.

[0045] Step 102: Obtain the fixed file location corresponding to the dependent file, and upload the calculation program file to a pre-created temporary working directory.

[0046] In one embodiment, since Hive will automatically read the configuration environment variable $HIVE_AUX_JARS_PATH and the file with the hive.aux.jars.path attribute in the configuration file hive-site.xml during the local resource upload stage and upload it to the temporary working directory of the HDFS cluster; for dependent files, in the prior art, multiple dependent files need to be repeatedly uploaded in each stage, and the file package volume of the dependent files is relatively large, seriously affecting the execution efficiency of YARN and thus affecting the Hive query speed; based on this, in this embodiment, after dividing multiple local resource data into dependent files and calculation program files, only the calculation program files are uploaded to the pre-built temporary working directory; for dependent files, their corresponding fixed file locations are obtained, avoiding the repeated upload of dependent files, improving the subsequent execution efficiency of YARN, and thus improving the Hive query speed.

[0047] Step 103: Submit the data query task and the fixed location of the file to the cluster resource management system, so that the cluster resource management system can obtain the dependent file based on the received fixed location of the file, download the calculation program file from the temporary working directory, and start an engine task based on the calculation program file and the dependent file to obtain the task processing data corresponding to the data query task.

[0048] In one embodiment, since the dependent file is no longer uploaded to the temporary working directory in the hdfs cluster, when the yarn-Client, which is a client that submits and executes hive jobs on the cluster resource management system yarn, submits a data query task to the cluster resource management system yarn, it also submits the fixed file address of the dependent file corresponding to the data query task to the cluster resource management system yarn, so that the cluster resource management system yarn can receive the data query task and the fixed file address of the dependent file corresponding to the data query task at the same time.

[0049] In one embodiment, since one sql statement of hive has n stages, and data query tasks and fixed file locations need to be submitted to the cluster resource management system at each stage. If the cluster resource management system needs to obtain the dependent file based on the received fixed file location each time, it will also cause repeated data acquisition, seriously affecting the execution efficiency of yarn and thus the hive query speed. Therefore, to avoid repeated acquisition of the dependent file, it is also set that before the cluster resource management system obtains the dependent file based on the received fixed file location, the cluster resource management system determines whether there is a dependent file in the local cache directory of the cluster resource management system based on the received fixed file location. If so, the dependent file in the local cache directory is called; otherwise, the dependent file is obtained based on the received fixed file location, so as to avoid repeated acquisition of the dependent file.

[0050] In one embodiment, when the cluster resource management system determines whether there is a dependent file in the local cache directory of the cluster resource management system based on the received fixed file location, the cluster resource management system first determines the source file path based on the fixed file location, and determines the source file system based on the source file path; obtains the target file path, and determines the target file system based on the target file path; determines whether the source file system and the target file system are the same. If so, it is determined that there is a dependent file in the local cache directory of the cluster resource management system; otherwise, it continues to determine whether there is a dependent file in the target file path. If so, it is determined that there is a dependent file in the local cache directory of the cluster resource management system.

[0051] Preferably, the source file path is the fixed file path of the dependent file; the source file system is a file system to which a file belongs determined based on the source file path.

[0052] Preferably, the target file path is the path of the dependent file to be processed or stored, and the target file system is a file system to which a file belongs determined based on the target file path.

[0053] Specifically, when the cluster resource management system performs file transfer or processing, it needs to determine specific read and write operations based on the source file path and the target file path. By judging whether the source file system and the target file system are the same, if the source file system and the target file system are the same, that is, the file is already under the same file system, then it can be determined that the dependent file exists in the local cache directory of the cluster resource management system, and the file cached locally can be directly used for processing. If the source file system and the target file system are different, it is necessary to continue to judge whether the dependent file exists in the target file path; if the dependent file exists in the target file path, then it can be determined that the dependent file exists in the local cache directory of the cluster resource management system. In this case, it may be necessary to copy the target file from its original file system to the local cache directory for subsequent use.

[0054] In one embodiment, when judging whether the dependent file exists in the local cache directory of the cluster resource management system, it is also necessary to further judge whether the cache function of the cluster resource management system is enabled. If the cache function of the cluster resource management system has been enabled and it is judged that the source file system and the target file system are the same, it is determined that the dependent file exists in the local cache directory of the cluster resource management system.

[0055] Preferably, when judging whether the dependent file exists in the local cache directory of the cluster resource management system, check in the local buffer directory ${configuration} / yarn / nm / filecache / . If the corresponding dependent file exists, it is considered that the dependent file exists in the local cache directory of the cluster resource management system.

[0056] In one embodiment, when it is determined that the dependent file does not exist in the local cache directory of the cluster resource management system, when the cluster resource management system obtains the dependent file based on the received fixed file location, it judges whether the dependent file exists in the file fixed location based on the received file fixed location. If so, it performs a file copy operation, copies the dependent file to a pre-created local resource object, generates a dependent file object, and stores the dependent file object in the local cache directory.

[0057] In one embodiment, the cluster resource management system also downloads the computing program file from the temporary working directory of the HDFS cluster. When YARN starts, it directly loads the cached dependency files and the downloaded computing program file into the engine task. By starting the engine task, the task processing data corresponding to the data query task is obtained and written to the corresponding directory.

[0058] As Figure 3 shown, Figure 3 FIG. is a schematic structural diagram of another embodiment of an execution method for a data query task provided by the present invention. First, a temporary working directory is created on the HDFS cluster. Through the hdfs-Client, the client of the Hadoop Distributed File System, when uploading the local resource data required for running the data query task, including dependency files, computing program files, etc., to the temporary working directory in the HDFS cluster, instead of uploading both the dependency files and the computing program files to the temporary working directory as in the prior art, only the computing program files are uploaded to the temporary working directory. For the dependency files, their corresponding fixed file positions are obtained, avoiding the repeated upload of dependency files and improving the speed of submitting tasks by Hive. Hive performs task scheduling and resource management through the cluster resource management system YARN. Therefore, the yarn-Client, the client for submitting and executing Hive jobs on the cluster resource management system YARN, submits the data query task and the fixed file positions of the dependency files corresponding to the data query task to the cluster resource management system YARN. At the same time, in the resource download stage, before starting the query task, the cluster resource management system YARN will, according to the task configuration, only download the computing program files required for task running from the HDFS cluster. For the dependency files, it first checks whether they exist in the local cache directory. If so, it directly calls them; otherwise, it obtains the corresponding dependency files based on the received fixed file positions, reducing the repeated download of dependency files and improving the speed of starting tasks by YARN. After the relevant resources are obtained, the corresponding engine task will be started, and this task will process the data according to the query requirements and write the processed results to the specified directory.

[0059] Embodiment 2, refer to Figure 4 , Figure 4 FIG. is a schematic structural diagram of an embodiment of an execution device for a data query task provided by the present invention. As Figure 4 shown, the device includes a local resource data acquisition module 401, a local resource data upload module 402, and a local resource data download module 403, specifically as follows:

[0060] The local resource data acquisition module 401 is configured to acquire multiple local resource data corresponding to a data query task. When it is determined that there are dependent files among the multiple local resource data, the multiple local resource data are divided into dependent files and calculation program files.

[0061] The local resource data upload module 402 is configured to acquire the fixed file location corresponding to the dependent file and upload the calculation program file to a pre-created temporary working directory.

[0062] The local resource data download module 403 is configured to submit the data query task and the fixed file location to the cluster resource management system, so that the cluster resource management system acquires the dependent file based on the received fixed file location, downloads the calculation program file from the temporary working directory, and starts an engine task based on the calculation program file and the dependent file to obtain the task processing data corresponding to the data query task.

[0063] In one embodiment, before the cluster resource management system in the local resource data download module 403 acquires the dependent file based on the received fixed file location, it further includes: the cluster resource management system determines whether the dependent file exists in the local cache directory of the cluster resource management system based on the received fixed file location. If so, it calls the dependent file in the local cache directory; otherwise, it acquires the dependent file based on the received fixed file location.

[0064] In one embodiment, for the cluster resource management system in the local resource data download module 403 to determine whether the dependent file exists in the local cache directory of the cluster resource management system based on the received fixed file location, it includes: the cluster resource management system determines the source file path based on the fixed file location and determines the source file system based on the source file path; acquires the target file path and determines the target file system based on the target file path; determines whether the source file system and the target file system are the same. If so, it determines that the dependent file exists in the local cache directory of the cluster resource management system; otherwise, it continues to determine whether the dependent file exists in the target file path. If so, it determines that the dependent file exists in the local cache directory of the cluster resource management system.

[0065] In one embodiment, in the local resource data download module 403, the cluster resource management system obtains the dependent file based on the received fixed position of the file, including: the cluster resource management system determines whether the dependent file exists in the received fixed position of the file. If so, it performs a file copying operation, copies the dependent file to a pre-created local resource object, generates a dependent file object, and stores the dependent file object in the local cache directory.

[0066] In one embodiment, the local resource data acquisition module 401 is further configured to, before determining that there is a dependent file in the multiple local resource data, respectively obtain the mode information corresponding to the multiple local resource data, and based on the mode information, determine whether there is a distributed file system file in the multiple local resource data. If so, it is determined that there is a dependent file in the multiple local resource data; otherwise, it is determined that there is no dependent file in the multiple local resource data.

[0067] In one embodiment, the local resource data acquisition module 401 is configured to obtain multiple local resource data corresponding to a data query task, including: obtaining a first target path under an environment variable configuration, and based on the first target path, obtaining multiple first archive files; obtaining a second target path under a configuration file, and based on the second target path, obtaining multiple second archive files; using the multiple first archive files and the multiple second archive files as the multiple local resource data corresponding to the data query task.

[0068] The execution device of the above data query task can implement the execution method of the data query task in the above method embodiment. The optional items in the above method embodiment also apply to this embodiment and will not be elaborated here.

[0069] Figure 5 It is a schematic structural diagram of a terminal device. As Figure 5 shown, the terminal device 5 in this embodiment includes: at least one processor 501 ( Figure 5 only one is shown in the figure), a memory 502, and a computer program 503 stored in the memory 502 and executable on at least one processor 501. When the processor 501 executes the computer program 503, it implements the steps in any of the above method embodiments.

[0070] The terminal device 5 may be a computing device such as a smart phone, a laptop computer, a tablet computer, and a desktop computer. The terminal device may include but is not limited to the processor 501 and the memory 502. Those skilled in the art can understand, Figure 5The above are merely examples of the terminal device 5, which do not constitute a limitation to the terminal device 5. It may include more or fewer components than those shown in the figure, or combine certain components, or have different components. For example, it may also include input / output devices, network access devices, etc.

[0071] The so-called processor 501 may be a central processing unit (CPU), and the processor 501 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0072] In some embodiments, the memory 502 may be an internal storage unit of the terminal device 5, such as the hard disk or memory of the terminal device 5. In some other embodiments, the memory 502 may also be an external storage device of the terminal device 5, such as a plug-in hard disk equipped on the terminal device 5, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 502 may also include both the internal storage unit and the external storage device of the terminal device 5. The memory 502 is used to store an operating system, application programs, a boot loader, data, and other programs, such as program codes of computer programs. The memory 502 may also be used to temporarily store data that has been output or is to be output.

[0073] In addition, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0074] An embodiment of the present application provides a computer program product, and when the computer program product runs on a terminal device, the terminal device is caused to implement the steps in each of the above method embodiments when executed.

[0075] In several embodiments provided in the present application, it can be understood that each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, the program segment, or the part of code includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in an order different from that marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved.

[0076] If a function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a terminal device to execute all or part of the steps of the methods in the various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0077] In summary, a method and device for executing a data query task provided by the present invention obtain multiple local resource data corresponding to the data query task. When it is determined that there are dependent files among the multiple local resource data, the multiple local resource data are divided into dependent files and calculation program files; obtain the fixed file locations corresponding to the dependent files, and upload the calculation program files to a pre-created temporary working directory; submit the data query task and the file fixed locations to the cluster resource management system, so that the cluster resource management system obtains the dependent files based on the received file fixed locations, downloads the calculation program files from the temporary working directory, and starts an engine task based on the calculation program files and the dependent files to obtain task processing data corresponding to the data query task; compared with the prior art, the technical solution of the present invention can avoid repeated uploading and downloading of dependent files and improve the task execution speed.

[0078] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and replacements can be made, and these improvements and replacements should also be regarded as the protection scope of the present invention.

Claims

1. A method for executing a data query task, characterized in that, Including: Obtain multiple local resource data corresponding to a data query task. When it is determined that there are dependent files among the multiple local resource data, divide the multiple local resource data into dependent files and calculation program files; Obtain the fixed file location corresponding to the dependent file, and upload the calculation program file to a pre-created temporary working directory; Submit the data query task and the file fixed location to the cluster resource management system, so that the cluster resource management system, based on the received file fixed location, obtains the dependent file, downloads the calculation program file from the temporary working directory, and starts an engine task based on the calculation program file and the dependent file to obtain the task processing data corresponding to the data query task.

2. The execution method of a data query task according to claim 1, characterized in that, Before the cluster resource management system obtains the dependent file based on the received file fixed location, it further includes: The cluster resource management system, based on the received file fixed location, determines whether the dependent file exists in the local cache directory of the cluster resource management system. If so, call the dependent file in the local cache directory; otherwise, obtain the dependent file based on the received file fixed location.

3. The execution method of a data query task according to claim 2, wherein The cluster resource management system, based on the received file fixed location, determines whether the dependent file exists in the local cache directory of the cluster resource management system, including: The cluster resource management system determines the source file path based on the file fixed location, and determines the source file system based on the source file path; obtains the target file path, and determines the target file system based on the target file path; determines whether the source file system and the target file system are the same. If so, it is determined that the dependent file exists in the local cache directory of the cluster resource management system; otherwise, continue to determine whether the dependent file exists in the target file path. If so, it is determined that the dependent file exists in the local cache directory of the cluster resource management system.

4. The method for executing a data query task according to claim 3, wherein The cluster resource management system, based on the received file fixed location, obtains the dependent file, including: The cluster resource management system, based on the received file fixed location, determines whether the dependent file exists in the file fixed location. If so, perform a file copy operation, copy the dependent file to a pre-created local resource object, generate a dependent file object, and store the dependent file object in the local cache directory.

5. The execution method of a data query task according to claim 1, characterized in that, Before it is determined that there are dependent files among the multiple local resource data, it further includes: Respectively obtain the mode information corresponding to the multiple local resource data, and based on the mode information, determine whether there are distributed file system files among the multiple local resource data. If so, it is determined that there are dependent files among the multiple local resource data; otherwise, it is determined that there are no dependent files among the multiple local resource data.

6. The method for executing a data query task according to claim 1, wherein Obtain multiple local resource data corresponding to a data query task, including: Obtain the first target path under the environment variable configuration, and based on the first target path, obtain multiple first archive files; Obtain the second target path under the configuration file, and based on the second target path, obtain multiple second archive files; Use the multiple first archive files and the multiple second archive files as multiple local resource data corresponding to the data query task.

7. An execution device for a data query task, characterized in that, It includes: A local resource data acquisition module, a local resource data upload module, and a local resource data download module; Among them, the local resource data acquisition module is used to obtain multiple local resource data corresponding to the data query task. When it is determined that there are dependent files among the multiple local resource data, the multiple local resource data are divided into dependent files and calculation program files; The local resource data upload module is used to obtain the fixed file location corresponding to the dependent file and upload the calculation program file to a pre-created temporary working directory; The local resource data download module is used to submit the data query task and the file fixed location to the cluster resource management system, so that the cluster resource management system obtains the dependent file based on the received file fixed location, downloads the calculation program file from the temporary working directory, and starts an engine task based on the calculation program file and the dependent file to obtain the task processing data corresponding to the data query task.

8. The execution device for a data query task according to claim 7, wherein Before the cluster resource management system in the local resource data download module obtains the dependent file based on the received file fixed location, it further includes: The cluster resource management system determines whether the dependent file exists in the local cache directory of the cluster resource management system based on the received file fixed location. If so, it calls the dependent file in the local cache directory; otherwise, it obtains the dependent file based on the received file fixed location.

9. A terminal device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the execution method of the data query task according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program. When the computer program runs, it controls the device where the computer-readable storage medium is located to execute the execution method of the data query task according to any one of claims 1 to 6.

Images