Data query method and device, electronic equipment and storage medium
By generating unique task identifiers for engine tasks in the Hadoop framework and querying intermediate files, the performance problems caused by the large amount of engine tasks are solved, and efficient resource utilization and query performance are achieved.
Patent Information
- Application Number
- CN202411804118.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-09
- Publication Date
- 2025-03-18
AI Technical Summary
In the Hadoop framework, data query requests lead to large amount of calculations of engine tasks and consume a lot of cluster resources, and cannot guarantee data query performance.
Avoid repeated execution of engine tasks by generating a unique task identity for each engine task and querying intermediate files based on the task identity.
It reduces the computational volume of engine tasks, reduces the consumption of cluster resources, and improves data query performance.
Smart Images
Figure CN119646034A_ABST
Abstract
Description
Background Art
[0002] With the continuous development of computer technology, data computing platforms based on the Hadoop (a distributed system infrastructure) framework are becoming more widely used. In the Hadoop framework, data query requests can be made based on Hive (a data warehouse tool).
[0003] In the related art, for data query requests with a large amount of data query, they can be parsed, compiled, and optimized to obtain a series of engine tasks, so as to minimize the reduction of query performance. However, since the actual engine task flow chart may be relatively complex, the associations between engine tasks may also be very complex. Therefore, when the data computing platform reaches a certain scale, a large number of data query requests are run on the platform every day. When the amount of data in the base table is large or the logic of the engine task itself is relatively complex, these data query requests will still result in a large amount of engine task calculation, thereby consuming more cluster resources and failing to guarantee data query performance.
[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0005] The present disclosure provides a data query method, device, electronic device and storage medium, which at least to a certain extent overcome the problem that the computational complexity of engine tasks in related technologies is large, thereby consuming more cluster resources and failing to guarantee data query performance.
[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.
[0007] According to one aspect of the present disclosure, a data query method is provided, comprising:
[0008] Determine each engine task based on the data query request;
[0009] Generate task identifiers corresponding to each engine task, wherein any engine task corresponds to one task identifier;
[0010] Obtain the corresponding intermediate file through each task identifier, wherein any intermediate file corresponds to one task identifier;
[0011] Determine the data query results based on each intermediate file.
[0012] In some exemplary embodiments, determining each engine task according to a data query request includes:
[0013] Determine each engine task and the dependencies between each engine task according to the data query request;
[0014] The step of generating task identifiers corresponding to various engine tasks includes:
[0015] If a first engine task has a dependency relationship with at least one second engine task, a task identifier corresponding to the first engine task is generated according to the first engine task and at least one second engine task, wherein the first engine task and the second engine task both belong to the engine tasks.
[0016] In some exemplary embodiments, generating a task identifier corresponding to the first engine task according to the first engine task and at least one second engine task includes:
[0017] Determine task information of the first engine task, wherein the task information of any engine task includes at least one of source table information and processing method information;
[0018] Serializing the task information of the first engine task to obtain an initial identifier corresponding to the first engine task;
[0019] A task identifier corresponding to the first engine task is generated according to the initial identifier corresponding to the first engine task and the task identifier corresponding to at least one second engine task.
[0020] In some exemplary embodiments, the data query method provided by the embodiments of the present disclosure further includes:
[0021] If the first engine task has no dependency relationship with any engine task, determining task information of the first engine task;
[0022] Serializing the task information of the first engine task to obtain an initial identifier corresponding to the first engine task;
[0023] The initial identifier corresponding to the first engine task is used as the task identifier.
[0024] In some exemplary embodiments, serializing the task information of the first engine task to obtain an initial identifier corresponding to the first engine task includes:
[0025] The task information of the first engine task is sorted and serialized based on a preset information sequence to obtain an initial identifier corresponding to the first engine task.
[0026] In some exemplary embodiments, generating a task identifier corresponding to the first engine task according to an initial identifier corresponding to the first engine task and a task identifier corresponding to at least one second engine task includes:
[0027] Sort the task identifiers corresponding to each second engine task based on a preset identifier order;
[0028] The initial identifier corresponding to the first engine task is added to the end of the sequence to obtain the task identifier corresponding to the first engine task.
[0029] In some exemplary embodiments, if the task execution order between the second engine tasks is serial execution, the preset identification order is the task execution order;
[0030] If the task execution order between the second engine tasks is parallel execution, the preset identification order is based on the task identification of each second engine task.
[0031] In some exemplary embodiments, obtaining corresponding intermediate files through each task identifier includes:
[0032] Determine whether the intermediate files corresponding to each task identifier are stored in the cache;
[0033] If an intermediate file corresponding to a first task identifier is stored in the cache, obtaining the intermediate file corresponding to the first task identifier from the cache, wherein the first task identifier is any task identifier;
[0034] If the intermediate file corresponding to the first task identifier is not stored in the cache, the engine task corresponding to the first task identifier is executed to obtain the intermediate file corresponding to the first task identifier.
[0035] In some exemplary embodiments, after executing the engine task corresponding to the first task identifier to obtain the intermediate file corresponding to the first task identifier, the method further includes:
[0036] Determining whether the cluster resources consumed by executing the engine task corresponding to the first task identifier meet a preset condition;
[0037] If the preset condition is met, the intermediate file corresponding to the first task identifier is cached.
[0038] According to another aspect of the present disclosure, there is also provided a data query device, comprising:
[0039] The engine task determination module is used to determine each engine task according to the data query request;
[0040] A task identifier generation module, used to generate task identifiers corresponding to each engine task, wherein each engine task corresponds to one task identifier;
[0041] An intermediate file acquisition module, used to acquire corresponding intermediate files through each task identifier, wherein any intermediate file corresponds to one task identifier;
[0042] The data query result determination module is used to determine the data query result according to each intermediate file.
[0043] In some exemplary embodiments, the engine task determination module is used to determine the various engine tasks and the dependencies between the various engine tasks according to the data query request;
[0044] A task identifier generation module is used to generate a task identifier corresponding to the first engine task based on the first engine task and at least one second engine task if the first engine task has a dependency relationship with at least one second engine task, wherein the first engine task and the second engine task both belong to the engine task.
[0045] In some exemplary embodiments, a task identifier generation module is used to determine the task information of the first engine task, wherein the task information of any engine task includes at least one of source table information and processing method information; serialize the task information of the first engine task to obtain an initial identifier corresponding to the first engine task; and generate a task identifier corresponding to the first engine task based on the initial identifier corresponding to the first engine task and the task identifier corresponding to at least one second engine task.
[0046] In some exemplary embodiments, the task identifier generation module is also used to determine the task information of the first engine task if the first engine task has no dependency relationship with any engine task; serialize the task information of the first engine task to obtain an initial identifier corresponding to the first engine task; and use the initial identifier corresponding to the first engine task as the task identifier.
[0047] In some exemplary embodiments, the task identifier generation module is used to sort and serialize the task information of the first engine task based on a preset information sequence to obtain an initial identifier corresponding to the first engine task.
[0048] In some exemplary embodiments, the task identifier generation module is used to sort the task identifiers corresponding to each second engine task based on a preset identifier sequence; and add the initial identifier corresponding to the first engine task to the end of the sequence to obtain the task identifier corresponding to the first engine task.
[0049] In some exemplary embodiments, if the task execution order between each second engine task is serial execution, the preset identification order is the task execution order; if the task execution order between each second engine task is parallel execution, the preset identification order is sorted based on the task identification of each second engine task.
[0050] In some exemplary embodiments, the intermediate file acquisition module is used to determine whether the intermediate files corresponding to each task identifier are stored in the cache; if the intermediate file corresponding to the first task identifier is stored in the cache, the intermediate file corresponding to the first task identifier is acquired from the cache, wherein the first task identifier is any task identifier; if the intermediate file corresponding to the first task identifier is not stored in the cache, the engine task corresponding to the first task identifier is executed to obtain the intermediate file corresponding to the first task identifier.
[0051] In some exemplary embodiments, the data query device provided by the embodiments of the present disclosure further includes:
[0052] The intermediate file caching module is used to determine whether the cluster resources consumed by executing the engine task corresponding to the first task identifier meet the preset conditions; if the preset conditions are met, the intermediate file corresponding to the first task identifier is cached.
[0053] According to another aspect of the present disclosure, an electronic device is also provided, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned data query methods by executing the executable instructions.
[0054] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data query method described in any one of the above is implemented.
[0055] According to another aspect of the present disclosure, a computer program product is provided, including: a computer program or instructions, wherein when the computer program or instructions are executed by a processor, any one of the above data query methods is implemented.
[0056] The technical solution provided in the embodiment of the present disclosure can generate a unique task identifier for each engine task and query the intermediate file based on the task identifier, so as to avoid executing the engine task again when the intermediate file has been stored. Therefore, the present disclosure can reduce the computing amount of the engine task, reduce the consumed cluster resources, and ensure data query performance.
[0057] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0059] Figure 1 A schematic diagram showing a system structure in an embodiment of the present disclosure;
[0060] Figure 2 A flow chart of a data query method in an embodiment of the present disclosure is shown;
[0061] Figure 3 A schematic diagram of a HiveQL execution process in an embodiment of the present disclosure is shown;
[0062] Figure 4 A schematic diagram showing an execution order of an engine task in an embodiment of the present disclosure;
[0063] Figure 5 A schematic diagram showing an initial identification of a generation engine task in an embodiment of the present disclosure;
[0064] Figure 6 A schematic diagram of a process for optimizing a data query request in an embodiment of the present disclosure is shown;
[0065] Figure 7 A schematic diagram of a data query device in an embodiment of the present disclosure is shown;
[0066] Figure 8 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown;
[0067] Fig. 9 A schematic diagram of a computer-readable storage medium in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0068] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0069] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.
[0070] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.
[0071] Figure 1 FIG. 1 shows an exemplary application system architecture diagram to which the data query method in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .
[0072] The terminal device 101 may send a data query request to the server 103. Then, the server 103 may determine each engine task according to the data query request.
[0073] Then, the server 103 may generate task identifiers corresponding to each engine task, wherein any engine task corresponds to one task identifier, and obtain corresponding intermediate files through each task identifier, wherein any intermediate file corresponds to one task identifier.
[0074] Finally, the server 103 may determine the data query result according to each intermediate file, and the server 103 may return the data query result to the terminal device 101 .
[0075] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and may be a wired network or a wireless network.
[0076] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.
[0077] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, wearable devices, augmented reality devices, virtual reality devices, etc.
[0078] The server 103 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 101. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.
[0079] Optionally, server 103 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.
[0080] Those skilled in the art will know that Figure 1The number of terminal devices 101, networks 102 and servers 103 is only illustrative, and any number of terminal devices 101, networks 102 and servers 103 may be provided according to actual needs. The embodiments of the present disclosure are not limited to this.
[0081] Under the above system architecture, a data query method is provided in an embodiment of the present disclosure, and the method can be executed by any electronic device with computing and processing capabilities.
[0082] In some embodiments, the data query method provided in the embodiments of the present disclosure can be executed by the server of the above-mentioned system architecture; in other embodiments, the data query method provided in the embodiments of the present disclosure can be implemented by the terminal device and the server in the above-mentioned system architecture through interaction.
[0083] Figure 2 A flow chart of a data query method in an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the data query method provided in the embodiment of the present disclosure includes the following steps S202 to S208.
[0084] S202: Determine each engine task according to the data query request.
[0085] In some possible implementations, the data query request can be input into a compiler for parsing and compilation. Then, the compilation result of the data query request can be optimized to obtain multiple engine tasks. Exemplarily, the data query request can be a HiveQL (Hive Query Language, a query language) query statement written by a user, and the input compiler can be a Hive compiler. In addition, each engine task obtained can include a MapReduce (map-reduce) task. Alternatively, the engine task can also include tasks such as Tez (a computing framework) and Spark (another computing framework).
[0086] The disclosed embodiment does not limit the method of obtaining the engine task according to the data query request. Exemplarily, the process of the compiler parsing the data query request may include lexical analysis, syntax analysis, etc., so as to convert the data query request into an abstract syntax tree. Then, the abstract syntax tree can be converted into a corresponding logical execution plan through compilation, and the logical execution plan can also be semantically analyzed and verified. Finally, the logical execution plan can be optimized and divided into corresponding physical plans, i.e., engine tasks, according to the corresponding execution engine.
[0087] In an exemplary embodiment, among the various engine tasks determined according to the data query request, different engine tasks may be executed based on the same or different execution engines, which is not limited in this embodiment of the present disclosure.
[0088] In some exemplary embodiments, determining each engine task according to a data query request includes: determining each engine task and dependencies between each engine task according to the data query request.
[0089] In an exemplary embodiment, each engine task may form a DAG (Directed Acyclic Graph). The execution order between each engine task may be serial or parallel. Exemplarily, if the execution of a first engine task depends on an intermediate file obtained by running a second engine task, then the first engine task has a dependency relationship with the second engine task, and the second engine task is a dependent task of the first engine task.
[0090] For example, a HiveQL execution process diagram provided by an embodiment of the present disclosure can be as follows: Figure 3 shown.
[0091] In this Figure 3 In the HiveQL input, it can be parsed, compiled, and optimized in the Hive compiler to generate a directed acyclic graph consisting of MapReduce tasks. Then the directed acyclic graph can be executed in the Hadoop execution engine.
[0092] like Figure 3 As shown, in the directed acyclic graph, engine task 1, engine task 2, and engine task 3 can be executed in parallel, that is, the execution order is the same. Running engine task 1 can obtain intermediate file 1, running engine task 2 can obtain intermediate file 2, and running engine task 3 can obtain intermediate file 3. Then, engine task 4 can be run according to intermediate file 1 and intermediate file 2 to obtain intermediate file 4, that is, engine task 4 has a dependency relationship with engine task 1 and engine task 2. Finally, intermediate file 3 and intermediate file 4 are merged and the result can be returned.
[0093] It should be noted that in the related technology, after obtaining the DAG, the DAG execution process can be executed in the Hadoop execution engine to obtain the data query result. However, for data query requests with a large amount of data query, the engine task will have a large amount of calculation, thereby consuming more cluster resources and failing to guarantee data query performance.
[0094] S204: Generate a task identifier corresponding to each engine task, wherein any engine task corresponds to one task identifier.
[0095] In an exemplary embodiment, the task identifier corresponding to the engine task can be generated according to all necessary information of the engine task to generate the intermediate file, so any engine task can correspond to a unique task identifier.
[0096] In a possible implementation, all necessary information for an engine task to generate an intermediate file may include at least one of an execution path, query and aggregation logic, execution parameters, and calculation methods. Therefore, the execution path, query and aggregation logic, execution parameters, and calculation methods of any engine task may be serialized to generate a task identifier corresponding to the engine task. Among them, the execution path of an engine task may include each dependent task of the engine task. Therefore, if the task identifier corresponding to any engine task is deserialized, the engine task can be obtained.
[0097] In some exemplary embodiments, a task identifier corresponding to each engine task is generated, including: if a first engine task has a dependency relationship with at least one second engine task, then a task identifier corresponding to the first engine task is generated based on the first engine task and at least one second engine task, wherein the first engine task and the second engine task both belong to engine tasks.
[0098] It should be noted that, since each second engine task is a dependent task of the first engine task, the second engine task is included in the execution path of the first engine task. Therefore, the task identifier corresponding to the first engine task can be generated through the first engine task and each second engine task.
[0099] In some exemplary embodiments, a task identifier corresponding to the first engine task is generated based on the first engine task and at least one second engine task, including: determining the task information of the first engine task, the task information of any engine task including at least one of source table information and processing method information; serializing the task information of the first engine task to obtain an initial identifier corresponding to the first engine task; and generating a task identifier corresponding to the first engine task based on the initial identifier corresponding to the first engine task and the task identifier corresponding to at least one second engine task.
[0100] In an exemplary embodiment, the source table information included in the task information of any engine task can be used to describe the source table required for the operation of the engine task. The processing method information included in the task information can include, for example, scanning tables, shuffles, sorting, conditional screening, joins, query and aggregation logic, execution parameters, custom functions, etc. Among them, custom functions can be used to define calculation methods.
[0101] It should be noted that the task information of any engine task is used to describe the information corresponding to the engine task itself, and may not include the task information corresponding to the dependent tasks of the engine task.
[0102] In an exemplary embodiment, the task information of the first engine task may be serialized to obtain an initial identifier corresponding to the first engine task, and then a task identifier corresponding to the first engine task may be generated according to the initial identifier corresponding to the first engine task and the task identifiers corresponding to each second engine task.
[0103] In a possible implementation, after determining each engine task according to a data query request, the initial identifier corresponding to each engine task can be determined first, and then the task identifier corresponding to each engine task can be determined in sequence according to the execution order of each engine task. Taking the first engine task as an example, since the second engine task is a dependent task of the first engine task, the execution order of the second engine task is before the first engine task. Therefore, when it is necessary to determine the task identifier corresponding to the first engine task, the initial identifier corresponding to the first engine task and the task identifiers corresponding to each second engine task can be directly obtained to generate the task identifier corresponding to the first engine task.
[0104] In another possible implementation, after determining each engine task according to the data query request, the initial identifier and task identifier corresponding to each engine task can be determined in sequence according to the execution order of each engine task. Taking the first engine task as an example, when it is necessary to determine the task identifier corresponding to the first engine task, the initial identifier corresponding to the first engine task can be determined first. Then, since the execution order of the second engine task is before the first engine task, the task identifiers corresponding to each second engine task can be directly obtained at this time to generate the task identifier corresponding to the first engine task.
[0105] In some exemplary embodiments, the task information of the first engine task is serialized to obtain an initial identifier corresponding to the first engine task, including: sorting and serializing the task information of the first engine task based on a preset information order to obtain an initial identifier corresponding to the first engine task.
[0106] The embodiments of the present disclosure do not limit the order of preset information, and the order of preset information can be set based on experience or application scenarios. For example, the order of preset information can be source table information, scan table, shuffle, sort, conditional screening, join, query and aggregation logic, execution parameters, and custom functions.
[0107] For example, if the source table information in the task information contains multiple source tables, each source table can be sorted according to the associated order. If there is no associated order, the source tables can be sorted according to the table name. In addition, if the task information contains multiple aggregation logics, the order of the aggregated fields can be used. And the returned fields can be sorted according to the field name.
[0108] It should be noted that the embodiment of the present disclosure sets a preset information order for the task information, which can ensure that the initial identifiers corresponding to different engine tasks will not be repeated, and can ensure that for any engine task, the initial identifiers generated multiple times are the same.
[0109] In some exemplary embodiments, a task identifier corresponding to the first engine task is generated based on an initial identifier corresponding to the first engine task and a task identifier corresponding to at least one second engine task, including: sorting the task identifiers corresponding to each second engine task based on a preset identifier order; adding the initial identifier corresponding to the first engine task to the end of the sequence to obtain the task identifier corresponding to the first engine task.
[0110] The embodiments of the present disclosure do not limit the order of preset identifiers. For example, the initial identifier may be composed of letters and / or numbers. Therefore, the order of preset identifiers may be sorted from small to large numbers, or may be sorted by first letter.
[0111] In some exemplary embodiments, if the task execution order between each second engine task is serial execution, the preset identification order is the task execution order; if the task execution order between each second engine task is parallel execution, the preset identification order is sorted based on the task identification of each second engine task.
[0112] For example, a schematic diagram of the execution order of an engine task may be as follows: Figure 4 shown.
[0113] In this Figure 4 In the example, engine task 1, engine task 2, and engine task 3 are executed in parallel, engine task 1→engine task 4→engine task 6 are executed in series, engine task 2 & engine task 3→engine task 5 are executed in series, and engine task 4 & engine task 5→engine task 6 are executed in series.
[0114] by Figure 4 Take engine task 4 in as an example, the dependent task of engine task 4 is engine task 1, so the task identifier of engine task 4 can be obtained by arranging in the order of the initial identifier of engine task 1 and the initial identifier of engine task 4. Take engine task 5 as an example, the dependent tasks of engine task 5 are engine task 2 and engine task 3, and engine task 2 and engine task 3 are executed in parallel. Therefore, the initial identifiers of engine task 2 and engine task 3 can be sorted according to the preset identifier order, and then the initial identifier of engine task 5 can be added at the end to obtain the task identifier of engine task 5.
[0115] In some exemplary embodiments, the data query method provided by the embodiments of the present disclosure may also include: if the first engine task has no dependency relationship with any engine task, determining the task information of the first engine task; serializing the task information of the first engine task to obtain an initial identifier corresponding to the first engine task; and using the initial identifier corresponding to the first engine task as the task identifier.
[0116] For example, Figure 4 Taking the engine task 1 in as an example, the engine task 1 has no dependency relationship with other engine tasks, so the initial identifier of the engine task 1 can be used as the task identifier.
[0117] S206, obtaining corresponding intermediate files through each task identifier, wherein any intermediate file corresponds to one task identifier.
[0118] In some exemplary embodiments, obtaining the corresponding intermediate file through each task identifier includes: determining whether the intermediate file corresponding to each task identifier is stored in the cache; if the intermediate file corresponding to the first task identifier is stored in the cache, obtaining the intermediate file corresponding to the first task identifier from the cache, wherein the first task identifier is any task identifier; if the intermediate file corresponding to the first task identifier is not stored in the cache, executing the engine task corresponding to the first task identifier to obtain the intermediate file corresponding to the first task identifier.
[0119] In an exemplary embodiment, the cache may include multiple intermediate files, each of which corresponds to a task identifier. For example, the intermediate file may be used as a value in the cache, and the corresponding task identifier may be used as a key in the cache.
[0120] In some possible implementations, it is possible to query in the cache whether the intermediate file cache corresponding to the first task identifier exists. If it exists, the engine task corresponding to the first task identifier is no longer executed, and the intermediate file in the cache is directly obtained. If it does not exist, the engine task corresponding to the first task identifier is executed to obtain the corresponding intermediate file.
[0121] It should be noted that, since different data query requests are split into engine tasks, the engine tasks may be repeated. For example, the input data, processing logic, and conditional parameters of the engine tasks may be exactly the same, that is, the same intermediate files may be generated after the engine tasks are executed. Therefore, the embodiment of the present disclosure can query the intermediate files according to the task identifier, thereby realizing the sharing of the intermediate files, avoiding the repeated operation of the engine tasks with the same logic, and avoiding the waste of resources.
[0122] In some exemplary embodiments, after executing the engine task corresponding to the first task identifier to obtain the intermediate file corresponding to the first task identifier, the data query method provided by the embodiment of the present disclosure may also include: determining whether the cluster resources consumed by executing the engine task corresponding to the first task identifier meet preset conditions; if the preset conditions are met, caching the intermediate file corresponding to the first task identifier.
[0123] In an exemplary embodiment, the preset condition may be that the cluster resources consumed by the engine task exceed a specified threshold. If the specified threshold is exceeded, it is determined that the preset condition is met, and the corresponding intermediate file is cached. In addition, the threshold may be determined based on experience or application scenarios, and the embodiments of the present disclosure do not limit this.
[0124] S208: Determine the data query result according to each intermediate file.
[0125] In an exemplary embodiment, Figure 4 For example, the intermediate file corresponding to engine task 6 can be used as the data query result. Alternatively, if there are multiple engine tasks executed in parallel at the end of the DAG composed of each engine task, the intermediate files corresponding to each engine task executed in parallel can be merged to obtain the data query result.
[0126] The method provided by the embodiment of the present disclosure can generate a unique task identifier for each engine task and query the intermediate file based on the task identifier, so as to avoid executing the engine task again when the intermediate file has been stored. Therefore, the present disclosure can reduce the computing amount of the engine task, reduce the consumed cluster resources, and ensure data query performance.
[0127] For example, a schematic diagram of an initial identification of a generation engine task provided by an embodiment of the present disclosure may be as follows: Figure 5 shown.
[0128] In this Figure 5 In the example, the initial identifier of engine task 1 may be 201b16b879b7, the initial identifier of engine task 2 may be 941d18cf186f, the initial identifier of engine task 3 may be 6a32840d50e0, the initial identifier of engine task 4 may be 1618dd1f0d36, the initial identifier of engine task 5 may be e93f3a3e915d, and the initial identifier of engine task 6 may be 67870cfc0070.
[0129] In an exemplary embodiment, after determining the initial identification of each engine task, the task identification of the corresponding engine task can be determined based on the initial identification of the engine task. For example, engine task 1, engine task 2, and engine task 3 have no dependent tasks, so the initial identification can be directly used as the task identification. The dependent task of engine task 4 is engine task 1, so the task identification can be determined based on the task execution order. Figure 5 In the task execution sequence shown, the task identifier of engine task 4 may be 201b16b879b7-1618dd1f0d36.
[0130] In addition, the dependent tasks of engine task 5 are engine task 2 and engine task 3. Since engine task 2 and engine task 3 are executed in parallel, engine task 2 and engine task 3 can be sorted according to the initial identifier and then the initial identifier corresponding to engine task 5 is added to the end of the sequence, and the final order obtained is: engine task 3, engine task 2, engine task 5. Therefore, the task identifier of engine task 5 is 6a32840d50e0-941d18cf186f-e93f3a3e915d. The dependent tasks of engine task 6 are engine task 4 and engine task 5, and engine task 4 and engine task 5 are sorted according to the preset identifier sequence. Exemplarily, the order can be engine task 4 first and engine task 5 last, and then the initial identifier of engine task 6 is added to the end of the sequence, and the final order obtained is engine task 1, engine task 4, engine task 3, engine task 2, engine task 5, engine task 6. Therefore, the task ID of engine task 6 is 201b16b879b7-1618dd1f0d36-6a32840d50e0-941d18cf186f-e93f3a3e915d-67870cfc0070.
[0131] For example, a schematic diagram of a process for optimizing a data query request provided by an embodiment of the present disclosure may be as follows: Figure 6 shown.
[0132] In this Figure 6 In the example, after HiveQL input is parsed, compiled, and optimized, engine tasks are generated. For example, the generated engine tasks may include engine task 1, engine task 2, and engine task 3.
[0133] Taking engine task 3 as an example, when engine task 3 starts, a task identifier of engine task 3 can be generated. Then, you can check whether the task identifier exists in the cache. If it exists, you can skip running engine task 3 and get the corresponding intermediate file from the cache. If it does not exist, run engine task 3 and determine whether the task size of engine task 3 exceeds the specified threshold. If it does, the task ends. If not, the intermediate file can be cached. After engine task 1, engine task 2, and engine task 3 are all completed, HiveQL ends.
[0134] It should be noted that the acquisition, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations, and various types of data such as personal identity data, operation data, behavioral data, etc. related to individuals, customers, and groups obtained in the embodiments of this disclosure have all been authorized.
[0135] Based on the same inventive concept, the present disclosure also provides a data query device in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.
[0136] Figure 7 A schematic diagram of a data query device in an embodiment of the present disclosure is shown. Figure 7 As shown, the device comprises:
[0137] The engine task determination module 701 is used to determine each engine task according to the data query request;
[0138] The task identifier generating module 702 is used to generate task identifiers corresponding to each engine task, wherein each engine task corresponds to one task identifier;
[0139] The intermediate file acquisition module 703 is used to acquire the corresponding intermediate file through each task identifier, wherein any intermediate file corresponds to one task identifier;
[0140] The data query result determination module 704 is used to determine the data query result according to each intermediate file.
[0141] In some exemplary embodiments, the engine task determination module 701 is used to determine each engine task and the dependency relationship between each engine task according to the data query request;
[0142] The task identifier generation module 702 is used to generate a task identifier corresponding to the first engine task based on the first engine task and at least one second engine task if the first engine task has a dependency relationship with at least one second engine task, wherein the first engine task and the second engine task both belong to the engine task.
[0143] In some exemplary embodiments, the task identifier generation module 702 is used to determine the task information of the first engine task, and the task information of any engine task includes at least one of source table information and processing method information; serialize the task information of the first engine task to obtain the initial identifier corresponding to the first engine task; and generate the task identifier corresponding to the first engine task based on the initial identifier corresponding to the first engine task and the task identifier corresponding to at least one second engine task.
[0144] In some exemplary embodiments, the task identifier generation module 702 is also used to determine the task information of the first engine task if the first engine task has no dependency relationship with any engine task; serialize the task information of the first engine task to obtain an initial identifier corresponding to the first engine task; and use the initial identifier corresponding to the first engine task as the task identifier.
[0145] In some exemplary embodiments, the task identifier generating module 702 is used to sort and serialize the task information of the first engine task based on a preset information sequence to obtain an initial identifier corresponding to the first engine task.
[0146] In some exemplary embodiments, the task identifier generation module 702 is used to sort the task identifiers corresponding to each second engine task based on a preset identifier sequence; and add the initial identifier corresponding to the first engine task to the end of the sequence to obtain the task identifier corresponding to the first engine task.
[0147] In some exemplary embodiments, if the task execution order between each second engine task is serial execution, the preset identification order is the task execution order; if the task execution order between each second engine task is parallel execution, the preset identification order is sorted based on the task identification of each second engine task.
[0148] In some exemplary embodiments, the intermediate file acquisition module 703 is used to determine whether the intermediate files corresponding to each task identifier are stored in the cache; if the intermediate file corresponding to the first task identifier is stored in the cache, the intermediate file corresponding to the first task identifier is acquired from the cache, wherein the first task identifier is any task identifier; if the intermediate file corresponding to the first task identifier is not stored in the cache, the engine task corresponding to the first task identifier is executed to obtain the intermediate file corresponding to the first task identifier.
[0149] In some exemplary embodiments, the data query device provided by the embodiments of the present disclosure further includes:
[0150] The intermediate file caching module is used to determine whether the cluster resources consumed by executing the engine task corresponding to the first task identifier meet the preset conditions; if the preset conditions are met, the intermediate file corresponding to the first task identifier is cached.
[0151] The device provided by the embodiment of the present disclosure can generate a unique task identifier for each engine task and query the intermediate file based on the task identifier, so as to avoid executing the engine task again when the intermediate file has been stored. Therefore, the present disclosure can reduce the computing amount of the engine task, reduce the consumed cluster resources, and ensure data query performance.
[0152] It should be noted that the examples and application scenarios implemented by the modules in the above-mentioned device embodiment are the same as those of the corresponding steps in the method embodiment, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules as part of the device can be executed in a computer system such as a set of computer executable instructions.
[0153] Those skilled in the art will appreciate that various aspects of the present disclosure may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."
[0154] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present disclosure, the electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above data query methods by executing the executable instructions. Since the principle of solving the problem in the electronic device embodiment is similar to that in the above method embodiment, the implementation of the electronic device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.
[0155] Refer to the following Figure 8 The electronic device 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0156] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810).
[0157] The storage unit stores a program code, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps of various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 810 can perform the following steps of the above method embodiment:
[0158] According to the data query request, each engine task is determined; a task identifier corresponding to each engine task is generated, wherein any engine task corresponds to a task identifier; the corresponding intermediate file is obtained through each task identifier, wherein any intermediate file corresponds to a task identifier; according to each intermediate file, the data query result is determined.
[0159] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .
[0160] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0161] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0162] The electronic device 800 may also communicate with one or more external devices 840 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0163] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0164] Based on the same inventive concept, the embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, any of the above data query methods is implemented. Since the principle of solving the problem in the embodiment of the computer-readable storage medium is similar to that in the above method embodiment, the implementation of the embodiment of the computer-readable storage medium can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.
[0165] Fig. 9 A schematic diagram of a computer-readable storage medium in an embodiment of the present disclosure is shown. Fig. 9 As shown, a program product capable of implementing the above method of the present disclosure is stored on the computer-readable storage medium 900. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.
[0166] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0167] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0168] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.
[0169] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).
[0170] Based on the same inventive concept, a computer program product is also provided in the embodiments of the present disclosure, including a computer program product, including: a computer program or an instruction, wherein when the computer program or the instruction is executed by a processor, the data query method of any one of the above method embodiments is implemented. Since the principle of solving the problem in the computer program product embodiment is similar to that in the above method embodiment, the implementation of the computer program product embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.
[0171] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0172] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.
[0173] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0174] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope of the present disclosure is indicated by the appended claims.
Claims
1. A data query method, characterized in that: include: Determine each engine task based on the data query request; Generate task identifiers corresponding to each engine task, wherein any engine task corresponds to one task identifier; Obtain the corresponding intermediate file through each task identifier, wherein any intermediate file corresponds to one task identifier; Determine the data query results based on each intermediate file.
2. The data query method according to claim 1, characterized in that: Determining each engine task according to the data query request includes: Determine each engine task and the dependencies between each engine task according to the data query request; The step of generating task identifiers corresponding to various engine tasks includes: If a first engine task has a dependency relationship with at least one second engine task, a task identifier corresponding to the first engine task is generated according to the first engine task and at least one second engine task, wherein the first engine task and the second engine task both belong to the engine tasks.
3. The data query method according to claim 2, characterized in that: The step of generating a task identifier corresponding to the first engine task according to the first engine task and at least one second engine task includes: Determine task information of the first engine task, wherein the task information of any engine task includes at least one of source table information and processing method information; Serializing the task information of the first engine task to obtain an initial identifier corresponding to the first engine task; A task identifier corresponding to the first engine task is generated according to the initial identifier corresponding to the first engine task and the task identifier corresponding to at least one second engine task.
4. The data query method according to claim 2, characterized in that: The method further comprises: If the first engine task has no dependency relationship with any engine task, determining task information of the first engine task; Serializing the task information of the first engine task to obtain an initial identifier corresponding to the first engine task; The initial identifier corresponding to the first engine task is used as the task identifier.
5. The data query method according to claim 3 or 4, characterized in that: The serializing the task information of the first engine task to obtain an initial identifier corresponding to the first engine task includes: The task information of the first engine task is sorted and serialized based on a preset information sequence to obtain an initial identifier corresponding to the first engine task.
6. The data query method according to claim 3, characterized in that: The generating, according to the initial identifier corresponding to the first engine task and the task identifier corresponding to at least one second engine task, a task identifier corresponding to the first engine task includes: Sort the task identifiers corresponding to each second engine task based on a preset identifier order; The initial identifier corresponding to the first engine task is added to the end of the sequence to obtain the task identifier corresponding to the first engine task.
7. The data query method according to claim 6, characterized in that: If the task execution order between the second engine tasks is serial execution, the preset identification order is the task execution order; If the task execution order between the second engine tasks is parallel execution, the preset identification order is based on the task identification of each second engine task.
8. The data query method according to claim 1, characterized in that: The step of obtaining the corresponding intermediate file through each task identifier includes: Determine whether the intermediate files corresponding to each task identifier are stored in the cache; If an intermediate file corresponding to a first task identifier is stored in the cache, obtaining the intermediate file corresponding to the first task identifier from the cache, wherein the first task identifier is any task identifier; If the intermediate file corresponding to the first task identifier is not stored in the cache, the engine task corresponding to the first task identifier is executed to obtain the intermediate file corresponding to the first task identifier.
9. The data query method according to claim 8, characterized in that: After executing the engine task corresponding to the first task identifier to obtain the intermediate file corresponding to the first task identifier, the method further includes: Determining whether the cluster resources consumed by executing the engine task corresponding to the first task identifier meet a preset condition; If the preset condition is met, the intermediate file corresponding to the first task identifier is cached.
10. A data query device, characterized in that: include: The engine task determination module is used to determine each engine task according to the data query request; A task identifier generation module, used to generate task identifiers corresponding to each engine task, wherein each engine task corresponds to one task identifier; An intermediate file acquisition module, used to acquire corresponding intermediate files through each task identifier, wherein any intermediate file corresponds to one task identifier; The data query result determination module is used to determine the data query result according to each intermediate file.
11. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the data query method described in any one of claims 1 to 9 by executing the executable instructions.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data query method according to any one of claims 1 to 9 is implemented.
13. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the data query method as described in any one of claims 1 to 9.