Data query method and apparatus

Abstract

The application discloses a data query method and device, and relates to the technical field of computers. A specific implementation of the method comprises the following steps: receiving a query request, selecting corresponding historical query sub-tasks according to a query task carried in the query request, and generating a historical task set; then determining a query task splitting strategy according to the historical task set, and splitting the query task into a plurality of query sub-tasks; executing the plurality of query sub-tasks in parallel, and performing ordered merging on query results corresponding to the plurality of query sub-tasks, so as to generate a query result corresponding to the query request. The implementation overcomes the technical problems of poor compatibility, slow query speed and unstable data query process of the existing query mode, and achieves the technical effect of improving the data query efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for data querying. Background Technology

[0002] In Java projects, there is a need to query and export data from Hive tables. Currently, there are two solutions for this scenario: one is to obtain data through Hive's own HiveServer / HiveServer2 service; the other is to obtain data through the open-source distributed SQL query engine Presto.

[0003] In implementing this invention, the prior art has at least the following problems: the first method, HiveServer / HiveServer2, is generally not allowed to query Hive table data directly for security reasons; the second method, Presto, is not compatible with data format and Hive SQL syntax, and is slow and unstable when exporting tables with large amounts of data. Summary of the Invention

[0004] In view of this, the present invention provides a method and apparatus for data querying, capable of submitting query SQL (Structured Query Language) to obtain exported data. Furthermore, to facilitate invocation and code reuse in Java projects, a JAR file (Java Archive, a software package file format typically used to aggregate a large number of Java class files, related metadata, and resource files into a single file for developing Java platform applications or libraries) named Hive-Client is encapsulated in this invention. Hive-Client is a module / client formed by encapsulating the Hive query function, which can be directly invoked by users through an interface. Simultaneously, to overcome the slow speed and instability issues when exporting large amounts of data from tables, intelligent task splitting and automatic merging based on historical query tasks are performed, improving data query efficiency.

[0005] To achieve the above objectives, according to one aspect of the present invention, a data query method is provided, comprising:

[0006] Receive a query request, which carries a query task;

[0007] Based on the query task, select historical query subtasks to generate a historical task set;

[0008] A query task splitting strategy is determined based on the historical task set, and the query task is split based on the strategy to generate multiple query subtasks.

[0009] performing a plurality of query sub-tasks to obtain a first query result corresponding to each query sub-task;

[0010] merging the first query results to generate a second query result corresponding to the query request.

[0011] Preferably, the determining of the query task splitting strategy according to the historical task set comprises:

[0012] determining a time query range and a query efficiency corresponding to a historical query sub-task in the historical task set;

[0013] determining the splitting period of the query task according to the time query range and the query efficiency.

[0014] Preferably, the determining of the splitting period of the query task according to the time query range and the query efficiency comprises:

[0015] selecting a predetermined number of historical query sub-tasks from the historical task set;

[0016] classifying the historical query sub-tasks selected from the historical task set according to the time query range, and calculating a first average query efficiency of each class of historical query sub-tasks;

[0017] selecting a time query range corresponding to a historical query sub-task classification with the highest first average query efficiency as the splitting period of the query task.

[0018] Preferably, the determining of the splitting period of the query task according to the time query range and the query efficiency comprises:

[0019] selecting a predetermined number of historical query sub-tasks from the historical task set;

[0020] classifying the historical query sub-tasks selected from the historical task set according to the time query range, and calculating a first average query efficiency of each class of historical query sub-tasks;

[0021] calculating an average value of the first average query efficiency to obtain a second average query efficiency;

[0022] determining a matched historical query sub-task classification according to the second average query efficiency, and taking a time query range thereof as the splitting period of the query task.

[0023] Preferably, the selecting of the historical query sub-task according to the query task to generate the historical task set comprises:

[0024] determining a value range of the splitting period according to the query target object, selecting a historical query sub-task based on the value range to generate the historical task set; and / or

[0025] According to the query field selection history query subtask of the query task, a history task set is generated.

[0026] Preferably, after receiving the query request, it is further judged whether the query time range of the query task is less than a preset threshold, and if yes, the task query is directly executed.

[0027] Preferably, in the execution of the plurality of query subtasks, for each query subtask, the following query steps are executed:

[0028] The query subtask is parsed to determine the task parameters and the execution parameters of the query engine;

[0029] According to the task parameters, an initial query result file and a corresponding field are created;

[0030] According to the execution parameters, the query engine executes the subtask query, and the query result is written into the initial query result file to generate the first query result.

[0031] Preferably, the first query result is merged to generate a second query result corresponding to the query request, including:

[0032] The plurality of query subtasks corresponding to the query task are set with an identifier;

[0033] According to the identifier, it is judged whether the plurality of query subtasks are correctly executed and the first query result is generated;

[0034] If yes, the first query result is merged.

[0035] Preferably, the subtask query step is encapsulated and called through an interface.

[0036] According to another aspect of the embodiment of the application, a data query device is provided, including:

[0037] A receiving module is configured to receive a query request, wherein the query request carries a query task;

[0038] A history task set generation module is configured to select a history query subtask according to the query task, and generate a history task set;

[0039] A task splitting module is configured to determine a query task splitting strategy according to the history task set, and split the query task based on the strategy to generate a plurality of query subtasks;

[0040] A subtask execution module is configured to execute the plurality of query subtasks to obtain a first query result corresponding to each query subtask;

[0041] The merging module merges the first query result to generate a second query result corresponding to the query request.

[0042] According to still another aspect of the embodiments of the present application, a data query device is provided, comprising:

[0043] one or more processors;

[0044] a storage device configured to store one or more programs,

[0045] When the one or more programs are executed by the one or more processors, the one or more processors implement the foregoing data query method.

[0046] According to still another aspect of the embodiments of the present application, a computer readable medium having a computer program stored thereon is provided, wherein the program, when executed by a processor, implements the foregoing data query method.

[0047] One embodiment of the above-mentioned application has the following advantages or beneficial effects: because the intelligent task splitting based on historical query tasks and the technical means of querying the split subtasks through the encapsulated client are adopted, the technical problems of poor compatibility, slow query speed and unstable data query process of the existing query method are overcome, and the technical effect of improving the data query efficiency is achieved.

[0048] The further effects of the above-mentioned non-conventional optional mode will be described in the following combined with the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings serve to better understand the present application and do not constitute an improper limitation thereto. Among them:

[0050] Figure 1 is a schematic diagram of the main process of the data query method according to an embodiment of the present application;

[0051] Figure 2 is a schematic diagram of the main process of the query subtask execution according to another embodiment of the present application;

[0052] Figure 3 is a schematic diagram of the main process of the query task execution according to still another embodiment of the present application;

[0053] Figure 4 is a schematic diagram of the partial query result of the query subtask according to still another embodiment of the present application;

[0054] Figure 5 is a schematic diagram of the main module of the data query device according to still another embodiment of the present application;

[0055] Figure 6 is an exemplary system architecture diagram to which embodiments of the present application can be applied;

[0056] Figure 7 is a structural schematic diagram of a computer system of a server suitable for implementing embodiments of the present application. DETAILED DESCRIPTION

[0057] Exemplary embodiments of the present application are described herein with reference to the accompanying drawings, which are meant to be exemplary. It should be understood that various changes and modifications to the embodiments described herein will be apparent to those skilled in the art without departing from the scope and spirit of the application. Similarly, it should be understood that the detailed description and specific examples, while indicating preferred embodiments of the application, are given by way of illustration only and are not meant to limit the scope of the application. The description of features and functions in the following description is not meant to limit the application to the specific described features and functions, but rather, the description is meant to provide an enabling description of the general nature of the application.

[0058] Figure 1 is a schematic diagram of the main flow of a data query method according to an embodiment of the present application, as shown in Figure 1 the method comprises:

[0059] Step S101: receiving a query request, the query request carrying a query task.

[0060] Taking data export of a large amount of Hive table data as an example, after receiving a data export query request of a user, by parsing the SQL query in the user request, the specific query task of the user can be determined, including query fields, query time range, query objects, and the like.

[0061] The form of a common SQL statement is as follows:

[0062] SELECT indicator 1, …, indicator n FROM tableA WHERE dt >= 'date 1' AND dt <= 'date 2'.

[0063] Wherein, indicator 1-indicator n are fields queried by the user; tableA is the target object of the user query, which is usually the target data table name of the query, i.e., in which data tables the above fields are queried; dt represents the query date, and "dt >= 'date 1' AND dt <= 'date 2'" indicates that the start time of the query is date 1 and the end time is date 2, and correspondingly, the query time range T = date 2-date 1.

[0064] Optionally, when the query time range of the query task obtained is less than a preset threshold, the task query can be directly executed without subsequent splitting of the query task.

[0065] Step S102: selecting historical query sub-tasks according to the query task, and generating a historical task set.

[0066] In the implementation of the present application, the historical query sub-tasks are analyzed first to obtain the start time, end time, query data table name, query time consumption, and the number of rows of the generated data file, and then the query task splitting strategy is determined based on the above information.

[0067] Further, the historical query sub-tasks can be filtered according to the query time range, the query target object, and other information of the query task determined in step 101 to generate a historical task set, so that the query task splitting strategy can be determined more accurately in step 103. The historical task set is composed of the filtered historical query sub-tasks.

[0068] In an embodiment of the present application, the historical query sub-tasks can be filtered according to the query target object in the query task, which is usually the query data table name. Since the data fields recorded in the same query data table are relatively fixed, and the amount of data generated every day is also considerable, the historical query sub-tasks with the same query data table as in the query task can be selected to generate the historical task set.

[0069] In addition, the historical query sub-tasks can also be filtered based on the query time range. Generally, the minimum and maximum values of the query time range that can be supported by the same query data table can be determined, for example, the query time range can be set to [5 days, 40 days], that is, the splitting period of the query task needs to be within the above range. Correspondingly, when filtering the historical query sub-tasks, the historical query sub-tasks with the query time range within the above range can be selected to generate the historical task set.

[0070] In another implementation of the present application, the historical query sub-tasks can also be filtered according to the query fields in the query task. Since the number of fields is different, the execution efficiency of the query task is different, and in the above selection process, the historical query sub-tasks matching the query fields in the query task can be selected to generate the historical task set. Specifically, it can be completely the same, or partially the same, or the query weight is set for the fields according to the average query time consumption of different fields, the weighted values of the query task and the historical sub-tasks are calculated respectively, and the historical sub-tasks close to the weighted value of the query task are selected to generate the historical task set.

[0071] It should be noted that the above-mentioned multiple ways of generating the historical task set can be used alone or together, and the present application does not make any limitation.

[0072] In addition, in order to more accurately determine the historical task set, a historical query subtask with an incomplete query time range can also be removed during selection. The historical query subtask with an incomplete query time range refers to a historical query subtask whose query time range is less than the split period and is not the same as the split period of other historical query tasks during the same historical query task splitting process. For example, the query time range of a historical query task is 43 days, and the split period is 20 days, so the historical query task can be split into two 20-day query subtasks and one 3-day query subtask. At the same time, by analyzing the split periods of other historical query tasks, it is found that, in addition to the above-mentioned 20 days, only 5 days, 10 days, and 30 days have split periods that are not the same as the query time range of the last subtask of 3 days. Therefore, when generating the historical task set, the historical query subtask with a query time range of 3 days can be filtered out.

[0073] Step S103: determining a query task splitting strategy according to the historical task set, and splitting the query task based on the strategy to generate a plurality of query subtasks.

[0074] The query task splitting strategy refers to the principle of splitting the query task, which can be directly specified by a user, or can be based on the fields of the query task and / or based on the query time range of the query task. Preferably, in order to make the time of parallel execution of the split subtasks close to each other, facilitating the subsequent merging of the query results generated by the subtasks, in the embodiments of the present application, the query task is preferably split based on the query time range of the query task.

[0075] In a specific implementation process, a binary variable [T, t] can be generated for each historical query subtask, where T represents the query time range, T=query end time-query start time; t represents the unit query time, t=query total time / T. The smaller the t value, the higher the query efficiency, and the larger the t value, the lower the query efficiency.

[0076] After determining the binary variable of each historical query subtask in the historical task set, the split period of the query task can be determined according to the query time range and the query efficiency in the binary variable.

[0077] In an embodiment, a predetermined number of historical query subtasks can be selected from the historical task set in order from high to low according to the query efficiency, the selected historical query subtasks can be classified according to the time query range, and the first average query efficiency of each category of historical query subtasks can be calculated. The time query range corresponding to the historical query subtask classification with the highest first average query efficiency is selected as the split period of the query task.

[0078] Table 1 shows an example of selecting 8 historical query sub-tasks with the minimum unit query time (the highest query efficiency) from the historical task set.

[0079] Table 1

[0080] Query time range (days) Unit query time consumption (minutes) 10 5 20 5 30 4 40 6 10 5.5 20 6.5 30 5 40 6

[0081] By querying the time range, the 8 historical query sub-tasks in Table 1 can be divided into 4 categories, and the first average unit query time corresponding to each category of historical query sub-tasks is calculated, as shown in Table 2.

[0082] Table 2

[0083]

[0084] Further, in Table 2, the query sub-task category with the minimum first average unit query time (4.5 minutes) is selected, and the corresponding query time range (30 days) is taken as the split period.

[0085] In another embodiment, considering the number of fields queried by each historical query sub-task and the different server loads during query execution, the first average unit query time can be further averaged to obtain a second average unit query time, and then the matched historical query sub-task category is determined according to the second average unit query time, and the time query range thereof is taken as the split period of the query task. For example, according to Table 2, the second average unit query time can be determined as: (5.25+5.75+4.5+6) / 4=5.375 (minutes), and then the matched historical query sub-task category (i.e. the historical query sub-task category with the first average unit query time closest to the second average unit query time) is determined according to the second average unit query time in Table 2, and the split period is determined as 10 days.

[0086] Step S104: performing a plurality of query sub-tasks to obtain a first query result corresponding to each query sub-task.

[0087] The application obtains export data by submitting a query SQL (Structured Query Language), and specifically, the SQL can be Hive-e or Hive-f. Hive-e or Hive-f is a Hive query command. Hive-e can directly execute an SQL statement, and Hive-f is used to execute a file formed by an SQL statement. Taking Hive-e SQL as an example, in the embodiment of the application, a Jar package Hive-Client is encapsulated, and each query subtask can execute a query task through the Hive-Client. Specifically, in the execution process, in order to facilitate the implementation of calling in a JAVA project and the implementation of code reuse, a sub-process can be created in the current JAVA process through a Runtime class to execute an SQL, such as Process process = Runtime.getRuntime().exec(command), that is, the thread of the current code calling line will pull up a process to run Hive-e SQL query on the current machine, and the query SQL is submitted to a Hive cluster to execute query. The command is command = {"hive", "-e", sql}, and then the query result can be obtained through a function process.getInputStream(), and written into a data file. The query process of the Hive-Client will be further described in subsequent embodiments of the application.

[0088] Since Hive-e SQL needs to be executed through the Hive-Client, the Hive access environment needs to be configured on the server executing the Hive-Client. Actually, it is too cumbersome and low in efficiency to deploy the Hive access environment on each JAVA project using the Hive-Client. Therefore, in the embodiment of the application, the Hive environment is configured on a specified server, a WEB service is developed based on the Jar of the Hive-Client, and RPC and HTTP service interfaces are provided, so that the user can directly call the interface, and the cost of using the Hive-Client is simplified.

[0089] Step S105: merging the first query result to generate a second query result corresponding to the query request.

[0090] After each subtask is completed by Hive-Client concurrently, the query result of each subtask can be merged to obtain the query result corresponding to the query task.

[0091] The above embodiment of the present application forms a splitting strategy based on historical query tasks, can realize intelligent splitting of the query task, and improves query efficiency.

[0092] Figure 2 Fig. 1 is a schematic diagram of the main process of executing a query subtask according to an embodiment of the present application.

[0093] Step 201: initialize an object based on query SQL parameters.

[0094] Specifically, the object can be initialized by defining a JAVA function, for example, HiveClient hiveClient = new HiveClient(parameters) is used to initialize the object, wherein the parameters parameters are object parameters encapsulating query SQL information.

[0095] Step 202: execute parameter analysis to determine the execution parameters of the query engine.

[0096] Specifically, the SQL can be analyzed by the function resolveHiveEnvProperties() to determine the execution parameters of the query engine, that is, different Hive execution parameters are set for different Hive tables, such as the number of MapReduce, jvm parameters, etc., to achieve the optimal execution effect.

[0097] Step 203: execute parameter analysis to determine the user task parameters.

[0098] The task parameters submitted by the user are parsed by the function resolveParameters() to obtain the task parameters including table header parameters, enumeration value conversion, etc.

[0099] Step 204: generating a data file and a header.

[0100] The function generateFileAndAddTitle() is used to create a data result file and set the table header data of the file based on the task parameters parsed in step 203.

[0101] Step 205: submitting execution of SQL to obtain results and write the data file.

[0102] The function executeSqlAndAppendData() is used to continuously write the query results into the data result file created in step 204 in an appending manner. Specifically, the SQL query can be performed in the distributed query mechanism of the Hive engine itself in the prior art, such as parsing the logical query plan corresponding to the subtask into a specific physical execution plan and distributing the specific physical execution plan to different threads on different nodes of the cluster for execution.

[0103] Preferably, in order to facilitate the use of the Hive-Client, the steps 202-205 can be further encapsulated to execute the query and return the query result by the function hiveClient.procedure(). The return result of the hiveClient.procedure() can be a status code, such as int returnCode = hiveClient.procedure(), which is used to determine the status of the current query, such as success, failure, pause, etc., and the query result can be directly written into the data file by executeSqlAndAppendData() in step 205. Through the above encapsulation, the user only needs to call the hiveClient.procedure() to obtain the query data file and the status of the corresponding query result.

[0104] The Hive-Client query method provided by the above embodiment of the application determines the execution parameters and task parameters of the query engine through parameter parsing, thereby being capable of more efficiently providing the query result, and overcomes the technical problems of poor compatibility, slow query speed and unstable query process of the prior query method. Meanwhile, through further encapsulation of the query process, the user can directly use the Hive-Client to execute the subtask query, thereby improving the user experience.

[0105] Figure 3 is a schematic diagram of the main process of the query task execution according to the embodiment of the application.

[0106] Take the following query task as an example.

[0107] SQL: SELECT partition, direct order volume FROM tableB WHERE dt >= '2020-10-01' AND dt <= '2020-12-29'.

[0108] According to the number of records generated by TableB every day, which is usually in the order of hundreds of millions, the maximum and minimum values of the query task splitting period are determined, for example, the minimum value is 5 days and the maximum value is 50 days, which means that when the query time range of the query task is less than 5 days, Hive-Client query can be directly called without splitting the task. When the query task needs to be split, the query time range of the split subtask should not be greater than 50 days to prevent the query success rate of the subtask from being affected due to the too long query time range. Therefore, according to the above SQL query task, it can be determined that the query time range of the current task is 90 days, which needs to be split.

[0109] The task splitting component is responsible for splitting the query task, which splits the task according to the intelligent splitting strategy. The selection of the historical task set and the determination of the splitting strategy are similar to steps 102 and 103 in the foregoing embodiments. For example, according to steps 102 and 103, the splitting period of the current query task can be determined as 30 days, and the task splitting component splits the current query task into 3 subtasks SQL1~SQL3, as follows:

[0110] SQL1: SELECT partition, direct order volume FROM tableB WHERE dt >= '2020-10-01' AND dt <= '2020-10-30'.

[0111] SQL2: SELECT partition, direct order volume FROM tableB WHERE dt >= '2020-10-31' AND dt <= '2020-11-29'.

[0112] SQL3: SELECT partition, direct order volume FROM tableB WHERE dt >= '2020-11-30' AND dt <= '2020-12-29'.

[0113] Each subtask executes the query of the subtask by calling the Hive-Client, and obtains the query result. The specific process is introduced in Embodiment 2. Taking the SQL1 task as an example, object initialization is performed by using HiveClient hiveClient = new HiveClient (parameters), and the information of the query SQL1 is encapsulated; then the query is executed and the query result and data file are returned by using int returnCode = hiveClient.procedure(). Figure 2 Similar to the embodiment in the background art, when hiveClient.procedure() is executed, the parameters are parsed, the execution parameters of the query engine and the user task parameters are determined, and then the data file and the header are generated according to the parsed parameters, and further, the SQL1 is executed to obtain the result and write the result into the data file. Figure 4 Part of the query result of the subtask SQL1 is shown. The table header includes "date", "partition" and "direct order quantity", and the corresponding query result is written into the table in an appended manner.

[0114] After the subtasks SQL1- SQL3 are executed correctly, the query results are merged by using the subtask automatic merging component, and the user is notified to download.

[0115] Figure 5 is a schematic diagram of main modules of the data query device according to an embodiment of the present application;

[0116] The data query device 500 comprises:

[0117] A receiving module 501 is configured to receive a query request, wherein the query request carries a query task;

[0118] A historical task set generating module 502 is configured to select historical query subtasks according to the query task, and generate a historical task set;

[0119] A task splitting module 503 is configured to determine a query task splitting strategy according to the historical task set, and split the query task based on the strategy, and generate a plurality of query subtasks;

[0120] A subtask executing module 504 is configured to execute the plurality of query subtasks, and obtain a first query result corresponding to each query subtask;

[0121] A merging module 505 is configured to merge the first query results, and generate a second query result corresponding to the query request.

[0122] Figure 6 An exemplary system architecture 600 to which the data query method or the data query device according to the embodiments of the present application can be applied is shown.

[0123] As shown in Figure 6 System architecture 600 can include terminal devices 601, 602, 603, network 604 and servers 605, 606, 607, 608. Network 604 is used to provide communication link medium between terminal devices 601, 602, 603 and servers 605. Network 604 can include various connection types, such as wired, wireless communication link or optical cable, etc. Server 605 can also be connected to servers 606, 607, 608 through any of the above connection modes, and servers 606, 607, 608 can be cluster nodes used to execute specific physical execution plans parsed by server 605 from logical query plans.

[0124] Users can use terminal devices 601, 602, 603 to interact with server 605 through network 604 to send query requests. Terminal devices 601, 602, 603 can access query services provided by server 605 through RPC and / or HTTP interface through WEB mode.

[0125] Terminal devices 601, 602, 603 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc.

[0126] Server 605 can be a server providing data query function, which is configured with Hive environment, and can develop a WEB service based on Hive-Client's Jar to provide data query function.

[0127] It should be noted that the data query method provided by the embodiment of the application is generally executed by server 605, and correspondingly, the data query device is generally provided in server 605.

[0128] It should be understood that Figure 6 the number of terminal devices, networks and servers in

[0129] Reference is made to Figure 7 which shows a structural schematic diagram of a computer system 700 of a terminal device suitable for implementing the embodiment of the application. Figure 7 The terminal device shown is only an example and should not bring any limitation to the function and use range of the embodiment of the application.

[0130] As shown in Figure 7As shown, the computer system 700 includes a central processing unit (CPU) 701 which can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 702 or loaded into a random access memory (RAM) 703 from a storage section 708. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0131] Connected to the I / O interface 705 are an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a display device such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as necessary. A removable recording medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 710 as necessary, so that a computer program read therefrom is installed into the storage section 708 as necessary.

[0132] In particular, the processes described above with reference to the flow charts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable recording medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-described functions defined in the system of the present disclosure are performed.

[0133] It should be noted that the computer-readable medium shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In the present application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0134] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code containing one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0135] The modules described in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware. The described modules can also be arranged in a processor, for example, can be described as: a processor comprising a receiving module, etc. Among them, the name of these units does not constitute a limitation on the unit itself in some cases, for example, the receiving module can also be described as "a module for receiving a query request".

[0136] As another aspect, the present application also provides a computer readable medium, which can be included in the device described in the above embodiments, or can exist independently without being assembled into the device. The above computer readable medium carries one or more programs, when the one or more programs are executed by the device, the device:

[0137] receiving a query request, the query request carrying a query task;

[0138] selecting a historical query subtask according to the query task, and generating a historical task set;

[0139] determining a query task splitting strategy according to the historical task set, and splitting the query task based on the strategy to generate a plurality of query subtasks;

[0140] executing the plurality of query subtasks to obtain a first query result corresponding to each query subtask;

[0141] merging the first query results to generate a second query result corresponding to the query request.

[0142] According to the technical scheme of the embodiments of the present application, the intelligent task splitting based on historical query tasks and the technical means of querying the split subtasks through the encapsulated client are adopted, which overcomes the technical problems of poor compatibility, slow data query speed and unstable query process of the existing query method, and achieves the technical effect of improving the data query efficiency.

[0143] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can occur depending on design requirements and other factors. Any modification, equivalent replacement and improvement made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A data query method, characterized in that, include: Receive a query request, which carries a query task; Based on the query task, select historical query subtasks to generate a historical task set; A query task splitting strategy is determined based on the historical task set, and the query task is split based on the strategy to generate multiple query subtasks. Execute multiple query subtasks and obtain the first query result for each query subtask; The first query results are merged to generate a second query result corresponding to the query request; Based on the query task, select historical query subtasks to generate a historical task set, including: setting query weights for fields based on the average query time of different fields, calculating the weighted value of the query task and the weighted value of the historical subtasks respectively, removing historical query subtasks with incomplete query time ranges, and selecting historical subtasks with weighted values ​​close to those of the query task to generate a historical task set.

2. The method according to claim 1, characterized in that, The strategy for determining query task splitting based on the historical task set includes: Determine the time range and query efficiency corresponding to the historical query subtasks in the historical task set; Based on the time query range and query efficiency, the splitting cycle of the query task is determined.

3. The method according to claim 2, characterized in that, The step of determining the splitting cycle of the query task based on the time query range and query efficiency includes: Select a predetermined number of historical query subtasks from the historical task set; The historical query subtasks selected from the historical task set are classified according to the time query range, and the first average query efficiency of each category of historical query subtasks is calculated. The time range corresponding to the historical query subtask category with the highest average query efficiency is selected as the splitting period for the query task.

4. The method according to claim 2, characterized in that, The step of determining the splitting cycle of the query task based on the time query range and query efficiency includes: Select a predetermined number of historical query subtasks from the historical task set; The historical query subtasks selected from the historical task set are classified according to the time query range, and the first average query efficiency of each category of historical query subtasks is calculated. Calculate the average of the first average query efficiency to obtain the second average query efficiency; Based on the second average query efficiency, the matching historical query subtasks are classified, and their time query range is used as the splitting period of the query task.

5. The method according to any one of claims 1-4, characterized in that, Based on the query task, select historical query subtasks to generate a historical task set, which also includes: The range of values ​​for the splitting period is determined based on the target object being queried. Based on the range of values, historical query subtasks are selected to generate a set of historical tasks.

6. The method according to claim 1, characterized in that, After receiving the query request, it further determines whether the query time range of the query task is less than a preset threshold. If so, the task query is executed directly.

7. The method according to any one of claims 1-4, characterized in that, When executing multiple query subtasks, the following query steps are performed for each query subtask: Analyze the query subtasks to determine the task parameters and the execution parameters of the query engine; Create an initial query result file and corresponding fields based on the task parameters; The query engine executes a subtask query based on the execution parameters and writes the query results into the initial query result file to generate the first query result.

8. The method according to any one of claims 1-4, characterized in that, The first query results are merged to generate a second query result corresponding to the query request, including: Set identifiers for multiple query subtasks corresponding to a query task; Based on the identifier, determine whether multiple query subtasks have been executed correctly and generate the first query result; If so, the results of the first query will be merged.

9. The method according to claim 7, characterized in that, The subtask query steps are encapsulated and called through an interface.

10. A data query device, characterized in that, include: The receiving module is used to receive query requests, which carry query tasks. The historical task set generation module selects historical query subtasks based on the query task to generate a historical task set; The task splitting module determines a query task splitting strategy based on the historical task set, and splits the query task based on the strategy to generate multiple query subtasks. The subtask execution module executes multiple query subtasks and obtains the first query result for each query subtask. The merging module merges the first query results to generate a second query result corresponding to the query request; Based on the query task, select historical query subtasks to generate a historical task set, including: setting query weights for fields based on the average query time of different fields, calculating the weighted value of the query task and the weighted value of the historical subtasks respectively, removing historical query subtasks with incomplete query time ranges, and selecting historical subtasks with weighted values ​​close to those of the query task to generate a historical task set.

11. A data query device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-9.

Images