Data query method, master control server and database system

Abstract

The embodiment of the invention provides a data query method, a master control server and a database system. The method can be used in the distributed database field. The method is applied to a master control server in a database system, the database system comprises the master control server and a plurality of server extensions, and the master control server is connected with the server extensions. The method comprises the steps of splitting a data query job into a plurality of data query sub-jobs in response to a data query job request; if the number of the data query sub-jobs is greater than or equal to a preset sub-job number threshold value, distributing the plurality of data query sub-jobs to a plurality of server extensions; wherein the plurality of data query sub-jobs are used for being executed in parallel by the plurality of server extensions to obtain execution results of the data query sub-jobs; and obtaining a result of the data query job according to the execution result of each data query sub-job. The method is used for achieving the effect of improving the efficiency of data query operation.

Description

Technical Field

[0001] The present application relates to the field of distributed databases, and in particular to a data query method, a master control server and a database system. Background Art

[0002] Batch processing is a data processing mode that can be used to perform large-scale and complex data processing jobs. Batch processing can be applied to the field of distributed databases. Through batch processing, large-scale data query jobs can be automatically executed in distributed databases.

[0003] In some technologies, when executing a data query job in a distributed database, batch processing is directly performed from the distributed database, and all data is queried from the distributed database at one time. In the above technologies, the execution efficiency of the data query job is low. Summary of the invention

[0004] The data query method, main control server and database system provided in the embodiments of the present application are used to achieve the effect of improving the efficiency of data query operations.

[0005] In a first aspect, an embodiment of the present application provides a data query method, which is applied to a master control server in a database system, wherein the database system includes a master control server and multiple server extensions, wherein the master control server is connected to the multiple server extensions respectively; the method includes:

[0006] In response to a data query job request, split the data query job into a plurality of data query sub-jobs;

[0007] If the number of data query sub-jobs is greater than or equal to a preset sub-job number threshold, multiple data query sub-jobs are assigned to multiple server extensions; wherein the multiple data query sub-jobs are used to be executed in parallel by the multiple server extensions to obtain execution results of the data query sub-jobs;

[0008] The result of the data query job is obtained according to the execution results of each data query sub-job.

[0009] In a possible implementation manner, the data query job request includes the splitting dimension of the data query job, the splitting strategy of the data query job, and the number of data query sub-jobs;

[0010] In response to a data query job request, the data query job is split into multiple data query sub-jobs, including:

[0011] In response to the data query job request, the data query job is split into a plurality of data query sub-jobs according to the splitting dimension of the data query job, the splitting strategy of the data query job, and the number of data query sub-jobs in the data query job request.

[0012] In a possible implementation, the splitting dimension includes at least one of the following: splitting by one field, splitting by multiple fields, and splitting by the entire row; the splitting strategy includes at least one of the following: hash splitting and range splitting.

[0013] In a possible implementation, allocating multiple data query sub-tasks to multiple server extensions includes:

[0014] Determine the extension sub-job quantity threshold according to the number of data query sub-jobs and the number of server extensions;

[0015] According to the extension sub-job quantity threshold, multiple data query sub-jobs are allocated to multiple server extensions; wherein the number of data query sub-jobs allocated to each server extension is less than or equal to the extension sub-job quantity threshold.

[0016] In a possible implementation, when the number of data query sub-jobs is greater than or equal to a preset sub-job number threshold, the data query sub-jobs are configured to be executed in parallel by the server extension based on the server extension threads.

[0017] In a possible implementation, the method further includes:

[0018] If the number of data query sub-jobs is less than a preset sub-job number threshold, multiple data query sub-jobs are assigned to multiple master control server threads for parallel execution to obtain execution results of the data query sub-jobs.

[0019] In a possible implementation, the database system corresponds to a data query job information table, and the method further includes:

[0020] Record the grouping characteristic values ​​and job distribution conditions corresponding to the data query sub-jobs in the data query job information table;

[0021] The job execution status, error stack, and IP address of the master server of the data query sub-job assigned to each master server thread are recorded in the data query job information table.

[0022] In a possible implementation, the database system corresponds to a data query job information table, which records the job execution status of the data query sub-job assigned to the server extension, the error stack, and the IP address of the server extension.

[0023] In a second aspect, an embodiment of the present application provides a data query device, including:

[0024] The splitting module is used to split the data query job into multiple data query sub-jobs in response to the data query job request.

[0025] The allocation module is used to allocate multiple data query sub-jobs to multiple server extensions if the number of data query sub-jobs is greater than or equal to a preset sub-job number threshold; wherein the multiple data query sub-jobs are used to be executed in parallel by multiple server extensions to obtain execution results of the data query sub-jobs.

[0026] The processing module is used to obtain the result of the data query job according to the execution results of each data query sub-job.

[0027] In a possible implementation, the data query job request includes the splitting dimension of the data query job, the splitting strategy of the data query job, and the number of data query sub-jobs.

[0028] In response to a data query job request, the data query job is split into multiple data query sub-jobs, and the splitting module is used to:

[0029] In response to the data query job request, the data query job is split into a plurality of data query sub-jobs according to the splitting dimension of the data query job, the splitting strategy of the data query job, and the number of data query sub-jobs in the data query job request.

[0030] In a possible implementation, the splitting dimension includes at least one of the following: splitting by one field, splitting by multiple fields, and splitting by the entire row; the splitting strategy includes at least one of the following: hash splitting and range splitting.

[0031] In a possible implementation, multiple data query sub-tasks are assigned to multiple server extensions, and the assignment module is used to:

[0032] Determine the extension sub-job quantity threshold according to the number of data query sub-jobs and the number of server extensions;

[0033] According to the extension sub-job quantity threshold, multiple data query sub-jobs are allocated to multiple server extensions; wherein the number of data query sub-jobs allocated to each server extension is less than or equal to the extension sub-job quantity threshold.

[0034] In a possible implementation, when the number of data query sub-jobs is greater than or equal to a preset sub-job number threshold, the data query sub-jobs are configured to be executed in parallel by the server extension based on the server extension threads.

[0035] In a possible implementation, the allocation module is further configured to:

[0036] If the number of data query sub-jobs is less than a preset sub-job number threshold, multiple data query sub-jobs are assigned to multiple master control server threads for parallel execution to obtain execution results of the data query sub-jobs.

[0037] In a possible implementation, the database system corresponds to a data query operation information table, and the processing module 303 is further used to:

[0038] Record the grouping characteristic values ​​and job distribution conditions corresponding to the data query sub-jobs in the data query job information table;

[0039] The job execution status, error stack, and IP address of the master server of the data query sub-job assigned to each master server thread are recorded in the data query job information table.

[0040] In a possible implementation, the database system corresponds to a data query job information table, which records the job execution status of the data query sub-job assigned to the server extension, the error stack, and the IP address of the server extension.

[0041] In a third aspect, an embodiment of the present application provides a master control server, including: a memory, a processor;

[0042] Memory stores computer-executable instructions;

[0043] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0044] In a fourth aspect, an embodiment of the present application provides a database system, comprising a main control server as provided in the third aspect above, and a plurality of server extensions, wherein the main control server is connected to the plurality of server extensions respectively.

[0045] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0046] In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0047] The data query method, master control server and database system provided by the embodiments of the present application split the data query job based on the request of the data query job in the master control server of the database system to obtain multiple data query sub-jobs; the master control server makes a judgment based on the number of data query sub-jobs, and if the number of data query sub-jobs exceeds a preset threshold, the data query sub-jobs are assigned to multiple server extensions connected to the master control server, and the multiple server extensions execute the data query sub-jobs in parallel to obtain the execution results of the data query sub-jobs; the execution results of the data query sub-jobs are integrated to obtain the execution results of the original data query job, so as to achieve the effect of improving the execution efficiency of the data query job. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0049] Figure 1 Schematic diagram of the data query method provided in this application Figure 1 ;

[0050] Figure 2 Schematic diagram of the data query method provided in this application Figure 2 ;

[0051] Figure 3 A schematic diagram of the structure of the data query device provided by this application;

[0052] Figure 4 A schematic diagram of the structure of the master control server provided for this application;

[0053] Figure 5 A schematic diagram of the structure of the database system provided for this application.

[0054] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0055] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0056] First, the terms involved in this application are explained:

[0057] Database system: may refer to a distributed database system, in which a master server and multiple server extensions can access the distributed database system, so that the master server and the server extensions can query or process the data information in the distributed database.

[0058] Hash splitting: refers to mapping data into a hash table so that the data can be evenly distributed in different locations, thereby achieving equal distribution of data. Specifically, the input data is mapped to a hash value of a fixed length through a hash function; wherein the hash function may include but is not limited to: MD5 message digest algorithm (Message Digest Algorithm 5, referred to as MD5), SHA-1 secure hash algorithm (Secure Hash Algorithm 1, referred to as SHA-1), SHA-256 secure hash algorithm (Secure Hash Algorithm 256, referred to as SHA-256).

[0059] Thread: refers to the smallest unit that can be used for operation and scheduling in an operating system. Thread is the actual operation unit of a process. Multiple threads can be executed concurrently in a process, and each thread can execute different tasks in parallel.

[0060] In database systems, the amount of data stored is large. Batch processing can accumulate a series of tasks or data and process them all at once. Therefore, batch processing can be used to perform large-scale and complex data processing jobs. In the field of distributed databases, batch processing can be used to automate large-scale data query jobs.

[0061] In some embodiments, when executing a data query operation in a distributed database, the master control server directly performs batch processing from the distributed database, that is, all required data is queried from the distributed database at one time. Specifically, when the master control server directly performs batch processing to execute the data query operation, all required data is fetched into the memory, which may cause the memory overflow of the master control server. In addition, the master control server directly queries all required data from the distributed database in serial processing, and the resource and performance utilization efficiency of the master control server is low, resulting in a long time for the master control server to execute the data query operation and low execution efficiency of the data query operation.

[0062] The data query method provided by the present application splits the data query job based on the request of the data query job in the main control server of the database system, thereby obtaining multiple data query sub-jobs; the main control server makes a judgment based on the number of data query sub-jobs, and if the number of data query sub-jobs exceeds a preset threshold, the data query sub-jobs are allocated to multiple server extensions connected to the main control server, and the multiple server extensions execute the data query sub-jobs in parallel to obtain the execution results of the data query sub-jobs; the execution results of the data query sub-jobs are integrated to obtain the execution results of the original data query job, thereby solving the technical problem of low execution efficiency of the data query job.

[0063] It should be noted that the data query method, master control server and database system provided in the present application can be used in the field of distributed databases, and can also be used in any field except the field of distributed databases. The application field of the data query method, master control server and database system provided in the present application is not limited.

[0064] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0065] The data query method provided by the present application is applied to a master control server in a database system, wherein the database system includes a master control server and multiple server extensions, wherein the master control server is connected to the multiple server extensions respectively. Exemplarily, when the master control server is connected to the multiple server extensions respectively, the connection can be made through a socket long connection, wherein a socket long connection refers to a transmission control protocol (Transmission Control Protocol, referred to as TCP connection) established between different network devices in network communication. After the data transmission is completed, it is not closed immediately, but remains open, so that subsequent data transmission can reuse the established connection without re-establishing the connection.

[0066] Figure 1 Schematic diagram of the data query method provided in this application Figure 1 .like Figure 1 As shown, the method includes:

[0067] Step 101: In response to a data query job request, split the data query job into multiple data query sub-jobs.

[0068] Exemplarily, the data query job request may include relevant information of the data query job, and the relevant information of the data query job may be input by a user or preset by the main control server.

[0069] In one example, the relevant information of the data query job is input by a user. Based on the relevant information of the data query job input by the user, a data query job request is generated. In response to the generated data query job request, the master control server determines the data query job to be executed this time according to the relevant information of the data query job in the data query job request. Furthermore, the master control server splits the data query job into multiple data query sub-jobs according to the relevant information of the data query job in the data query job request.

[0070] For example, a data query job needs to query data in a user table named custom_info. There are 10 million data in the user table, and the data query job is to query the 10 million data in the user table. The user inputs the relevant information of the data query job, which may include the number of splits of the data query job as 10, that is, the data query job is split into 10 data query sub-jobs according to the data query job request, and the data query sub-job is to query the 1 million data in the user table.

[0071] In one example, the relevant information of the data query job is preset by the master control server. Based on the preset relevant information of the data query job, a data query job request can be generated through the user's operation on the master control server. Among them, the user's operation on the master control server may include but is not limited to: clicking the start button, setting the start execution time of the data query job, etc. In response to the generated data query job request, the master control server determines the data query job to be executed this time according to the relevant information of the data query job in the data query job request. In addition, the master control server splits the data query job into multiple data query sub-jobs according to the relevant information of the data query job in the data query job request.

[0072] Step 102: If the number of data query sub-jobs is greater than or equal to a preset sub-job number threshold, multiple data query sub-jobs are allocated to multiple server extensions.

[0073] The multiple data query sub-jobs are used to be executed in parallel by multiple server branches to obtain execution results of the data query sub-jobs.

[0074] Exemplarily, multiple data query sub-jobs can be obtained based on step 101. If the number of data query sub-jobs is greater than or equal to the preset sub-job number threshold, the master server allocates multiple data query sub-jobs to multiple server extensions. Among them, the sub-job number threshold can be any positive integer greater than 2. Further, the setting of the sub-job number threshold can be set according to the resources and performance of the master server. For example, the master server can open 20 threads for parallel execution in the process of executing the data query sub-job, and the sub-job number threshold can be set to 21. When the number of split sub-jobs is greater than the sub-job number threshold, the 20 threads that the master server can open cannot execute more than 20 data query sub-jobs in parallel, and there will be a situation where one thread has to execute multiple data query sub-jobs, and the efficiency of data query is low. Therefore, when the number of data query sub-jobs is greater than or equal to the sub-job number threshold, the data query sub-job is allocated to multiple server extensions for execution.

[0075] Specifically, when the master server assigns multiple data query sub-jobs to multiple server extensions, if the number of data query sub-jobs is less than or equal to the number of server extensions, one data query sub-job can be assigned to one server extension; if the number of data query sub-jobs is greater than the number of server extensions, multiple data query sub-jobs can be assigned to one server extension.

[0076] After the server extension receives the assigned data query sub-job, multiple server extensions can execute these data query sub-jobs in parallel, thereby obtaining the execution result of each data query sub-job respectively.

[0077] Step 103: Obtain the result of the data query job according to the execution results of each data query sub-job.

[0078] Exemplarily, the main control server is connected to multiple server extensions respectively. Therefore, after each server extension executes in parallel and obtains the execution result of the assigned data query sub-job, the execution result of the assigned data query sub-job is returned to the main control server, and the main control server integrates the execution results of each data query sub-job to obtain the result of the data query job.

[0079] The data query method provided in the embodiment of the present application, through the main control server in the database system, based on the request of the data query job, responds to the request to split the data query job into multiple sub-jobs, and distributes these sub-jobs to multiple server extensions connected to the main control server, and these server extensions execute the sub-jobs in parallel, respectively obtain the execution results of each sub-job, and return them to the main control server, and the main control server integrates the means to obtain the results of the data query job, thereby realizing the splitting of the data query job and executing the split sub-jobs in parallel, thereby improving the efficiency of the main control server in the database system in executing the data query job.

[0080] Based on the above embodiments, it can be known that when the master control server is split, the split is performed according to the information in the data query operation request. This embodiment specifically explains how the master control server is split.

[0081] Exemplarily, the data query job request includes the splitting dimension of the data query job, the splitting strategy of the data query job, and the number of data query sub-jobs.

[0082] Furthermore, the splitting dimension of the data query job may include at least one of the following: splitting by one field, splitting by multiple fields, and splitting by the entire row.

[0083] In addition, the splitting strategy of the data query job includes at least one of the following: hash splitting and range splitting.

[0084] In response to the data query job request, the data query job is split into a plurality of data query sub-jobs according to the splitting dimension of the data query job, the splitting strategy of the data query job, and the number of data query sub-jobs in the data query job request.

[0085] In one example, the master server performs splitting based on a split dimension of splitting by a field. For example, a data query job queries the data in a user table named custom_info, and the user table includes multiple fields and data under each field. For example, one of the fields of the account opening institution named create_branch in the user table may include information of multiple account opening institutions, such as institution A, institution B, institution C, etc. It can be understood that each piece of data in the user table has the information of the corresponding account opening institution.

[0086] Based on the account opening institution information in the account opening institution field, it is possible to split by range. That is, the data in the user table is split according to different account opening institutions. Specifically, according to the information of the account opening institution corresponding to the data in the user table, all data belonging to institution A can be divided into one group, all data belonging to institution B can be divided into one group, and all data belonging to institution C can be divided into one group. And so on, until all data are grouped. Each group of data can be used as a data query sub-job. Each data query sub-job obtained by splitting has a corresponding grouping feature value. In this example, the grouping feature value of the first data query sub-job can be institution A, the grouping feature value of the second data query sub-job can be institution B, and the grouping feature value of the third data query sub-job can be institution C.

[0087] In another example, the master server performs splitting based on a splitting dimension of splitting by multiple fields. Based on the above example, in addition to the account opening institution field named create_branch, the user table may also include a creation time field named create_date, and the creation time field may include information on the creation time corresponding to each piece of data.

[0088] Based on the creation time information in the creation time field, it can be split by range. Specifically, the creation time information in the creation time field is divided into multiple time periods, such as a first time period, a second time period, and a third time period. In the first time period, all data belonging to organization A is divided into a group, all data belonging to organization B is divided into a group, and all data belonging to organization C is divided into a group. In the second time period, all data belonging to organization A is divided into a group, all data belonging to organization B is divided into a group, and all data belonging to organization C is divided into a group. And so on, until all data are grouped. Each group of data can be used as a data query sub-job. Each data query sub-job obtained by splitting has a corresponding grouping feature value. In this example, the grouping characteristic value of the first data query sub-job may be the first time period and institution A, the grouping characteristic value of the second data query sub-job may be the first time period and institution B, the grouping characteristic value of the third data query sub-job may be the first time period and institution C, the grouping characteristic value of the fourth data query sub-job may be the second time period and institution A, the grouping characteristic value of the fifth data query sub-job may be the second time period and institution B, the grouping characteristic value of the sixth data query sub-job may be the second time period and institution C, the grouping characteristic value of the seventh data query sub-job may be the third time period and institution A, the grouping characteristic value of the eighth data query sub-job may be the third time period and institution B, and the grouping characteristic value of the ninth data query sub-job may be the third time period and institution C.

[0089] In another example, the master control server performs splitting based on the splitting dimension of splitting by whole row. The data information corresponding to different fields in each data in the user table can be spliced ​​to obtain the data information string corresponding to each data. Based on the data information string corresponding to each data, hash splitting can be performed. Specifically, a hash function is selected, and the data information string corresponding to each data is calculated by the hash function to obtain the hash value corresponding to the data information string. Then, the grouping feature value of the data query sub-job is determined according to the number of data query sub-jobs and the hash value corresponding to the data information string.

[0090] For example, if there are 10 million data in the user table, there are 10 million hash values, and the number of data query sub-jobs is 100. The hash value corresponding to each data is modulo 100, that is, the remainder obtained by dividing the hash value corresponding to each data by 100 is calculated. It can be understood that the remainder obtained should be an integer from 0 to 99. That is, the data query job can be divided into 100 data query sub-jobs by the result of the modulo operation of the hash value corresponding to each data. The grouping feature value corresponding to each data query sub-job is the remainder obtained by the modulo operation of the hash value corresponding to the data.

[0091] For example, the entire row of data included in the first data query sub-job is the entire row of data whose corresponding hash value modulo 100 operation result is 0, and the entire row of data included in the second data query sub-job is the entire row of data whose corresponding hash value modulo 100 operation result is 1. And so on, until the 100th data query sub-job including the entire row of data whose hash value modulo 100 operation result is 99 is obtained.

[0092] It should be noted that when the master server splits according to the splitting dimension of splitting by one field or by multiple fields, it can also split by the splitting strategy of hash splitting. Exemplarily, one of the fields in the user table is user information, and the user information field may include user information corresponding to each piece of data. Further, the user information may be reflected by the user's ID identifier. Therefore, the corresponding hash value may be obtained by performing a hash operation on the user's ID identifier in each piece of data, and the corresponding hash value may be modulo-operated according to the number of data query sub-jobs, and the data with the same remainder may be divided into data to be queried by a data query sub-job, and the same remainder may be used as the grouping feature value of the data query sub-job.

[0093] In the above embodiment, by using different splitting dimensions and different splitting strategies, it is possible to split a data query job into multiple data query sub-jobs. This lays the foundation for subsequently assigning the data query sub-jobs to server extensions for parallel execution. Furthermore, by using the splitting strategy of hash splitting, the amount of data included in different sub-jobs can be made more even, thereby improving the stability of the database system and the execution efficiency of the data query sub-jobs.

[0094] Based on the above embodiments, after the data query sub-jobs are split, they can be assigned to different server extensions for parallel execution. This embodiment specifically explains how the master control server assigns data query sub-jobs and how to execute sub-jobs.

[0095] Figure 2 Schematic diagram of the data query method provided in this application Figure 2 .like Figure 2 As shown, when allocating multiple data query sub-jobs to multiple server extensions, the method may include:

[0096] Step 201: Determine the extension sub-job quantity threshold according to the number of data query sub-jobs and the number of server extensions.

[0097] Exemplarily, the main control server can determine the extension sub-job quantity threshold according to the number of data query sub-jobs and the number of server extensions. The extension sub-job quantity threshold is used to limit the maximum number of data query sub-jobs that can be assigned to each server extension.

[0098] Specifically, a load balancing algorithm can be built into the memory of the master server, and the processor of the master server can obtain the number of server extensions currently connected to the master server. Through the load balancing algorithm, it is determined how many data query sub-jobs can be assigned to each server extension at most. For example, the number of data query sub-jobs in the current data query job request is recorded as N, the number of server extensions currently connected to the master server is recorded as M, and the extension sub-job number threshold is recorded as max_job_count. The extension sub-job number threshold can be calculated using the following formula:

[0099] max_job_count = ceil[N / M]

[0100] In the formula, ceil represents a round-up function. For example, the number of data query sub-jobs in the current data query job request is 100, and the number of server extensions currently connected to the master server is 11. Substituting into the above formula, the threshold value of the number of extension sub-jobs is 10.

[0101] Step 202: Allocate multiple data query sub-jobs to multiple server extensions according to the extension sub-job quantity threshold.

[0102] The number of data query sub-jobs allocated to each server extension is less than or equal to the extension sub-job number threshold.

[0103] Exemplarily, after the master server splits the data query job into data query sub-jobs according to the splitting dimension, splitting strategy and the number of data query sub-jobs, it assigns multiple data query sub-jobs to multiple server extensions according to the extension sub-job number threshold, so that each server extension executes the assigned data query sub-job.

[0104] Specifically, the number of data query sub-jobs assigned to each server extension can be different or the same. However, the number of data query sub-jobs assigned to each server extension cannot exceed the calculated extension sub-job number threshold.

[0105] In a possible implementation, when the number of data query sub-jobs is greater than or equal to a preset sub-job number threshold, the master control server assigns the data query sub-jobs to multiple server extensions for execution. The data query sub-jobs are used to be executed in parallel by the server extensions based on the server extension threads.

[0106] It can be understood that each server extension may be assigned to one data query sub-job, or may be assigned to multiple data query sub-jobs.

[0107] In one example, the server extension is assigned to a data query sub-job, and the server extension can directly execute the data query sub-job.

[0108] In another example, a server extension is assigned to multiple data query sub-jobs. The server extension can create multiple server extension threads in the process of executing a data query sub-job. For example, the number of data query sub-jobs is 100, and the number of server extensions is 20. It can be understood that each server extension will be assigned to 5 data query sub-jobs. Taking the first server extension among the 20 server extensions as an example, after the first server extension receives the 5 data query sub-jobs sent by the master server, 5 server extension threads are opened in the thread pool of the first server extension, and the 5 data query sub-jobs are assigned to the 5 server extension threads one by one. The first server extension executes the 5 server extension threads in parallel, thereby obtaining the execution results of the 5 data query sub-jobs assigned to the first server extension.

[0109] Optionally, when allocating data query sub-jobs to server extensions, a polling mechanism can be set. The polling mechanism means that if a data query sub-job is not received by the target server extension when it is allocated, the data query sub-job that has not been received will be allocated again, and the data query sub-job that has not been received can be allocated to other server extensions connected to the master server and executed by other server extensions.

[0110] Furthermore, when executing the assigned data query sub-task, the first server extension can specify the data query conditions through the WHERE clause in the Structured Query Language (SQL statement for short). The WHERE clause is used to filter the result set and only return records that meet specific conditions.

[0111] It can be understood that the first server extension inputs the grouping characteristic values ​​of the five data query sub-jobs corresponding to the first server extension in the WHERE clause. So that when the first server extension accesses the database to execute the data query sub-job, it only queries the assigned data query sub-job. For example, the data query job obtains 100 data query sub-jobs through hash splitting, and 5 data query sub-jobs are assigned to the first server extension. The grouping characteristic values ​​of the five assigned data query sub-jobs can be 0, 1, 2, 3, and 4. Through the WHERE clause, according to these grouping characteristic values, the data corresponding to the grouping characteristic values ​​are queried respectively.

[0112] It should be noted that the process of executing the assigned data query sub-job by the other 19 server extensions is similar to the process of executing the assigned data query sub-job by the first server extension, and will not be repeated here. Each server extension obtains the execution result of the assigned data query sub-job and returns it to the master server, so that the master server integrates the results of the original data query job.

[0113] It should also be noted that this example is only an optional example. For a server extension, one thread may be used to serially execute multiple data query sub-jobs, which is not limited in this embodiment.

[0114] In a possible implementation, if the number of data query sub-jobs is less than a preset sub-job number threshold, multiple data query sub-jobs are assigned to multiple master control server threads for parallel execution to obtain execution results of the data query sub-jobs.

[0115] Exemplarily, when the number of data query sub-jobs is less than a preset sub-job number threshold, the master server can execute the multiple data query sub-jobs obtained by splitting. Specifically, in the thread pool of the master server, a number of master server threads corresponding to the number of data query sub-jobs is created, and the data query sub-jobs are assigned to the master server threads one by one. For example, the sub-job number threshold is 20, and the number of data query sub-jobs obtained by splitting is 10. The master server creates 10 master server threads, and assigns the 10 data query sub-jobs to the 10 master server threads one by one. The master server executes the 10 master server threads in parallel, thereby obtaining the execution results of the 10 data query sub-jobs.

[0116] In the above embodiment, the extension sub-job number threshold is determined by the number of data query sub-jobs and the number of server extensions. When the master server assigns data query sub-jobs to server extensions, it is ensured that the number of data query sub-jobs assigned to each server extension does not exceed the extension sub-job number threshold. It can avoid that a server extension is assigned too many data query sub-jobs, resulting in excessive burden on the server extension to execute sub-jobs. By opening a number of server extension threads corresponding to the number of data query sub-jobs assigned to the server extension, the efficiency of the server extension to execute sub-jobs can be further improved, thereby improving the efficiency of the overall data query job execution. When the number of data query sub-jobs does not exceed the preset sub-job number threshold, multi-threaded parallel execution can be directly enabled through the master server. For the case where the number of sub-jobs is small, unnecessary data transmission can be avoided, and multiple master server threads can be directly enabled through the master server for parallel execution, thereby improving the execution efficiency of the data query job.

[0117] Based on any of the above embodiments, in the process of executing a data query job, in order to prevent the split data query sub-jobs from being repeatedly assigned and executed, the distribution and execution status of the data query sub-jobs can be recorded in a data table in the database. This embodiment explains how to record the distribution, execution status and other information of the data query job in the data table.

[0118] In one example, the database system corresponds to a data query operation information table. Based on the corresponding data query operation information table stored in the database system, the main control server and multiple server extensions can access the data query operation information table in the database system to record information.

[0119] Exemplarily, the main control server records the grouping characteristic values ​​and job distribution conditions corresponding to the data query sub-job in the data query job information table.

[0120] For example, the master server splits the data query job into 20 data query sub-jobs according to the splitting dimension of a field through the splitting strategy of hash splitting. It can be understood that the grouping characteristic values ​​of these 20 data query sub-jobs are integers between 0 and 19, and each integer corresponds to a grouping characteristic value of a data query sub-job. In the data query job information table, a field for grouping characteristic values ​​is set, and the grouping characteristic values ​​of the 20 data query sub-jobs are filled in respectively, so as to determine each data query sub-job through these grouping characteristic values.

[0121] In the data query job information table, a job distribution status field can also be set. In the job distribution status field, it is recorded whether the data query sub-job corresponding to the grouping characteristic value has been assigned to the master server thread or server extension. If the master server successfully assigns the data query sub-job corresponding to the grouping characteristic value to the master server thread or server extension, the job distribution status of the data query sub-job corresponding to the grouping characteristic value is modified in the job distribution status field to "allocated".

[0122] In a possible implementation, if the number of data query sub-jobs is less than a preset sub-job number threshold, the master server allocates the split data query sub-jobs to multiple master server threads, and each master server thread executes the allocated data query sub-job.

[0123] Exemplarily, the master control server records the job execution status, error stack, and IP address of the master control server of each master control server thread that is assigned to the data query sub-job in the data query job information table.

[0124] For example, in the data query job information table, a job execution status field is set. After the master server thread is assigned to the data query sub-job, the job execution status of the data query sub-job corresponding to the grouping characteristic value is modified to "pending execution" in the job execution status field. After the master server thread is executed, the job execution status of the data query sub-job corresponding to the grouping characteristic value is modified to "execution success" or "execution failure" according to the actual execution status.

[0125] Optionally, in the data query job information table, you can also set the error stack field and the master server IP address field. In the error stack field, you can record the failure information when the master server thread fails to execute the data query sub-job. In the master server IP address field, you can record the IP address of the master server when accessing the database.

[0126] In a possible implementation, if the number of data query sub-jobs is greater than or equal to a preset sub-job number threshold, the main control server distributes the split data query sub-jobs to multiple server extensions, and each server extension executes the distributed data query sub-job.

[0127] Exemplarily, the data query job information table records the job execution status of the data query sub-job assigned to the server extension, the error stack, and the IP address of the server extension.

[0128] For example, in the data query job information table, a job execution status field is set. After the server extension is assigned to the data query sub-job, the job execution status of the data query sub-job corresponding to the grouping characteristic value is modified to "pending execution" in the job execution status field. After the server extension finishes executing, the job execution status of the data query sub-job corresponding to the grouping characteristic value is modified to "execution success" or "execution failure" according to the actual execution status.

[0129] Optionally, in the data query job information table, you can also set the error stack field and the server extension IP address field. In the error stack field, you can record the failure information when the server extension fails to execute the data query sub-job. In the server extension IP address field, you can record the IP address of the server extension when accessing the database.

[0130] It should be noted that, no matter whether the main control server or the server extension modifies the information of the fields in the data query job information table, the data row corresponding to the data query sub-job of the corresponding grouping characteristic value is modified.

[0131] In the above embodiment, by setting up a corresponding data query job information table in the database system and allowing the main control server and the server extension to modify the information of the fields in the data query information job table respectively, it is possible to directly feedback the distribution and execution status of each data query sub-job, prevent the data query sub-job from being executed repeatedly or missed, and indirectly improve the efficiency and reliability of the data query job execution.

[0132] The data query method provided by the embodiment of the present application uses a master server in a database system and multiple server extensions connected to the master server. The master server splits the data query job to obtain multiple data query sub-jobs, which are assigned to multiple server extensions or to multiple master server threads, so that multiple data query sub-jobs are executed in parallel, and execution results are obtained respectively. Finally, the master server is used to integrate the results of the original data query job. The time consumption of data query job execution can be reduced; in addition, by executing these data query sub-jobs in parallel, the efficiency of data query job execution can be improved. Furthermore, a data query information table is set in the database system, and the distribution and execution of data query sub-jobs are recorded according to the execution of the master server thread or server extension, so as to prevent repeated execution or missed execution of data query sub-jobs, and indirectly improve the efficiency and reliability of data query job execution.

[0133] Figure 3 The data query device 30 provided in this embodiment can be applied to the processor of the master server in the database system, and the database system includes the master server and multiple server extensions, wherein the master server is connected to the multiple server extensions respectively. Figure 3 As shown, the data query device 30 provided in this embodiment includes:

[0134] The splitting module 301 is used to split the data query job into multiple data query sub-jobs in response to a data query job request.

[0135] The allocation module 302 is used to allocate multiple data query sub-jobs to multiple server extensions if the number of data query sub-jobs is greater than or equal to a preset sub-job number threshold; wherein the multiple data query sub-jobs are used to be executed in parallel by multiple server extensions to obtain execution results of the data query sub-jobs.

[0136] The processing module 303 is used to obtain the result of the data query job according to the execution results of each data query sub-job.

[0137] In a possible implementation, the data query job request includes the splitting dimension of the data query job, the splitting strategy of the data query job, and the number of data query sub-jobs.

[0138] In response to the data query job request, the data query job is split into multiple data query sub-jobs, and the splitting module 301 is used to:

[0139] In response to the data query job request, the data query job is split into a plurality of data query sub-jobs according to the splitting dimension of the data query job, the splitting strategy of the data query job, and the number of data query sub-jobs in the data query job request.

[0140] In a possible implementation, the splitting dimension includes at least one of the following: splitting by one field, splitting by multiple fields, and splitting by the entire row; the splitting strategy includes at least one of the following: hash splitting and range splitting.

[0141] In a possible implementation, multiple data query sub-tasks are assigned to multiple server extensions, and the assignment module 302 is used to:

[0142] Determine the extension sub-job quantity threshold according to the number of data query sub-jobs and the number of server extensions;

[0143] According to the extension sub-job quantity threshold, multiple data query sub-jobs are allocated to multiple server extensions; wherein the number of data query sub-jobs allocated to each server extension is less than or equal to the extension sub-job quantity threshold.

[0144] In a possible implementation, when the number of data query sub-jobs is greater than or equal to a preset sub-job number threshold, the data query sub-jobs are configured to be executed in parallel by the server extension based on the server extension threads.

[0145] In a possible implementation, the allocation module 302 is further configured to:

[0146] If the number of data query sub-jobs is less than a preset sub-job number threshold, multiple data query sub-jobs are assigned to multiple master control server threads for parallel execution to obtain execution results of the data query sub-jobs.

[0147] In a possible implementation, the database system corresponds to a data query job information table, and the processing module 303 is further used to:

[0148] Record the grouping characteristic values ​​and job distribution conditions corresponding to the data query sub-jobs in the data query job information table;

[0149] The job execution status, error stack, and IP address of the master server of the data query sub-job assigned to each master server thread are recorded in the data query job information table.

[0150] In a possible implementation, the database system corresponds to a data query job information table, which records the job execution status of the data query sub-job assigned to the server extension, the error stack, and the IP address of the server extension.

[0151] The data query device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0152] Figure 4 This is a schematic diagram of the structure of the master control server provided for this application. Figure 4 As shown, the main control server 40 provided in this embodiment includes: at least one processor 401 and a memory 402. Optionally, the main control server 40 also includes a communication component 403. The processor 401, the memory 402 and the communication component 403 are connected via a bus 404.

[0153] In a specific implementation process, at least one processor 401 executes the computer-executable instructions stored in the memory 402, so that at least one processor 401 executes the above method.

[0154] The specific implementation process of the processor 401 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0155] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0156] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0157] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0158] Figure 5 This is a schematic diagram of the structure of the database system provided in this application. Figure 5 As shown, the database system 50 includes: a main control server 40 as provided in the above embodiment, and a plurality of server extensions 501 , wherein the main control server 40 is connected to the plurality of server extensions 501 respectively.

[0159] It should be noted that Figure 5 The number of server extensions 501 shown in the figure is only an example, and the embodiment of the present application does not limit the number of server extensions in the database system.

[0160] The processor of the main control server in the database system provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0161] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0162] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0163] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0164] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0165] The division of modules is only a logical function division. There may be other divisions in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, which can be electrical, mechanical or other forms.

[0166] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0167] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0168] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0169] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0170] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application, are not limited to the precise structures described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data query method, characterized in that: The method is applied to a master control server in a database system, wherein the database system comprises the master control server and a plurality of server extensions, wherein the master control server is connected to the plurality of server extensions respectively; the method comprises: In response to a data query job request, split the data query job into a plurality of data query sub-jobs; If the number of the data query sub-jobs is greater than or equal to a preset sub-job number threshold, the multiple data query sub-jobs are allocated to the multiple server extensions; wherein the multiple data query sub-jobs are used to be executed in parallel by the multiple server extensions to obtain the execution results of the data query sub-jobs; The result of the data query job is obtained according to the execution results of each of the data query sub-jobs.

2. The method according to claim 1, characterized in that The data query job request includes the splitting dimension of the data query job, the splitting strategy of the data query job, and the number of data query sub-jobs; In response to a data query job request, the data query job is split into multiple data query sub-jobs, including: In response to the data query job request, the data query job is split into a plurality of data query sub-jobs according to the splitting dimension of the data query job, the splitting strategy of the data query job, and the number of data query sub-jobs in the data query job request.

3. The method according to claim 2, characterized in that The splitting dimension includes at least one of the following: splitting by one field, splitting by multiple fields, and splitting by the entire row; the splitting strategy includes at least one of the following: hash splitting and range splitting.

4. The method according to claim 1, characterized in that: Allocating the plurality of data query sub-tasks to the plurality of server extensions comprises: Determine a threshold value of the number of extension sub-jobs according to the number of the data query sub-jobs and the number of server extensions; According to the extension sub-job quantity threshold, the multiple data query sub-jobs are allocated to the multiple server extensions; wherein the number of data query sub-jobs allocated to each server extension is less than or equal to the extension sub-job quantity threshold.

5. The method according to claim 1, characterized in that When the number of the data query sub-jobs is greater than or equal to a preset sub-job number threshold, the data query sub-jobs are configured to be executed in parallel by the server extension based on the server extension threads.

6. The method according to claim 1, characterized in that The method further comprises: If the number of the data query sub-jobs is less than a preset sub-job number threshold, the multiple data query sub-jobs are assigned to multiple master control server threads for parallel execution to obtain execution results of the data query sub-jobs.

7. The method according to any one of claims 1 to 6, characterized in that The database system corresponds to a data query operation information table, and the method further includes: Recording the grouping characteristic values ​​and job distribution conditions corresponding to the data query sub-job in the data query job information table; The job execution status, error stack, and IP address of the master control server of the data query sub-job assigned to each master control server thread are recorded in the data query job information table.

8. The method according to any one of claims 1 to 6, characterized in that The database system corresponds to a data query job information table, and the data query job information table records the job execution status of the data query sub-job assigned to the server extension, the error stack, and the IP address of the server extension.

9. A master control server, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 8.

10. A database system, characterized in that: The database system comprises: a main control server as claimed in claim 9, and a plurality of server extensions, wherein the main control server is connected to the plurality of server extensions respectively.

Images