Data processing method and apparatus based on replication group, data query method and apparatus based on replication group, and device and medium

Abstract

The present application relates to a data processing method and apparatus based on a replication group, a data query method and apparatus based on a replication group, and a device and a medium. The data query method comprises: acquiring the name of a target task that a request is made to query, and storage partitions that need to be queried; on the basis of the storage partitions that need to be queried, determining, for a corresponding replication group, n storage partitions that need to be queried, wherein the replication group comprises a data node cluster made up of at least M data servers, which is responsible for reading and writing services of M*N data partitions; and determining data nodes capable of operating normally in the replication group, and allocating query tasks of the n storage partitions to the data nodes capable of operating normally in the replication group, such that the data nodes capable of operating normally perform a data query in the allocated corresponding storage partitions. By means of the present application, parallel queries of a single query task can be performed among all nodes in a replication group, so that the efficiency of a database processing reading requests can be significantly improved.

Description

Data processing and query method, device, equipment and medium based on replication group

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 17, 2023, with application number 202311045285.6, and invention name “Data processing, query method, device, equipment and medium based on replication group”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application is applicable to the field of data processing, and in particular relates to a data processing and query method, apparatus, device and medium based on a replication group. Background Art

[0003] The node-level master-slave distribution rule in the prior art is to store data in the master node (the node refers to the server) and the backup node. When a data query is required for a target task, the usual data query method is to obtain the target task and storage partition to be queried, and then assign all query tasks for all storage partitions to the master node, leaving the other backup nodes idle. The inventors realized that because the master node bears all query work, the data query efficiency is limited. Moreover, when the master node fails, after the master-slave switch is performed, the original backup node will continue to bear all storage partition query work, which still cannot solve the problem of low data query efficiency.

[0004] Therefore, how to improve the query efficiency of existing data query methods has become an urgent problem to be solved.

[0005] Summary of the Invention

[0006] In view of this, embodiments of the present application provide a data processing and query method, apparatus, device, and medium based on a replication group to solve the problem of low query efficiency of existing data query methods.

[0007] In a first aspect, a data query method based on a replication group is provided, the data query method comprising the following steps:

[0008] Get the name of the target task and the storage partition to be queried.

[0009] Determine n storage partitions to be queried for the corresponding replication group based on the storage partition to be queried, where n ≥ 1; the replication group includes a data node cluster consisting of at least M data servers, which is responsible for read and write services for M*N data partitions, where M ≥ 2, N ≥ n ≥ 1, and N is the number of data partitions set in each of the data servers;

[0010] Determine the data nodes in the replication group that can work normally, and assign the query tasks of the n storage partitions to the data nodes in the replication group that can work normally, so that each of the data nodes that can work normally performs data query in the corresponding assigned storage partition.

[0011] In a second aspect, a data processing method based on a replication group is provided, comprising the following steps of a data query method:

[0012] Get the name of the target task and the storage partition to be queried.

[0013] Determine n storage partitions to be queried for the corresponding replication group based on the storage partition to be queried, where n ≥ 1; the replication group includes a data node cluster consisting of at least M data servers, which is responsible for read and write services for M*N data partitions, where M ≥ 2, N ≥ n ≥ 1, and N is the number of data partitions set in each of the data servers;

[0014] Determine the data nodes in the replication group that can work normally, and assign the query tasks of the n storage partitions to the data nodes in the replication group that can work normally, so that each of the data nodes that can work normally performs data query in the assigned corresponding storage partition;

[0015] The data processing method further comprises the following steps of a data writing method:

[0016] Get the name of the target task to be written, as well as the value to be inserted, modified, or deleted during the writing process of the target task;

[0017] According to the name of the target task that needs to be written, determine the replication group that executes the writing of the target task, and send the execution plan for writing the target task to the master node of the corresponding replication group, so that the master node of the corresponding replication group writes the target task according to the execution plan; the master node is a data node responsible for the read service of N data partitions and the write service of all data partitions in the replication group.

[0018] In a third aspect, a data query device based on a replication group is provided, wherein the data query includes:

[0019] The data collection module is used to obtain the name of the target task requested for query and the storage partition to be queried;

[0020] A storage partitioning module is configured to determine n storage partitions to be queried for a corresponding replication group based on the storage partition to be queried, where n ≥ 1; the replication group includes a data node cluster consisting of at least M data servers, which are responsible for read and write services for M*N data partitions, where M ≥ 2 and N ≥ n ≥ 1;

[0021] The task allocation module is used to determine the data nodes that can work normally in the replication group, and allocate the query tasks of n storage partitions to the data nodes that can work normally in the replication group, so that each of the data nodes that can work normally can perform data query in the corresponding allocated storage partition.

[0022] In a fourth aspect, an embodiment of the present application provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps of the data query method are implemented:

[0023] Get the name of the target task and the storage partition to be queried.

[0024] Determine n storage partitions to be queried for the corresponding replication group based on the storage partition to be queried, where n ≥ 1; the replication group includes a data node cluster consisting of at least M data servers, which is responsible for read and write services for M*N data partitions, where M ≥ 2, N ≥ n ≥ 1, and N is the number of data partitions set in each of the data servers;

[0025] Determine the data nodes in the replication group that can work normally, and assign the query tasks of the n storage partitions to the data nodes in the replication group that can work normally, so that each of the data nodes that can work normally performs data query in the corresponding assigned storage partition.

[0026] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the following data query method are implemented:

[0027] Get the name of the target task and the storage partition to be queried.

[0028] Determine n storage partitions to be queried for the corresponding replication group based on the storage partition to be queried, where n ≥ 1; the replication group includes a data node cluster consisting of at least M data servers, which is responsible for read and write services for M*N data partitions, where M ≥ 2, N ≥ n ≥ 1, and N is the number of data partitions set in each of the data servers;

[0029] Determine the data nodes in the replication group that can work normally, and assign the query tasks of the n storage partitions to the data nodes in the replication group that can work normally, so that each of the data nodes that can work normally performs data query in the corresponding assigned storage partition.

[0030] Alternatively, when the computer program is executed by a processor, the steps of the following data processing method are implemented:

[0031] The method includes the steps of a data query method and a data writing method, wherein the steps of the data query method include:

[0032] Get the name of the target task and the storage partition to be queried.

[0033] Determine n storage partitions to be queried for the corresponding replication group based on the storage partition to be queried, where n ≥ 1; the replication group includes a data node cluster consisting of at least M data servers, which is responsible for read and write services for M*N data partitions, where M ≥ 2, N ≥ n ≥ 1, and N is the number of data partitions set in each of the data servers;

[0034] Determine the data nodes in the replication group that can work normally, and assign the query tasks of the n storage partitions to the data nodes in the replication group that can work normally, so that each of the data nodes that can work normally performs data query in the assigned corresponding storage partition;

[0035] The steps of the data writing method include:

[0036] Get the name of the target task to be written, as well as the value to be inserted, modified, or deleted during the writing process of the target task;

[0037] According to the name of the target task that needs to be written, determine the replication group that executes the writing of the target task, and send the execution plan for writing the target task to the master node of the corresponding replication group, so that the master node of the corresponding replication group writes the target task according to the execution plan; the master node is a data node responsible for the read service of N data partitions and the write service of all data partitions in the replication group.

[0038] In a sixth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the following steps of the data query method:

[0039] Get the name of the target task and the storage partition to be queried.

[0040] Determine n storage partitions to be queried for the corresponding replication group based on the storage partition to be queried, where n ≥ 1; the replication group includes a data node cluster consisting of at least M data servers, which is responsible for read and write services for M*N data partitions, where M ≥ 2, N ≥ n ≥ 1, and N is the number of data partitions set in each of the data servers;

[0041] Determine the data nodes in the replication group that can work normally, and assign the query tasks of the n storage partitions to the data nodes in the replication group that can work normally, so that each of the data nodes that can work normally performs data query in the corresponding assigned storage partition.

[0042] Alternatively, the terminal device is caused to perform the following steps of the data processing method:

[0043] The method includes the steps of a data query method and a data writing method, wherein the steps of the data query method include:

[0044] Get the name of the target task and the storage partition to be queried.

[0045] Determine n storage partitions to be queried for the corresponding replication group based on the storage partition to be queried, where n ≥ 1; the replication group includes a data node cluster consisting of at least M data servers, which is responsible for read and write services for M*N data partitions, where M ≥ 2, N ≥ n ≥ 1, and N is the number of data partitions set in each of the data servers;

[0046] Determine the data nodes in the replication group that can work normally, and assign the query tasks of the n storage partitions to the data nodes in the replication group that can work normally, so that each of the data nodes that can work normally performs data query in the assigned corresponding storage partition;

[0047] The steps of the data writing method include:

[0048] Get the name of the target task to be written, as well as the value to be inserted, modified, or deleted during the writing process of the target task;

[0049] According to the name of the target task that needs to be written, determine the replication group that executes the writing of the target task, and send the execution plan for writing the target task to the master node of the corresponding replication group, so that the master node of the corresponding replication group writes the target task according to the execution plan; the master node is a data node responsible for the read service of N data partitions and the write service of all data partitions in the replication group.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] The data processing, query method, device, equipment and medium of the present application, when there is a target task of requesting a query, such as querying a table, after knowing the storage partitions that may be involved in storing the table, the query task of the storage partition can be distributed to all data nodes that can work normally in all replication groups as much as possible, so that a single query task can be queried in parallel among all nodes in the replication group, which can significantly improve the efficiency of processing read requests of the database. In addition, the present application is based on the data query of the existing node resources of the replication group, ensuring high data query efficiency while not requiring additional computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0053] FIG1 is a schematic diagram of an application environment of a data query method based on a replication group provided by an embodiment of the present application;

[0054] FIG2 is a flow chart of a data query method based on a replication group according to an embodiment of the present application;

[0055] FIG2-1 is a schematic diagram of a replication group structure provided in an embodiment of the present application;

[0056] 3 is a schematic diagram of a process for determining n storage partitions to be queried for a corresponding replication group according to an embodiment of the present application;

[0057] FIG4 is a schematic diagram of a process for allocating query tasks of n storage partitions to data nodes according to an embodiment of the present application;

[0058] FIG5 is a flow chart of another data query method provided in an embodiment of the present application;

[0059] FIG6 is a flow chart of another data query method provided in an embodiment of the present application;

[0060] FIG6-1 is a schematic diagram of a replication group structure with unavailable data nodes provided by an embodiment of the present application;

[0061] 7 is a flow chart of a data processing method based on a replication group according to an embodiment of the present application;

[0062] FIG8 is a flow chart of another data processing method based on replication groups provided in an embodiment of the present application;

[0063] FIG9 is a schematic structural diagram of a data query device provided by an embodiment of the present application;

[0064] FIG10 is a schematic structural diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0066] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0067] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0068] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0069] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0070] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0071] It should be understood that the size of the serial numbers of the steps in the following embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0072] In order to illustrate the technical solution of the present application, specific embodiments are provided below.

[0073] The first embodiment of the present application provides a replication group-based data query method that can be applied in an application environment such as Figure 1 , wherein a client communicates with a server. The client includes, but is not limited to, a palmtop computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cloud terminal device, a personal digital assistant (PDA), and other terminal devices. The server can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0074] Referring to FIG2 , which is a flow chart of a data query method based on a replication group provided by one embodiment of the present application, the data query method can be applied to the server in FIG1 . The computer device corresponding to the server is connected to the corresponding database, rule base, etc. to obtain the corresponding data in the database. The computer device can also be connected to the client to collect data and control instructions sent by the client user. As shown in FIG2 , the data query method can include the following steps:

[0075] S201, obtaining the name of the target task requested for query and the storage partition to be queried.

[0076] The target task of the query request may be a distribution table, and the data in the distribution table has been stored in advance in the storage partitions of each data node in one or several replication groups.

[0077] In this step, the client (CLENT) parses the DQL statement to obtain the name of the target task requested and the storage partition to be queried, and then sends it to the routing server (ROUTER) connected to each replication group.

[0078] S202, according to the storage partitions that need to be queried, determine the n storage partitions that need to be queried for the corresponding replication group, where n≥1; the replication group includes a data node cluster consisting of at least M data servers, which is responsible for the read and write services of M*N data partitions, where M≥2, N≥n≥1, and N is the number of data partitions set in each of the data servers.

[0079] The query target task provided by the client may involve several storage partitions, each of which is maintained by different replication groups. As shown in Figure 2-1, there are three replication groups: Group 1, Group 2, and Group 3. Each replication group includes a data node cluster consisting of three data servers, M = 3, and each data node has four storage partitions, N = 4. Taking Group 1 as an example, it includes data nodes node 1-1, node 1-2, and node 1-3. Data node 1-1 has four storage partitions: p1, p2, p3, and p4; data node 1-2 has four storage partitions: p1-r, p2-r, p3-r, and p4-r; and data node 1-3 has four storage partitions: p1-r, p2-r, p3-r, and p4-r.

[0080] As shown in Figure 2-1, the routing server (ROUTER) obtains the 12 storage partitions that need to be queried from the client (CLENT). Then, based on the specific 12 storage partitions that need to be queried, the routing server (ROUTER) can determine the n storage partitions that each replication group needs to query, where n = 4. That is, the first replication group Group1 needs to query 4 storage partitions, the second replication group Group2 needs to query 4 storage partitions, and the third replication group Group1 needs to query 4 storage partitions.

[0081] S203, determining the data nodes in the replication group that can work normally, and assigning the query tasks of the n storage partitions to the data nodes in the replication group that can work normally, so that each of the data nodes that can work normally performs data query in the assigned corresponding storage partition.

[0082] When all data nodes within each replication group are functioning properly, query tasks for n storage partitions are distributed to all data nodes within each replication group. For example, as shown in Figure 2-1, the routing server (ROUTER) detects that each replication group has three functioning data nodes. Therefore, query tasks within each replication group are redistributed among the data nodes to fully utilize the computing resources of each data node. To maximize balance, two data nodes each scan one storage partition, while another data node scans the remaining two partitions. This allows all nodes to be utilized for parallel querying.

[0083] Specifically, the query allocation rules for the first replication group Group1 that needs to query 4 storage partitions are: assign the data node node1-1 in the first replication group to perform the query task of storage partitions p1 and p2, assign the data node node1-2 in the first replication group to perform the query task of storage partition p3-r, and assign the data node node1-3 in the first replication group to perform the query task of storage partition p4-r.

[0084] The query allocation rule for the second replication group Group2 that needs to query 4 storage partitions is: assign the query task of storage partition p5 to data node node2-1 in the second replication group, assign the query task of storage partitions p6-r and p7-r to data node node2-2 in the second replication group, and assign the query task of storage partition p8-r to data node node2-3 in the second replication group.

[0085] The query allocation rule for the third replication group Group1 that needs to query 4 storage partitions is: assign the query task of storage partition p9 to data node node3-1 in the third replication group, assign the query task of storage partition p10-r to data node node3-2 in the second replication group, and assign the query task of storage partitions p11-r and p12-r to data node node3-3 in the second replication group.

[0086] In other implementations, query tasks for storage partition p1 may be assigned only to data node node1-1 within the first replication group, while query tasks for storage partition p2 may be assigned to data node node1-2 or data node node1-3. Similarly, query tasks for storage partition p6-r may be assigned only to data node node2-2 within the second replication group, while query tasks for storage partition p7-r may be assigned to data node node2-1 or data node node2-3. Similarly, query tasks for storage partition p11-r may be assigned only to data node node3-3 within the second replication group, while query tasks for storage partition p12-r may be assigned to data node node3-1 or data node node3-2.

[0087] In the above scenario of assigning query tasks to 12 storage partitions, as another implementation, when the target tasks to be queried are different, the number of storage partitions involved in the target tasks is different. For example, when the routing server (ROUTER) obtains from the client (CLENT) that it needs to query n storage partitions, n = 1, 2, ..., 9, it can assign a query task for each storage partition to each of the three data nodes in the three replication groups; when n = 10 or 11, it can first assign a query task for each storage partition to each of the three data nodes in the three replication groups, and then randomly assign the query tasks for the remaining one or two storage partitions to any two data nodes.

[0088] The data query method of this embodiment fully utilizes the readability of all data nodes in the replication group. When there is a target task to request a query, such as querying a table, after knowing the several storage partitions that may be involved in storing the table, it is equivalent to setting query tasks for several storage partitions. The query tasks of all storage partitions can be distributed as much as possible to all data nodes that can work normally in all replication groups, so that a single query task can be queried in parallel among all nodes in the replication group, which can significantly improve the efficiency of the database in processing read requests. In addition, this application is based on the data query of the existing node resources of the replication group, which ensures high data query efficiency without the need for additional computing resources.

[0089] In one embodiment, as shown in FIG3 , in step S202 , determining n storage partitions to be queried for a corresponding replication group according to the storage partition to be queried includes:

[0090] S301, obtaining N1 storage partitions to be queried and the query filter condition of the target task.

[0091] The query filter condition for the target task can be the partition key value of a distribution table. For example, a partitioned table partTab1 uses col1 as the partition key and uses range partitioning (i.e., each partition contains data whose partition expression value falls within a given range). Data with the partition key col1 in the range [0, 100] belongs to partition 1, and data with the remaining range belongs to partition 2. The query filter condition for this partitioned table partTab1 can be col1 = 99.

[0092] S302 : Select N2 storage partitions that meet the query filter condition from the N1 storage partitions, where N1 ≥ N2.

[0093] When using the partition key value as the filter condition, it is possible to calculate whether the filtered range falls within the partition key range of some partitions. For example, if the query filter condition for partition table partTab1 is col1 = 99, the partitioning algorithm can calculate that the value 99 belongs to partition 1.

[0094] In this step, by setting the query filter condition, the number of N1 storage partitions to be queried can be reduced to N2 storage partitions, thereby narrowing the query scope of the storage partitions.

[0095] S303 , among the N2 storage partitions, determine the replication group responsible for querying each storage partition, thereby determining n storage partitions that each replication group needs to query, where N2>n.

[0096] The small number of N2 storage partitions determined after filtering by the query filter criteria can be evenly distributed to all functioning data nodes in the corresponding replication group. For example, when N2 = 2, the query task for one storage partition can be assigned to the functioning data node in the first replication group in Figure 2-1, and the query task for the other storage partition can be assigned to the functioning data node in the second replication group in Figure 2-1.

[0097] According to the preset query filtering conditions, the embodiment of the present application can reduce the number of storage partitions that need to be queried. Only a small number of query tasks of storage partitions need to be sent to the corresponding data nodes to obtain the correct results, thereby realizing partition pruning and improving data query efficiency to a certain extent.

[0098] In one embodiment, referring to FIG. 4 , in step S203 , determining the data nodes in the replication group that can operate normally, and assigning the query tasks of the n storage partitions to the data nodes in the replication group that can operate normally includes:

[0099] S401 , obtaining the current status identifier of each data node in the replication group, and identifying m data nodes whose current status is identified as available as data nodes that can work normally in the replication group, where m≤M.

[0100] The routing server (ROUTER) records the status of all data nodes, which are categorized as available or unavailable. One method for changing the status includes: the backend periodically sends messages to each data node. Upon confirming the data node's survival, the status of the corresponding data node is updated based on the received message. Another method for changing the status includes: after sending a service message to a data node, if an error message indicating a network anomaly or the data node does not exist is returned, the data node's status is changed to unavailable.

[0101] S402, when n / m is a positive integer ratio, evenly distribute the query tasks of the n storage partitions to the m data nodes, and each data node is allocated n / m query tasks of the storage partitions.

[0102] For example, when n=3 and m=3, the query tasks of the three storage partitions are evenly distributed to the three data nodes, that is, each data node is allocated a query task of one storage partition.

[0103] The data query method of this embodiment first checks the availability status of all data nodes when querying the distribution table, and then evenly distributes the query tasks of each storage partition to all available data nodes for query, so as to ensure the reliability of data query.

[0104] In one embodiment, as shown in FIG5 , after determining m data nodes whose current status is marked as available in step S401, the data query method further includes:

[0105] S501, when n / m is a non-positive integer ratio, among the query tasks of the n storage partitions, select a preset number k of query tasks of the storage partitions in turn and assign them to the m data nodes, and the preset number k is an integer value obtained by rounding the non-positive integer ratio n / m.

[0106] For example, when n=4 and m=3, n / m is a non-positive integer ratio. Then, among the query tasks of the four storage partitions, the query task of one storage partition is selected in turn and assigned to each data node.

[0107] S502: Randomly select (nm*k) data nodes from the m data nodes, and assign the query tasks of the remaining (nm*k) storage partitions to the (nm*k) data nodes in a one-to-one correspondence.

[0108] For example, when n=4, m=3, and k=1, after selecting the query task of one storage partition and assigning it to each data node in turn, one data node is randomly selected from the three data nodes, and the query task of the remaining storage partition is assigned one-to-one to the randomly selected data node.

[0109] The data query method of this embodiment distributes the query tasks of each storage partition as evenly as possible to all available data nodes for query, so that the number of query tasks of the storage partition allocated to each data node is roughly equal (that is, the deviation in the number of tasks allocated to the storage partition is small). The query tasks can be dispersed to different data nodes, avoiding the problem of low resource utilization of data nodes in this replication group, thereby improving resource utilization in a distributed database system.

[0110] In one embodiment, as shown in FIG6 , after step S203 , the data query method further includes the following steps:

[0111] S601 , when each of the data nodes performs a data query in the corresponding allocated storage partition, the current status identifier of the data node is detected in real time.

[0112] For example, during the detection process of a data node changing from an available state to an unavailable state and then to an available state again, the routing server (ROUTER) will first send a business message to this data node. After returning the execution failure, it is detected that the data node is unavailable and the status mark is immediately modified to an unavailable state. Subsequent business messages will not be sent to the data node. Instead, the status of the data node will be checked regularly through the background thread. When the data node status becomes available, the background thread can detect that the data node has recovered to an available state and immediately update the node status to an available state. Subsequently, business messages can continue to be sent to the data node for execution.

[0113] S602, when it is detected that the current status identifier of a data node changes from an available state to an unavailable state, re-determine the data nodes that can work normally in the replication group, and assign the query tasks of the n storage partitions to the data nodes that can work normally in the replication group.

[0114] For example, as shown in Figure 6-1, during the query process of a target task (such as the T1 distribution table), it is necessary to query all storage partitions of the T1 distribution table. According to the above steps S201 to S203, the query tasks of 12 storage partitions are assigned to the data nodes that can work normally in each replication group, that is, it is necessary to use the data nodes on 3 replication groups for query, and the data nodes of each replication group need to scan 4 storage partitions.

[0115] Then, according to the above step S601, the routing server (ROUTER) detects that each replication group has 3 nodes, but the current status identifiers of some of the data nodes are unavailable. For example, the status identifiers of the data node node1-3 of the first replication group Group1, the data node node2-2 of the second replication group Group2, and the data node node3-2 and data node node3-3 of the third replication group Group3 all become unavailable. Therefore, when executing this step, the partitions that need to be scanned by each replication group are evenly distributed to the available data nodes in the group.

[0116] For example, as shown in Figure 6-1, the first replication group Group1 and the second replication group Group2 each have two available data nodes, node1-1, node1-2, node2-1, and node2-3. Therefore, query tasks for two storage partitions are assigned to each available data node. The third replication group Group3 has only one available data node, node3-1, so query tasks for the four storage partitions in the replication group are all assigned to this data node.

[0117] In the data query method of this embodiment, when each data node in each replication group concurrently queries the distribution table, if a data node within a replication group fails, the remaining data nodes within the replication group will share the query task for the storage partition responsible for the failed data node without any impact on other replication groups. This improves resource utilization and ensures overall fault tolerance. Furthermore, as long as at least one data node in each replication group is available, the query for the target task can be executed on the currently available node reselected within the replication group without error.

[0118] This application can maintain the status of all data nodes on all replication groups, and dynamically generate a query execution plan on how to allocate storage partitions to the corresponding data nodes based on the status identifiers of the data nodes, maximize the use of the capabilities of the data nodes in the available state, and try to ensure that the execution of query tasks does not report errors due to failures of individual data nodes, thereby improving the availability of the database and achieving successful execution of query tasks.

[0119] In one embodiment, referring to FIG7 , a flow chart of a data processing method based on a replication group is provided. The data processing method may include a data query step and a data writing step, wherein the data query step includes:

[0120] Step S701: Obtain the name of the target task requested for query and the storage partition to be queried;

[0121] Step S702: Based on the storage partition to be queried, n storage partitions to be queried are determined for the corresponding replication group, where n≥1; the replication group includes a data node cluster consisting of at least M data servers, which are responsible for read and write services for M*N data partitions, where M≥2, N≥n≥1, and N is the number of data partitions set in each of the data servers;

[0122] Step S703, determine the data nodes that can work normally in the replication group, and assign the query tasks of the n storage partitions to the data nodes that can work normally in the replication group, so that each of the data nodes that can work normally can perform data query in the corresponding assigned storage partition.

[0123] Among them, the contents of steps S701 to S703 are the same as those of the above-mentioned steps S201 to S203. Please refer to the description of steps S701 to S703 and will not be repeated here.

[0124] The data writing step in the data processing method includes:

[0125] Step S704: Obtain the name of the target task to be written and the value to be inserted, modified or deleted during the process of writing the target task;

[0126] The DML statement is parsed by the client (CLENT) to obtain the name of the target task requested for query and the value to be inserted, modified or deleted, and then sent to the routing server (ROUTER) connected to each replication group.

[0127] Step S705: Determine the replication group that executes the writing of the target task according to the name of the target task that needs to be written, and send the execution plan for writing the target task to the master node of the corresponding replication group, so that the master node of the corresponding replication group writes the target task according to the execution plan; the master node is a data node responsible for the read service of N data partitions and the write service of all data partitions in the replication group.

[0128] In this embodiment, the data nodes of the replication group are pre-divided into primary nodes and backup nodes. Specifically, one data server is selected from the data node cluster of each replication group as the primary node, and the remaining servers are selected as backup nodes. The primary node is responsible for writing M*N data partitions, and the primary node and all backup nodes are responsible for reading N partitions.

[0129] In the data processing method of this embodiment, the pre-selected master node is responsible for writing all target tasks, and other backup nodes do not need to be responsible for writing target tasks.

[0130] In one embodiment, as shown in FIG8 , after step S705, the data processing method further includes the following steps:

[0131] Step S801: When it is detected that a node failure occurs in the primary node of the replication group during the process of writing the target task, a data node is reselected from each backup node in the replication group as the primary node.

[0132] The state identifier of the master node is obtained in real time to determine whether the master node has a node failure. Specifically, when the state identifier of the master node is detected as available, it is determined that the master node has not experienced a node failure; when the state identifier of the master node is detected as unavailable, it is determined that the master node has experienced a node failure.

[0133] Step S802 : Sending the execution plan for writing the target task to the master node re-elected in the replication group, so that the master node re-elected in the replication group writes the target task according to the execution plan.

[0134] The rule for re-election of the master node within the replication group is as follows: Based on the Raft protocol adopted within the replication group, a standby node with the latest data synchronization is elected within the replication group as the new master node.

[0135] The data processing method of this embodiment relies on the high availability capability of target task writing to the master node and the standby node switching capability. By real-time detection of the status of the master node, it is possible to ensure that when a master node failure occurs, a new master node can be quickly selected from the standby nodes, and the target task writing can be successfully executed.

[0136] Corresponding to the data query method in the above embodiment, FIG9 shows a block diagram of a data query device based on a replication group provided in one embodiment of the present application. The data query device is applied to a terminal device, which connects to the target database via a preset application program interface. When the target database is driven to execute a corresponding task, a corresponding task log is generated, which can be collected through the API. For ease of explanation, only the portion relevant to the embodiment of the present application is shown.

[0137] Referring to FIG9 , the data query device includes:

[0138] The data acquisition module 91 is used to obtain the name of the target task requested for query and the storage partition to be queried;

[0139] The storage partition module 92 is configured to determine n storage partitions to be queried for the corresponding replication group based on the storage partition to be queried, where n≥1; the replication group includes a data node cluster consisting of at least M data servers, which are responsible for read and write services for M*N data partitions, where M≥2 and N≥n≥1;

[0140] The first task assignment module 93 is used to determine the data nodes that can work normally in the replication group, and assign the query tasks of n storage partitions to the data nodes that can work normally in the replication group, so that each of the data nodes that can work normally can perform data query in the corresponding assigned storage partition.

[0141] Optionally, the storage partition module includes:

[0142] A collection unit, configured to obtain N1 storage partitions to be queried and query filter conditions for the target task;

[0143] A filtering unit, configured to select N2 storage partitions that meet the query filtering condition from the N1 storage partitions, where N1 ≥ N2;

[0144] The replication group query unit is used to determine the replication group responsible for querying each storage partition among the N2 storage partitions, thereby determining the n storage partitions that each replication group needs to query, N2>n.

[0145] Optionally, the first task assignment module includes:

[0146] a node identification unit, configured to obtain a current status identifier of each data node in the replication group, and identify m data nodes whose current status is identified as available as data nodes that can normally operate in the replication group, where m≤M;

[0147] The first node allocation unit is used to evenly allocate the query tasks of the n storage partitions to the m data nodes when n / m is a positive integer ratio, and each data node is allocated n / m query tasks of the storage partitions.

[0148] Optionally, the first task assignment module further includes:

[0149] A second node allocation unit is configured to, when n / m is a non-positive integer ratio, sequentially select query tasks of a preset number k of the storage partitions from the query tasks of the n storage partitions and allocate them to the m data nodes, where the preset number k is an integer value obtained by rounding the non-positive integer ratio n / m;

[0150] The third node allocation unit randomly selects (nm*k) data nodes from the m data nodes, and allocates the query tasks of the remaining (nm*k) storage partitions to the (nm*k) data nodes one by one.

[0151] Optionally, the data query device further includes:

[0152] A status detection module is used to detect the current status identifier of the data node in real time during the data query process of each data node in the corresponding allocated storage partition;

[0153] The second task allocation module is used to re-determine the data nodes that can work normally in the replication group when it is detected that the current status identifier of the data node changes from an available state to an unavailable state, and allocate the query tasks of the n storage partitions to the data nodes that can work normally in the replication group.

[0154] In one embodiment, corresponding to the data query method of the above embodiment, a data processing device based on a replication group is further provided, comprising:

[0155] The first data acquisition module is used to obtain the name of the target task requested for query and the storage partition to be queried;

[0156] A storage partitioning module is configured to determine n storage partitions to be queried for a corresponding replication group based on the storage partition to be queried, where n ≥ 1; the replication group includes a data node cluster consisting of at least M data servers, which are responsible for read and write services for M*N data partitions, where M ≥ 2 and N ≥ n ≥ 1;

[0157] The query task allocation module is used to determine the data nodes that can work normally in the replication group, and allocate the query tasks of n storage partitions to the data nodes that can work normally in the replication group, so that each of the data nodes that can work normally can perform data query in the corresponding allocated storage partition.

[0158] The second data acquisition module is used to obtain the name of the target task to be written and the values ​​to be inserted, modified or deleted during the process of writing the target task;

[0159] A write task allocation module is used to determine the replication group that executes the writing of the target task according to the name of the target task that needs to be written, and send the execution plan for writing the target task to the master node of the corresponding replication group, so that the master node of the corresponding replication group writes the target task according to the execution plan; the master node is a data node responsible for the read service of N data partitions and the write service of all data partitions in the replication group.

[0160] Optionally, the data processing device further includes:

[0161] The fault detection module is used to reselect a data node from each backup node in the replication group as the master node when it is detected that a node failure occurs in the master node of the replication group during the execution of writing the target task.

[0162] The master node switching module is used to send the execution plan for writing the target task to the master node re-elected in the replication group, so that the master node re-elected in the replication group writes the target task according to the execution plan.

[0163] It should be noted that the information interaction, execution process and other contents between the above modules are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0164] Figure 10 is a schematic diagram of the structure of a terminal device provided in one embodiment of the present application. As shown in Figure 10, the terminal device of this embodiment includes: at least one processor (only one is shown in Figure 10), a memory, and a computer program stored in the memory and executable by the at least one processor. When the processor executes the computer program, it implements the steps of any of the above-mentioned data query method or data processing method embodiments.

[0165] The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that FIG5 is merely an example of a terminal device and does not limit the terminal device. The terminal device may include more or fewer components than shown in the figure, or may combine certain components or different components. For example, it may also include a network interface, a display screen, and an input device.

[0166] The processor may be a CPU, other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or any conventional processor, etc.

[0167] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be the internal memory of the terminal device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium can be the hard disk of the terminal device. In other embodiments, it can also be an external storage device of the terminal device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. equipped with the terminal device. Furthermore, the memory can also include both the internal storage unit of the terminal device and an external storage device. The memory is used to store the operating system, application programs, boot loaders (BootLoader), data, and other programs, such as the program code of computer programs. The memory can also be used to temporarily store data that has been output or is about to be output.

[0168] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0169] In one embodiment, one or more readable storage media storing computer-readable instructions are provided, and the computer-readable storage media stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors implement the steps of the data query method or data processing method in the above-mentioned embodiment, such as steps S201-S203 shown in Figure 2, or Figures 3 to 6, and the steps shown in Figures 7 and 8. To avoid repetition, they are not repeated here. Alternatively, when the processor executes the readable storage medium, the functions of each module / unit in this embodiment of the data query device are implemented, such as the functions of the data acquisition module 91, the storage partition module 92, and the first task allocation module 93 shown in Figure 9. To avoid repetition, they are not repeated here. The readable storage medium in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0170] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a readable storage medium, and the readable storage medium can be stored in a non-volatile computer-readable storage medium. When the readable storage medium is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0171] The present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed through a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiment when executing it.

[0172] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0173] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0174] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal devices and methods can be implemented in other ways. For example, the device / terminal device embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0175] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0176] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. Data query method based on replication group, where: The data query method comprises the following steps: Get the name of the target task for the query request and the storage partition to be queried; According to the storage partition to be queried, n storage partitions to be queried are determined for the corresponding replication group, where n≥1; the replication group includes a data node cluster consisting of at least M data servers, which is responsible for the read and write services of M*N data partitions, where M≥2, N≥n≥1, and N is the number of data partitions set in each of the data servers; Determine the data nodes in the replication group that can work normally, and assign the query tasks of the n storage partitions to the data nodes in the replication group that can work normally, so that each of the data nodes that can work normally performs data query in the corresponding assigned storage partition.

2. The data query method based on replication group according to claim 1, wherein: According to the storage partition to be queried, determining n storage partitions to be queried for the corresponding replication group includes: Obtain N1 storage partitions to be queried and the query filter conditions of the target task; Among the N1 storage partitions, select N2 storage partitions that meet the query filtering condition, N1≥N2; Among the N2 storage partitions, the replication group responsible for querying each storage partition is determined, thereby determining the n storage partitions that each replication group needs to query, N2>n.

3. The data query method based on replication group according to claim 1, wherein: Determining the data nodes in the replication group that can work normally, and assigning the query tasks of the n storage partitions to the data nodes in the replication group that can work normally includes: Obtaining the current status identifier of each data node in the replication group, and identifying m data nodes whose current status is identified as available as data nodes that can work normally in the replication group, m≤M; When n / m is a positive integer ratio, the query tasks of the n storage partitions are evenly distributed to the m data nodes, and each of the data nodes is allocated n / m query tasks of the storage partitions.

4. The data query method based on replication group according to claim 3, wherein: After determining the m data nodes whose current status is marked as being in an available state, the method further includes: When n / m is a non-positive integer ratio, among the query tasks of the n storage partitions, a preset number k of query tasks of the storage partitions are selected in turn and assigned to the m data nodes, where the preset number k is an integer value obtained by rounding the non-positive integer ratio n / m; Randomly select (nm*k) data nodes from the m data nodes, and assign the query tasks of the remaining (nm*k) storage partitions to the (nm*k) data nodes one by one.

5. The data query method based on replication group according to claim 1, wherein: The data query method further includes: During the process of each of the data nodes performing data query in the corresponding allocated storage partition, real-time detection of the current status identifier of the data node; When it is detected that the current status identifier of a data node changes from an available state to an unavailable state, the data nodes that can work normally in the replication group are re-determined, and the query tasks of the n storage partitions are assigned to the data nodes that can work normally in the replication group.

6. A data processing method based on a replication group, wherein: The data processing method comprises the following steps of a data query method: Get the name of the target task for the query request and the storage partition to be queried; According to the storage partition to be queried, n storage partitions to be queried are determined for the corresponding replication group, where n≥1; the replication group includes a data node cluster consisting of at least M data servers, which is responsible for the read and write services of M*N data partitions, where M≥2, N≥n≥1, and N is the number of data partitions set in each of the data servers; Determine the data nodes in the replication group that can work normally, and assign the query tasks of the n storage partitions to the data nodes in the replication group that can work normally, so that each of the data nodes that can work normally performs data query in the assigned corresponding storage partition; The data processing method also includes the following steps of a data writing method: Get the name of the target task to be written, as well as the value to be inserted, modified or deleted during the process of writing the target task; According to the name of the target task that needs to be written, determine the replication group that executes the writing of the target task, and send the execution plan for writing the target task to the master node of the corresponding replication group, so that the master node of the corresponding replication group writes the target task according to the execution plan; the master node is a data node responsible for the read service of N data partitions and the write service of all data partitions in the replication group.

7. The data processing method based on replication groups according to claim 6, wherein: The data processing method further includes: When it is detected that a node failure occurs in the master node of the replication group during the process of writing the target task, a data node is reselected from each standby node in the replication group as the master node; The execution plan for writing the target task is sent to the master node re-elected in the replication group, so that the master node re-elected in the replication group writes the target task according to the execution plan.

8. The data processing method based on replication groups according to claim 6, wherein: According to the storage partition to be queried, determining n storage partitions to be queried for the corresponding replication group includes: Obtain N1 storage partitions to be queried and the query filter conditions of the target task; Among the N1 storage partitions, select N2 storage partitions that meet the query filtering condition, N1≥N2; Among the N2 storage partitions, the replication group responsible for querying each storage partition is determined, thereby determining the n storage partitions that each replication group needs to query, N2>n.

9. The data processing method based on replication groups according to claim 6, wherein: Determining the data nodes in the replication group that can work normally, and assigning the query tasks of the n storage partitions to the data nodes in the replication group that can work normally includes: Obtaining the current status identifier of each data node in the replication group, and identifying m data nodes whose current status is identified as available as data nodes that can work normally in the replication group, m≤M; When n / m is a positive integer ratio, the query tasks of the n storage partitions are evenly distributed to the m data nodes, and each of the data nodes is allocated n / m query tasks of the storage partitions.

10. The data processing method based on replication groups according to claim 9, wherein: After determining the m data nodes whose current status is marked as being in an available state, the method further includes: When n / m is a non-positive integer ratio, among the query tasks of the n storage partitions, a preset number k of query tasks of the storage partitions are selected in turn and assigned to the m data nodes, where the preset number k is an integer value obtained by rounding the non-positive integer ratio n / m; Randomly select (nm*k) data nodes from the m data nodes, and assign the query tasks of the remaining (nm*k) storage partitions to the (nm*k) data nodes one by one.

11. The data processing method based on replication groups according to claim 6, wherein: The data query method further comprises the following steps: During the process of each of the data nodes performing data query in the corresponding allocated storage partition, real-time detection of the current status identifier of the data node; When it is detected that the current status identifier of a data node changes from an available state to an unavailable state, the data nodes that can work normally in the replication group are re-determined, and the query tasks of the n storage partitions are assigned to the data nodes that can work normally in the replication group.

12. A data query device based on a replication group, wherein: The data query device comprises: The data collection module is used to obtain the name of the target task requested for query and the storage partition to be queried; A storage partition module is used to determine n storage partitions to be queried for a corresponding replication group according to the storage partition to be queried, where n≥1; the replication group includes a data node cluster consisting of at least M data servers, which is responsible for the read and write services of M*N data partitions, where M≥2, N≥n≥1; The task allocation module is used to determine the data nodes that can work normally in the replication group, and allocate the query tasks of the n storage partitions to the data nodes that can work normally in the replication group, so that each of the data nodes that can work normally can perform data query in the corresponding allocated storage partition.

13. A terminal device, wherein: The terminal device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the following steps of the data query method when executing the computer program: Get the name of the target task for the query request and the storage partition to be queried; According to the storage partition to be queried, n storage partitions to be queried are determined for the corresponding replication group, where n≥1; the replication group includes a data node cluster consisting of at least M data servers, which is responsible for the read and write services of M*N data partitions, where M≥2, N≥n≥1, and N is the number of data partitions set in each of the data servers; Determine the data nodes in the replication group that can work normally, and assign the query tasks of the n storage partitions to the data nodes in the replication group that can work normally, so that each of the data nodes that can work normally performs data query in the corresponding assigned storage partition.

14. The terminal device according to claim 13, wherein: According to the storage partition to be queried, determining n storage partitions to be queried for the corresponding replication group includes: Obtain N1 storage partitions to be queried and the query filter conditions of the target task; Among the N1 storage partitions, select N2 storage partitions that meet the query filtering condition, N1≥N2; Among the N2 storage partitions, the replication group responsible for querying each storage partition is determined, thereby determining the n storage partitions that each replication group needs to query, N2>n.

15. The terminal device according to claim 13, wherein: Determining the data nodes in the replication group that can work normally, and assigning the query tasks of the n storage partitions to the data nodes in the replication group that can work normally includes: Obtaining the current status identifier of each data node in the replication group, and identifying m data nodes whose current status is identified as available as data nodes that can work normally in the replication group, m≤M; When n / m is a positive integer ratio, the query tasks of the n storage partitions are evenly distributed to the m data nodes, and each of the data nodes is allocated n / m query tasks of the storage partitions.

16. The terminal device according to claim 14, wherein: After determining the m data nodes whose current status is marked as being in an available state, the method further includes: When n / m is a non-positive integer ratio, among the query tasks of the n storage partitions, a preset number k of query tasks of the storage partitions are selected in turn and assigned to the m data nodes, where the preset number k is an integer value obtained by rounding the non-positive integer ratio n / m; Randomly select (nm*k) data nodes from the m data nodes, and assign the query tasks of the remaining (nm*k) storage partitions to the (nm*k) data nodes one by one.

17. The terminal device according to claim 13, wherein: When the processor executes the computer program, it is also used to implement the following steps of the data query method: During the process of each of the data nodes performing data query in the corresponding allocated storage partition, real-time detection of the current status identifier of the data node; When it is detected that the current status identifier of a data node changes from an available state to an unavailable state, the data nodes that can work normally in the replication group are re-determined, and the query tasks of the n storage partitions are assigned to the data nodes that can work normally in the replication group.

18. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, the following steps of the data query method are implemented: Get the name of the target task for the query request and the storage partition to be queried; According to the storage partition to be queried, n storage partitions to be queried are determined for the corresponding replication group, where n≥1; the replication group includes a data node cluster consisting of at least M data servers, which is responsible for the read and write services of M*N data partitions, where M≥2, N≥n≥1, and N is the number of data partitions set in each of the data servers; Determine the data nodes in the replication group that can work normally, and assign the query tasks of the n storage partitions to the data nodes in the replication group that can work normally, so that each of the data nodes that can work normally performs data query in the corresponding assigned storage partition.

19. The computer-readable storage medium of claim 18, wherein: According to the storage partition to be queried, determining n storage partitions to be queried for the corresponding replication group includes: Obtain N1 storage partitions to be queried and the query filter conditions of the target task; Among the N1 storage partitions, select N2 storage partitions that meet the query filtering condition, N1≥N2; Among the N2 storage partitions, the replication group responsible for querying each storage partition is determined, thereby determining the n storage partitions that each replication group needs to query, N2>n.

20. The computer-readable storage medium of claim 18, wherein: Determining the data nodes in the replication group that can work normally, and assigning the query tasks of the n storage partitions to the data nodes in the replication group that can work normally includes: Obtaining the current status identifier of each data node in the replication group, and identifying m data nodes whose current status is identified as available as data nodes that can work normally in the replication group, m≤M; When n / m is a positive integer ratio, the query tasks of the n storage partitions are evenly distributed to the m data nodes, and each of the data nodes is allocated n / m query tasks of the storage partitions.

Images