Distributed system data query method, electronic equipment and readable storage medium

Abstract

The invention discloses a distributed system data query method, electronic equipment and a readable storage medium. The method comprises the steps that the distribution condition of all data fragments divided by a data set on nodes in a distributed system is acquired; the data fragments comprise a main fragment and an auxiliary fragment which are distributed at different nodes; based on the distribution condition, obtaining a query path of the queried data fragments, and obtaining a node selection number of each node in the query path; the query path corresponds to a main fragment and a node corresponding to the main fragment or an auxiliary fragment and a node corresponding to the auxiliary fragment; on the basis of the query path and the node selection number, adjusting the query path of at least part of the data fragments which can be queried and generating the query path of at least part of the data fragments which are not queried, so as to obtain a target path for querying at least part of the data fragments; and querying the data fragments included in the data set from the main fragments or the auxiliary fragments on the corresponding nodes according to the target path. According to the scheme, the consistency of query data of the distributed system and the balance of node utilization can be improved.

Description

Technical Field

[0001] This application relates to the field of data query technology, and in particular to a distributed system data query method, electronic device, and readable storage medium. Background Technology

[0002] With the significant increase in data storage demands, distributed storage systems, characterized by their excellent scalability, high availability, and fault tolerance, have been widely adopted. These systems divide datasets into multiple smaller data shards and distribute these shards across different nodes to achieve data storage. However, inconsistencies still exist between stored data shards and queries, and achieving balanced utilization of nodes during large-scale data queries remains challenging. Therefore, improving the consistency of data queries and the balanced utilization of nodes in distributed systems has become a pressing issue. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide a distributed system data query method, electronic device, and readable storage medium that can improve the consistency of data query in a distributed system and the balance of node utilization.

[0004] To address the aforementioned technical problems, this application provides a first aspect of a distributed system data query method, comprising: obtaining the distribution status of all data shards of a dataset on nodes in a distributed system; wherein the data shards include primary shards and secondary shards, and the primary shards and the secondary shards are distributed on different nodes; based on the distribution status, obtaining query paths for queryable data shards, and obtaining the number of selected nodes for each node in the query path; wherein the query path corresponds to the primary shard and its corresponding node or the secondary shard and its corresponding node; based on the query path and the number of selected nodes, adjusting the query paths for at least some queryable data shards and generating query paths for at least some unqueried data shards, thereby obtaining a target path for querying at least some data shards; wherein the data shards included in the dataset are queried from the primary shard or the secondary shard on the corresponding node according to the target path.

[0005] To address the aforementioned technical problems, a second aspect of this application provides an electronic device comprising: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor invokes the program data to execute the method described in the first aspect.

[0006] To address the aforementioned technical problems, a third aspect of this application provides a computer-readable storage medium storing program data thereon, wherein the program data, when executed by a processor, implements the method described in the first aspect.

[0007] The beneficial effects of this application are as follows: Unlike existing technologies, this application obtains all data shards into which the dataset is divided, determines the distribution of all data shards on nodes in the distributed system, wherein each data shard includes a primary shard and a secondary shard, and the primary and secondary shards are distributed on different nodes. Based on the distribution, query paths for queryable data shards are obtained, wherein the query path corresponds to either the primary shard and its corresponding nodes or the secondary shard and its corresponding nodes, that is, a data shard corresponds to one of the query paths, and the number of selected nodes for each node in the query path is obtained. Based on the query path and the number of selected nodes, the query paths of at least some queryable data shards are adjusted so that the corresponding data shards select another node, thereby releasing some occupied nodes. The released nodes are then used to generate corresponding query paths for at least some unqueried data shards, resulting in the target path for querying at least some data shards. This adjusts the number of selected nodes on each node to make the number of selected nodes more balanced. For each data shard, the target path for the data shard is selected from the two query paths, thereby fixing the path for querying the data shards, improving the consistency of data querying in the distributed system, and ensuring the balanced utilization of nodes during large-scale data queries by querying according to the target path. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating one implementation method of the distributed system data query method of this application; Figure 2 This is a flowchart illustrating another implementation of the distributed system data query method of this application; Figure 3 This is a schematic diagram illustrating an application scenario of one implementation of the distribution of primary and secondary shards on the node of this application. Figure 4 This is a schematic diagram illustrating an application scenario of one implementation of the network flow model of this application; Figure 5 This is a schematic diagram illustrating an application scenario of selecting a query path implementation method in the network flow model of this application; Figure 6 This is a schematic diagram illustrating an application scenario of one implementation method for releasing residual capacity using the network flow model of this application; Figure 7 This is a schematic diagram illustrating an application scenario of one implementation method of adjusting the query path using the network flow model of this application; Figure 8This is a schematic diagram of the structure of one embodiment of the electronic device of this application; Figure 9 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0009] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments, and different implementation methods can be adaptively combined. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0010] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper means two or more.

[0011] The distributed system data query method provided in this application is used to perform data query in a distributed system, and its corresponding execution subject is a processing unit capable of data processing.

[0012] Please see Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the distributed system data query method of this application. The method includes: S101: Obtain the distribution of all data shards in the dataset across nodes in the distributed system; where data shards include primary shards and secondary shards, and the primary shards and secondary shards are distributed on different nodes.

[0013] Specifically, the dataset is divided into all data shards, and the distribution of all data shards on the nodes of the distributed system is determined. Each data shard includes a primary shard and a secondary shard, and the primary shard and the secondary shard are distributed on different nodes.

[0014] It should be noted that when the dataset is stored in the distributed system, it is divided into multiple data shards. In order to ensure the high availability and fault tolerance of the system, the data shards usually correspond to primary shards and secondary shards, and the primary shards and secondary shards are distributed on different nodes. When the data corresponding to the data shard is written, it is first written to the primary shard, and then synchronized to the secondary shard by the primary shard.

[0015] It is understandable that there is a time difference in the synchronization from the primary shard to the secondary shard. If data is queried from the primary shard but the secondary shard has not yet synchronized the data, and then data is queried from the secondary shard, the result of the second query will be inconsistent with the result of the first query.

[0016] In one embodiment, the primary and secondary shards corresponding to the dataset on each node in the distributed system are obtained, a first mapping relationship is established between each node and the primary and secondary shards, and based on the first mapping relationship, the distribution of all data shards of the dataset on the nodes in the distributed system is determined.

[0017] In some implementation scenarios, each node has a primary shard list and a secondary shard list for each dataset. Based on the primary shard list and secondary shard list on the node, a first mapping relationship is established between each node and the primary and secondary shards. Based on the first mapping relationship, the distribution of all data shards of the dataset on the nodes of the distributed system is determined.

[0018] In one implementation, the nodes corresponding to the primary and secondary shards of each data shard are obtained, a second mapping relationship between each data shard and the nodes is established, and based on the second mapping relationship, the distribution of all data shards of the dataset on the nodes of the distributed system is determined.

[0019] In some implementation scenarios, each data shard corresponds to a record table, which is used to record the nodes where the primary and secondary shards are located. Based on the record tables of all data shards corresponding to the dataset, a second mapping relationship between each data shard and the node is established. Based on the second mapping relationship, the distribution of all data shards of the dataset on the nodes of the distributed system is determined.

[0020] S102: Based on the distribution, obtain the query path of the queryable data shards, and get the number of selected nodes for each node in the query path; where the query path corresponds to the main shard and its corresponding nodes or the secondary shard and its corresponding nodes.

[0021] Specifically, based on the distribution, query paths for queryable data shards are obtained. Each query path corresponds to a primary shard and its corresponding nodes or a secondary shard and its corresponding nodes. In other words, each data shard corresponds to a query path, and the number of selected nodes for each node in the query path is obtained.

[0022] In one embodiment, based on the distribution, a primary shard and its corresponding node or a secondary shard and its corresponding node are randomly selected sequentially for each data shard to obtain a query path corresponding to at least some of the data shards. The data shards for which a query path is obtained are the queryable data shards. The number of times a node is selected in the query path corresponding to all queryable data shards is taken as the node selection count.

[0023] In some implementation scenarios, based on the distribution, a binary classification model is used to randomly select a primary shard and its corresponding node or a secondary shard and its corresponding node for each data shard, thereby obtaining a query path corresponding to at least some of the data shards.

[0024] In one implementation, the latest node selection count for each node is obtained, where the initial node selection count is zero. Based on the distribution and the latest node selection count, query paths are selected for each data shard in the order of data shards until all queryable data shards have obtained query paths, thus determining the final node selection count. During the traversal, the query paths for data shards are selected in a way that minimizes the number of node selections among all nodes.

[0025] In some implementation scenarios, the latest number of selected nodes for each node is obtained. Based on the distribution and the latest number of selected nodes, a probability model is used to select query paths for the data shards in the order of data sharding, until all queryable data shards have obtained query paths. The probability of the probability model selecting a query path is negatively correlated with the number of selected nodes of the two nodes corresponding to each data shard.

[0026] It should be noted that there is a limit to the number of nodes that can be selected. If the number of selected data for some nodes has reached the limit, subsequent data shards corresponding to those nodes cannot be queried. Therefore, when initially determining the query path, some data shards may not receive a query path.

[0027] S103: Based on the query path and the number of selected nodes, adjust the query path of at least some queryable data shards and generate the query path of at least some unqueried data shards to obtain the target path for querying at least some data shards; wherein, according to the target path, query the data shards included in the dataset from the primary or secondary shards on the corresponding nodes.

[0028] Specifically, based on the query path and the number of selected nodes, the query path of at least some queryable data shards is adjusted so that the corresponding data shards select another node, thereby releasing some occupied nodes. The released nodes are then used to generate corresponding query paths for at least some unqueried data shards, thus obtaining the target path for querying at least some data shards.

[0029] It is understandable that adjusting the number of selected nodes on each node to make the number of selected nodes more balanced, and selecting the target path of the data shard from two query paths for each data shard, thereby fixing the path for querying the data shard, can improve the consistency of data query in the distributed system, and querying according to the target path can ensure the balanced utilization of nodes when querying large-scale data.

[0030] In one implementation, nodes are sorted according to their selected number. Starting from the node with the largest selected number, a query path associated with another node is selected for the corresponding data shard to release the corresponding node. Unqueried data shards that match the released nodes are identified, and corresponding query paths are generated for the corresponding data shards. The selected number of nodes is continuously updated until the difference in the selected number between any two nodes is less than the selected number difference threshold. The query path corresponding to each data shard obtained after the final adjustment is used as the target path for querying the data shard.

[0031] In one implementation, the difference in the number of selected nodes between any two nodes is obtained. Based on the difference in the number of selected nodes, a query path associated with another node is selected for the partial data shard corresponding to the node with the larger number of selected nodes, thereby releasing some nodes. The released nodes are used to generate query paths for at least some unqueried data shards, and the number of selected nodes and the difference in the number of selected nodes are continuously updated until the difference in the number of selected nodes between any two nodes is less than the difference in the number of selected nodes threshold. The query path corresponding to each data shard obtained after the final adjustment is used as the target path for querying the data shard.

[0032] It should be noted that in some scenarios, all data shards are queryable data shards, meaning the number of unqueried data shards is zero. In this case, after releasing some nodes, the query path of the data shards can be adjusted to minimize the difference in the number of selected nodes between any two nodes, resulting in a more balanced node load.

[0033] In some implementation scenarios, the final query path is determined by traversing the query paths that can be selected by the data shards, counting the number of selected nodes for each node under different selections, and the difference in the number of selected nodes between nodes.

[0034] In some implementation scenarios, by constructing a network flow model associated with data shards and nodes, the network flow model includes a source and a sink, the source and each data shard, the data shard and the corresponding node, and each node and the sink are connected by an edge with a corresponding residual capacity. The maximum flow result of the network flow model is obtained, thereby determining the final query path.

[0035] Understandably, due to the upper limit on the number of nodes that can be selected, and the limitations on the distribution of primary and secondary shards corresponding to data shards across nodes, it may be impossible to ensure that all shards obtain a query path. Therefore, in different scenarios, it may be that all data shards obtain the target path, or it may be that only some data shards obtain the target path. The above embodiment maximizes the number of data shards that obtain the target path. Furthermore, when the primary and secondary shards corresponding to all data shards are evenly distributed across nodes, each data shard can obtain the target query path.

[0036] It should be noted that the above logic for determining the target path applies to each new dataset in the distributed system; that is, the above steps are performed for each new dataset.

[0037] The above scheme obtains all data shards into which the dataset is divided and determines the distribution of all data shards across nodes in the distributed system. Each data shard includes a primary shard and a secondary shard, which are distributed across different nodes. Based on the distribution, query paths for queryable data shards are obtained. Each query path corresponds to either the primary shard and its corresponding nodes or the secondary shard and its corresponding nodes; that is, each data shard corresponds to one query path. The number of nodes selected for each node in the query path is obtained. Based on the query paths and the number of nodes selected, the query paths for at least some queryable data shards are adjusted so that the corresponding data shards select another node, thereby releasing some occupied nodes. The released nodes are then used to generate corresponding query paths for at least some unqueried data shards, obtaining the target paths for querying at least some data shards. This adjusts the number of nodes selected across nodes to make the node selection more balanced. For each data shard, the target path is selected from the two query paths, thus fixing the query path for the data shard, improving the consistency of data queries in the distributed system, and ensuring balanced node utilization during large-scale data queries by querying according to the target path.

[0038] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the distributed system data query method of this application, the method including: S201: Obtain the distribution of all data shards in the dataset across nodes in the distributed system; where data shards include primary shards and secondary shards, and primary shards and secondary shards are distributed on different nodes.

[0039] Specifically, the dataset is divided into all data shards, and the distribution of all data shards on the nodes of the distributed system is determined. Each data shard includes a primary shard and a secondary shard, and the primary shard and the secondary shard are distributed on different nodes.

[0040] Optionally, the primary and secondary shards are distributed across nodes in the distributed system according to the following steps: obtaining the total number of primary and secondary shards and the total number of nodes; based on the total number of primary and secondary shards and the total number of nodes, distributing the primary and secondary shards of each data shard to different nodes; wherein the total number of shards on a node corresponding to the primary and secondary shards is such that the difference between the total number of shards on any two nodes is less than a difference threshold.

[0041] Specifically, the total number of primary and secondary shards and the total number of nodes are obtained. Based on the total number of primary and secondary shards and the total number of nodes, all primary and secondary shards are evenly distributed across the nodes of the distributed system. The total number of shards for primary and secondary shards on each node is obtained, so that the difference between the total number of shards on any two nodes is less than the difference threshold. This ensures that all primary and secondary shards corresponding to the dataset are evenly distributed across the nodes, increasing the probability of maintaining node load balancing when selecting a target path.

[0042] In a specific implementation scenario, please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating an application scenario of one implementation of the distribution of primary and secondary shards on a node in this application. Taking 12 data shards as an example, those without black borders are primary shards, and those with black borders are secondary shards. Each primary shard corresponds to one secondary shard, and the secondary and primary shards are distributed on different nodes, thereby improving fault tolerance. Furthermore, the number of shards on each node is kept as consistent as possible. Figure 3 Taking the distribution of primary and / or secondary shards on nodes as an example, each node has 6 shards. Some nodes can all be primary shards, some nodes can all be secondary shards, and some nodes can include both primary and secondary shards. This application does not impose specific restrictions on this.

[0043] Understandably, the difference threshold can usually be set to a positive integer, so that when the total number of primary and secondary shards is divisible by the total number of nodes, they are evenly distributed across all nodes; when the total number of primary and secondary shards is not divisible by the total number of nodes, the remaining primary shards and / or secondary shards are respectively assigned to some nodes.

[0044] In one embodiment, obtaining the distribution of all data shards of the dataset across nodes in a distributed system includes performing the following steps on the dataset: obtaining the primary and secondary shards on each node, establishing a first mapping relationship between each node and the primary and secondary shards, obtaining the nodes where the primary and secondary shards of each data shard are distributed, and establishing a second mapping relationship between each data shard and nodes; and determining the distribution of all data shards of the dataset across nodes in the distributed system based on the first and second mapping relationships.

[0045] Specifically, obtain node information in the distributed system, including the total number of nodes N and node identifiers, and determine the number of primary shards P_i and secondary shards R_i on each node, where i is the node identifier and 1≤i≤N. Construct the first mapping relationship between nodes and shards, and record the primary and secondary shards stored on each node.

[0046] Optionally, a node mapping table node_map is created, where node_map[i] records the list of primary shards and the list of replica shards stored on node i.

[0047] Similarly, obtain sharding information, including the total number of primary and secondary shards M, shard identifiers, and the nodes where the primary and secondary shards of each shard are located. Construct a second mapping relationship from shards to nodes and record the primary and secondary shards stored on each node.

[0048] Optionally, a shard mapping table shard_map is created, where shard_map[j] records the primary node and replica node where the primary shard of shard j is located, and j is the shard identifier, 1≤j≤M.

[0049] It is understandable that, based on the first and second mapping relationships, the distribution of data shards in the dataset on the nodes can be determined, thus obtaining a more accurate distribution through the two-dimensional mapping relationship.

[0050] S202: Based on the distribution, construct a network flow model that matches all data shards and all nodes; wherein, the network flow model includes a source node and a sink node, the source node and each data shard, the data shard and the corresponding node, and each node and the sink node are connected by an edge, and the edge corresponds to residual capacity.

[0051] Specifically, based on the distribution, corresponding edges are set for each data shard and its corresponding node, and a source node is set on one side of the data shard and a sink node is set on the other side of the node. Corresponding edges are set for the source node and each data shard, and corresponding edges are set for each node and the sink node. Initial residual capacity is set for all edges to obtain a network flow model that matches the data shards and nodes.

[0052] In one embodiment, based on the distribution, a network flow model matching all data shards and all nodes is constructed, including: constructing a bipartite graph model between all data shards and all nodes based on the distribution; wherein, in the bipartite graph model, data shards are associated with nodes corresponding to primary and secondary shards; a source node and a sink node are set on both sides of the bipartite graph model, and forward edge capacity and reverse edge capacity are set for the source node and each data shard, the data shard and the corresponding node, and each node and the sink node, to obtain the network flow model; wherein, the residual capacity is the difference between the forward edge capacity and the reverse edge capacity.

[0053] Specifically, based on the distribution, a bipartite graph model is constructed between all data shards and their corresponding nodes. In the bipartite graph model, the nodes corresponding to the data shards are also the nodes corresponding to the primary shard and the secondary shard. Therefore, a data shard is associated with two nodes, and thus corresponding edges are set.

[0054] Furthermore, in the bipartite graph model, a source node is set on one side of the data fragment, and a sink node is set on one side of the node. Corresponding edges are set for the source node and each data fragment, and corresponding edges are set for each node and the sink node. Thus, by constructing a bipartite graph model, the accuracy of the network flow model is ensured.

[0055] Understandably, each edge in the network flow model has an initial residual capacity, which is obtained by the difference between the forward edge capacity and the reverse edge capacity. The forward edge capacity corresponds to the capacity from the source to the sink, and the reverse edge capacity corresponds to the capacity from the sink to the source. The initial reverse edge capacity can be uniformly set to zero. Thus, by setting the forward edge capacity on each edge, the residual capacity can be determined, and the network flow model can be constructed.

[0056] In some implementation scenarios, forward edge capacity and reverse edge capacity are set for the source node and each data shard, the data shard and its corresponding node, and each node and sink node. This includes: setting forward edge capacity and reverse edge capacity for the source node and each data shard based on the association relationship between the source node and each data shard; setting forward edge capacity and reverse edge capacity for the data shard and its corresponding node based on the association relationship between the data shard and its corresponding node; and setting forward edge capacity and reverse edge capacity for each node and sink node based on the total number of data shards and the total number of nodes.

[0057] Specifically, based on the relationship between the source node and each data shard, i.e., the requirement that each data shard must be selected, forward edge capacity and reverse edge capacity are set for the source node and each data shard. Based on the relationship between the data shard and the corresponding node, i.e., the connectivity between the data shard and the corresponding node, forward edge capacity and reverse edge capacity are set for the data shard and the corresponding node. Based on the total number of data shards and the total number of nodes, forward edge capacity and reverse edge capacity are set for the node and the sink node. Thus, by using the residual capacity between the node and the sink node, the number of data shards that each node can connect to is constrained, increasing the probability that the final selected query path can be evenly distributed across different nodes.

[0058] Understandably, please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating an application scenario of one implementation of the network flow model of this application. In this diagram, s is the origin, t is the sink, u_j is a data fragment, and multiple data fragments together form a fragment set. v_i is a node, and multiple nodes together form a node set. Figure 4 In this diagram, the first value on each edge represents the capacity of the reverse edge, and the second value represents the capacity of the forward edge. For example, 0 / 1 indicates a residual capacity of 1. The residual capacity of the edges between data shards and their corresponding nodes is set to 1, indicating that the data shards and their corresponding nodes are connected. An edge with a residual capacity of 1 is established from the source node to each data shard, indicating that each data shard needs to be selected once. An edge with a residual capacity of k is established from each node to the sink node t, indicating that each node can select at most k data shards. Here, the value of k is greater than or equal to the quotient of the total number of data shards and the total number of nodes, and k is an integer.

[0059] S203: Based on residual capacity, select the edges that can connect from the source to the sink in the network flow model as the query path, and obtain the number of selected nodes for each node in the query path.

[0060] Specifically, based on residual capacity, starting from the source node in the network flow model, select connectable edges, that is, edges with residual capacity greater than 0, until the sink node is reached to obtain the query path, and obtain the number of selected nodes for each node in all query paths that the network flow model can currently obtain.

[0061] Please see Figure 5 , Figure 5 This is a schematic diagram of an application scenario for selecting a query path in the network flow model of this application. When the corresponding edge is selected, the capacity of the reverse edge increases, thereby reducing the residual capacity on the corresponding edge. Since the residual capacity of the source node s and any data shard is 1, when a data shard selects one of the nodes as the query path, the data shard and the other node will not be selected.

[0062] It should be noted that due to the randomness in the initial selection of the query path, and there is a limit on the residual capacity between each node and the sink node. Therefore, some data shards may not be able to obtain a query path because the residual capacity between the corresponding node and the sink node has all become zero, and thus the maximum flow result cannot be obtained.

[0063] S204: Based on the number of selected nodes, release the residual capacity on the edges corresponding to at least some of the query paths in the network flow model. Select query paths associated with another node for at least some of the queryable data shards from the network flow model. Use the released residual capacity and its corresponding nodes to select query paths for at least some of the unqueried data shards in the network flow model, and update the number of selected nodes and the residual capacity.

[0064] Specifically, based on the number of selected nodes, release the residual capacity on the edges corresponding to at least some of the query paths in the network flow model, so as to reselect query paths in the network flow model. Thus, select query paths associated with another node for at least some of the queryable data shards from the network flow model. Use the released residual capacity and its corresponding nodes to select query paths for at least some of the unqueried data shards in the network flow model, and thus update the number of selected nodes and the residual capacity.

[0065] Specifically, please refer to Figure 6 , Figure 6 is a schematic diagram of an application scenario of releasing the residual capacity of the network flow model of the present application. Among them, the red edges in the figure are the reverse edges corresponding to each edge with traffic, and the capacity becomes the opposite of the corresponding consumed traffic, where n < k. When the residual capacity is released, the query paths can be reselected according to the released residual capacity. When releasing the residual capacity, it is possible to choose to release the residual capacity on the edges corresponding to some of the query paths, or choose to release the residual capacity on the edges corresponding to all query paths.

[0066] In one embodiment, based on the number of selected nodes, releasing the residual capacity on the edges corresponding to at least some of the query paths in the network flow model and selecting query paths associated with another node for at least some of the queryable data shards from the network flow model includes: obtaining the difference in the number of selected nodes between the selected node numbers, determining the query paths to be released in the network flow model based on the difference in the number of selected nodes and the nodes on the unselected edges corresponding to the queryable data shards, and releasing the residual capacity on the edges corresponding to the corresponding query paths; selecting query paths associated with another node for the data shards on the released query paths from the network flow model.

[0067] Specifically, the difference in the number of selected nodes between any two nodes is obtained. Based on the difference in the number of selected nodes and the nodes on the unselected edges corresponding to the queryable data shards, the impact of selecting another node on the number of selected nodes is evaluated. The query paths to be released in the network flow model are determined, and the residual capacity on the edges corresponding to the query paths is released so that the number of selected nodes is more even after the query paths are adjusted.

[0068] Furthermore, from the network flow model, a query path associated with another node is selected for the data shards on the released query paths. The remaining capacity after release and its corresponding nodes are used to generate query paths for the unqueried data shards, thereby updating the number of selected nodes and the remaining capacity, thus balancing the number of selected nodes among the nodes and obtaining more query paths.

[0069] Understandably, by repeatedly releasing residual capacity and continuously adjusting query paths, the goal is to obtain the result with the most query paths in the network flow model, i.e., the maximum flow result. This ensures that each data shard can perform data queries and that the load on the nodes is relatively balanced.

[0070] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating an application scenario of one implementation method of adjusting the query path using the network flow model of this application. Figure 7 The blue query path is the readjusted query path, obtained through comparison. Figure 5 and Figure 7 It can be seen that the data shard u_3 originally selected node v_1 as the query path. After releasing the residual capacity, the data shard u_3 reselected node v_i, making the number of selected nodes more balanced and obtaining the maximum flow result.

[0071] S205: In response to obtaining the maximum flow result with the most query paths in the network flow model, select the query path from the maximum flow result for the corresponding data shard as the target path for querying the data shard.

[0072] Specifically, when the maximum flow result with the most query paths in the network flow model is obtained, the query path in the maximum flow result is selected as the target path for the corresponding data shard, thereby fixing the query method of the corresponding data shard as the target path and ensuring query consistency.

[0073] Please see Figure 8 , Figure 8This is a schematic diagram of an embodiment of the electronic device of this application. The electronic device 30 includes a memory 301 and a processor 302 coupled to each other. The memory 301 stores program data (not shown in the figure). The processor 302 calls the program data to implement the method in any of the above embodiments. For the description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0074] Please see Figure 9 , Figure 9 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 40 stores program data 400. When the program data 400 is executed by a processor, it implements the method in any of the above embodiments. For a detailed description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0075] It should be noted that the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0076] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0077] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] The above description is merely an embodiment of this application and does not limit the scope of protection of this application. Any equivalent structural or procedural transformations made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of protection of this application.

Claims

1. A data query method for a distributed system, characterized in that, The method includes: Obtain the distribution of all data shards in the dataset across nodes in the distributed system; wherein the data shards include primary shards and secondary shards, and the primary shards and the secondary shards are distributed on different nodes; Based on the distribution, the query path of the queryable data shards is obtained, and the number of selected nodes for each node in the query path is obtained; wherein, the query path corresponds to the main shard and its corresponding node or the sub-shard and its corresponding node; Based on the query path and the number of selected nodes, the query path of at least some queryable data shards is adjusted and the query path of at least some unqueried data shards is generated to obtain the target path for querying at least some data shards; wherein, the data shards included in the dataset are queried from the primary shard or the secondary shard on the corresponding node according to the target path.

2. The distributed system data query method according to claim 1, characterized in that, The step of obtaining the distribution of all data shards of the dataset across nodes in the distributed system includes performing the following steps on the dataset: Obtain the primary shard and the secondary shard on each node, establish a first mapping relationship between each node and the primary shard and the secondary shard, obtain the nodes distributed by the primary shard and the secondary shard of each data shard, and establish a second mapping relationship between each data shard and the node; Based on the first mapping relationship and the second mapping relationship, the distribution of all data shards of the dataset on the nodes of the distributed system is determined.

3. The distributed system data query method according to claim 1, characterized in that, The process of obtaining query paths for queryable data shards based on distribution conditions, and acquiring the number of selected nodes for each node in the query path, includes: Based on the distribution, a network flow model matching all the data shards and all the nodes is constructed; wherein, the network flow model includes a source node and a sink node, the source node is connected to each of the data shards, the data shards are connected to the corresponding nodes, and each node is connected to the sink node, and the edges correspond to residual capacity; Based on the residual capacity, an edge that can connect from the source to the sink is selected from the network flow model as the query path, and the number of selected nodes for each node in the query path is obtained.

4. The distributed system data query method according to claim 3, characterized in that, The step of constructing a network flow model that matches all the data shards and all the nodes based on the distribution includes: Based on the distribution, a bipartite graph model is constructed between all the data shards and all the nodes; wherein, in the bipartite graph model, the data shards are associated with the nodes corresponding to the primary shard and the secondary shard; A source node and a sink node are set on both sides of the bipartite graph model. Forward edge capacity and reverse edge capacity are set for the source node and each data fragment, the data fragment and the corresponding node, and each node and the sink node to obtain the network flow model; wherein, the residual capacity is the difference between the forward edge capacity and the reverse edge capacity.

5. The distributed system data query method according to claim 4, characterized in that, The step of setting forward edge capacity and reverse edge capacity for the source node and each data shard, the data shard and the corresponding node, and each node and the sink node includes: Based on the association between the source node and each data shard, the forward edge capacity and the reverse edge capacity are set for the source node and each data shard; and, Based on the association between the data shards and the corresponding nodes, the forward edge capacity and the reverse edge capacity are set for the data shards and the corresponding nodes; and, Based on the total number of data shards and the total number of nodes of all the data shards, the forward edge capacity and the reverse edge capacity are set for each node and the sink.

6. The distributed system data query method according to claim 3, characterized in that, The step of adjusting the query paths of at least some queryable data shards and generating query paths for at least some unqueried data shards based on the query path and the number of selected nodes, to obtain the target path for querying at least some data shards, includes: Based on the number of selected nodes, release the residual capacity on the edges corresponding to at least a portion of the query paths in the network flow model, select a query path associated with another node for at least a portion of the queryable data shards in the network flow model, use the released residual capacity and its corresponding nodes to select query paths for at least a portion of the unqueried data shards in the network flow model, and update the number of selected nodes and the residual capacity. In response to obtaining the maximum flow result with the most query paths in the network flow model, the query path in the maximum flow result is selected as the target path for querying the corresponding data shard.

7. The distributed system data query method according to claim 6, characterized in that, The step of releasing the residual capacity on the edges corresponding to at least a portion of the query paths in the network flow model based on the selected node count, and selecting a query path associated with another node from the network flow model for at least a portion of the queryable data shards, includes: Obtain the difference in the number of selected nodes between the selected nodes. Based on the difference in the number of selected nodes and the nodes on the unselected edges corresponding to the queryable data shards, determine the query paths to be released in the network flow model and release the residual capacity on the edges corresponding to the query paths. From the network flow model, select a query path associated with another node for the data shard on the released query path.

8. The distributed system data query method according to any one of claims 1-7, characterized in that, The primary shard and the secondary shard are distributed across nodes in the distributed system according to the following steps: Obtain the total number of primary and secondary shards and the total number of nodes for all primary and secondary shards; Based on the total number of primary and secondary shards and the total number of nodes, the primary and secondary shards of each data shard are distributed to different nodes; wherein, each node has a total number of node shards corresponding to the primary and secondary shards, and the difference between the total number of node shards on any two nodes is less than a difference threshold.

9. An electronic device, characterized in that, include: A memory and a processor are coupled to each other, wherein the memory stores program data, and the processor invokes the program data to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium storing program data thereon, characterized in that, When the program data is executed by the processor, it implements the method as described in any one of claims 1-8.

Images