Data fragmentation processing method and device, equipment and medium

Abstract

The invention provides a data fragmentation processing method and device, equipment and a medium, and the method comprises the steps: obtaining table identification information and to-be-added data in a data adding request in response to the received data adding request; based on table identification information in the newly added data request, determining a target data table from a target node group of the database cluster, the target node group comprising at least one database node in the database cluster, and the at least one database node being pre-specified by a user; and storing the data to be newly added into the target data table.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data slicing processing method, device, equipment, and medium. Background Art

[0002] With the advancement of technology, the amount of data in various fields is growing explosively. A single node can no longer meet the storage needs of large amounts of data. Therefore, distributed database operations have emerged, which provide an effective solution for processing and storing massive data.

[0003] In related technologies, data is usually stored on different nodes of a distributed database to improve overall performance. However, data storage nodes are usually specified when a table is created, which means that when users create tables in large quantities, they need to determine the distribution nodes of the table one by one based on the configuration and remaining space on the nodes. This operation is cumbersome and prone to errors. Summary of the invention

[0004] In view of the above problems, the present invention provides a data sharding processing method, apparatus, device, and medium.

[0005] According to a first aspect of the present invention, a data sharding processing method is provided, comprising: in response to a received request for new data, obtaining table identification information and data to be added in the request for new data; determining a target data table based on the table identification information in the request for new data, wherein the target data table is stored in a target node group of a database cluster, the target node group including at least one database node pre-specified by a user in the database cluster; and storing the data to be added in the target data table.

[0006] According to an embodiment of the present invention, before the above-mentioned response to the received request for new data and obtaining the table identification information and the data to be added in the above-mentioned request for new data, the above-mentioned method also includes: in response to the reception of the request for new data table, obtaining the to-be-processed table information of the to-be-processed table in the above-mentioned request for new data table and the library identification information for identifying the database corresponding to the above-mentioned to-be-processed table; when it is determined that the above-mentioned node identification information matches the information of the target node group pre-specified by the above-mentioned user, storing the above-mentioned to-be-processed table in the above-mentioned target node group; when it is determined that the above-mentioned node identification information does not match the information of the target node group pre-specified by the above-mentioned user, determining the table node group that matches the above-mentioned to-be-processed table from the above-mentioned database cluster based on the above-mentioned node identification information; and storing the above-mentioned to-be-processed table in the above-mentioned table node group.

[0007] According to an embodiment of the present invention, the target node group is determined in the following manner: in response to receiving a database addition request from a user, sending multiple database clusters of different types to the user, wherein the multiple database clusters are stored on different node groups; in response to receiving a cluster identifier representing the target database cluster from the user, sending a database node group matching the target database cluster to the user; and in response to receiving a node group identifier representing the target node group from the user, determining the target node group.

[0008] According to an embodiment of the present invention, the above method also includes: in response to receiving a first query data request, obtaining the first table identification information and the first data identification to be queried in the above first query data request; based on the above first table identification information, determining the target data table from the above target node group of the above database cluster; based on the above first data identification to be queried, querying from the above target data table to obtain the first target query data.

[0009] According to an embodiment of the present invention, the target node group of the above-mentioned database cluster includes at least one database management node, and the above-mentioned determining the target data table from the target node group of the database cluster based on the table identification information in the above-mentioned new data request includes: obtaining a mapping relationship between the table identification and the target node group, wherein the mapping relationship between the table identification and the target node group is stored in the cache of the above-mentioned database management node; based on the mapping relationship between the table identification and the target node group and the table identification information in the above-mentioned new data request, determining the above-mentioned target data table from the above-mentioned target node group of the above-mentioned database cluster.

[0010] According to an embodiment of the present invention, the cache of the above-mentioned database management node also stores a hash mapping value group of the target node group, and the above-mentioned hash mapping value group includes a hash value range corresponding to each of the above-mentioned database nodes; the above-mentioned storing the above-mentioned data to be added into the above-mentioned target data table includes: using a hash algorithm to obtain a hash value of the data to be added based on the above-mentioned data to be added; determining the target storage node based on the hash mapping value of the above-mentioned target node group and the hash value of the above-mentioned data to be added, wherein the above-mentioned target storage node includes at least one of the above-mentioned database nodes in the above-mentioned target node group; storing the above-mentioned data to be added in the above-mentioned target storage node.

[0011] According to an embodiment of the present invention, the above method also includes: in response to the received second query data request, obtaining the second table identification information in the above second query data request, the second data identification to be queried and the mapping relationship between the table identification in the above cache and the target node group; determining the sub-target node group based on the above second table identification information and the mapping relationship between the above table identification and the target node group; for each database node in the above sub-target node group, determining the sub-target data table from the above database node based on the above second data identification to be queried, and obtaining a sub-target data table group; determining a candidate query data group that matches the above second data identification to be queried from the above sub-target data table group; and obtaining the second target query data based on the above candidate query data group.

[0012] The second aspect of the present invention provides a data sharding processing device, including: an acquisition module, used to respond to a received new data request and obtain the table identification information and the data to be added in the above-mentioned new data request; a determination module, used to determine the target data table from the target node group of the database cluster based on the table identification information in the above-mentioned new data request, wherein the above-mentioned target node group includes at least one database node in the above-mentioned database cluster, and the above-mentioned at least one database node is pre-specified by the user; a storage module, used to store the above-mentioned data to be added in the above-mentioned target data table.

[0013] According to an embodiment of the present invention, the above-mentioned device also includes: a new request acquisition module, which is used to respond to receiving a new data table request, obtain the to-be-processed table information of the to-be-processed table in the above-mentioned new data table request, and the library identification information used to identify the database corresponding to the above-mentioned to-be-processed table; a new data table storage module, which is used to store the above-mentioned to-be-processed table in the above-mentioned target node group when it is determined that the above-mentioned node identification information matches the information of the target node group pre-specified by the above-mentioned user; a table node group determination module, which is used to determine the table node group matching the above-mentioned to-be-processed table from the above-mentioned database cluster based on the above-mentioned node identification information when it is determined that the above-mentioned node identification information does not match the information of the target node group pre-specified by the above-mentioned user; a table storage module, which is used to store the above-mentioned to-be-processed table in the above-mentioned table node group.

[0014] According to an embodiment of the present invention, the target node group is determined by the following submodules: a cluster sending submodule, for sending multiple database clusters of different types to the user in response to receiving a database addition request from the user, wherein the multiple database clusters are stored on different node groups; a node sending submodule, for sending multiple database nodes matching the target database cluster to the user in response to receiving a cluster identifier representing the target database cluster from the user; and a node group determination submodule, for determining the target node group in response to receiving a node group identifier representing the target node group from the user.

[0015] According to an embodiment of the present invention, the above-mentioned device also includes: a first query module, used to obtain the first table identification information and the first data identification to be queried in the above-mentioned first query data request in response to receiving the first query data request; a first data table determination module, used to determine the target data table from the above-mentioned target node group of the above-mentioned database cluster based on the above-mentioned first table identification information; the first query module is used to query the first target query data from the above-mentioned target data table based on the above-mentioned first data identification to be queried.

[0016] According to an embodiment of the present invention, the target node group of the above-mentioned database cluster includes at least one database management node, and the above-mentioned determination module includes: a mapping relationship acquisition submodule, used to obtain the mapping relationship between the table identifier and the target node group, wherein the mapping relationship between the above-mentioned table identifier and the target node group is stored in the cache of the above-mentioned database management node; a data table determination submodule, used to determine the above-mentioned target data table from the above-mentioned target node group of the above-mentioned database cluster based on the mapping relationship between the above-mentioned table identifier and the target node group and the table identifier information in the above-mentioned new data request.

[0017] According to an embodiment of the present invention, the cache of the above-mentioned database management node also stores a hash mapping value group of the target node group, and the above-mentioned hash mapping value group includes a hash value range corresponding to each of the above-mentioned database nodes; the above-mentioned storage module includes: a hash value calculation submodule, which is used to use a hash algorithm to obtain the hash value of the data to be added based on the above-mentioned data to be added; a target storage node determination submodule, which is used to determine the target storage node based on the hash mapping value of the above-mentioned target node group and the hash value of the above-mentioned data to be added, wherein the above-mentioned target storage node includes at least one of the above-mentioned database nodes in the above-mentioned target node group; a data to be added submodule, which is used to store the above-mentioned data to be added in the above-mentioned target storage node.

[0018] According to an embodiment of the present invention, the above-mentioned device also includes: a second query module, which is used to respond to the received second query data request and obtain the second table identification information in the above-mentioned second query data request, the second data identification to be queried and the mapping relationship between the table identification in the above-mentioned cache and the target node group; a sub-target determination module, which is used to determine the sub-target node group based on the above-mentioned second table identification information and the mapping relationship between the above-mentioned table identification and the target node group; a sub-target data table determination module, which is used to determine the sub-target data table from the above-mentioned database node based on the above-mentioned second data identification to be queried for each database node in the above-mentioned sub-target node group, and obtain a sub-target data table group; a candidate query data group determination module, which is used to determine a candidate query data group that matches the above-mentioned second data identification to be queried from the above-mentioned sub-target data table group; a second query module, which is used to obtain the second target query data based on the above-mentioned candidate query data group.

[0019] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0020] The fourth aspect of the present invention further provides a computer-readable storage medium on which a computer program or instruction is stored, and the steps of the above method are implemented when the above computer program or instruction is executed by a processor.

[0021] The fifth aspect of the present invention also provides a computer program product, including a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0022] According to an embodiment of the present invention, by obtaining the table identification information in the request when adding new data, the corresponding target data table is determined from the target node group pre-specified by the user, and the data is stored in the target data table. When the user adds new data to the distributed database, the data can be directly stored based on the node group preset by the user, which simplifies user operations, improves user experience, and further reduces the computing power of hardware devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0024] Figure 1 A diagram showing an application scenario of a data sharding processing method and device according to an embodiment of the present invention;

[0025] Figure 2 A flow chart of a data sharding processing method according to an embodiment of the present invention is shown;

[0026] Figure 3 A flow chart of determining a target node group according to an embodiment of the present invention is shown;

[0027] Figure 4 A structural block diagram of a data sharding processing device according to an embodiment of the present invention is shown; and

[0028] Figure 5 A block diagram of an electronic device suitable for implementing a data slicing processing method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0029] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0030] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0031] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0032] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0033] With the increasing amount of data, storing a large amount of data on a single node can no longer meet the system's requirements in terms of performance load and storage capacity. Based on this, in a distributed database system, data is usually specified to be distributed and stored on different shards to improve the overall performance of the system. However, for a distributed database, data storage nodes are generally specified when a table is created. For a large number of tables that need to be built on a database, distribution nodes need to be specified when each table is created. For users, the operation is cumbersome and prone to errors.

[0034] In view of this, the present invention provides a data sharding processing method, which includes: in response to a received request for new data, obtaining table identification information and data to be added in the request for new data; based on the table identification information in the request for new data, determining a target data table from a target node group of a database cluster, wherein the target node group includes at least one database node in the database cluster, and at least one database node is pre-specified by a user; and storing the data to be added in the target data table.

[0035] Figure 1 A diagram showing an application scenario of a data sharding processing method and device according to an embodiment of the present invention is shown.

[0036] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0037] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0038] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0039] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0040] It should be noted that the data sharding processing method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the data storage device provided in the embodiment of the present invention can generally be set in the server 105. The data sharding processing method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data storage device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0042] The following will be based on Figure 1 The scene described by Figure 2~Figure 3 The data fragmentation processing method according to the embodiment of the present invention is described in detail.

[0043] Figure 2 A flow chart of a data sharding processing method according to an embodiment of the present invention is shown.

[0044] like Figure 2 As shown, the data sharding processing method of this embodiment includes operations S210 to S230.

[0045] In operation S210, in response to the received new data request, table identification information and the data to be added in the new data request are obtained.

[0046] Exemplarily, the request for adding new data may be, for example, "INSERT INTO table_name VALUES (value1, value2)", wherein the above "table_name" is table identification information, and "value1, value2" is the data to be added.

[0047] In operation S220, a target data table is determined based on the table identification information in the new data request.

[0048] The target data table is stored in a target node group of a database cluster, and the target node group includes at least one database node pre-specified by a user in the database cluster.

[0049] According to an embodiment of the present invention, the above-mentioned data sharding processing method can be applied to a distributed database. The distributed database has multiple nodes. When building a table for storage, the corresponding node can be selected according to user needs to store the table.

[0050] According to an embodiment of the present invention, the target node group may be specified when a user creates a database, or may be specified when a user creates a table, and the present invention does not impose any limitation on this.

[0051] In operation S230, the data to be added is stored in the target data table.

[0052] According to an embodiment of the present invention, by obtaining the table identification information in the request when adding new data, the corresponding target data table is determined from the target node group pre-specified by the user, and the data is stored in the target data table. When the user adds new data to the distributed database, the data can be directly stored based on the node group preset by the user, which simplifies user operations, improves user experience, and further reduces the computing power of hardware devices.

[0053] According to an embodiment of the present invention, before the above-mentioned response to the received request for new data and obtaining the table identification information and the data to be added in the request for new data, the above-mentioned method may also include: in response to the receipt of the request for new data table, obtaining the to-be-processed table information of the to-be-processed table in the request for new data table and the node identification information used to identify the storage space of the to-be-processed table; when it is determined that the node identification information matches the information of the target node group pre-specified by the user, storing the to-be-processed table in the target node group; when it is determined that the node identification information does not match the information of the target node group pre-specified by the user, determining a table node group that matches the to-be-processed table from the database cluster based on the node identification information; and storing the to-be-processed table in the table node group.

[0054] According to an embodiment of the present invention, the information of the target node group pre-specified by the user may be stored in a system table of a distributed database.

[0055] Exemplarily, for a request to add a new data table, if the node identification information specified in the request is the same as the information of the target node group pre-specified by the user, the data table can be distributed on the specified nodes; if the node identification information specified in the request is different from the information of the target node group pre-specified by the user, a table node group matching the table to be processed is searched from the database cluster, and the data table is distributed on the table node group; if there is no node group matching the table to be processed in the database, the data table is distributed on all nodes.

[0056] According to an embodiment of the present invention, by judging the node identifier in the request for a new data table when creating a table, the user can flexibly determine the distribution nodes of the data table; further, when the node identifier does not exist, the table node group is automatically found in the distributed database according to the node identifier, and the data table is distributed on the nodes corresponding to the table node group, so that the user only needs to consider the configuration and node remaining space issues once when creating a large number of tables, which simplifies user operations while improving user experience.

[0057] According to an embodiment of the present invention, a target node group is determined in the following manner: in response to receiving a database addition request from a user, multiple database clusters of different types are sent to the user; in response to receiving a cluster identifier representing the target database cluster from the user, a database node group matching the target database cluster is sent to the user, wherein the multiple database clusters are stored on different node groups; and in response to receiving a node group identifier representing the target node group from the user, the target node group is determined.

[0058] Figure 3 A target node group determination flow chart according to an embodiment of the present invention is shown.

[0059] like Figure 3 As shown, the user first sends a database addition request to the database, and the database cluster queries its own configuration and the remaining storage space of the current node and other information and returns it to the user. The aforementioned information can be, for example: Configuration A: "Database Cluster 1; Database Cluster 2", Configuration B: "Database Cluster 3, Database Cluster 4"; the user selects the database cluster according to his own needs and returns the corresponding cluster identifier to the database cluster. The database cluster receives the returned cluster identifier, queries the node corresponding to the cluster, and returns the multiple nodes included in the cluster to the user for selection. The user can form a node group with multiple nodes and return the corresponding node group identifier to the database cluster so that the mapping relationship between the database and the node group is stored in the system table of the database.

[0060] According to an embodiment of the present invention, the user can also directly specify the distribution node group or distribution node when building a database after logging in according to his own needs, wherein the distribution node group is pre-set by the user before building the database, and the database building statement can be, for example, "CREATE DATABASE database_name opt_with created_opt_list OptSubCluser", wherein "database_name" represents the database name, and "OptSubCluser" represents the specified node or node group, which can be represented by "OptSubCluster ::= OptSubClusterInternal | EMPTY", wherein "OptSubClusterInternal" can be represented as "OptSubClusterInternal ::= TO NODEpgxcnodes | TO GROUP_P pgxcgroup_name", "pgxcnodes" represents the node, which can be a collection type data, and "pgxcgroup_name" represents the node group name. It should be noted that the node or node group can be not specified when building the database, so that the table can be distributed to all nodes in the subsequent table building operation.

[0061] According to an embodiment of the present invention, by sending different database clusters to users when they add a new database, and providing database nodes for users to make further selections after they select a cluster, and finally determining the target node group based on the node group identifier selected by the user, the flexibility of user configuration can be improved, the diverse needs of users can be met, and the user experience can be further improved.

[0062] According to an embodiment of the present invention, the above method also includes: in response to receiving a first query data request, obtaining the first table identification information and the first data identification to be queried in the first query data request; based on the first table identification information, determining the target data table from the target node group of the database cluster; and obtaining the first target query data by querying from the target data table based on the first data identification to be queried.

[0063] According to an embodiment of the present invention, after receiving a query request from a user, the entire target data table can be determined in the target node group of the database cluster, and then based on the first data to be queried in the query request, the first target query data can be queried from the above-mentioned entire target database, and the first target query data can be returned to the user.

[0064] According to an embodiment of the present invention, the target node group of the database cluster includes at least one database management node, and based on the table identification information in the new data request, determining the target data table from the target node group of the database cluster includes: obtaining a mapping relationship between the table identification and the target node group, wherein the mapping relationship between the table identification and the target node group is stored in the cache of the database management node; based on the mapping relationship between the table identification and the target node group and the table identification information in the new data request, determining the target data table from the target node group of the database cluster.

[0065] According to an embodiment of the present invention, the above method can be applied to a database management node. When determining a target data table from a target node group in a database cluster, the target data table can be determined for data storage through a mapping relationship between a table identifier cached in the database management node and the target node group.

[0066] Exemplarily, access to system tables can be accomplished through a cache framework, which is used to store and manage cache data in an application. When reading data, it first checks whether the data exists in the cache. If it exists and is not expired, the data is obtained directly from the cache, avoiding the overhead of re-obtaining data from the data source.

[0067] According to an embodiment of the present invention, the mapping relationship between the table identifier and the target node group is stored in the cache of the central node, and the corresponding mapping relationship is directly read from the cache and added when new data is added, thereby reducing hardware resource consumption while ensuring the efficiency of adding new data. Furthermore, the mapping relationship between the library identifier and the target node group is also cached in the cache of the central node, which can ensure table building performance and reduce resource consumption when users add a large number of data tables.

[0068] According to an embodiment of the present invention, the cache of the database management node also stores a hash mapping value group of the target node group, the hash mapping value group includes a hash value range corresponding to each database node; storing the data to be added in the target data table includes: using a hash algorithm to obtain a hash value of the data to be added based on the data to be added; determining a target storage node based on the hash mapping value of the target node group and the hash value of the data to be added, wherein the target storage node includes at least one database node in the target node group; and storing the data to be added in the target storage node.

[0069] Exemplarily, for a target node group, it includes two storage nodes: the hash mapping value group corresponding to node A can be 1~3, and the hash mapping value group corresponding to node B can be 4~8. When storing data, the hash mapping of the data to be added is performed and the hash value is 2, then the data to be added can be stored in node A.

[0070] According to an embodiment of the present invention, the above method may also include: in response to a received second query data request, obtaining the second table identification information in the second query data request, the second data identification to be queried, and the mapping relationship between the table identification in the cache and the target node group; determining the sub-target node group based on the second table identification information and the mapping relationship between the table identification and the target node group; for each database node in the sub-target node group, determining the sub-target data table from the database node based on the second data identification to be queried, and obtaining a sub-target data table group; determining a candidate query data group that matches the second data identification to be queried from the sub-target data table group; and obtaining the second target query data based on the candidate query data group.

[0071] Exemplarily, for the second query data request received, the second table identification information is table A, and the mapping relationship between the table identification in the cache and the target node group is: data table: table A, node group: node 1, node 2; then the sub-target node group can be determined as node 1 and node 2 through the second table identification information. For node 1, the sub-target data table can be determined from the shard data of multiple data tables stored on the node based on the data identification to be queried, and then the candidate query data group in the sub-target data table group is obtained. Finally, the multiple candidate query groups obtained by querying on node 1 and node 2 are integrated to obtain the second target query data.

[0072] Based on the above data sharding processing method, the present invention also provides a data sharding processing device. Figure 4 The device is described in detail.

[0073] Figure 4 A structural block diagram of a data sharding processing device according to an embodiment of the present invention is shown.

[0074] like Figure 4 As shown, the data sharding processing device 400 of this embodiment includes an acquisition module 410 , a determination module 420 , and a storage module 430 .

[0075] The acquisition module 410 is used to respond to the received new data request and acquire the table identification information and the data to be added in the new data request. In one embodiment, the acquisition module 410 can be used to perform the operation S210 described above, which will not be described in detail here.

[0076] The determination module 420 is used to determine the target data table based on the table identification information in the new data request, wherein the target data table is stored in the target node group of the database cluster, and the target node group includes at least one database node pre-specified by the user in the database cluster. In one embodiment, the determination module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0077] The storage module 430 is used to store the data to be added into the target data table. In one embodiment, the storage module 430 can be used to perform the operation S230 described above, which will not be described in detail here.

[0078] According to an embodiment of the present invention, the above-mentioned device also includes: a new request acquisition module, which is used to respond to receiving a new data table request, obtain the to-be-processed table information of the to-be-processed table in the above-mentioned new data table request, and the library identification information used to identify the database corresponding to the above-mentioned to-be-processed table; a new data table storage module, which is used to store the above-mentioned to-be-processed table in the above-mentioned target node group when it is determined that the above-mentioned node identification information matches the information of the target node group pre-specified by the above-mentioned user; a table node group determination module, which is used to determine the table node group matching the above-mentioned to-be-processed table from the above-mentioned database cluster based on the above-mentioned node identification information when it is determined that the above-mentioned node identification information does not match the information of the target node group pre-specified by the above-mentioned user; a table storage module, which is used to store the above-mentioned to-be-processed table in the above-mentioned table node group.

[0079] According to an embodiment of the present invention, the above-mentioned target node group is determined by the following sub-modules: a cluster sending sub-module, which is used to send multiple database clusters of different types to the above-mentioned user in response to receiving a database addition request from the user, and the above-mentioned multiple database clusters are stored on different node groups; a node sending sub-module, which is used to send a database node group matching the above-mentioned target database cluster to the above-mentioned user in response to receiving a cluster identifier representing the target database cluster from the user; and a node group determination sub-module, which is used to determine the above-mentioned target node group in response to receiving a node group identifier representing the target node group from the user.

[0080] According to an embodiment of the present invention, the above-mentioned device also includes: a first query module, used to obtain the first table identification information and the first data identification to be queried in the above-mentioned first query data request in response to receiving the first query data request; a first data table determination module, used to determine the target data table from the above-mentioned target node group of the above-mentioned database cluster based on the above-mentioned first table identification information; the first query module is used to query the first target query data from the above-mentioned target data table based on the above-mentioned first data identification to be queried.

[0081] According to an embodiment of the present invention, the target node group of the above-mentioned database cluster includes at least one database management node, and the above-mentioned determination module includes: a mapping relationship acquisition submodule, used to obtain the mapping relationship between the table identifier and the target node group, wherein the mapping relationship between the above-mentioned table identifier and the target node group is stored in the cache of the above-mentioned database management node; a data table determination submodule, used to determine the above-mentioned target data table from the above-mentioned target node group of the above-mentioned database cluster based on the mapping relationship between the above-mentioned table identifier and the target node group and the table identifier information in the above-mentioned new data request.

[0082] According to an embodiment of the present invention, the cache of the above-mentioned database management node also stores a hash mapping value group of the target node group, and the above-mentioned hash mapping value group includes a hash value range corresponding to each of the above-mentioned database nodes; the above-mentioned storage module includes: a hash value calculation submodule, which is used to use a hash algorithm to obtain the hash value of the data to be added based on the above-mentioned data to be added; a target storage node determination submodule, which is used to determine the target storage node based on the hash mapping value of the above-mentioned target node group and the hash value of the above-mentioned data to be added, wherein the above-mentioned target storage node includes at least one of the above-mentioned database nodes in the above-mentioned target node group; a data to be added submodule, which is used to store the above-mentioned data to be added in the above-mentioned target storage node.

[0083] According to an embodiment of the present invention, the above-mentioned device also includes: a second query module, which is used to respond to the received second query data request and obtain the second table identification information in the above-mentioned second query data request, the second data identification to be queried and the mapping relationship between the table identification in the above-mentioned cache and the target node group; a sub-target determination module, which is used to determine the sub-target node group based on the above-mentioned second table identification information and the mapping relationship between the above-mentioned table identification and the target node group; a sub-target data table determination module, which is used to determine the sub-target data table from the above-mentioned database node based on the above-mentioned second data identification to be queried for each database node in the above-mentioned sub-target node group, and obtain a sub-target data table group; a candidate query data group determination module, which is used to determine a candidate query data group that matches the above-mentioned second data identification to be queried from the above-mentioned sub-target data table group; a second query module, which is used to obtain the second target query data based on the above-mentioned candidate query data group.

[0084] According to an embodiment of the present invention, any multiple modules among the acquisition module 410, the determination module 420, and the storage module 430 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the acquisition module 410, the determination module 420, and the storage module 430 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 410, the determination module 420, and the storage module 430 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0085] Figure 5 A block diagram of an electronic device suitable for implementing a data slicing processing method according to an embodiment of the present invention is shown.

[0086] like Figure 5As shown, the electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0087] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0088] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that the computer program read therefrom is installed into the storage portion 508 as needed.

[0089] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0090] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0091] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data slicing processing method provided by the embodiment of the present invention.

[0092] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 501. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0093] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0094] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0095] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0096] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0097] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or combinations fall within the scope of the present invention.

[0098] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A data sharding processing method, characterized in that: The method comprises: In response to the received new data request, obtaining table identification information and the data to be added in the new data request; Determine a target data table based on the table identification information in the new data request, wherein the target data table is stored in a target node group of a database cluster, and the target node group includes at least one database node pre-specified by a user in the database cluster; The data to be added is stored in the target data table.

2. The method according to claim 1, characterized in that Before obtaining the table identification information and the newly added data in the newly added data request in response to the received newly added data request, the method further includes: In response to receiving a request for adding a new data table, obtaining to-be-processed table information of the to-be-processed table in the request for adding a new data table and database identification information for identifying a database corresponding to the to-be-processed table; When it is determined that the node identification information matches the information of the target node group pre-specified by the user, storing the to-be-processed table in the target node group; In the case where it is determined that the node identification information does not match the information of the target node group pre-specified by the user, determining a table node group matching the table to be processed from the database cluster based on the node identification information; The table to be processed is stored in the table node group.

3. The method according to claim 1, characterized in that The target node group is determined in the following manner: In response to receiving a database addition request from a user, sending a plurality of database clusters of different types to the user, wherein the plurality of database clusters are stored on different node groups; In response to receiving a cluster identifier representing a target database cluster from a user, sending a database node group matching the target database cluster to the user; as well as In response to receiving a node group identifier representing a target node group from a user, the target node group is determined.

4. The method according to claim 1, characterized in that: The method further comprises: In response to receiving a first query data request, obtaining first table identification information and a first to-be-queried data identification in the first query data request; Based on the first table identification information, determining a target data table from the target node group of the database cluster; Based on the first to-be-queried data identifier, first target query data is obtained by querying from the target data table.

5. The method according to claim 1, characterized in that The target node group of the database cluster includes at least one database management node, and determining the target data table from the target node group of the database cluster based on the table identification information in the new data request includes: Acquire a mapping relationship between a table identifier and a target node group, wherein the mapping relationship between the table identifier and the target node group is stored in a cache of the database management node; Based on the mapping relationship between the table identifier and the target node group and the table identifier information in the new data request, the target data table is determined from the target node group of the database cluster.

6. The method according to claim 5, characterized in that The cache of the database management node also stores a hash mapping value group of the target node group, wherein the hash mapping value group includes a mapping range corresponding to each of the database nodes; The storing the to-be-added data in the target data table includes: Using a hash algorithm, a hash value of the data to be added is obtained based on the data to be added; Determine a target storage node based on the hash mapping value of the target node group and the hash value of the data to be added, wherein the target storage node includes at least one of the database nodes in the target node group; The data to be added is stored in the target storage node.

7. The method according to claim 5, characterized in that The method further comprises: In response to the received second query data request, obtaining the second table identification information in the second query data request, the second to-be-queried data identification, and the mapping relationship between the table identification in the cache and the target node group; Determine a sub-target node group based on the second table identification information and a mapping relationship between the table identification and the target node group; For each database node in the sub-target node group, determining a sub-target data table from the database node based on the second to-be-queried data identifier to obtain a sub-target data table group; Determine a candidate query data group matching the second to-be-queried data identifier from the sub-target data table group; Based on the candidate query data group, second target query data is obtained.

8. A data sharding processing device, characterized in that: The device comprises: An acquisition module, configured to, in response to a received request for newly added data, acquire table identification information and data to be newly added in the request for newly added data; A determination module, configured to determine a target data table based on the table identification information in the new data request, wherein the target data table is stored in a target node group of a database cluster, and the target node group includes at least one database node pre-specified by a user in the database cluster; The storage module is used to store the data to be added into the target data table.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images