A data processing method, device and system for distributed database

Abstract

The embodiment of the present invention provides a data processing method, device and system for a distributed database. The method is applied to a management server in a distributed database system, and the management server records the file identifiers of multiple sub-files stored in a distributed manner for a target file, as well as the corresponding relationship between the file identifier of each sub-file and the sharding node where the sub-file is located; the method includes: determining the target file identifier of a sub-file to be migrated among multiple sub-files; based on the target file identifier and the corresponding relationship, determining the sharding node where the sub-file to be migrated is located as the initial node; migrating the sub-file with the target file identifier in the initial node to the expansion node, which is a sharding node pre-set in the distributed data system when the system is expanded. Through this solution, the problem of large computing overhead and log overhead in the process of distributed database expansion can be solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device and system for a distributed database. Background Art

[0002] In a distributed database system, the management server stores any file in multiple shard nodes, thus forming a storage format of logical file + partition file. For example, a table file is stored in multiple shard nodes, and each shard node stores part of the file content of the table file.

[0003] Elastic expansion is an essential basic function of distributed database systems. After adding expansion nodes in a distributed database system, that is, adding new sharding nodes, the data content of the files is redistributed to complete the expansion. Specifically, the data content of the files in at least one sharding node is migrated to the newly added expansion node.

[0004] In the related art, after determining the query conditions for filtering the data to be migrated, the management server reads the data that meets the query conditions from the shard nodes at the row level, and then writes the read data to the newly added expansion node.

[0005] It can be seen that since the data that meets the query conditions is read at the row level, the computing overhead and log overhead during the expansion process are relatively large. Summary of the invention

[0006] The purpose of the embodiments of the present invention is to provide a data processing method, device and system for a distributed database to solve the problem of large computing overhead and log overhead in the expansion process of the prior art, and at the same time flexibly expand the distributed database system to avoid waste of hardware resources. The specific technical solution is as follows:

[0007] The purpose of the embodiments of the present invention is to provide a data processing method, device and system for a distributed database to solve the problem of large computing overhead and log overhead in the expansion process of the prior art. The specific technical solution is as follows:

[0008] In a first aspect, an embodiment of the present invention provides a data processing method for a distributed database, which is applied to a management server in a distributed database system, wherein the management server records file identifiers of a plurality of sub-files of a target file that are distributedly stored, and a correspondence between the file identifier of each sub-file and the shard node where the sub-file is located; the method comprises:

[0009] Determine a target file identifier of a subfile to be migrated among the multiple subfiles;

[0010] Based on the target file identifier and the corresponding relationship, determine the shard node where the sub-file to be migrated is located as the initial node;

[0011] The subfile with the target file identifier in the initial node is migrated to the expansion node, and the expansion node is a shard node pre-set in the distributed data system when the system is expanded.

[0012] Optionally, after the step of migrating the subfile having the target file identifier in the initial node to the expansion node, the method further includes:

[0013] The shard node corresponding to the target file identifier in the corresponding relationship is changed to the expansion node.

[0014] Optionally, the migrating the subfile having the target file identifier in the initial node to the expansion node includes:

[0015] A migration instruction for the sub-file with the target file identifier is sent to the initial node, so that after receiving the migration instruction, the initial node determines the expansion node to which the sub-file with the file identifier is to be migrated, and transfers the sub-file with the target file identifier to the expansion node through a file transfer method.

[0016] Optionally, the method further comprises: after the transmission of the sub-file with the target file identifier is completed, detecting whether there is incremental data for the sub-file with the target file identifier in the initial node relative to the sub-file with the target file identifier in the expansion node;

[0017] If incremental data exists, the incremental data is written into the subfile with the target file identifier in the initial node.

[0018] Optionally, the determining a target file identifier of a sub-file to be migrated among the multiple files includes:

[0019] According to the principle of load balancing, determine the target shard node from which the sub-files are to be migrated, and the target number of sub-files to be migrated from the target shard node;

[0020] From the corresponding relationship, the file identifier corresponding to the target shard node is determined, and the target number of file identifiers are selected from the determined file identifiers to obtain the target file identifiers of the sub-files to be migrated among the multiple sub-files.

[0021] Optionally, the method further comprises: after receiving an access request for the target file, determining, according to a predetermined file splitting rule, a file identifier of a sub-file to be accessed by the access request as an identifier to be utilized; wherein the file splitting rule is a rule for splitting the target file into the plurality of sub-files for distributed storage;

[0022] Sending a transaction processing request to the shard node corresponding to the to-be-utilized identifier in the corresponding relationship, so that the shard node corresponding to the to-be-utilized identifier responds to the transaction processing request;

[0023] The transaction processing request is used to execute the request content in the access request on the subfile having the identifier to be utilized.

[0024] In a second aspect, an embodiment of the present invention provides a distributed database system, comprising: a management server and a plurality of shard nodes, wherein the management server records file identifiers of a plurality of sub-files stored in a distributed manner for a target file, and a corresponding relationship between the file identifier of each sub-file and the shard node where the sub-file is located;

[0025] Each shard node is used to store a sub-file for the target file;

[0026] The management server is used to determine the target file identifier of the sub-file to be migrated among the multiple sub-files; based on the target file identifier and the corresponding relationship, determine the shard node where the sub-file to be migrated is located as the initial node; migrate the sub-file with the target file identifier in the initial node to the expansion node, and the expansion node is a shard node pre-set in the distributed data system when the system is expanded.

[0027] Optionally, the management server is further used for:

[0028] After the sub-file with the target file identifier in the initial node is migrated to the expansion node, the shard node corresponding to the target file identifier in the corresponding relationship is changed to the expansion node.

[0029] Optionally, the management server migrates the subfile with the target file identifier in the initial node to the expansion node, specifically:

[0030] Sending a migration instruction for the sub-file having the target file identifier to the initial node;

[0031] The initial node is further configured to determine, after receiving the migration instruction, an expansion node to which the subfile with the target file identifier is to be migrated, and transfer the subfile with the target file identifier to the expansion node by means of file transfer.

[0032] Optionally, the management server is further used for:

[0033] After the transmission of the sub-file with the target file identifier is completed, detecting whether there is incremental data for the sub-file with the target file identifier in the initial node relative to the sub-file with the target file identifier in the expansion node;

[0034] If there is incremental data, the incremental data is written into the sub-file with the target file identifier in the expansion node.

[0035] Optionally, the management server determines a target file identifier of a sub-file to be migrated among the multiple sub-files, specifically:

[0036] According to the principle of load balancing, determine the target shard node from which the sub-files are to be migrated, and the target number of sub-files to be migrated from the target shard node;

[0037] From the corresponding relationship, the file identifier corresponding to the target shard node is determined, and the target number of file identifiers are selected from the determined file identifiers to obtain the target file identifiers of the sub-files to be migrated among the multiple sub-files.

[0038] Optionally, the management server is further used for:

[0039] After receiving an access request for the target file, determine the file identifier of the sub-file to be accessed by the access request as the identifier to be used according to a predetermined file splitting rule; send a transaction processing request to the sharding node corresponding to the identifier to be used in the corresponding relationship; wherein the file splitting rule is a rule for splitting the target file into the plurality of sub-files for distributed storage; and the transaction processing request is used to execute the request content in the access request on the sub-file with the identifier to be used;

[0040] The shard node corresponding to the identifier to be utilized is also used to respond to the transaction processing request.

[0041] In a third aspect, an embodiment of the present invention provides a data processing device for a distributed database, characterized in that it is applied to a management server in a distributed database system, wherein the management server records file identifiers of multiple sub-files of a target file that are distributed and stored, and a corresponding relationship between the file identifier of each sub-file and the shard node where the sub-file is located; the device comprises:

[0042] A file identification determination module, used to determine a target file identification of a sub-file to be migrated among the multiple sub-files;

[0043] An initial node determination module, used to determine, based on the target file identifier and the corresponding relationship, the shard node where the sub-file to be migrated is located as the initial node;

[0044] The file migration module is used to migrate the sub-file with the target file identifier in the initial node to the expansion node, and the expansion node is a shard node pre-set in the distributed data system when the system is expanded.

[0045] In a fourth aspect, an embodiment of the present invention provides a management server, characterized in that it includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0046] Memory, used to store computer programs;

[0047] The processor is used to implement the steps of the data processing method for a distributed database provided in the first aspect when executing the program stored in the memory.

[0048] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method for a distributed database provided in the first aspect above is implemented.

[0049] An embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned data processing methods for a distributed database.

[0050] Beneficial effects of the embodiments of the present invention:

[0051] In an embodiment of the present invention, the management server records the file identifiers of multiple sub-files of the target file that are distributed and stored, as well as the corresponding relationship between the file identifier of each sub-file and the shard node where the sub-file is located. In this way, when redistributing data for the target file, the management server can determine the target file identifier of the sub-file to be migrated; then, based on the target file identifier and the corresponding relationship, determine the shard node where the sub-file to be migrated is located as the initial node; and migrate the sub-file with the target file identifier in the initial node to the expansion node. It can be seen that in this solution, the management server can select the sub-file to be migrated by recording the file identifiers of each sub-file, and perform file migration on the sub-file to be migrated, without reading the data content at the row level. Therefore, compared with the related art, this solution can solve the problem of large computing overhead and log overhead in the process of expanding the distributed database.

[0052] In addition, in this solution, the number of expansion nodes can be set according to actual needs. Then, after adding the set number of expansion nodes, some sub-files can be migrated to each expansion node by sub-file migration, thereby achieving effective data redistribution. It can be seen that this solution can flexibly expand the distributed database system and avoid the waste of hardware resources while ensuring effective data redistribution. For example: In this solution, the distributed database system can be expanded in a step-by-step manner, thereby greatly reducing the hardware cost compared to doubling the expansion.

[0053] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0055] Figure 1 A flow chart of a data processing method for a distributed database provided by an embodiment of the present invention;

[0056] FIG. 2( a ) is a schematic diagram of sub-file distribution before a distributed database is expanded using a data processing method for a distributed database provided by an embodiment of the present invention;

[0057] FIG2( b ) is a schematic diagram of sub-file distribution after the distributed database is expanded using a data processing method for a distributed database provided by an embodiment of the present invention;

[0058] Figure 3 Another flow chart of a data processing method for a distributed database provided by an embodiment of the present invention;

[0059] Figure 4 Another flow chart of a data processing method for a distributed database provided by an embodiment of the present invention;

[0060] Figure 5 A schematic diagram of the structure of a data processing system for a distributed database provided by an embodiment of the present invention;

[0061] Figure 6 A schematic diagram of the structure of a data processing device for a distributed database provided by an embodiment of the present invention;

[0062] Figure 7A schematic diagram of the structure of a management server provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field based on this application belong to the scope of protection of the present invention.

[0064] In the related art, the management server in the distributed database system distributes and stores any file in multiple shard nodes based on a predetermined splitting rule to form a distributed database. The splitting rule is a rule for determining the shard nodes where the file content of the file is located. In this way, for the access process of the distributed database, if the management server receives an access request for a certain file, such as a write request or a read request, it can determine the shard node where the file content to be accessed is located based on the splitting rule, and send a transaction processing request to the determined shard node, wherein the transaction processing request is used to instruct the execution of the request content in the access request. For the above-mentioned method, what is visible to the management server are the various shard nodes, while the storage form of the data content of any file in the shard node is invisible to the management server. For example, the management server cannot know whether the shard node stores the data content of a certain file in one or more files.

[0065] Correspondingly, since what is visible to the management server is the shard node, in the related technology, when the distributed database is expanded, the management server needs to determine the query conditions for filtering the data to be migrated, and after determining the query conditions for filtering the data to be migrated, read the data that meets the query conditions from the shard node at the row level, and then write the read data to the newly added expansion node.

[0066] However, since the management server reads the data that meets the query conditions line by line, the computing overhead and log overhead in the expansion process will be large. Therefore, in order to reduce the computing overhead and log overhead of the distributed database in the expansion process, the embodiment of the present invention provides a data processing method for the distributed database. Among them, in the related art, the computing overhead and log overhead in the expansion process may include but are not limited to: RedoLog (i.e., redo log file), UndoLog (i.e., log file for recording logical changes of data), binlog (i.e., log file in binary format for recording changes of data blocks), and Double Write (i.e., a writing method in which part of the data exists in the memory and the other part exists in the shared table space of the disk when writing data).

[0067] First, a data processing method for a distributed database provided by an embodiment of the present invention is introduced below.

[0068] A data processing method for a distributed database provided by an embodiment of the present invention is applied to a management server in a distributed database system, and the distributed database system also includes multiple sharding nodes. The management server is used to manage each sharding node and use the sharding node to respond to access requests for target files sent by external access devices; and in specific applications, the management server can also be called a proxy server, a transaction server or a management node, etc. The sharding node is used to store part of the data content of the target file, and the sharding node can include at least one storage node. If the sharding node includes multiple storage nodes, the multiple storage nodes include master-slave nodes, thereby ensuring storage stability and security. It can be understood that for a distributed database system, the number of target files can be one or more, and the target file belongs to a logical file, that is, a file visible to external users; and the target file can be a data table, but it is certainly not limited to this.

[0069] In order to solve the problems of the prior art, in an embodiment of the present invention, the management server records the file identifiers of multiple sub-files of the target file that are distributed and stored, as well as the corresponding relationship between the file identifier of each sub-file and the sharding node where the sub-file is located. In other words, in this solution, the sub-files of the target file in the sharding node are visible to the management server, that is, the management server can know which content is stored in which sub-file and located in which sharding node.

[0070] It should be noted that in order for the management server to record the above content, a file splitting rule can be pre-set for the target file. The target file can be split into multiple sub-files for distributed storage through the file splitting rule, and each sub-file can be deployed in multiple shard nodes. Correspondingly, in the process of accessing the target file, the management server can use the file splitting rule to respond to the access request, and then perform a specific access process in combination with the following embodiments. Exemplarily, the file splitting rule can be: split the file according to the specified field in the target file, the number of hash values ​​that can be calculated for each field value under the specified field is the number of multiple sub-files, and different hash values ​​correspond to different sub-files; or, the number of each field value under the specified field corresponds to different sub-files, and different field values ​​correspond to different sub-files; and so on.

[0071] In addition, it can be understood that the multiple sub-files in distributed storage are all physical files. No matter how many sub-files there are, only one logical file, ie, the target file, is visible to the outside.

[0072] The data processing method for a distributed database provided by an embodiment of the present invention may include the following steps:

[0073] Determine a target file identifier of a subfile to be migrated among the multiple subfiles;

[0074] Based on the target file identifier and the corresponding relationship, determine the shard node where the sub-file to be migrated is located as the initial node;

[0075] Migrate the sub-files with the target file identifier in the initial node to the expanded node.

[0076] It can be seen that in this solution, the management server can select the sub-files to be migrated by recording the file identifiers of each sub-file, and perform file migration on the sub-files to be migrated, without having to read the data content at the row level. Therefore, compared with related technologies, this solution can solve the problem of large computing overhead and log overhead during the expansion of distributed databases.

[0077] The following describes a data processing method for a distributed database provided by an embodiment of the present invention in conjunction with the accompanying drawings.

[0078] like Figure 1 As shown, a data processing method for a distributed database provided by an embodiment of the present invention may include the following steps:

[0079] S101, determining a target file identifier of a sub-file to be migrated among multiple sub-files;

[0080] When the distributed database reaches the trigger condition for expansion, it is necessary to add an expansion node in the distributed database system and redistribute the data content of the file to complete the expansion. Specifically, the data content of the file in at least one shard node is migrated to the newly added expansion node. The trigger condition can be set according to actual needs. For example, the expansion condition can be that the storage capacity of the distributed database reaches a specified amount of data, or reaches a predetermined time point, etc.

[0081] In this embodiment, after adding an expansion node, when redistributing data, the target file identifier of the subfile to be migrated among multiple subfiles can be first determined, and then the target file identifier is used to complete the expansion by migrating the subfile. The number of target file identifiers can be one or more, and the processing process of migrating the subfile based on each target file identifier is the same.

[0082] There are multiple implementation methods for determining the target file identifier of the sub-file to be migrated.

[0083] Exemplarily, in one implementation, determining the target file identifier of a subfile to be migrated from a plurality of subfiles may include: outputting a human-computer interaction interface, in which the number of subfiles of the target file stored in each sharding node is displayed; obtaining the target sharding node selected by a person for migrating the subfiles based on the human-computer interaction interface, and the target number of subfiles to be migrated from the target sharding node; determining the file identifier corresponding to the target sharding node from the corresponding relationship, selecting the target number of file identifiers from the determined file identifiers, and obtaining the target file identifier of the subfile to be migrated from the plurality of subfiles. It can be seen that in this implementation, the relevant person can select which sharding nodes to migrate the subfiles from according to actual needs.

[0084] Exemplarily, in another implementation, determining the target file identifier of a subfile to be migrated from a plurality of subfiles may include: determining the target shard node from which the subfile is to be migrated and the target number of subfiles to be migrated from the target shard node according to the load balancing principle; determining the file identifier corresponding to the target shard node from the corresponding relationship, selecting the target number of file identifiers from the determined file identifiers, and obtaining the target file identifier of the subfile to be migrated from the plurality of subfiles. It can be seen that in this implementation, the management server may determine which shard nodes to migrate the subfiles from based on the load balancing principle, wherein the load balancing principle enables the number of subfiles of the target file stored in each shard node to be balanced.

[0085] S102, based on the target file identifier and the corresponding relationship, determining the shard node where the sub-file to be migrated is located as the initial node;

[0086] The management server records the correspondence between the file identifier of each sub-file and the shard node where the sub-file is located. Therefore, after the target file identifier of the sub-file to be migrated is determined, it is possible to determine on which shard node the sub-file to be migrated is located based on the target file identifier and the correspondence, thereby obtaining the initial node.

[0087] S103, migrating the sub-file with the target file identifier in the initial node to the expansion node, where the expansion node is a shard node pre-set in the distributed data system when the system is expanded.

[0088] In this embodiment, in order to solve the problems of the prior art, the capacity is expanded by migrating sub-files. Therefore, after obtaining the target file identifier and determining the initial node, the sub-file with the target file identifier in the initial node can be migrated to the expanded node.

[0089] Furthermore, in order to ensure that the expanded node is used to respond to the access request for the migrated sub-files in the future, the management server can change the shard node corresponding to the target file identifier in the corresponding relationship to the expanded node after migrating the sub-file with the target file identifier in the initial node to the expanded node. Specifically, the management server records the corresponding relationship between the file identifier of each sub-file and the node where the sub-file is located. The corresponding relationship can be the distribution information of the nodes where each sub-file is located recorded by the management server. After migrating the sub-file with the target file identifier to the expanded node, the distribution information of the migrated sub-file of the initial node is deleted, and the distribution information of the migrated sub-file in the expanded node is added.

[0090] There are multiple implementation methods for migrating the subfile with the target file identifier in the initial node to the expansion node.

[0091] Exemplarily, in one implementation, the management server may send a query transaction request to the initial node, read the entire contents of the subfile with the target file identifier from the initial node, and then write the entire contents into the subfile with the target file identifier in the expansion node, thereby achieving the migration of the subfile with the target file identifier in the initial node to the expansion node. In addition, after reading the entire contents of the subfile with the target file identifier from the initial node, until the migration is completed, the subfile with the target file identifier in the initial node may generate incremental data. For this reason, after writing the entire contents read into the expansion node, the management server may read incremental data from the initial node, and synchronize the incremental data read into the subfile with the target file identifier in the expansion node.

[0092] Other implementations of migrating the subfile having the target file identifier in the initial node to the expanded node will be introduced later in conjunction with other embodiments.

[0093] In order to facilitate understanding of the data processing method for distributed databases of the present invention, it is introduced below in conjunction with Figure 2(a) and Figure 2(b). Among them, Figure 2(a) is a distribution diagram of multiple sub-files before expansion, and Figure 2(b) is a distribution diagram of multiple sub-files after expansion. As shown in Figures 2(a) and (b), shard node 1 and shard node 2 are shard nodes before expansion, and shard node 3 is a pre-added expansion node; in Figure 2(a), the management server splits the target file into sub-files 1-8 according to a predetermined file splitting rule, and sub-files 1-4 are stored in shard node 1, and sub-files 5-8 are stored in shard node 2, and the management server records the file identifier of each sub-file and the corresponding relationship between the file identifier of each sub-file and the shard node where the sub-file is located. At this time, the management server records the distribution information of each sub-file:

[0094] Tblname:Table

[0095] g1:[Table_1,Table_2,Table_3,Table_4]

[0096] g2:[Table_5,Table_6,Table_7,Table_8]

[0097] Among them, g1 and g2 represent shard nodes 1 and 2, and Table_1 to Table8 represent the sub-files after splitting.

[0098] When capacity expansion is required, according to the load balancing principle, a sub-file is migrated out from shard node 1 and shard node 2 respectively, and the file identifier of sub-file 2 in shard node 1 and the file identifier of sub-file 5 in shard node 2 are both used as target file identifiers; then, the management server can issue a migration instruction carrying the file identifier of sub-file 2 to shard node 1 as the initial node, and issue a migration instruction carrying the file identifier of sub-file 5 to shard node 2 as the initial node; accordingly, shard node 1 transfers sub-file 2 to shard node 3 through file transfer, and shard node 2 transfers sub-file 5 to shard node 3 through file transfer, forming a distribution diagram of sub-files as shown in Figure 2(b).

[0099] The management server adds or deletes the sub-file distribution information of the corresponding shard. The distribution information of each sub-file recorded after addition or deletion is:

[0100] Tblname:Table

[0101] g1:[Table_1,Table_3,Table_4]

[0102] g2:[Table_6,Table_7,Table_8]

[0103] g3:[Table_2,Table_5]

[0104] Among them, g3 is a pre-added expansion node, that is, shard node 3, and Table_2 and Table_5 are the target sub-files of migration, that is, sub-file 2 and sub-file 5.

[0105] In the embodiment of the present invention, the management server can select the sub-files to be migrated by recording the file identifiers of each sub-file, and perform file migration on the sub-files to be migrated, without having to read the data content at the row level. Therefore, compared with the related art, this solution can solve the problem of large computing overhead and log overhead in the process of distributed database expansion.

[0106] In addition, in this solution, the number of expansion nodes can be set according to actual needs. Then, after adding the set number of expansion nodes, some sub-files can be migrated to each expansion node by sub-file migration, thereby achieving effective data redistribution. It can be seen that this solution can flexibly expand the distributed database system and avoid the waste of hardware resources while ensuring effective data redistribution. For example: In this solution, the distributed database system can be expanded in a step-by-step manner, thereby greatly reducing the hardware cost compared to doubling the expansion.

[0107] Alternatively, in another embodiment of the present invention, Figure 3 As shown, the above S103 may include the following steps:

[0108] S1031, sending a migration instruction for the sub-file with the target file identifier to the initial node, so that after receiving the migration instruction, the initial node determines the expansion node to which the sub-file with the target file identifier is to be migrated, and transfers the sub-file with the target file identifier to the expansion node through a file transfer method.

[0109] In this embodiment, the management server instructs the initial node to migrate the sub-file with the target file identifier by issuing a migration instruction, wherein the migration instruction may at least include the target file identifier, thereby indicating the sub-file to be migrated from the initial node through the target file identifier.

[0110] There are multiple ways to determine the expansion node to which the subfile with the target file identifier is to be migrated. Exemplarily, in one implementation, the migration instruction also carries the communication identification information of the expansion node, so that the initial node can determine the expansion node to which the subfile with the target file identifier is to be migrated through the migration instruction. Exemplarily, in another implementation, the communication identification information of the expansion node is pre-configured in the initial node, so that the initial node can determine the expansion node to which the subfile with the target file identifier is to be migrated from the pre-configured information.

[0111] In addition, after the initial node and the expansion node establish a wired or wireless communication connection, the sub-file with the target file identifier is transferred to the expansion node through a file transfer method. The initial node and the expansion node may be located in the same local area network, but it is not limited to this. In addition, the file transfer method may include but is not limited to: FTP (File Transfer Protocol).

[0112] In this embodiment, the management server can select the sub-files to be migrated by recording the file identifiers of each sub-file, and perform file migration on the sub-files to be migrated by issuing migration instructions, without having to read the data content at the row level. Therefore, compared with related technologies, this solution can solve the problem of large computing overhead and log overhead during the expansion of distributed databases.

[0113] In addition, when the initial node transfers the sub-file with the target file identifier to the expansion node, the sub-file with the target file identifier in the initial node may generate incremental data. Figure 3 In another embodiment of the present invention, as shown in Figure 4 As shown, the method may further include the following S104-S105:

[0114] S104, after the transmission of the sub-file with the target file identifier is completed, detecting whether there is incremental data for the sub-file with the target file identifier in the initial node relative to the sub-file with the target file identifier in the expansion node;

[0115] There are multiple implementation methods for detecting whether there is incremental data, which are not limited in the embodiments of the present invention. Exemplarily, in one implementation method, the method for detecting whether there is incremental data may include: based on designated log data, detecting whether there is a write request for a sub-file with a target file identifier after the transmission moment of the sub-file with the target file identifier; if so, determining that there is incremental data. The designated log data may include: log data generated about a sub-file with a target file identifier during the transmission process of the sub-file with the target file identifier.

[0116] S105: If there is incremental data, write the incremental data into the sub-file with the target file identifier in the expansion node.

[0117] Exemplarily, in one implementation, the management server may query the updated data content during the transmission of the subfile with the target file identifier from the subfile with the target file identifier in the initial node by sending a query transaction request for the subfile with the target file identifier to the initial node, thereby writing the queried data content to the subfile with the target file identifier in the expansion node. The updated data content is the incremental data, which may specifically include: the content added to the subfile before migration, and the content modified to the subfile before migration. In addition, the query condition in the query transaction request may also be determined based on the above-mentioned specified log data.

[0118] In this embodiment, compared with related technologies, this solution can not only solve the problem of large computing overhead and log overhead during the expansion of distributed databases, but also ensure the integrity of redistributed data.

[0119] In addition, optionally, in another embodiment of the present invention, a data processing method for a distributed database may further include the following steps A1-A2:

[0120] Step A1, after receiving an access request for a target file, determining the file identifier of a sub-file to be accessed by the access request according to a predetermined file splitting rule as the identifier to be used; wherein the file splitting rule is a rule for splitting the target file into a plurality of sub-files for distributed storage;

[0121] The access request may be a write request or a read request, and the write request is used to insert data or modify data.

[0122] Exemplarily, the target file is a data table file. After receiving a write request for the data table file, the file identifier of the sub-file to which the row record to be written in the write request belongs is determined according to a predetermined file splitting rule as the identifier to be used; after receiving a read request for the data table file, the file identifier of the sub-file to which the row record to be read belongs is determined according to a predetermined file splitting rule as the identifier to be used.

[0123] Exemplarily, the file splitting rule may be: split the file according to the specified field in the data table file, the number of hash values ​​of each field value under the specified field is the number of multiple sub-files, and different hash values ​​correspond to different sub-files; accordingly, after receiving an access request for the data table file, determine the hash value of the field value of the specified field in the access request, and through the hash value, determine the file identifier of the sub-file to which the row record to be accessed belongs.

[0124] Step A2: Send a transaction processing request to the shard node corresponding to the identifier to be used in the corresponding relationship, so that the shard node corresponding to the identifier to be used responds to the transaction processing request.

[0125] The transaction processing request is used to execute the request content in the access request on the sub-file with the identifier to be used.

[0126] In this embodiment, compared with related technologies, this solution can not only solve the problem of large computing overhead and log overhead during the expansion of distributed databases, but also ensure effective access to target files.

[0127] Corresponding to the above method embodiment, the embodiment of the present invention also provides a distributed database system, such as Figure 5 As shown, the system may include: a management server 510 and multiple sharding nodes 520, wherein the management server 510 records file identifiers of multiple sub-files in distributed storage for a target file, and a corresponding relationship between the file identifier of each sub-file and the sharding node 520 where the sub-file is located;

[0128] Each shard node 520 is used to store a sub-file for the target file;

[0129] The management server 510 is used to determine the target file identifier of the sub-file to be migrated among the multiple sub-files; based on the target file identifier and the corresponding relationship, determine the shard node 520 where the sub-file to be migrated is located as the initial node; migrate the sub-file with the target file identifier in the initial node to the expansion node, and the expansion node is a shard node pre-set in the distributed data system when the system is expanded.

[0130] It can be seen that in this solution, the management server can select the sub-files to be migrated by recording the file identifiers of each sub-file, and perform file migration on the sub-files to be migrated, without having to read the data content at the row level. Therefore, compared with related technologies, this solution can solve the problem of large computing overhead and log overhead during the expansion of distributed databases.

[0131] Optionally, the management server 510 is further used for:

[0132] After the sub-file with the target file identifier in the initial node is migrated to the expansion node, the shard node corresponding to the target file identifier in the corresponding relationship is changed to the expansion node.

[0133] Optionally, the management server 510 migrates the subfile with the target file identifier in the initial node to the expansion node, specifically:

[0134] Sending a migration instruction for the sub-file having the target file identifier to the initial node;

[0135] The initial node is further configured to determine, after receiving the migration instruction, an expansion node to which the subfile with the target file identifier is to be migrated, and transfer the subfile with the target file identifier to the expansion node by means of file transfer.

[0136] Optionally, the management server 510 is further used for:

[0137] After the transmission of the sub-file with the target file identifier is completed, detecting whether there is incremental data for the sub-file with the target file identifier in the initial node relative to the sub-file with the target file identifier in the expansion node;

[0138] If there is incremental data, the incremental data is written into the sub-file with the target file identifier in the expansion node.

[0139] Optionally, the management server determines a target file identifier of a sub-file to be migrated among the multiple sub-files, specifically:

[0140] According to the principle of load balancing, determine the target shard node from which the sub-files are to be migrated, and the target number of sub-files to be migrated from the target shard node;

[0141] From the corresponding relationship, the file identifier corresponding to the target shard node is determined, and the target number of file identifiers are selected from the determined file identifiers to obtain the target file identifiers of the sub-files to be migrated among the multiple sub-files.

[0142] Optionally, the management server is further used for:

[0143] After receiving an access request for the target file, determine the file identifier of the sub-file to be accessed by the access request as the identifier to be used according to a predetermined file splitting rule; send a transaction processing request to the sharding node corresponding to the identifier to be used in the corresponding relationship; wherein the file splitting rule is a rule for splitting the target file into the plurality of sub-files for distributed storage; and the transaction processing request is used to execute the request content in the access request on the sub-file with the identifier to be used;

[0144] The shard node corresponding to the identifier to be utilized is also used to respond to the transaction processing request.

[0145] For the relevant description content of each device in the system, please refer to the corresponding content in the above method embodiment, which will not be repeated here.

[0146] Corresponding to the above method embodiment, the embodiment of the present invention further provides a data processing device for a distributed database, which is applied to a management server in a distributed database system, wherein the management server records file identifiers of a plurality of sub-files stored in a distributed manner for a target file, and a corresponding relationship between the file identifier of each sub-file and the shard node where the sub-file is located; Figure 6 As shown, the device comprises:

[0147] A file identification determination module 610, configured to determine a target file identification of a sub-file to be migrated among the multiple sub-files;

[0148] An initial node determination module 620, configured to determine, based on the target file identifier and the corresponding relationship, a shard node where the subfile to be migrated is located as the initial node;

[0149] The file migration module 630 is used to migrate the sub-file with the target file identifier in the initial node to the expansion node, and the expansion node is a shard node pre-set in the distributed data system when the system is expanded.

[0150] It can be seen that in this solution, the management server can select the sub-files to be migrated by recording the file identifiers of each sub-file, and perform file migration on the sub-files to be migrated, without having to read the data content at the row level. Therefore, compared with related technologies, this solution can solve the problem of large computing overhead and log overhead during the expansion of distributed databases.

[0151] Optionally, the device further comprises:

[0152] The changing module is used to change the shard node corresponding to the target file identifier in the corresponding relationship to the expanded node after the file migration module 630 migrates the subfile with the target file identifier in the initial node to the expanded node.

[0153] Optionally, the file migration module 630 is specifically used to:

[0154] A migration instruction for the sub-file with the target file identifier is sent to the initial node, so that after receiving the migration instruction, the initial node determines the expansion node to which the sub-file with the target file identifier is to be migrated, and transfers the sub-file with the target file identifier to the expansion node through a file transfer method.

[0155] Optionally, the device further comprises:

[0156] A detection module, configured to detect whether there is incremental data in the sub-file with the target file identifier in the initial node relative to the sub-file with the target file identifier in the expansion node after the transmission of the sub-file with the target file identifier is completed;

[0157] A writing module is used to write the incremental data into the sub-file with the target file identifier in the expansion node if there is incremental data.

[0158] Optionally, the file identification determination module 610 includes:

[0159] A determination submodule is used to determine, according to the principle of load balancing, a target sharding node from which the subfiles are to be migrated and a target number of subfiles to be migrated from the target sharding node;

[0160] A selection submodule is used to determine the file identifier corresponding to the target shard node from the corresponding relationship, select the target number of file identifiers from the determined file identifiers, and obtain the target file identifier of the subfile to be migrated from the multiple subfiles.

[0161] Optionally, the device further comprises:

[0162] A determination module, configured to, after receiving an access request for the target file, determine, according to a predetermined file splitting rule, a file identifier of a sub-file to be accessed by the access request as an identifier to be utilized; wherein the file splitting rule is a rule for splitting the target file into the plurality of sub-files for distributed storage;

[0163] A request issuing module, used for issuing a transaction processing request to the shard node corresponding to the to-be-utilized identifier in the corresponding relationship, so that the shard node corresponding to the to-be-utilized identifier responds to the transaction processing request;

[0164] The transaction processing request is used to execute the request content in the access request on the subfile having the identifier to be utilized.

[0165] The embodiment of the present invention also provides a management server such as Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0166] Memory 703, used for storing computer programs;

[0167] The processor 701 is used to implement the steps of any method provided above in the embodiment of the present invention when executing the program stored in the memory 703.

[0168] The communication bus mentioned in the above management server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0169] The communication interface is used for communication between the above electronic device and other devices.

[0170] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0171] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0172] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data processing methods for a distributed database are implemented.

[0173] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes any one of the data processing methods for a distributed database in the above embodiments.

[0174] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0175] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0176] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0177] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data processing method for a distributed database, characterized in that: A management server applied to a distributed database system, wherein the management server records file identifiers of a plurality of distributedly stored sub-files for a target file, and a corresponding relationship between the file identifier of each sub-file and the shard node where the sub-file is located; the method comprises: Determine a target file identifier of a subfile to be migrated among the multiple subfiles; Based on the target file identifier and the corresponding relationship, determine the shard node where the sub-file to be migrated is located as the initial node; Migrating the subfile with the target file identifier in the initial node to the expansion node, where the expansion node is a shard node pre-set in the distributed database system when the system is expanded; The determining of the target file identifier of the sub-file to be migrated among the multiple files includes: According to the principle of load balancing, determine the target shard node from which the sub-files are to be migrated, and the target number of sub-files to be migrated from the target shard node; From the corresponding relationship, the file identifier corresponding to the target shard node is determined, and the target number of file identifiers are selected from the determined file identifiers to obtain the target file identifiers of the sub-files to be migrated among the multiple sub-files.

2. The method according to claim 1, characterized in that After the step of migrating the subfile having the target file identifier in the initial node to the expansion node, the method further includes: The shard node corresponding to the target file identifier in the corresponding relationship is changed to the expansion node.

3. The method according to claim 1 or 2, characterized in that: The step of migrating the subfile having the target file identifier in the initial node to the expansion node includes: A migration instruction for the sub-file with the target file identifier is sent to the initial node, so that after receiving the migration instruction, the initial node determines the expansion node to which the sub-file with the target file identifier is to be migrated, and transfers the sub-file with the target file identifier to the expansion node through a file transfer method.

4. The method according to claim 3, characterized in that The method further comprises: After the transmission of the sub-file with the target file identifier is completed, detecting whether there is incremental data for the sub-file with the target file identifier in the initial node relative to the sub-file with the target file identifier in the expansion node; If there is incremental data, the incremental data is written into the sub-file with the target file identifier in the expansion node.

5. The method according to claim 1 or 2, characterized in that: The method further comprises: After receiving an access request for the target file, determining the file identifier of the sub-file to be accessed by the access request as the identifier to be utilized according to a predetermined file splitting rule; wherein the file splitting rule is a rule for splitting the target file into the plurality of sub-files for distributed storage; Sending a transaction processing request to the shard node corresponding to the to-be-utilized identifier in the corresponding relationship, so that the shard node corresponding to the to-be-utilized identifier responds to the transaction processing request; The transaction processing request is used to execute the request content in the access request on the subfile having the identifier to be utilized.

6. A distributed database system, characterized in that: include: A management server and a plurality of sharding nodes, wherein the management server records file identifiers of a plurality of sub-files of a target file that are distributedly stored, and a corresponding relationship between the file identifier of each sub-file and the sharding node where the sub-file is located; Each shard node is used to store a sub-file for the target file; The management server is used to determine the target file identifier of the subfile to be migrated among the multiple subfiles; based on the target file identifier and the corresponding relationship, determine the shard node where the subfile to be migrated is located as the initial node; migrate the subfile with the target file identifier in the initial node to the expansion node, which is a shard node pre-set in the distributed database system when the system is expanded; The management server determines the target file identifier of the sub-file to be migrated among the multiple sub-files, specifically: According to the principle of load balancing, determine the target shard node from which the sub-files are to be migrated, and the target number of sub-files to be migrated from the target shard node; From the corresponding relationship, the file identifier corresponding to the target shard node is determined, and the target number of file identifiers are selected from the determined file identifiers to obtain the target file identifiers of the sub-files to be migrated among the multiple sub-files.

7. The system according to claim 6, characterized in that The management server is also used for: After the sub-file with the target file identifier in the initial node is migrated to the expansion node, the shard node corresponding to the target file identifier in the corresponding relationship is changed to the expansion node.

8. The system according to claim 6 or 7, characterized in that: The management server migrates the subfile with the target file identifier in the initial node to the expansion node, specifically: Sending a migration instruction for the sub-file having the target file identifier to the initial node; The initial node is further configured to determine, after receiving the migration instruction, an expansion node to which the subfile with the target file identifier is to be migrated, and transfer the subfile with the target file identifier to the expansion node by means of file transfer.

9. The system according to claim 8, characterized in that The management server is also used for: After the transmission of the sub-file with the target file identifier is completed, detecting whether there is incremental data for the sub-file with the target file identifier in the initial node relative to the sub-file with the target file identifier in the expansion node; If there is incremental data, the incremental data is written into the sub-file with the target file identifier in the expansion node.

10. The system according to claim 6 or 7, characterized in that: The management server is also used for: After receiving an access request for the target file, determining, according to a predetermined file splitting rule, a file identifier of a sub-file to be accessed by the access request as an identifier to be utilized; Sending a transaction processing request to the sharding node corresponding to the identifier to be used in the corresponding relationship; wherein the file splitting rule is a rule for splitting the target file into the plurality of sub-files for distributed storage; and the transaction processing request is used to execute the request content in the access request on the sub-file with the identifier to be used; The shard node corresponding to the identifier to be utilized is also used to respond to the transaction processing request.

11. A data processing device for a distributed database, characterized in that: A management server applied to a distributed database system, wherein the management server records file identifiers of a plurality of distributedly stored sub-files of a target file, and a corresponding relationship between the file identifier of each sub-file and the shard node where the sub-file is located; The device comprises: A file identification determination module, used to determine a target file identification of a sub-file to be migrated among the multiple sub-files; An initial node determination module, used to determine, based on the target file identifier and the corresponding relationship, the shard node where the sub-file to be migrated is located as the initial node; A file migration module, used to migrate the subfile with the target file identifier in the initial node to the expansion node, where the expansion node is a shard node pre-set in the distributed database system when the system is expanded; The file identification determination module is specifically used for: According to the principle of load balancing, determine the target shard node from which the sub-files are to be migrated, and the target number of sub-files to be migrated from the target shard node; From the corresponding relationship, the file identifier corresponding to the target shard node is determined, and the target number of file identifiers are selected from the determined file identifiers to obtain the target file identifiers of the sub-files to be migrated among the multiple sub-files.

12. A management server, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-5 when executing a program stored in a memory.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images