A data processing method, apparatus and system for distributed database systems

Abstract

This invention provides a data processing method, apparatus, and system for a distributed database system. The method is applied to a management server in the distributed database system. The management server obtains file information from each shard node. The file information of each shard node at least indicates associated sub-files within the sub-files stored by that shard node. These associated sub-files are multiple sub-files belonging to different files but with a relational relationship. Based on the obtained file information and according to a predetermined file selection rule, a file to be migrated is selected from the sub-files included in at least one shard node. The file selection rule includes: for the associated sub-file, using all sub-files contained in the associated sub-file as the selection granularity. The file to be migrated is then migrated to the expansion node. This solution can improve the execution efficiency of access commands in the expanded distributed database system.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method, apparatus and system for distributed database systems. Background Technology

[0002] In a distributed database system, the management server distributes any file as a physical sub-table across multiple shard nodes, thus forming a storage format of logical files + partitioned files.

[0003] Elastic scaling is an essential basic function of distributed database systems. Specifically, when a new scaling node is added to a distributed database system (i.e., a new shard node is added), the data content of the files is redistributed to complete the scaling.

[0004] In related technologies, the individual sub-files in the original shard nodes may be distributed across different shard nodes after scaling. This presents the following problem: Before scaling, the management server could push access commands to each shard node for response and aggregate the results reported by each shard node. However, after scaling, the management server needs to read the relevant data from each shard node and then use that data to respond to commands, significantly reducing the efficiency of the access command response process. Summary of the Invention

[0005] The purpose of this invention is to provide a data processing method, apparatus, and system for distributed database systems, which improves the execution efficiency of access commands in expanded distributed database systems. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of the present invention provide a data processing method for a distributed database system, applied to a management server in the distributed database system, wherein the distributed database system further includes multiple shard nodes; the method includes:

[0007] Obtain file information for each shard node; wherein, the file information for each shard node is used to indicate at least the associated sub-files in the sub-files stored by that shard node, and the associated sub-files are multiple sub-files that belong to different files but have an association relationship;

[0008] According to predetermined file selection rules, based on the obtained file information, files to be migrated are selected from the sub-files included in at least one shard node; wherein, the file selection rules include: for the associated sub-file, when selecting files, all sub-files contained in the associated sub-file are used as the selection granularity;

[0009] The files to be migrated are migrated to the expansion node; wherein, the expansion node is a shard node pre-added in the distributed database system when the system is expanded.

[0010] Optionally, the step of selecting files to be migrated from the sub-files included in at least one shard node according to predetermined file selection rules and based on the acquired file information includes:

[0011] Based on the specified index values ​​of each shard node, at least one shard node for which files need to be migrated is determined, as well as the amount of data to be migrated corresponding to the at least one shard node; wherein, the specified index values ​​are used to characterize the storage space usage.

[0012] For each of the at least one sharded nodes, according to a predetermined file selection rule, an independent sub-file and / or associated sub-file that matches the amount of data to be migrated corresponding to the sharded node is selected from the sub-files stored in that node, and these are used as the files to be migrated for that node; wherein, the independent sub-file is a sub-file that has no association relationship with other sub-files.

[0013] Optionally, the step of selecting independent sub-files and / or associated sub-files from the sub-files stored by the node, according to predetermined file selection rules, that match the amount of data to be migrated corresponding to the node, as the files to be migrated for that node, includes:

[0014] Determine the first data volume of the associated sub-files indicated by the file information of the node, and the second data volume of the independent sub-files in the shard node;

[0015] According to the predetermined file selection rules, based on the first data volume and the second data volume, independent sub-files and / or associated sub-files that match the amount of data to be migrated corresponding to the node are selected from the sub-files stored in the node and used as the files to be migrated for the node.

[0016] Optionally, migrating the file to be migrated to the expansion node includes:

[0017] A migration instruction for the file to be migrated is issued to the shard node to which the file to be migrated belongs, so that the shard node receiving the migration instruction can migrate the file to be migrated to the expansion node with sub-files as the migration granularity.

[0018] Optionally, the method for determining the associated sub-files indicated by the file information of each shard node includes:

[0019] Identify the related files among the target files, where the target file is the file to which the sub-file stored in the shard node belongs;

[0020] By using the sub-files of the target file contained in the associated file located in the shard node, the associated sub-files in the shard node are determined.

[0021] Optionally, determining the associated files among the target files that have a relationship includes:

[0022] Retrieve the associated file declaration information provided by the user;

[0023] Based on the associated file declaration information, identify the associated files that have a relationship among the target files.

[0024] Optionally, determining the associated sub-files in the shard node using the sub-files of the target file contained in the associated file, located in the shard node, includes:

[0025] Identify the subfiles stored in the shard node that match the predetermined matching rules among the subfiles belonging to each specified file, and use them as associated subfiles;

[0026] The specified file is the target file contained in the associated file, and the predetermined matching rules include: having the same sub-file identifier, or having matching column content.

[0027] Optionally, before determining which sub-files belonging to each specified file stored in the shard node conform to a predetermined matching rule and are used as associated sub-files, the determination method further includes:

[0028] Detect the number of sub-files belonging to each specified file stored in the shard node;

[0029] If the detected quantity is 1 for all of them, the subfiles belonging to each of the specified files will be identified as associated subfiles.

[0030] If the detected number is greater than 1, then the step of determining the sub-files that conform to the predetermined matching rules among the sub-files stored by the shard node and belonging to each specified file is executed, and these sub-files are used as associated sub-files.

[0031] Secondly, embodiments of the present invention provide a distributed database system, including: a management server and multiple shard nodes;

[0032] Each shard node is used to determine the file information of the shard node; and to report the file information of the shard node to the management server; wherein, the file information of the shard node is used at least to indicate the associated sub-files in the sub-files stored by the shard node, and the associated sub-files are multiple sub-files that belong to different files but have an association relationship;

[0033] The management server is used to obtain file information from each shard node; select files to be migrated from the sub-files included in at least one shard node according to a predetermined file selection rule and based on the obtained file information; and migrate the files to be migrated to the expansion node; wherein, the file selection rule includes: for the associated sub-file, when selecting files, using all sub-files included in the associated sub-file as the selection granularity; the expansion node is a shard node pre-added in the distributed database system when the system is expanded.

[0034] Optionally, the management server selects files to be migrated from the sub-files included in at least one shard node according to predetermined file selection rules and based on the acquired file information, specifically as follows:

[0035] Based on the specified index values ​​of each shard node, at least one shard node for which files need to be migrated is determined, as well as the amount of data to be migrated corresponding to the at least one shard node; wherein, the specified index values ​​are used to characterize the storage space usage.

[0036] For each of the at least one sharded nodes, according to a predetermined file selection rule, select an independent sub-file and / or associated sub-file from the sub-files stored by that node that matches the amount of data to be migrated corresponding to that node, and use it as the file to be migrated for that node;

[0037] The independent sub-file is a sub-file that is not associated with other sub-files.

[0038] Optionally, the management server selects, according to predetermined file selection rules, independent sub-files and / or associated sub-files from the sub-files stored by the node that match the amount of data to be migrated corresponding to the node, as the files to be migrated for that node, including:

[0039] Determine the first data volume of the associated sub-files indicated by the file information of the node, and the second data volume of the independent sub-files in the shard node;

[0040] According to the predetermined file selection rules, based on the first data volume and the second data volume, independent sub-files and / or associated sub-files that match the amount of data to be migrated corresponding to the node are selected from the sub-files stored in the node and used as the files to be migrated for the node.

[0041] Optionally, the management server migrates the files to be migrated to the expansion node, including:

[0042] A migration instruction for the file to be migrated is issued to the shard node to which the file to be migrated belongs, so that the shard node receiving the migration instruction can migrate the file to be migrated to the expansion node with sub-files as the migration granularity.

[0043] Optionally, each shard node determines its file information, including:

[0044] Identify the related files among the target files, where the target file is the file to which the sub-file stored in the shard node belongs;

[0045] Using the sub-files of the target file contained in the associated file located in the shard node, determine the associated sub-files in the shard node;

[0046] Based on the identified associated sub-files, generate the file information for that shard node.

[0047] Optionally, each shard node determines related files among the target files that have a relationship, including:

[0048] Retrieve the associated file declaration information provided by the user;

[0049] Based on the associated file declaration information, identify the associated files that have a relationship among the target files.

[0050] Optionally, each shard node determines the associated sub-files within the shard node using the sub-files of the target file contained in the associated file, including:

[0051] Identify the subfiles stored in the shard node that match the predetermined matching rules among the subfiles belonging to each specified file, and use them as associated subfiles;

[0052] The specified file is the target file contained in the associated file, and the predetermined matching rules include: having the same sub-file identifier, or having matching column content.

[0053] Optionally, before each shard node determines which sub-files belonging to each specified file and conforming to a predetermined matching rule are used as associated sub-files, it is further used for:

[0054] Detect the number of sub-files belonging to each specified file stored in the shard node;

[0055] If the detected quantity is 1 for all of them, the subfiles belonging to each of the specified files will be identified as associated subfiles.

[0056] If the detected number is greater than 1, then the step of determining the sub-files that conform to the predetermined matching rules among the sub-files stored by the shard node and belonging to each specified file is executed, and these sub-files are used as associated sub-files.

[0057] Thirdly, embodiments of the present invention provide a data processing apparatus for a distributed database system, applied to a management server in the distributed database system; the distributed database system further includes multiple shard nodes; the apparatus includes:

[0058] The acquisition module is used to acquire file information of each shard node; wherein, the file information of any shard node is used to indicate at least the associated sub-files in the sub-files stored by that shard node, and the associated sub-files are multiple sub-files that belong to different files but have an association relationship;

[0059] The selection module is used to select files to be migrated from the sub-files included in at least one shard node based on the acquired file information according to a predetermined file selection rule; wherein, the file selection rule includes: for the associated sub-file, when selecting files, all sub-files included in the associated sub-file are used as the selection granularity;

[0060] The migration module is used to migrate the file to be migrated to the expansion node; wherein the expansion node is a shard node that is pre-added in the distributed database system when the system is expanded.

[0061] Fourthly, embodiments of the present invention provide a management server, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0062] Memory, used to store computer programs;

[0063] When a processor executes a program stored in memory, it implements the steps of the data processing method for a distributed database system provided in the first aspect above.

[0064] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the data processing method for a distributed database system provided in the first aspect.

[0065] This invention also provides a computer program product containing instructions that, when run on a computer, causes the computer to execute the data processing method for a distributed database system provided in any of the first aspects described above.

[0066] Beneficial effects of the embodiments of the present invention:

[0067] In this embodiment of the invention, the management server in the distributed database system obtains file information from each shard node. The file information of any shard node at least indicates associated sub-files within the sub-files stored by that shard node. These associated sub-files are multiple sub-files belonging to different files but with a relational relationship. Based on the obtained file information and according to a predetermined file selection rule, a file to be migrated is selected from the sub-files included in at least one shard node. This file selection rule includes: for associated sub-files, using all sub-files contained in the associated sub-file as the selection granularity; and migrating the file to be migrated to an expansion node. This expansion node is a shard node pre-added to the distributed database system during system expansion. Therefore, in this solution, when selecting files to be migrated, the management server uses all sub-files contained in the associated sub-file as the selection granularity. Since the associated sub-files are still distributed in the same node, after expansion, the management server can still push the access command down to each shard node for response. Therefore, compared with related technologies, this solution can improve the execution efficiency of access commands in the expanded distributed database system.

[0068] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0070] Figure 1 This is a schematic diagram illustrating the principles of existing data processing methods for distributed database systems.

[0071] Figure 2 A flowchart illustrating a data processing method for a distributed database system provided in an embodiment of the present invention;

[0072] Figure 3 This is a schematic diagram of a data processing method for a distributed database system provided in an embodiment of the present invention;

[0073] Figure 4 This is another flowchart of a data processing method for a distributed database system provided in an embodiment of the present invention;

[0074] Figure 5This is a schematic diagram of the structure of a data processing system for a distributed database system provided in an embodiment of the present invention;

[0075] Figure 6 This is a schematic diagram of the structure of a data processing device for a distributed database system provided in an embodiment of the present invention;

[0076] Figure 7 This is a schematic diagram of the structure of a management server provided in an embodiment of the present invention. Detailed Implementation

[0077] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0078] In related technologies, the management server in a distributed database system splits the file to be stored into multiple sub-files based on predetermined splitting rules, and distributes these sub-files across multiple shard nodes to form a distributed database. Sub-files belonging to various files stored in a shard node may have the same sub-file identifier or have matching column contents or other relationships.

[0079] Before scaling up, for a given access command, the management server would push the command down to each shard node. Each shard node would respond using the relationships between its sub-files and then aggregate the results reported by all shard nodes to complete the response to the access command. However, after scaling up, because the related sub-files in the original shard nodes were split across different shard nodes, for the same access command, the management server would need to read the relevant data from each shard node and then use that data to respond to the command. This would not fully utilize the computing power of each shard node, thus significantly reducing the efficiency of the response process for that access command.

[0080] For example, such as Figure 1As shown, a distributed database system includes: a management server, shard node 1, shard node 2, and a pre-added expansion node 1. The distributed database system stores three files in the form of tables: TableA for storing account information, TableB for storing order information, and TableC for storing points information. Each table stores an associated column: userid. During distributed storage, these three tables are hash-sharded according to userid, and each table is split into six sub-tables: TableA is split into TableA_1 to TableA_6, TableB into TableB_1 to TableB_6, and TableC into TableC_1 to TableC_6. The storage configuration of each sub-table is as follows. Figure 1 As shown. At this point, although each data table is split into multiple sub-tables, account information, order information, and points information with the same userid are still in the same shard node. That is, TableA_1, TableB_1, and TableC_1 are in the same shard node, and TableA_6, TableB_6, and TableC_6 are also in the same shard node.

[0081] When the distributed database system responds to an SQL access command: "Select B.Orders from A join Bon A.userid = B.userid where A.name = 'XXX'", the purpose of this command is to retrieve the content of the "userid" column corresponding to the "name" column being "XXX" in Table A, and then find the corresponding content of the "Orders" column in Table B based on the content of the "userid" column. Before scaling up, the management server can push this access command down to shard node 1 and shard node 2. Each shard node executes the access command to perform the "join" calculation, and the results reported by each shard node are aggregated to complete the response to the access command. After scaling, related sub-tables were migrated to different shard nodes. For example, TableA_1 was migrated to a newly added shard node, while TableB_1 and TableC_1 remained in shard node 1. Now, for the aforementioned access commands, the management server needs to read the relevant data from each shard node and use that data to respond to the command. The "join" calculation can no longer be pushed down to each shard node for execution and aggregation, increasing the SQL computation load and reducing execution efficiency.

[0082] To address the aforementioned problems, this embodiment provides a data processing method, apparatus, and system for distributed database systems. The following section first introduces a data processing method for distributed database systems provided by this embodiment.

[0083] The data processing method for a distributed database system provided in this embodiment of the invention is applied to a management server in the distributed database system, which further includes multiple shard nodes; the method includes:

[0084] Obtain file information for each shard node; wherein, the file information of any shard node is used to indicate at least the associated sub-files in the sub-files stored by that shard node, and the associated sub-files are multiple sub-files that belong to different files but have an association relationship;

[0085] According to predetermined file selection rules, based on the obtained file information, files to be migrated are selected from the sub-files included in at least one shard node; wherein, the file selection rules include: for the associated sub-file, when selecting files, all sub-files contained in the associated sub-file are used as the selection granularity;

[0086] The files to be migrated are migrated to the expansion node; wherein, the expansion node is a shard node pre-added in the distributed database system when the system is expanded.

[0087] In this invention, the aforementioned files can also be called logical files, and sub-files can be called physical sub-files. For example, a data table can be one of the aforementioned files; in this case, the data table can be called a logical table, and the sub-tables of the data table can be called physical sub-tables. Furthermore, the files involved in this embodiment are not limited to data tables; any file that can be used in a distributed database system can be considered as the files involved in this embodiment. In this solution, when the management server selects files to be migrated, it uses all sub-files contained in the associated sub-file as the selection granularity. Thus, since the associated sub-files are still distributed on the same node, after scaling up, the management server can still push the access command down to each shard node for response. Therefore, compared to related technologies, this solution can improve the execution efficiency of the expanded distributed database system for access commands.

[0088] The following description, in conjunction with the accompanying drawings, introduces a data processing method for a distributed database system provided by an embodiment of the present invention.

[0089] like Figure 2As shown, the data processing method for a distributed database system provided in this embodiment of the invention is applied to a management server in the distributed database system; the distributed database system further includes multiple shard nodes; the method includes the following steps:

[0090] S201, Obtain file information for each shard node; wherein, the file information for each shard node is used to indicate at least the associated sub-files in the sub-files stored by that shard node, and the associated sub-files are multiple sub-files that belong to different files but have an association relationship;

[0091] After scaling up nodes to a distributed database system, data migration, or data redistribution, is required. Considering the potential relationships between sub-files stored in sharded nodes, this solution first retrieves file information from each sharded node during data migration to understand the relationships between sub-files within each sharded node, and then executes the subsequent migration process. Sharded nodes are also called storage nodes in the distributed database system, and the management server is also called a management node or compute node.

[0092] For example, the file information of each shard node may include a mapping relationship of file identifiers of associated sub-files, such that the sub-file represented by each file identifier in the mapping relationship is an associated sub-file. For example, the file information of each shard node may include an array of file identifiers of associated sub-files, such that the sub-file represented by each file identifier in the array is an associated sub-file. It should be emphasized that, provided that the file information of each shard node is at least used to indicate the associated sub-files among the sub-files stored in that shard node, the specific form of the file information is not limited in this embodiment of the invention.

[0093] Understandably, the file information of each shard node can also be used to indicate the independent sub-files stored within that shard node. Independent sub-files are those that are not associated with other sub-files. Thus, after obtaining the file information of each shard node, the associated sub-files and independent sub-files within each shard node can be identified based on this information. For each shard node's file information, if it indicates both associated and independent sub-files, then, for example, the file information of each shard node could include: an array of file identifiers for associated sub-files and file identifiers for independent sub-files; or, the file information of each shard node could include: a mapping relationship of file identifiers for associated sub-files and file identifiers for independent sub-files, and so on.

[0094] Furthermore, after determining the associated sub-files in each shard node, the file information for that shard node can be generated. The associated sub-files in each shard node can be determined by the management server, which can then use these determined sub-files to determine the file information for each shard node. Alternatively, the associated sub-files in each shard node can be determined by each shard node itself, in which case each shard node can generate its own file information based on its determined sub-files and report it to the management server. Optionally, the method for determining the associated sub-files indicated by the file information of each shard node can include steps A1-A2:

[0095] A1, Identify the related files among the target files, where the target file is the file to which the sub-file stored in the shard node belongs;

[0096] For example, a distributed database system might store three files as tables: TableA for account information, TableB for order information, and TableC for points information. Each table stores a related column: userid. Additionally, a distributed database system can also store other files that are not directly related. TableA, TableB, and TableC then form a linked file system. Figure 1 In this context, TableA_1 is stored in shard node 1, so TableA is the target file to which TableA_1 stored in shard node 1 belongs. Similarly, other sub-files stored in each shard node also have their corresponding target files.

[0097] For example, identifying related files among the target files may include:

[0098] Retrieve the associated file declaration information provided by the user;

[0099] Based on the associated file declaration information, related files with relationships are identified among the target files. Users can provide associated file declaration information specifying which files are related. The management server can then identify the related files among the target files based on this information. In this implementation, because user assistance is used to determine the associated sub-files, the identified sub-files better reflect the user's actual needs. For example, users can declare and cancel associated file declaration information via an SQL interface; that is, the SQL interface can provide users with settings for declaring and canceling associated file declaration information.

[0100] like Figure 3As shown, `Table_group:TableA\TableB\TableC` indicates that TableA, TableB, and TableC are related files. When declaring multiple related tables, the related file declaration information can be stored in each shard node as a two-dimensional array. For example, to declare TableA, TableB, and TableC; TableE, TableF, and TableG; and TableX, TableY, and TableZ as related files, it can be recorded as follows:

[0101] Table_group:

[0102] {

[0103] Table_group1:TableA\TableB\TableC

[0104] Table_group2:TableE\TableF\TableG

[0105] Table_group3:TableX\TableY\TableZ

[0106] }

[0107] It should be emphasized that the above-described method of determining associated files based on user-provided associated file declaration information is merely an example and should not be construed as limiting the present invention. For example, it is possible to identify whether each target file belongs to an associated file based on whether there are identical column contents among the target files, and so on.

[0108] A2, using the sub-files of the target file contained in the associated file located in the shard node, determine the associated sub-files in the shard node.

[0109] The specific implementation process of this step can be as follows:

[0110] The subfiles stored in the shard node that belong to the subfiles of each specified file and conform to the predetermined matching rules are identified as associated subfiles; wherein, the specified file is the target file contained in the associated file, and the predetermined matching rules include: having the same subfile identifier, or having matching column content.

[0111] In the example above, the associated file consisting of TableA, TableB, and TableC contains three target files: TableA, TableB, and TableC. These three files are distributed and stored across shard node 1 and shard node 2, and the sub-files split according to the same hash splitting rule contain matching column content. After splitting, the associated sub-files can be assigned the same sub-file identifier by naming them with the same suffix, such as TableA_1, TableB_1, and TableC_1 all having the same suffix "_1". Then, by identifying the common suffix, the sub-files stored on each shard node that belong to each specified file and match the predetermined matching rules can be identified as associated sub-files. Furthermore, virtual associated sub-files can be constructed to mark the sub-files stored on each shard node that belong to each specified file and match the predetermined matching rules. For example, TableA_1, TableB_1, and TableC_1 in the example above can be defined as Table_group_index1.

[0112] S202, according to a predetermined file selection rule, based on the obtained file information, select the file to be migrated from the sub-files included in at least one shard node; wherein, the file selection rule includes: for the associated sub-file, when selecting the file, using all sub-files included in the associated sub-file as the selection granularity;

[0113] The files to be migrated include related sub-files and unrelated sub-files. The selection rules for this file ensure that when migrating files during distributed database system expansion, no single sub-file within a related sub-file is migrated; instead, all sub-files contained within the related sub-files are selected. This guarantees that all related sub-files are always stored on the same shard node.

[0114] S203, migrate the file to be migrated to the expansion node; wherein, the expansion node is a shard node pre-added in the distributed database system when the system is expanded.

[0115] For example, such as Figure 3 By identifying subfiles belonging to specified files stored in each shard node that share the same suffix, those subfiles that conform to predetermined matching rules are designated as associated subfiles. For example, TableA_1, TableB_1, and TableC_1 form an associated subfile; TableA_2, TableB_2, and TableC_2 form an associated subfile, and so on, up to the suffix "_6". When selecting files to be migrated, all subfiles contained in the associated subfiles are used as the selection granularity. Figure 3 In the process, the associated subfiles consisting of TableA_1, TableB_1, and TableC_1 bound together, and the associated subfiles consisting of TableA_4, TableB_4, and TableC_4 bound together, are migrated to the expansion node, instead of as... Figure 1 In the process, related sub-files were not protected, resulting in the related sub-files not being in the same shard node after expansion.

[0116] For example, the management server can issue a migration instruction to the shard node to which the file to be migrated belongs, so that the shard node receiving the migration instruction can migrate the file to be migrated to the expansion node at the sub-file level.

[0117] Migration commands can carry file identifiers of the files to be migrated. Each shard node migrates the files to be migrated to the expansion node according to the sub-files specified in the migration command, with the migration granularity being the sub-files. Optionally, shard nodes can transfer the sub-files to be migrated to the expansion node via file transfer to save the computational overhead of the management server.

[0118] Understandably, in this solution, the number of expansion nodes can be set according to actual needs. Then, after adding the set number of expansion nodes, some sub-files can be migrated to each expansion node through sub-file migration.

[0119] In this embodiment, when selecting files to be migrated, the management server uses all subfiles contained within a related subfile as the selection granularity. Since related subfiles remain distributed across the same node, after scaling up, the management server can still push the access command to each shard node for response. Therefore, compared to related technologies, this solution can improve the execution efficiency of the expanded distributed database system for access commands.

[0120] Alternatively, in another embodiment of the invention, such as Figure 4 As shown, the above-mentioned S202 may include the following steps:

[0121] S2021, Based on the specified index values ​​of each shard node, determine at least one shard node for which files need to be migrated, and the amount of data to be migrated corresponding to the at least one shard node; wherein, the specified index values ​​are used to characterize the storage space usage.

[0122] Each shard node has a fixed storage space, and each sub-file within it occupies a certain amount of storage space. When scaling up a distributed database system, the amount of data to be migrated can be comprehensively considered based on the shard node storage space size, storage space usage, and the size of each sub-file. For example, if the storage space size of each shard node is the same, the specified metric value can be the space usage of each shard node; while if the storage space size of each shard node is different, the specified metric value can be the space utilization rate of each shard node.

[0123] The management server can determine, according to the load balancing principle and the specified index values ​​of each shard node, at least one shard node that needs to be migrated, as well as the amount of data to be migrated corresponding to the at least one shard node, but it is not limited to this.

[0124] For example, in a distributed database system, there are two shard nodes with the same storage space. Shard node 1 stores 5G of data and shard node 2 stores 4G of data. When expanding, an expansion node 1 with the same storage space as the initial shard nodes is added in advance. Then, according to the load balancing principle, it can be determined that 2G of data will be migrated from shard node 1 and 1G of data will be migrated from shard node 2 to expansion node 1, so that the storage space of the three nodes after expansion is 3G.

[0125] S2022, for each of the at least one sharded nodes, according to a predetermined file selection rule, select an independent sub-file and / or associated sub-file from the sub-files stored by the node that matches the amount of data to be migrated corresponding to the node, and use it as the file to be migrated for the node; wherein, the independent sub-file is a sub-file that has no association relationship with other sub-files.

[0126] It should be noted that selecting independent sub-files and / or associated sub-files from the sub-files stored by the node that match the amount of data to be migrated for that node specifically means: selecting at least one sub-file from the sub-files stored by the node, where the data volume of the at least one sub-file is not less than the amount of data to be migrated; and the at least one sub-file may contain only independent sub-files, only associated sub-files, or both. Furthermore, before executing S2022, the management server can obtain the data volume of the stored sub-files from each shard node, and then select the files to be migrated based on the obtained data volume of the sub-files.

[0127] For example, this step may include steps B1-B2:

[0128] B1, determine the first data volume of the associated sub-file indicated by the file information of the node, and the second data volume of the independent sub-file in the shard node;

[0129] Each shard node may contain associated subfiles and independent subfiles without any association. Once associated subfiles are identified, all subfiles within an associated subfile can be treated as a whole, and the data volume of that associated subfile, i.e., the size of the shard node's storage space, is considered as the first data volume. The data volume of each independent subfile within that shard node is considered as the second data volume.

[0130] B2. According to the predetermined file selection rules, based on the first data volume and the second data volume, select independent sub-files and / or associated sub-files from the sub-files stored in the node that match the amount of data to be migrated corresponding to the shard node, and use them as the files to be migrated for the node.

[0131] The file selection rules include: when selecting the associated sub-file, all sub-files contained in the associated sub-file are used as the selection granularity.

[0132] In this scheme, the amount of data to be migrated can be comprehensively considered based on the specified index value of the shard node, the size of each first data volume and each second data volume. For example, the independent sub-files and / or associated sub-files that the shard node should migrate can be determined according to the principle of load balancing.

[0133] In this embodiment of the invention, when selecting files to be migrated, the management server uses all subfiles contained within a related subfile as the selection granularity. Since related subfiles remain distributed across the same node, after scaling up, the management server can still push the access command to each shard node for response. Therefore, compared to related technologies, this solution can improve the execution efficiency of access commands in the expanded distributed database system. Furthermore, by determining at least one shard node for the files to be migrated, and the corresponding amount of data to be migrated for that at least one shard node, based on specified metric values ​​of each shard node, the resource utilization of each shard node can be improved, and it is applicable to data migration between shard nodes with the same or different storage spaces.

[0134] Optionally, in another embodiment of the present invention, before determining the sub-files that conform to the predetermined matching rules among the sub-files stored by the shard node and belonging to each specified file as associated sub-files, the determination method further includes the step of:

[0135] Detect the number of sub-files belonging to each specified file stored in the shard node;

[0136] If the detected quantity is 1 for all of them, the subfiles belonging to each of the specified files will be identified as associated subfiles.

[0137] If the detected number is greater than 1, then the step of determining the sub-files that conform to the predetermined matching rules among the sub-files stored by the shard node and belonging to each specified file is executed, and these sub-files are used as associated sub-files.

[0138] Specifically, when a target file is stored in a distributed database, it may be split into multiple sub-files and distributed across various shard nodes. A single shard node may store multiple sub-files of the target file, or it may store only one sub-file. Therefore, after identifying the related files among the target files, before determining which sub-files belonging to the specified files stored on the shard node match the predetermined matching rules and are considered as related sub-files, the number of sub-files belonging to the specified file stored on the shard node is first checked. If the detected number is 1, the sub-file belonging to the specified file is directly identified as the related sub-file.

[0139] In this embodiment of the invention, when selecting files to be migrated, the management server uses all subfiles contained in the associated subfiles as the selection granularity. Before expansion, the management server can push down access commands to each shard node for response. Since the associated subfiles are still distributed in the same node, after expansion, the management server can still push down the access command to each shard node for response. Therefore, compared with related technologies, this solution can improve the execution efficiency of the expanded distributed database system for access commands. Furthermore, before determining which subfiles belonging to each specified file and stored in the shard node conform to predetermined matching rules as associated subfiles, the number of subfiles belonging to each specified file stored in the shard node is detected. If the detected number is 1 for all subfiles, the subfiles belonging to each specified file can be directly determined as associated subfiles, thereby further reducing the computational overhead during the execution of this solution.

[0140] Corresponding to the above method embodiments, this invention also provides a distributed database system, such as... Figure 5 As shown, the system may include: a management server 510 and multiple shard nodes 520, wherein:

[0141] Each shard node 520 is used to determine the file information of the shard node; and to report the file information of the shard node to the management server; wherein, the file information of the shard node is used at least to indicate the associated sub-files in the sub-files stored by the shard node, and the associated sub-files are multiple sub-files that belong to different files but have an association relationship;

[0142] Management server 510 is used to obtain file information of each shard node; according to a predetermined file selection rule, based on the obtained file information, select files to be migrated from the sub-files included in at least one shard node; migrate the files to be migrated to the expansion node; wherein, the file selection rule includes: for the associated sub-file, when selecting files, all sub-files included in the associated sub-file are used as the selection granularity; the expansion node is a shard node pre-added in the distributed database system when the system is expanded.

[0143] As can be seen, in this solution, when selecting files to be migrated, the management server uses all subfiles contained within a related subfile as the selection granularity. This ensures that, since related subfiles remain distributed across the same node, after scaling, the management server can still push the access command to each shard node for response. Therefore, compared to related technologies, this solution improves the execution efficiency of access commands in the scaled distributed database system.

[0144] Optionally, the management server 510 selects files to be migrated from the sub-files included in at least one shard node according to a predetermined file selection rule and based on the acquired file information. Specifically:

[0145] Based on the specified index values ​​of each shard node, at least one shard node for which files need to be migrated is determined, as well as the amount of data to be migrated corresponding to the at least one shard node; wherein, the specified index values ​​are used to characterize the storage space usage.

[0146] For each of the at least one sharded nodes, according to a predetermined file selection rule, select an independent sub-file and / or associated sub-file from the sub-files stored by that node that matches the amount of data to be migrated corresponding to that node, and use it as the file to be migrated for that node;

[0147] The independent sub-file is a sub-file that is not associated with other sub-files.

[0148] Optionally, the management server 510 selects, according to predetermined file selection rules, independent sub-files and / or associated sub-files from the sub-files stored by the node that match the amount of data to be migrated corresponding to the node, as the files to be migrated for that node, including:

[0149] Determine the first data volume of the associated sub-files indicated by the file information of the node, and the second data volume of the independent sub-files in the shard node;

[0150] According to the predetermined file selection rules, based on the first data volume and the second data volume, independent sub-files and / or associated sub-files that match the amount of data to be migrated corresponding to the node are selected from the sub-files stored in the node and used as the files to be migrated for the node.

[0151] Optionally, the management server migrates the files to be migrated to the expansion node, including:

[0152] A migration instruction for the file to be migrated is issued to the shard node to which the file to be migrated belongs, so that the shard node receiving the migration instruction can migrate the file to be migrated to the expansion node with sub-files as the migration granularity.

[0153] Optionally, each shard node 520 determines the file information of that shard node, including:

[0154] Identify the related files among the target files, where the target file is the file to which the sub-file stored in the shard node belongs;

[0155] Using the sub-files of the target file contained in the associated file located in the shard node, determine the associated sub-files in the shard node;

[0156] Based on the identified associated sub-files, generate the file information for that shard node.

[0157] Optionally, each shard node 520 determines the associated files that have a relationship among the target files, including:

[0158] Retrieve the associated file declaration information provided by the user;

[0159] Based on the associated file declaration information, identify the associated files that have a relationship among the target files.

[0160] Optionally, each shard node 520 uses the sub-files of the target file contained in the associated file located within that shard node to determine the associated sub-files in that shard node, including:

[0161] Identify the subfiles stored in the shard node that match the predetermined matching rules among the subfiles belonging to each specified file, and use them as associated subfiles;

[0162] The specified file is the target file contained in the associated file, and the predetermined matching rules include: having the same sub-file identifier, or having matching column content.

[0163] Optionally, before each shard node 520 determines the sub-files stored in that shard node that belong to each specified file and conform to the predetermined matching rules, it is also used for:

[0164] Detect the number of sub-files belonging to each specified file stored in the shard node;

[0165] If the detected quantity is 1 for all of them, the subfiles belonging to each of the specified files will be identified as associated subfiles.

[0166] If the detected number is greater than 1, then the step of determining the sub-files that conform to the predetermined matching rules among the sub-files stored by the shard node and belonging to each specified file is executed, and these sub-files are used as associated sub-files.

[0167] For descriptions of the devices in the system, please refer to the corresponding content in the above method embodiments, which will not be repeated here.

[0168] Corresponding to the above method embodiments, this invention also provides a data processing apparatus for a distributed database system, applied to a management server in the distributed database system, wherein the distributed database system further includes multiple shard nodes; such as Figure 6 As shown, the device includes:

[0169] The acquisition module 610 is used to acquire file information of each shard node; wherein, the file information of any shard node is used to indicate at least the associated sub-files in the sub-files stored by the shard node, and the associated sub-files are multiple sub-files that belong to different files but have an association relationship;

[0170] The selection module 620 is used to select files to be migrated from the sub-files included in at least one shard node based on the acquired file information according to a predetermined file selection rule; wherein, the file selection rule includes: for the associated sub-file, when selecting files, all sub-files included in the associated sub-file are used as the selection granularity;

[0171] Migration module 630 is used to migrate the file to be migrated to the expansion node; wherein the expansion node is a shard node pre-added in the distributed database system when the system is expanded.

[0172] As can be seen, in this solution, when selecting files to be migrated, the management server uses all subfiles contained within a related subfile as the selection granularity. This ensures that, since related subfiles remain distributed across the same node, after scaling, the management server can still push the access command to each shard node for response. Therefore, compared to related technologies, this solution improves the execution efficiency of access commands in the scaled distributed database system.

[0173] Optionally, module 620 is selected, including:

[0174] The first determining submodule is used to determine at least one shard node that needs to migrate files, and the amount of data to be migrated corresponding to the at least one shard node, based on the specified indicator values ​​of each shard node; wherein, the specified indicator values ​​are used to characterize the storage space usage.

[0175] The selection submodule is used to select, for each of the at least one sharded nodes, an independent subfile and / or associated subfile that matches the amount of data to be migrated corresponding to the sharded node from the subfiles stored in the node according to a predetermined file selection rule, as the file to be migrated for that node; wherein, the independent subfile is a subfile that has no association relationship with other subfiles.

[0176] Optionally, select sub-modules, including:

[0177] The data volume determination unit is used to determine the first data volume of the associated sub-files indicated by the file information of the node, and the second data volume of the independent sub-files in the shard node;

[0178] The selection unit is configured to select, according to predetermined file selection rules, independent sub-files and / or associated sub-files that match the amount of data to be migrated corresponding to the node from the sub-files stored in the node, based on the first data amount and the second data amount, as the files to be migrated for the node.

[0179] Optionally, the migration module 630 is specifically used for:

[0180] A migration instruction for the file to be migrated is issued to the shard node to which the file to be migrated belongs, so that the shard node receiving the migration instruction can migrate the file to be migrated to the expansion node with sub-files as the migration granularity.

[0181] Optionally, the method for determining the associated sub-files indicated by the file information of each shard node includes:

[0182] Identify the related files among the target files, where the target file is the file to which the sub-file stored in the shard node belongs;

[0183] By using the sub-files of the target file contained in the associated file located in the shard node, the associated sub-files in the shard node are determined.

[0184] Optionally, determining the associated files among the target files that have a relationship includes:

[0185] Retrieve the associated file declaration information provided by the user;

[0186] Based on the associated file declaration information, identify the associated files that have a relationship among the target files.

[0187] Optionally, determining the associated sub-files in the shard node using the sub-files of the target file contained in the associated file, located in the shard node, includes:

[0188] The subfiles stored in the shard node that belong to the subfiles of each specified file and conform to the predetermined matching rules are identified as associated subfiles; wherein, the specified file is the target file contained in the associated file, and the predetermined matching rules include: having the same subfile identifier, or having matching column content.

[0189] Optionally, before determining which sub-files belonging to each specified file stored in the shard node conform to a predetermined matching rule and are used as associated sub-files, the determination method further includes:

[0190] Detect the number of sub-files belonging to each specified file stored in the shard node;

[0191] If the detected quantity is 1 for all of them, the subfiles belonging to each of the specified files will be identified as associated subfiles.

[0192] If the detected number is greater than 1, then the step of determining the sub-files that conform to the predetermined matching rules among the sub-files stored by the shard node and belonging to each specified file is executed, and these sub-files are used as associated sub-files.

[0193] This invention also provides a management server, such as... Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0194] Memory 703 is used to store computer programs;

[0195] When the processor 701 executes the program stored in the memory 703, it implements the steps of any of the methods provided in the embodiments of the present invention.

[0196] The communication bus mentioned in the management server above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0197] The communication interface is used for communication between the aforementioned management server and other devices.

[0198] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0199] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0200] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described data processing methods for a distributed database system.

[0201] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the data processing methods for a distributed database system described above.

[0202] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0203] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0204] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0205] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A data processing method for a distributed database system, characterized in that, Management servers used in distributed database systems; The distributed database system further includes multiple shard nodes; the method includes: Obtain file information for each shard node; wherein, the file information for each shard node is used to indicate at least the associated sub-files in the sub-files stored by that shard node, and the associated sub-files are multiple sub-files that belong to different files but have an association relationship; According to predetermined file selection rules, based on the acquired file information, files to be migrated are selected from the sub-files included in at least one shard node; wherein, the file selection rules include: for the associated sub-file, when selecting files, all sub-files included in the associated sub-file are used as the selection granularity; the files to be migrated include associated sub-files and sub-files without association, and when associated sub-files are included, the included associated sub-files are selected from the sub-files included in at least one shard node; The files to be migrated are migrated to the expansion node; wherein, the expansion node is a shard node pre-added in the distributed database system when the system is expanded.

2. The method according to claim 1, characterized in that, The step of selecting files to be migrated from the sub-files included in at least one shard node according to predetermined file selection rules and based on the acquired file information includes: Based on the specified index values ​​of each shard node, at least one shard node for which files need to be migrated is determined, as well as the amount of data to be migrated corresponding to the at least one shard node; wherein, the specified index values ​​are used to characterize the storage space usage. For each of the at least one sharded nodes, according to a predetermined file selection rule, an independent sub-file and / or associated sub-file that matches the amount of data to be migrated corresponding to the sharded node is selected from the sub-files stored in that node, and these are used as the files to be migrated for that node; wherein, the independent sub-file is a sub-file that has no association relationship with other sub-files.

3. The method according to claim 2, characterized in that, The step of selecting, according to predetermined file selection rules, independent sub-files and / or associated sub-files from the sub-files stored by the node that match the amount of data to be migrated corresponding to the node, and using these as the files to be migrated for the node, includes: Determine the first data volume of the associated sub-files indicated by the file information of the node, and the second data volume of the independent sub-files in the shard node; According to the predetermined file selection rules, based on the first data volume and the second data volume, independent sub-files and / or associated sub-files that match the amount of data to be migrated corresponding to the node are selected from the sub-files stored in the node and used as the files to be migrated for the node.

4. The method according to claim 1, characterized in that, The process of migrating the files to be migrated to the expansion node includes: A migration instruction for the file to be migrated is issued to the shard node to which the file to be migrated belongs, so that the shard node receiving the migration instruction can migrate the file to be migrated to the expansion node with sub-files as the migration granularity.

5. The method according to any one of claims 1-4, characterized in that, The methods for determining the associated sub-files indicated by the file information of each shard node include: Identify the related files among the target files, where the target file is the file to which the sub-file stored in the shard node belongs; By using the sub-files of the target file contained in the associated file located in the shard node, the associated sub-files in the shard node are determined.

6. The method according to claim 5, characterized in that, The process of identifying related files among the target files includes: Retrieve the associated file declaration information provided by the user; Based on the associated file declaration information, identify the associated files that have a relationship among the target files.

7. The method according to claim 5, characterized in that, The step of determining the associated sub-files in the shard node using the sub-files of the target file contained in the associated file includes: Identify the subfiles stored in the shard node that match the predetermined matching rules among the subfiles belonging to each specified file, and use them as associated subfiles; The specified file is the target file contained in the associated file, and the predetermined matching rules include: having the same sub-file identifier, or having matching column content.

8. The method according to claim 7, characterized in that, Before determining which sub-files belonging to each specified file and stored in the shard node conform to the predetermined matching rules, and using them as associated sub-files, the determination method further includes: Detect the number of sub-files belonging to each specified file stored in the shard node; If the detected quantity is 1 for all of them, the subfiles belonging to each of the specified files will be identified as associated subfiles. If the detected number is greater than 1, then the step of determining the sub-files that conform to the predetermined matching rules among the sub-files stored by the shard node and belonging to each specified file is executed, and these sub-files are used as associated sub-files.

9. A distributed database system, characterized in that, include: Management server and multiple shard nodes; Each shard node is used to determine the file information of that shard node; The file information of the shard node is reported to the management server; wherein, the file information of the shard node is used at least to indicate the associated sub-files in the sub-files stored by the shard node, and the associated sub-files are multiple sub-files that belong to different files but have an association relationship; The management server is used to obtain file information from each shard node; select files to be migrated from the sub-files included in at least one shard node according to a predetermined file selection rule and based on the obtained file information; and migrate the files to be migrated to the expansion node; wherein, the file selection rule includes: for the associated sub-file, when selecting files, all sub-files included in the associated sub-file are used as the selection granularity; the expansion node is a shard node pre-added in the distributed database system when the system is expanded; the files to be migrated include associated sub-files and sub-files without association, and when associated sub-files are included, the included associated sub-files are selected from the sub-files included in at least one shard node.

10. The system according to claim 9, characterized in that, The management server selects files to be migrated from the sub-files included in at least one shard node according to predetermined file selection rules and based on the acquired file information. Specifically: Based on the specified index values ​​of each shard node, at least one shard node for which files need to be migrated is determined, as well as the amount of data to be migrated corresponding to the at least one shard node; wherein, the specified index values ​​are used to characterize the storage space usage. For each of the at least one sharded nodes, according to a predetermined file selection rule, select an independent sub-file and / or associated sub-file from the sub-files stored by that node that matches the amount of data to be migrated corresponding to that node, and use it as the file to be migrated for that node; The independent sub-file is a sub-file that is not associated with other sub-files.

11. The system according to claim 10, characterized in that, The management server selects, according to predetermined file selection rules, independent sub-files and / or associated sub-files from the sub-files stored by the node that match the amount of data to be migrated corresponding to the node, as the files to be migrated for that node, including: Determine the first data volume of the associated sub-files indicated by the file information of the node, and the second data volume of the independent sub-files in the shard node; According to the predetermined file selection rules, based on the first data volume and the second data volume, independent sub-files and / or associated sub-files that match the amount of data to be migrated corresponding to the node are selected from the sub-files stored in the node and used as the files to be migrated for the node.

12. The system according to claim 9, characterized in that, The management server migrates the files to be migrated to the expansion node, including: A migration instruction for the file to be migrated is issued to the shard node to which the file to be migrated belongs, so that the shard node receiving the migration instruction can migrate the file to be migrated to the expansion node with sub-files as the migration granularity.

13. The system according to any one of claims 9-12, characterized in that, Each shard node determines its file information, including: Identify the related files among the target files, where the target file is the file to which the sub-file stored in the shard node belongs; Using the sub-files of the target file contained in the associated file located in the shard node, determine the associated sub-files in the shard node; Based on the identified associated sub-files, generate the file information for that shard node.

14. The system according to claim 13, characterized in that, Each shard node identifies related files within each target file, including: Retrieve the associated file declaration information provided by the user; Based on the associated file declaration information, identify the associated files that have a relationship among the target files.

15. The system according to claim 13, characterized in that, Each shard node uses the sub-files of the target file contained in the associated file located within that shard node to determine the associated sub-files in that shard node, including: Identify the subfiles stored in the shard node that match the predetermined matching rules among the subfiles belonging to each specified file, and use them as associated subfiles; The specified file is the target file contained in the associated file, and the predetermined matching rules include: having the same sub-file identifier, or having matching column content.

16. The system according to claim 15, characterized in that, Before each shard node determines which sub-files belonging to each specified file and conforming to a predetermined matching rule are used as associated sub-files, it is also used for: Detect the number of sub-files belonging to each specified file stored in the shard node; If the detected quantity is 1 for all of them, the subfiles belonging to each of the specified files will be identified as associated subfiles. If the detected number is greater than 1, then the step of determining the sub-files that conform to the predetermined matching rules among the sub-files stored by the shard node and belonging to each specified file is executed, and these sub-files are used as associated sub-files.

17. A data processing apparatus for a distributed database system, characterized in that, Management servers used in distributed database systems; The distributed database system further includes multiple sharding nodes; the device includes: The acquisition module is used to acquire file information of each shard node; wherein, the file information of any shard node is used to indicate at least the associated sub-files in the sub-files stored by that shard node, and the associated sub-files are multiple sub-files that belong to different files but have an association relationship; The selection module is used to select files to be migrated from the sub-files included in at least one shard node based on the acquired file information and according to predetermined file selection rules. The file selection rules include: for the associated sub-files, when selecting files, using all sub-files contained in the associated sub-file as the selection granularity; the files to be migrated include associated sub-files and sub-files without association, and when associated sub-files are included, the included associated sub-files are selected from the sub-files included in at least one shard node. The migration module is used to migrate the file to be migrated to the expansion node; wherein the expansion node is a shard node that is pre-added in the distributed database system when the system is expanded.

18. A management server, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-8.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-8.

Images