Data processing method, system, device, and medium

Abstract

Embodiments of the present application provide a data processing method, system, device and medium. The method comprises: in response to a capacity expansion request, determining a first data group stored in a distributed system to be expanded and a corresponding first directory file; splitting the first directory file to generate at least one first group directory item; wherein each first group directory item corresponds to target data of a logical group in the first data group; and writing the target data corresponding to the first group directory item to a second data group after expansion according to a preset capacity expansion rule. When expanding the capacity, the logical group capacity is expanded by using the newly created directory item, and the corresponding computing node is expanded for the first group directory item corresponding to each logical group, without migrating the target data in the first data group, which can improve the data processing capability of the distributed system and effectively improve the capacity expansion efficiency. The distributed system service does not need to be interrupted, and the capacity is expanded step by step, effectively improving the capacity expansion efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a data processing method, system, device and medium. BACKGROUND

[0002] In a distributed system, especially in a massive data scenario, a cluster composed of computing nodes of various scales is used for parallel computing to shorten the data processing time of the overall system.

[0003] As the data scale continues to grow, the data managed by each node is more and more, and the data processing is slower and slower. At this time, it is necessary to split and expand. One part of the data in the storage node is split into multiple data groups according to certain rules, and the computing node is also expanded into multiple. After expansion, the one-to-one correspondence between the computing node and the data group is still maintained. However, in the process of splitting and expanding the distributed system, the distributed system cannot provide services to the outside. SUMMARY

[0004] To solve or improve the problems in the prior art, embodiments of the present application provide a data processing method, system, device and medium.

[0005] In a first aspect, in one embodiment of the present application, a data processing method is provided. The method applied to a storage node of a distributed system comprises:

[0006] In response to an expansion request, determining a first data group stored in the distributed system to be expanded and a corresponding first directory file;

[0007] Splitting the first directory file to generate at least one first group directory item; wherein each first group directory item corresponds to target data of a logical group in the first data group;

[0008] According to a preset expansion rule, writing the target data of the logical group corresponding to the first group directory item into a second data group after expansion.

[0009] In a second aspect, in one embodiment of the present application, a data processing method is provided. The method is applied to a computing node of a distributed system, and comprises:

[0010] Based on an expansion request, establishing at least one computing node and at least one first group directory item corresponding to a first data group in the same distributed system; wherein the first group directory item is obtained by splitting a first directory file corresponding to the first data group, and each first group directory item corresponds to target data of a logical group in the first data group;

[0011] After the target data of the logical group corresponding to the first group directory item is written into the second data group after expansion according to the preset expansion rule in the distributed system, an association relationship between the computing node and the second data group after expansion is established.

[0012] In a third aspect, an embodiment of the present application provides a data processing system, comprising:

[0013] a storage node configured to execute the data processing method in the first aspect;

[0014] a computing node configured to execute the data processing method in the second aspect.

[0015] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor; wherein,

[0016] the memory is configured to store a program;

[0017] the processor is coupled to the memory and is configured to execute the program stored in the memory, so as to implement the method in the first aspect or implement the method in the second aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a non-transitory machine readable storage medium, which stores executable code, when the executable code is executed by a processor of an electronic device, the processor executes the data processing method in the first aspect or the data processing method in the second aspect.

[0019] The technical scheme provided by the embodiments of the present application determines a first data group in a distributed system when there is an expansion demand of the distributed system. The first data group in the same distributed system is expanded, so as to avoid cross-system data migration and copying during expansion, and effectively reduce resource overhead generated during expansion. After the first data group and the first directory file corresponding to the first data group are determined, the first directory file is split according to a preset rule, and at least one first group directory item is obtained. The first group directory item obtained here corresponds to target data of each logical group in the first data group, that is, during expansion, the logical group expansion is realized by using the newly created directory item, and the computing node corresponding to the first group directory item of each logical group is expanded, without the need of migrating, copying or other operations on the target data in the first data group, so as to effectively improve the expansion efficiency while improving the data processing capability of the distributed system. Through the above scheme, when the distributed system is expanded, the service of the distributed system does not need to be interrupted, and the expansion is step by step, so as to effectively improve the expansion efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced as follows. Obviously, the accompanying drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort based on these drawings.

[0021] Figure 1 The flowchart of the data processing method provided by the embodiments of the present application is shown in the figure.

[0022] Figure 2 The schematic diagram of the logical grouping illustrated by the embodiments of the present application is shown in the figure.

[0023] Figure 3 The flowchart of the capacity expansion method provided by the embodiments of the present application is shown in the figure.

[0024] Figure 4 The flowchart of another data processing method provided by the embodiments of the present application is shown in the figure.

[0025] Figure 5 The schematic diagram of the distributed system capacity expansion illustrated by the embodiments of the present application is shown in the figure.

[0026] Figure 6 The schematic diagram of the data processing system provided by the embodiments of the present application is shown in the figure.

[0027] Figure 7 The structural schematic diagram of the data processing device provided by the embodiments of the present application is shown in the figure.

[0028] Figure 8 The structural schematic diagram of another data processing device provided by the embodiments of the present application is shown in the figure.

[0029] Figure 9 The structural schematic diagram of an electronic device provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0030] In order to make the person skilled in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the accompanying drawings of the embodiments of the present application.

[0031] In some of the flowcharts described in the specification, claims, and drawings described above, multiple operations are described in a particular, sequential order. However, unless otherwise specified, the operations can be performed in any order, or in parallel, or can be omitted. The numbering of operations presented in the specification, claims, and drawings described above is merely illustrative and does not necessarily represent an execution sequence. Unless otherwise specified, the numbering of operations is not intended to reflect an execution sequence. Skilled artisans will appreciate that the order of execution can differ from that which is described. Further, some of the operations can not be executed, or can be executed in a different order, or can be omitted, without departing from the scope of the disclosure. Additionally, the description of the embodiments in the specification, claims, and drawings described above is merely illustrative and does not necessarily represent the only way of implementing the disclosure. Based on the embodiments described in the disclosure, skilled artisans will appreciate that all other embodiments obtained without creative work are within the scope of the disclosure.

[0032] In the existing split expansion scheme of the distributed system, in order to avoid the problem of data read-write error occurring in the expansion process, it is usually necessary to suspend part or all of the service capabilities in the distributed system. However, when data migration or copying of the distributed system is performed, especially when cross-system migration or copying is involved, the larger the amount of data to be expanded, the more distributed system resources are consumed, and the longer the time is consumed, which will directly affect the user experience. Therefore, a solution is needed that can effectively improve the efficiency of the distributed system expansion.

[0033] Terminology:

[0034] Hard link: (hard link) is a computer file system in which multiple file directory entries equally share the same file storage unit. Hard links must be in the same file system.

[0035] Copy on write: (copy on write) is an optimization strategy in the field of computer programming. The core is that if multiple requesters access the same resource at the same time, they will all get the pointer or reference of the resource, and only when the requester tries to modify the resource, a dedicated copy will be copied to the requester. The resource seen by other requesters remains unchanged.

[0036] Shard split: (shard split) in the field of parallel computing, data replicas and computing nodes are one-to-one. When the computing node is to be increased, the data replica is also split into multiple subgroups according to certain rules, and the computing node is corresponding to it.

[0037] The technical solutions implemented by the present application will be explained in conjunction with specific embodiments.

[0038] AsFigure 1 A flowchart of a data processing method provided by an embodiment of the present application is shown. The execution subject of the method can be a storage node in a distributed system. From Figure 1 It can be seen that the method specifically comprises the following steps:

[0039] 101: In response to a capacity expansion request, determining a first data group stored in the distributed system to be expanded and a corresponding first directory file.

[0040] 102: Splitting the first directory file to generate at least one first group directory item; wherein each first group directory item corresponds to target data of a logical group in the first data group.

[0041] 103: According to a preset capacity expansion rule, writing the target data of the logical group corresponding to the first group directory item into a second data group after expansion.

[0042] It should be noted that, in the technical solution of the present application, in order to reduce the resource overhead and network burden caused by the process of migrating and copying target data in the first data group during capacity expansion, the first directory file is split in a hard link manner of the distributed system to obtain at least one first group directory item. The plurality of first group directory items equally share the data in the first data group. That is, at this time, any computing node can access any data in the first data group through any first group directory item. At the same time, any computing node updates the data in the first data group through the corresponding first group directory item, and other first group directory items also share the updated data.

[0043] The target data in the logical group corresponding to the at least one first group directory item obtained by splitting is different from each other, and there is no intersection or duplicate data between the target data in each logical group. Each logical group has its corresponding first group directory item and corresponding computing node, that is, each logical group can be regarded as an independent data group in a logical form, realizing logical form expansion, and effectively improving the data processing capability of each logical group. Of course, although this logical form expansion can temporarily meet or alleviate the data processing pressure of the distributed system, it still needs to realize physical expansion to be a real expansion.

[0044] For example, as Figure 2 A schematic diagram of a logical group illustrated by an embodiment of the present application is shown. From Figure 2As can be seen, assuming that the first directory file is split to obtain the first grouped directory item A1, the second grouped directory item A2 and the third grouped directory item A3, the data contained in the first data group includes the target data B1, the target data B2 and the target data B3. The first data group is split to obtain the logical group C1, the logical group C2 and the logical group C3. Among them, the logical group C1 contains the target data B1, the logical group C2 contains the target data B2, and the logical group C3 contains the target data B3. The target data contained in each logical group is different. While the storage node is expanded in logical form, the computing node is also expanded to obtain the computing node D1, the computing node D2 and the computing node D3. An association relationship between the computing node D1 and the first grouped directory item A1 is established, an association relationship between the computing node D2 and the first grouped directory item A2 is established, and an association relationship between the computing node D3 and the third grouped directory item A3 is established. Through the above scheme, the corresponding relationship between each computing node and each logical group is established, the number of computing nodes is increased, the logical form expansion is realized, and the data processing capacity of the distributed system is improved.

[0045] As described above, obtaining the logical group is only to realize the logical form expansion, but it does not realize the physical expansion of the first data group. When subsequent splitting expansion is performed on the first data group, the logical group can be used as a basic expansion unit for expansion. That is, the target data of each logical group is respectively written into the expanded second data group corresponding thereto. Since multiple logical groups are obtained through logical form expansion, the replication migration can be performed according to the logical group without stopping the overall service capability of the distributed system (for example, temporarily suspending the data processing capability of a certain logical group in batches, such as temporarily suspending the logical group currently copying the target data for a short time). The migration demand can be met without stopping the overall service of the distributed system.

[0046] When the physical expansion is performed based on the logical group, the target data in the logical group corresponding to the first grouped directory item is written into the expanded second data group. The second data group here is different from the first data group. Since in the present scheme, the logical group is expanded into at least one second data group one by one, the original first data group needs to be retained in order to meet the data processing demand of other logical groups that have not been expanded.

[0047] It should be noted that the second data group obtained by expansion can be multiple, and each data group has its corresponding computing node. The first data group and the second data group are in the same distributed system, and the first data group and the second data group can be in the same storage node or in different storage nodes.

[0048] The expansion rule has two types, one is to expand according to the logical grouping order, and the other is to set the visible filtering condition for write-time expansion. The specific examples will be illustrated in the following embodiments, which will not be repeated here.

[0049] When the expansion rule is to set the visible filtering condition for write-time expansion, it specifically includes the following steps: in response to a data update request for the first grouping directory item, based on the visible filtering condition, the target data of the logical grouping corresponding to the first grouping directory item is written to the expanded second data grouping.

[0050] The data update request includes a data write request, a data modification request, a data deletion request, and the like for the target data corresponding to the first grouping directory item. The data update request can be understood as a request with a true modification demand for the target data. The request that does not have a modification impact on the target data is not a data update request.

[0051] In the present scheme, only when there is a data update request for the first grouping directory item, the expansion operation for the target data corresponding to the first grouping directory item is performed. Specifically, if multiple requesters access the same target data at the same time, these requesters will all obtain the pointer or reference of the target data, and only the real requester who tries to modify the target data will copy a dedicated copy (the second data grouping) of the logical grouping (where the target data is located) to the requester (the computing node) to meet the demand of the requester, while the resources (the first data grouping and the target data in other logical groupings) seen by other requesters remain unchanged. Unnecessary copying can be delayed or even avoided to occupy less memory, and a shared memory can save space. In other words, when expanding, the first data grouping with a large amount of data is split into multiple logical groupings, each of which has a corresponding computing node, which can meet the expansion demand of multiple computing nodes in a logical form and improve the data processing capability to a certain extent. When there is an update demand for a target data in the first data grouping, physical expansion is performed based on the logical grouping where the updated target data is located, without expanding other logical groupings without update demand, realizing batch expansion of a small amount of data, and expanding the logical grouping with actual data processing demand, which can effectively reduce or alleviate the expansion pressure.

[0052] The visible filtering condition refers to a rule that restricts the computing node to have only directional data processing capability for the specified target data. That is, the target data meeting the visible filtering condition is only visible to the computing node (that is, the computing node is allowed to process the visible target data (such as data processing such as addition, deletion, modification, and query)), and is not visible to other computing nodes, and the computing node is also not visible to the target data of the logical grouping corresponding to other computing nodes.

[0053] For example, the target data visible to the computing node can be restricted by setting the where shardkey statement as the visible filter condition. Specifically, if the where shardkey statement is not added as the visible filter condition, all the data in the storage nodes in the distributed system will be traversed in the execution process. In other words, if the computing node is not restricted by the visible filter condition, when the target data in a certain logical group is updated, the data processing process of the computing node without the visible filter condition can be affected, for example, when the certain logical group is expanded, the computing node without the visible filter condition cannot update the target data in the expanded logical group.

[0054] Based on the visible constraint condition, when the expansion is performed, only the target data in the logical group can be expanded, and the other target data in the first data group (that is, the target data in other logical groups) will not be expanded, and the normal work of other logical groups will not be affected. In addition, when the targeted expansion is performed, the expansion action is performed only when there is an actual data update request for the target data corresponding to the first group directory item, unnecessary memory copying can be delayed or even avoided, a shared memory can be saved, and space can be saved. The corresponding physical group is created for the target data with real data processing needs, so as to further improve the data processing capacity.

[0055] The generation method of the visible filter condition includes the following steps: after generating at least one first group directory item, a first association relationship between a computing node and the first group directory item is established according to a split expansion rule. Based on the association relationship, the visible filter condition is generated to restrict the computing node to perform targeted data processing on the target data in the logical group corresponding to the first group directory item.

[0056] As described above, after the first directory file is split, a plurality of first group directory items are obtained, and each first group directory item corresponds to a logical group. In other words, the computing node can perform data processing on the target data in the logical group through the first group directory item having the first association relationship.

[0057] The split expansion rule is to calculate the remainder of the slice key of the array to be split according to the expansion number, for example, shardkey%2, which means that the expansion number is 2, and the remainder of the slice key is calculated by 2. Two first group directory items can be obtained. At the same time, the computing node is also expanded to obtain two computing nodes. The first association relationship between the first group directory item and the computing node is established. Before the expansion, the computing node can perform data processing on the target data in the logical group having the first association relationship.

[0058] For example, the first catalog file is split to obtain two first group directory items dentry:t1.bin and dentry:t2.bin. To meet the expansion requirement, two computing nodes are configured for the two first group directory items, and the corresponding visible filtering conditions shardkey%2=0 and shardkey%2=1 are set for the two computing nodes according to the split expansion rule. The first association relationship between shardkey%2=0 and dentry:t1.bin is established, and the first association relationship between shardkey%2=1 and dentry:t2.bin is established. That is, by setting the visible filtering condition, the routing behavior of the computing node is restricted, and only the target data in the logical group visible to the computing node can be directed to data processing. The directed data processing can be understood as that, in the case of setting the visible filtering condition of the computing node, the target data having the first association relationship with the computing node can be executed by the computing node to perform the data processing task.

[0059] As Figure 3 A flowchart of an expansion method provided by an embodiment of the present application is shown in FIG. 3. As can be seen from FIG. 3, the method comprises the following steps: Figure 3 As can be seen from FIG. 3, the method comprises the following steps:

[0060] 301: Based on the visible filtering condition, the target data corresponding to the first group directory item visible to the computing node is determined.

[0061] 302: The target data in the logical group corresponding to the first group directory item is read.

[0062] 303: The update data corresponding to the data update request is merged with the target data to obtain merged target data.

[0063] 304: The merged target data is written into the second data group after expansion.

[0064] 305: If there is at least one unfinished expansion task in the plurality of first group directory items corresponding to the first data group, at least one first group directory item of the unfinished expansion task and the corresponding visible filtering condition are retained.

[0065] 306: At least one first group directory item of the completed expansion task and the corresponding visible filtering condition are deleted.

[0066] In actual application, since the visible filtering condition is set for the computing node, after receiving the data update request, the corresponding logical group needs to be determined by the computing node visible to the target data to be updated by the data update request. Then, the target data in the logical group corresponding to the first group directory item is read by the computing node. The data to be updated by the data update request is merged with the currently read target data as the merged target data. Since the use demand of the target data in the logical group is high, the logical group can be physically expanded, and the merged target data is written into the second data group obtained by expansion. The second data group and the first data group are physically separated different data groups. Through the above embodiment, the first data group is changed from a logical group to a physical group, and at the same time, the first data group does not need to stop the distributed service capability, reducing the influence on user use demand in the expansion process.

[0067] As described above, in the case of splitting the first data group to obtain multiple logical groups, the physical group is executed according to the visible filtering condition for the logical group with actual data update demand. Since there are multiple logical groups, the data update request of the computing node may not occur at the same time, otherwise, there are logical groups that have not completed the expansion task, but these logical groups and the corresponding computing nodes still need to ensure that they can normally provide service capability for users, therefore, the computing node that has not completed the expansion, the first group directory item, the visible filtering condition set for the computing node, and the first association relationship between the computing node and the first group directory item will be retained.

[0068] For the logical group that has completed the expansion task, since the physical expansion has been completed, it is not necessary to set the logical group and set the visible filtering condition, therefore, the visible filtering condition of the computing node that has completed the expansion task, the first group directory item can be deleted, and the second group directory item for satisfying the computing node to access the second data group is re-created.

[0069] Through the above scheme, logical grouping is performed first to realize logical form expansion, and then physical expansion is performed for the logical group with actual data processing demand. The expansion demand can be met, and the influence of the distributed system on providing service capability for users during expansion is reduced (that is, the distributed system service does not need to be completely stopped), and the expansion efficiency is improved.

[0070] When the expansion rule is write-time expansion of setting the visible filtering condition, the method comprises the following steps: determining the directory order of at least one first group directory item. According to the directory order, the target data of the logical group corresponding to the first group directory item is written into the second data group after expansion.

[0071] After the logical grouping is established based on the above-mentioned scheme, the computing nodes can access each logical group without setting visible filtering conditions for the computing nodes, which means that the routing path of the computing nodes cannot be constrained. Therefore, when physical expansion is performed, targeted expansion cannot be performed on target data that has data update requirements. In this embodiment, the obtained first grouping directory items can be sorted to obtain a directory order (the sorting rule can be set according to user needs). When physical expansion is performed, expansion is performed one by one according to the directory order. Through the above-mentioned scheme, logical form expansion is performed first, and then batch physical expansion is performed. In this way, the expansion demand is met, and the impact of stopping the service of the distributed system on users during the expansion process is reduced.

[0072] In one or more embodiments of the present application, the first directory file is split to generate at least one first grouping directory item, including: splitting the first directory file according to the expansion quantity; and generating at least one first grouping directory item corresponding to the first data group.

[0073] The expansion quantity can be set according to user expansion needs. For example, if the data quantity in the first data group in the distributed system to be expanded is relatively large, multiple logical groups can be split equally or unequally, and each logical group corresponds to a first grouping directory item. It should be noted that when logical form expansion is performed, the multiple first grouping directory items obtained all correspond to the same first data group.

[0074] Because the multiple first grouping directory items obtained when the first directory file is split have a hard link relationship with the first data group, during the logical form expansion process, no data migration and copying operations are involved, the expansion operation can be quickly completed, and the impact of the expansion process on the service capability of the distributed system can be effectively reduced.

[0075] For example, the first directory file is split into two first grouping directory items, which can be set as two directories dentry:t1.bin and dentry:t2.bin, and two visible filtering conditions shardkey%2=0 and shardkey%2=1 are set. Further, the correspondence between the two directories dentry:t1.bin and dentry:t2.bin and the first data group is established. Because physical grouping has not been actually performed at this time, the two logical groups corresponding to dentry:t1.bin and dentry:t2.bin have a hard link relationship. During the establishment of the grouping directory items, no migration and copying of the first data group are required.

[0076] In one or more embodiments of the present application, after the target data of the logical group corresponding to the first group directory item is written into the second data group after expansion, the method further comprises: generating a second group directory item corresponding to the second data group; and establishing a second association relationship between the expanded computing node and the second group directory item, so that the expanded computing node performs data processing on the second data group through the second group directory item.

[0077] In actual application, after a logical group completes the expansion task, the target data corresponding to the logical group is written into a second data group, and the computing node and the directory corresponding to the logical group also need to be updated. The second association relationship between the computing node and the second data group needs to be re-established, and the first association relationship needs to be released. Meanwhile, the first group directory item of the logical group and the visible filtering condition set for the computing node need to be deleted, and a second group directory item of the logical group needs to be re-established, so that the expanded computing node performs data processing on the second data group through the second group directory item.

[0078] In the above manner, the association relationship between the data group and the computing node in the logical form expansion is modified, the association relationship between the computing node and the second data group after physical expansion can be quickly established, the rapid expansion is realized, and the adverse effect on users caused by stopping part of the service capability of the distributed system during the expansion process is reduced.

[0079] Based on the same idea, the present application also provides another data processing method. As shown in Figure 4 The flowchart of another data processing method provided by the present application is shown in FIG. 4. The execution subject of the method can be a computing node in a distributed system. As shown in Figure 4 It can be seen from FIG. 4 that the method comprises the following steps:

[0080] 401: based on an expansion request, establishing at least one computing node and at least one first group directory item corresponding to a first data group in the same distributed system; wherein the first group directory item is obtained by splitting a first directory file corresponding to the first data group, and each first group directory item corresponds to target data of a logical group in the first data group.

[0081] 402: after the target data of the logical group corresponding to the first group directory item is written into a second data group after expansion in the distributed system according to a preset expansion rule, establishing an association relationship between the computing node and the second data group.

[0082] In practical applications, when the distributed system is scaled up, the computing nodes also need to be scaled up accordingly, so that each scaled-up second data group has its corresponding computing node. The data processing requirements of the scaled-up distributed system are met. At the same time, each computing node is associated with a corresponding first group directory item, and logical form scaling is realized. In subsequent physical scaling according to the scaling rules for each logical group. One of the physical scaling methods is as follows: in response to a data update request for the first group directory item, based on the visible filtering condition, the target data in the logical group corresponding to the first group directory item is subjected to directional data processing, so that the storage node writes the target data of the logical group corresponding to the first group directory item to the second data group after scaling.

[0083] The computing node further performs physical scaling on the logical group in which the target data with data update requirements is located on the basis of logical form scaling based on logical groups, and writes the scaled target data into the second data group after scaling. The second data group mentioned here is equivalent to the first data group obtained by sharding.

[0084] In the above manner, after the computing node establishes an association with the first group directory item obtained by logical scaling, logical form scaling is realized, and then physical scaling is performed. The scaling task can be completed without stopping all services of the distributed system.

[0085] In order to facilitate understanding, the present scheme will be illustrated below in conjunction with specific embodiments. For example, Figure 5 A schematic diagram of the present application for illustrating the scaling of the distributed system.

[0086] From Figure 5 It can be seen from the above that in the existing distributed system, there is one computing node and one distributed system. In the distributed system, there is a first data group (inode&data in the figure), and there is a first directory file dentry.t1.bin. When the scaling operation is initiated, the first directory file to be scaled is first split to create multiple first group directory items by the distributed system hard link method, which are dentry.t1.bin and dentry.t2.bin in the figure. Since only directory items are created and the data content in the first data group is not copied, this process can be completed quickly.

[0087] According to the expansion rule, different visible filtering conditions are set on the new data packet, as shown in the figure, one computing node DB-app1 sets the visible filtering condition where shardkey%2=0, indicating that the target data in the logical group corresponding to the shardkey remainder of 0 is visible, and the other computing node DB-app2 sets the visible filtering condition where shardkey%2=1, indicating that the target data in the logical group corresponding to the shardkey remainder of 1 is visible. So that the computing nodes under different logical groups can only read the target data in the logical group, forming a logical data packet. Thus, the logical expansion is completed, and the query request of all data packets can be quickly responded.

[0088] When data needs to be written or updated, on the basis of the above logical grouping, the target data of the logical group is read, and the target data and the new write data or update data are merged and written to the expanded new grouping inode&data, forming a new grouping data file. At the same time, the first grouping directory item and the corresponding visible filtering condition created by the original hard link are deleted.

[0089] Based on the same idea, the embodiment of the application also provides a data processing system. As Figure 6 The schematic diagram of a data processing system provided by the embodiment of the application is shown in FIG. 1. As can be seen from Figure 6 The system is a distributed system, and the data processing system includes:

[0090] The storage node 61 is configured to determine a first data packet and a corresponding first directory file stored in the distributed system to be expanded in response to an expansion request;

[0091] The first directory file is split to generate at least one first grouping directory item; each first grouping directory item corresponds to the target data of a logical group in the first data packet.

[0092] According to a preset expansion rule, the target data of the logical group corresponding to the first grouping directory item is written to a second data packet after expansion.

[0093] The storage node 61 is configured to write, in response to a data update request for the first grouping directory item, the target data of the logical group corresponding to the first grouping directory item to the second data packet after expansion based on a visible filtering condition.

[0094] The generation mode of the visible filtering condition required by the computing node includes:

[0095] After generating the at least one first grouping directory item, a first association relationship between the computing node and the first grouping directory item is established according to a split expansion rule.

[0096] Based on the association relationship, the visible filter condition is generated, which restricts the computing node to perform directional data processing on target data in the logical group corresponding to the first group directory item.

[0097] The storage node 61 is configured to determine a directory order of at least one of the first group directory items.

[0098] According to the directory order, the target data of the logical group corresponding to the first group directory item is written into the second data group after expansion.

[0099] The storage node 61 is configured to split the first directory file according to the expansion quantity.

[0100] At least one of the first group directory items corresponding to the first data group is generated.

[0101] The storage node 61 is configured to determine, based on the visible filter condition, the target data corresponding to the first group directory item that is visible to the computing node.

[0102] The target data in the logical group corresponding to the first group directory item is read.

[0103] The update data corresponding to the data update request is merged with the target data to obtain merged target data.

[0104] The merged target data is written into the second data group after expansion.

[0105] The storage node 61 is configured to, if there is at least one unfinished expansion task in a plurality of first group directory items corresponding to the first data group, retain at least one of the first group directory items of the unfinished expansion task and the corresponding visible filter condition.

[0106] At least one of the first group directory items of the completed expansion task and the corresponding visible filter condition are deleted.

[0107] The storage node 61 is configured to generate a second group directory item corresponding to the second data group.

[0108] A second association relationship between the expansion obtained computing node and the second group directory item is established, so that the expansion obtained computing node performs data processing on the second data group through the second group directory item.

[0109] The computing node 62 is configured to, based on the expansion request, establish an association between at least one computing node and at least one first group directory item corresponding to a first data group in the same distributed system; wherein the first group directory item is obtained by splitting the first directory file corresponding to the first data group, and each first group directory item corresponds to target data of a logical group in the first data group.

[0110] After the target data of the logical group corresponding to the first group directory item is written into a second data group expanded according to a preset expansion rule, the computing node and the second data group are associated.

[0111] The computing node 62 is configured to, in response to a data update request for the first group directory item, perform targeted data processing on the target data in the logical group corresponding to the first group directory item based on a visible filtering condition, so that the storage node writes the target data of the logical group corresponding to the first group directory item into a second data group expanded.

[0112] Based on the same idea, the embodiment also provides a data processing device. As shown in Figure 7 A structural schematic diagram of a data processing device provided by the embodiment of the present application is shown in FIG. 1. Figure 7 As can be seen from FIG. 1, the device comprises:

[0113] The determining module 71 is configured to, in response to an expansion request, determine a first data group and a corresponding first directory file stored in a distributed system to be expanded.

[0114] The generating module 72 is configured to split the first directory file to generate at least one first group directory item; wherein each first group directory item corresponds to target data of a logical group in the first data group.

[0115] The writing module 73 is configured to write the target data of the logical group corresponding to the first group directory item into a second data group expanded according to a preset expansion rule.

[0116] The determining module 71 is configured to, in response to a data update request for the first group directory item, write the target data of the logical group corresponding to the first group directory item into a second data group expanded based on a visible filtering condition.

[0117] The generating module 72 is configured to, after generating at least one first group directory item, establish a first association between a computing node and the first group directory item according to a split expansion rule.

[0118] generate the visible filter condition for constraining the computing node to perform directional data processing on target data in the logical group corresponding to the first group directory entry according to the association relationship.

[0119] a writing module 73, configured to determine a directory order of at least one of the first group directory entries;

[0120] write target data in the logical group corresponding to the first group directory entry into the second data group after expansion according to the directory order.

[0121] a generating module 72, configured to split the first directory file according to the expansion quantity;

[0122] generate at least one of the first group directory entries corresponding to the first data group.

[0123] a writing module 73, configured to determine the target data corresponding to the first group directory entry that is visible to the computing node based on the visible filter condition;

[0124] read target data in the logical group corresponding to the first group directory entry;

[0125] merge the update data corresponding to the data update request with the target data to obtain merged target data;

[0126] write the merged target data into the second data group after expansion.

[0127] a writing module 73, configured to, if there is at least one unfinished expansion task in a plurality of the first group directory entries corresponding to the first data group, retain at least one of the first group directory entries of the unfinished expansion task and the corresponding visible filter condition;

[0128] delete at least one of the first group directory entries of the completed expansion task and the corresponding visible filter condition.

[0129] a writing module 73, configured to generate a second group directory entry corresponding to the second data group;

[0130] establish a second association relationship between the computing node obtained after expansion and the second group directory entry, so that the computing node obtained after expansion performs data processing on the second data group through the second group directory entry.

[0131] Based on the same idea, the embodiment also provides another data processing apparatus. As shown in Figure 8 Another data processing apparatus provided by the embodiment of the present application is shown in the structural schematic diagram. The apparatus is applied to a computing node. As can be seen from Figure 8 The apparatus comprises:

[0132] The establishing module 81 is configured to establish at least one computing node and at least one first group directory item corresponding to a first data group in the same distributed system based on the capacity expansion request, wherein the first group directory item is obtained by splitting a first directory file corresponding to the first data group, and each first group directory item corresponds to target data of a logical group in the first data group.

[0133] The writing module 82 is configured to establish an association between the computing node and a second data group after writing target data of the logical group corresponding to the first group directory item into the second data group according to a preset capacity expansion rule.

[0134] The data processing module 83 is configured to perform directional data processing on target data in the logical group corresponding to the first group directory item based on a visible filtering condition in response to a data update request for the first group directory item, so that the storage node writes the target data of the logical group corresponding to the first group directory item into the second data group.

[0135] An embodiment of the present application further provides an electronic device. The electronic device is a master node electronic device in a computing unit. As shown in Figure 9 An embodiment of the present application further provides an electronic device. The electronic device is a master node electronic device in a computing unit. As shown in

[0136] The storage 901 is configured to store a program.

[0137] The processor 902 is coupled with the storage and is configured to execute the program stored in the storage, so as to:

[0138] In response to a capacity expansion request, determining a first data group and a corresponding first directory file stored in a distributed system to be expanded;

[0139] Splitting the first directory file to generate at least one first group directory item, wherein each first group directory item corresponds to target data of a logical group in the first data group.

[0140] According to a preset capacity expansion rule, writing target data of the logical group corresponding to the first group directory item into a second data group after expansion.

[0141] The processor 902 is configured to write target data of the logical group corresponding to the first group directory item into a second data group after expansion based on a visible filtering condition in response to a data update request for the first group directory item.

[0142] The processor 902 is configured to, after generating at least one first group directory item, establish a first association relationship between a computing node and the first group directory item according to a split scaling rule;

[0143] Based on the association relationship, the visible filtering condition is generated to restrict the computing node to perform targeted data processing on target data in the logical group corresponding to the first group directory item.

[0144] The processor 902 is configured to determine a directory order of at least one first group directory item;

[0145] According to the directory order, the target data of the logical group corresponding to the first group directory item is written into the second data group after scaling.

[0146] The processor 902 is configured to split the first directory file according to the scaling number;

[0147] At least one first group directory item corresponding to the first data group is generated.

[0148] The processor 902 is configured to determine the target data corresponding to the first group directory item that is visible to the computing node based on the visible filtering condition;

[0149] The target data in the logical group corresponding to the first group directory item is read;

[0150] The update data corresponding to the data update request is merged with the target data to obtain merged target data;

[0151] The merged target data is written into the second data group after scaling.

[0152] The processor 902 is configured to, if there is at least one unfinished scaling task in a plurality of first group directory items corresponding to the first data group, retain at least one first group directory item of the unfinished scaling task and the corresponding visible filtering condition;

[0153] At least one first group directory item of the completed scaling task and the corresponding visible filtering condition are deleted.

[0154] The processor 902 is configured to generate a second group directory item corresponding to the second data group;

[0155] The second association relationship between the computing node obtained by scaling and the second group directory item is established, so that the computing node obtained by scaling performs data processing on the second data group through the second group directory item.

[0156] The above-mentioned memory 901 can be configured to store other various data to support operations on the electronic device. Examples of these data include instructions for any application program or method operating on the electronic device. The memory can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0157] Further, the processor 902 in the embodiment can be specifically: a programmable switching processing chip, wherein a data replication engine is configured in the programmable switching processing chip, and the data replication engine can replicate received data.

[0158] In addition to the above functions, the processor 902 can also implement other functions when executing programs in the memory. For details, refer to the descriptions in the foregoing embodiments. Further, as shown in the figure, the electronic device also includes other components such as a power supply component 904. Figure 9

[0159] The embodiment of the present application also provides a non-transitory machine readable storage medium, wherein the non-transitory machine readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes the method according to the corresponding embodiment. Figure 1 The method according to the corresponding embodiment.

[0160] Based on the above embodiments, when there is a capacity expansion demand in the distributed system, a first data group in the distributed system is determined. The first data group in the same distributed system is expanded, so as to avoid cross-system data migration and copying during expansion, and the resource overhead generated during expansion can be effectively reduced. After the first data group and the first directory file corresponding to the first data group are determined, the first directory file is split according to a preset rule, and at least one first group directory item is obtained. The first group directory item obtained here corresponds to the target data of each logical group in the first data group, that is, during expansion, logical group expansion is realized by using the new directory item, and the corresponding computing node is expanded according to the first group directory item corresponding to each logical group. No migration, copying or other operations are required for the target data in the first data group, so that the expansion efficiency can be effectively improved while improving the data processing capacity of the distributed system. Through the above scheme, when the distributed system is expanded, the service of the distributed system does not need to be interrupted, and the expansion efficiency is effectively improved.

[0161] ​In addition, when the user needs to modify the application related function (for example, the user finds that the application has a fault), the modification requirement can be sent to the corresponding console, and then the modification requirement of the user can be met by modifying the adaptation parameter, without the need to adjust the underlying code of the application or the cloud operating system, so that the application maintenance efficiency can be effectively improved.

[0162] The device embodiments described above are only schematic, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0163] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and the necessary general hardware platform, and of course, can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, and the computer software products can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0164] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method applied to a storage node in a distributed system, the method comprising: In response to a scaling request, determine the data files to be split in the distributed system to be scaled up; Multiple first group directory entries are created for the data file using hard links in the file system; wherein the data in the data file is logically split into multiple data groups, and each first group directory entry corresponds to the target data of a data group in the data file. According to the preset expansion rules, the target data of the data group corresponding to the first group directory item is written into the data group to form a group data file.

2. The method according to claim 1, wherein writing the target data of the data group corresponding to the first group directory item into the data group to form a group data file according to a preset expansion rule includes: In response to a data update request for the first group directory item, the target data of the data group corresponding to the first group directory item is written into the data group based on the visibility filtering conditions to form a group data file.

3. The method according to claim 2, wherein the visible filter conditions are generated in a manner comprising: After generating at least one first group directory entry, a first association relationship is established between the computing node and the first group directory entry according to the split expansion rules; Based on the aforementioned association, visible filtering conditions are generated to constrain the computing node to perform targeted data processing on the target data in the logical group corresponding to the first group directory item.

4. The method according to claim 1, wherein writing the target data of the data group corresponding to the first group directory item into the data group to form a group data file according to a preset expansion rule includes: Determine the directory order of at least one of the first group directory entries; According to the directory order, the target data of the data group corresponding to the first group directory item is written into the data group to form a group data file.

5. The method according to claim 1, wherein creating multiple first group directory entries for the data file using file system hard links includes: Using hard links in the file system, a first group directory entry is created for the data file to expand its size.

6. The method according to claim 2, wherein writing the target data of the data group corresponding to the first group directory item into the data group to form a group data file based on the visibility filtering condition includes: Based on the visibility filtering conditions, the target data corresponding to the first group directory item that is visible to the computing node is determined; Read the target data from the data group corresponding to the first group directory item; The updated data corresponding to the data update request is merged with the target data to obtain the merged target data; The target data to be merged is written into the data group to form a grouped data file.

7. The method according to claim 6, further comprising: If at least one of the multiple first group directory entries corresponding to the data file has not completed the expansion task, then at least one first group directory entry with the incomplete expansion task and the corresponding visibility filtering condition are retained. Delete at least one of the first group directory entries that has completed the expansion task and the corresponding visible filter conditions.

8. The method according to claim 1, after writing the target data of the data group corresponding to the first group directory entry into the data group to form a group data file, further includes: Generate the second group directory entry corresponding to the grouped data file; A second association is established between the expanded computing nodes and the second group directory entry, so that the expanded computing nodes can process the group data files through the second group directory entry.

9. A data processing method applied to a computing node in a distributed system, the method comprising: Based on the expansion request, at least one computing node and multiple first group directory entries corresponding to the first data group in the same distributed system are established; wherein, the multiple first group directory entries are created by using file system hard links for the data files to be split stored in the distributed system to be expanded, and the data in the data files is logically split into multiple data groups, and each first group directory entry corresponds to the target data of a data group in the data file. In a distributed system, according to a preset expansion rule, the target data of the data group corresponding to the first group directory entry is written into the data group to form a group data file, and then the association between the computing node and the data group is established.

10. The method of claim 9, further comprising: In response to a data update request for the first group directory item, based on visibility filtering conditions, targeted data processing is performed on the target data in the data group corresponding to the first group directory item, so that the storage node writes the target data of the data group corresponding to the first group directory item into the data group to form a group data file.

11. A data processing system, the system comprising: A storage node for performing the method according to any one of claims 1 to 8; A computing node for performing the method of any one of claims 9 to 10.

12. An electronic device, comprising a memory and a processor; wherein, The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program stored in the memory to implement the method of any one of claims 1 to 8, or the method of any one of claims 9 to 10.

13. A non-transitory machine-readable storage medium storing executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as claimed in any one of claims 1 to 8, or the method as claimed in any one of claims 9 to 10.

Images