Data processing method and server
By sharding the data and regenerating the correspondence between the server and the batch executor group, the problem that the batch executor could only process fixed servers was solved, thus achieving efficient and accurate data processing and rational utilization of resources.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INDUSTRIAL AND COMMERCIAL BANK OF CHINA
- Filing Date
- 2022-04-25
- Publication Date
- 2026-07-10
AI Technical Summary
The existing file server and batch executor have a fixed mapping relationship, which means that the batch executor can only process data from a specific server, resulting in data processing errors and improper resource configuration.
By sharding the data to be processed, a mapping relationship between the batch executor group and the sharded data is generated, and the mapping relationship between the server and the batch executor group is redefined based on the allocation rules, thus avoiding fixed mapping limitations.
It achieves high efficiency and accuracy in data processing, avoids data processing errors, and enables the rational use of resources.
Smart Images
Figure CN114816751B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data processing method and server. Background Technology
[0002] Businesses generate batches of data during their daily operations. Once stored on a file server, this data is processed by batch executors mounted on the file server, based on processing needs. The data is then categorized and stored in appropriate databases. However, as businesses grow, their data volume surges. To ensure efficient data processing, it's necessary to reconfigure the resources of existing file servers and batch executors to better utilize current resources to handle the increased business data, thereby saving manpower, material resources, and financial resources.
[0003] However, because there is a fixed mapping relationship between the existing file server and the batch executor, the batch executor can only read or write data from the file server with which it has a mapping relationship in order to process the business data on the file server. Therefore, after reconfiguring the resources, the batch executor will be unable to process the data on the corresponding file server, resulting in data processing errors. Summary of the Invention
[0004] This application provides a data processing method in which the batch executor group is no longer limited to a fixed mapping relationship with the server. The batch executor group only needs to determine the fragmented data to be processed based on the pre-set correspondence between the batch executor group and the fragmented data, and read and write the fragmented data from the corresponding server based on the regenerated correspondence between the batch executor and the server to process the fragmented data. This avoids the situation where the batch executor can only process data on the server with a fixed mapping relationship, which would lead to data processing errors.
[0005] In a first aspect, this application provides a data processing method, which is executed on any server in a server cluster, comprising:
[0006] The data to be processed is split into multiple target data fragments.
[0007] Based on the first correspondence, the target batch executor group corresponding to each target data segment is determined. The first correspondence includes the correspondence between multiple data segments and multiple batch executor groups. The target batch executor group includes at least two batch executors, and the batch executors store the execution logic code for processing data.
[0008] Based on the allocation rules, the target server corresponding to each target shard data is determined from the server cluster, and a second correspondence is generated so that the target shard data stored on the corresponding target server can be processed by the target batch executor group. The second correspondence includes the correspondence between multiple target servers and multiple target batch executor groups.
[0009] Secondly, this application also provides a server, the server comprising:
[0010] The processing module is used to perform fragment processing on the data to be processed, and obtain multiple target fragment data;
[0011] The determination module is used to determine the target batch executor group corresponding to each target data segment based on the first correspondence relationship. The first correspondence relationship includes the correspondence between multiple data segments and multiple batch executor groups. The target batch executor group includes at least two batch executors, and the batch executors store the execution logic code for processing data.
[0012] The generation module is used to determine the target server corresponding to each target shard data from the server cluster based on the allocation rules, and generate a second correspondence so that the target shard data stored on the corresponding target server can be processed by the target batch executor group. The second correspondence includes the correspondence between multiple target servers and multiple target batch executor groups.
[0013] Thirdly, this application also provides another type of server, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.
[0014] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the methods described above.
[0015] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.
[0016] This application provides a data processing method and server. The data processing method is executed on any server in a server cluster. The method includes: splitting the data to be processed into multiple target data segments; determining the target batch executor group corresponding to each target data segment based on a first correspondence between the multiple data segments and multiple batch executor groups; determining the target server corresponding to each target data segment from the server cluster based on allocation rules, and generating a second correspondence between the multiple target servers and multiple target batch executor groups, so that the target batch executor groups can process the target data segment stored on the corresponding target servers. This application maps batch executor groups to data segments, so that when processing data, the batch executor groups only process the corresponding data segments. This avoids the problem that the batch executor groups cannot specifically process target data segments because the target data segments are not stored on servers with a fixed mapping relationship to the batch executor groups. This avoids data processing errors, achieves efficient and accurate data processing, and realizes the rational utilization of resources. Attached Figure Description
[0017] Figure 1 This is a diagram illustrating the application environment of a data processing method in one embodiment.
[0018] Figure 2 This is a flowchart illustrating a data processing method in one embodiment;
[0019] Figure 3 This is a flowchart illustrating the data processing steps in one embodiment;
[0020] Figure 4 This is a flowchart illustrating the data processing method in another embodiment;
[0021] Figure 5 This is a flowchart illustrating the data processing method in another embodiment;
[0022] Figure 6 This is a flowchart illustrating the data processing method in another embodiment;
[0023] Figure 7 This is a flowchart illustrating the data processing method in another embodiment;
[0024] Figure 8 This is a flowchart illustrating the data processing method in another embodiment;
[0025] Figure 9 This is a structural block diagram of a server in one embodiment;
[0026] Figure 10 This is a diagram of the internal structure of a server in one embodiment. Detailed Implementation
[0027] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0028] The data processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 sends the data to be processed to any server 104 in the server cluster via the network. Server 104 processes the received data by segmenting it using the corresponding batch executor, obtaining multiple target segmented data. Based on the correspondence between segmented data and batch executors, it determines the target batch executor group corresponding to each segmented data. Simultaneously, based on allocation rules, it determines the target server corresponding to each segmented data, storing the segmented data on the corresponding target server. The batch executor group is no longer limited to a fixed mapping relationship with servers. The batch executor group only needs to determine the segmented data to be processed based on a pre-set correspondence between the batch executor group and the segmented data, and reads and writes the segmented data from the corresponding server based on the regenerated correspondence between the batch executor and the server to process the segmented data. This avoids the situation where the batch executor can only process data on servers with fixed mapping relationships, leading to data processing errors. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc. The server 104 can be implemented using a server cluster consisting of multiple servers.
[0029] In one embodiment, such as Figure 2 As shown, a data processing method is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:
[0030] Step S202: The data to be processed is split into multiple target data fragments.
[0031] The data to be processed can be of different types or the same type; examples include user identity data, user transaction data, corporate business data, savings business data, and credit card business data. Each server in the server cluster can process data of the same type; servers in different server clusters can process data of different types. If the data to be processed is of the same type, it can be received by any server in the server cluster. Since each server in the server cluster has at least two batch executors mounted on it, and each batch executor stores the execution logic code for processing data, the data stored on the server can be processed accordingly through these batch executors. Therefore, the data to be processed, which has a certain amount of data, can be segmented and processed to obtain multiple target data segments by any batch executor mounted on the server. The data sizes of the multiple target data segments can be the same or different. The data to be processed can be divided into fragments based on a pre-set number of fragments, based on the correlation between the data to be processed, or based on the size of the data to be processed and the storage space of the existing server, etc. This application does not limit the scope of the fragments.
[0032] It should also be noted here that after the data to be processed is fragmented, the sequence number of each target fragment can be randomly assigned, or the sequence number can be assigned according to the generation time of each target fragment in chronological order, or the sequence number can be assigned according to the size of the target fragment in ascending or descending order. This application does not limit this.
[0033] For example, the first server in the first server cluster receives 100GB of customer identity data transmitted from a terminal device. The first server processes the 100GB of customer identity data according to a data sharding rule that divides the data into 10 equal shards using a first batch executor, obtaining 10 target data shards. These 10 target data shards can be divided into shards according to their size, resulting in target data shards 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10. Each target data shard can contain 10GB of data.
[0034] Step S204: Based on the first correspondence, determine the target batch executor group corresponding to each target data segment. The first correspondence includes the correspondence between multiple data segments and multiple batch executor groups. The target batch executor group includes at least two batch executors, and the batch executors store the execution logic code for processing data.
[0035] The first correspondence is preset and can be a table, a document, a piece of code, etc. It can be stored on any one server in the server cluster, on every server in the cluster, or on a subset of servers. If the first correspondence is stored on only one server in the cluster, it can be stored in the server's shared memory address for easy access by other servers. The first correspondence includes correspondences between multiple data shards and multiple batch executor groups: this correspondence can be between the tag of the data shard and the tag of the batch executor group; it can also be between the name of the data shard and the name of the batch executor group; it can also be between the sequence number of the data shard and the sequence number of the batch executor group, etc., which are not limited in this application. Therefore, after determining the tag, name, and sequence number of the target data shard, querying the first correspondence allows for the rapid acquisition of the target batch executor group corresponding to the target data shard.
[0036] For example, the first correspondence may include the correspondence between the names of the sharded data and the names of the batch executor groups. Based on the first batch executor, after sharding the data to be processed, 10 target sharded data are obtained. These 10 target sharded data can be named as the first target sharded data, the second target sharded data, the third target sharded data, the fourth target sharded data, the fifth target sharded data, the sixth target sharded data, the seventh target sharded data, the eighth target sharded data, the ninth target sharded data, and the tenth target sharded data. If the first target data segment corresponds to the first batch executor group, then the first batch executor group is the target batch executor group corresponding to the first target data segment; if the second target data segment corresponds to the second batch executor group, then the second batch executor group is the target batch executor group corresponding to the second target data segment; if the third target data segment corresponds to the third batch executor group, then the third batch executor group is the target batch executor group corresponding to the third target data segment; if the fourth target data segment corresponds to the fourth batch executor group, then the fourth batch executor group is the target batch executor group corresponding to the fourth target data segment; if the fifth target data segment corresponds to the fifth batch executor group, then the fifth batch executor group is the target batch executor group corresponding to the fifth target data segment. The sixth target fragment data corresponds to the sixth batch executor group, so the sixth batch executor group is the target batch executor group corresponding to the sixth target fragment data; the seventh target fragment data corresponds to the seventh batch executor group, so the seventh batch executor group is the target batch executor group corresponding to the seventh target fragment data; the eighth target fragment data corresponds to the eighth batch executor group, so the eighth batch executor group is the target batch executor group corresponding to the eighth target fragment data; the ninth target fragment data corresponds to the ninth batch executor group, so the ninth batch executor group is the target batch executor group corresponding to the ninth target fragment data; the tenth target fragment data corresponds to the tenth batch executor group, so the tenth batch executor group is the target batch executor group corresponding to the tenth target fragment data.
[0037] The first batch executor group may include four batch executors, such as batch executor A, batch executor B, batch executor C, and batch executor D.
[0038] The second batch executor group may include four batch executors; the third batch executor group may include six batch executors; the fourth batch executor group may include two batch executors; the fifth batch executor group may include six batch executors; the sixth batch executor group may include four batch executors; the seventh batch executor group may include four batch executors; the eighth batch executor group may include four batch executors; the ninth batch executor group may include six batch executors; and the tenth batch executor group may include ten batch executors (the batch executors included in each batch executor group will not be listed here).
[0039] Step S206: Based on the allocation rules, determine the target server corresponding to each target shard data from the server cluster and generate a second correspondence so that the target shard data stored on the corresponding target server can be processed by the target batch executor group. The second correspondence includes the correspondence between multiple target servers and multiple target batch executor groups.
[0040] The allocation rules can be, for example, based on the data size of each target shard, the sequence number of each shard, or the generation time of each target shard, etc., and this application does not limit the specific allocation rules. After determining the corresponding target server for each target shard from the server cluster using the allocation rules, the correspondence between the target server and the target batch executor group can be obtained based on the target batch executor group corresponding to the target shard data. Since the correspondence between the batch executor group and the server in this application is generated after each piece of data to be processed, the correspondence between the generated batch executor group and the server can be different for different pieces of data to be processed. There is no fixed mapping relationship between the batch executor group and the server. Therefore, when processing data, the batch executor group only processes the target shard data that has a corresponding relationship with it. This avoids the problem that the batch executor group cannot process the target shard data specifically because the target shard data is not stored on a server with a fixed mapping relationship with the batch executor group, thus preventing data processing errors.
[0041] For example, server allocation can be based on the data size of each target fragment; the first target fragment can be allocated to a first server; the second target fragment to a second server; the third target fragment to a third server; the fourth target fragment to a fourth server; the fifth target fragment to a fifth server; the sixth target fragment to a sixth server; the seventh target fragment to a seventh server; the eighth target fragment to an eighth server; the ninth target fragment to a ninth server; and the tenth target fragment to a tenth server.
[0042] Furthermore, a second correspondence can be obtained based on the target batch executors corresponding to each target fragment data obtained above. The second correspondence is the correspondence between the target batch executor group and the server: this correspondence can be between the tag of the target batch executor group and the tag of the server; it can also be between the name of the target batch executor group and the name of the server; it can also be between the sequence number of the target batch executor group and the sequence number of the server, etc., and this application does not limit this. Therefore, after determining the tag, name, sequence number, etc. of the target batch executor group, querying the second correspondence allows for the rapid acquisition of the server corresponding to the target batch executor group.
[0043] For example, if the second correspondence is a correspondence between the names of servers and target batch executor groups, then after determining the target batch executor group for the target sharded data based on the first correspondence, and allocating target servers for the target sharded data according to the sharding rules, the following second correspondence can be generated:
[0044] The first server corresponds to the first batch executor group; the second server corresponds to the second batch executor group; the third server corresponds to the third batch executor group; the fourth server corresponds to the fourth batch executor group; the fifth server corresponds to the fifth batch executor group; the sixth server corresponds to the sixth batch executor group; the seventh server corresponds to the seventh batch executor group; the eighth server corresponds to the eighth batch executor group; the ninth server corresponds to the ninth batch executor group; and the tenth server corresponds to the tenth batch executor group.
[0045] This application provides a data processing method executed on any server in a server cluster. The method includes: segmenting the data to be processed to obtain multiple target segmented data; determining the target batch executor group corresponding to each target segmented data based on a first correspondence between the multiple segmented data and multiple batch executor groups; determining the target server corresponding to each target segmented data from the server cluster based on allocation rules, and generating a second correspondence between the multiple target servers and multiple target batch executor groups, so that the target batch executor groups can process the target segmented data stored on the corresponding target servers. This application maps batch executor groups to segmented data, ensuring that when processing data, the batch executor groups only process the corresponding segmented data. This avoids the problem of the batch executor groups being unable to process the target segmented data specifically because the target segmented data is not stored on a server with a fixed mapping relationship to the batch executor group. This prevents data processing errors, achieves efficient and accurate data processing, and simultaneously achieves the effect of rational resource utilization.
[0046] In one embodiment, such as Figure 3 As shown, Figure 3 An optional method embodiment for determining the target batch executor group corresponding to each target data segment provided in this application embodiment includes:
[0047] Step S302: Obtain the sequence number of the target fragment data.
[0048] The first correspondence includes the correspondence between the sequence numbers of multiple data shards and the sequence numbers of multiple batch executor groups. The first server may shard the data to be processed to obtain multiple target data shards, assign sequence numbers to each target data shard, and then determine the corresponding target batch executor group based on the sequence number of each target data shard. For example, the first server may obtain the sequence number of the first target data shard as A, the sequence number of the second target data shard as B, the sequence number of the third target data shard as C, the sequence number of the fourth target data shard as D, and so on.
[0049] Step S304: Determine the sequence number of the target batch executor group corresponding to the sequence number of the target fragment data from the first correspondence relationship based on the sequence number of the target fragment data.
[0050] Because the first correspondence includes the relationship between the sequence numbers of multiple data shards and the sequence numbers of multiple batch executor groups, once the sequence number of the target data shard is obtained, the sequence number of the target batch executor group corresponding to the target data shard can be determined by searching the first correspondence. Since this first correspondence can be stored in each server of the server cluster, or stored in one server and accessible to other servers at any time, this application can quickly determine the sequence number of the corresponding target batch executor group for each target data shard once the sequence number of the target data shard is obtained, thereby improving data processing efficiency.
[0051] For example, if the data segment with serial number A corresponds to the batch executor group with serial number 1, then the batch executor group 1 is the target batch executor group for the target data segment A; if the data segment with serial number B corresponds to the batch executor group with serial number 2, then the batch executor group 2 is the target batch executor group for the target data segment B; if the data segment with serial number C corresponds to the batch executor group with serial number 3, then the batch executor group 3 is the target batch executor group for the target data segment B; and so on.
[0052] Step S306: Determine at least two batch executors corresponding to the sequence number of the target batch executor group as the target batch executor group corresponding to the target fragment data.
[0053] Finally, based on the sequence number of the target batch executor group obtained above, the target batch executor group used to process the target fragment data can be determined. This target batch executor group includes at least two batch executors.
[0054] The data processing method provided in this application can quickly obtain the sequence number of the target batch executor group corresponding to the target data segment by using the sequence number of the segmented data and the first correspondence, which can improve the efficiency of data processing.
[0055] In one embodiment, such as Figure 4 As shown, Figure 4 An optional method embodiment for determining target servers for each target data shard, provided in this application, includes:
[0056] Step S402: Obtain the data size of each target data shard and the remaining space size of each server in the server cluster;
[0057] Step S404: Compare the data size of each target fragment with the remaining space size of each server in the server cluster to obtain the comparison result;
[0058] Step S406: Determine the target server corresponding to each target fragment data based on the comparison results.
[0059] The allocation rules described above can be varied. In this embodiment, the allocation of servers for the data shards is based on the data size of each shard and the remaining space of the server. This way, the servers allocated to the shards can better match the size of the shards and avoid situations where the server cannot store the shards.
[0060] For example, the size of the first target data fragment is 10G, the size of the second target data fragment is 30G, the size of the third target data fragment is 40G, the size of the fourth target data fragment is 50G, the size of the fifth target data fragment is 60G, the size of the sixth target data fragment is 15G, the size of the seventh target data fragment is 19G, the size of the eighth target data fragment is 35G, the size of the ninth target data fragment is 25G, and the size of the tenth target data fragment is 55G.
[0061] The first server has 12GB of free space, the second server has 35GB of free space, the third server has 50GB of free space, the fourth server has 70GB of free space, the fifth server has 100GB of free space, the sixth server has 80GB of free space, the seventh server has 50GB of free space, the eighth server has 60GB of free space, the ninth server has 120GB of free space, and the tenth server has 150GB of free space.
[0062] Therefore, the following can be done: allocate the first target fragment data to the first server; allocate the second target fragment data to the second server; allocate the third target fragment data to the third server; allocate the fourth target fragment data to the fourth server; allocate the fifth target fragment data to the tenth server; allocate the sixth target fragment data to the seventh server; allocate the seventh target fragment data to the eighth server; allocate the eighth target fragment data to the ninth server; allocate the ninth target fragment data to the sixth server; and allocate the tenth target fragment data to the fifth server.
[0063] In one embodiment, such as Figure 5 As shown, Figure 5 Another optional method embodiment for determining the target server for each target fragment of data, provided for embodiments of this application, includes:
[0064] Step S502: Obtain the number of target shard data and the number of servers in the server cluster;
[0065] Step S504: Perform modulo calculation on the number of target fragment data and the number of servers in the server cluster to obtain the modulo calculation result;
[0066] Step S506: Determine the target server corresponding to each target data segment based on the modulus calculation results.
[0067] The above method is applicable when the target fragments obtained after fragmenting the data to be processed are all of different sizes. Servers can be allocated to the target fragments based on their size. Alternatively, if the data to be processed can be fragmented equally according to its size, servers can be allocated based on the number of target fragments and the number of servers in the server cluster. Specifically, a modulo operation can be performed on the number of target fragments and the number of servers to determine the server number corresponding to each target fragment. This controls index out-of-bounds errors between the target fragments and the servers, avoiding allocation chaos caused by random allocation.
[0068] For example, if there are 3 target data fragments and 5 servers, the first target data fragment (5 / 1 = 5 with a remainder of 0) can be allocated to the first server; the second target data fragment (5 / 2 = 2 with a remainder of 1) can also be allocated to the first server; and the third target data fragment (5 / 3 = 1 with a remainder of 2) can also be allocated to the second server.
[0069] In one embodiment, such as Figure 6 As shown, Figure 6 Another optional method embodiment for determining the target server for each target fragment of data, provided for embodiments of this application, includes:
[0070] Step S602: Obtain the sequence number of the target shard data and the sequence number of the server in the server cluster;
[0071] Step S604: Perform a hash mapping between the sequence number of the target fragment data and the sequence number of the server in the server cluster to obtain the mapping result;
[0072] Step S606: Determine the server sequence number corresponding to each target data shard based on the mapping result;
[0073] Step S608: Determine the server corresponding to each server sequence number as the target server corresponding to each target shard data.
[0074] The allocation rule can also be determined based on hash mapping. Hash mapping is a method that maps one value to another using a function. It involves inputting the sequence number of the target shard data and the sequence number of the server in the server cluster into a function used for hash mapping, and then determining the server number corresponding to the target shard data based on the output. Because hash mapping is very fast, it can improve the speed of allocating servers to each target shard data, thereby further improving the data processing rate.
[0075] For example, the sequence numbers of the target fragment data (A, B, C, D, E, F, G, H, I, J) and the sequence numbers of the server (1, 2, 3, 4, 5, 6, 7, 8, 9, 10) can be input into the target function to obtain the mapping results of A-1, B-2, C-3, D-4, E-5, F-6, G-7, H-8, I-9, J-10, and finally obtain the target server corresponding to each target fragment data.
[0076] Optionally, the target fragment data can be stored on the corresponding target server.
[0077] After assigning each target fragment of data to its corresponding target server according to the allocation rules, the data can be distributed as follows: target fragment A is sent to server 1; target fragment B is sent to server 2; target fragment C is sent to server 3; target fragment D is sent to server 4; target fragment E is sent to server 5; target fragment F is sent to server 6; target fragment G is sent to server 7; target fragment H is sent to server 8; target fragment I is sent to server 9; and target fragment J is sent to server 10. The corresponding target fragment data is then stored on each target server.
[0078] In one embodiment, such as Figure 7 As shown, Figure 7 Another optional method embodiment for determining a target server for target fragment data provided in this application embodiment includes:
[0079] Step S702: If the target shard data does not have a corresponding target server, then add a new server to the server cluster.
[0080] Step S704: Store the target shard data in the new server, and add the correspondence between the new server and the target batch executor group corresponding to the target shard data to the second correspondence.
[0081] In cases where the allocation rules fail to assign corresponding target servers to each data shard, this might be due to an excessively large volume of data to be processed, resulting in a large number of target shards, while the number of target servers in the server cluster is insufficient, or the remaining storage space on the target servers is inadequate. To ensure data processing efficiency, new servers can be added, and the target shard data can be allocated to these new servers and stored in the corresponding memory addresses, preventing data processing delays. It should be noted that after adding a new server, the mapping between the new server and the corresponding target batch executor for the target shard data can be added to a second mapping. This ensures that the target batch executor corresponding to the target shard data can read and write the target shard data from the correct target server for processing.
[0082] In one embodiment, such as Figure 8 As shown, Figure 8 Another optional method embodiment for determining a target batch executor group for target fragment data provided in this application embodiment includes:
[0083] Step S802: If the target fragment data does not have a corresponding target batch executor group, then add a new batch executor group;
[0084] Step S804: Add the correspondence between the target shard data and the new batch executor group to the first correspondence; and add the correspondence between the target server corresponding to the target shard data and the new batch executor group to the second correspondence.
[0085] If the amount of data to be processed is large, splitting the data into chunks can yield a significant amount of target chunk data. However, if no target batch executor group corresponds to some or individual target chunk data in the first mapping, adding a target batch executor group can solve the problem of the target chunk data not being processed in a timely manner, thus improving data processing efficiency. Similarly, when a new batch executor group is added, the mapping between the new batch executor group and the chunk data can be added to the first mapping for subsequent use. Simultaneously, the mapping between the target server allocated to the target chunk data and the new batch executor can be added to the second mapping, enabling the new batch executor to read and write the target chunk data from the correct target server for processing.
[0086] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0087] Based on the same inventive concept, this application also provides a server for implementing the above-mentioned solution. The solution provided by this server is similar to the solution described in the above method; therefore, the specific limitations of one or more server embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.
[0088] In one embodiment, such as Figure 9 As shown, a server 900 is provided, including: a processing module 902, a determining module 904, and a determining generation module 906, wherein:
[0089] Processing module 902 is used to perform fragment processing on the data to be processed to obtain multiple target fragment data;
[0090] The determination module 904 is used to determine the target batch executor group corresponding to each target data segment according to the first correspondence relationship. The first correspondence relationship includes the correspondence relationship between multiple data segments and multiple batch executor groups. The target batch executor group includes at least two batch executors, and the batch executors store the execution logic code for processing data.
[0091] The generation module 906 is used to determine the target server corresponding to each target shard data from the server cluster based on the allocation rules, and generate a second correspondence so that the target shard data stored on the corresponding target server can be processed by the target batch executor group. The second correspondence includes the correspondence between multiple target servers and multiple target batch executor groups.
[0092] In one embodiment, the determining module 904 is specifically used to obtain the sequence number of the target fragment data; determine the sequence number of the target batch executor group corresponding to the sequence number of the target fragment data from the first correspondence relationship based on the sequence number of the target fragment data; and determine at least two batch executors corresponding to the sequence number of the target batch executor group as the target batch executor group corresponding to the target fragment data.
[0093] In one embodiment, the generation module 906 is specifically used to obtain the data size of each target shard data and the remaining space size of each server in the server cluster; compare the data size of each target shard data with the remaining space size of each server in the server cluster to obtain the comparison result; and determine the target server corresponding to each target shard data based on the comparison result.
[0094] In one embodiment, the generation module 906 is specifically used to obtain the number of target shard data and the number of servers in the server cluster; perform modulo calculation on the number of target shard data and the number of servers in the server cluster to obtain the modulo calculation result; and determine the target server corresponding to each target shard data based on the modulo calculation result.
[0095] In one embodiment, the generation module 906 is specifically used to obtain the sequence number of the target shard data and the sequence number of the server in the server cluster; perform a hash mapping on the sequence number of the target shard data and the sequence number of the server in the server cluster to obtain a mapping result; determine the server sequence number corresponding to each target shard data based on the mapping result; and determine the server corresponding to each server sequence number as the target server corresponding to each target shard data.
[0096] In one embodiment, the server further includes a storage module for storing the target shard data on the corresponding target server.
[0097] In one embodiment, the server further includes an adding module.
[0098] The module adds a new server to the server cluster when the target shard data does not have a corresponding target server; it stores the target shard data in the new server and adds the correspondence between the new server and the target batch executor group corresponding to the target shard data to the second correspondence.
[0099] In one embodiment, the adding module is further configured to: add a new batch executor group when there is no corresponding target batch executor group for the target shard data; add the correspondence between the target shard data and the new batch executor group to a first correspondence; and add the correspondence between the target server corresponding to the target shard data and the new batch executor group to a second correspondence.
[0100] The modules in the aforementioned server can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of the computer device as software, so that the processor can invoke and execute the operations corresponding to each module.
[0101] In one embodiment, a server is provided, the internal structure of which can be as follows: Figure 10 As shown, the server includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The server's memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The server's database stores target fragment data. The server's network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.
[0102] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0103] In one embodiment, a server is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps:
[0104] The data to be processed is split into multiple target data fragments.
[0105] Based on the first correspondence, the target batch executor group corresponding to each target data segment is determined. The first correspondence includes the correspondence between multiple data segments and multiple batch executor groups. The target batch executor group includes at least two batch executors, and the batch executors store the execution logic code for processing data.
[0106] Based on the allocation rules, the target server corresponding to each target shard data is determined from the server cluster, and a second correspondence is generated so that the target shard data stored on the corresponding target server can be processed by the target batch executor group. The second correspondence includes the correspondence between multiple target servers and multiple target batch executor groups.
[0107] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0108] Obtain the sequence number of the target fragment data; determine the sequence number of the target batch executor group corresponding to the sequence number of the target fragment data from the first correspondence relationship based on the sequence number of the target fragment data; determine at least two batch executors corresponding to the sequence number of the target batch executor group as the target batch executor group corresponding to the target fragment data.
[0109] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0110] Obtain the data size of each target shard and the remaining space size of each server in the server cluster; compare the data size of each target shard with the remaining space size of each server in the server cluster to obtain the comparison result; determine the target server corresponding to each target shard based on the comparison result.
[0111] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0112] Obtain the number of target data shards and the number of servers in the server cluster; perform modulo calculation on the number of target data shards and the number of servers in the server cluster to obtain the modulo calculation result; determine the target server corresponding to each target data shard based on the modulo calculation result.
[0113] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0114] Obtain the sequence number of the target shard data and the sequence number of the server in the server cluster; perform a hash mapping on the sequence number of the target shard data and the sequence number of the server in the server cluster to obtain the mapping result; determine the server sequence number corresponding to each target shard data based on the mapping result; and determine the server corresponding to each server sequence number as the target server corresponding to each target shard data.
[0115] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0116] The target fragment data is stored on the corresponding target server.
[0117] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0118] If the target shard data does not have a corresponding target server, add a new server to the server cluster; store the target shard data in the new server, and add the correspondence between the new server and the target batch executor group corresponding to the target shard data to the second correspondence.
[0119] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0120] If the target shard data does not have a corresponding target batch executor group, add a new batch executor group; add the correspondence between the target shard data and the new batch executor group to the first correspondence; and add the correspondence between the target server corresponding to the target shard data and the new batch executor group to the second correspondence.
[0121] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0122] The data to be processed is split into multiple target data fragments.
[0123] Based on the first correspondence, the target batch executor group corresponding to each target data segment is determined. The first correspondence includes the correspondence between multiple data segments and multiple batch executor groups. The target batch executor group includes at least two batch executors, and the batch executors store the execution logic code for processing data.
[0124] Based on the allocation rules, the target server corresponding to each target shard data is determined from the server cluster, and a second correspondence is generated so that the target shard data stored on the corresponding target server can be processed by the target batch executor group. The second correspondence includes the correspondence between multiple target servers and multiple target batch executor groups.
[0125] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0126] Obtain the sequence number of the target fragment data; determine the sequence number of the target batch executor group corresponding to the sequence number of the target fragment data from the first correspondence relationship based on the sequence number of the target fragment data; determine at least two batch executors corresponding to the sequence number of the target batch executor group as the target batch executor group corresponding to the target fragment data.
[0127] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0128] Obtain the data size of each target shard and the remaining space size of each server in the server cluster; compare the data size of each target shard with the remaining space size of each server in the server cluster to obtain the comparison result; determine the target server corresponding to each target shard based on the comparison result.
[0129] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0130] Obtain the number of target data shards and the number of servers in the server cluster; perform modulo calculation on the number of target data shards and the number of servers in the server cluster to obtain the modulo calculation result; determine the target server corresponding to each target data shard based on the modulo calculation result.
[0131] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0132] Obtain the sequence number of the target shard data and the sequence number of the server in the server cluster; perform a hash mapping on the sequence number of the target shard data and the sequence number of the server in the server cluster to obtain the mapping result; determine the server sequence number corresponding to each target shard data based on the mapping result; and determine the server corresponding to each server sequence number as the target server corresponding to each target shard data.
[0133] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0134] The target fragment data is stored on the corresponding target server.
[0135] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0136] If the target shard data does not have a corresponding target server, add a new server to the server cluster; store the target shard data in the new server, and add the correspondence between the new server and the target batch executor group corresponding to the target shard data to the second correspondence.
[0137] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0138] If the target shard data does not have a corresponding target batch executor group, add a new batch executor group; add the correspondence between the target shard data and the new batch executor group to the first correspondence; and add the correspondence between the target server corresponding to the target shard data and the new batch executor group to the second correspondence.
[0139] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:
[0140] The data to be processed is split into multiple target data fragments.
[0141] Based on the first correspondence, the target batch executor group corresponding to each target data segment is determined. The first correspondence includes the correspondence between multiple data segments and multiple batch executor groups. The target batch executor group includes at least two batch executors, and the batch executors store the execution logic code for processing data.
[0142] Based on the allocation rules, the target server corresponding to each target shard data is determined from the server cluster, and a second correspondence is generated so that the target shard data stored on the corresponding target server can be processed by the target batch executor group. The second correspondence includes the correspondence between multiple target servers and multiple target batch executor groups.
[0143] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0144] Obtain the sequence number of the target fragment data; determine the sequence number of the target batch executor group corresponding to the sequence number of the target fragment data from the first correspondence relationship based on the sequence number of the target fragment data; determine at least two batch executors corresponding to the sequence number of the target batch executor group as the target batch executor group corresponding to the target fragment data.
[0145] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0146] Obtain the data size of each target shard and the remaining space size of each server in the server cluster; compare the data size of each target shard with the remaining space size of each server in the server cluster to obtain the comparison result; determine the target server corresponding to each target shard based on the comparison result.
[0147] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0148] Obtain the number of target data shards and the number of servers in the server cluster; perform modulo calculation on the number of target data shards and the number of servers in the server cluster to obtain the modulo calculation result; determine the target server corresponding to each target data shard based on the modulo calculation result.
[0149] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0150] Obtain the sequence number of the target shard data and the sequence number of the server in the server cluster; perform a hash mapping on the sequence number of the target shard data and the sequence number of the server in the server cluster to obtain the mapping result; determine the server sequence number corresponding to each target shard data based on the mapping result; and determine the server corresponding to each server sequence number as the target server corresponding to each target shard data.
[0151] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0152] The target fragment data is stored on the corresponding target server.
[0153] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0154] If the target shard data does not have a corresponding target server, add a new server to the server cluster; store the target shard data in the new server, and add the correspondence between the new server and the target batch executor group corresponding to the target shard data to the second correspondence.
[0155] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0156] If the target shard data does not have a corresponding target batch executor group, add a new batch executor group; add the correspondence between the target shard data and the new batch executor group to the first correspondence; and add the correspondence between the target server corresponding to the target shard data and the new batch executor group to the second correspondence.
[0157] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0158] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0159] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A data processing method, characterized in that, The method is executed on any server in the server cluster, including: The data to be processed is split into multiple target data fragments. Based on the first correspondence, the target batch executor group corresponding to each target data segment is determined. The first correspondence includes the correspondence between multiple data segments and multiple batch executor groups. The target batch executor group includes at least two batch executors, and the batch executor stores the execution logic code for processing data. Based on the allocation rules, the target server corresponding to each target shard data is determined from the server cluster, and a second correspondence is generated so that the target shard data stored on the corresponding target server can be processed by the target batch executor group. The second correspondence includes the correspondence between multiple target servers and multiple target batch executor groups. The step of determining the target server corresponding to each target shard of data from the server cluster based on the allocation rules includes: Obtain the data size of each target data segment and the remaining space size of each server in the server cluster; The size of each target data fragment is compared with the remaining space size of each server in the server cluster to obtain the comparison result; Based on the comparison results, determine the target server corresponding to each target data fragment; or... Obtain the number of the target data fragments and the number of servers in the server cluster; The modulo operation is performed on the number of target data fragments and the number of servers in the server cluster to obtain the modulo operation result. The target server corresponding to each target data fragment is determined based on the modulo calculation result; or... Obtain the sequence number of the target data shard and the sequence number of the server in the server cluster; A hash mapping is performed between the sequence number of the target fragment data and the sequence number of the server in the server cluster to obtain the mapping result; Based on the mapping results, the server sequence number corresponding to each target data segment is determined; The server corresponding to each of the server serial numbers is determined as the target server corresponding to each of the target fragment data.
2. The method according to claim 1, characterized in that, The first correspondence includes the correspondence between the sequence numbers of multiple data shards and the sequence numbers of multiple batch executor groups. The step of determining the target batch executor group corresponding to each target data shard based on the first correspondence includes: Obtain the sequence number of the target fragment data; Based on the sequence number of the target fragment data, determine the sequence number of the target batch executor group corresponding to the sequence number of the target fragment data from the first correspondence relationship; At least two batch executors corresponding to the sequence number of the target batch executor group are identified as the target batch executor group corresponding to the target fragment data.
3. The method according to claim 1, characterized in that, The method further includes: The target fragment data is stored on the corresponding target server.
4. The method according to claim 3, characterized in that, The method further includes: If the target shard data does not have a corresponding target server, then a new server is added to the server cluster; The target shard data is stored in the new server, and the correspondence between the new server and the target batch executor group corresponding to the target shard data is added to the second correspondence.
5. The method according to claim 1, characterized in that, The method further includes: If the target fragment data does not have a corresponding target batch executor group, then add a new batch executor group; The correspondence between the target shard data and the new batch executor group is added to the first correspondence; and the correspondence between the target server corresponding to the target shard data and the new batch executor group is added to the second correspondence.
6. A data processing apparatus, characterized in that, The data processing device includes: The processing module is used to perform fragment processing on the data to be processed, and obtain multiple target fragment data; The determining module is used to determine the target batch executor group corresponding to each target data segment according to the first correspondence relationship. The first correspondence relationship includes the correspondence relationship between multiple data segments and multiple batch executor groups. The target batch executor group includes at least two batch executors. The batch executor stores the execution logic code for processing data. The generation module is used to determine the target server corresponding to each target shard data from the server cluster based on the allocation rules, and generate a second correspondence so that the target shard data stored on the corresponding target server can be processed by the target batch executor group. The second correspondence includes the correspondence between multiple target servers and multiple target batch executor groups. The determination and generation module is specifically used to obtain the data size of each target data segment and the remaining space size of each server in the server cluster; The size of each target data fragment is compared with the remaining space size of each server in the server cluster to obtain the comparison result; Based on the comparison results, the target server corresponding to each target data segment is determined; The determination and generation module is specifically used to obtain the number of target fragment data and the number of servers in the server cluster; The modulo operation is performed on the number of target data fragments and the number of servers in the server cluster to obtain the modulo operation result. Based on the modulo calculation results, the target server corresponding to each target data segment is determined; The determination and generation module is specifically used to obtain the sequence number of the target fragment data and the sequence number of the server in the server cluster; A hash mapping is performed between the sequence number of the target fragment data and the sequence number of the server in the server cluster to obtain the mapping result; Based on the mapping results, the server sequence number corresponding to each target data segment is determined; The server corresponding to each of the server serial numbers is determined as the target server corresponding to each of the target fragment data.
7. A server comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.