Nested distributed coding calculation system and method
By introducing nested distributed encoding computing in the distributed computing system, the encoding transmission between the central node and the distributed cluster is solved, and more efficient communication and computing performance is achieved.
Patent Information
- Application Number
- CN202410951769.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-07-16
AI Technical Summary
Existing distributed computing systems have high communication loads in the encoding computing process, especially when communicating across clusters, ignoring in-rack multicast opportunities, resulting in unnecessary communication loads.
A nested distributed encoding computing system is adopted to allocate input files and output functions through central nodes, and encoding transmission within and between clusters is realized in a distributed cluster to reduce communication load. The specific steps include: the central node allocates input files and output functions for the distributed cluster, the users in the distributed cluster perform mapping and encoding processing, the central node conducts inter-cluster communication, and reduces the communication load through exclusive or linear encoding.
It significantly reduces the communication load of distributed computing systems and reduces the computational complexity, which improves the scalability and robustness of the system compared with traditional solutions.
Smart Images

Figure CN119011578A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of distributed cluster processing technology, and in particular to a nested distributed coding computing system and method. Background Art
[0002] Distributed computing systems are on the rise in modern communication systems. Frameworks such as MapReduce are widely used to process a large number of computing tasks in parallel. MapReduce is a programming model for parallel operations on large data sets. MapReduce-type tasks consist of three phases: map, shuffle, and reduce. In the map phase, distributed users map the assigned input files to the intermediate value variables required to calculate the output function. In the shuffle phase, these intermediate value variables are exchanged between all users so that each user obtains the intermediate value variables required for the reduction phase. In the reduction phase, each user aggregates all received intermediate value variables and executes the reduction function on these intermediate value variables to generate the output results. However, traditional MapReduce solutions have high communication costs related to data transmission in the shuffle phase.
[0003] The existing technology uses coded distributed computing to alleviate the distributed communication bottleneck. Coded distributed computing can greatly reduce the communication cost of the shuffle stage by introducing additional calculations in the mapping stage and utilizing coded transmission in the shuffle stage. In order to improve the scalability, robustness and manageability of the system, practical large-scale distributed computing systems usually consist of multiple clusters, each containing multiple computing nodes. However, most of the current work focuses on the single-cluster case, where all worker threads are gathered in the same cluster. For cross-cluster coded computing, people often focus on reducing cross-cluster traffic, while ignoring multicast opportunities in intra-rack communication, which leads to unnecessary communication load.
[0004] In summary, how to reduce the communication load of distributed computing systems for encoding calculations has become a problem that needs to be solved urgently. Summary of the invention
[0005] The present invention provides a nested distributed coding computing system, method, electronic device and computer-readable storage medium, the main purpose of which is to solve the problem of how to reduce the communication load of the distributed computing system for coding computing, which has become an urgent problem to be solved.
[0006] To achieve the above object, the present invention provides a nested distributed coding computing system, the system comprising a central node and multiple distributed clusters, each distributed cluster comprising multiple users:
[0007] The central node is used to distribute input files and output functions to users in the distributed cluster;
[0008] The user in the distributed cluster is used to perform mapping according to the allocated input file to obtain an initial intermediate value variable, perform intra-cluster communication according to the initial intermediate value variable, obtain a first intermediate value variable and send it to the central node;
[0009] The central node is configured to perform inter-cluster communication with the distributed cluster according to the first intermediate value variable sent by a user in the distributed cluster;
[0010] The users in the distributed cluster are used to calculate the second intermediate value variable obtained through inter-cluster communication and the initial intermediate value variable according to the allocated output function to obtain a distributed calculation result.
[0011] In one embodiment of the present invention, when the central node executes the step of allocating input files and output functions to users in the distributed cluster, the steps include:
[0012] Evenly distribute the user assigned input file to each distributed cluster;
[0013] Allocate the input files obtained from each distributed cluster to each corresponding user;
[0014] Distributing the output function evenly to all distributed clusters;
[0015] The output function obtained by the distributed cluster allocation is allocated to each corresponding user.
[0016] In one embodiment of the present invention, when the user in the distributed cluster performs mapping according to the allocated input file to obtain the initial intermediate value variable, the step includes:
[0017] Selecting target distributed clusters one by one from the distributed clusters, and selecting target users one by one from the target distributed clusters;
[0018] Obtaining a target input file allocated to the target user;
[0019] Mapping is performed according to the target input file using a plurality of mapping functions to obtain corresponding initial intermediate value variables.
[0020] In one embodiment of the present invention, when the user in the distributed cluster performs the step of performing intra-cluster communication according to the initial intermediate value variable to obtain the first intermediate value variable and send it to the central node, the step includes:
[0021] Selecting target distributed clusters one by one from the distributed clusters, and selecting target users and adjacent users one by one from the target distributed clusters;
[0022] Obtaining initial intermediate value variables corresponding to the target user and the adjacent users;
[0023] The target user performs XOR encoding according to the corresponding initial intermediate value variable to obtain a corresponding first encoded message and sends it to the adjacent user;
[0024] The neighboring user decodes the encoded message to obtain the intra-cluster communication intermediate value variable transmitted by the target user.
[0025] In one embodiment of the present invention, when the target user performs the step of performing XOR encoding according to the corresponding initial intermediate value variable to obtain the corresponding first encoded message and sending it to the neighboring user, the step includes:
[0026] Determining directly useful intermediate values and auxiliary intermediate values from the corresponding initial intermediate value variables;
[0027] XOR encoding is performed on the direct useful intermediate value and the auxiliary intermediate value to obtain a corresponding first encoded message.
[0028] In one embodiment of the present invention, when all users in the target distributed cluster execute the step of obtaining the first intermediate value variable and sending it to the central node, the step includes:
[0029] Determining a slice size for dividing the encoded message;
[0030] Slicing the first intermediate value variable according to the slice size to obtain multiple slice values;
[0031] XOR encoding is performed according to the slice value to obtain a second encoded message.
[0032] In one embodiment of the present invention, when the central node executes the step of performing inter-cluster communication with the distributed cluster according to the first intermediate value variable sent by a user in the distributed cluster, the step includes:
[0033] Obtaining the first intermediate value variable transmitted by all users in each distributed cluster as the intermediate value variable to be processed;
[0034] Performing linear coding according to the intermediate value variables to be processed to obtain multiple groups of downlink coded messages;
[0035] The downlink encoded message is sent to users in the corresponding distributed cluster.
[0036] In one embodiment of the present invention, the second intermediate value variable obtained by the inter-cluster communication according to the output function obtained by allocation is obtained by the following steps: the user in the distributed cluster decodes the downlink encoded message according to the initial intermediate value variable and the first intermediate value variable to obtain the second intermediate value variable.
[0037] In one embodiment of the present invention, when the user in the distributed cluster performs the step of calculating the second intermediate value variable obtained by inter-cluster communication and the initial intermediate value variable according to the output function obtained by the distribution to obtain the distributed computing result, the step includes:
[0038] Determine a target intermediate value variable required by a user in a distributed cluster according to the second intermediate value variable, the first intermediate value variable, and the initial intermediate value variable;
[0039] Substitute the target intermediate value variable into the output function to obtain the distributed computing result corresponding to the user in the distributed cluster.
[0040] In order to solve the above problem, the present invention also provides a nested distributed coding calculation method, the method comprising:
[0041] The central node distributes input files and output functions to users in the distributed cluster;
[0042] The users in the distributed cluster perform mapping according to the allocated input files to obtain initial intermediate value variables, perform intra-cluster communication according to the initial intermediate value variables, obtain first intermediate value variables and send them to the central node;
[0043] The central node performs inter-cluster communication with the distributed cluster according to the first intermediate value variable sent by the user in the distributed cluster;
[0044] The users in the distributed cluster calculate the second intermediate value variable obtained through inter-cluster communication and the initial intermediate value variable according to the allocated output function to obtain a distributed calculation result.
[0045] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:
[0046] at least one processor; and,
[0047] a memory communicatively connected to the at least one processor; wherein,
[0048] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the functions of the nested distributed coding computing system described above.
[0049] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the functions of the nested distributed coding computing system described above.
[0050] The present invention takes into account the multicast opportunities in intra-rack communications, directly conducts intra-cluster communications through users in a distributed cluster, and realizes inter-cluster communications between distributed clusters through a central node, while realizing coded transmission of intra-cluster and inter-cluster communications, reducing the communication load in two stages, which can greatly reduce the communication load and reduce the computational complexity compared to traditional solutions. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 A system architecture diagram of a nested distributed coding computing system provided by an embodiment of the present invention;
[0052] Figure 2 A schematic diagram of a flow chart of a nested distributed coding calculation method provided by an embodiment of the present invention;
[0053] Figure 3 A schematic diagram of the structure of an electronic device for implementing the nested distributed coding computing system at a user or a central node in a distributed cluster provided by an embodiment of the present invention.
[0054] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0055] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0056] like Figure 1 , which is a system architecture diagram of a nested distributed coding computing system 100 provided in one embodiment of the present invention.
[0057] The nested distributed coding computing system proposed by the present invention includes a central node 101 and multiple distributed clusters 102 , and each distributed cluster 102 includes multiple users 103 .
[0058] In the embodiment of the present invention, the central node 101 is used to distribute input files and output functions to users 103 in the distributed cluster 102 ; the distributed cluster 102 also includes a switch 104 , and the users 103 in the distributed cluster 102 communicate with the central node 101 through the switch 104 .
[0059] Among them, the nested distributed coding computing system can be a multi-cluster MapReduce computing system, the central node 101 is a central server, connected to multiple distributed clusters 102 to achieve cross-cluster communication between users 103 of the distributed clusters 102; the distributed cluster 102 has any number of users 103 and computing loads, each cluster contains a group of users 103 who want to jointly calculate a MapReduce task, and each user 103 can directly "communicate within the cluster" with users 103 in the same cluster.
[0060] The present invention can greatly reduce the communication load and achieve the optimal cross-cluster communication load. In addition, compared with the traditional coding calculation scheme, the calculation complexity can be significantly reduced.
[0061] In the embodiment of the present invention, when the central node 101 executes the step of allocating input files and output functions to the user 103 in the distributed cluster 102, the steps include: evenly distributing the input files allocated to the user 103 to each distributed cluster 102; distributing the input files allocated to each distributed cluster 102 to each corresponding user 103; evenly distributing the output functions to all distributed clusters 102; and distributing the output functions allocated to the distributed clusters to each corresponding user 103.
[0062] Among them, the input file can be first divided into multiple non-overlapping sub-files, and after the multiple sub-files are evenly distributed to each distributed cluster 102, each distributed cluster 102 can obtain the same number of non-overlapping input files; then, according to the input files obtained by each distributed cluster 102, each user 103 in the cluster is evenly distributed, so that all users 103 in a cluster can obtain the same number of input files.
[0063] Likewise, after the output functions are evenly distributed to each user 103 in all distributed clusters 102 , each user 103 can obtain the same number of output functions.
[0064] In an actual application scenario of the present invention, the nested distributed coding computing system includes C=3 distributed clusters 102, each distributed cluster 102 has K=3 users 103, has N=9 input files, Q=9 output functions, and the computing load of the distributed cluster 102 is r1=2. The computing load in each cluster is the same, which is r2=2.
[0065] After the central node 101 distributes input files to the users 103 in the distributed cluster 102, the input files distributed to the three users 103 in the first distribution are expressed as follows: The input files allocated to the three users 103 in the second distribution are expressed as: The input files allocated to the three users 103 in the third distribution are expressed as: as well as 1-9 represents 9 input files.
[0066] After the central node 101 assigns the output function to the user 103 in the distributed cluster 102, the output function assigned to each user 103 in the three distributed clusters 102 is expressed as: W1 (1) =1,W2 (1) =2,W3 (1) =3,W1 (2) =4,W2 (2) =5,W3 (2) =6,W1 (3) =7,W2 (3) =8 and W3 (3) =9, where 1-9 represents 9 output functions.
[0067] In the embodiment of the present invention, the user 103 in the distributed cluster 102 is used to perform mapping according to the allocated input file to obtain an initial intermediate value variable, perform intra-cluster communication according to the initial intermediate value variable, obtain a first intermediate value variable and send it to the central node 101.
[0068] Among them, multiple mapping functions can be used to map each user 103 in the distributed cluster 102 to obtain multiple intermediate value variables corresponding to each user 103; the mapping function and the output function are not necessarily in a one-to-one correspondence, and there may be multiple mapping functions corresponding to one output function.
[0069] Specifically, when the user 103 in the distributed cluster 102 executes the step of mapping according to the allocated input file to obtain the initial intermediate value variable, it includes: selecting the target distributed cluster 102 one by one from the distributed cluster 102, and selecting the target user 103 one by one from the target distributed cluster 102; obtaining the target input file allocated to the target user 103; using multiple mapping functions to map according to the target input file to obtain the corresponding initial intermediate value variable.
[0070] In an actual application scenario of the present invention, each user 103 in the distributed cluster 102 is mapped, and the initial intermediate value variable obtained is expressed as in, It is represented as the initial intermediate value variable being generated based on the nth input file and being applied to the user 103 corresponding to the kth output function in the cth distributed cluster 102 .
[0071] Furthermore, intra-cluster communication is divided into two parallel processes: direct transmission of useful intermediate values and transmission of auxiliary intermediate values.
[0072] Specifically, when the user 103 in the distributed cluster 102 executes the step of performing intra-cluster communication according to the initial intermediate value variable, the steps include: selecting a target distributed cluster 102 from the distributed cluster 102 one by one, and selecting a target user and an adjacent user from the target distributed cluster 102 one by one; obtaining the initial intermediate value variables corresponding to the target user and the adjacent user; the target user performs XOR encoding according to the corresponding initial intermediate value variables to obtain a corresponding first encoded message and sends it to the adjacent user; the adjacent user decodes the encoded message to obtain the intra-cluster communication intermediate value variable transmitted by the target user.
[0073] Furthermore, when the target user performs the step of performing XOR encoding according to the corresponding initial intermediate value variable to obtain the corresponding first encoded message and sending it to the adjacent user, it includes: determining a directly useful intermediate value and an auxiliary intermediate value from the corresponding initial intermediate value variable; performing XOR encoding according to the directly useful intermediate value and the auxiliary intermediate value to obtain the corresponding first encoded message.
[0074] In detail, the corresponding initial intermediate value variables include direct useful intermediate values and auxiliary intermediate values corresponding to the adjacent users.
[0075] In an actual application scenario of the present invention, taking the first user 103 in the first distributed cluster 102 as an example, the user 103 is taken as the target user, and the second user and the third user in the first distributed cluster 102 are adjacent users. The input file allocated to the target user is The input file obtained by neighboring users is The initial intermediate value variable obtained after mapping is wait;
[0076] The corresponding direct useful intermediate value can be determined by the input file corresponding to the first adjacent user (the second user in the first distributed cluster 102) as The auxiliary intermediate value is wait;
[0077] Through the input file corresponding to the second adjacent user (the third user in the first distributed cluster 102), its corresponding direct useful intermediate value can be determined as The auxiliary intermediate value is wait.
[0078] Specifically, the present invention can have two encoding methods to realize the encoding of the directly useful intermediate value and the auxiliary intermediate value:
[0079] The first encoding method integrates the initial intermediate value variables sent to adjacent users through an XOR operation and sends them to adjacent users. For example, the target user sends To the first adjacent user and the second adjacent user, to achieve direct transmission of useful intermediate values; the target user sends To the first adjacent user and the second adjacent user to achieve the transmission of auxiliary intermediate values.
[0080] The second encoding method is to decompose each initial intermediate value variable into two non-overlapping and equal-sized parts, encode them separately and send them to adjacent users. For example, Decompose into two non-overlapping equal parts Will Decompose into two non-overlapping equal parts Target users send and To the first neighboring user and the second neighboring user, the first neighboring user sends and The second neighbor sends and To realize the transmission of direct useful intermediate values; the transmission of auxiliary intermediate values is realized by the same steps as above.
[0081] Furthermore, after receiving the first coded message sent by the target user and the second adjacent user, the first adjacent user can decode the required directly useful intermediate value and and auxiliary intermediate values and After receiving the first coded message sent by the target user and the first adjacent user, the second adjacent user can decode the required direct useful intermediate value and and auxiliary intermediate values and
[0082] Furthermore, all users in the target distributed cluster 102 execute the step of obtaining a first intermediate value variable and sending it to the central node, including: determining a slice size for dividing the encoded message; slicing the first intermediate value variable according to the slice size to obtain multiple slice values; and performing XOR encoding according to the slice value to obtain a second encoded message.
[0083] The present invention slices the coded message sent by the user 103 in the distributed cluster 102 into uniform size, so that the central node 101 can decode the received uplink coded message easily; and the load of the uplink communication can be reduced by XOR encoding the slice value.
[0084] Taking the first distributed cluster 102 as an example, when the second coded message is sent to the central node 101 to achieve uplink communication, the first user 103 sends as well as To the central node 101; the second user 103 sends as well as To the central node 101; the third user 103 sends as well as To the central node 101.
[0085] In the embodiment of the present invention, the central node 101 is used to perform inter-cluster communication with the distributed cluster 102 according to the first intermediate value variable sent by the user 103 in the distributed cluster 102 .
[0086] Specifically, when the central node 101 executes the step of communicating between clusters with the distributed cluster 102 according to the first intermediate value variable sent by the user 103 in the distributed cluster 102, the step includes: obtaining the first intermediate value variable transmitted by all users 103 in each distributed cluster 102 as the intermediate value variable to be processed; performing linear encoding according to the intermediate value variable to be processed to obtain multiple groups of downlink encoded messages; and sending the downlink encoded message to the user 103 in the corresponding distributed cluster 102.
[0087] Among them, a linear combination can be performed according to the output function to be acted on in the intermediate value variables to be processed, so as to transmit the intermediate value variables corresponding to the output function to be acted on to the user 103 corresponding to the output function.
[0088] Taking the second user 103 in the first distributed cluster 102 as an example, the multiple groups of downlink coded messages obtained according to linear coding are expressed as follows:
[0089]
[0090] By sending the downlink coded message to the second user 103 in the first distributed cluster 102, the user 103 obtains the allocated intermediate value variables required for the output function calculation.
[0091] In the embodiment of the present invention, the user 103 in the distributed cluster 102 is used to calculate the second intermediate value variable obtained by inter-cluster communication and the initial intermediate value variable according to the allocated output function to obtain a distributed calculation result.
[0092] Specifically, the second intermediate value variable obtained by the inter-cluster communication according to the allocated output function is obtained by the following steps: the user 103 in the distributed cluster 102 decodes the downlink encoded message according to the initial intermediate value variable and the first intermediate value variable to obtain the second intermediate value variable.
[0093] Taking the second user 103 in the first distributed cluster 102 as an example, the user 103 obtains the intermediate value variable Then, the relevant information transmitted from the inter-cluster communication can be The required second intermediate variable is decoded from the two linear combinations and
[0094] In an embodiment of the present invention, when the user 103 in the distributed cluster 102 executes the step of calculating the second intermediate value variable obtained by inter-cluster communication and the initial intermediate value variable according to the allocated output function to obtain a distributed computing result, the step includes: determining the target intermediate value variable required by the user 103 in the distributed cluster 102 according to the second intermediate value variable, the first intermediate value variable and the initial intermediate value variable; substituting the target intermediate value variable into the output function to obtain the distributed computing result corresponding to the user 103 in the distributed cluster 102.
[0095] Each user 103 in the distributed cluster 102 is assigned a corresponding output function, and the target intermediate value variable substituted into the output function for calculation is: the intermediate value variable related to the output function obtained by each user 103 in each distributed cluster 102 based on all input files mapping.
[0096] The present invention takes into account the multicast opportunities in intra-rack communications, directly conducts intra-cluster communications through users 103 in the distributed cluster 102, and realizes inter-cluster communications between distributed clusters 102 through the central node 101, while realizing coded transmission of intra-cluster and inter-cluster communications, reducing the communication load in two stages, which can greatly reduce the communication load and reduce the computational complexity compared to traditional solutions.
[0097] The embodiment of the present application provides a nested distributed coding calculation method. The execution subject of the nested distributed coding calculation method includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the system provided by the embodiment of the present application. In other words, the nested distributed coding calculation method can be executed by software or hardware installed in a terminal device or a server device. Cluster, etc.
[0098] Reference Figure 2 FIG. 1 is a flow chart of a nested distributed coding calculation method provided by an embodiment of the present invention. In this embodiment, the nested distributed coding calculation method includes:
[0099] S1, the central node distributes input files and output functions to users in the distributed cluster;
[0100] S2. The users in the distributed cluster perform mapping according to the allocated input files to obtain initial intermediate value variables, perform intra-cluster communication according to the initial intermediate value variables, obtain first intermediate value variables and send them to the central node;
[0101] S3. The central node performs inter-cluster communication with the distributed cluster according to the first intermediate value variable sent by the user in the distributed cluster;
[0102] S4. The users in the distributed cluster calculate the second intermediate value variable obtained through inter-cluster communication and the initial intermediate value variable according to the allocated output function to obtain a distributed calculation result.
[0103] like Figure 3 , which is a schematic diagram of the structure of an electronic device for implementing a nested distributed coding computing system at a user or a central node in a distributed cluster provided by an embodiment of the present invention.
[0104] An embodiment of the present invention provides an electronic device for implementing a nested distributed coding computing system.
[0105] The electronic device 200 may include a processor 201, a memory 202, a communication bus 203 and a communication interface 204, and may also include a computer program stored in the memory 202 and executable on the processor 201, such as a program for the steps executed by a central node in a nested distributed coding computing method or a computer program for the steps executed by a user in a distributed cluster.
[0106] In some embodiments, the processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips.
[0107] In some embodiments, the memory 202 may be an internal storage unit of the electronic device, such as a mobile hard disk of the electronic device. The memory 202 may be used not only to store application software and various data installed in the electronic device, such as the code of the nested distributed coding computing program, but also to temporarily store data that has been output or is to be output.
[0108] The communication bus 203 can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 202 and at least one processor 201, etc.
[0109] The communication interface 204 is used for communication between the electronic device and other devices, and includes a network interface and a user interface.
[0110] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.
[0111] The nested distributed coding computing program stored in the memory 202 in the electronic device 200 is a combination of multiple instructions. When running in the processor 201, it can implement the functions performed by the central node in the above-mentioned nested distributed coding computing system or the functions performed by the users in the distributed cluster.
[0112] Specifically, the specific implementation system of the processor 201 for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.
[0113] Furthermore, if the modules / units integrated in the electronic device 200 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.
[0114] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor of an electronic device, it can implement the functions performed by the central node in the above-mentioned nested distributed coding computing system or the functions performed by users in a distributed cluster.
[0115] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and systems can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.
[0116] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0117] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.
[0118] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.
[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.
Claims
1. A nested distributed coding computing system, characterized in that: The system includes a central node and multiple distributed clusters, each distributed cluster includes multiple users: The central node is used to distribute input files and output functions to users in the distributed cluster; The user in the distributed cluster is used to perform mapping according to the allocated input file to obtain an initial intermediate value variable, perform intra-cluster communication according to the initial intermediate value variable, obtain a first intermediate value variable and send it to the central node; The central node is configured to perform inter-cluster communication with the distributed cluster according to the first intermediate value variable sent by a user in the distributed cluster; The users in the distributed cluster are used to calculate the second intermediate value variable obtained through inter-cluster communication and the initial intermediate value variable according to the allocated output function to obtain a distributed calculation result.
2. The nested distributed coding computing system according to claim 1, characterized in that: When the central node executes the step of allocating input files and output functions to users in the distributed cluster, the step includes: Evenly distribute the user assigned input file to each distributed cluster; Allocate the input files obtained from each distributed cluster to each corresponding user; Distributing the output function evenly to all distributed clusters; The output function obtained by the distributed cluster allocation is allocated to each corresponding user.
3. The nested distributed coding computing system according to claim 1, characterized in that: When the user in the distributed cluster performs mapping according to the allocated input file to obtain the initial intermediate value variable, the step includes: Selecting target distributed clusters one by one from the distributed clusters, and selecting target users one by one from the target distributed clusters; Obtaining a target input file allocated to the target user; Mapping is performed according to the target input file using a plurality of mapping functions to obtain corresponding initial intermediate value variables.
4. The nested distributed coding computing system according to claim 1, characterized in that: When the user in the distributed cluster performs the step of performing intra-cluster communication according to the initial intermediate value variable, the step includes: Selecting target distributed clusters one by one from the distributed clusters, and selecting target users and adjacent users one by one from the target distributed clusters; Obtaining initial intermediate value variables corresponding to the target user and the adjacent users; The target user performs XOR encoding according to the corresponding initial intermediate value variable to obtain a corresponding first encoded message and sends it to the adjacent user; The neighboring user decodes the encoded message to obtain the intra-cluster communication intermediate value variable transmitted by the target user.
5. The nested distributed coding computing system according to claim 4, characterized in that: When the target user performs the step of performing XOR encoding according to the corresponding initial intermediate value variable to obtain the corresponding first encoded message and sending it to the adjacent user, it includes: Determining directly useful intermediate values and auxiliary intermediate values from the corresponding initial intermediate value variables; XOR encoding is performed on the direct useful intermediate value and the auxiliary intermediate value to obtain a corresponding first encoded message.
6. The nested distributed coding computing system according to claim 1, characterized in that: When all users in the target distributed cluster execute the step of obtaining the first intermediate value variable and sending it to the central node, the step includes: Determining a slice size for dividing the encoded message; Slicing the first intermediate value variable according to the slice size to obtain multiple slice values; XOR encoding is performed according to the slice value to obtain a second encoded message.
7. The nested distributed coding computing system according to claim 1, characterized in that: When the central node executes the step of performing inter-cluster communication with the distributed cluster according to the first intermediate value variable sent by the user in the distributed cluster, the step includes: Obtaining the first intermediate value variable transmitted by all users in each distributed cluster as the intermediate value variable to be processed; Performing linear coding according to the intermediate value variables to be processed to obtain multiple groups of downlink coded messages; The downlink encoded message is sent to users in the corresponding distributed cluster.
8. The nested distributed coding computing system according to claim 7, characterized in that: The second intermediate value variable obtained by inter-cluster communication according to the allocated output function is obtained by the following steps: the user in the distributed cluster decodes the downlink encoded message according to the initial intermediate value variable and the first intermediate value variable to obtain the second intermediate value variable.
9. The nested distributed coding computing system according to claim 1, characterized in that: When the user in the distributed cluster performs the step of calculating the second intermediate value variable obtained by inter-cluster communication and the initial intermediate value variable according to the output function obtained by the distribution to obtain the distributed computing result, the step includes: Determine a target intermediate value variable required by a user in a distributed cluster according to the second intermediate value variable, the first intermediate value variable, and the initial intermediate value variable; Substitute the target intermediate value variable into the output function to obtain the distributed computing result corresponding to the user in the distributed cluster.
10. A nested distributed coding computing method, characterized in that: The method comprises: The central node distributes input files and output functions to users in the distributed cluster; The users in the distributed cluster perform mapping according to the allocated input files to obtain initial intermediate value variables, perform intra-cluster communication according to the initial intermediate value variables, obtain first intermediate value variables and send them to the central node; The central node performs inter-cluster communication with the distributed cluster according to the first intermediate value variable sent by the user in the distributed cluster; The users in the distributed cluster calculate the second intermediate value variable obtained through inter-cluster communication and the initial intermediate value variable according to the allocated output function to obtain a distributed calculation result.
Citation Information
Patent Citations
Coding MapReduce method for median length isomerism
CN111490795A
Coding MapReduce-oriented Shuffle performance optimization method and system under Rack architecture
CN114844781A
Discrete task-oriented joint source channel coding framework
CN116599628A
Mapping epigenetic surprisal data througth hadoop type distributed file systems
US20140236977A1
System and method for distributed computation using heterogeneous computing nodes
WO2014207759A2
