A nested distributed encoding computing system and method

Abstract

The present application relates to the technical field of distributed cluster processing, and discloses a nested distributed encoding computing system and method, the system comprising a central node and a plurality of distributed clusters: the central node allocates input files and output functions to users in the distributed clusters; the users in the distributed clusters map according to the input files obtained by allocation to obtain initial intermediate value variables, and communicate within the clusters according to the initial intermediate value variables to obtain first intermediate value variables; the central node communicates with the distributed clusters according to the first intermediate value variables sent by the users in the distributed clusters; and the users in the distributed clusters calculate distributed computing results according to the second intermediate value variables obtained by inter-cluster communication, the first intermediate value variables and the initial intermediate value variables according to the output functions obtained by allocation. The present application can simultaneously realize encoding transmission of intra-cluster and inter-cluster communication, reduce the communication load of two stages, and greatly reduce the communication load.

Description

Technical Field

[0001] The present invention relates to the field of distributed cluster processing technology, and in particular to a nested distributed coding computing system and method. Background Art

[0002] Distributed computing systems are on the rise in modern communication systems. Frameworks such as MapReduce are widely used to parallelize large computational tasks. MapReduce is a programming model for parallel operations on large datasets. MapReduce tasks consist of three phases: map, shuffle, and reduce. In the map phase, distributed users map assigned input files to intermediate variables required to compute the output function. In the shuffle phase, these intermediate variables are exchanged among all users, allowing each user to obtain the intermediate variables required for the reduce phase. In the reduce phase, each user aggregates all received intermediate variables and executes the reduction function on them to generate the output. However, traditional MapReduce solutions incur high communication costs associated with data transmission during the shuffle phase.

[0003] Existing technologies use coded distributed computing to alleviate the distributed communication bottleneck. By introducing additional computation in the mapping phase and utilizing coded transmission in the shuffling phase, coded distributed computing can significantly reduce the communication cost in the shuffling phase. To improve the scalability, robustness, and manageability of the system, practical large-scale distributed computing systems usually consist of multiple clusters, each containing multiple computing nodes. However, most current work focuses on the single-cluster case, where all worker threads are clustered in the same cluster. Cross-cluster coded computing often focuses on reducing cross-cluster traffic while ignoring multicast opportunities in intra-rack communication, which leads to unnecessary communication load.

[0004] In summary, how to reduce the communication load of distributed computing systems for coding calculations has become an urgent problem to be solved. Summary of the Invention

[0005] The present invention provides a nested distributed coding computing system, method, electronic device and computer-readable storage medium, the main purpose of which is to solve the problem of how to reduce the communication load of the distributed computing system for coding computing, which has become an urgent problem to be solved.

[0006] To achieve the above objectives, the present invention provides a nested distributed coding computing system, which includes a central node and multiple distributed clusters, each of which includes multiple users:

[0007] The central node is used to distribute input files and output functions to users in the distributed cluster;

[0008] The users in the distributed cluster are configured to perform mapping according to the allocated input files to obtain initial intermediate value variables, perform intra-cluster communication according to the initial intermediate value variables, obtain first intermediate value variables and send them to the central node;

[0009] The central node is configured to perform inter-cluster communication with the distributed cluster according to the first intermediate value variable sent by a user in the distributed cluster;

[0010] The users in the distributed cluster are configured to calculate the second intermediate value variable obtained through inter-cluster communication and the initial intermediate value variable according to the allocated output function to obtain a distributed calculation result.

[0011] In one embodiment of the present invention, when the central node executes the step of allocating input files and output functions to users in the distributed cluster, the steps include:

[0012] Evenly distribute the user-assigned input file to each distributed cluster;

[0013] Allocate the input files obtained from each distributed cluster to each corresponding user;

[0014] Distributing the output function evenly to all distributed clusters;

[0015] The output function obtained by the distributed cluster allocation is allocated to each corresponding user.

[0016] In one embodiment of the present invention, when a user in the distributed cluster performs mapping according to the allocated input file to obtain an initial intermediate value variable, the step includes:

[0017] Selecting target distributed clusters one by one from the distributed clusters, and selecting target users one by one from the target distributed clusters;

[0018] Obtaining a target input file allocated to the target user;

[0019] Mapping is performed according to the target input file using multiple mapping functions to obtain corresponding initial intermediate value variables.

[0020] In one embodiment of the present invention, when a user in the distributed cluster performs intra-cluster communication according to the initial intermediate value variable to obtain a first intermediate value variable and sends the first intermediate value variable to the central node, the step includes:

[0021] Selecting target distributed clusters one by one from the distributed clusters, and selecting target users and adjacent users one by one from the target distributed clusters;

[0022] Obtaining initial intermediate value variables corresponding to the target user and the adjacent users;

[0023] The target user performs XOR encoding according to the corresponding initial intermediate value variable to obtain a corresponding first encoded message and sends it to the adjacent user;

[0024] The neighboring user decodes the encoded message to obtain the intra-cluster communication intermediate value variable transmitted by the target user.

[0025] In one embodiment of the present invention, when the target user performs the step of performing XOR encoding according to the corresponding initial intermediate value variable to obtain the corresponding first encoded message and sends it to the neighboring user, the step includes:

[0026] Determining a directly useful intermediate value and an auxiliary intermediate value from the corresponding initial intermediate value variables;

[0027] XOR encoding is performed on the direct useful intermediate value and the auxiliary intermediate value to obtain a corresponding first encoded message.

[0028] In one embodiment of the present invention, when all users in the target distributed cluster execute the step of obtaining a first intermediate value variable and sending it to the central node, the step includes:

[0029] Determine the slice size for dividing the encoded message;

[0030] Slicing the first intermediate value variable according to the slice size to obtain multiple slice values;

[0031] XOR encoding is performed according to the slice value to obtain a second encoded message.

[0032] In one embodiment of the present invention, when the central node performs the step of performing inter-cluster communication with the distributed cluster according to the first intermediate value variable sent by a user in the distributed cluster, the step includes:

[0033] Obtaining the first intermediate value variable transmitted by all users in each distributed cluster as the intermediate value variable to be processed;

[0034] Performing linear coding according to the intermediate value variables to be processed to obtain multiple groups of downlink coded messages;

[0035] The downlink encoded message is sent to users in the corresponding distributed cluster.

[0036] In one embodiment of the present invention, the second intermediate value variable obtained by inter-cluster communication according to the output function obtained by distribution is obtained by the following steps: the user in the distributed cluster decodes the downlink encoded message according to the initial intermediate value variable and the first intermediate value variable to obtain the second intermediate value variable.

[0037] In one embodiment of the present invention, when a user in the distributed cluster performs a calculation based on the assigned output function on the second intermediate value variable obtained through inter-cluster communication and the initial intermediate value variable to obtain a distributed computing result, the step includes:

[0038] Determine a target intermediate value variable required by a user in a distributed cluster according to the second intermediate value variable, the first intermediate value variable, and the initial intermediate value variable;

[0039] Substitute the target intermediate value variable into the output function to obtain the distributed computing result corresponding to the user in the distributed cluster.

[0040] In order to solve the above problems, the present invention further provides a nested distributed coding calculation method, which includes:

[0041] The central node distributes input files and output functions to users in the distributed cluster;

[0042] The users in the distributed cluster perform mapping according to the allocated input files to obtain initial intermediate value variables, perform intra-cluster communication according to the initial intermediate value variables, obtain first intermediate value variables and send them to the central node;

[0043] The central node performs inter-cluster communication with the distributed cluster according to the first intermediate value variable sent by the user in the distributed cluster;

[0044] The users in the distributed cluster calculate the second intermediate value variable obtained through inter-cluster communication and the initial intermediate value variable according to the allocated output function to obtain a distributed calculation result.

[0045] In order to solve the above problem, the present invention further provides an electronic device, comprising:

[0046] at least one processor; and,

[0047] a memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the functions of the nested distributed coding computing system described above.

[0049] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the functions of the above-mentioned nested distributed coding computing system.

[0050] The present invention takes into account the multicast opportunities in intra-rack communication, directly conducts intra-cluster communication through users in the distributed cluster, and realizes inter-cluster communication between distributed clusters through the central node, while realizing coded transmission of intra-cluster and inter-cluster communication, reducing the communication load in two stages, which can greatly reduce the communication load and reduce the computational complexity compared with traditional solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A system architecture diagram of a nested distributed coding computing system provided by one embodiment of the present invention;

[0052] Figure 2 A schematic diagram of a flow chart of a nested distributed coding calculation method provided by one embodiment of the present invention;

[0053] Figure 3 A schematic structural diagram of an electronic device for implementing the nested distributed coding computing system at a user or central node in a distributed cluster provided by an embodiment of the present invention.

[0054] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0056] like Figure 1 , which is a system architecture diagram of a nested distributed coding computing system 100 provided in one embodiment of the present invention.

[0057] The nested distributed coding computing system proposed in the present invention includes a central node 101 and multiple distributed clusters 102 , and each distributed cluster 102 includes multiple users 103 .

[0058] In this embodiment of the present invention, the central node 101 is used to distribute input files and output functions to users 103 in the distributed cluster 102 . The distributed cluster 102 also includes a switch 104 , through which users 103 in the distributed cluster 102 communicate with the central node 101 .

[0059] Among them, the nested distributed coding computing system can be a multi-cluster MapReduce computing system, the central node 101 is a central server, connected to multiple distributed clusters 102 to realize cross-cluster communication between users 103 of the distributed clusters 102; the distributed cluster 102 has any number of users 103 and computing loads, each cluster contains a group of users 103 who want to jointly calculate a MapReduce task, and each user 103 can directly conduct "intra-cluster communication" with users 103 in the same cluster.

[0060] The present invention can significantly reduce the communication load and achieve optimal cross-cluster communication load. In addition, compared with traditional coding calculation solutions, it can significantly reduce the computational complexity.

[0061] In an embodiment of the present invention, when the central node 101 executes the step of allocating input files and output functions to the user 103 in the distributed cluster 102, the steps include: evenly distributing the input files allocated to the user 103 to each distributed cluster 102; allocating the input files allocated to each distributed cluster 102 to each corresponding user 103; evenly distributing the output functions to all distributed clusters 102; and allocating the output functions allocated to the distributed clusters to each corresponding user 103.

[0062] Among them, the input file can be first divided into multiple non-overlapping sub-files, and then the multiple sub-files are evenly distributed to each distributed cluster 102, so that each distributed cluster 102 can obtain the same number of non-overlapping input files; then, based on the input files obtained by each distributed cluster 102, each user 103 in the cluster is evenly distributed, so that all users 103 in a cluster can obtain the same number of input files.

[0063] Likewise, after the output functions are evenly distributed to each user 103 in all distributed clusters 102 , each user 103 can obtain the same number of output functions.

[0064] In an actual application scenario of the present invention, the nested distributed coding computing system includes C=3 distributed clusters 102, each distributed cluster 102 has K=3 users 103, N=9 input files, Q=9 output functions, and the computing load of the distributed cluster 102 is r1=2. The computing load in each cluster is the same, which is r2=2.

[0065] After the central node 101 distributes input files to the users 103 in the distributed cluster 102, the input files distributed to the three users 103 in the first distribution are represented as follows: The input files allocated to the three users 103 in the second distribution are expressed as: The input files allocated to the three users 103 in the third distribution are expressed as: as well as 1-9 represents 9 input files.

[0066] After the central node 101 assigns output functions to the users 103 in the distributed clusters 102, the output functions assigned to each user 103 in the three distributed clusters 102 are expressed as: W1 (1) =1,W2 (1) =2,W3 (1) =3,W1 (2) =4,W2 (2) =5,W3 (2) =6,W1 (3) =7,W2 (3) =8 and W3 (3) =9, where 1-9 represents 9 output functions.

[0067] In an embodiment of the present invention, the user 103 in the distributed cluster 102 is used to perform mapping according to the allocated input file to obtain initial intermediate value variables, perform intra-cluster communication according to the initial intermediate value variables, obtain the first intermediate value variable and send it to the central node 101.

[0068] Among them, multiple mapping functions can be used to map each user 103 in the distributed cluster 102 to obtain multiple intermediate value variables corresponding to each user 103; the mapping function and the output function are not necessarily in a one-to-one correspondence, and there may be multiple mapping functions corresponding to one output function.

[0069] Specifically, when the user 103 in the distributed cluster 102 executes the step of mapping according to the allocated input file to obtain the initial intermediate value variable, it includes: selecting the target distributed cluster 102 one by one from the distributed cluster 102, and selecting the target user 103 one by one from the target distributed cluster 102; obtaining the target input file allocated to the target user 103; using multiple mapping functions to map according to the target input file to obtain the corresponding initial intermediate value variable.

[0070] In a practical application scenario of the present invention, each user 103 in the distributed cluster 102 is mapped, and the initial intermediate value variable obtained is expressed as in, It is represented as the initial intermediate value variable being generated based on the nth input file and will act on the user 103 corresponding to the kth output function in the cth distributed cluster 102 .

[0071] Furthermore, intra-cluster communication is divided into two parallel processes: direct transmission of useful intermediate values ​​and transmission of auxiliary intermediate values.

[0072] Specifically, when the user 103 in the distributed cluster 102 executes the step of performing intra-cluster communication according to the initial intermediate value variable, the steps include: selecting a target distributed cluster 102 from the distributed cluster 102 one by one, and selecting a target user and an adjacent user from the target distributed cluster 102 one by one; obtaining the initial intermediate value variables corresponding to the target user and the adjacent user; the target user performs XOR encoding according to the corresponding initial intermediate value variables to obtain a corresponding first encoded message and sends it to the adjacent user; the adjacent user decodes the encoded message to obtain the intra-cluster communication intermediate value variable transmitted by the target user.

[0073] Furthermore, when the target user performs the step of performing XOR encoding according to the corresponding initial intermediate value variable to obtain the corresponding first encoded message and sends it to the adjacent user, it includes: determining a directly useful intermediate value and an auxiliary intermediate value from the corresponding initial intermediate value variable; performing XOR encoding according to the directly useful intermediate value and the auxiliary intermediate value to obtain the corresponding first encoded message.

[0074] In detail, the corresponding initial intermediate value variables include direct useful intermediate values ​​and auxiliary intermediate values ​​corresponding to the adjacent users.

[0075] In an actual application scenario of the present invention, taking the first user 103 in the first distributed cluster 102 as an example, the user 103 is used as the target user, and the second and third users in the first distributed cluster 102 are adjacent users. The input file allocated to the target user is The input file obtained by neighboring users is The initial intermediate value variable obtained after mapping is wait;

[0076] The corresponding direct useful intermediate value can be determined by the input file corresponding to the first adjacent user (the second user in the first distributed cluster 102) as The auxiliary intermediate value is wait;

[0077] The corresponding direct useful intermediate value can be determined by the input file corresponding to the second adjacent user (the third user in the first distributed cluster 102) as The auxiliary intermediate value is etc.

[0078] Specifically, the present application can exist two kinds of encoding mode to realize the encoding of the direct useful intermediate value and the auxiliary intermediate value:

[0079] The first encoding mode, by means of exclusive or operation to send to the initial intermediate value variable of adjacent user and send to adjacent user, for example, the target user sends To the first adjacent user and the second adjacent user, in order to realize the transmission of direct useful intermediate value; the target user sends To the first adjacent user and the second adjacent user, in order to realize the transmission of auxiliary intermediate value.

[0080] The second encoding mode, by means of decomposing each initial intermediate value variable into two mutually non overlapping and equal size parts, respectively, encoding and sending to adjacent user, for example, the target user sends Decomposed into two mutually non overlapping and equal size parts The target user sends Decomposed into two mutually non overlapping and equal size parts The target user sends And To the first adjacent user and the second adjacent user, the first adjacent user sends And The second adjacent user sends And In order to realize the transmission of direct useful intermediate value; the transmission of auxiliary intermediate value is realized by the above steps.

[0081] Further, the first adjacent user can decode the required direct useful intermediate value And And auxiliary intermediate value And After receiving the first encoding message sent by the target user and the second adjacent user, the second adjacent user can decode the required direct useful intermediate value And And auxiliary intermediate value And

[0082] Further, all users in the target distributed cluster 102 in the step of obtaining the first intermediate value variable and sending to the central node, comprising: determining the slice size of the partition encoding message; according to the slice size, the first intermediate value variable is sliced to obtain a plurality of slice values; according to the slice value, the exclusive or encoding is obtained.

[0083] The application makes the central node 101 decode conveniently after receiving the uplink coded message by slicing the coded message sent by the user 103 in the distributed cluster 102 into uniform size; and the load of uplink communication is reduced by XOR coding the sliced value.

[0084] Taking the first distributed cluster 102 as an example, when the second coded message is sent to the central node 101 to realize the uplink communication, the first user 103 sends and to the central node 101; the second user 103 sends and to the central node 101; and the third user 103 sends and to the central node 101.

[0085] In the embodiment of the application, the central node 101 is configured to perform inter-cluster communication with the distributed cluster 102 according to the first intermediate value variable sent by the user 103 in the distributed cluster 102.

[0086] Specifically, when performing the step of performing inter-cluster communication with the distributed cluster 102 according to the first intermediate value variable sent by the user 103 in the distributed cluster 102, the central node 101 comprises: obtaining the first intermediate value variable transmitted by all users 103 in each distributed cluster 102 as a to-be-processed intermediate value variable; performing linear coding according to the to-be-processed intermediate value variable to obtain a plurality of groups of downlink coded messages; and sending the downlink coded messages to the users 103 in the corresponding distributed cluster 102.

[0087] The to-be-processed intermediate value variable can be linearly combined according to the output function to be acted on, so as to transmit the intermediate value variable corresponding to the output function to be acted on to the user 103 corresponding to the output function.

[0088] Taking the second user 103 in the first distributed cluster 102 as an example, the plurality of groups of downlink coded messages obtained according to the linear coding are expressed as:

[0089]

[0090] By sending the downlink coded message to the second user 103 in the first distributed cluster 102, the user 103 further obtains the intermediate value variable required for the output function calculation.

[0091] In the embodiment of the present invention, the user 103 in the distributed cluster 102 is configured to calculate the second intermediate value variable obtained through inter-cluster communication and the initial intermediate value variable according to the allocated output function to obtain a distributed calculation result.

[0092] Specifically, the second intermediate value variable obtained by the inter-cluster communication according to the output function obtained by the distribution is obtained by the following steps: the user 103 in the distributed cluster 102 decodes the downlink encoded message according to the initial intermediate value variable and the first intermediate value variable to obtain the second intermediate value variable.

[0093] Taking the second user 103 in the first distributed cluster 102 as an example, the user 103 obtains the intermediate value variable Then, the relevant information can be transmitted from the inter-cluster communication The second intermediate variable required is decoded from the two linear combinations and

[0094] In an embodiment of the present invention, when the user 103 in the distributed cluster 102 executes the step of calculating the second intermediate value variable obtained by inter-cluster communication and the initial intermediate value variable according to the allocated output function to obtain a distributed computing result, the step includes: determining the target intermediate value variable required by the user 103 in the distributed cluster 102 according to the second intermediate value variable, the first intermediate value variable and the initial intermediate value variable; substituting the target intermediate value variable into the output function to obtain the distributed computing result corresponding to the user 103 in the distributed cluster 102.

[0095] Each user 103 in the distributed cluster 102 is assigned a corresponding output function, and the target intermediate value variable substituted into the output function for calculation is: the intermediate value variable related to the output function obtained by each user 103 in each distributed cluster 102 based on all input files.

[0096] The present invention takes into account the multicast opportunities in intra-rack communication, directly conducts intra-cluster communication through users 103 in the distributed cluster 102, and realizes inter-cluster communication between distributed clusters 102 through the central node 101, while realizing coded transmission of intra-cluster and inter-cluster communication, reducing the communication load of the two stages, which can greatly reduce the communication load and reduce the computational complexity compared with traditional solutions.

[0097] The present invention provides a nested distributed coding computing method. The execution subject of the nested distributed coding computing method includes, but is not limited to, at least one electronic device such as a server or a terminal that can be configured to execute the system provided by the present invention. In other words, the nested distributed coding computing method can be executed by software or hardware installed on a terminal device or a server device, a cluster, etc.

[0098] Reference Figure 2 FIG. 1 is a flow chart of a nested distributed coding calculation method according to an embodiment of the present invention. In this embodiment, the nested distributed coding calculation method includes:

[0099] S1, the central node distributes input files and output functions to users in the distributed cluster;

[0100] S2. Users in the distributed cluster perform mapping according to the allocated input files to obtain initial intermediate value variables, perform intra-cluster communication based on the initial intermediate value variables, obtain first intermediate value variables, and send them to the central node;

[0101] S3. The central node performs inter-cluster communication with the distributed cluster according to the first intermediate value variable sent by the user in the distributed cluster;

[0102] S4. The users in the distributed cluster calculate the second intermediate value variable obtained through inter-cluster communication and the initial intermediate value variable according to the allocated output function to obtain a distributed calculation result.

[0103] like Figure 3 , which is a structural diagram of an electronic device for implementing a nested distributed coding computing system at a user or central node in a distributed cluster provided by an embodiment of the present invention.

[0104] An embodiment of the present invention provides an electronic device for implementing a nested distributed coding computing system.

[0105] The electronic device 200 may include a processor 201, a memory 202, a communication bus 203 and a communication interface 204, and may also include a computer program stored in the memory 202 and executable on the processor 201, such as a program for the steps executed by a central node in a nested distributed coding computing method or a computer program for the steps executed by a user in a distributed cluster.

[0106] In some embodiments, the processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of integrated circuits packaged with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips.

[0107] In some embodiments, the memory 202 may be an internal storage unit of the electronic device, such as a removable hard drive of the electronic device. The memory 202 may be used not only to store application software installed on the electronic device and various data, such as code of a nested distributed coding computing program, but also to temporarily store data that has been output or is about to be output.

[0108] The communication bus 203 can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to implement connection and communication between the memory 202 and at least one processor 201, etc.

[0109] The communication interface 204 is used for communication between the electronic device and other devices, and includes a network interface and a user interface.

[0110] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0111] The nested distributed coding computing program stored in the memory 202 in the electronic device 200 is a combination of multiple instructions. When running in the processor 201, it can realize the functions performed by the central node in the above-mentioned nested distributed coding computing system or the functions performed by users in the distributed cluster.

[0112] Specifically, the specific implementation system of the processor 201 for the above instructions can refer to the description of the relevant steps in the corresponding embodiment in the accompanying drawings, which will not be repeated here.

[0113] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.

[0114] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor of an electronic device, it can implement the functions performed by the central node in the above-mentioned nested distributed coding computing system or the functions performed by users in a distributed cluster.

[0115] In the several embodiments provided herein, it should be understood that the disclosed devices, apparatuses, and systems may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical functional division, and actual implementation may employ other division methods.

[0116] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0117] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0118] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. Terms such as "first" and "second" are used to indicate names and do not imply any particular order.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A nested distributed coding computing system, characterized in that: The system includes a central node and multiple distributed clusters, each of which includes multiple users: The central node is used to distribute input files and output functions to users in the distributed cluster; The users in the distributed cluster are used to perform mapping according to the allocated input files to obtain initial intermediate value variables; The users in the distributed cluster are further configured to perform intra-cluster communication according to the initial intermediate value variable, obtain a first intermediate value variable, and send the first intermediate value variable to the central node, and the steps include: Selecting target distributed clusters one by one from the distributed clusters, and selecting target users and adjacent users one by one from the target distributed clusters; Obtaining initial intermediate value variables corresponding to the target user and the adjacent user; performing XOR encoding on the target user according to the corresponding initial intermediate value variables to obtain a corresponding first encoded message and sending the message to the adjacent user; The neighboring user decodes the encoded message to obtain the intra-cluster communication intermediate value variable transmitted by the target user; The central node is configured to perform inter-cluster communication with the distributed cluster according to the first intermediate value variable sent by a user in the distributed cluster, and the steps include: Obtaining the first intermediate value variable transmitted by all users in each distributed cluster as the intermediate value variable to be processed; Performing linear coding according to the intermediate value variables to be processed to obtain multiple groups of downlink coded messages; Sending the downlink encoded message to users in the corresponding distributed cluster; The users in the distributed cluster are further configured to calculate the second intermediate value variable obtained through inter-cluster communication and the initial intermediate value variable according to the allocated output function to obtain a distributed calculation result, and the steps include: Determine a target intermediate value variable required by a user in a distributed cluster according to the second intermediate value variable, the first intermediate value variable, and the initial intermediate value variable; Substituting the target intermediate value variable into the output function to obtain the distributed computing result corresponding to the user in the distributed cluster; The second intermediate variable is obtained by the following steps: The users in the distributed cluster decode the downlink coded message according to the initial intermediate value variable and the first intermediate value variable to obtain a second intermediate value variable.

2. The nested distributed coding computing system according to claim 1, characterized in that: When the central node executes the steps of allocating input files and output functions to users in the distributed cluster, the steps include: Evenly distribute the user-assigned input file to each distributed cluster; Allocate the input files obtained from each distributed cluster to each corresponding user; Distributing the output function evenly to all distributed clusters; The output function obtained by the distributed cluster allocation is allocated to each corresponding user.

3. The nested distributed coding computing system according to claim 1, characterized in that: When a user in the distributed cluster performs mapping according to the allocated input file to obtain an initial intermediate value variable, the step includes: Selecting target distributed clusters one by one from the distributed clusters, and selecting target users one by one from the target distributed clusters; Obtaining a target input file allocated to the target user; Mapping is performed according to the target input file using multiple mapping functions to obtain corresponding initial intermediate value variables.

4. The nested distributed coding computing system according to claim 1, characterized in that: When the target user performs the step of performing XOR encoding according to the corresponding initial intermediate value variable to obtain the corresponding first encoded message and sends it to the neighboring user, the step includes: Determining a directly useful intermediate value and an auxiliary intermediate value from the corresponding initial intermediate value variables; XOR encoding is performed on the direct useful intermediate value and the auxiliary intermediate value to obtain a corresponding first encoded message.

5. The nested distributed coding computing system according to claim 1, characterized in that: When all users in the target distributed cluster execute the step of obtaining a first intermediate value variable and sending it to the central node, the step includes: Determine the slice size for dividing the encoded message; Slicing the first intermediate value variable according to the slice size to obtain multiple slice values; XOR encoding is performed according to the slice value to obtain a second encoded message.

6. A nested distributed coding calculation method, characterized in that: The method comprises: The central node distributes input files and output functions to users in the distributed cluster; The users in the distributed cluster perform mapping according to the allocated input files to obtain initial intermediate value variables, perform intra-cluster communication according to the initial intermediate value variables, obtain first intermediate value variables and send them to the central node, and the steps include: Selecting target distributed clusters one by one from the distributed clusters, and selecting target users and adjacent users one by one from the target distributed clusters; Obtaining initial intermediate value variables corresponding to the target user and the adjacent user; performing XOR encoding on the target user according to the corresponding initial intermediate value variables to obtain a corresponding first encoded message and sending the message to the adjacent user; The neighboring user decodes the encoded message to obtain the intra-cluster communication intermediate value variable transmitted by the target user; The central node performs inter-cluster communication with the distributed cluster according to the first intermediate value variable sent by the user in the distributed cluster, and the steps include: Obtaining the first intermediate value variable transmitted by all users in each distributed cluster as the intermediate value variable to be processed; Performing linear coding according to the intermediate value variables to be processed to obtain multiple groups of downlink coded messages; Sending the downlink encoded message to users in the corresponding distributed cluster; The users in the distributed cluster calculate the second intermediate value variable obtained by inter-cluster communication and the initial intermediate value variable according to the allocated output function to obtain a distributed calculation result, and the steps include: Determine a target intermediate value variable required by a user in a distributed cluster according to the second intermediate value variable, the first intermediate value variable, and the initial intermediate value variable; Substituting the target intermediate value variable into the output function to obtain the distributed computing result corresponding to the user in the distributed cluster; The second intermediate variable is obtained by the following steps: The users in the distributed cluster decode the downlink coded message according to the initial intermediate value variable and the first intermediate value variable to obtain a second intermediate value variable.

Images