A large strip coding computing task allocation method for coding distributed computing

Abstract

The application belongs to the field of computer distributed computing, and specifically discloses a large-strip coding computing task allocation method for coding distributed computing, which comprises the following steps: based on the computing task allocation parameters adopted in the current execution of distributed coding, a plurality of original computing task groups are allocated to each coding node to form sub-computing tasks of each coding node; and based on the coding time length fed back by each coding node, the computing task allocation parameters adopted in the next execution of distributed coding are determined in a load balancing manner. In the process of each execution of distributed coding, the coding time length of each coding node is obtained, the coding time length can reflect the computing power of the node, and the computing task allocation parameters adopted in the next execution of distributed coding can be determined in a load balancing manner based on the coding time length fed back by each coding node. With the increase of the number of times of executing distributed coding, the plurality of coding nodes can tend to a more balanced state, and the coding efficiency can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer distributed computing, and more particularly, relates to a large strip coding computing task allocation method for coding distributed computing. BACKGROUND

[0002] With the rapid development of big data analysis applications in the information age and artificial intelligence, massive data and large-scale computing tasks are generated, and distributed processing computing tasks can effectively maximize resource utilization and reduce processing time. The distributed computing system with ten thousand cores in parallel will inevitably have a high failure frequency. In order to ensure reliability, a certain fault-tolerant mechanism is needed. Coding distributed computing applies coding theory to distributed computing systems, uses the redundant computing results generated by the coding mechanism to calculate the final result, and also does not cause large storage and computing resource overhead. In the coding theory, Reed-Solomon (RS) code is a very classic MDS property error correction code, which is widely used in various mainstream distributed systems. Specifically, in the coding distributed computing system, the user's computing task is divided into k original computing task groups. When encoding by RS code, the k original computing task groups are encoded into n groups (which include k original computing task groups and m encoding check groups, k+m=n, generally k is greater than or equal to m), and the n computing task groups are distributed to different user task computing nodes for calculation. There are a total of n user task computing nodes (one user task computing node is assigned to one computing task group), and when k user task computing nodes have completed the calculation, the final calculation result can be constructed (that is, any m user task computing node failure can be tolerated).

[0003] The k original computing task groups and the m encoding check groups form a strip, and in the case of a large k (for example, k is greater than 100), the strip is a large strip. As a new coding technology proposed in recent years, the large strip refers to a strip with large k and n. Research has found that the coding throughput (the number of strips that can be output per unit time) decreases as the value of k increases, which means that the large strip coding technology may generate a large coding overhead and become a bottleneck of the coding distributed computing system. For large strip coding, how to improve the coding efficiency is a technical problem to be solved in the field. SUMMARY

[0004] In view of the defects of the prior art, the purpose of the present application is to improve the coding efficiency for large strip coding.

[0005] To achieve the above purpose, in a first aspect, the present application provides a large strip coding computing task allocation method for coding distributed computing, comprising:

[0006] According to the computing task distribution parameter used in the current distributed encoding, the multiple original computing task groups are distributed to the encoding nodes to form sub-computing tasks of the encoding nodes;

[0007] According to the encoding time fed back by the encoding nodes, the computing task distribution parameter used in the next distributed encoding is determined in a load balancing manner;

[0008] The multiple original computing task groups are obtained by splitting a matrix multiplication computing task, the sub-computing tasks are used to instruct the encoding nodes to perform encoding on the distributed original computing task groups according to the encoding mode of the RS code to obtain an encoding check group, the computing task distribution parameter is used to represent the quantity ratio between the original computing task groups distributed to different encoding nodes, and the original computing task groups and the encoding check group form a strip corresponding to the matrix multiplication computing task.

[0009] In a possible implementation, the computing task distribution parameter used in the next distributed encoding is determined in a load balancing manner according to the encoding time fed back by the encoding nodes, and includes:

[0010] According to the encoding time fed back by the encoding nodes, the maximum encoding time of the current distributed encoding is determined;

[0011] In the case that the current distributed encoding is the first distributed encoding, the computing task distribution parameter used in the next distributed encoding is determined according to the encoding time fed back by the encoding nodes and the computing task distribution parameter used in the current distributed encoding;

[0012] In the case that the maximum encoding time of the current distributed encoding is less than or equal to the maximum encoding time of the last distributed encoding, the computing task distribution parameter used in the next distributed encoding is determined according to the encoding time fed back by the encoding nodes and the computing task distribution parameter used in the current distributed encoding;

[0013] In the case that the maximum encoding time of the current distributed encoding is greater than the maximum encoding time of the last distributed encoding, if it is determined that the condition of updating the computing task distribution parameter is met, the computing task distribution parameter used in the next distributed encoding is determined according to the encoding time fed back by the encoding nodes and the computing task distribution parameter used in the current distributed encoding, and if it is determined that the condition of updating the computing task distribution parameter is not met, the computing task distribution parameter used in the last distributed encoding is determined as the computing task distribution parameter used in the next distributed encoding;

[0014] The probability of occurrence of the condition satisfying the updating of the computing task allocation parameter decreases with an increase in the number of times of performing the distributed encoding.

[0015] In a possible implementation, the determining of the condition satisfying the updating of the computing task allocation parameter includes determining that the following inequality is true:

[0016] e -a×c×dE÷Tb >rand();

[0017] wherein a and c are constants, dE is a difference between the maximum encoding duration of the current time of performing the distributed encoding and the maximum encoding duration of the last time of performing the distributed encoding, Tb is a variable gradually decreasing with an increase in the number of times of performing the distributed encoding, and rand() is a random number between 0 and 1.

[0018] In a possible implementation, a and c are determined by the following steps:

[0019] When determining whether the inequality is true for the first time, a and c are determined according to e -a×c×dE÷Tb = 0.5.

[0020] In a possible implementation, the determining of the computing task allocation parameter for the next time of performing the distributed encoding based on the encoding duration fed back by each encoding node and the computing task allocation parameter used for the current time of performing the distributed encoding includes determining the computing task allocation parameter for the next time of performing the distributed encoding by the following formula:

[0021]

[0022] wherein k represents the number of groups of the plurality of original computing task groups, D i represents the number of groups of the original computing task group to which the i th encoding node is allocated at the current time of performing the distributed encoding, t i represents the encoding duration of the i th encoding node at the current time of performing the distributed encoding, X represents the total number of encoding nodes, and D' i represents the number of groups of the original computing task group to which the i th encoding node is allocated at the next time of performing the distributed encoding in the case where the number of groups of the plurality of original computing task groups is k.

[0023] In a second aspect, the present application provides an encoding distributed computing system, comprising a plurality of encoding nodes and a master node applying the method described in any of the possible implementation manners of the first aspect, and each encoding node is in communication connection with the master node.

[0024] In a third aspect, the present application provides a large-strip encoding computing task allocation device for encoding distributed computing, comprising:

[0025] The computing task group distribution module is configured to distribute a plurality of original computing task groups to each encoding node based on a computing task distribution parameter used in the current execution of the distributed encoding, so as to form sub-computing tasks of each encoding node;

[0026] The distribution parameter determination module is configured to determine the computing task distribution parameter used in the next execution of the distributed encoding in a load balancing manner based on the encoding time length fed back by each encoding node;

[0027] The plurality of original computing task groups are obtained by splitting a matrix multiplication computing task, the sub-computing task is used to instruct the encoding node to perform encoding on the distributed original computing task group according to the encoding mode of the RS code to obtain an encoding check group, the computing task distribution parameter is used to represent the quantity ratio between the original computing task groups distributed to different encoding nodes, and the original computing task group and the encoding check group form a strip corresponding to the matrix multiplication computing task.

[0028] In a fourth aspect, the present application provides a / image signal generator / network device / transmitter / terminal / base station / industrial computer, comprising: at least one memory for storing programs; at least one processor for executing the programs stored in the memory, and when the programs stored in the memory are executed, the processor is configured to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0029] In a fifth aspect, the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and when the computer program runs on the processor, the processor executes the method described in the first aspect or any possible implementation manner of the first aspect.

[0030] In a sixth aspect, the present application provides a computer program product, and when the computer program product runs on the processor, the processor executes the method described in the first aspect or any possible implementation manner of the first aspect.

[0031] It can be understood that the beneficial effects of the above-mentioned second aspect to sixth aspect can be referred to the related description in the first aspect, and will not be repeated here.

[0032] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects:

[0033] For encoding nodes in an encoding distributed computing system, different encoding nodes generally have different computing power (heterogeneous relationship between different encoding nodes). By obtaining the encoding time of each encoding node in each execution of distributed encoding, the encoding time can reflect the computing power of the node. Based on the encoding time fed back by each encoding node, the computing task allocation parameters used in the next execution of distributed encoding can be determined in a load balancing manner. As the number of times of executing distributed encoding increases, the multiple encoding nodes can tend to a more balanced state, avoiding overloading of some encoding nodes and underloading of the last encoding node. The computing power resources of each encoding node can be more efficiently utilized, the maximum encoding time (the largest one among the encoding times fed back by each encoding node) is reduced, thereby reducing the overall encoding time (i.e. reducing the time for generating m encoding check groups), and the encoding efficiency can be effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is one of the flowcharts of the large strip encoding computing task allocation method for encoding distributed computing provided by the embodiments of the present application;

[0035] Figure 2 is another flowchart of the large strip encoding computing task allocation method for encoding distributed computing provided by the embodiments of the present application;

[0036] Figure 3 is a parallel encoding distributed computing framework diagram provided by the embodiments of the present application;

[0037] Figure 4 is a diagram of the number of computing task groups of each encoding node and the encoding time provided by the embodiments of the present application;

[0038] Figure 5 is a structural diagram of the large strip encoding computing task allocation device for encoding distributed computing provided by the embodiments of the present application;

[0039] Figure 6 is a structural diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0041] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0042] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0043] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0044] Figure 1 This is one of the flow charts of the method for allocating large-strip coding computing tasks for distributed coding computing provided by the embodiment of the present application, such as Figure 1 As shown, the execution subject of the method may be an electronic device, such as a server, etc. The method includes the following steps S101 and S102.

[0045] Step S101, based on the computing task allocation parameters used in the current distributed encoding execution, multiple original computing task groups are allocated to each encoding node to form sub-computing tasks of each encoding node;

[0046] Step S102: Based on the encoding duration fed back by each encoding node, a computing task allocation parameter to be used for the next distributed encoding is determined in a load balancing manner;

[0047] Among them, multiple original computing task groups are obtained by splitting the matrix multiplication computing task, the sub-computation task is used to instruct the encoding node to perform encoding on the assigned (one or more) original computing task groups according to the encoding method of Reed-Solomon (RS) code to obtain the encoding verification group, and the computing task allocation parameter is used to represent the quantity ratio between the original computing task groups assigned to different encoding nodes. The original computing task group and the encoding verification group form the strip corresponding to the matrix multiplication computing task.

[0048] The current time in the current distributed encoding execution may be any time, for example, the current time may be the first time, the second time, the third time, or the tenth time.

[0049] The matrix multiplication calculation task is a task of performing a multiplication operation on two matrices. Specifically, the two matrices are used as two operands of a binary operator, and the binary operator is specifically a multiplication operator.

[0050] For distributed coding, specifically, the coding tasks of multiple original computing task groups may be assigned to multiple coding nodes for processing.

[0051] The RS code used in the embodiments of the present application may be a Cauchy RS code or an optimized Cauchy RS code.

[0052] Cauchy RS codes use the Cauchy matrix as their generator matrix, which offers superior performance, especially in hardware implementation, compared to the Vandermonde matrix used in traditional RS codes. Optimized Cauchy RS codes are based on basic RS codes but with an optimized generator matrix, resulting in fewer "1"s and fewer XOR operations. These optimized Cauchy RS erasure codes also have lower encoding time overhead.

[0053] The next time in the next distributed encoding execution can be the time after the current time. For example, if the current time is the first time, the next time is the second time. For another example, if the current time is the third time, the next time is the fourth time.

[0054] For the calculation task allocation parameter, when the number of coding nodes is X, the calculation task allocation parameter can be expressed as [D1, D2, ..., D X ], D1+D2+…+D X = k, D1 represents the number of original computing task groups assigned to the first coding node when the number of original computing task groups is k, D2 represents the number of original computing task groups assigned to the second coding node when the number of original computing task groups is k, and so on. X Indicates the number of original computing task groups assigned to the Xth coding node when the number of original computing task groups is k. It can be seen that the computing task allocation parameter can represent the ratio of the number of original computing task groups assigned to different coding nodes. For example, for the first and second coding nodes, the ratio of the number of original computing task groups assigned to these two different coding nodes is D1:D2.

[0055] Here, an example is given to illustrate how to split the matrix multiplication calculation task into k original calculation task groups and form the sub-computation tasks of each encoding node. The matrix on the left side of the binary operator and the matrix on the right side of the binary operator in the matrix multiplication calculation task can be divided into k original calculation task groups, and each of the k groups includes a sub-matrix transversely cut from the left matrix and a sub-matrix longitudinally cut from the right matrix. For any two groups in the k groups, the sub-matrix cut by one group does not overlap with the sub-matrix cut by the other group. After obtaining the k original calculation task groups, the k groups can be allocated to the encoding nodes according to the calculation task allocation parameters to form the sub-computation tasks of the encoding nodes. If the number of encoding nodes is X, the number of original calculation task groups allocated to each encoding node can be represented as [D1, D2, …, DX]. X Correspondingly, the sub-computation task of the first encoding node is to perform encoding on the D1 original calculation task groups allocated thereto, the sub-computation task of the second encoding node is to perform encoding on the D2 original calculation task groups allocated thereto, and so on, and the sub-computation task of the Xth encoding node is to perform encoding on the DX original calculation task groups allocated thereto. X

[0056] After forming the sub-computation tasks of the encoding nodes, the sub-computation tasks can be sent to the corresponding encoding nodes, and one encoding node is allocated one sub-computation task. The encoding nodes perform encoding calculation according to the sub-computation tasks. Timers can be configured on the encoding nodes to count the encoding time of each encoding node.

[0057] Optionally, examples are given here to illustrate how to count the encoding time of each encoding node. For example:

[0058] (1) Count the time length of the encoding node for further splitting of the sub-computation task after receiving the sub-computation task;

[0059] (2) Count the time length of the encoding node for creating multiple threads and using the multiple threads for parallel encoding;

[0060] (3) Count the time length of the encoding node for merging the calculation task verification groups of the multiple threads.

[0061] (4) Construct a one-dimensional array, each element of which corresponds to the encoding time of an encoding node, indirectly representing the computing power of the encoding node.

[0062] Optionally, the encoding node can use multiple cores on the processor for parallel encoding.

[0063] ​One encoding node includes one or more processors, one processor includes multiple cores, multiple cores refer to multiple cores (nuclei) on one processor, and the specific technical means of "parallel encoding" is to further split the sub-tasks after receiving the sub-computing tasks, create multiple threads, and use multiple threads for parallel encoding.

[0064] It can be understood that, for the encoding nodes in the distributed encoding system, different encoding nodes generally have different computing power (heterogeneous relationship between different encoding nodes), by obtaining the encoding time of each encoding node in each execution of distributed encoding, the encoding time can reflect the computing power of the node, based on the encoding time fed back by each encoding node, the computing task allocation parameters used in the next execution of distributed encoding can be determined in a load balancing manner, as the number of times of executing distributed encoding increases, multiple encoding nodes can tend to a more balanced state, avoiding overloading of some encoding nodes and avoiding overloading of the last encoding node, which can more efficiently utilize the computing power resources of each encoding node, reduce the maximum encoding time (the maximum item in the encoding time fed back by each encoding node), thereby reducing the overall encoding time (i.e. reducing the time for generating m encoding check groups), which can effectively improve the encoding efficiency.

[0065] In a possible implementation, the above-mentioned determination of the computing task allocation parameters used in the next execution of distributed encoding in a load balancing manner based on the encoding time fed back by each encoding node includes:

[0066] determining the maximum encoding time of the current execution of distributed encoding based on the encoding time fed back by each encoding node (for example, statistics of the encoding time fed back by each encoding node, and selecting the maximum item as the maximum encoding time);

[0067] In the case that the current execution of distributed encoding is the first execution of distributed encoding, the computing task allocation parameters used in the next execution of distributed encoding are determined based on the encoding time fed back by each encoding node and the computing task allocation parameters used in the current execution of distributed encoding;

[0068] In the case that the maximum encoding time of the current execution of distributed encoding is less than or equal to the maximum encoding time of the last execution of distributed encoding (the last execution is the one before the current execution, for example, in the case that the current execution is the third time, the last execution is the second time), the computing task allocation parameters used in the next execution of distributed encoding are determined based on the encoding time fed back by each encoding node and the computing task allocation parameters used in the current execution of distributed encoding;

[0069] In a case that the maximum encoding duration of the current time of performing distributed encoding is greater than the maximum encoding duration of the last time of performing distributed encoding, if a condition of updating the computing task allocation parameter is met, the computing task allocation parameter used in the next time of performing distributed encoding is determined based on the encoding duration fed back by each encoding node and the computing task allocation parameter used in the current time of performing distributed encoding, and if the condition of updating the computing task allocation parameter is not met, the computing task allocation parameter used in the last time of performing distributed encoding is determined as the computing task allocation parameter used in the next time of performing distributed encoding.

[0070] The occurrence probability of the condition of updating the computing task allocation parameter decreases with the increase of the number of times of performing distributed encoding.

[0071] It can be understood that after the maximum encoding duration of the current time of performing distributed encoding (as the current encoding state) is obtained, the maximum encoding duration of the current time of performing distributed encoding and the maximum encoding duration of the last time of performing distributed encoding (as the last encoding state) can be compared. If the maximum encoding duration of the current time of performing distributed encoding is less than or equal to the maximum encoding duration of the last time of performing distributed encoding, it indicates that the current encoding state is better than the last encoding state, and the computing task allocation parameter has the possibility of further optimization. In this case, the computing task allocation parameter can be continuously updated, that is, the encoding duration fed back by each encoding node and the computing task allocation parameter used in the current time of performing distributed encoding are taken as the basis to update the computing task allocation parameter in a load balancing manner, and the updated computing task allocation parameter is taken as the computing task allocation parameter used in the next time of performing distributed encoding.

[0072] If the maximum encoding duration of the current time of performing distributed encoding is greater than the maximum encoding duration of the last time of performing distributed encoding, it indicates that the current encoding state is not better than the last encoding state. However, considering that the computing task allocation parameter has a greater possibility of further optimization (that is, the plurality of encoding nodes tend to be in a more balanced state by updating the computing task allocation parameter) in the process of performing distributed encoding for a plurality of times, a condition of updating the computing task allocation parameter can be set. The occurrence probability of the condition of updating the computing task allocation parameter decreases with the increase of the number of times of performing distributed encoding. In this way, even if the current encoding state is not better than the last encoding state in the process of performing distributed encoding for a plurality of times, the computing task allocation parameter will be continuously updated with a greater probability (for example, the probability value is close to 50%) and the computing task allocation parameter used in the last time of performing distributed encoding will be taken as the computing task allocation parameter used in the next time of performing distributed encoding with a smaller probability.

[0073] After several previous distributed coding executions, in the subsequent distributed coding execution process, if the maximum coding time of the current distributed coding execution is greater than the maximum coding time of the previous distributed coding execution, in this case, the computing task allocation parameters are less likely to be further optimized, and the computing task allocation parameters will continue to be updated with a smaller probability. Correspondingly, the computing task allocation parameters used in the previous distributed coding execution will be used as the computing task allocation parameters used in the next distributed coding execution with a greater probability (that is, the computing task allocation parameters will not be updated).

[0074] Therefore, by updating the computing task allocation parameters when the current encoding state is better than the previous encoding state, and by updating the computing task allocation parameters according to a certain probability when the current encoding state is not better than the previous encoding state, it can be ensured that as the number of distributed encoding executions increases, multiple encoding nodes can tend to a more balanced state, reduce the maximum encoding time, and effectively improve the encoding efficiency.

[0075] In one possible implementation, determining that a condition for updating the computing task allocation parameter is satisfied includes determining that the following inequality holds:

[0076] e -a×c×dE÷Tb >rand();

[0077] Where a and c are constants, dE is the difference between the maximum encoding time of the current distributed encoding and the maximum encoding time of the previous distributed encoding, Tb is a variable that gradually decreases as the number of distributed encoding executions increases, and rand() is a random number between 0 and 1.

[0078] In a possible implementation, the method further includes determining a and c by the following steps:

[0079] When first judging whether the inequality is true, according to e -a×c×dE÷Tb =0.5, determine a and c.

[0080] It should be noted that the above inequality is only determined when the maximum encoding time of the current distributed coding execution is greater than the maximum encoding time of the previous distributed coding execution. It can be seen that when determining whether the above inequality is established, dE is a positive number.

[0081] For the first time determining whether the inequality is established, since in the case of the current time performing distributed encoding being the first time performing distributed encoding, the computing task allocation parameter adopted by the next time performing distributed encoding is determined based on the encoding time length fed back by each encoding node and the computing task allocation parameter adopted by the current time performing distributed encoding, thus, when the distributed encoding is performed for the first time, the determination of whether the inequality is established is not involved, and it can be seen that the first time determining whether the inequality is established occurs for the first time when the distributed encoding is performed for the second time (but it is not limited that the first time determining whether the inequality is established occurs for the first time when the distributed encoding is performed for the second time).

[0082] For example, when the maximum encoding time length of the second time performing distributed encoding is greater than the maximum encoding time length of the first time (the first time) performing distributed encoding, the determination of whether the inequality is established is the first time in this case, that is, the first time determining occurs when the distributed encoding is performed for the second time.

[0083] For another example, when the maximum encoding time length of the second time performing distributed encoding is less than the maximum encoding time length of the first time (the first time) performing distributed encoding, the determination of whether the inequality is established is not needed in this case; and when the maximum encoding time length of the third time performing distributed encoding is greater than the maximum encoding time length of the second time performing distributed encoding, the determination of whether the inequality is established is the first time in this case, that is, the first time determining occurs when the distributed encoding is performed for the third time.

[0084] Optionally, Tb=f(nt), nt represents the number of times of performing distributed encoding, f(nt) is a monotone decreasing function about nt, and f(nt)>0.

[0085] Next, how to determine a and c is described by assuming that Tb=f(nt), the first time determining occurs when the distributed encoding is performed for the second time (nt=2), and dE=6.93, Tb=f(2)=10, in this case, if a=1 and c=1 are set, e -a×c×dE÷Tb = e -1×1×6.93÷10 = e -0.693 ≈0.5, thus, in this case, it can be determined that a=1 and c=1.

[0086] Next, the characteristic that the probability of the inequality being established decreases with the increase of the number of times of performing distributed encoding is described by assuming that Tb=f(nt), f(nt)>0, a=1 and c=1, dE is fixed (dE is a positive number), f(nt) is a monotone decreasing function about nt with the increase of the number of times of performing distributed encoding (nt gradually increases), Tb gradually decreases and is a positive number, it can be seen that -a×c×dE÷Tb is a negative number and the negative part becomes more negative, and then e -a×c×dE÷Tb the value of e-a×c×dE÷Tb The probability of rand() being true gradually decreases, that is, the probability of satisfying the condition of updating the calculation task allocation parameter decreases as the number of times of performing distributed encoding increases.

[0087] It can be understood that, considering the possibility that the calculation task allocation parameter can be further optimized in the process of performing distributed encoding several times, by setting e -a×c×dE÷Tb The value of e is approximately equal to 0.5, and rand() is a random number between 0 and 1. Therefore, even if the current encoding state is not better than the last encoding state, there is a relatively large probability (close to 50%) of updating the calculation task allocation parameter. By updating the calculation task allocation parameter as much as possible in the process of performing distributed encoding several times, the multiple encoding nodes can tend to a more balanced state.

[0088] In a possible implementation, the calculation task allocation parameter used in the next execution of distributed encoding is determined based on the encoding duration fed back by each encoding node and the calculation task allocation parameter used in the current execution of distributed encoding, and includes determining the calculation task allocation parameter used in the next execution of distributed encoding by the following formula:

[0089]

[0090] Wherein, k represents the number of groups of the multiple original calculation task groups, D i represents the number of groups of the original calculation task group allocated to the i-th encoding node in the current execution of distributed encoding, t i represents the encoding duration of the i-th encoding node in the current execution of distributed encoding, and X represents the total number of encoding nodes, D' i represents the number of groups of the original calculation task group allocated to the i-th encoding node in the next execution of distributed encoding, under the condition that the number of groups of the multiple original calculation task groups is k.

[0091] It can be understood that D i ×1 / t i can represent the computing power of the i-th encoding node. If D i ×1 / t i The higher the value of D iThe higher the value of D is, the more original computing task groups are allocated to the encoding node with more computing resources, and the less original computing task groups are allocated to the encoding node with less computing resources, so that the multiple encoding nodes tend to be more balanced.

[0092] The large-strip encoding computing task allocation method for encoding distributed computing provided by the present application is described below in combination with several examples, but is not limited to the present application.

[0093] Figure 2 is a flowchart of the large-strip encoding computing task allocation method for encoding distributed computing provided by the present application, as shown in Figure 2 The method comprises the following steps S201 to S206.

[0094] S201, based on the matrix multiplication computing task generated by the user, forming each sub-computing task.

[0095] Specifically, when the distributed encoding is performed for the first time, the computing task uniform allocation method can be used to uniformly allocate the multiple original computing task groups to each encoding node to form the sub-computing task of each encoding node.

[0096] The number of original computing task groups allocated to each encoding node can be represented as [D1, D2, …, D X ]. When the distributed encoding is performed for the first time, since the computing task uniform allocation method is used, for any two encoding nodes, for example, the i-th encoding node and the j-th encoding node, the number of original computing task groups allocated to the i-th encoding node D i is equal to the number of original computing task groups allocated to the j-th encoding node D j .

[0097] S202, sending each sub-computing task to the corresponding encoding node for encoding calculation.

[0098] Specifically, each sub-computing task is sent to the corresponding encoding node, and each encoding node uses multi-threading to perform parallel encoding on the sub-computing task on multiple cores.

[0099] S203, statistics the encoding duration of each node.

[0100] Specifically, the passive updating method is adopted, and each encoding node calculates the encoding duration after the encoding is completed, and transmits the encoding duration to the master node in the form of a message. The master node calculates the encoding duration of each encoding node and records it in the form of a one-dimensional array.

[0101] S204, taking the maximum encoding duration as the current encoding state (hereinafter referred to as state).

[0102] S205, determining the next calculation task allocation parameter according to the current state and the previous state.

[0103] When the state is transferred, it is determined whether the current state is better than the previous state. If the current state is better than the previous state, or the following inequality is established when the previous state is better, the current state is accepted:

[0104] e -a×c×dE÷Tb >rand();

[0105] If the above inequality is not established when the previous state is better, the current state is not accepted, and the value is returned to the previous state (determining the calculation task allocation parameter used in the previous distributed encoding as the calculation task allocation parameter used in the next distributed encoding).

[0106] It can be understood that, by using the above inequality, the probability of still using the current state in the case of an unsatisfactory current state is related to time, and the update is frequent at the beginning and reduced at the end.

[0107] If the current state is accepted, the calculation task allocation parameter needs to be updated, and the calculation task allocation parameter used in the next distributed encoding is determined by the following formula:

[0108]

[0109] S206, performing the next distributed encoding (steps S201-S205 are for performing the current distributed encoding).

[0110] Figure 3 is a parallel encoding distributed computing framework provided by an embodiment of the present application, as shown in Figure 3 (n, k) = (9, 8), m = 1, and the number of encoding nodes is 4. After the matrix multiplication calculation task generated by the user is divided and allocated to different encoding nodes for encoding calculation, the encoding calculation duration is fed back in the form of a message to adjust the calculation task allocator.

[0111] For the parallel encoding distributed computing framework, "parallel encoding" means that encoding is performed in parallel by multiple encoding nodes, and "distributed" means that multiple encoding nodes are in a distributed relationship.

[0112] As shown in Figure 3As shown, the master node divides the user-generated matrix multiplication computing task task into k original computing task groups, and then assigns the k original computing task groups to each encoding node according to the computing task assignment parameter, to form four sub-computing tasks task1, task2, task3 and task4.

[0113] Figure 4 is a schematic diagram of the number of computing task groups of each encoding node and the encoding duration provided by the embodiment of the present application, as shown in Figure 4 each sub-computing task contains 2 original computing task groups (i.e. 2 original computing task groups need to be executed for encoding calculation), and the encoding results of each sub-computing task are merged to obtain task p , task p contains 1 computing task group (i.e. m = 1 encoding check group).

[0114] As shown in Figure 3 , after receiving the sub-computing task, each encoding node uses an encoder (which belongs to a linear operation encoder) to encode the sub-computing task in a multi-core manner.

[0115] The master node counts the encoding duration of each encoding node, as shown in Figure 4 , the encoding duration corresponding to task1 is 1 unit time, the encoding duration corresponding to task2 is 1 unit time, the encoding duration corresponding to task3 is 2 unit times, and the encoding duration corresponding to task4 is 2 unit times, and then a one-dimensional matrix [1 1 2 2] is constructed to represent the encoding duration of the four encoding nodes.

[0116] By counting the encoding duration of each encoding node, the maximum encoding duration of 2 can be determined as the state of the current time;

[0117] If it is the first time to perform distributed encoding, the state of the current time is unconditionally accepted, i.e. in the case that the current time to perform distributed encoding is the first time to perform distributed encoding, based on the encoding duration fed back by each encoding node and the computing task assignment parameter used in the current time to perform distributed encoding, the computing task assignment parameter used in the next time to perform distributed encoding is determined. If it is not the first time to perform distributed encoding, it is necessary to compare with the state of the previous time, and according to the comparison result, the computing task assignment parameter used in the next time to perform distributed encoding is determined.

[0118] Thus, if it is the first time to perform distributed encoding, the calculation task distribution parameters need to be updated according to the encoding time length of each node (as the calculation task distribution parameters adopted in the next time to perform distributed encoding), and it is known by calculation using the above formula that, in the next time to perform distributed encoding, if the number of original calculation task groups is k = 8, the number of calculation task groups allocated to the encoding node 1 should be D'1 = 8 x (1 x 2 ÷ (1 x 2 + 1 x 2 + 0.5 x 2 + 0.5 x 2) ≈ 3 (rounded result), and by analogy, 3, 1, and 1 calculation task groups are allocated to the encoding node 2, the encoding node 3, and the encoding node 4 respectively, that is, [3 3 1 1]. It should be noted that for any encoding node, for example, the ith encoding node, if D' i i ≈0 (rounded result), the number of calculation task groups allocated to other encoding nodes needs to be adjusted to ensure that the ith encoding node is allocated at least one calculation task group.

[0119] The large strip encoding calculation task distribution device for encoding distributed calculation provided in the present application will be described below. The large strip encoding calculation task distribution device for encoding distributed calculation described below can be correspondingly referred to the large strip encoding calculation task distribution method for encoding distributed calculation described above.

[0120] Figure 5 FIG. 1 is a structural schematic diagram of the large strip encoding calculation task distribution device for encoding distributed calculation provided in an embodiment of the present application. As shown in FIG. 1, the device comprises: Figure 5

[0121] A calculation task group distribution module 10, configured to distribute a plurality of original calculation task groups to each encoding node based on the calculation task distribution parameters adopted in the current time to perform distributed encoding, to form sub-calculation tasks of each encoding node.

[0122] A distribution parameter determination module 20, configured to determine the calculation task distribution parameters adopted in the next time to perform distributed encoding in a load balancing manner based on the encoding time length fed back by each encoding node.

[0123] Wherein, the plurality of original calculation task groups are obtained by splitting the matrix multiplication calculation task, the sub-calculation task is used to instruct the encoding node to perform encoding on the allocated original calculation task group according to the encoding mode of the RS code to obtain an encoding check group, the calculation task distribution parameter is used to represent the quantity ratio between the original calculation task groups allocated to different encoding nodes, and the original calculation task group and the encoding check group form a strip corresponding to the matrix multiplication calculation task.

[0124] It can be understood that the detailed function implementation of each unit / module described above can be referred to the description in the foregoing method embodiments, which will not be described here. ​

[0125] It should be understood that the above device is used to execute the method in the above embodiment, the corresponding program module in the device, the implementation principle and technical effect are similar to the description in the above method, the working process of the device can refer to the corresponding process in the above method, and details are not described here.

[0126] Based on the method in the above embodiment, the embodiment of the present application provides an electronic device, Figure 6 is a structural schematic diagram of an electronic device provided by the embodiment of the present application, as Figure 6 indicated, the electronic device can include a processor (Processor) 810, a communications interface (Communications Interface) 820, a memory (Memory) 830 and a communications bus 840, wherein the processor 810, the communications interface 820, the memory 830 complete mutual communication through the communications bus 840. The processor 810 can call the logical instructions in the memory 830 to execute the method in the above embodiment.

[0127] In addition, the logical instructions in the memory 830 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or in the form of a part or the technical solutions of the present application. The computer software product is stored in a storage medium, includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application.

[0128] Based on the method in the above embodiment, the embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, when the computer program runs on the processor, so that the processor executes the method in the above embodiment.

[0129] Based on the method in the above embodiment, the embodiment of the present application provides a computer program product, when the computer program product runs on the processor, so that the processor executes the method in the above embodiment.

[0130] It can be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0131] The method steps in the embodiments of the present application can be implemented in the form of hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0132] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in or transmitted by a computer readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0133] It can be understood that various numerical numbers involved in the embodiments of the present application are only distinguished for convenience of description, and are not used to limit the scope of the embodiments of the present application.

[0134] Those skilled in the art easily understand that the above only describes the preferred embodiments of the present application and is not used to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A large stripe encoding computation task allocation method for encoding distributed computing, characterized in that, The method comprises the following steps: based on the computing task allocation parameter adopted in the current execution of distributed encoding, allocating a plurality of original computing task groups to each encoding node to form sub-computing tasks of each encoding node; based on the encoding time length fed back by each encoding node, determining the computing task allocation parameter adopted in the next execution of distributed encoding in a load balancing manner; wherein the plurality of original computing task groups are obtained by splitting matrix multiplication computing tasks, the sub-computing tasks are used to instruct the encoding nodes to perform encoding on the allocated original computing task groups according to the encoding mode of RS code to obtain encoding check groups, the computing task allocation parameter is used to represent the quantity ratio between the original computing task groups allocated to different encoding nodes, and the original computing task groups and the encoding check groups form the strip corresponding to the matrix multiplication computing task; the step of determining the computing task allocation parameter adopted in the next execution of distributed encoding based on the encoding time length fed back by each encoding node in a load balancing manner comprises the following steps: based on the encoding time length fed back by each encoding node, determining the maximum encoding time length of the current execution of distributed encoding; in the case that the current execution of distributed encoding is the first execution of distributed encoding, based on the encoding time length fed back by each encoding node and the computing task allocation parameter adopted in the current execution of distributed encoding, determining the computing task allocation parameter adopted in the next execution of distributed encoding; in the case that the maximum encoding time length of the current execution of distributed encoding is less than or equal to the maximum encoding time length of the last execution of distributed encoding, based on the encoding time length fed back by each encoding node and the computing task allocation parameter adopted in the current execution of distributed encoding, determining the computing task allocation parameter adopted in the next execution of distributed encoding; in the case that the maximum encoding time length of the current execution of distributed encoding is greater than the maximum encoding time length of the last execution of distributed encoding, if it is determined that the condition of updating the computing task allocation parameter is met, based on the encoding time length fed back by each encoding node and the computing task allocation parameter adopted in the current execution of distributed encoding, determining the computing task allocation parameter adopted in the next execution of distributed encoding, and if it is determined that the condition of updating the computing task allocation parameter is not met, determining the computing task allocation parameter adopted in the last execution of distributed encoding as the computing task allocation parameter adopted in the next execution of distributed encoding; wherein the occurrence probability of the condition of updating the computing task allocation parameter decreases with the increase of the number of times of executing distributed encoding.

2. The method of claim 1, wherein, the step of determining that the condition of updating the computing task allocation parameter is met comprises determining that the following inequality is true: ； wherein, and is a constant, is a difference between the maximum encoding duration of the current time of performing distributed encoding and the maximum encoding duration of the last time of performing distributed encoding, is a variable that gradually decreases as the number of times of performing distributed encoding increases, is a random number between 0 and 1.

3. The method of claim 2, wherein, Also included are determining and : At the first time of judging whether the inequality is established, according to , determine and .

4. The method of claim 1, wherein the method further comprises: the step of determining the computing task allocation parameter adopted in the next execution of distributed encoding based on the encoding time length fed back by each encoding node and the computing task allocation parameter adopted in the current execution of distributed encoding comprises determining the computing task allocation parameter adopted in the next execution of distributed encoding by the following formula: ； in, Indicates the number of groups of multiple original computing task groups, Indicates the number of times distributed coding is currently executed. The number of original computing task groups to which the encoding nodes are assigned, Indicates the number of times distributed coding is currently executed. The encoding time of each encoding node, Indicates the total number of coding nodes, Indicates that the number of groups in multiple original computing task groups when distributed coding is executed next time is In the case of The number of original computing task groups to which the encoding nodes are assigned.

5. An encoding distributed computing system, comprising: The method comprises the following steps: a plurality of encoding nodes and a master node applying the large strip encoding computing task allocation method for encoding distributed computing according to any one of claims 1-4, each encoding node being in communication connection with the master node.

6. A large stripe encoding computing task allocation apparatus for encoding distributed computing, characterized by, The method comprises the following steps: The computing task group distribution module is configured to distribute a plurality of original computing task groups to each encoding node based on a computing task distribution parameter used in the current execution of distributed encoding, so as to form sub-computing tasks of each encoding node; The distribution parameter determination module is configured to determine the computing task distribution parameter used in the next execution of distributed encoding in a load balancing manner based on the encoding time fed back by each encoding node; The plurality of original computing task groups are obtained by splitting a matrix multiplication computing task, the sub-computing tasks are used to instruct the encoding nodes to perform encoding on the distributed original computing task groups in a manner of RS code to obtain encoding check groups, the computing task distribution parameter is used to represent the quantity ratio between the original computing task groups distributed to different encoding nodes, and the original computing task groups and the encoding check groups form a strip corresponding to the matrix multiplication computing task. The determination of the computing task distribution parameter used in the next execution of distributed encoding in a load balancing manner based on the encoding time fed back by each encoding node comprises: determining the maximum encoding time of the current execution of distributed encoding based on the encoding time fed back by each encoding node; in the case that the current execution of distributed encoding is the first execution of distributed encoding, determining the computing task distribution parameter used in the next execution of distributed encoding based on the encoding time fed back by each encoding node and the computing task distribution parameter used in the current execution of distributed encoding; in the case that the maximum encoding time of the current execution of distributed encoding is less than or equal to the maximum encoding time of the last execution of distributed encoding, determining the computing task distribution parameter used in the next execution of distributed encoding based on the encoding time fed back by each encoding node and the computing task distribution parameter used in the current execution of distributed encoding; in the case that the maximum encoding time of the current execution of distributed encoding is greater than the maximum encoding time of the last execution of distributed encoding, if it is determined that a condition for updating the computing task distribution parameter is met, determining the computing task distribution parameter used in the next execution of distributed encoding based on the encoding time fed back by each encoding node and the computing task distribution parameter used in the current execution of distributed encoding, and if it is determined that the condition for updating the computing task distribution parameter is not met, determining the computing task distribution parameter used in the last execution of distributed encoding as the computing task distribution parameter used in the next execution of distributed encoding; wherein the occurrence probability of the condition for updating the computing task distribution parameter decreases with the increase of the number of times of executing distributed encoding.

7. An electronic device, comprising: The computer program product comprises: at least one memory for storing a computer program; at least one processor for executing the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute the method of any one of claims 1-4.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: When the computer program runs on the processor, the processor is caused to execute the method of any one of claims 1-4.

9. A computer program product, characterised in that, When the computer program product runs on the processor, the processor is caused to execute the method of any one of claims 1-4.

Images