Information processing device and information processing method

Abstract

To achieve collective communication in which some information processing devices among a plurality of information processing devices connected to switches in a communication network participate.SOLUTION: A plurality of information processing devices are connected to a first switch among a plurality of switches, and any information processing device of the plurality of information processing devices includes a storage section and a communication section. The storage section stores communication destination information based on a plurality of bit strings related to communication destinations of collective communication, and the communication section communicates with an information processing device connected to a second switch among the plurality of switches on the basis of the communication destination information. Some information processing devices including the information processing device having the storage section and the communication section among the plurality of information processing devices participate in the collective communication. The plurality of bit strings related to the communication destinations of the collective communication is selected from a bit string set related to the communication destinations of the plurality of information processing devices on the basis of the number of some information processing devices.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to information processing. [Background technology]

[0002] When performing parallel distributed processing in HPC (High-Performance Computing), the network topology of the communication network connecting servers or switches is important for efficient data transmission and reception at low cost. Network topology represents the connection form of servers or switches.

[0003] In relation to parallel distributed processing, an information processing system that avoids the occurrence of route contention when performing All-Reduce communication in a full mesh system is known (see, for example, Patent Document 1). A distributed deep learning system that can perform distributed deep learning processing at higher speed is also known (see, for example, Patent Document 2 and Patent Document 3). An abstraction library that enables scalable distributed machine learning is also known (see, for example, Patent Document 4). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-8648 [Patent Document 2] JP 2020-77300 A [Patent Document 3] International Publication No. 2021 / 111490 [Patent Document 4] US Patent Application Publication No. 2018 / 0293492 Summary of the Invention [Problem to be solved by the invention]

[0005] The information processing system in Patent Document 1 includes multiple switches with a full-mesh network topology, with multiple servers connected to each switch. Full-mesh refers to a connection topology in which each switch in a communication network is directly connected to every other switch. A full-mesh network topology allows more servers to be connected with fewer switches.

[0006] When All-Reduce communication is performed between servers in this information processing system, each server generates a bit string indicating the switch to which the destination server is connected based on the communication table, and communicates with the destination server using the generated bit string, thereby avoiding route contention.

[0007] All-Reduce refers to the process of sharing the results of a pre-specified operation on data held by all servers, and All-Reduce communication refers to communication carried out for All-Reduce. Route contention refers to the simultaneous transmission of multiple packets in the same direction along a single route in a communication network.

[0008] According to the information processing system of Patent Document 1, it is possible to avoid route contention in All-Reduce communication using all servers. However, depending on the target of parallel distributed processing, there may be cases where it is desired to submit a partial job using only an appropriate number of servers according to the calculation size. For this reason, it is desirable to be able to flexibly change the number of servers participating in All-Reduce communication.

[0009] This problem occurs not only when a server in a full-mesh communication network performs All-Reduce communication, but also when information processing devices (computers) in various communication networks perform various collective communications. Collective communications are one-to-many, many-to-one, or many-to-many communications performed among multiple communication entities such as information processing devices.

[0010] In one aspect, the present invention aims to realize collective communication in which some of a plurality of information processing devices connected to a switch in a communication network participate. [Means for solving the problem]

[0011] In one scheme, a plurality of information processing devices are connected to a first switch among a plurality of switches, and any one of the plurality of information processing devices includes a storage unit and a communication unit.

[0012] The storage unit stores communication destination information based on a plurality of bit strings relating to communication destinations of collective communication, and the communication unit communicates with an information processing device connected to a second switch among the plurality of switches based on the communication destination information.

[0013] Among the plurality of information processing devices, some of the information processing devices, including an information processing device having a storage unit and a communication unit, participate in collective communication. A plurality of bit strings relating to communication destinations of the collective communication are selected based on the number of the part of the information processing devices from a set of bit strings relating to communication destinations of the plurality of information processing devices. [Effects of the Invention]

[0014] According to one aspect, it is possible to realize collective communication in which some of a plurality of information processing devices connected to a switch in a communication network participate. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 2 is a functional configuration diagram of an information processing device. [Figure 2] FIG. 1 is a configuration diagram of an information processing system. [Figure 3] FIG. 10 is a configuration diagram of an information processing system when k=2. [Figure 4] FIG. 2 is a functional configuration diagram of a control server. [Figure 5] FIG. 1 is a diagram illustrating a primitive polynomial. [Figure 6] FIG. 10 is a diagram illustrating a first example of a set of bit strings. [Figure 7] FIG. 10 is a diagram illustrating a second example of a set of bit strings. [Figure 8] FIG. 10 is a diagram showing a communication table. [Figure 9] FIG. 2 is a functional configuration diagram of a server. [Figure 10] FIG. 10 is a diagram illustrating switches included in an information processing system when k=4. [Figure 11] FIG. 10 is a diagram illustrating communication in communication phase 2. [Figure 12] FIG. 10 is a diagram illustrating communication in communication phase 4. [Figure 13] FIG. 10 illustrates pseudocode. [Figure 14] 10 is a flowchart of All-Reduce communication. [Figure 15] FIG. 2 is a hardware configuration diagram of a first information processing device. [Figure 16] FIG. 2 is a hardware configuration diagram of a second information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, embodiments will be described in detail with reference to the drawings.

[0017] Fig. 1 shows an example of a functional configuration of an information processing device according to an embodiment. An information processing device 101 in Fig. 1 is one of a plurality of information processing devices, and includes a storage unit 111 and a communication unit 112. The plurality of information processing devices are connected to a first switch among a plurality of switches.

[0018] The storage unit 111 stores communication destination information based on a plurality of bit strings relating to communication destinations of collective communication, and the communication unit 112 communicates with an information processing device connected to a second switch among the plurality of switches based on the communication destination information.

[0019] Of the multiple information processing devices, some of the information processing devices, including the information processing device 101, participate in the collective communication. A plurality of bit strings related to the communication destinations of the collective communication are selected based on the number of the partial information processing devices from a set of bit strings related to the communication destinations of the multiple information processing devices.

[0020] According to the information processing device 101 in FIG. 1, collective communication can be realized in which some of the information processing devices participate among a plurality of information processing devices connected to a switch in a communication network.

[0021] Fig. 2 shows an example of the configuration of an information processing system including the information processing device 101 in Fig. 1. The information processing system in Fig. 2 includes switches 201-1 to 201-d (d is an integer of 2 or more) and servers 202-j-1 to 202-jr (j = 1 to d, r is an integer of 2 or more).

[0022] The servers 202-ji (i=1 to r) correspond to the information processing devices 101 in FIG. k (k is an integer greater than or equal to 1), and r=d-1=2 k It is -1.

[0023] The switches 201-1 to 201-d are connected to one another via a communication network 203. The network topology of the communication network 203 is a full mesh. The switch 201-j may be an Infiniband-based switch.

[0024] The number of switches 201-j included in the information processing system is d, and the number of servers 202-ji connected to each switch 201-j is r. Therefore, the number of servers 202-ji included in the information processing system is d*r.

[0025] The r servers 202-ji connected to each switch 201-j communicate via the switch 201-j. The server 202-ji connected to the switch 201-j and the server 202-hi connected to the switch 201-h (h≠j) communicate via the switch 201-j, the communication network 203, and the switch 201-h.

[0026] Each server 202-ji performs parallel distributed processing while communicating with other servers 202-ji. The parallel distributed processing may be distributed machine learning.

[0027] The network topology of the communication network 203 is a full mesh, and d=2 k and r=2 k When the value is -1, it is possible to easily realize All-Reduce communication that avoids the occurrence of route contention.

[0028] Fig. 3 shows an example of the configuration of an information processing system when k = 2. The information processing system in Fig. 3 includes switches 201-1 to 201-4 and servers 202-j-1 to 202-j-3 (j = 1 to 4).

[0029] In this case, the information processing system includes four switches 201-j, and three servers 202-ji are connected to each switch 201-j, so that the information processing system includes 12 servers 202-ji.

[0030] In collective communication performed in an information processing system, all r servers 202-ji connected to each switch 201-j may participate in the collective communication, or only some of the servers 202-ji may participate in the collective communication. The collective communication may be All-Reduce communication, All-Gather communication, etc.

[0031] In a collective communication in which only some of the servers 202-ji connected to each switch 201-j participate, any of the servers 202-ji connected to any of the switches 201-j operates as a control server. The control server generates a communication table indicating the communication destinations of the collective communication and distributes it to each server 202-ji participating in the collective communication. The control server may be a server 202-ji participating in the collective communication, or may be a server 202-ji not participating in the collective communication.

[0032] 4 shows an example of the functional configuration of the control server 401. The control server 401 in FIG.

[0033] Before collective communication is started, the generating unit 411 generates a bit string set 421 and stores it in the storage unit 413. The bit string set 421 corresponds to a bit string set related to the communication destinations of a plurality of information processing devices.

[0034] The generator 411 can generate the bit string set 421 by, for example, a greedy algorithm based on the circular method described in Patent Document 1. In this case, the bit string set 421 includes d-1 bit strings corresponding to integers from 1 to d-1, and each bit string has a length of k bits. When the d-1 bit strings included in the bit string set 421 are arranged in a circle, any k consecutive bit strings are linearly independent.

[0035] When k bit strings consisting of k bits are said to be linearly independent, this means that when one or more bit strings are selected from those bit strings and their exclusive OR is calculated, the result will never be 0, regardless of the combination of bit strings selected. However, it is assumed that the exclusive OR of one bit string is that bit string itself.

[0036] For example, no matter how one or more bit strings are selected from the three bit strings 001, 010, and 100, the exclusive OR will not be 0. Therefore, 001, 010, and 100 are linearly independent.

[0037] On the other hand, if 1010, 1111, and 0101 are selected from the four bit strings 1010, 1111, 0101, and 1011, the exclusive OR becomes 0000. Therefore, 1010, 1111, 0101, and 1011 are not linearly independent.

[0038] The generator 411 generates a Galois field GF(2 k ) can also be used to generate the bit string set 421. A Galois field is sometimes called a finite field.

[0039] In this case, the generation unit 411 selects a primitive polynomial m(x) in advance, and generates a primitive polynomial m(x) in GF(2 k 2 of 2 k Generate polynomials and bit strings corresponding to each of the elements.

[0040] The four arithmetic operations are defined for the generated polynomials and bit strings. Addition and subtraction are the same as normal addition and subtraction. Multiplication is defined using polynomial multiplication and division. For example, for bit string B1=b1 k-1 b1 k-2 b10 and bit string B2=b2 k-1 b2 k-2 ···b20 is expressed by the following polynomial:

[0041] B1 b1 k-1 x k-1 +b1 k-2 x k-2 +···+b10 B2 b2 k-1 x k-1 +b2 k-2 x k-2 +···+b20

[0042] The multiplication result of B1 and B2 is defined as a bit string corresponding to the remainder when the multiplication result of these polynomials is divided by m(x).

[0043] Division is defined as multiplication of the polynomial corresponding to the dividend by the inverse of the polynomial corresponding to the divisor. Since polynomial f(x) and m(x), the minimal polynomial of f(x), are relatively prime, there exist polynomials A(x) and B(x) that satisfy the following equation.

[0044] f(x)A(x)+m(x)B(x)=1 (1)

[0045] In this case, A(x) can be used as the inverse of f(x).

[0046] Figure 5 shows examples of primitive polynomials m(x) when k = 1 to 5. When k = 3 and k = 4, there are two types of m(x), and when k = 5, there are six types of m(x).

[0047] Figure 6 shows the relationship between GF(2 k ) is generated using k=4 and m(x)=x 4 +x+1. Polynomial x q is defined using the index q and is modified using the condition m(x)=0.

[0048] The bit string is the transformed polynomial x q The first (most significant) bit value corresponds to x 3 The second bit represents the coefficient of the term x 2 The third bit represents the coefficient of the term x 1 The fourth (least significant) bit value is x 0 represents the coefficient of the term.

[0049] For example, the polynomial x 4 The transformed polynomial is 1+x. In this case, x 3 and x 2 The coefficient of the term is 0, and x 1 and x 0 The coefficient of the term is 1. Therefore, the polynomial x 4 The corresponding bit string is 0011.

[0050] The integer represents the integer corresponding to the bit string, and is calculated by converting the bit string to a decimal number. For example, the integer corresponding to the bit string 0011 is 3.

[0051] GF(2 k 2 of 2 k Among the bit strings corresponding to each element, the polynomial x 0 ~x 2^k-2 The corresponding 2 k When k-1 bit strings are arranged in a circle, any k consecutive bit strings are linearly independent. The reason for this is explained below.

[0052] polynomial x 0 ~x 2k-2 The corresponding 2 k -1 bit string, k consecutive polynomials x t ~x t+k-1 Assuming that the corresponding bit strings are not linearly independent, there exist s integers I1, I2, . . . , Is that satisfy the following equation:

[0053] t≦I1,I2,...,Is≦t+k-1 (2) x I1 +x I2 +···+x Is =0 (3)

[0054] Both sides of equation (3) are x t Dividing by gives the following equation:

[0055] x I1-t +x I2-t +···+x Is-t =0 (4)

[0056] On the other hand, the following equation is obtained from equation (2):

[0057] 0≦I1-t,I2-t,...,Is-t≦k-1 (5)

[0058] For an integer I such that 0≦I≦k-1, x IThe bit string corresponding to (1) is the bit string 0···010···0, where the I-th least significant bit is 1 and all other bits are 0. Therefore, if equation (5) holds, then equation (4) does not hold.

[0059] This contradiction shows that the initial assumption is incorrect. Therefore, for any k consecutive polynomials x t ~x t+k-1 For each k bits, the corresponding k bit sequences are linearly independent.

[0060] Therefore, the generation unit 411 generates the polynomial x 0 ~x 2^k-2 The corresponding 2 k 6, the bit string set 421 includes 15 bit strings corresponding to q=0 to 14, and the bit string 0000 corresponding to q=-∞ is excluded from the bit string set 421.

[0061] Figure 7 shows the relationship between GF(2 k ) where k=4 and m(x)=x 4 +x 3 +1. The generating unit 411 adopts 15 bit strings corresponding to q=0 to 14 as the bit string set 421.

[0062] By using such a bit string set 421 to determine the communication destination of each server 202-ji, it is possible to avoid the occurrence of path contention when all d-1 servers 202-ji connected to each switch 201-j participate in All-Reduce communication.

[0063] Next, the generating unit 411 generates a communication table 422 using the bit string set 421 and stores it in the storage unit 413. The communication table 422 corresponds to communication destination information based on a plurality of bit strings relating to communication destinations of collective communication.

[0064] When n servers 202-j-i (where k ≤ n < d - 1) among the d - 1 servers 202-j-i connected to each switch 201-j participate in group communication, the generation unit 411 selects n bit strings included in the bit string set 421 as a partial bit string set. Then, the generation unit 411 generates a communication table 422 using the selected n bit strings.

[0065] When the bit string set 421 is generated using GF(2 k ), the generation unit 411 selects, as a partial bit string set, the bit strings corresponding to n elements respectively specified based on n among the 2 k elements of GF(2 k ).

[0066] For example, when n is a divisor of 2 k - 1, the generation unit 411 selects one element every p calculated by the following formula from GF(2<s k ) to identify n elements corresponding to the polynomials x 0 , x p , x 2p , ···, and x (n-1)p respectively.

[0067] p = (2 k - 1) / n (6)

[0068] When the bit strings corresponding to the respectively identified n elements are arranged circularly, any consecutive k bit strings are linearly independent.

[0069] For example, when k = 4 and n = 5, p = 3. Therefore, when the bit string set 421 in FIG. 7 is generated, the following bit strings corresponding to the polynomials x 0 , x 3 , x 6 , x 9 , and x 12 are selected as a partial bit string set.

[0070] x 0 0001 x 3 1000 x 6 1111 x 9 0101 x 12 0011

[0071] GF(2 k ) to generate a bit string set 421, k If it is a divisor of -1, a partial bit string set can be easily generated.

[0072] Fig. 8 shows an example of the communication table 422. The communication table 422 in Fig. 8 is a table with n rows and d columns. 1 to n indicate communication phases, and ID1 to IDd indicate identification information of the switch 201-j (j = 1 to d). As IDj, a k-bit bit string corresponding to the integer j-1 is used. Hereinafter, the switch 201-j indicated by IDj may be referred to as switch IDj.

[0073] Cell C(u,j) in row u, column j (u=1 to n) of communication table 422 includes communication destination identification information indicating the communication destination of each of n servers 202-j-1 to 202-jn connected to switch ID j in communication phase u. Therefore, C(u,j) includes n pieces of communication destination identification information. Each piece of communication destination identification information represents the identification information of switch 201-h connected to server 202-hi, which is the communication destination of server 202-ji.

[0074] The generation unit 411 calculates the communication destination identification information ID(u,j,i) (u=1 to n, j=1 to d, i=1 to n) of the server 202-ji connected to the switch IDj in the communication phase u using the following formula and records it in the communication table 422.

[0075] ID(u,j,i)=IDj xor a(u+i) (7)

[0076] a(i) represents the ith bit string among the n bit strings included in the bit string subset, and xor represents the exclusive OR of bit strings. Note that the index i of a(i) is defined cyclically, so that a(i + n) = a(i).

[0077] By calculating ID(u,j,i) using equation (7), the communication destination of each server 202-ji in collective communication can be easily determined.

[0078] The generation unit 411 transmits the communication table 422 via the communication unit 412 to each server 202-ji participating in the collective communication.

[0079] Fig. 9 shows an example of the functional configuration of each server 202-ji participating in collective communication. The server 901 in Fig. 9 includes a communication unit 412 and a storage unit 413. The communication unit 412 receives a communication table 422 from the control server 401, and the storage unit 413 stores the received communication table 422.

[0080] Next, we will explain the procedure of collective communication in the information processing system of Fig. 2. As an example, All-Reduce communication in which n servers 202-ji connected to each switch 201-j participate is performed in the following procedure.

[0081] (P1) The communication units 412 of the n servers 202-ji connected to each switch 201-j perform All-Reduce communication between the n servers 202-ji connected to the same switch 201-j.

[0082] (P2) The communication units 412 of the n servers 202-ji connected to each switch 201-j perform All-Reduce communication with the server 202-hi connected to a different switch 201-h. In this case, communications from communication phase 1 to communication phase n are executed in order.

[0083] In the communication phase u, the communication unit 412 of the i-th server 202-ji connected to the switch 201-j acquires ID(u,j,i) from C(u,j) in the communication table 422. Then, the communication unit 412 transmits and receives data to and from the i-th server 202-hi connected to the switch 201-h indicated by the acquired ID(u,j,i).

[0084] The control server 401 may be included in the servers 202-ji participating in the All-Reduce communication.

[0085] As described above, among a(u+1) to a(u+n) used to calculate ID(u,j,i) to ID(u,j,n), any k consecutive bit strings are linearly independent. Therefore, by calculating ID(u,j,i) using equation (7) and executing the communication procedure described above, the calculation results of the data held by d*n servers 202-ji can be shared among all servers.

[0086] As a result, All-Reduce communication is realized in which only n servers 202-ji participate among the d-1 servers 202-ji connected to each switch 201-j.

[0087] Fig. 10 shows an example of switch 201-j included in an information processing system when k = 4. The information processing system in Fig. 10 includes switches 201-1 to 201-16. A 4-bit bit string corresponding to integer j-1 is assigned as IDj to each switch 201-j.

[0088] Each circle connected to the switch 201-7 represents one of the 15 servers 202-7-i, and the number i (i=1 to 15) inside the circle represents the i-th server 202-7-i. Note that the 15 servers 202-ji connected to the switches 201-1 to 201-6 and the switches 201-8 to 201-16, respectively, are omitted.

[0089] When n=5, servers 202-j-1 to 202-j-5 connected to each switch 201-j participate in the All-Reduce communication, and servers 202-j-6 to 202-j-15 do not participate in the All-Reduce communication. As the partial bit string set, "0001, 1000, 1111, 0101, 0011" is used, and as a(1) to a(5), the following bit strings are used.

[0090] a(1)=0001 a(2)=1000 a(3)=1111 a(4)=0101 a(5)=0011

[0091] In this case, the communication table 422 is a table with 5 rows and 16 columns, and in step (P2), communications from communication phase 1 to communication phase 5 are executed in order.

[0092] 11 shows an example of communication in communication phase 2 performed by servers 202-7-1 to 202-7-5 connected to switch 0110 in the information processing system of FIG. 10. ID(2,7,1) to ID(2,7,5) included in C(2,7) of communication table 422 are calculated as follows using equation (7):

[0093] ID(2,7,1)=0110 xor a(2+1) =0110 xor a(3) =0110 xor 1111 =1001 (11)

[0094] ID(2,7,2)=0110 xor a(2+2) =0110 xor a(4) =0110 xor 0101 =0011 (12)

[0095] ID(2,7,3)=0110 xor a(2+3) =0110 xor a(5) =0110 xor 0011 =0101 (13)

[0096] ID(2,7,4)=0110 xor a(2+4) =0110 xor a(1) =0110 xor 0001 =0111 (14)

[0097] ID(2,7,5)=0110 xor a(2+5) =0110 xor a(2) =0110 xor 1000 =1110 (15)

[0098] Therefore, server 202-7-1 connected to switch 0110 communicates with server 202-10-1 connected to switch 1001 indicated by ID (2,7,1). Server 202-7-2 connected to switch 0110 communicates with server 202-4-2 connected to switch 0011 indicated by ID (2,7,2).

[0099] Server 202-7-3 connected to switch 0110 communicates with server 202-6-3 connected to switch 0101 indicated by ID (2,7,3). Server 202-7-4 connected to switch 0110 communicates with server 202-8-4 connected to switch 0111 indicated by ID (2,7,4).

[0100] The server 202-7-5 connected to the switch 0110 communicates with the server 202-15-5 connected to the switch 1110 indicated by the ID (2,7,5).

[0101] 12 shows an example of communication in communication phase 4 performed by servers 202-14-1 to 202-14-5 connected to switch 1101 in the information processing system of FIG. 10. ID(4,14,1) to ID(4,14,5) included in C(4,14) of communication table 422 are calculated as follows using equation (7):

[0102] ID(4,14,1)=1101 xor a(4+1) =1101 xor a(5) =1101 xor 0011 =1110 (16)

[0103] ID(4,14,2)=1101 xor a(4+2) =1101 xor a(1) =1101 xor 0001 =1100 (17)

[0104] ID(4,14,3)=1101 xor a(4+3) =1101 xor a(2) =1101 xor 1000 =0101 (18)

[0105] ID(4,14,4)=1101 xor a(4+4) =1101 xor a(3) =1101 xor 1111 =0010 (19)

[0106] ID(4,14,5)=1101 xor a(4+5) =1101 xor a(4) =1101 xor 0101 =1000 (20)

[0107] Therefore, server 202-14-1 connected to switch 1101 communicates with server 202-15-1 connected to switch 1110 indicated by ID (4,14,1). Server 202-14-2 connected to switch 1101 communicates with server 202-13-2 connected to switch 1100 indicated by ID (4,14,2).

[0108] Server 202-14-3 connected to switch 1101 communicates with server 202-6-3 connected to switch 0101 indicated by ID (4,14,3). Server 202-14-4 connected to switch 1101 communicates with server 202-3-4 connected to switch 0010 indicated by ID (4,14,4).

[0109] The server 202-14-5 connected to the switch 1101 communicates with the server 202-9-5 connected to the switch 1000 indicated by the ID (4,14,5).

[0110] The method of selecting a partial bit string set using p in equation (6) is k This is valid when n is a divisor of -1. k If n is not a divisor of −1, the generator 411 can select a partial bit string set from the bit string set 421 by a greedy algorithm using n.

[0111] Fig. 13 shows an example of pseudocode for selecting a partial bitstring set by the greedy method using n. In Fig. 13, B represents the bitstring set 421, and a represents the partial bitstring set.

[0112] For example, when k=3, the generator 411 can generate the following bit string set 421.

[0113] B={001,010,011,100,101,110,111} (21)

[0114] When n=5, the generator 411 generates a set of partial bitstreams in the following procedure in accordance with the pseudocode in FIG.

[0115] (P11) The generation unit 411 deletes 001 from B and adds 001 to the end of a. As a result, a=(001) and B={010, 011, 100, 101, 110, 111}. Then, the generation unit 411 recursively calls f().

[0116] (P12) The generation unit 411 deletes 011 from B and adds 011 to the end of a. As a result, a=(001,011) and B={010,100,101,110,111}. Then, the generation unit 411 recursively calls f().

[0117] (P13) The generation unit 411 deletes 010 from B and adds 010 to the end of a, resulting in a=(001,011,010) and B={100,101,110,111}.

[0118] However, the exclusive OR of the last three terms of a, 001, 011, and 010, is 000, so 001, 011, and 010 are not linearly independent. Therefore, the generation unit 411 deletes 010 from the end of a and adds 010 to B. As a result, a = (001, 011) and B = {010, 100, 101, 110, 111} again.

[0119] (P14) The generation unit 411 deletes 100 from B and adds 100 to the end of a. As a result, a = (001, 011, 100) and B = {010, 101, 110, 111}. Because the last three terms of a, 001, 011, and 100, are linearly independent, the generation unit 411 recursively calls f().

[0120] (P15) The generation unit 411 deletes 010 from B and adds 010 to the end of a. As a result, a = (001, 011, 100, 010) and B = {101, 110, 111}. Because the last three terms of a, 011, 100, and 010, are linearly independent, the generation unit 411 recursively calls f().

[0121] (P16) The generation unit 411 deletes 101 from B and adds 101 to the end of a. As a result, a = (001, 011, 100, 010, 101) and B = {110, 111}. The number of terms in a is 5, and when the five bit strings included in a are arranged in a circle, any three consecutive terms are linearly independent. Therefore, the generation unit 411 selects the five bit strings included in a as a partial bit string set and ends the process.

[0122] The greedy algorithm using n works up to about k=6. When k=6, the number of servers 202-ji included in the information processing system is 2 6 *(2 6 -1)≒2 12 = 4096. According to the pseudocode in Figure 13, when n is 2 k Even if a bit string is not a divisor of -1, a partial bit string set can be generated.

[0123] 14 is a flowchart showing an example of All-Reduce communication in the information processing system of FIG. 2. First, the generation unit 411 of the control server 401 generates a Galois field GF(2 k ) or a greedy method based on a circular method to generate a set of bit strings 421 (step 1401).

[0124] Next, the generating unit 411 generates a communication table 422 using the bit string set 421 (step 1402), and transmits the communication table 422 to each server 202-ji participating in the All-Reduce communication via the communicating unit 412 (step 1403).

[0125] Next, the communication unit 412 of each server 202-ji executes All-Reduce communication using MPI (Message Passing Interface) (step 1404). In step 1404, the communication of the above-mentioned procedure (P1) and procedure (P2) is executed.

[0126] Fig. 15 shows an example of the hardware configuration of a first information processing device used as the control server 401 in Fig. 4. The information processing device in Fig. 15 includes a CPU (Central Processing Unit) 1501, a memory 1502, an input device 1503, an output device 1504, an auxiliary storage device 1505, a media drive device 1506, and an interface 1507. These components are hardware and are connected to each other by a bus 1508.

[0127] The memory 1502 is, for example, a semiconductor memory such as a read-only memory (ROM) or a random access memory (RAM), and stores programs and data used in processing. The memory 1502 may operate as the storage unit 413 in FIG.

[0128] The CPU 1501 (processor) operates as the generating unit 411 in FIG. 4 by executing a program using the memory 1502, for example.

[0129] The input device 1503 is, for example, a keyboard, a pointing device, etc., and is used for inputting instructions or information from a user or operator. The output device 1504 is, for example, a display device, a printer, etc., and is used for sending inquiries or instructions to a user or operator and outputting processing results. The processing results may be the results of parallel distributed processing.

[0130] The auxiliary storage device 1505 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, a tape device, or the like. The auxiliary storage device 1505 may be a hard disk drive or a solid state drive (SSD). The information processing device stores programs and data in the auxiliary storage device 1505 and can use them by loading them into the memory 1502. The auxiliary storage device 1505 may operate as the storage unit 413 in FIG. 4.

[0131] The medium drive device 1506 drives the portable recording medium 1509 and accesses the recorded contents thereof. The portable recording medium 1509 is a memory device, a flexible disk, an optical disk, a magneto-optical disk, etc. The portable recording medium 1509 may be a CD-ROM (Compact Disk Read Only Memory), a DVD (Digital Versatile Disk), a USB (Universal Serial Bus) memory, etc. A user or operator can store programs and data in the portable recording medium 1509 and load them into the memory 1502 for use.

[0132] In this way, the computer-readable recording medium that stores the program and data used in the processing is a physical (non-transitory) recording medium such as the memory 1502, the auxiliary storage device 1505, or the portable recording medium 1509.

[0133] The interface 1507 is a communication circuit connected to the communication network 203 and performs data conversion associated with communication. The interface 1507 operates as the communication unit 412 in Fig. 4. The information processing device can receive programs and data from an external communication network (not shown) via the interface 1507 and load them into the memory 1502 for use.

[0134] 15, some components may be omitted depending on the purpose or conditions of the information processing device. For example, if an interface with a user or operator is not required, the input device 1503 and the output device 1504 may be omitted. If the portable recording medium 1509 is not used, the medium drive device 1506 may be omitted.

[0135] Fig. 16 shows an example of the hardware configuration of a second information processing device used as the server 901 in Fig. 9. The information processing device in Fig. 16 includes a CPU 1601, a memory 1602, and an interface 1603. These components are hardware and are connected to each other via a bus 1604.

[0136] The memory 1602 is, for example, a semiconductor memory such as a ROM or RAM, and stores programs and data used in processing. The memory 1602 operates as the storage unit 413 in FIG.

[0137] The CPU 1601 executes a program using the memory 1602, for example, to perform parallel distributed processing.

[0138] The interface 1603 is a communication circuit that is connected to the communication network 203 and performs data conversion associated with communication. The interface 1603 operates as the communication unit 412 in FIG.

[0139] The configuration of the information processing device 101 in FIG. 1 is merely an example, and some of the components may be omitted or changed depending on the use or conditions of the information processing device 101.

[0140] 2 and 3 are merely examples, and some of the components may be omitted or changed depending on the purpose or conditions of the information processing system. The switch 201-j may be a switch for Ethernet (registered trademark) communication or wireless communication.

[0141] The configurations of the control server 401 in FIG. 4 and the server 901 in FIG. 9 are merely examples, and some of the components may be omitted or changed depending on the application or conditions of the information processing system.

[0142] The configurations of the information processing devices in FIGS. 15 and 16 are merely examples, and some of the components may be omitted or changed depending on the use or conditions of the information processing device.

[0143] The flowchart in FIG. 14 is merely an example, and some of the processing may be omitted or changed depending on the configuration or conditions of the information processing system.

[0144] The primitive polynomial shown in Fig. 5 is just an example, and a different primitive polynomial is used when k is 6 or greater. The bit string set 421 shown in Fig. 6 and Fig. 7 is just an example, and the bit string set 421 changes depending on the value of k. The communication table 422 shown in Fig. 8 is just an example, and a communication table 422 of a different format may also be used.

[0145] 10 to 12 are merely examples, and All-Reduce communication changes depending on the values ​​of k and n. The pseudocode shown in Fig. 13 is merely an example, and the Greedy algorithm may be implemented using other pseudocode.

[0146] Equations (1) to (21) are merely examples, and collective communication may be performed using other equations.

[0147] Although the disclosed embodiments and their advantages have been described in detail, those skilled in the art may make various modifications, additions, and omissions without departing from the scope of the invention as clearly set forth in the claims.

[0148] The following notes are further provided regarding the embodiment described with reference to FIGS. (Appendix 1) Any one of the plurality of information processing devices, the plurality of information processing devices are connected to a first switch among the plurality of switches; Any of the information processing devices includes: a storage unit that stores communication destination information based on a plurality of bit strings relating to communication destinations of collective communication; a communication unit that communicates with an information processing device connected to a second switch among the plurality of switches based on the communication destination information; Equipped with Among the plurality of information processing devices, some information processing devices including any one of the information processing devices participate in the collective communication; The information processing device, wherein the plurality of bit strings are selected from a set of bit strings related to communication destinations of the plurality of information processing devices, based on the number of the part of information processing devices. (Appendix 2) The information processing device described in Appendix 1, characterized in that the communication destination information includes a second bit string indicating the second switch, obtained by exclusive ORing a first bit string indicating the first switch with any one of the multiple bit strings. (Appendix 3) the set of bit strings is a set of bit strings related to a communication destination when the plurality of information processing devices participate in collective communication, each bit string included in the set of bit strings has a length of k bits; k is an integer equal to or greater than 2, When the bit strings included in the bit string set are arranged in a circle, any k consecutive bit strings are linearly independent; 3. The information processing device according to claim 1, wherein when the plurality of bit strings are arranged in a circle, any k consecutive bit strings are linearly independent. (Appendix 4) The number of the plurality of switches is 2 k and the plurality of switches are connected in a full mesh topology, The number of information processing devices connected to each of the plurality of switches is 2 k -1. The information processing device according to claim 3. (Appendix 5) each bit string included in the bit string set corresponds to an element of a Galois field generated from a k-th degree primitive polynomial; 5. The information processing device according to claim 3, wherein each bit string included in the plurality of bit strings corresponds to an element identified based on the number of the part of information processing devices among the plurality of elements of the Galois field. (Appendix 6) 5. The information processing device according to any one of claims 1 to 4, wherein the plurality of bit strings are selected from the set of bit strings by a greedy algorithm using the number of the partial information processing devices. (Appendix 7) Any one of a plurality of information processing devices connected to a first switch among the plurality of switches communicates with an information processing device connected to a second switch among the plurality of switches based on communication destination information based on a plurality of bit strings related to communication destinations of collective communication; Among the plurality of information processing devices, some information processing devices including any one of the information processing devices participate in the collective communication; The information processing method, wherein the plurality of bit strings are selected from a set of bit strings related to communication destinations of the plurality of information processing devices, based on the number of the part of information processing devices. (Appendix 8) The information processing method described in Appendix 7, characterized in that the communication destination information includes a second bit string indicating the second switch, obtained by exclusive ORing a first bit string indicating the first switch with any one of the multiple bit strings. (Appendix 9) the set of bit strings is a set of bit strings related to a communication destination when the plurality of information processing devices participate in collective communication, each bit string included in the set of bit strings has a length of k bits; k is an integer equal to or greater than 2, When the bit strings included in the bit string set are arranged in a circle, any k consecutive bit strings are linearly independent; 9. The information processing method according to claim 7 or 8, wherein when the plurality of bit strings are arranged in a circle, any k consecutive bit strings are linearly independent. (Appendix 10) The number of the plurality of switches is 2 k and the plurality of switches are connected in a full mesh topology, The number of information processing devices connected to each of the plurality of switches is 2 k -1. The information processing method according to claim 9, (Appendix 11) each bit string included in the bit string set corresponds to an element of a Galois field generated from a k-th degree primitive polynomial; 11. The information processing method according to claim 9, wherein each bit string included in the plurality of bit strings corresponds to an element, among a plurality of elements of the Galois field, that is identified based on the number of the part of information processing devices. (Appendix 12) The information processing method according to any one of appendices 7 to 10, wherein the plurality of bit strings are selected from the set of bit strings by a greedy algorithm using the number of the part of information processing devices. [Explanation of symbols]

[0149] 101 Information processing equipment 111, 413 Storage section 112, 412 Communications Department 201-1~201-d Switches 202-1-1 to 202-1-r, 202-2-1 to 202-2-r, 202-d-1 to 202-dr, 901 server 203 Communication Network 401 Control Server 411 Generation part 421 Bit String Set 422 Report Card 1501, 1601 CPU 1502, 1602 memory 1503 Input Device 1504 Output Device 1505 Auxiliary storage device 1506 Media drive unit 1507, 1603 Interface 1508, 1604 bus 1509 Portable recording media

Claims

1. Any one of the plurality of information processing devices, the plurality of information processing devices are connected to a first switch among the plurality of switches; Any of the information processing devices includes: a storage unit that stores communication destination information based on a plurality of bit strings relating to communication destinations of collective communication; a communication unit that communicates with an information processing device connected to a second switch among the plurality of switches based on the communication destination information; Equipped with Among the plurality of information processing devices, some information processing devices including any one of the information processing devices participate in the collective communication; the plurality of bit sequences are selected from a set of bit sequences related to communication destinations of the plurality of information processing devices based on the number of the part of information processing devices; the set of bit strings is a set of bit strings related to a communication destination when the plurality of information processing devices participate in collective communication, each bit string included in the set of bit strings has a length of k bits; k is an integer of 2 or more, When the bit strings included in the bit string set are arranged in a circle, any k consecutive bit strings are linearly independent; When the plurality of bit strings are arranged in a circle, any k consecutive bit strings are linearly independent; each bit string included in the bit string set corresponds to an element of a Galois field generated from a primitive polynomial of degree k; Each bit string included in the plurality of bit strings corresponds to an element, among a plurality of elements of the Galois field, that is identified based on the number of the part of information processing devices.

2. 2. The information processing device according to claim 1, wherein the communication destination information includes a second bit string indicating the second switch, which is obtained by exclusive ORing a first bit string indicating the first switch with any one of the plurality of bit strings.

3. the number of the plurality of switches is 2k, the plurality of switches are connected in a full mesh topology, 2. The information processing device according to claim 1, wherein the number of information processing devices connected to each of the plurality of switches is 2k-1.

4. 4. The information processing device according to claim 1, wherein the plurality of bit strings are selected from the set of bit strings by a greedy algorithm using the number of the partial information processing devices.

5. Any one of a plurality of information processing devices connected to a first switch among the plurality of switches communicates with an information processing device connected to a second switch among the plurality of switches based on communication destination information based on a plurality of bit strings related to communication destinations of collective communication; Among the plurality of information processing devices, some information processing devices including any one of the information processing devices participate in the collective communication; the plurality of bit sequences are selected from a set of bit sequences related to communication destinations of the plurality of information processing devices based on the number of the part of information processing devices; the set of bit strings is a set of bit strings related to a communication destination when the plurality of information processing devices participate in collective communication, each bit string included in the set of bit strings has a length of k bits; k is an integer of 2 or more, When the bit strings included in the bit string set are arranged in a circle, any k consecutive bit strings are linearly independent; When the plurality of bit strings are arranged in a circle, any k consecutive bit strings are linearly independent; each bit string included in the bit string set corresponds to an element of a Galois field generated from a primitive polynomial of degree k; an information processing method, wherein each bit string included in the plurality of bit strings corresponds to an element, among a plurality of elements of the Galois field, that is identified based on the number of the part of information processing devices.

Images