A communication policy configuration method, apparatus and related device

Abstract

The application provides a communication strategy configuration method and device and related equipment. A management node determines a grouping manner of a plurality of computing nodes distributed in at least two data centers, and determines a plurality of communication strategies corresponding to the grouping manner. Each communication strategy includes a communication algorithm of the computing nodes in a same communication group and an inter-group communication algorithm between the computing nodes of different communication groups. Each communication strategy is used to indicate a method for the plurality of computing nodes to implement a communication operator. The management node determines a time delay corresponding to each communication strategy according to each communication strategy and the communication operator, and then determines a target grouping manner and a target communication strategy of the plurality of computing nodes according to the time delay corresponding to each communication strategy. Different communication strategies result in different communication data amounts in a same data center and different communication data amounts between different data centers, thereby resulting in different time delays. Through the above method, an efficient method for cross-data-center communication can be determined.

Description

Technical Field

[0001] This application relates to the field of data center technology, and in particular to a communication strategy configuration method, apparatus and related equipment. Background Technology

[0002] Large-scale computing in data centers requires a large number of computing nodes. For example, in the field of Artificial Intelligence (AI), the number of model parameters is growing rapidly, and the demand for computing power for model training or inference is increasing dramatically. However, due to limitations such as power resources, the computing power of a single data center (DC) can no longer meet the computing needs. Currently, the industry uses the resources of multiple data centers to jointly train or infer models. However, the distance between different data centers can be hundreds or even thousands of kilometers. The latency of a single long-distance data transmission across data centers is relatively long, and multiple long-distance data transmissions across data centers occur during a single training or inference process, further increasing the latency. Data transmission across data centers is essentially communication between computing nodes in different data centers. Communication between different computing nodes can be achieved through various communication algorithms, each with different latency. Therefore, determining an efficient communication algorithm is of great significance. Summary of the Invention

[0003] This application provides a communication strategy configuration method, apparatus, and related equipment, which can improve the efficiency of cross-data center collaborative computing.

[0004] In a first aspect, this application provides a communication strategy configuration method, comprising: a management node determining a grouping method for multiple computing nodes in a computing system, wherein the grouping method is used to divide the multiple computing nodes into at least one communication group; then, the management node determining multiple communication strategies corresponding to the grouping method, wherein each of the multiple communication strategies includes an intra-group communication algorithm between multiple computing nodes within the same communication group and an inter-group communication algorithm between computing nodes in different communication groups; each communication strategy is used to instruct the multiple computing nodes to implement a method of communication operators; the management node determines the latency corresponding to each communication strategy based on each communication strategy and the aforementioned communication operators, and determines the target grouping method and target communication strategy for the multiple computing nodes based on the latency corresponding to each communication strategy. The multiple computing nodes are distributed across at least two data centers, with computing nodes located within the same data center connected via a first communication network, and different data centers connected via a second communication network.

[0005] For multiple computing nodes executing communication operators, these nodes can be divided into one or more communication groups. Different communication algorithms can be selected for communication between computing nodes within the same group and between computing nodes in different groups. By combining inter-group and intra-group communication algorithms, various communication strategies can be obtained. Different communication strategies result in different amounts of communication data within the same data center and different amounts of communication data between different data centers, leading to varying latency. By grouping multiple computing nodes and determining multiple communication strategies under this grouping method, and then determining the latency of each communication strategy based on the communication operator to be executed and the different communication strategies, a target grouping method and communication strategy with lower latency can be selected for the multiple computing nodes to implement the corresponding communication operator. Therefore, this method can determine an efficient method for cross-data center data communication.

[0006] The aforementioned communication operators include reduce-scatter communication operators, all-gather communication operators, or all-reduce communication operators; the aforementioned intra-group communication algorithms include any one of the following: ring algorithm, Halving-Doubling (HD) algorithm, recursive HD algorithm, or nonuniform hierarchical ring (NHR) algorithm; the aforementioned inter-group communication algorithms include any one of the following: ring algorithm, HD algorithm, recursive HD algorithm, or NHR algorithm.

[0007] In one possible implementation, when the management node determines the latency corresponding to each communication strategy based on each communication strategy and communication operator, the management node determines the latency corresponding to each communication strategy based on the communication data volume, communication operator, each communication strategy, and bandwidth information; the bandwidth information includes the bandwidth of the first communication network and the bandwidth of the second communication network.

[0008] Given a fixed amount of data to be sent by each computing node, the different communication operators to be implemented and the different communication strategies for implementing the communication operators will lead to different amounts of communication data within the same data center and across data centers. Therefore, when determining the latency corresponding to each strategy, the latency of implementing the communication operator for each strategy can be accurately obtained based on the amount of communication data, the communication operator, each communication strategy, and bandwidth information.

[0009] In one possible implementation, the management node determines the latency corresponding to each communication strategy based on the communication data volume, the communication operator, each communication strategy, and bandwidth information, including: determining the communication data volume within the same data center and the cross-data center communication data volume corresponding to each communication strategy based on the communication data volume, the communication operator, and each communication strategy; and determining the latency corresponding to each communication strategy based on the communication data volume within the same data center, the cross-data center communication data volume corresponding to each communication strategy, and bandwidth information.

[0010] Given a fixed grouping method and communication data volume, the communication data volume within the same data center and the cross-data center communication data volume corresponding to each communication strategy can be determined based on the communication operator, intra-group communication algorithm, and inter-group communication algorithm. The intra-group communication latency can be determined based on the communication data volume within the same data center corresponding to each communication strategy and the bandwidth information of the same data center. The inter-group communication latency can be determined based on the cross-data center communication data volume and bandwidth information corresponding to each communication strategy, thereby obtaining the latency corresponding to each communication strategy.

[0011] In one possible implementation, the aforementioned multiple computing nodes correspond to multiple grouping methods, and each of the multiple grouping methods corresponds to multiple communication strategies; the aforementioned multiple grouping methods include grouping multiple computing nodes into one communication group and dividing them into n communication groups within a data center; wherein, dividing them into n communication groups within a data center means dividing the multiple computing nodes located in the same data center into n communication groups, where n is a positive integer and each of the n communication groups includes at least two computing nodes.

[0012] In one possible implementation, the above method further includes: the management node generating multiple tasks for the first computing node according to the target grouping method and the target communication strategy, and distributing the multiple tasks to the first computing node; wherein, each of the multiple tasks is used to indicate the data to be sent by the first computing node and the target computing node of the data, the first computing node and the target computing node belong to multiple computing nodes in the above computing system, the first computing node and the target computing node may belong to the same communication group or different communication groups, the first computing node and the target computing node may belong to the same data center or different data centers.

[0013] For a communication operator, after the management node determines the target grouping method and target communication strategy for implementing the communication operator according to the above method, it can determine the process of implementing the communication operator, including which computing node should send which data to which computing node at each step. The management node generates the task corresponding to each computing node at each step, including which target node to send which data to, and then distributes it to the computing node to implement the above communication operator.

[0014] In one possible implementation, each of the plurality of computing nodes includes a plurality of symbol storage areas, each of which stores a terminator to indicate the end of a data segment. The method further includes: a first computing node performing a first task, writing first data into a first storage area of ​​a target computing node and writing a first terminator indicating the completion of the first data transmission into the first symbol storage area; the first computing node performing a second task, writing second data into a second storage area of ​​the target computing node and writing a second terminator indicating the completion of the second data transmission into the second symbol storage area; the target computing node determining that the first symbol storage area stores the first terminator, determining that it has received the first data, and deleting the first terminator so that the first symbol storage area can store terminators corresponding to other data again.

[0015] If the receiving compute node has only one symbol store, it needs to frequently query this symbol store. Upon confirming that the symbol store contains an end-of-database character, it must delete the symbol store and send an acknowledgment message to the sender to notify it that it has received the complete data before sending subsequent data. Otherwise, the sender's subsequent packets will lack a corresponding storage area for the end-of-database character, leading to reception problems. In cross-datacenter communication, if the sender only sends the next data packet after receiving an acknowledgment message from the receiver, it will result in high latency. By setting multiple symbol stores on the receiving compute node, sufficient symbol store space can be ensured. The sender can send the next data packet even without receiving an acknowledgment message from the receiver, preventing data reception problems caused by insufficient symbol store resources.

[0016] Secondly, this application provides a communication strategy configuration device, comprising: a grouping module, a strategy formulation module, and a strategy selection module. The grouping module is used to determine the grouping method of multiple computing nodes in a computing system. The grouping method is used to divide the multiple computing nodes into at least one communication group. The multiple computing nodes are distributed in at least two data centers. Computing nodes located in the same data center are connected via a first communication network, and different data centers are connected via a second communication network. The strategy formulation module is used to determine multiple communication strategies corresponding to the grouping method. Each communication strategy includes an intra-group communication algorithm between multiple computing nodes within the same communication group and an inter-group communication algorithm between computing nodes in different communication groups. Each communication strategy is used to instruct multiple computing nodes on a method for implementing a communication operator. The strategy selection module is used to determine the latency corresponding to each communication strategy based on each communication strategy and the communication operator. The strategy selection module is also used to determine the target grouping method and target communication strategy for the multiple computing nodes based on the latency corresponding to each communication strategy.

[0017] In one possible implementation, the aforementioned communication operators include reduce-scatter communication operators, allgather communication operators, or allreduce communication operators; the aforementioned intra-group communication algorithms include any one of the ring algorithm, HD algorithm, recursive HD algorithm, or NHR algorithm; and the inter-group communication algorithms include any one of the ring algorithm, HD algorithm, recursive HD algorithm, or NHR algorithm.

[0018] In one possible implementation, the strategy selection module is specifically used to: determine the latency corresponding to each communication strategy based on the amount of communication data, the communication operator, each communication strategy, and bandwidth information; the bandwidth information includes the bandwidth of the first communication network and the bandwidth of the second communication network.

[0019] In one possible implementation, the strategy selection module is specifically used to: determine the communication data volume within the same data center and the cross-data center communication data volume corresponding to each communication strategy based on the communication data volume, communication operator, and each communication strategy; and determine the latency corresponding to each communication strategy based on the communication data volume within the same data center, the cross-data center communication data volume corresponding to each communication strategy, and bandwidth information.

[0020] In one possible implementation, multiple computing nodes correspond to multiple grouping methods, and each of the multiple grouping methods corresponds to multiple communication strategies. The multiple grouping methods include grouping multiple computing nodes into one communication group and dividing them into n communication groups within a data center. Here, dividing them into n communication groups within a data center means dividing the computing nodes located in the same data center into n communication groups, where n is a positive integer and each of the n communication groups includes at least two computing nodes.

[0021] In one possible implementation, the above-mentioned apparatus further includes a task generation module, which generates multiple tasks for the first computing node according to the target grouping method and the target communication strategy, and sends the multiple tasks to the first computing node; wherein, each of the multiple tasks is used to indicate the data sent by the first computing node and the target computing node of the data, and the first computing node and the target computing node belong to the multiple computing nodes.

[0022] Thirdly, this application provides a computing device including a processor and a memory, wherein the processor executes instructions stored in the memory to implement the method as described in the first aspect or any possible implementation thereof.

[0023] Fourthly, this application provides a computer-readable storage medium, characterized in that it includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method described in the first aspect or any possible implementation thereof.

[0024] Fifthly, this application provides a computer program product, which includes a computer program that, when run by a computing device, implements the method as described in the first aspect or any possible implementation of the first aspect.

[0025] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of a ring communication algorithm provided in this application;

[0027] Figure 2 This is a schematic diagram of a recursive binary search and doubling algorithm provided in this application;

[0028] Figure 3 This is a schematic diagram of a computing system provided in this application;

[0029] Figure 4 This is a flowchart illustrating a communication strategy configuration method provided in this application;

[0030] Figure 5 This is a schematic diagram showing the connection of multiple computing nodes in a communication plane provided in this application;

[0031] Figure 6 This is a schematic diagram of a grouping of computing nodes provided in this application;

[0032] Figure 7 This is a schematic diagram of a reduce-scatter communication process provided in this application;

[0033] Figure 8 This is a schematic flowchart of a communication strategy configuration device provided in this application;

[0034] Figure 9 This is a schematic diagram of a computing device provided in this application. Detailed Implementation

[0035] With the rapid increase in the number of AI model parameters, the demand for computing power for model training and inference has grown dramatically. However, due to limitations such as power resources, the computing power of a single data center (DC) is limited. Currently, the industry uses resources from multiple data centers to jointly train large models. Compared with training and inference in a single data center, the performance of model training and inference in a multi-data center is affected by the distance and bandwidth between data centers. The distance between data centers can reach hundreds or even thousands of kilometers. Due to cost design considerations, the network bandwidth between data centers is limited, and the latency of a single long-distance data transmission across data centers (referred to as cross-domain communication) is relatively long. In a single training or inference process, multiple cross-domain communications may need to be performed, further increasing latency and reducing training or inference efficiency.

[0036] For example, see Figure 1 , Figure 1 This is a schematic diagram of a ring communication algorithm provided in this application. Figure 1 In this system, the computing system comprises eight nodes, nodes 0-7. Nodes 0-3 are located in the first data center (DC1), and nodes 4-7 are located in DC2. In the ring communication algorithm, the eight nodes form a ring, and each node has two adjacent nodes. For example, nodes 1 and 7 are two adjacent nodes of node 0, and nodes 2 and 4 are two adjacent nodes of node 3. When using... Figure 1The computing system shown, in implementing communication operators such as reduce-scatter, all-gather, or all-reduce using the ring algorithm, requires each computing node to execute the ring algorithm multiple times. During each execution, each node sends a portion of its data to one adjacent node and receives data from another adjacent node. For example, node 0 sends data to node 1 and simultaneously receives data from node 7; node 3 sends data to node 4 and simultaneously receives data from node 2. Therefore, in implementing these communication operators using the ring algorithm, there are cross-domain communication and intra-domain communication (i.e., communication within the same data center). Since the bandwidth between different computing nodes within the same data center is greater than the bandwidth between different data centers, cross-domain communication is less efficient than intra-domain communication, slowing down intra-domain communication. Furthermore, implementing these communication operators using the ring algorithm requires multiple executions of the ring algorithm, each involving inter-domain communication, which significantly increases latency.

[0037] Furthermore, during the execution of communication operators, there may be situations where multiple compute nodes within one data center simultaneously engage in cross-domain communication with multiple compute nodes within another data center. When the bandwidth between data centers is limited, congestion can occur on these inter-data center links. For example, if using... Figure 1 The computational system shown implements each communication operator using the Halving-Doubling (HD) algorithm. Each computation node in the system needs to execute the HD algorithm three times; that is, each computation node needs three steps to implement each communication operator. For example, see [link to documentation]. Figure 2 , Figure 2 This is a schematic diagram of a recursive binary search and doubling algorithm provided in this application. In the first step of executing the HD algorithm, each computing node in DC1 interacts with one computing node in DC2. For example, node 0 interacts with node 4, and node 1 interacts with node 5. The second and third steps involve data interaction between nodes within the same DC. If the bandwidth between DC1 and DC2 is low, congestion may occur on the link between DC1 and DC2, leading to increased latency.

[0038] To address the aforementioned issues, this application provides a communication strategy configuration method applicable to scenarios where multiple data centers jointly perform model training or inference. For multiple computing nodes executing communication operators across data centers, different grouping methods can be used to divide these nodes into multiple communication groups, such as two computing nodes per group or three computing nodes per group. For intra-group communication algorithms within the same communication group and inter-group communication algorithms between different communication groups, ring communication algorithms or HD communication algorithms can be selected. After determining the grouping method for the multiple computing nodes, different communication strategies can be obtained by combining different inter-group and intra-group communication algorithms. For different communication strategies, the latency corresponding to each strategy can be determined based on information such as communication data volume and bandwidth. Then, the communication strategy with lower latency is selected as the scheme for multiple computing nodes to implement communication operators. By using different grouping methods and different intra-group and inter-group communication algorithms, different communication strategies for implementing communication operators can be obtained, thereby determining a communication strategy with lower latency for multiple computing nodes to implement communication operators. This improves the efficiency of cross-data center communication and enhances the efficiency of multi-data center training or inference.

[0039] The communication strategy configuration method provided in this application is applied to a computing system comprising multiple data centers, wherein each data center includes multiple computing devices, and each computing device includes multiple computing nodes. The network topology of each data center can be any of the following: tree network topology, ring network topology, or star network topology; this application does not specifically limit the network topology of the data centers. See also Figure 3 , Figure 3 This is a schematic diagram of a computing system provided in this application. The computing system includes two data centers (DCs), DC1 and DC2, each DC including six computing devices, and each computing device including at least one computing node. Figure 3 (Not shown in the image). Each computing device is connected to an access layer switch. Figure 3 Each access layer switch connects to three computing devices, and each computing node is connected to an access layer switch via a link. Access layer switches connect to aggregation layer switches, and aggregation layer switches connect to routers. Different data centers are connected via routers and an interconnect network. It should be understood that... Figure 3 The computing system described herein is merely an example. This application does not specify the number of data centers (DCs) included in the computing system, the number of computing devices included in each DC, or the number of computing nodes included in each computing device. It also does not limit the network topology between computing devices and switches, or between switches and other devices.

[0040] The aforementioned computing devices can be servers, and computing nodes can be graphics processing units (GPUs), neural network processing units (NPUs), tensor processing units (TPUs), or other dedicated AI processing chips.

[0041] In this application, different devices within the same data center are connected through a first communication network, wherein the first communication network includes any one or more of the following: links between computing nodes and switches, links between switches, or links between switches and routers; different data centers are connected through a second communication network.

[0042] The following description, in conjunction with the accompanying drawings, introduces the communication policy configuration method provided in this application, which is applicable to, for example... Figure 3 The computing system shown, such as Figure 4 As shown, Figure 4 This is a flowchart illustrating a communication strategy configuration method provided in this application, which includes the following steps S401 to S404.

[0043] S401. The management node determines the grouping method for multiple compute nodes.

[0044] In this application, each computing device has a management node. The management node can be the central processing unit (CPU) in the computing device, or any computing node in the computing device. This application does not make any specific limitation on this.

[0045] The aforementioned multiple computing nodes refer to computing nodes located on the same communication plane within a computing system. Here, the communication plane refers to the computing nodes within the system that need to exchange data during the execution of communication operators. For example, multiple computing nodes on the same communication plane are computing nodes that exchange data with each other during the execution of reduce-scatter, all-gather, or all-reduce communication operators.

[0046] It should be noted that in this application, when a service needs to be executed through the computing system, each management node, upon receiving the service, can convert it into computational and communication operators that can be executed by the computing nodes. For example, if the service is a model training service, the management node uses deep learning frameworks such as MindSpore, PyTorch, or TensorFlow to convert the model training service into computational and communication operators that can be executed by the computing nodes. Then, based on the topology information of the computing system, the management node divides the computing nodes included in the computing system into one or more communication planes. Typically, the same communication plane includes the same number of computing nodes in each data center (DC). Computing nodes belonging to the same communication plane within a DC can be computing nodes located on the same computing device or computing nodes located on different computing devices. For example, Figure 3 In the cluster shown, if each access layer switch connects to 3 computing devices, and each data center includes two access layer switches, then each data center (DC) includes 6 computing devices, numbered 0 to 6. The computing nodes within the same-numbered computing devices in two DCs can be grouped into a single communication plane, resulting in 6 communication planes for the computing system. If each computing device includes 8 computing nodes, then each communication plane includes 16 computing nodes. For example... Figure 3 In the computing system shown, if each communication plane includes one computing node in each computing device, then a communication plane includes 12 computing nodes; if each computing device includes 8 computing nodes, then the computing system includes 8 communication planes.

[0047] The connectivity between multiple computing nodes within each communication plane can be simplified to connections between multiple computing nodes and switching equipment within the same data center, and connections between data centers via switching equipment and communication networks. See also Figure 5 , Figure 5 This is a schematic diagram illustrating the connection of multiple computing nodes in a communication plane provided in this application. If each communication plane includes... Figure 3 Each computing device shown has one computing node. Each communication plane comprises 12 computing nodes across two data centers (DCs). If the six computing nodes in DC1 are A0–A5 and the six computing nodes in DC2 are B1–B5, then the connection relationships between the 12 computing nodes can be simplified as follows: Figure 5 As shown, computing nodes located in the same DC on the same communication plane are connected to a switch, and different DCs establish connections through the switch and communication network.

[0048] In this application, grouping refers to dividing multiple computing nodes within the same communication plane into one or more communication groups, with each group including at least two computing nodes within the same communication plane. Grouping methods include no grouping and dividing the data center (DC) into n communication groups. No grouping means treating multiple computing nodes within a communication plane as a single communication group; dividing the data center into n communication groups means dividing computing nodes located within the same DC within a communication plane into n communication groups, where n is a positive integer and each of the n communication groups includes at least two computing nodes. That is, if the number of computing nodes located within the same DC in a communication plane is m, then n is a divisor of m excluding m, meaning n can take multiple values. For example, if m is 6, then n can take the values ​​1, 2, and 3. Therefore, if dividing the data center into n communication groups includes k grouping methods, then the communication plane can have k+1 grouping methods.

[0049] For example, see Figure 6 , Figure 6 This is a schematic diagram illustrating the grouping of computing nodes provided in this application. (The above...) Figure 5 One of the communication planes consists of 12 communication nodes. When the grouping mode is not grouped, the 12 computing nodes A0 to A5 and B0 to B5 are divided into one communication group. When the grouping method is to divide the DC into n communication groups, since a communication plane includes 6 computing nodes, the value of n includes 1, 2, and 3. If n is 1, then A0 to A5 are divided into one communication group, and B0 to B5 are divided into another communication group. If n is 2, then the computing nodes in a DC are divided into 2 communication groups, and each communication group includes 3 computing nodes in a DC. For example, A0 to A2 are divided into one communication group, A3 to A5 into another, B0 to B2 into another, and B3 to B5 into another. If n is 3, then the computing nodes in a DC are divided into 3 communication groups, and each communication group includes 2 computing nodes in a DC. For example, A0 and A1 are divided into one communication group, and A2 and A3 into another. Therefore, the above communication plane includes 4 grouping methods.

[0050] It should be noted that before determining the various grouping methods corresponding to multiple compute nodes, the management node obtains the cluster information of the computing system. This cluster information includes the topology and bandwidth information of the computing system. The topology information includes the number of computing devices in each data center and the number of compute nodes included in each computing device; the bandwidth information includes the bandwidth of the first communication network and the bandwidth of the second communication network. The bandwidth of the first communication network includes, but is not limited to, one or more of the following: bandwidth between compute nodes and switches, bandwidth between switches, or bandwidth between switches and routers. It should be understood that the bandwidth information is related to the network topology of the cluster. Figure 3 In the network topology shown, the bandwidth of the first communication network includes the bandwidth between computing nodes and switches, the bandwidth between switches, and the bandwidth between switches and routers.

[0051] In one possible implementation, the aforementioned cluster information can be obtained by the computing nodes in the cluster through self-negotiation, for example, by obtaining the network topology of the cluster through a network topology discovery algorithm to obtain the aforementioned topology information; the aforementioned cluster information can also be configured by the user, and this application does not impose specific restrictions.

[0052] S402. The management node determines the various communication strategies corresponding to the grouping method.

[0053] In this application, the communication algorithms used by computing nodes to implement communication operators include, but are not limited to, the ring algorithm, the HD algorithm, recursive HD (RHD), and the nonuniform hierarchical ring (NHR) algorithm. After grouping multiple computing nodes on the same communication plane, the communication between computing nodes during the execution of communication operators includes intra-group communication between computing nodes within the same communication group and inter-group communication between computing nodes in different communication groups. In this application, the intra-group communication algorithm can be any of the above-mentioned communication algorithms, and the inter-group communication algorithm can also be any of the above-mentioned communication algorithms.

[0054] After determining the various grouping methods for multiple computing nodes in the same communication plane, the management node determines various communication strategies for the communication plane based on the aforementioned various communication algorithms and grouping methods. Each communication strategy includes intra-group communication algorithms between computing nodes within the same communication group and inter-group communication algorithms between computing nodes in different communication groups.

[0055] After determining the grouping method used within the same communication plane, since multiple computing nodes within the same communication plane correspond to various grouping methods, there are multiple intra-group communication algorithms and inter-group communication algorithms. Different grouping methods lead to different amounts of communication data within and between groups, different intra-group communication algorithms lead to different amounts of communication within and between groups, and different inter-group communication algorithms also lead to different amounts of communication within and between groups. Therefore, for each of the various grouping methods for multiple computing nodes, various communication strategies can be obtained by permuting and combining the communication algorithms that can be used for intra-group communication and inter-group communication. For example, communication strategies include, but are not limited to (using the ring algorithm for intra-group communication and the ring algorithm for inter-group communication), (using the HD algorithm for intra-group communication and the HD algorithm for inter-group communication), (using the ring algorithm for intra-group communication and the HD algorithm for inter-group communication), and (using the HD algorithm for intra-group communication and the ring algorithm for inter-group communication), etc., which will not be listed here.

[0056] For example, for Figure 6 The four grouping methods shown indicate that, after determining that the communication plane has four grouping methods, if the communication algorithm includes the ring algorithm and the HD algorithm, for the non-grouping method, the communication algorithm between computing nodes can be either the ring algorithm or the HD algorithm. That is, the non-grouping scheme corresponds to two communication strategies. For the DC divided into 1 communication group, 2 communication groups, and 3 communication groups, there are three grouping schemes, each of which corresponds to four communication strategies: using the ring algorithm for both intra-group and inter-group communication, using the HD algorithm for both intra-group and inter-group communication, using the ring algorithm for intra-group communication and the HD algorithm for inter-group communication, and using the HD algorithm for intra-group communication and the ring algorithm for inter-group communication. Therefore, the above three grouping schemes correspond to 12 communication strategies, and the communication plane corresponds to a total of 14 communication strategies.

[0057] S403. The management node determines the cross-domain communication data volume and intra-domain communication data volume corresponding to each of the above multiple communication strategies.

[0058] After determining the various communication strategies corresponding to each grouping method of the aforementioned multiple computing nodes, the management node, for a communication operator to be executed by a computing node, can determine the cross-domain communication data volume and intra-domain communication data volume corresponding to each communication strategy, after determining the grouping method and the various communication strategies corresponding to the grouping method. Here, cross-domain communication data volume refers to the communication data volume across data centers, and intra-domain communication data volume refers to the communication data volume within the same data center. In this application, the amount of data sent by a computing node to other computing nodes within the same data center during the implementation of the communication operator is considered as the intra-domain communication data volume.

[0059] For example, with Figure 5 The example shown illustrates how 12 compute nodes on the same communication plane execute the reduce-scatter communication operator, using a 1 / 2 grouping method within the DC, with both intra-group and inter-group communication algorithms employing the ring algorithm. This example demonstrates how the management node determines the amount of cross-domain and intra-domain communication data.

[0060] When implementing the reduce-scatter communication operator, multiple communication groups first perform reduce-scatter within their respective groups; that is, each communication group consists of multiple computing nodes forming a ring. For example... Figure 7 As shown, Figure 7 This is a schematic diagram of a reduce-scatter communication process provided in this application. Figure 7 Taking three computing nodes, A0, A1, and A2, as an example, since a communication group includes three computing nodes, the data in each computing node is divided into three equal parts: the three parts in A0 are a0, b0, and c0; the three parts in A1 are a1, b1, and c1; and the three parts in A2 are a2, b2, and c2. The data size of each part is N / 3, where N is the amount of data in each computing node. These three computing nodes complete reduce-scatter communication in the following two steps:

[0061] (1) A0 sends a0 to A1, A1 sends b1 to A2, and A2 sends c2 to A0. In this step, each computing node sends N / 3 data and receives N / 3 data. After receiving the data, each computing node will perform addition. That is, A0 will perform addition of c0 and c2 to get c3, A1 will perform addition of a0 and a1 to get a3, and A2 will perform addition of b1 and b2 to get b3. After the addition is performed, the number of data in each computing node is 2N / 3.

[0062] (2) A0 sends c3 to A1, A1 sends a3 to A2, and A2 sends b3 to A0. In this step, the amount of data sent by each computing node is N / 3, and the amount of data received is also N / 3. After receiving the data, each computing node will perform addition. That is, A0 will perform the addition operation of b0 and b3 to get b4, A1 will perform the addition operation of c1 and c3 to get c4, and A2 will perform the addition operation of a2 and a3 to get a4. After the addition operation is completed, the amount of data in each computing node is N / 3.

[0063] During the reduce-scatter process within each communication group, each compute node sends 2N / 3 of the data to and receives 2N / 3 of the data to an adjacent compute node. Since intra-group communication is intra-domain communication, the amount of data sent and received by each compute node during the intra-group reduce-scatter process is the intra-domain communication data volume. After the complete intra-group reduce-scatter, each compute node has N / 3 of the data.

[0064] After reduce-scatter is completed between compute nodes within each communication group, multiple communication groups then perform inter-group reduce-scatter communication. Because inter-group reduce-scatter communication involves data interaction between compute nodes across different data centers (DCs), it is called cross-domain communication. During inter-group reduce-scatter communication, each ring includes one compute node from each communication group. Since the compute nodes within a DC are divided into two communication groups, each group containing three compute nodes, the communication plane comprises four communication groups, one ring contains four compute nodes, and the communication plane consists of three rings. For example... Figure 8 As shown, Figure 8 This is a ring diagram for cross-domain communication provided in this application. A0, A3, B0 and B3 form one ring, A1, A4, B1 and B4 form another ring, and A2, A5, B2 and B5 form yet another ring.

[0065] Since each ring consists of 4 computing nodes, the N / 3 data in each computing node is divided into 4 equal parts, with each part containing N / 12 data. (Refer to the above.) Figure 7 In the communication process, the four compute nodes in each ring need to complete reduce-scatter communication in three steps. That is, all three rings in the communication plane require three steps for inter-group reduce-scatter communication. Each compute node sends N / 12 data per step and receives N / 12 data per step. For example... Figure 8As shown, in each ring, there are two cross-domain communications in each step. For example, in a ring composed of A0, A3, B0, and B3, each step includes A3 sending N / 12 data to B0 and B3 sending N / 12 data to A0. Therefore, the cross-domain communication data volume in each step of each ring is 2N / 12. The communication plane includes 3 rings, so the cross-domain communication data volume in each step of the communication plane is 6N / 12. Since it takes 3 steps to complete the reduce-scatter communication between groups, the cross-domain communication data volume in the process of cross-group reduce-scatter communication is 18N / 12, and the unidirectional cross-domain communication data volume is 9N / 12.

[0066] In the aforementioned inter-group reduce-scatter communication process, one compute node sends data to another compute node within the same data center (DC). Since each compute node sends N / 12 data per step, and the inter-group reduce-scatter communication requires three steps, the total amount of data a compute node sends to another compute node within the same DC during this process is 3N / 12. For example, in DC1, A0 sends data to A3, A1 sends data to A4, and A2 sends data to A5.

[0067] In summary, when implementing the reduce-scatter communication operator, if 1 / 2 grouping within the DC is used, and the ring algorithm is used for both intra-group and inter-group communication, the amount of cross-domain communication data is 9N / 12, and the amount of intra-domain communication data for each computing node is 2N / 3 + 3N / 12 = 11N / 12.

[0068] Based on the above method, the management node can determine the cross-domain communication data volume and the intra-domain communication data volume of each computing node corresponding to each communication strategy. For example, if multiple computing nodes are not grouped, the cross-domain communication data volume is 11N / 12, and the intra-domain communication data volume of each computing node is 11N / 12; if the DC is divided into one communication group, the cross-domain communication data volume is 6N / 12, and the intra-domain communication data volume of each computing node is 11N / 12; if the DC is divided into three communication groups, the cross-domain communication data volume is 10N / 12, and the intra-domain communication data volume of each computing node is 11N / 12.

[0069] S404. The management node determines the latency corresponding to different communication strategies based on the cross-domain communication data volume, the intra-domain communication data volume, and bandwidth information corresponding to each communication strategy.

[0070] After determining the cross-domain and intra-domain communication data volumes of a grouping method and a communication strategy under that grouping method using the above method, the management node, for a combination of a grouping method and a communication strategy under that grouping method, such as a combination of a first grouping method and a first communication strategy, determines the cross-domain communication latency corresponding to the combined communication scheme based on the cross-domain communication data volume corresponding to the first communication strategy and the bandwidth between data centers (DCs). Based on the intra-domain communication data volume of each compute node in the first communication strategy and the bandwidth between the compute node and the switch, the management node determines the intra-domain communication latency of the combined communication scheme. Based on the intra-domain communication latency and the cross-domain communication latency, the latency corresponding to the first grouping method and the first communication strategy is obtained. It should be understood that due to different data center topologies, intra-domain communication latency is related to the data center topology. For example, if all compute nodes in a communication plane within a DC are connected to a single switch, then the intra-domain communication latency is related to the bandwidth between the compute node and the switch. If the compute nodes in a communication plane within a DC are connected to different switches, then the intra-domain communication latency is related to the bandwidth between the compute node and the switch, as well as the bandwidth between the switches.

[0071] The above-mentioned multiple computing nodes can be grouped in various ways, and each grouping method corresponds to multiple communication strategies. Based on the multiple grouping methods and the multiple communication strategies corresponding to each grouping method, multiple combination schemes can be obtained. Each combination scheme includes a grouping method and a communication strategy. The management node can determine the latency corresponding to each combination scheme according to the above method and obtain multiple latency.

[0072] S405. The management node determines the target grouping method and target communication strategy of the above-mentioned multiple computing nodes based on the latency corresponding to each communication strategy.

[0073] Using the above method, the management node can determine the latency corresponding to each combination scheme, and then determine the target grouping method and target communication strategy based on the latency corresponding to multiple combination schemes. For example, if a combination scheme divides the computing nodes in each data center into a group, and the communication strategy is that both the intra-group communication algorithm and the inter-group communication algorithm use the ring algorithm, the latency of implementing the communication operator under this combination scheme is less than the latency corresponding to other combination schemes. The management node determines the target grouping method as dividing the data center into 1 communication group, and the target communication strategy as using the ring algorithm for intra-group communication and the ring algorithm for inter-group communication.

[0074] After determining the target grouping method and target communication strategy, the management node generates a task flow for the first computing node based on the target communication strategy and distributes the task flow to the first computing node. This task flow includes multiple tasks, each of which indicates the data the first computing node needs to send and the target computing node for that data. Both the first computing node and the target computing node belong to the same communication plane, and the management node and the first computing node reside on the same computing device.

[0075] In the process of implementing communication operators on computing nodes, each computing node sends data to another computing node multiple times, and each computing node also receives data sent multiple times by other computing nodes. Each data transmission from each computing node is followed by a terminator to indicate the end of the data transmission. The receiving computing node determines that it has received a data transmission based on the terminator. In one possible implementation, each computing node includes multiple symbol storage areas, each used to store the terminator corresponding to the received data, for example, ... Figure 7 In a communication group, when reduce-scatter communication is implemented, compute node A0 will send data twice to compute node A1, and compute node A1 will send data twice to compute node A2.

[0076] When a compute node sends multiple data sets to another compute node, for example, a first compute node sends multiple data sets to a target compute node, the first compute node includes a first end-of-data marker at the end of the first data set. The first compute node writes the first data set to the target compute node's first storage area and the first end-of-data marker to the target compute node's first symbol storage area. Then, when the first compute node is ready to send the next data set, it sends a second data set to the target compute node. The first compute node includes a second end-of-data marker at the end of the second data set. The first compute node writes the second data set to the target compute node's second storage area and the second end-of-data marker to the target compute node's second symbol storage area. The target compute node periodically polls multiple symbol storage areas. If a symbol storage area contains an end-of-data marker, it means the target compute node has received a data set. The target compute node deletes the end-of-data marker from that symbol storage area so that the first compute node can reuse that symbol storage area to write the end-of-data marker for another data set.

[0077] For the sake of simplicity, the above method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application. Therefore, other reasonable combinations of steps that those skilled in the art can conceive of based on the above description also fall within the scope of protection of this application.

[0078] The above text combined Figures 1 to 7 This application provides a detailed description of the communication policy configuration method. The following section will combine... Figure 8 and Figure 9 The communication strategy configuration device and computing device provided in this application are described separately.

[0079] See Figure 8 , Figure 8 This is a schematic diagram of a communication policy configuration device provided in this application. The device includes a grouping module 810, a policy formulation module 820, and a policy selection module 830. The grouping module 810 determines the grouping method for multiple computing nodes in a computing system. The grouping method is used to divide the multiple computing nodes into at least one communication group; wherein, the grouping method includes dividing the multiple computing nodes into one communication group and dividing them into n communication groups within a data center. The grouping method for computing nodes can be referred to the above. Figure 6 The relevant details will not be repeated here. The aforementioned computing nodes are distributed across at least two data centers. Computing nodes located within the same data center are connected via a first communication network, while nodes in different data centers are connected via a second communication network.

[0080] The strategy formulation module 820 is used to determine multiple communication strategies corresponding to the grouping method. Each of the multiple communication strategies includes intra-group communication algorithms between multiple computing nodes within the same communication group and inter-group communication algorithms between computing nodes in different communication groups. Each communication strategy is used to instruct multiple computing nodes on how to implement communication operators.

[0081] The strategy selection module 830 is used to determine the latency corresponding to each communication strategy based on each communication strategy and communication operator, and to determine the target grouping method and target communication strategy for multiple computing nodes based on the latency corresponding to each communication strategy. The strategy selection module 830 determines the latency corresponding to each communication strategy based on the communication data volume, communication operator, each communication strategy, and bandwidth information; the bandwidth information includes the bandwidth of the first communication network and the bandwidth of the second communication network. Specifically, the strategy selection module 830 determines the communication data volume within the same data center and the communication data volume across data centers corresponding to each communication strategy based on the communication data volume within the same data center and the communication data volume across data centers corresponding to each communication strategy, and determines the latency corresponding to each communication strategy based on the communication data volume within the same data center, the communication data volume across data centers, and bandwidth information.

[0082] In one possible implementation, the communication strategy configuration device further includes a task generation module 840. For a communication operator, after determining the target grouping method and target communication strategy for implementing the communication operator according to the above method, the process of implementing the communication operator can be determined, including which computing node should send which data to which computing node at each step. The task generation module 840 is used to generate multiple tasks for the first computing node according to the target grouping method and target communication strategy, and to distribute the multiple tasks to the first computing node. Each of the multiple tasks is used to indicate the data to be sent by the first computing node and the target computing node of the data. The first computing node and the target computing node belong to multiple computing nodes.

[0083] The specific method for implementing communication policy configuration using the communication policy configuration device can be referred to the above. Figures 1-7 The relevant descriptions in the corresponding method embodiments are not repeated here. It should be understood that... Figure 8 This application merely illustrates one possible division method for the communication strategy configuration device. In practical applications, there may be other ways to divide the communication strategy configuration, and this application does not impose any specific limitations.

[0084] This application also provides a computing device, such as Figure 9 As shown, Figure 9This is a schematic diagram of a computing device provided in this application. The computing device 900 includes a bus 902, a processor 904, a memory 906, a communication interface 908, and computing nodes 910. The processor 904, memory 906, communication interface 908, and computing nodes 910 communicate with each other via the bus 902. It should be understood that this application does not limit the number of processors 904 and computing nodes 910 in the computing device 900. The processor 904 may be a central processing unit (CPU), or may include a CPU and other hardware chips. The hardware chips may be of various types, such as any one of a programmable logic device (PLD), a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), or a digital signal processor (DSP). The computing device 900 may include one or more of the above-mentioned types of hardware chips, or may include multiple types of the above-mentioned hardware chips; this application does not make specific limitations. The compute node 910 can be a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or other dedicated AI processing chips.

[0085] The 902 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 9 The bus 902 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 902 may include a path for transmitting information between various components of the computing device 900 (e.g., memory 906, processor 904, communication interface 908).

[0086] Memory 906 may include volatile memory, such as random access memory (RAM). Memory 906 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0087] The memory 906 stores executable program code, and the processor 904 executes the executable program code to implement... Figures 3 to 7 The method described in the corresponding method embodiment. That is, the memory 906 stores program code for the functions of the aforementioned management node, thereby implementing... Figures 3 to 7 The corresponding method embodiment is the method implemented by the management node. The program code includes one or more software modules, and the one or more software modules include... Figure 8 The grouping module 810, strategy formulation module 820, and strategy selection module 830 shown are executed by the processor 904, which carries executable program code. Figures 3-7 The corresponding communication strategy configuration process will not be elaborated here.

[0088] The communication interface 908 can be a wired or wireless interface, used for communicating with other modules or devices. For example, it can receive the aforementioned traceability request, or receive configuration information input by the user through the configuration interface. The wired interface can be an Ethernet interface, a local interconnect network (LIN), etc., while the wireless interface can be a cellular network interface or a wireless LAN interface.

[0089] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on a computing device, it causes the computing device to perform… Figures 3 to 7 The method described in the corresponding method embodiment.

[0090] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to implement… Figures 3 to 7The methods described in the corresponding method embodiments. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A communication strategy configuration method, characterized in that, The method includes: The management node determines the grouping method of multiple computing nodes in the computing system. The grouping method is used to divide the multiple computing nodes into at least one communication group. The multiple computing nodes are distributed in at least two data centers. The computing nodes located in the same data center are connected through a first communication network, and the computing nodes in different data centers are connected through a second communication network. The management node determines multiple communication strategies corresponding to the grouping method. Each of the multiple communication strategies includes an intra-group communication algorithm between multiple computing nodes within the same communication group and an inter-group communication algorithm between computing nodes in different communication groups. Each communication strategy is used to instruct the multiple computing nodes on how to implement the communication operator. The management node determines the latency corresponding to each communication strategy based on each communication strategy and the communication operator; The management node determines the target grouping method and target communication strategy of the multiple computing nodes based on the latency corresponding to each communication strategy.

2. The method according to claim 1, characterized in that, The communication operators include the reduce-scatter communication operator, the allgather communication operator, or the allreduce communication operator; the intra-group communication algorithm includes any one of the ring algorithm, the binary and doubling HD algorithm, the recursive HD algorithm, or the unbalanced hierarchical ring NHR algorithm; the inter-group communication algorithm includes any one of the ring algorithm, the HD algorithm, the recursive HD algorithm, or the NHR algorithm.

3. The method according to claim 2, characterized in that, The management node determines the latency corresponding to each communication strategy based on each communication strategy and the communication operator, including: The management node determines the latency corresponding to each communication strategy based on the amount of communication data, the communication operator, each communication strategy, and bandwidth information; the bandwidth information includes the bandwidth of the first communication network and the bandwidth of the second communication network.

4. The method according to claim 3, characterized in that, The management node determines the latency corresponding to each communication strategy based on the communication data volume, the communication operator, each communication strategy, and bandwidth information, including: The management node determines the communication data volume within the same data center for each communication strategy and the cross-data center communication data volume for each communication strategy based on the communication data volume, the communication operator, and each communication strategy. The management node determines the latency corresponding to each communication strategy based on the amount of communication data within the same data center corresponding to each communication strategy, the amount of communication data across data centers corresponding to each communication strategy, and the bandwidth information.

5. The method according to any one of claims 1-4, characterized in that, The multiple computing nodes correspond to multiple grouping methods, and each of the multiple grouping methods corresponds to multiple communication strategies. The multiple grouping methods include grouping the multiple computing nodes into one communication group and dividing them into n communication groups within the data center. Here, dividing them into n communication groups within the data center means dividing the multiple computing nodes located in the same data center into n communication groups, where n is a positive integer and each of the n communication groups includes at least two computing nodes.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: The management node generates multiple tasks for the first computing node according to the target grouping method and the target communication strategy, and sends the multiple tasks to the first computing node; wherein, each of the multiple tasks is used to indicate the data sent by the first computing node and the target computing node of the data, and the first computing node and the target computing node belong to the multiple computing nodes.

7. The method according to claim 6, characterized in that, Each of the plurality of computing nodes includes a plurality of symbol storage areas, each of the plurality of symbol storage areas being used to store a terminator, the terminator being used to indicate the end of a segment of data; The method further includes: The first computing node performs a first task, writes the first data into the first storage area of ​​the target computing node, and writes a first end symbol indicating that the first data has been sent into the first symbol storage area. The first computing node performs a second task, writing the second data into the second storage area of ​​the target computing node, and writing a second end symbol indicating that the second data has been sent into the second symbol storage area. The target computing node determines that the first symbol storage area stores the first end symbol, determines that it has received the first data, and deletes the first end symbol so that the first symbol storage area can store the end symbol corresponding to other data again.

8. A communication strategy configuration device, characterized in that, include: A grouping module is used to determine the grouping method of multiple computing nodes in a computing system. The grouping method is used to divide the multiple computing nodes into at least one communication group. The multiple computing nodes are distributed in at least two data centers. Computing nodes located in the same data center are connected through a first communication network, and computing nodes in different data centers are connected through a second communication network. The strategy formulation module is used to determine multiple communication strategies corresponding to the grouping method. Each of the multiple communication strategies includes an intra-group communication algorithm between multiple computing nodes within the same communication group and an inter-group communication algorithm between computing nodes in different communication groups. Each strategy is used to instruct the multiple computing nodes on how to implement the communication operator. The strategy selection module is used to determine the time delay corresponding to each communication strategy based on each communication strategy and the communication operator; The strategy selection module is also used to determine the target grouping method and target communication strategy of the multiple computing nodes based on the latency corresponding to each communication strategy.

9. The apparatus according to claim 7, characterized in that, The communication operators include the reduce-scatter communication operator, the allgather communication operator, or the allreduce communication operator; the intra-group communication algorithm includes any one of the ring algorithm, the binary and doubling HD algorithm, the recursive HD algorithm, or the unbalanced hierarchical ring NHR algorithm; the inter-group communication algorithm includes any one of the ring algorithm, the HD algorithm, the recursive HD algorithm, or the NHR algorithm.

10. The apparatus according to claim 8 or 9, characterized in that, The strategy selection module is specifically used for: Based on the amount of communication data, the communication operator, each communication strategy, and bandwidth information, the latency corresponding to each communication strategy is determined; the bandwidth information includes the bandwidth of the first communication network and the bandwidth of the second communication network.

11. The apparatus according to claim 10, characterized in that, The strategy selection module is specifically used for: Based on the communication data volume, the communication operator, and each communication strategy, determine the communication data volume within the same data center corresponding to each communication strategy and the cross-data center communication data volume corresponding to each communication strategy; The latency corresponding to each communication strategy is determined based on the amount of communication data within the same data center for each communication strategy, the amount of communication data across data centers for each communication strategy, and the bandwidth information.

12. The apparatus according to any one of claims 8-11, characterized in that, The multiple computing nodes correspond to multiple grouping methods, and each of the multiple grouping methods corresponds to multiple communication strategies. The multiple grouping methods include grouping the multiple computing nodes into one communication group and dividing them into n communication groups within the data center. Here, dividing them into n communication groups within the data center means dividing the multiple computing nodes located in the same data center into n communication groups, where n is a positive integer and each of the n communication groups includes at least two computing nodes.

13. The apparatus according to any one of claims 8-12, characterized in that, The device also includes a task generation module; The task generation module is used to generate multiple tasks for the first computing node according to the target grouping method and the target communication strategy, and to send the multiple tasks to the first computing node; wherein, each of the multiple tasks is used to indicate the data sent by the first computing node and the target computing node of the data, and the first computing node and the target computing node belong to the multiple computing nodes.

14. A computing device, characterized in that, It includes a processor and a memory, the processor executing instructions stored in the memory to implement the method as claimed in any one of claims 1 to 7.

15. A computer-readable storage medium, characterized in that, It includes computer program instructions that, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1 to 7.

Images