Task processing methods, devices and related equipment

Abstract

A task processing method, apparatus, and related equipment are disclosed, relating to the field of computing technology. The method involves determining M corresponding to tasks performed by computing units within multiple computing nodes of a computing cluster, where each communication relationship represents a data transmission relationship between some computing units; acquiring the physical topology information of the computing cluster; and for N communication relationships among the M relationships that require data forwarding through the same-hop network device, setting a forwarding path within the same-hop network device for each communication relationship based on the physical topology information of the computing cluster. The same-hop network device forwards data according to the set forwarding path. In this way, the traffic generated by multiple computing units corresponding to the N communication relationships can be distributed across multiple forwarding paths of the network devices, thereby improving the communication efficiency between multiple computing nodes and ultimately improving the overall efficiency of the computing cluster in executing tasks.

Description

Technical Field

[0001] This application relates to the field of computing technology, and in particular to a task processing method, apparatus and related equipment. Background Technology

[0002] With the development of technologies such as artificial intelligence (AI) and high-performance computing (HPC), the scale of computing clusters used to process AI or HPC tasks is constantly expanding. This makes the communication efficiency between multiple computing nodes participating in task processing a key factor determining the overall efficiency of the computing cluster. Generally, the lower the communication efficiency between multiple computing nodes, the lower the efficiency of the computing cluster in processing tasks.

[0003] Currently, a common approach is to use flow-based equal-cost multipath routing (ECMP) to distribute different traffic between multiple computing nodes across multiple different paths for forwarding. Specifically, this involves performing a hash operation on the traffic's five-tuple (source IP address, destination IP address, source port, destination port, and communication protocol) and selecting the appropriate path from multiple options based on the hash value. Since the information in the five-tuples for different traffic types typically differs, the hash values ​​calculated based on these five-tuples will vary, resulting in different paths used for forwarding the traffic. This allows for the distribution of different traffic types across different paths for forwarding.

[0004] However, the method of distributing traffic based on ECMP can easily lead to some paths becoming overloaded while others are idle. This can affect the communication efficiency between multiple computing nodes, resulting in a lower overall efficiency of the computing cluster in processing tasks. Summary of the Invention

[0005] This application provides a task processing method to improve the overall efficiency of a computing cluster in processing tasks. Furthermore, this application also provides a corresponding task processing apparatus, computing device, computer-readable storage medium, and computer program product.

[0006] Firstly, this application provides a task processing method applied to a computing cluster. Multiple computing nodes in the cluster communicate via multiple network devices. The computing nodes can be, for example, servers, and the network devices can be, for example, electrical switches. Multiple computing units within the computing nodes run tasks, such as tasks for training or inference on AI models. The task processing method can be executed by a corresponding task processing device. Specifically, during the execution of the task processing method, the task processing device determines M (M is a positive integer) communication relationships corresponding to the task. Each of these M communication relationships represents the data transmission relationship between some computing units among the multiple computing units when executing the task. Furthermore, the task processing device also acquires the physical topology information of the computing cluster, which can be used to indicate the connection relationships between the multiple network devices and the multiple computing nodes in the computing cluster. Therefore, for N (N is a positive integer) communication relationships that need to forward data through the same hop network device out of M communication relationships, the task processing device sets a forwarding path in the same hop network device for each of the N communication relationships according to the physical topology information of the computing cluster; wherein, when forwarding task data, the same hop network device forwards data according to the forwarding path set for the N communication relationships.

[0007] Since multiple computing units corresponding to N communication relationships all need to forward data through the same hop network device, and the task processing device sets an appropriate forwarding path for each of the N communication relationships, the traffic generated by the multiple computing units corresponding to the N communication relationships during task execution can be distributed across multiple forwarding paths of the network device. This effectively avoids some forwarding paths becoming overloaded while others are idle, achieving traffic load balancing across multiple forwarding paths. When the traffic load on multiple forwarding paths is relatively balanced, the traffic generated by multiple computing nodes during task execution can usually be forwarded more quickly, thereby improving the communication efficiency between multiple computing nodes and thus improving the overall efficiency of the computing cluster in executing the task. Simultaneously, the fact that multiple forwarding paths of the network device can all forward traffic, and that the traffic load on multiple forwarding paths is relatively balanced, effectively improves the efficiency of data forwarding by the network device, that is, improves the overall performance of data forwarding by multiple network devices.

[0008] In one possible implementation, the M communication relationships determined by the task processing device are either M communication domains during task execution or M parallel communication domains (or parallel domains) during task execution. Each parallel communication domain represents the communication relationship between different computing units executing subtasks of a task in parallel across multiple computing nodes. Thus, by distributing the traffic corresponding to multiple communication domains or multiple parallel communication domains to different forwarding paths of the network device, the overall efficiency of multiple computing units within the M communication domains / parallel communication domains in executing tasks can be improved.

[0009] In one possible implementation, the same-hop network device includes a first queue, and the N communication relationships include both first and second communication relationships. When the task processing device sets a forwarding path in the same-hop network device for each of the N communication relationships based on the physical topology information of the computing cluster, specifically, it can map the first queue to both the first and second communication relationships within the same-hop network device. The time periods during which the multiple computing units corresponding to the first communication relationship generate traffic during task execution do not overlap with the time periods during which the multiple computing units corresponding to the second communication relationship generate traffic during task execution. The first queue is used to cache data corresponding to both the first and second communication relationships, and the cached data is forwarded through a forwarding path set for the same-hop network device. Thus, when the network device forwards communication data in the queue, even if the queue is mapped to multiple different communication relationships, the network device can send data corresponding to different communication relationships sequentially at different time periods based on the same forwarding path. This avoids the problem of traffic congestion for multiple communication relationships when a single queue maps to multiple communication relationships.

[0010] In one possible implementation, the same-hop network device includes a second queue and a third queue. Among the N communication relationships, there are third and fourth communication relationships. The same-hop network device forwards data cached in the second queue and data cached in the third queue through the forwarding path where the target port is located. Specifically, when the task processing device sets the forwarding path in the same-hop network device for each of the N communication relationships based on the physical topology information of the computing cluster, it can determine the traffic size corresponding to the third communication relationship and the traffic size corresponding to the fourth communication relationship. Based on the physical topology information of the computing cluster, it maps the second queue to the third communication relationship and the third queue to the fourth communication relationship within the same-hop network device. The second queue is used to cache data corresponding to the third communication relationship, and the third queue is used to cache data corresponding to the fourth communication relationship. Thus, the task processing device can configure a first weight for the traffic of the third communication relationship and a second weight for the traffic of the fourth communication relationship based on the traffic size corresponding to the third and fourth communication relationships. The first and second weights are used to indicate the ratio between the traffic of the third and fourth communication relationships forwarded by the same-hop network device within a certain time period. In this way, when the network device forwards traffic for multiple communication relationships using the same forwarding path, it can forward traffic simultaneously based on the weights corresponding to each communication relationship. This can prevent the network device from forwarding traffic for one communication relationship for a long time, causing the traffic for other communication relationships to remain unforwarded for a long time. This can avoid the problem of traffic from some communication relationships being blocked on the network device.

[0011] In one possible implementation, the tasks performed by the computing cluster include AI model tasks, such as training or inference tasks for the AI ​​model. The task processing device can then adjust the parallelism of the AI ​​model within the computing cluster, or the communication operators executed by multiple computing nodes during the execution of the AI ​​model, based on the forwarding paths set for N communication relationships. In this way, by adjusting the parallelism of the AI ​​model within the cluster, or by adjusting the communication operators to be executed, the task processing device can optimize the deployment of the AI ​​model within the computing cluster. This makes the traffic generated by the computing cluster during tasks more adaptable to the traffic forwarding capabilities of multiple network devices, thereby further improving the efficiency of the computing cluster in executing tasks.

[0012] Secondly, this application provides a task processing apparatus applied to a computing cluster. Multiple computing nodes in the cluster communicate data via multiple network devices. Multiple computing units within the computing nodes run tasks. The task processing apparatus includes: a determining module for determining M communication relationships corresponding to a task, where each of the M communication relationships represents a data transmission relationship between some computing units among the multiple computing units during task execution, and M is a positive integer; an acquiring module for acquiring physical topology information of the computing cluster; and a setting module for setting a forwarding path in the same-hop network device for each of the N communication relationships that need to forward data through the same-hop network device, based on the physical topology information of the computing cluster, where N is a positive integer. The same-hop network device forwards task data according to the forwarding path set for the N communication relationships.

[0013] In one possible implementation, the M communication relationships are either M communication domains during task execution or M parallel communication domains during task execution, where each parallel communication domain is a communication relationship between different computing units when a subtask of a task is executed in parallel across multiple computing nodes.

[0014] In one possible implementation, the same-hop network device includes a first queue, and the N communication relationships include a first communication relationship and a second communication relationship. When the setting module sets a forwarding path in the same-hop network device for each of the N communication relationships according to the physical topology information of the computing cluster, it is specifically used to: map the first queue to the first communication relationship and the second communication relationship respectively in the same-hop network device according to the physical topology information of the computing cluster; the time period during which multiple computing units corresponding to the first communication relationship generate traffic during the execution of the task does not overlap with the time period during which multiple computing units corresponding to the second communication relationship generate traffic during the execution of the task; the first queue is used to cache the data corresponding to the first communication relationship and the data corresponding to the second communication relationship; the data cached in the first queue is forwarded through a forwarding path set for the same-hop network device.

[0015] In one possible implementation, the same-hop network device includes a second queue and a third queue, and the N communication relationships include a third communication relationship and a fourth communication relationship. The same-hop network device forwards the data cached in the second queue and the data cached in the third queue through the forwarding path where the target port is located. When the setting module sets the forwarding path in the same-hop network device for each of the N communication relationships according to the physical topology information of the computing cluster, it is specifically used to: determine the traffic size corresponding to the third communication relationship and the traffic size corresponding to the fourth communication relationship; map the second queue to the third communication relationship and the third queue to the fourth communication relationship in the same-hop network device according to the physical topology information of the computing cluster, the second queue is used to cache the data corresponding to the third communication relationship, the third queue is used to cache the data corresponding to the fourth communication relationship, and the data cached in the second queue and the third queue are forwarded through the forwarding path where the target port is located set for the same-hop network device; configure a first weight for the traffic of the third communication relationship and a second weight for the traffic of the fourth communication relationship according to the traffic size corresponding to the third communication relationship and the traffic size corresponding to the fourth communication relationship, the first weight and the second weight are used to indicate the ratio between the traffic of the third communication relationship and the traffic of the fourth communication relationship forwarded by the same-hop network device in a certain period of time.

[0016] In one possible implementation, the task is a task that includes an artificial intelligence (AI) model, and the task processing device further includes: an adjustment module, used to adjust the parallel mode of the AI ​​model in the computing cluster or the communication operators executed by multiple computing nodes during the execution of the AI ​​model, based on the forwarding paths set for N communication relationships.

[0017] The task processing apparatus provided in the second aspect corresponds to the task processing method provided in the first aspect. Therefore, the technical effects of any implementation in the second aspect can be found in the description of the relevant technical effects of the corresponding implementation in the first aspect.

[0018] Thirdly, this application provides an AI model deployment method. The method is applied to a computing cluster, in which different computing nodes communicate with each other through multiple network devices. The multiple computing nodes are used to run AI models. The method includes: acquiring physical topology information of the computing cluster; determining multiple forwarding paths that can be provided by the multiple network devices based on the physical topology information of the computing cluster, wherein each forwarding path is used to forward a portion of the traffic generated by the multiple computing nodes during the running of the AI ​​model; and generating a deployment strategy for the AI ​​model based on the multiple forwarding paths, wherein the deployment strategy is used to deploy the AI ​​model on the multiple computing nodes in the computing cluster.

[0019] Fourthly, this application provides an AI model deployment device applied to a computing cluster. Different computing nodes in the computing cluster communicate with each other via multiple network devices. These multiple computing nodes are used to run AI models. The device includes: an acquisition module for acquiring physical topology information of the computing cluster; a determination module for determining multiple forwarding paths that can be provided by the multiple network devices based on the physical topology information of the computing cluster, each of the multiple forwarding paths being used to forward a portion of the traffic generated during the running of the AI ​​model by the multiple computing nodes; and a generation module for generating a deployment strategy for the AI ​​model based on the multiple forwarding paths, the deployment strategy being used to deploy the AI ​​model on the multiple computing nodes in the computing cluster.

[0020] Fourthly, this application provides a computing device, which includes a processor and a memory. The processor and the memory communicate with each other. The processor executes instructions stored in the memory to cause the computing device to perform a task processing method as described in the first aspect or any implementation thereof. It should be noted that the memory may be integrated into the processor or may be independent of the processor. The computing device may also include a bus. The processor is connected to the memory via the bus. The memory may include readable storage and random access memory.

[0021] Fifthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computing device, cause the computing device to perform the operational steps of the task processing method described in the first aspect or any implementation thereof.

[0022] In a sixth aspect, this application provides a computer program product containing instructions that, when run on a computing device, causes the computing device to perform the operational steps of the task processing method described in the first aspect or any implementation thereof.

[0023] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0024] Figure 1 A schematic diagram of the structure of an exemplary computing cluster provided in this application

[0025] Figure 2 A flowchart illustrating a task processing method provided in this application;

[0026] Figure 3 This is a schematic diagram illustrating the process of distributing traffic generated by the computing cluster 10 to multiple forwarding paths, as provided in this application.

[0027] Figure 4 This is a schematic diagram illustrating the aggregation of information reported by each computing unit to obtain the final result.

[0028] Figure 5 This is a schematic diagram illustrating how traffic can be distributed to different forwarding paths by associating parallel communication domains with queues.

[0029] Figure 6 A flowchart illustrating an AI model deployment method provided in this application;

[0030] Figure 7 A diagram illustrating the deployment of AI models using pipeline parallelism, data parallelism, and tensor parallelism simultaneously;

[0031] Figure 8 A schematic diagram of the structure of a task processing device provided in this application;

[0032] Figure 9 This is a schematic diagram of the hardware structure of a computing device provided in this application. Detailed Implementation

[0033] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a method of distinction used in describing objects with the same attributes in the embodiments of this application.

[0034] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0035] See Figure 1 This is a schematic diagram of the structure of an exemplary computing cluster 10. Figure 1 As shown, the computing cluster 10 includes multiple computing nodes, multiple network devices, and at least one service node. Figure 1 The following example illustrates the use of a computing cluster 10, which includes 4 computing nodes (compute nodes 1 to 4), 6 network devices (network devices 1 to 6), and 200 service nodes.

[0036] In computing cluster 10, a computing node refers to a node with data computing capabilities, such as a server or other computing device. Each computing node may include one or more computing units. Figure 2This example uses a compute node comprising four compute units, such as compute node 1 including compute units 1 through 4. In practical applications, different compute units deployed within the same compute node can communicate via a bus, such as a compute express link (CXL) bus.

[0037] The computing units in a computing node can be implemented using a processor. For example, a processor can be any type of processor or any combination thereof, such as a central processing unit (CPU), an accelerator, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system-on-chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, or a data processing unit (DPU). An accelerator can be, for example, a graphics processing unit (GPU), a neural network processing unit (NPU), or a tensor processing unit (TPU).

[0038] The computing cluster 10 can run at least one service, such as an AI task or a high-performance computing (HPC) task. For example, multiple computing nodes in the computing cluster 10 can be used to train an AI model, or multiple computing nodes in the computing cluster 10 can be used to run an AI model and support the AI ​​model inference, such as supporting large language model (LLM) inference to output human-computer dialogue text. For example, the AI ​​model can be an LLM model, a large language model meta AI (LLaMA) model, a bidirectional encoder representations from transformers (BERT) model, or a generative pre-trained Transformer 3 (GPT-3) model, or other types of models, such as GPT-4, etc., without limitation.

[0039] Multiple network devices in computing cluster 10 enable communication between different computing nodes within the cluster. For example, computing node 1 in cluster 10 can send data to computing node 3 via network device 1, network device 5, and network device 3. Exemplarily, network devices can be, for example, electrical switches. An electrical switch can be a switch that uses electrical signals for data exchange, and these signals can be transmitted via twisted-pair or coaxial cables. Alternatively, network devices can be other devices with data forwarding capabilities; this is not limited. In practical applications, multiple network devices can be deployed in multiple layers. For example, taking a two-layer deployment as an example... Figure 1 As shown, some network devices can act as leaf nodes, while others can act as spine nodes. Furthermore, different leaf nodes can interact with each other through spine nodes.

[0040] Service node 200 can be implemented using a processor, or using a computing device that includes a processor. For example... Figure 1As shown, service node 200 may include control device 201 and service device 202. Control device 201 is used to control and manage multiple network devices, such as controlling the paths available for forwarding traffic between different network devices and configuring routing tables in the network devices. Service device 202 can be used to provide control device 201 with communication relationships between different computing nodes during the execution of tasks in computing cluster 10; or, when the task executed by computing cluster 10 specifically includes an AI model, service device 202 can be used to provide computing cluster 10 with deployment strategies for that AI model to guide its deployment within computing cluster 10. In practical applications, service node 200 can be a node in the management network (used to manage network devices) corresponding to the network device, or it can be a node independent of the management network; there is no limitation on this.

[0041] Both the control device 201 and the service device 202 can be implemented in software or hardware. Taking the service device 202 as an example, when implemented in software, the service device 202 can be a process running on the service node 200. When implemented in hardware, the service device 202 can be implemented using a processor in the service node 200, such as any processor or any combination thereof, including CPU, ASIC, PLD, CPLD, FPGA, GAL, SoC, SDI chip, AI chip, and DPU (Data Processing Unit).

[0042] During the processing of tasks using multiple computing nodes in computing cluster 10 (such as training / inference tasks for AI models or HPC tasks), communication traffic typically occurs between different computing nodes. For example, during the backpropagation phase of a model training scenario, different computing nodes can exchange their calculated gradient parameters so that each computing node can update the parameters in the AI ​​model based on the gradient data calculated by multiple computing nodes. At this time, when multiple network devices forward traffic to these different computing nodes, if the forwarding path is selected by hashing the five-tuple information of the traffic, the same hash value may be calculated based on the five-tuple information of different traffic, resulting in the same path being used to forward different traffic. This can easily lead to excessive traffic load on that path, causing congestion in the data communication traffic between different computing nodes on that forwarding path. Consequently, the communication efficiency between different computing nodes is reduced due to traffic congestion on the forwarding path, thus reducing the overall efficiency of multiple computing nodes in processing tasks (i.e., reducing the overall efficiency of computing cluster 10 in processing the task). For example, the overall training efficiency of processing the task may be lower due to the longer communication time between different computing nodes. While some paths experience traffic overload, other paths may become idle due to a lack of forwarding traffic, resulting in low overall data forwarding performance across multiple network devices.

[0043] Based on this, Figure 1 In the computing cluster 10 shown, the service node 200 can distribute the traffic generated during the AI ​​business operation of the computing cluster 10 to different forwarding paths through computing network collaboration.

[0044] Specifically, during the task processing of the computing cluster 10, the service device 202 can determine M (M is a positive integer) communication relationships corresponding to the task. Each of these M communication relationships represents the data transmission relationship between some computing units among the multiple computing units participating in the task. For example, each communication relationship can be a communication domain or a parallel domain, thus, multiple computing units belonging to the same communication domain / parallel domain have a need to communicate with each other. Then, the service device 202 can provide these multiple domains to the control device 201. At the same time, the control device 201 can also obtain the physical topology information of the computing cluster 10, which can indicate the connection relationships between multiple computing nodes and multiple network devices. For example, the control device 201 can generate the physical topology information of the computing cluster 10 based on the connection relationships between different network devices and the connection relationships between network devices and computing nodes. Therefore, for N (N is a positive integer less than M) communication relationships that need to forward data through the same hop network device out of M communication relationships, the control device 201 can set a forwarding path in the same hop network device for each of the N communication relationships according to the physical topology information of the computing cluster 10. For example, it can set forwarding path 1 for some of the N communication relationships and forwarding path 2 for another part of the communication relationships, so that the network device can use forwarding path 1 to forward the data corresponding to some communication relationships and forwarding path 2 to forward the data corresponding to the other part of the communication relationships. In this way, when the same hop network device forwards the data of the task (that is, the data corresponding to the N communication relationships), it can forward the data according to the forwarding path set for the N communication relationships.

[0045] Since multiple computing units corresponding to N communication relationships all need to forward data through the same hop network device, and the service node 200 sets an appropriate forwarding path for each of the N communication relationships, the traffic generated by the multiple computing units corresponding to the N communication relationships during task execution can be distributed across multiple forwarding paths of the network device. This effectively avoids traffic overload on some forwarding paths of the network device while other forwarding paths are idle, achieving traffic load balancing across multiple forwarding paths. When the traffic load on multiple forwarding paths is relatively balanced (compared to situations with large load differences), the traffic generated during task execution by the computing cluster 10 can usually be forwarded more quickly (without excessive communication time between some computing nodes in the computing cluster 10 due to traffic congestion on the forwarding paths), thereby improving the communication efficiency between multiple computing nodes and thus improving the overall efficiency of the computing cluster 10 in executing the task. Simultaneously, the fact that multiple forwarding paths of the network device can forward traffic, and that the traffic load on multiple forwarding paths is relatively balanced, effectively improves the efficiency of data forwarding by the network device, that is, improves the overall performance of data forwarding by multiple network devices.

[0046] It is worth noting that the above Figure 1 The computing cluster 10 shown is merely an illustrative example and is not intended to limit the scope of the application. For instance, in other possible implementations, the computing cluster 10 may include more types of nodes, such as scheduling nodes and computing power service nodes. The scheduling node can be used to schedule AI services to one or more computing nodes in the computing cluster 10 for execution. The computing power service node can be used to manage the computing power in the computing cluster 10, such as by putting some computing nodes in the computing cluster 10 into hibernation to reduce power consumption. Furthermore, in other possible computing clusters, the control device 201 and the service device 202 can be deployed independently within the computing cluster; for example, the control device 201 and the service device 202 can be implemented through separate servers.

[0047] For ease of understanding, embodiments of the task processing method provided in this application will be described below with reference to the accompanying drawings.

[0048] See Figure 2 , Figure 2 This is a flowchart illustrating a task processing method provided in an embodiment of this application. This method can be applied to... Figure 2 The computing cluster 10 described above can also be applied to other suitable computing clusters. For ease of explanation, this embodiment uses an application... Figure 2 The example shown is a computing cluster 10, and multiple network devices are adjusted based on information from the computing cluster 10 when performing AI services.

[0049] in, Figure 2 The task processing method shown may specifically include the following steps.

[0050] S201: Service node 200 determines M communication relationships corresponding to the tasks executed by multiple computing nodes in computing cluster 10. Each of the M communication relationships represents the data transmission relationship between some computing units in the multiple computing nodes when executing the task.

[0051] That is, each communication relationship can be used to indicate which computing units among the multiple computing units participating in the execution of the task need to interact with each other and which computing units do not need to interact with each other.

[0052] For example, each communication relationship can specifically be a communication domain or a parallel communication domain during task execution. A communication domain refers to a set of multiple computing units in the computing cluster 10 that participate in executing the AI ​​task, and this set defines the communication relationships between these computing units. For instance, these multiple computing units can communicate data according to the rules corresponding to one or more communication operators. In practical applications, a communication domain can include identifiers of multiple computing units to indicate which computing units belong to that communication domain. Thus, by obtaining information such as the communication domain for the AI ​​task, the service node 200 can determine which computing units in the computing cluster 10 need to communicate. Furthermore, a communication domain can also include other types of information, such as communication domain identifiers. For example, during task processing, multiple computing units in the computing cluster 10 may be divided into different communication domains, such as different computing units executing multiple different communication operators corresponding to the task. Therefore, the service node 200 can distinguish between different communication domains based on their identifiers. In practical applications, the communication domain can also include information such as the weight of communication data between different computing nodes (this weight can be used to indicate the proportion of data in multiple communication domains when network devices forward data in parallel), and there are no restrictions on this.

[0053] Communication operators refer to the operators executed to achieve data exchange (or data synchronization) between different computing units. Examples include broadcast and all-reduce operators. Thus, by obtaining the communication operators executed by the computing units, the service node 200 can determine the data interaction method between different computing units. For instance, when multiple computing units in a communication domain execute a broadcast operator, data from one computing unit is sent to each of the other computing units in that communication domain, resulting in unidirectional traffic from one computing unit to multiple computing units in that communication domain. Conversely, when multiple computing units in a communication domain execute an all-reduce operator, one computing unit retrieves data from the other computing units in that communication domain, reduces the data from all the computing units in that communication domain, and then sends the reduced result to each of the other computing units in that communication domain, resulting in bidirectional traffic between one computing unit and multiple computing units.

[0054] A parallel communication domain (also called a parallel domain) refers to the communication relationship between different computing units in multiple computing nodes participating in the execution of a task, when the subtasks of that task are executed in parallel. Taking the task of training an AI model as an example, different network layers in the AI ​​model can be deployed on different computing nodes, allowing these multiple computing nodes to execute the training process for different network layers in parallel (each subtask is a training task for one network layer of the AI ​​model). Typically, a parallel communication domain can include one or more communication domains.

[0055] Taking distributed training of an AI model as an example, multiple computing nodes in computing cluster 10 can simultaneously deploy the same network layer of the AI ​​model. During the training process, these multiple computing nodes can train the network layer separately using different parts of the same dataset. In each training iteration, these multiple computing nodes can execute the same communication operator, allowing them to send their calculated gradients to each other. Each computing node can then aggregate the gradients calculated by the multiple computing nodes, such as calculating the average of the gradients. Based on the aggregated gradients, each computing node can update the parameters in the network layer. This model training method can be called data parallelism. In this case, multiple computing nodes deploying the same network layer and executing the same communication operator constitute a communication domain. For multiple network layers in the AI ​​model, they can be deployed and trained in parallel according to the above method. Thus, multiple communication domains trained using this data parallelism method can constitute a parallel communication domain.

[0056] The following describes a non-limited implementation method for determining M communication relationships for service node 200.

[0057] In a first possible implementation, since multiple computing nodes in the computing cluster 10 collaboratively execute the same task, each computing unit within a computing node can be responsible for executing a subtask of that task. Thus, multiple computing units in the computing cluster 10 participating in the execution of one or more tasks can each report initial communication information for their respective subtasks. This initial communication information can be, for example, the communication domain or parallel communication domain to which each computing unit belongs, or it can report both the communication domain and parallel communication domain information of the computing node. Therefore, the service device 202 in the service node 200 can aggregate the initial communication information reported by each computing unit to obtain M communication relationships corresponding to the task (i.e., the global communication domain and / or parallel communication domain information when the computing cluster 10 executes the task). Furthermore, the initial communication information reported by each computing unit can include not only the communication domain and / or parallel communication domain, but also the communication operators to be executed by the computing unit, and the amount of communication data when the computing unit interacts with other computing units within the communication domain / parallel communication domain, etc., without limitation.

[0058] For example, such as Figure 3 As shown, a computing node can deploy multiple computing units. The computing node itself can be a server (such as an Atlas server), and the computing units within it can be accelerators (such as GPUs). In this case, one or more containers can run within the computing node. Taking running a single container as an example, this container can run an acceleration component, such as a component named "ascend speed." This acceleration component can collect the communication domain, parallel communication domain, required communication operators, and the amount of communication data to be exchanged for each computing node. Each computing unit can report this information to the acceleration component, which then provides this information to a proxy component in the server (such as a proxy component named "NetMind Agent"). In practical applications, such as... Figure 3 As shown, the acceleration component can be configured with runtime tools, such as a runtime tool named "NetMind RumTime". Furthermore, the acceleration component can use this runtime tool to provide the agent components in the computing nodes with information such as the communication domain, parallel communication domain, communication operators, and communication data volume of each computing node.

[0059] Then, the proxy component can aggregate the information reported by each computing unit and forward the aggregated information to the service device 202. For example, for the eight computing units in the computing cluster 10, the information reported by each computing unit includes the identifier of the communication domain, the identifier of the parallel communication domain, the identifier of the computing unit, the identifier of the computing node (such as a server) to which the computing unit belongs, the identifier of the communication operator to be executed (such as the name of the communication operator), and the amount of communication data to be exchanged, such as... Figure 4 As shown, the proxy component can aggregate the information reported by each computing unit to obtain an aggregation result. Thus, the service device 202 can receive the aggregation results provided by the proxy components in each computing node and further perform aggregation operations on the aggregation results provided by each proxy component to obtain the final aggregation result, which can be the aforementioned M communication relationships. For example, the service device 202 may include a service process, which may be, for example, a process supporting the NetMind platform, and this service process can be responsible for performing further aggregation operations on the aggregation results provided by the proxy components in each computing node. The implementation method of the service device 202 aggregating the aggregation results provided by each proxy component is related to the aggregation... Figure 4 The information reported by the multiple computing units shown is implemented in a similar way, so it will not be described in detail here.

[0060] In a second possible implementation, the computing cluster 10 may further include a task scheduling node ( Figure 1 (Not shown in the image), during the process of scheduling a task to multiple computing units in the computing cluster 10, the task scheduling node can parse the task to determine the subtasks that different computing units participating in the task execution need to execute. It can also locally store information such as the communication domain, parallel communication domain, communication operators executed, and the amount of data required for communication for each computing unit executing its subtasks. Therefore, when the service device 202 needs to obtain M communication relationships, it can access them from the task scheduling node, for example, by accessing the task scheduling node's local hard drive.

[0061] It is understood that the above-mentioned implementation method of the service device 202 to obtain the M communication relationships corresponding to the tasks in the computing cluster 10 is only an example. In actual application, the service device 202 can also obtain the M communication relationships based on other methods, and there is no limitation on this.

[0062] S202: Service node 200 obtains the physical topology information of computing cluster 10.

[0063] In this embodiment, the following two non-limiting implementation examples for obtaining the physical topology information of computing cluster 10 are provided.

[0064] In the first implementation, the control device 201 in the service node 200 is responsible for monitoring and managing multiple network devices. Therefore, the control device 201 can query and obtain information such as the connection relationships between different network devices, the connection relationships between network devices and computing nodes, and queues within the network devices, and use this information to generate the physical topology information of the computing cluster 10, such as... Figure 3 As shown. For example, control device 201 can query the routing tables and configuration information of each network device, and determine whether there is a connection between different network devices, whether there is a connection between the network device and each computing node, and determine the attribute information of objects such as physical links and queues used to establish connections between multiple network devices and multiple computing nodes based on the routing tables between the network devices. Thus, control device 201 can generate the topology of multiple network devices based on the connection relationships between different network devices, the connection relationships between network devices and computing nodes, and information such as queues in the network devices.

[0065] In the second implementation, the control device 201 can maintain the physical topology between multiple network devices and multiple computing nodes. For example, when configuring routing tables for each network device based on the physical topology, the control device 201 can locally store the physical topology between the multiple network devices and multiple computing nodes, or it can save the physical topology locally during the networking process between the multiple network devices and multiple computing nodes. Therefore, the control device 201 can retrieve the physical topology from the locally stored information.

[0066] It is understood that the above implementation is only an example. In other embodiments, the control device 201 may also obtain the physical topology information of the computing cluster 10 in other ways, and there is no limitation on this.

[0067] In the first implementation example, the physical topology information acquired by the control device 201 can be the global physical topology information of the computing cluster 10. That is, the physical topology information can indicate the connection relationship between all network devices and all computing nodes in the computing cluster 10, so that a path for forwarding the traffic generated when the computing cluster 10 runs the task can be selected from all paths formed by all network devices.

[0068] In the second implementation example, since in actual application scenarios, not all network devices in computing cluster 10 may be able to be used to forward traffic for the same task, for example, different tenants may require isolation of the resources (i.e., network devices) used by multiple network devices to forward traffic belonging to that tenant.

[0069] Based on this, the physical topology information acquired by the control device 201 can be a partial physical topology information of the computing cluster 10. For example, the control device 201 can determine the allocated forwarding paths in the computing cluster 10, such as by accessing locally stored resource allocation records. These forwarding paths are used to forward traffic from other tenants. Therefore, the control device 201 can determine the physical topology of the computing cluster 10 that does not include the allocated forwarding path, based on the global physical topology of the computing cluster 10 and the allocated forwarding paths. In other words, it can delete the physical topology corresponding to the allocated forwarding path from the global physical topology, so that the forwarding paths in the remaining physical topology can be used to forward the traffic of the task currently being processed by the computing cluster 10.

[0070] S203: For N communication relationships among M communication relationships that need to forward data through the same hop network device, the service node 200 sets a forwarding path in the same hop network device for each of the N communication relationships according to the physical topology information of the computing cluster 10. When forwarding the data of the task, the same hop network device forwards the data according to the forwarding path set for the N communication relationships.

[0071] In this system, service node 200 can configure multiple forwarding paths for N communication relationships, with one forwarding path configured for each communication relationship. In practical applications, when the number of communication relationships is less than or equal to the number of forwarding paths, service node 200 can configure one forwarding path for each communication relationship, and different forwarding paths can be configured for different communication relationships. When the number of communication relationships exceeds the number of forwarding paths, service node 200 can configure the same forwarding path for some communication relationships and a different forwarding path for others, with each forwarding path corresponding to one or more communication relationships.

[0072] In this embodiment, the service device 202 can determine the communication requirements between different computing units participating in the execution of the task and the paths for forwarding data supported by multiple network devices based on M communication relationships and the physical topology information of the computing cluster 10. Thus, the service device 202 can distribute the traffic between different computing units to different forwarding paths.

[0073] In one possible implementation, the control device 201 can determine queues among multiple network devices in the computing cluster 10 based on the physical topology information of the computing cluster 10. Then, the control device 201 can determine N communication relationships among the M communication relationships that require data forwarding through the same hop network device, based on the M communication relationships and the physical topology information. For example, such as... Figure 1As shown, the traffic generated by multiple computing units in computing node 1 (which needs to be forwarded to computing units in other computing nodes) all need to be forwarded through network device 1. Thus, one or more communication relationships to which multiple computing units in computing node 1 belong are communication relationships that need to forward data through network device 1.

[0074] Then, the control device 201 can map multiple queues in the same-hop network device to the N communication relationships based on the physical topology information of the computing cluster 10. Each queue is used to cache data corresponding to one or more of the N communication relationships, and the data cached in each queue is forwarded through a forwarding path set for the network device. For ease of distinction and description, the aforementioned same-hop network device will be referred to as the target network device below.

[0075] For example, the control device 201 can first determine the association between queues in the target network device and N communication relationships. Each queue can be associated with one or more communication relationships, thus allowing the queue to cache data from one or more communication relationships associated with it. Specifically, it can cache communication data between multiple computing nodes involved in the one or more communication relationships. Since each queue in the target network device can be bound to a port of the target network device, the communication data cached by each queue can be forwarded through the port bound to that queue, i.e., forwarded through the forwarding path of that port. This determines the association between queues in the target network device and the N communication relationships, i.e., it determines multiple forwarding paths for forwarding the communication data of the N communication relationships. Since different communication relationships can be associated with different queues in the target network device, the data of the N communication relationships can be forwarded through different forwarding paths of the target network device, thereby distributing traffic to different forwarding paths. In practical applications, different communication relationships can be associated with queues corresponding to different ports. This enables traffic isolation between data from different communication relationships (traffic from different communication domains / parallel communication domains can be forwarded through different forwarding paths), thus avoiding conflicts when the target network device forwards traffic from different communication relationships. For each network device in the computing cluster 10, the control device 201 can use the above method to associate the queue with the communication relationship in that network device, so as to distribute the traffic transmitted to that network device to different forwarding paths for forwarding.

[0076] The following examples, with reference to the accompanying drawings, illustrate that the communication relationship can specifically be a parallel communication domain.

[0077] Assume that computing cluster 10 includes, for example, Figure 5The network devices 501 to 505 are shown, and the multiple computing nodes in the computing cluster 10 that participate in executing the AI ​​model include computing nodes 511 to 516. Network devices 501 to 503 are connected via network device 504 or network device 505, and computing nodes 511 and 512 are connected to network devices 501, 513, 514, 515, and 516, which can all be connected to network devices 502 and 503, as shown. Figure 5 As shown in the diagram. Network device 504 can be configured with queue 1 and queue 2, and network device 505 can be configured with queue 3 and queue 4. Assume that computing nodes 511, 513, and 514 belong to parallel communication domain A, and computing nodes 512, 515, and 516 belong to parallel communication domain B.

[0078] Assume the traffic in parallel communication domain A consists of communication data 'a' sent by computing node 511 to computing nodes 513 and 514 respectively; and the traffic in parallel communication domain B consists of communication data 'b' sent by computing node 512 to computing nodes 515 and 516 respectively. Then, control device 201 can determine, based on the topology of parallel communication domains A and B corresponding to the AI ​​model and multiple network devices, that queue 1 in network device 504 is associated with parallel communication domain A, and queue 4 in network device 505 is associated with parallel communication domain B. Thus, when multiple network devices forward traffic from computing nodes 511 to 516, queue 1 in network device 504 can be used to cache communication data from parallel communication domain A, and queue 4 in network device 505 can be used to cache communication data from parallel communication domain B. In this way, during the traffic forwarding process, multiple network devices can forward communication data 'a' from parallel communication domain A to network device 502 via queue 1, and then network device 502 forwards it to computing nodes 513 and 514 belonging to parallel communication domain A. Simultaneously, communication data 'B' from parallel communication domain B can be forwarded to network device 503 via queue 4, and then network device 503 forwards it to computing nodes 515 and 516 belonging to parallel communication domain B. Correspondingly, queues in other network devices will also be associated with either parallel communication domain A or parallel communication domain B to ensure that communication data can be successfully forwarded to the corresponding computing nodes.

[0079] In this way, the traffic of multiple parallel communication domains can be distributed to forwarding path A (network device 501 → network device 504 → network device 502) and forwarding path B (network device 501 → network device 505 → network device 503). This avoids the situation where the traffic of both parallel communication domain A and parallel communication domain B is forwarded through forwarding path A, causing traffic overload on forwarding path A and idle traffic on forwarding path B.

[0080] Furthermore, the control device 201 can also determine the link bandwidth between different network devices based on physical topology information. For example, for network device 1 and network device 5, the control device 201 can determine the link bandwidth (i.e., the total bandwidth of all physical links) between network device 1 and network device 5 based on the number of physical links between them and the bandwidth of a single physical link. Then, the control device 201 can determine the association between queues in multiple network devices and M communication relationships based on the traffic volume of each communication relationship and the link bandwidth between different network devices, so that when multiple network devices forward traffic using the corresponding forwarding path according to the association, the traffic load can not exceed the link bandwidth of the forwarding path.

[0081] In practical applications, a single network device has a limited number of queues. However, when there are many communication relationships, some (or all) of the queues in the network device can be associated with multiple communication relationships. When a queue is associated with multiple communication relationships, the traffic corresponding to these multiple communication relationships can be traffic with large time intervals. That is, the traffic generated by the computing units corresponding to the multiple communication relationships during the execution of the task does not conflict in timing.

[0082] In specific implementation, for N communication relationships that need to forward data through the same hop network device (target network device), the control device 201 can predict the order in which the computing units corresponding to the N communication relationships generate traffic during the execution of the task, and associate the same queue in the target network device with multiple communication relationships among the N communication relationships based on this order and the physical topology information of the computing cluster 10. In this way, when forwarding communication data in the queue, the target network device can send the communication data generated by the computing units corresponding to different communication relationships during the execution of the task at different time periods based on the same forwarding path. This avoids the problem of traffic congestion among multiple communication relationships when a single queue is associated with multiple communication relationships.

[0083] Still with Figure 5The computing cluster 10 and its communication relationships are illustrated using a parallel communication domain as an example. Assume the tasks in computing cluster 10 specifically include AI models, and that these AI models are partitioned and deployed within the cluster using a pipelined parallel approach. Parallel communication domains A and B are two parallel communication domains constructed based on this pipelined parallel model partitioning method. In this case, computing nodes in parallel communication domain A and parallel communication domain B can generate traffic at different times. For example, computing nodes in parallel communication domain A might execute the first network layer of the AI ​​model, while computing nodes in parallel communication domain B might execute the last network layer. Thus, during the forward computation phase of model training, multiple network devices will forward traffic to parallel communication domain B only after completing traffic forwarding for parallel communication domain A; similarly, during the backward propagation phase of model training, multiple network devices will forward traffic to parallel communication domain A only after completing traffic forwarding for parallel communication domain B. Then, control device 201 can determine that queue 1 in network device 504 is simultaneously associated with parallel communication domain A and parallel communication domain B. At this time, the traffic of parallel communication domain A and the traffic of parallel communication domain B will be forwarded to the corresponding computing nodes through forwarding path A (network device 501 → network device 504 → network device 502). However, during the forward computation phase, while forwarding communication data of parallel communication domain A using forwarding path A, computing node 512 in parallel communication domain B does not generate communication data b. Furthermore, after multiple network devices complete the forwarding of communication data a in parallel communication domain A, computing node 512 in parallel communication domain B will generate communication data b. At this time, multiple network devices will again use forwarding path A to forward communication data b in parallel communication domain B. Thus, even when queue 1 is simultaneously associated with parallel communication domain A and parallel communication domain B, the problems of traffic congestion or traffic overload in parallel communication domain A and parallel communication domain B on the forwarding path can still be avoided.

[0084] In practical applications, the same port (hereinafter referred to as the target port) in a target network device may be bound to multiple queues. That is, the communication data cached in each of these multiple queues will be forwarded through the target port (i.e., through the forwarding path where the target port is located). In this case, the control device 201 can not only determine the communication relationship associated with each queue based on the physical topology information of the computing cluster 10, but also determine the weight of the queue. This weight is used to indicate the proportion of communication data in the queue relative to the total communication data forwarded when the network device forwards the communication data cached in multiple queues through the target port within a certain period of time. At the same time, the ratio between the weights of the multiple queues bound to the target port is also the ratio between the traffic of the multiple domains associated with the multiple queues forwarded by the network device using the target port within a certain period of time.

[0085] As mentioned above Figure 5 Taking the illustrated architecture as an example, assume that queues 1 and 2 in network device 504 are both bound to port Q in network device 504. That is, network device 504 uses port Q to forward the communication data cached in queues 1 and 2. After associating queue 1 with parallel communication domain A and queue 2 with parallel communication domain B, if the computing nodes in parallel communication domain A and parallel communication domain B generate traffic simultaneously, the control device 201 can also determine the weights of queues 1 and 2. Assuming the weight of queue 1 is 0.5 and the weight of queue 2 is 0.5, then when queues 1 and 2 simultaneously cache communication data, network device 504 can use port Q to alternately forward data in queues 1 and 2. For example, after forwarding a piece of communication data in queue 1 using port Q, network device 504 can use port Q to forward a piece of communication data in queue 2, and then use port Q to continue forwarding the next piece of communication data in queue 1. In this cycle, network device 504 can forward communication data from multiple queues in parallel. The ratio between the traffic of parallel communication domain A and parallel communication domain B forwarded by network device 504 using port Q within a certain period of time is 1:1 (that is, 0.5:0.5). This can prevent communication data of one of the parallel communication domains from not being forwarded by multiple network devices for a long time.

[0086] As an implementation example, assuming the communication relationship is specifically a parallel communication domain (or communication domain), the service device 202 can determine the weight value of the queues based on the traffic volume corresponding to the parallel communication domain (i.e., the amount of communication data generated by the computing units within the parallel communication domain). For example, when the traffic volume of parallel communication domain A is 4MB (megabytes) and the traffic volume of parallel communication domain B is 1MB, the service device 202 can determine the weight of queue 1 as 0.8 and the weight of queue 2 as 0.2. Then, the service device 202 can provide the weights of each queue to the control device 201 so that the control device 201 can configure the weights of each queue in the network device accordingly. In this way, during the subsequent forwarding of communication data in queues 1 and 2 by the network device 504, it can forward 4 data packets in queue 1 using port Q, then forward 1 data packet in queue 2 using port Q, and then forward the data packets in queue 1 again using port Q. And so on, the network device 504 can use the same forwarding path to forward traffic from multiple parallel communication domains in parallel.

[0087] Of course, the above is an exemplary description based on the parallel communication domain. In other embodiments, the control device 201 may also associate the queue with the communication domain at the communication domain level, and there is no limitation on this.

[0088] After determining the association between queues and N communication relationships in the target network device based on the above method, the control device 201 can map the queues to the N communication relationships in the target network device according to the association relationship, thereby realizing the mapping of queues to N communication relationships in the target network device. For example, the control device 201 can map the queues to the N communication relationships in the target network device according to the association relationship. Figure 5 The network device 504 shown maps queue 1 to parallel communication domain A and queue 2 to parallel communication domain 2; or, the control device 201 can map the above according to the association relationship. Figure 5 The queue 1 in the network device 504 shown is mapped to parallel communication domain A and parallel communication domain B, respectively. For other network devices, the control device 201 can map the queues in the network device to the communication relationships that require the network device to forward data in a similar manner, which will not be elaborated here.

[0089] In this way, after configuring multiple network devices, multiple network devices can provide multiple forwarding paths to forward the traffic generated when the computing cluster 10 processes tasks, thereby distributing the traffic to different forwarding paths, achieving traffic load balancing across multiple forwarding paths, improving the communication efficiency between multiple computing nodes, and thus improving the overall efficiency of the computing cluster 10 in executing tasks.

[0090] The above Figure 2 The illustrated embodiment uses the example of service node 200 optimizing the logical topology between multiple network devices based on the communication needs (i.e., multiple communication relationships corresponding to tasks) between multiple computing nodes in computing cluster 10. In other embodiments, when the task running in computing cluster 10 specifically includes an AI model, service node 200 can also adjust the deployment of the AI ​​model in computing cluster 10 according to the logical topology of multiple network devices, so that multiple network devices can achieve load balancing of traffic on multiple forwarding paths when forwarding traffic of the AI ​​model. The following will combine... Figure 6 This section describes the process of deploying an AI model in computing cluster 10.

[0091] See Figure 6 The diagram illustrates a flowchart of an AI model deployment method, which may include the following steps.

[0092] S601: Service node 200 obtains the physical topology information of computing cluster 10.

[0093] The specific implementation of step S601 can be found in the relevant description of step S202 above, and will not be repeated here.

[0094] S602: The service node 200 determines multiple forwarding paths that can be provided by multiple network devices in the computing cluster 10 based on the physical topology information of the computing cluster 10. Each of these multiple forwarding paths is used to forward part of the traffic generated by multiple computing nodes in the computing cluster 10 during the running of AI models.

[0095] In practice, the service node 200 can traverse the physical topology of multiple network devices and use all the paths in the physical topology that can be used to forward traffic as available paths for forwarding traffic generated by the AI ​​model during its operation. Each forwarding path can be used to forward a portion of the traffic corresponding to the AI ​​model.

[0096] S603: Service node 200 generates a deployment strategy for the AI ​​model based on multiple forwarding paths. This deployment strategy is used to deploy the AI ​​model on multiple computing nodes in computing cluster 10.

[0097] In this embodiment, the AI ​​model may not yet be deployed in the computing cluster 10. In this case, the service node 200 can generate a deployment strategy for the AI ​​model to guide its deployment in the computing cluster 10. Alternatively, the AI ​​model may already be distributed across multiple computing nodes in the computing cluster 10. Therefore, the service node 200 can optimize the deployment of the AI ​​model in the computing cluster 10 by generating a deployment strategy.

[0098] In one possible implementation, the deployment strategy to be generated includes a parallel strategy and a communication strategy for the AI ​​model in the computing cluster 10.

[0099] Parallelism strategies can be used to indicate how an AI model operates in a computing cluster. This parallelism can include at least one of pipelining parallelism, data parallelism, and tensor parallelism, or other types of parallelism. Pipelining parallelism refers to deploying different network layers of the AI ​​model to different computing nodes (or computing units), allowing multiple different computing nodes to run different network layers of the AI ​​model in parallel, such as... Figure 7 As shown; typically, different pipeline parallelism methods can generate different amounts of traffic and have different requirements for forwarding paths. Data parallelism refers to deploying the same network layer of an AI model across multiple different computing nodes (or computing units), thereby inputting multiple different input data into these multiple different computing nodes, enabling the AI ​​model to complete the training or inference process for larger input data, such as... Figure 7As shown; typically, the traffic requirements for forwarding paths can differ depending on the data parallelism method. Tensor parallelism refers to dividing the parameters of an AI model into multiple tensors, with each tensor deployed to different computing units (or nodes), thus allowing multiple tensors to be executed in parallel based on different computing units, such as... Figure 7 As shown; typically, the traffic generated by different tensor parallelism methods may have different requirements for forwarding paths. For example, when tensor parallelism is performed across nodes, traffic between different computing nodes will need to be forwarded through network devices. However, when tensor parallelism is performed using multiple accelerator cards (computing nodes) within the same computing node, multiple accelerators can communicate with each other through the internal bus of the computing node, thus eliminating the need for traffic to be forwarded through network devices.

[0100] Communication strategies can be used to instruct different computing units involved in executing an AI model on the communication operators they execute. Typically, when multiple computing units execute different communication operators, they may communicate data in different ways, which can easily result in different traffic patterns.

[0101] Then, service node 200 determines the value space of parallelism methods and communication operators for the AI ​​model, and determines the combination space of deployment strategies based on these two value spaces. Thus, service node 200 can search the combination space for combinations of parallelism methods and communication operators that are compatible with multiple forwarding paths (which can be used to forward traffic for the AI ​​model) from multiple network devices. This can be done, for example, by using a heuristic search algorithm. Based on this, corresponding parallelism and communication strategies are generated, i.e., deployment strategies for the AI ​​model are generated. Subsequently, deploying the AI ​​model in computing cluster 10 based on the searched deployment strategies allows the traffic generated during the operation of the AI ​​model in computing cluster 10 to be distributed across multiple forwarding paths in multiple network devices. This achieves traffic load balancing across multiple forwarding paths, improves communication efficiency between multiple computing nodes, and ultimately improves the overall training / inference efficiency of the AI ​​model running on multiple computing nodes.

[0102] Furthermore, based on the deployment strategy generated by service node 200, service node 200 or computing cluster 10 can deploy the AI ​​model or adjust the deployment for that AI model. For ease of understanding, the following explanation uses service node 200 executing the deployment strategy as an example.

[0103] In practical applications, the deployment strategy generated by service node 200 may include only parallel strategy or communication strategy, or it may be other strategies that can affect the deployment of AI models in computing cluster 10, without any limitation.

[0104] S604: Service node 200 deploys the AI ​​model to multiple computing nodes in computing cluster 10 according to the deployment strategy, or adjusts the deployment of the AI ​​model in the computing cluster according to the deployment strategy.

[0105] For example, when the deployment strategy includes a parallel strategy and a communication strategy, the service node 200 (specifically, the service device 202 within the service node 200) can determine the parallel mode of the AI ​​model according to the parallel strategy, and then perform corresponding segmentation of the AI ​​model according to the parallel mode. Then, the service node 200 can configure the communication operators to be executed by each of the multiple computing nodes according to the communication strategy. In this way, the service node 200 can deploy the AI ​​model to the computing cluster 10.

[0106] Alternatively, if the AI ​​model has already been deployed to computing cluster 10 before the deployment strategy is generated, and the deployment strategy includes both parallelism and communication strategies, service node 200 can obtain the deployment information of the AI ​​model in computing cluster 10. For example, multiple computing units participating in running the AI ​​model can report their identifiers and executed communication operators to service node 200, or service node 200 can obtain deployment information for the AI ​​model from the scheduling node. Then, service node 200 can compare the differences between the deployment information of the AI ​​model in computing cluster 10 and the deployment strategy, i.e., determine the differences between the current parallelism and communication operators of the AI ​​model in computing cluster 10 and those indicated by the deployment strategy. Based on these differences, service node 200 can adjust the deployment of the AI ​​model in computing cluster 10. For example, service node 200 can increase the parallelism of the AI ​​model or change the communication operators executed by some computing units.

[0107] In this way, since the deployment strategy is generated by the service node 200 adapting to multiple forwarding paths in multiple network devices, after the service node 200 determines or adjusts the deployment of the AI ​​model in the computing cluster 10 according to the deployment strategy, the traffic generated by the AI ​​model when running in the computing cluster 10 can be distributed to multiple forwarding paths in multiple network devices, thereby achieving traffic load balancing on multiple forwarding paths.

[0108] Accordingly, the service node 200 can configure multiple network devices to enable these devices to forward traffic generated during the operation of the AI ​​model based on the multiple forwarding paths. Therefore, this embodiment may also include the following steps.

[0109] S605: Service node 200 obtains multiple communication relationships corresponding to the AI ​​model. Each of these multiple communication relationships represents the data transmission relationship between some computing units among multiple computing units when executing a task.

[0110] For example, the communication relationship may be a parallel communication domain or a communication domain.

[0111] In this embodiment, after deploying the AI ​​model to the computing cluster 10 or adjusting the deployment of the AI ​​model in the computing cluster 10, the service node 200 can obtain information such as multiple parallel communication domains or communication domains corresponding to the AI ​​model. The specific implementation method can be found in the relevant part of step S201 above, and will not be repeated here.

[0112] Among them, it is assumed that the AI ​​model is based on Figure 7 The computing cluster 10 shown is deployed in a distributed manner. The service node 200 can determine multiple computing units running the AI ​​model in a data-parallel manner based on the information reported by each computing node. It then associates these multiple computing units according to the pipeline parallelism of the AI ​​models, and associates computing units located on the same computing node according to tensor parallelism. This allows it to obtain the communication requirements between all computing units participating in running the AI ​​model, i.e., the multiple communication relationships corresponding to the AI ​​model. In practical applications, these multiple communication relationships corresponding to the AI ​​model can be stored as a task domain (Job domain) information (when multiple different AI models are running in the computing cluster 10, the service node 200 can generate different Job domains for different AI models). Furthermore, when the AI ​​model in the computing cluster 10 finishes running, such as when the AI ​​model training is complete or the AI ​​model is removed from the computing cluster 10, the computing cluster 10 can notify the service node 200 to delete the Job task corresponding to that AI model.

[0113] S606: Service node 200 maps queues to communication relationships in multiple network devices corresponding to multiple forwarding paths.

[0114] In this process, after the service node 200 maps the queues of multiple network devices on multiple forwarding paths to multiple communication relationships corresponding to the AI ​​model, the traffic between different computing units corresponding to each communication relationship can be cached in the queue mapped to the communication relationship, and then forwarded through the port bound to the queue (i.e. the forwarding path where the port is located), thereby distributing the traffic between computing nodes corresponding to different communication relationships to different forwarding paths.

[0115] In this embodiment, the specific implementation of step S605 can be found in the above description. Figure 2The descriptions of mapping queues to communication relationships in the network devices in the illustrated embodiments are not repeated here.

[0116] It should be noted that in practical application scenarios, the above can also be applied to... Figure 2 as well as Figure 6 The embodiments shown are combined. For example, based on the above... Figure 2 In the illustrated embodiment, based on the optimization of the logical topology of multiple network devices, the service node 200 can also, based on the above... Figure 6 The illustrated embodiment adjusts the deployment of the AI ​​model in the computing cluster 10. For example, the service node 200 can adjust the deployment of the AI ​​model in the computing cluster based on the forwarding paths set by the aforementioned N communication relationships. This could involve adjusting the parallelism of the AI ​​model in the computing cluster 10, adjusting the communication operators executed by multiple computing nodes in the computing cluster 10 during the execution of the AI ​​model, or simultaneously adjusting both the parallelism of the AI ​​model in the computing cluster 10 and the communication operators executed by the computing nodes during the execution of the AI ​​model.

[0117] It is worth noting that other reasonable combinations of steps that can be conceived by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0118] The above combination Figures 1 to 7 The task processing method provided in the embodiments of this application will be introduced. Next, the structure of the network device configuration device and computing device provided in the embodiments of this application will be described with reference to the accompanying drawings.

[0119] See Figure 8 The diagram shows a schematic of a task processing device. Figure 8 The task processing device 800 shown is applied to a computing cluster, in which multiple computing nodes communicate with each other through multiple network devices, and multiple computing units in the multiple computing nodes are used to run tasks, such as AI tasks or HPC tasks.

[0120] like Figure 8 As shown, the network device configuration device 800 includes:

[0121] The determination module 801 is used to determine the M communication relationships corresponding to the task. Each of the M communication relationships represents the data transmission relationship between some computing units among multiple computing units when the task is executed, and M is a positive integer.

[0122] The acquisition module 802 is used to acquire the physical topology information of the computing cluster;

[0123] The configuration module 803 is used to set a forwarding path in the same hop network device for each of the N communication relationships that need to forward data through the same hop network device, based on the physical topology information of the computing cluster. N is a positive integer. When forwarding task data, the same hop network device forwards the data according to the forwarding path set for the N communication relationships.

[0124] In one possible implementation, the M communication relationships are either M communication domains during task execution or M parallel communication domains during task execution, where each parallel communication domain is a communication relationship between different computing units when a subtask of a task is executed in parallel across multiple computing nodes.

[0125] In one possible implementation, when setting a forwarding path in the same-hop network device for each of the N communication relationships based on the physical topology information of the computing cluster, the setting module 803 is specifically used for:

[0126] Identify multiple queues within the same-hop network device; based on the physical topology information of the computing cluster, map the multiple queues to N communication relationships within the same-hop network device. Each queue is used to cache data corresponding to at least one of the N communication relationships, and the data cached by each queue is forwarded through a forwarding path set for the same-hop network device.

[0127] In one possible implementation, the same-hop network device includes a first queue, and the N communication relationships include a first communication relationship and a second communication relationship;

[0128] When setting module 803 maps multiple queues to N communication relationships in the same hop network device, it is specifically used for:

[0129] In the same hop network device, the first queue is mapped to the first communication relationship and the second communication relationship respectively. The time period during which multiple computing units corresponding to the first communication relationship generate traffic during the execution of the task does not overlap with the time period during which multiple computing units corresponding to the second communication relationship generate traffic during the execution of the task.

[0130] In one possible implementation, the same-hop network device includes a second queue and a third queue, and the N communication relationships include a third communication relationship and a fourth communication relationship. The same-hop network device forwards the data cached in the second queue and the data cached in the third queue through the target port.

[0131] When setting module 803 maps multiple queues to N communication relationships in the same hop network device, it is specifically used for:

[0132] Determine the traffic volume corresponding to the third communication relationship and the traffic volume corresponding to the fourth communication relationship;

[0133] In the same hop network device, the second queue is mapped to the third communication relationship, and the third queue is mapped to the fourth communication relationship;

[0134] Based on the traffic volume corresponding to the third communication relationship and the traffic volume corresponding to the fourth communication relationship, a first weight is configured for the traffic of the third communication relationship and a second weight is configured for the traffic of the fourth communication relationship. The first weight and the second weight are used to indicate the ratio between the traffic of the third communication relationship and the traffic of the fourth communication relationship forwarded by the same hop network device within a certain period of time.

[0135] In one possible implementation, the task is a task that includes an AI model, and the task processing device 800 further includes:

[0136] The generation module 804 is used to generate a deployment strategy for the AI ​​model based on the forwarding paths set for N communication relationships.

[0137] Adjustment module 805 is used to adjust the deployment of AI models in the computing cluster according to the deployment strategy.

[0138] In one possible implementation, the deployment strategy includes a parallel strategy or a communication strategy. The parallel strategy is used to adjust the parallelism of the AI ​​model in the computing cluster, and the communication strategy is used to adjust the communication operators executed by multiple computing nodes during the execution of the AI ​​model.

[0139] because Figure 8 The task processing device 800 shown corresponds to the above. Figure 2 Service node 200 in the illustrated embodiment, therefore Figure 8 For the specific implementation of the task processing device 800 shown and its technical effects, please refer to the above. Figure 2 The relevant descriptions in the illustrated embodiments are not repeated here.

[0140] Figure 9 This application provides a schematic diagram of the hardware structure of a computing device 900, which, for example, can implement the above-described... Figure 2 Service node 200, etc., in the illustrated embodiment.

[0141] like Figure 9As shown, the computing device 900 includes a processor 901, a memory 902, and a communication interface 903. The processor 901, memory 902, and communication interface 903 communicate via a bus 904, or via other means such as wireless transmission. The memory 902 stores instructions, and the processor 901 executes the instructions stored in the memory 902. Further, the computing device 900 may also include a memory unit 905, which is connected to the processor 901, the storage medium 902, and the communication interface 903 via the bus 904. The memory 902 stores program code, and the processor 901 can read the program code stored in the memory 902 into the memory unit 905 and execute the program code in the memory unit 905, causing the processor 901 to perform the following operations:

[0142] Determine M communication relationships corresponding to the task. Each of the M communication relationships represents the data transmission relationship between some computing units among multiple computing units when the task is executed. M is a positive integer.

[0143] Obtain the physical topology information of the computing cluster;

[0144] For N communication relationships out of M communication relationships that need to forward data through the same hop network device, a forwarding path in the same hop network device is set for each of the N communication relationships according to the physical topology information of the computing cluster, where N is a positive integer; wherein, when forwarding task data, the same hop network device forwards data according to the forwarding path set for the N communication relationships.

[0145] It should be understood that in this embodiment, the processor 901 can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete device assemblies, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0146] The memory 902 may include read-only memory and random access memory, and provides instructions and data to the processor 901. The memory 902 may also include non-volatile random access memory.

[0147] The memory 902 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0148] The communication interface 903 is used to communicate with other devices connected to the computing device 900. The bus 904 may include a data bus, a power bus, a control bus, and a status signal bus, etc. However, for clarity, all buses are labeled as bus 904 in the figure.

[0149] It should be understood that the computing device 900 according to the embodiments of this application may correspond to the service node 200 in the embodiments of this application, and may correspond to the execution of the embodiments of this application. Figure 2 The method executed by the service node 200 in the illustrated method, and the above and other operations and / or functions implemented by the computing device 900, are respectively for the purpose of implementing... Figure 2 The process of the corresponding methods in [the document] will not be elaborated here for the sake of brevity.

[0150] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the above-described task processing method.

[0151] This application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0152] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0153] The computer program product can be a software installation package. When any of the aforementioned task processing methods is required, the computer program product can be downloaded and executed on a computing device.

[0154] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0155] The terminology used in the above embodiments is for the purpose of describing specific embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, “one or more” refers to one, two, or more; the character “ / ” generally indicates that the preceding and following objects are in an “or” relationship. In the embodiments of this application, “simultaneously” means within the same time period, including situations where they are at the same moment.

[0156] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0157] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A task processing method, characterized in that, The method is applied to a computing cluster, wherein multiple computing nodes in the computing cluster communicate with each other through multiple network devices, and multiple computing units in the multiple computing nodes are used to run tasks. The method includes: M communication relationships are determined corresponding to the task, where each of the M communication relationships represents the data transmission relationship between some of the multiple computing units when the task is executed, and M is a positive integer; Obtain the physical topology information of the computing cluster; For N communication relationships among the M communication relationships that need to forward data through the same hop network device, a forwarding path in the same hop network device is set for each of the N communication relationships according to the physical topology information of the computing cluster, where N is a positive integer; wherein, when forwarding the data of the task, the same hop network device forwards the data according to the forwarding path set for the N communication relationships.

2. The method according to claim 1, characterized in that, The M communication relationships are either M communication domains when executing the task, or M parallel communication domains when executing the task. Each parallel communication domain is a communication relationship between different computing units when the subtasks of the task are executed in parallel among the multiple computing nodes.

3. The method according to claim 1 or 2, characterized in that, The same-hop network device includes a first queue, and the N communication relationships include a first communication relationship and a second communication relationship; The step of setting a forwarding path in the same-hop network device for each of the N communication relationships based on the physical topology information of the computing cluster includes: Based on the physical topology information of the computing cluster, the first queue is mapped to the first communication relationship and the second communication relationship in the same-hop network device. The time period during which multiple computing units corresponding to the first communication relationship generate traffic during the execution of the task does not overlap with the time period during which multiple computing units corresponding to the second communication relationship generate traffic during the execution of the task. The first queue is used to cache the data corresponding to the first communication relationship and the data corresponding to the second communication relationship. The data cached in the first queue is forwarded through a forwarding path set for the same-hop network device.

4. The method according to claim 1 or 2, characterized in that, The same-hop network device includes a second queue and a third queue, and the N communication relationships include a third communication relationship and a fourth communication relationship. The same-hop network device forwards the data cached in the second queue and the data cached in the third queue through the forwarding path where the target port is located. The step of setting a forwarding path in the same-hop network device for each of the N communication relationships based on the physical topology information of the computing cluster includes: Determine the traffic volume corresponding to the third communication relationship and the traffic volume corresponding to the fourth communication relationship; Based on the physical topology information of the computing cluster, the second queue is mapped to the third communication relationship in the same-hop network device, and the third queue is mapped to the fourth communication relationship. The second queue is used to cache the data corresponding to the third communication relationship, and the third queue is used to cache the data corresponding to the fourth communication relationship. Based on the traffic volume corresponding to the third communication relationship and the traffic volume corresponding to the fourth communication relationship, a first weight is configured for the traffic of the third communication relationship and a second weight is configured for the traffic of the fourth communication relationship. The first weight and the second weight are used to indicate the ratio between the traffic of the third communication relationship and the traffic of the fourth communication relationship forwarded by the same hop network device within a certain period of time.

5. The method according to any one of claims 1 to 4, characterized in that, The task includes an artificial intelligence (AI) model, and the method further includes: Based on the forwarding paths set for the N communication relationships, adjust the parallel mode of the AI ​​model in the computing cluster or the communication operators executed by the multiple computing nodes during the execution of the AI ​​model.

6. A task processing device, characterized in that, The apparatus is applied to a computing cluster, wherein multiple computing nodes in the computing cluster communicate data with each other through multiple network devices, and multiple computing units in the multiple computing nodes are used to run tasks. The apparatus includes: A determining module is used to determine M communication relationships corresponding to the task, where each of the M communication relationships represents a data transmission relationship between some of the multiple computing units when the task is executed, and M is a positive integer; The acquisition module is used to acquire the physical topology information of the computing cluster; The configuration module is used to configure a forwarding path in the same-hop network device for each of the N communication relationships that need to forward data through the same-hop network device, based on the physical topology information of the computing cluster, where N is a positive integer; wherein, when forwarding the data of the task, the same-hop network device forwards the data according to the forwarding path configured for the N communication relationships.

7. The apparatus according to claim 6, characterized in that, The M communication relationships are either M communication domains when executing the task, or M parallel communication domains when executing the task. Each parallel communication domain is a communication relationship between different computing units when the subtasks of the task are executed in parallel among the multiple computing nodes.

8. The apparatus according to claim 6 or 7, characterized in that, The same-hop network device includes a first queue, and the N communication relationships include a first communication relationship and a second communication relationship; When the setting module sets the forwarding path in the same-hop network device for each of the N communication relationships based on the physical topology information of the computing cluster, it is specifically used for: Based on the physical topology information of the computing cluster, the first queue is mapped to the first communication relationship and the second communication relationship in the same-hop network device. The time period during which multiple computing units corresponding to the first communication relationship generate traffic during the execution of the task does not overlap with the time period during which multiple computing units corresponding to the second communication relationship generate traffic during the execution of the task. The first queue is used to cache the data corresponding to the first communication relationship and the data corresponding to the second communication relationship. The data cached in the first queue is forwarded through a forwarding path set for the same-hop network device.

9. The apparatus according to claim 6 or 7, characterized in that, The same-hop network device includes a second queue and a third queue, and the N communication relationships include a third communication relationship and a fourth communication relationship. The same-hop network device forwards the data cached in the second queue and the data cached in the third queue through the forwarding path where the target port is located. When the setting module sets the forwarding path in the same-hop network device for each of the N communication relationships based on the physical topology information of the computing cluster, it is specifically used for: Determine the traffic volume corresponding to the third communication relationship and the traffic volume corresponding to the fourth communication relationship; Based on the physical topology information of the computing cluster, the second queue is mapped to the third communication relationship and the third queue is mapped to the fourth communication relationship in the same-hop network device. The second queue is used to cache the data corresponding to the third communication relationship and the third queue is used to cache the data corresponding to the fourth communication relationship. The data cached by the second queue and the third queue are forwarded through the forwarding path where the target port is located, which is set for the same-hop network device. Based on the traffic volume corresponding to the third communication relationship and the traffic volume corresponding to the fourth communication relationship, a first weight is configured for the traffic of the third communication relationship and a second weight is configured for the traffic of the fourth communication relationship. The first weight and the second weight are used to indicate the ratio between the traffic of the third communication relationship and the traffic of the fourth communication relationship forwarded by the same hop network device within a certain period of time.

10. The apparatus according to any one of claims 6 to 9, characterized in that, The task is one that includes an artificial intelligence (AI) model, and the device further includes: The adjustment module is used to adjust the parallel mode of the AI ​​model in the computing cluster or the communication operators executed by the multiple computing nodes during the execution of the AI ​​model, based on the forwarding paths set for the N communication relationships.

11. A computing device, characterized in that, Including processor and memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the steps of the method as described in any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on at least one computing device, cause the at least one computing device to perform the steps of the method as described in any one of claims 1 to 5.

13. A computer program product containing instructions, characterized in that, When it is run on at least one computing device, it causes the at least one computing device to perform the method as described in any one of claims 1 to 5.

Images