Device cluster operation method and related apparatus

Abstract

The present application relates to the field of computer technology, and provides a device cluster operation method and a related apparatus. Devices in different device clusters can communicate by means of communication links between physical cards of the devices, and the quantities of devices comprised in device clusters can be dynamically adjusted on the basis of real-time load conditions of the device clusters. For example, when determining that the quantity of devices in a first device cluster needs to be adjusted, a computing device C may send a first notification to a first device. When receiving the first notification sent from the computing device C, the first device may establish or disconnect communication links between physical cards of the first device and physical cards of a second device on the basis of the first notification. By means of the described process, the quantity of devices comprised in the first device cluster can be dynamically adjusted, and there is no need to configure a large number of redundant resources in the first device cluster, thereby reducing resource waste.

Description

Method for operating a cluster of devices and related apparatus

[0001] Cross-reference to Related Applications

[0002] This application claims priority to the Chinese Patent Application No. 202410860098.1, filed on June 27, 2024, and entitled "A Method for Operating a Cluster of Devices and Related Apparatus", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of computers, and in particular to a method for operating a cluster of devices and related apparatus. BACKGROUND

[0004] With the development of artificial intelligence (AI) technology, large language models (LLMs) are increasingly widely used in people's daily work and life. The inference process of an LLM mainly includes two stages: a prefill stage and a decoder stage. When performing the inference process of an LLM, a first cluster of devices can be used to perform a prefill task in the prefill stage, and a second cluster of devices can be used to perform a decoding task in the decoder stage, so as to separate the prefill task and the decoding task and reduce interference between them.

[0005] In some application scenarios, an LLM can be deployed in a cluster of computing devices in the cloud. The cluster of computing devices can include a first cluster of devices and a second cluster of devices. The cluster of computing devices can receive a user request text transmitted by a terminal device through a network, perform inference based on the user request text, and feed back an inference result to the terminal device.

[0006] In actual application, the amount of received user requests can suddenly increase for a short time. In order to cope with the computing power required when the amount of user requests increases, a large number of computing devices need to be configured in the first cluster of devices and the second cluster of devices. At most times, the amount of user requests is usually small, and at this time, there will be a large number of redundant resources in the first cluster of devices or the second cluster of devices, resulting in resource waste. SUMMARY

[0007] The present application provides a method for operating a cluster of devices and related apparatus, which is beneficial to dynamically adjusting the number of devices in the first cluster of devices or the second cluster of devices.

[0008] In a first aspect, the present application provides a device cluster running method, which can be executed by a first device in a device cluster, or a chip, a chip system or a circuit in the first device. The device cluster running method can include: receiving, by the first device, a first notification, and based on the first notification, establishing or disconnecting a communication link between a physical card of the first device and a physical card of a second device. The first notification is sent by a fifth device when the fifth device determines that the number of devices in the first device cluster needs to be adjusted according to the real-time load of the first device cluster, and the first notification is used to indicate that the first device and the second device adjust the communication connection relationship. The first device belongs to the first device cluster, and the second device belongs to the second device cluster. The devices in the first device cluster are used to execute the first stage of the model inference process, and the devices in the second device cluster are used to execute the second stage of the model inference process. The fifth device is a computing device, which is used to monitor the load and running state of the computing devices in the first device cluster and the second device cluster, and adjust the number of devices in the first device cluster and the second device cluster.

[0009] The device cluster running method provided by the present application can enable the devices in different device clusters to communicate through the communication link between the physical cards of the devices, and dynamically adjust the number of devices included in the device cluster according to the real-time load of the device cluster. For example, when the fifth device determines that the number of devices in the first device cluster needs to be adjusted, the fifth device can send a first notification to the first device. The first device receives the first notification sent by the fifth device, and based on the first notification, establishes or disconnects the communication link between the physical card of the first device and the physical card of the second device. Through the above process, the number of devices included in the first device cluster can be dynamically adjusted, and a large number of redundant resources do not need to be set in the first device cluster. For example, a small number of computing devices can be initially set in the first device cluster to process daily user requests. When the user request volume increases, the number of computing devices in the first device cluster can be increased to share the load of the existing computing devices in the first device cluster and speed up the processing of user requests. When the user request volume decreases, the number of computing devices in the first device cluster can be reduced. The computing devices that exit from the first device cluster can be used to process other businesses, thereby reducing resource waste.

[0010] In an optional implementation, the first device is configured to perform the first stage of the model inference process by using N physical cards, and the second device is configured to perform the second stage of the model inference process by using N physical cards. The first device can include more than or equal to N physical cards, and the second device can include more than or equal to N physical cards. The N physical cards in the first device have a corresponding relationship with the N physical cards in the second device. When the first notification is used to indicate that the communication connection between the first device and the second device is established, based on the first notification, each of the N physical cards in the first device sends a link establishment request to the corresponding physical card in the second device to establish a communication link with the corresponding physical card. The link establishment request sent by each of the N physical cards includes information of the corresponding physical card in the second device.

[0011] In the above implementation, the physical cards in the first device have a one-to-one corresponding relationship with the physical cards in the second device, and the link establishment request sent by the physical card in the first device carries information of the corresponding physical card in the second device, so that a one-to-one communication link can be established between the physical card in the first device and the physical card in the second device.

[0012] In an optional implementation, the first device is configured to perform the first stage of the model inference process by using N physical cards, and the second device is configured to perform the second stage of the model inference process by using N physical cards. The first device can include more than or equal to N physical cards, and the second device can include more than or equal to N physical cards. The N physical cards in the first device have a corresponding relationship with the N physical cards in the second device. When the first notification is used to indicate that the communication connection between the first device and the second device is established, based on the first notification, each of the N physical cards in the first device sends a link establishment request to the corresponding physical card in the second device to establish a communication link with the corresponding physical card. The link establishment request sent by each of the N physical cards includes information of the corresponding physical card in the second device.

[0013] In an optional implementation, the established communication link is a remote direct data access (RDMA) link. The first physical card in the first device is configured to transmit the processing result of the first stage to the corresponding physical card in the second device through the RDMA link, or the first physical card in the first device is configured to receive the processing result of the second stage transmitted by the corresponding physical card in the second device through the RDMA link. The first physical card can be any physical card in the first device.

[0014] In the above implementation, the communication link established between the physical card in the first device and the physical card in the second device is an RDMA link, so that the physical card in the first device and the physical card in the second device can realize fast data transmission through the RDMA link.

[0015] In an optional implementation, the first device can disconnect the communication link between the physical card of the first device and the physical card of the second device upon receiving a second notification. The second notification is sent by the fifth device upon determining that the second device is faulty.

[0016] In the above implementation, the fifth device can send a notification to the first device having a communication connection with the second device upon determining that the second device is faulty, so that the first device actively disconnects the communication link between the physical card of the first device and the physical card of the second device, thereby isolating only the faulty computing device and not affecting the business processing of other computing devices in the device cluster.

[0017] In an optional implementation, the first device can send a disconnection request from the physical card of the first device to the physical card of the second device upon receiving a third notification. The third notification is sent by the fifth device upon determining that the number of devices in the first device cluster and the second device cluster needs to be adjusted. The first device establishes a communication link between the physical card of the first device and the physical card of the third device in response to the connection request sent by the physical card of the third device. The third device belongs to the first device cluster.

[0018] Through the above process, the computing devices in the first device cluster and the computing devices in the second device cluster can directly switch roles without restarting the computing devices, thereby reducing the impact on the model inference business, and even achieving almost no impact on the model inference business.

[0019] In a second aspect, the present application provides a device cluster operation method, which can be executed by a fifth device in a computing device cluster, or a chip, a chip system or a circuit in the fifth device. The fifth device is a computing device, which is used to monitor the load and running state of the computing devices in a first device cluster and a second device cluster, and adjust the number of devices in the first device cluster and the second device cluster. The device cluster operation method can include: the fifth device sends a first notification to the first device upon determining that the number of devices in the first device cluster needs to be adjusted. The first notification is used to instruct the first device to adjust the communication connection relationship with the second device. The first device belongs to the first device cluster, and the second device belongs to the second device cluster. The devices in the first device cluster are used to execute a first stage of a model inference process, and the devices in the second device cluster are used to execute a second stage of the model inference process.

[0020] In an optional implementation, the fifth device can monitor the load of the first device cluster, and determine whether the number of devices in the first device cluster needs to be adjusted based on the load of the first device cluster.

[0021] In an optional implementation, the first notification carries a correspondence between N physical cards of the first device and N physical cards of the second device.

[0022] In an optional implementation, the fifth device can send a second notification to the first device when determining that the second device is faulty. The second notification is used to instruct to disconnect the communication link between the physical card of the first device and the physical card of the second device.

[0023] In an optional implementation, the fifth device can send a third notification to the first device when determining that the number of devices included in the first device cluster and the second device cluster needs to be adjusted. The third notification is used to instruct the first device to change from the first device cluster to the second device cluster.

[0024] In an optional implementation, the fifth device can send a fourth notification to the second device when determining that the number of devices included in the first device cluster and the second device cluster needs to be adjusted. The fourth notification is used to instruct the second device to change from the second device cluster to the first device cluster.

[0025] In a third aspect, the present application provides a device cluster running method, which can be executed by the second device in the device cluster, or a chip, a chip system or a circuit in the second device. The device cluster running method can include: the second device establishes or disconnects a communication link between a physical card of the second device and a physical card of the first device in response to a first request sent by the physical card of the first device. The first request is sent by the physical card of the first device based on a first notification received by the first device. The first notification is sent by the fifth device when determining that the number of devices in the first device cluster needs to be adjusted, and is used to instruct the first device and the second device to adjust the communication connection relationship. The first device belongs to the first device cluster, and the second device belongs to the second device cluster. The devices in the first device cluster are used to execute the first stage of the model inference process, and the devices in the second device cluster are used to execute the second stage of the model inference process. The fifth device is a computing device, which is used to monitor the load condition and the running state of the computing devices in the first device cluster and the second device cluster, and to adjust the number of devices in the first device cluster and the second device cluster.

[0026] In an optional implementation, the second device disconnects the communication link between the physical card of the second device and the physical card of the first device when receiving the fourth notification. The fourth notification is sent by the fifth device when determining that the number of devices included in the first device cluster and the second device cluster needs to be adjusted. The second device sends a link establishment request to the physical card of the fourth device to establish a communication link between the physical card of the second device and the physical card of the fourth device. The fourth device belongs to the second device cluster.

[0027] In a fourth aspect, the present application provides a device cluster running apparatus, which can be applied to a first device. The device cluster running apparatus can include:

[0028] a notification receiving module, configured to receive a first notification; the first notification is sent by a fifth device when it is determined that the number of devices in a first device cluster needs to be adjusted; the first notification is used to instruct the first device and a second device to adjust a communication connection relationship; the first device belongs to the first device cluster, and the second device belongs to a second device cluster; the devices in the first device cluster are used to execute a first stage of a model inference process; the devices in the second device cluster are used to execute a second stage of the model inference process;

[0029] a link adjusting module, configured to, based on the first notification, establish or disconnect a communication link between a physical card of the first device and a physical card of the second device.

[0030] In an optional implementation, the first device and the second device each include N physical cards; the first device is used to execute the first stage of the model inference process through the N physical cards; the second device is used to execute the second stage of the model inference process through the N physical cards. The N physical cards of the first device have a corresponding relationship with the N physical cards of the second device. When the first notification is used to instruct the first device and the second device to establish a communication connection, based on the first notification, each of the N physical cards of the first device sends a link establishment request to a corresponding physical card of the second device to establish a communication link with the corresponding physical card; the link establishment request sent by each of the N physical cards includes information of the corresponding physical card of the second device.

[0031] In an optional implementation, the established communication link is an RDMA link; the first physical card of the first device is used to transmit a processing result of the first stage to the corresponding physical card of the second device through the RDMA link; or the first physical card of the first device is used to receive a processing result of the second stage transmitted by the corresponding physical card of the second device through the RDMA link, wherein the first physical card can be understood as any one of the physical cards of the first device.

[0032] In an optional implementation, the link adjusting module can also be used to, when a second notification is received, disconnect the communication link between the physical card of the first device and the physical card of the second device. The second notification is sent by the fifth device when it is determined that the second device is faulty.

[0033] In a fifth aspect, the present application provides a device cluster running apparatus, which can be applied to a fifth device. The device cluster running apparatus can include:

[0034] The notification sending module is configured to send a first notification to the first device when it is determined that the number of devices in the first device cluster needs to be adjusted, and the first notification is used to instruct the first device to adjust the communication connection relationship with the second device. The first device belongs to the first device cluster, and the second device belongs to the second device cluster. The devices in the first device cluster are used to perform the first stage of the model inference process, and the devices in the second device cluster are used to perform the second stage of the model inference process.

[0035] In an optional implementation, the device cluster running apparatus can further include a notification generating module, which can be configured to monitor the load of the first device cluster, determine whether the number of devices in the first device cluster needs to be adjusted based on the load of the first device cluster, and generate the first notification when it is determined that the number of devices in the first device cluster needs to be adjusted.

[0036] In an optional implementation, the first notification carries a correspondence between the N physical cards of the first device and the N physical cards of the second device.

[0037] In an optional implementation, the fifth device can send a second notification to the first device when it is determined that the second device is faulty. The second notification is used to instruct to disconnect the communication link between the physical card of the first device and the physical card of the second device.

[0038] In an optional implementation, the notification sending module can be further configured to send a third notification to the first device when it is determined that the number of devices included in the first device cluster and the second device set needs to be adjusted. The third notification is used to instruct the first device to change from the first device cluster to the second device cluster.

[0039] In an optional implementation, the notification sending module can be further configured to send a fourth notification to the second device when it is determined that the number of devices included in the first device cluster and the second device set needs to be adjusted. The fourth notification is used to instruct the second device to change from the second device cluster to the first device cluster.

[0040] In a sixth aspect, the present application provides a device cluster running apparatus, which can be applied to a second device. The device cluster running apparatus can include:

[0041] The link adjustment module is configured to, in response to a first request sent by the physical card of the first device, establish or disconnect a communication link between the physical card of the second device and the physical card of the first device; the first request is sent by the physical card of the first device based on a first notification received by the first device; the first notification is sent by a fifth device when it is determined that the number of devices in the first device cluster needs to be adjusted; the first notification is used to instruct the first device and the second device to adjust the communication connection relationship; the first device belongs to the first device cluster, and the second device belongs to the second device cluster; the devices in the first device cluster are used to perform a first stage of a model inference process; and the devices in the second device cluster are used to perform a second stage of the model inference process.

[0042] In an optional implementation, the link adjustment module is further configured to, when the fourth notification is received, disconnect the communication link between the physical card of the second device and the physical card of the first device. The fourth notification is sent by the fifth device when it is determined that the number of devices contained in the first device cluster and the second device cluster needs to be adjusted. A link establishment request is sent from the physical card of the second device to the physical card of a fourth device to establish a communication link between the physical card of the second device and the physical card of the fourth device, and the fourth device belongs to the second device cluster.

[0043] In a seventh aspect, the present application provides a computing device, which is a first device, comprising a host and a physical card connected to the host;

[0044] The host is configured to receive a first notification; the first notification is sent by a fifth device when it is determined that the number of devices in the first device cluster needs to be adjusted according to the real-time load of the first device cluster; the first notification is used to instruct the first device and the second device to adjust the communication connection relationship; the first device belongs to the first device cluster, and the second device belongs to the second device cluster; the devices in the first device cluster are used to perform a first stage of a model inference process; and the devices in the second device cluster are used to perform a second stage of the model inference process;

[0045] The physical card is configured to, based on the first notification, establish or disconnect a communication link with the physical card of the second device.

[0046] In an eighth aspect, the present application provides a computing device, comprising a processor and a memory; the memory stores computer executable instructions; and the processor is configured to execute the computer executable instructions stored in the memory, so that the computing device performs any one of the methods provided in the second aspect.

[0047] In a ninth aspect, the present application provides a computing device, which is a second device, comprising a host and a physical card connected to the host;

[0048] The host is configured to communicate with a fifth device;

[0049] The physical card is configured to, in response to a first request sent by the physical card of the first device, establish or disconnect a communication link between the physical card of the second device and the physical card of the first device; the first request is sent by the physical card of the first device based on a first notification received by the first device; the first notification is sent by a fifth device when it is determined that the number of devices in a first device cluster needs to be adjusted; the first notification is used to instruct the first device and the second device to adjust a communication connection relationship; the first device belongs to the first device cluster, and the second device belongs to a second device cluster; the devices in the first device cluster are used to perform a first stage of a model inference process; and the devices in the second device cluster are used to perform a second stage of the model inference process.

[0050] In a tenth aspect, the present application provides a computing device cluster, including a fifth device, at least one first device, and at least one second device. The fifth device can be the computing device provided in the eighth aspect, the first device can be the computing device provided in the seventh aspect, and the second device can be the computing device provided in the ninth aspect.

[0051] In an eleventh aspect, the present application provides a computer readable storage medium, which stores computer executable instructions for causing a computer to perform any of the methods provided in the first aspect.

[0052] In a twelfth aspect, the present application provides a computer readable storage medium, which stores computer executable instructions for causing a computer to perform any of the methods provided in the second aspect.

[0053] In a thirteenth aspect, the present application provides a computer readable storage medium, which stores computer executable instructions for causing a computer to perform any of the methods provided in the third aspect.

[0054] In a fourteenth aspect, the present application provides a computer program product, which contains computer executable instructions for causing a computer to perform any of the methods provided in the first aspect.

[0055] In a fifteenth aspect, the present application provides a computer program product, which contains computer executable instructions for causing a computer to perform any of the methods provided in the second aspect.

[0056] In a sixteenth aspect, the present application provides a computer program product, which contains computer executable instructions for causing a computer to perform any of the methods provided in the third aspect.

[0057] The technical effects achieved by any one of the second aspect to the sixteenth aspect described above can refer to the description of the beneficial effects of the first aspect described above, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0058] FIG. 1 is a schematic diagram of an application scenario of an embodiment of the present application;

[0059] FIG. 2 is a schematic diagram of a computing device cluster provided by an embodiment of the present application;

[0060] FIG. 3 is an interaction diagram between devices in a process of adding an incremental node provided by an embodiment of the present application;

[0061] FIG. 4 is a schematic diagram of a computing device cluster after adding an incremental node provided by an embodiment of the present application;

[0062] FIG. 5 is an interaction diagram between devices in a process of adding a full node provided by an embodiment of the present application;

[0063] FIG. 6 is a schematic diagram of a computing device cluster after adding a full node provided by an embodiment of the present application;

[0064] FIG. 7 is an interaction diagram between devices in a process of reducing an incremental node provided by an embodiment of the present application;

[0065] FIG. 8 is an interaction diagram between devices in a process of reducing a full node provided by an embodiment of the present application;

[0066] FIG. 9 is an interaction diagram between devices in a process of isolating a faulty incremental node provided by an embodiment of the present application;

[0067] FIG. 10 is an interaction diagram between devices in a process of isolating a faulty full node provided by an embodiment of the present application;

[0068] FIG. 11 is an interaction diagram between devices in a process of converting an incremental node into a full node provided by an embodiment of the present application;

[0069] FIG. 12 is a schematic diagram of a computing device cluster after converting an incremental node into a full node provided by an embodiment of the present application;

[0070] FIG. 13 is an interaction diagram between devices in a process of converting a full node into an incremental node provided by an embodiment of the present application;

[0071] FIG. 14 is a schematic diagram of a computing device cluster after converting a full node into an incremental node provided by an embodiment of the present application;

[0072] FIG. 15 is a structural schematic diagram of a device cluster running apparatus provided by an embodiment of the present application;

[0073] FIG. 16 is a structural schematic diagram of another device cluster running apparatus provided by an embodiment of the present application;

[0074] FIG. 17 is a structural schematic diagram of another device cluster running apparatus provided by an embodiment of the present application;

[0075] FIG. 18 is a structural schematic diagram of a computing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the drawings. The terms used in the implementation part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0077] Before introducing the specific solutions provided by the embodiments of the present application, some terms in the present application are explained and described, so as to facilitate understanding by those skilled in the art, and the terms in the present application are not limited.

[0078] (1) LLM: LLM is a large-scale language model, which is composed of an artificial neural network with tens of billions of weights or more parameters, and is pre-trained using unsupervised learning on a large amount of unlabeled corpus to generate a general LLM model, which can then be fine-tuned under different specific tasks. Illustratively, LLM can be used for generative tasks such as text generation, machine translation, dialogue generation, etc. LLM is usually based on the Transformer architecture. The Transformer architecture is one of the key components of LLM, and the core of the Transformer architecture is a deep neural network structure including multiple encoders and decoders. LLM can complete generative tasks by stacking multiple layers of Transformer architecture.

[0079] (2) Cloud computing: Cloud computing is a kind of distributed computing, which refers to dividing a huge data computing program into countless small units through a network "cloud", and then processing and analyzing these small program units through a computing device cluster composed of multiple computing devices to obtain results and return them to users (including enterprise users and individual users). The computing device cluster composed of multiple computing devices can be referred to as a cloud computing system. The computing devices in the cloud computing system can be referred to as cloud servers. In the embodiments of the present application, the data computing program can be a computer program for inference using LLM.

[0080] In the embodiments of the present application, "multiple" refers to two or more. In view of this, "multiple" can also be understood as "at least two" in the embodiments of the present application. "At least one" can be understood as one or more, for example, one, two or more. For example, including at least one means including one, two or more, and does not limit which ones are included, for example, including at least one of A, B and C, then the included can be A, B, C, A and B, A and C, B and C, or A and B and C. "And / or" describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / ", unless otherwise specified, generally indicates that the front and rear associated objects are in an "or" relationship.

[0081] Unless otherwise stated, the ordinal numbers "first", "second", etc. mentioned in the embodiments of the present application are used to distinguish a plurality of objects, and are not used to limit the order, time sequence, priority or importance of the plurality of objects.

[0082] With the development of AI technology, LLM is applied more and more widely in people's daily work and life. LLM has very strong reasoning ability and can process and generate natural language text. LLM usually has tens of billions or even hundreds of billions of parameters, so it can handle complex natural language understanding and generation tasks.

[0083] The reasoning process of LLM mainly includes two stages: pre-filling stage and decoding stage. The pre-filling stage is used to calculate based on the user request text to obtain the intermediate state. The decoding stage is used to automatically regress to generate a predicted value based on the intermediate state output by the pre-filling stage until the stop condition is met, and finally output the reasoning result. When performing the reasoning process of LLM, the pre-filling task of the pre-filling stage can be performed by the first device cluster, and the decoding task of the decoding stage can be performed by the second device cluster, so as to separate the pre-filling task and the decoding task for execution, and reduce the interference between them.

[0084] In some application scenarios, LLM can be deployed in a cloud computing device cluster, which can also be referred to as a cloud computing system. FIG. 1 shows a schematic diagram of an application scenario of the embodiments of the present application. As shown in FIG. 1, a plurality of terminal devices are connected with the computing device cluster 100, and FIG. 1 only exemplarily shows three terminal devices: terminal device 210, terminal device 220 and terminal device 230. In actual application scenarios, the number of terminal devices can be more than three. The terminal device can be, but is not limited to, a computer, a tablet computer, a vehicle-mounted terminal, a smart medical terminal, a smart power grid terminal, a mobile terminal, etc. The terminal device can be connected with the cloud computing device cluster 100 through a network, and send a user request text to the cloud computing device cluster 100.

[0085] The computing device cluster 100 receives a user request text sent by a terminal device of a user, performs a model inference process based on the user request text, and feeds back an inference result to the terminal device of the user. The computing device cluster 100 includes a plurality of computing devices, which can be distributed in the same region or in different regions. The plurality of computing devices included in the computing device cluster 100 can be distributed in the same availability zone (AZ) or in different AZs. Similarly, the plurality of computing devices included in the computing device cluster 100 can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. The plurality of computing devices can be any combination of servers, application specific integrated circuits (ASICs), programmable logic devices (PLDs), complex programmable logical devices (CPLDs), field-programmable gate arrays (FPGAs), generic array logic (GALs), and the like.

[0086] As shown in FIG. 1, the computing device cluster 100 can include a device cluster A and a device cluster B, and can also include a computing device C and an idle device. Among them, the device cluster A includes a plurality of computing devices such as a computing device A1 and a computing device A2, and the plurality of computing devices in the device cluster A can be a plurality of cloud servers. The device cluster B includes a plurality of computing devices such as a computing device B1, a computing device B2, and a computing device B3, and the plurality of computing devices in the device cluster B can also be a plurality of cloud servers. The plurality of computing devices in the device cluster A can be used to perform a pre-padding stage of a model inference process, and the plurality of computing devices in the device cluster B can be used to perform a decoding stage of the model inference process. In the following introduction, the computing device used to perform the pre-padding stage of the model inference process can be referred to as a full node, and the device cluster A includes a plurality of full nodes; the computing device used to perform the decoding stage of the model inference process can be referred to as an incremental node, and the device cluster B includes a plurality of incremental nodes. The ratio of the full nodes to the incremental nodes in the computing device cluster 100 shown in FIG. 1 is 2:3. One full node can establish a communication connection with a plurality of incremental nodes, for example, the computing device A1 can establish a communication connection with the computing device B1, the computing device B2, and the computing device B3. One incremental node can establish a communication connection with a plurality of full nodes. For example, the computing device B1 can establish a communication connection with the computing device A1 and the computing device A2.

[0087] The computing device C can correspond to the fifth device described above, and is used to monitor the load condition and running state of the computing devices in the first device cluster and the second device cluster, and control the running of the computing devices in the first device cluster and the second device cluster, so the computing device C can be referred to as a master device or a control device. The computing device C can be one or more cloud servers, and is used to run a cloud management platform provided by a cloud vendor. The cloud management platform can be connected to a terminal device of a user through the Internet, receive a user request text sent by the terminal device of the user, distribute the user request text to one or more devices in the device cluster A, perform a pre-padding task on the user request text by the one or more devices in the device cluster A, obtain an intermediate state, and transmit the intermediate state to one or more devices in the device cluster B. The one or more devices in the device cluster B perform a decoding task on the obtained intermediate state, obtain an inference result, and feed back the inference result to the terminal device of the user through the cloud management platform in the computing device C.

[0088] In actual application, the user request quantity received by the computing device cluster can be short-time surges, in order to cope with the computing power required when the user request quantity surges, a large number of computing devices need to be configured in the device cluster A and the device cluster B. At most times, the user request quantity is usually small, at this time, the device cluster A or the device cluster B will have a large amount of redundant resources, resulting in resource waste.

[0089] Based on this, an equipment cluster running method is provided in the embodiments of the present application. In the embodiments of the present application, the structure of the computing equipment in the computing equipment cluster 100 can be as shown in FIG. 2. The computing equipment 120 shown in FIG. 2 can include a host and multiple physical cards, such as physical card a0-physical card a7. Among them, the host 121 can include a processor 1211, a memory 1212 and a communication interface 1213. The communication interface 1213 is used for communication with devices other than the host 121. For example, the communication interface 1213 can be used for the computing equipment 120 to communicate with the computing equipment C, receive the AI task sent by the computing equipment C, which can be generated by the computing equipment C based on the received user request text; or the communication interface 1213 can be used for transmitting the inference result of the AI task to the computing equipment C.

[0090] The processor 1211 is the operation core and control core of the host 121, and can be a central processing unit (CPU), or other specific integrated circuits. The processor 1211 can also be other general-purpose processors, digital signal processors (DSPs), ASICs, FPGAs or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. In practical applications, the host 121 can be provided with multiple processors. The processor 1211 includes one or more processor cores. An operating system and other software programs are installed in the processor 1211, so that the processor 1211 can realize access to the memory 1212 and various bus devices, and the bus device refers to a device that transmits data with the processor 1211 through a bus, such as a physical card, etc.

[0091] The processor 1211 is connected with the memory 1212 through a double data rate (DDR) bus or other types of buses. The memory 1212 is the main memory of the host 121. The memory 1212 can be used to store various running software in the operating system, data received from other electronic devices, and intermediate states or inference results of calculations, etc. In order to improve the access speed of the processor 1211, the memory 1212 needs to have the advantage of fast access speed. Exemplarily, the memory 1212 can adopt a dynamic random access memory (DRAM). In addition to the DRAM, the memory 1212 can also be other random access memories, such as a static random access memory (SRAM), etc. In addition, the memory 1212 can also be a read only memory (ROM). As for the read only memory, for example, it can be a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), etc. The number and type of the memory 1212 are not limited in the embodiment.

[0092] The physical card is used to perform the model inference process of the AI task. The processor 1211 sends the received AI task to the physical card, and the physical card performs the model inference process of the AI task. Exemplarily, the physical card can be, but is not limited to, a neural-network processing unit (NPU), a data processing unit (DPU), a graphics processing unit (GPU), etc. The physical cards a0-a7 can be directly inserted into the card slot on the mainboard of the host 121, and exchange data with the processor 1211 through a bus. The bus can be a peripheral component interconnect extended (PCIe) bus, a compute express link (CXL) bus, a universal serial bus (USB) protocol bus, or a bus of other protocols.

[0093] The computing device 130 shown in FIG. 2 can include a host 131 and multiple physical cards, such as physical card b0 to physical card b7. The host 131 can include a processor 1311, a memory 1312, and a communication interface 1313. The structure and role of the host 131 can refer to the host 121 for implementation, and will not be described here.

[0094] The physical cards b0 to b7 in the computing device 130 have a one-to-one correspondence with the physical cards a0 to a7 in the computing device 120. For example, the physical card b0 corresponds to the physical card a0, that is, a communication link is established between the physical card a0 and the physical card b0, the physical card b1 corresponds to the physical card a1, that is, a communication link is established between the physical card a1 and the physical card b1, and so on. The physical card b7 corresponds to the physical card a7, that is, a communication link is established between the physical card a7 and the physical card b7. The communication link between the physical cards can be referred to as a point-to-point communication link.

[0095] In some embodiments, the computing device 120 shown in FIG. 2 can be any one of the computing devices in the device cluster A shown in FIG. 1, and the computing device 130 shown in FIG. 2 can be any one of the computing devices in the device cluster B shown in FIG. 1. The physical cards in the computing devices are used to perform model inference processes of AI tasks. The computing device 120 and the computing device 130 shown in FIG. 2 each include 8 physical cards. Taking the computing device including 8 physical cards as an example, the computing device 120 is used to perform a pre-padding task in the model inference process, and the pre-padding task can be divided into 8 parts, and each physical card in the computing device 120 is used to perform a pre-padding task. The computing device 130 is used to perform a decoding task in the model inference process, and the decoding task can be divided into 8 parts, and each physical card in the computing device 130 is used to perform a decoding task. Taking the physical card a0 and the physical card b0 as an example, the physical card a0 corresponds to the physical card b0, and the physical card a0 can transmit the intermediate state obtained by performing the pre-padding task to the physical card b0, and the physical card b0 performs the decoding task based on the intermediate state transmitted by the physical card a0.

[0096] In other embodiments, the computing device 120 shown in FIG. 2 can be any one of the computing devices in the device cluster B shown in FIG. 1, and the computing device 130 shown in FIG. 2 can be any one of the computing devices in the device cluster A shown in FIG. 1. Still taking the physical card a0 and the physical card b0 as an example, in this embodiment, the physical card b0 corresponds to the physical card a0, and the physical card b0 can transmit the intermediate state obtained by performing the pre-padding task to the physical card a0, and the physical card a0 performs the decoding task based on the intermediate state transmitted by the physical card b0.

[0097] In the embodiments of the present application, the computing device C can also be used to adjust the number of devices in the first device cluster and the second device cluster. The computing device C can dynamically adjust the number of devices included in the device cluster A or the device cluster B according to the real-time load of the device cluster A and the device cluster B. When the computing device C determines to adjust the number of devices in the first device cluster, the computing device C can send a first notification to the first device, and the first notification is used to instruct the first device to adjust the communication connection relationship with the second device. The first device belongs to the first device cluster, and the second device belongs to the second device cluster. The first device cluster can be the device cluster A, and the second device cluster can be the device cluster B; or the first device cluster can be the device cluster B, and the second device cluster can be the device cluster A. The devices in the first device cluster are used to execute the first stage of the model inference process; and the devices in the second device cluster are used to execute the second stage of the model inference process. The first stage can be understood as the pre-filling stage described above, and the second stage can be understood as the decoding stage described above. After receiving the first notification sent by the computing device C, the first device can establish or disconnect the communication link between the physical card of the first device and the physical card of the second device based on the first notification. Through the above process, the computing device cluster can dynamically adjust the number of devices included in the device cluster A or the device cluster B according to the change of the user request quantity, without the need to set a large number of redundant resources in the device cluster A and the device cluster B. For example, a small number of computing devices can be first set in the device cluster A and the device cluster B to process daily user requests, and when the user request quantity increases sharply, the computing devices can be added to the device cluster A or the device cluster B to share the load of the existing computing devices in the device cluster A or the device cluster B, so as to speed up the processing of user requests. When the user request quantity decreases, the number of computing devices in the device cluster A or the device cluster B can be reduced, and the computing devices withdrawn from the device cluster A or the device cluster B can be used to process other businesses, so as to reduce resource waste.

[0098] When all the nodes in the device cluster A establish the communication connection with the incremental nodes in the device cluster B, the client / server mode can be adopted, and the client unidirectionally initiates the chain establishment request to the server, and the server only needs to listen to the chain establishment request.

[0099] In some embodiments, the all nodes in the device cluster A can act as the server, and the incremental nodes in the device cluster B can act as the client, and the incremental nodes send the request to the all nodes.

[0100] FIG. 3 exemplarily shows an interaction flow diagram between the computing device C and each computing device in the process of adding the incremental nodes. As shown in FIG. 3, the process can include the following steps:

[0101] S301, the computing device C determines to increase the number of computing devices in the device cluster B.

[0102] During the running of the device cluster A and the device cluster B, the computing device C can monitor the load of each computing device in the device cluster A and the device cluster B in real time, and determine whether the number of computing devices in the device cluster A needs to be increased or whether the number of computing devices in the device cluster B needs to be increased according to the load of each computing device when the user request volume increases.

[0103] For example, the computing device C can monitor the load of each computing device in the device cluster B in real time, and when it is monitored that the load of the device cluster B exceeds a first load threshold, it can be determined that the number of computing devices in the device cluster B needs to be increased. The load of the device cluster B can be an average of the loads of the plurality of computing devices included in the device cluster B, or the load of the device cluster B can be a minimum value in the loads of the plurality of computing devices included in the device cluster B. The first load threshold can be a preset value. For example, the first load threshold can be 70% or 80%.

[0104] S302, the computing device C sends a link establishment notification to the computing device B4.

[0105] When the computing device C determines that the number of computing devices in the device cluster B needs to be increased, the computing device B4 can be notified to join the device cluster B. The computing device B4 can be a randomly selected idle device from the idle devices of the computer cluster by the computing device C. The computing device B4 is deployed with an LLM model, and the computing device C can control the computing device B4 to start the running of the LLM model and match a corresponding computing device for the computing device B4 in the device cluster A, and send a link establishment notification to the computing device B4. Assuming that the device cluster A includes the computing device A1 and the computing device A2, the link establishment notification is used to instruct the computing device B4 to establish a communication connection with the computing device A1 and / or the computing device A2.

[0106] This embodiment takes the link establishment notification used to instruct the computing device B4 to establish a communication connection with the computing device A1 as an example. Assuming that the computing device B4 and the computing device A1 each include 8 physical cards. The link establishment notification can carry the correspondence between the 8 physical cards of the computing device B4 and the 8 physical cards of the computing device A1. For example, the computing device B4 includes 8 physical cards, which are physical card B 40 ~ physical card B 47 ; the computing device A1 includes 8 physical cards, which are physical card A 10 ~ physical card A 17 ; the link establishment notification can carry the information of the 8 physical cards of the computing device B4 and the information of the 8 physical cards of the computing device A1, wherein the information of the physical card B 40 corresponds to the information of the physical card A 10 , the information of the physical card B 41information of the physical card A 11 information of the physical card B 42 information of the physical card A 12 information of the physical card B 47 information of the physical card A 17 information of the physical card B. The information of each physical card can be an internet protocol (IP) address of the physical card or a port identifier of the physical card.

[0107] It should be noted that the above embodiments take the example that the computing device B4 and the computing device A1 each include 8 physical cards, and in other embodiments, the computing device B4 and the computing device A1 can include the same number of physical cards, which can be less than or more than 8, or the computing device B4 and the computing device A1 can include different numbers of physical cards. For example, the computing device B4 includes 8 physical cards, which are physical card B 40 ~ physical card B 47 , and the computing device A1 includes 10 physical cards, which are physical card A 10 ~ physical card A 19 . The 8 physical cards (physical card A 10 ~ physical card A 17 ) in the computing device A1 have a one-to-one correspondence with the 8 physical cards (physical card B 40 ~ physical card B 47 ) of the computing device B4, and the remaining two physical cards (physical card A 18 and physical card A 19 ) can not perform model inference services; or the 8 physical cards (physical card A 12 ~ physical card A 19 ) in the computing device A1 have a one-to-one correspondence with the 8 physical cards (physical card B 40 ~ physical card B 47 ) of the computing device B4, and the remaining two physical cards (physical card A 10 and physical card A 11 ) can not perform model inference services. For another example, the computing device B4 includes 5 physical cards, which are physical card B 40 ~ physical card B 44 , and the computing device A1 includes 4 physical cards, which are physical card A 10 ~ physical card A 13 . The 4 physical cards (physical card B 40 ~ physical card B 43 ) in the computing device B4 have a one-to-one correspondence with the 4 physical cards (physical card A 10 and physical card A 13) have a one-to-one correspondence, and the remaining one physical card (physical card B 44 ) can not perform model inference services.

[0108] In other embodiments, if the link establishment notification is used to indicate that the computing device B4 establishes a communication connection with the computing device A1 and the computing device A2, the computing device A2 includes 8 physical cards, which are physical card A 20 ~ physical card A 27 . The link establishment notification can carry two sets of correspondence relationships. One set of correspondence relationships is between the 8 physical cards of the computing device B4 and the 8 physical cards of the computing device A1. The other set of correspondence relationships is between the 8 physical cards of the computing device B4 and the 8 physical cards of the computing device A2. The correspondence relationship between the 8 physical cards of the computing device B4 and the 8 physical cards of the computing device A2 can include: the information of the physical card B 40 corresponds to the information of the physical card A 20 , the information of the physical card B 41 corresponds to the information of the physical card A 21 , the information of the physical card B 42 corresponds to the information of the physical card A 22 , and so on, and the information of the physical card B 47 corresponds to the information of the physical card A 27 .

[0109] S303, the computing device B4 establishes a communication connection with the computing device A1.

[0110] This embodiment is described by taking the link establishment notification used to indicate that the computing device B4 establishes a communication connection with the computing device A1 as an example. The computing device B4 receives the link establishment notification sent by the computing device C, and establishes a communication link between the physical cards of the computing device B4 and the physical cards of the computing device A1 based on the link establishment notification. For example, the computing device B4 receives the link establishment notification sent by the computing device C, and the host of the computing device B4 can send link establishment instructions to the 8 physical cards of the computing device B4 according to the correspondence relationship carried by the link establishment notification. The link establishment instruction sent to any kind of physical card carries the information of the physical card of the computing device A1 corresponding to the physical card. For example, the host of the computing device B4 can carry the information of the physical card A 40 in the link establishment instruction sent to the physical card B 10 , to indicate that the physical card B 40 corresponds to the physical card A 10The communication link can be a remote direct memory access (RDMA) link, also referred to as a point-to-point communication link or a Raw point-to-point communication link in an embodiment. In another embodiment, the communication link can also be other data transmission links, such as a direct memory access (DMA) link, etc. Similarly, the host of the computing device B4 sends a link establishment instruction to the physical card B 41 The link establishment instruction sent can carry the information of the physical card A 11 , so as to instruct the physical card B 41 to establish a communication link with the physical card A 11 , and so on.

[0111] The following takes the establishment of a communication link between the physical card B 40 and the physical card A 10 as an example to illustrate the establishment process of the communication link.

[0112] The process can include the following steps: the physical card B 40 receives the link establishment instruction sent by the host of the computing device B4, and sends a link establishment request to the physical card A 10 based on the information of the physical card A 10 carried in the link establishment instruction, the link establishment request carrying the information of the physical card A 10 and the link information of the link to be established, the physical card A 10 receives the link establishment request sent by the physical card B 40 , and establishes a point-to-point communication link with the physical card B 40 based on the link information carried in the link establishment request. The link information can be the identification information of the link. Further, the computing device B4 starts the running of the LLM model, and when receiving the decoding task of the LLM model, can execute the decoding task of the LLM model through the eight physical cards, each physical card starting a user application process for executing a decoding task, and the user application process starting a point-to-point communication application programming interface (API) through a starting function HcclRawOpen(), the point-to-point communication API also referred to as HcclRawAPI, and the HcclRawAPI constructing the data structure required by the program running. The physical card B 40 receives the link establishment instruction sent by the host of the computing device B4, and transmits the IP address or port address of the physical card A 10 carried in the link establishment instruction to the HcclRawAPI of the physical card B 40 through a connection function HcclRawConnenct(), and the physical card B 40HcclRawAPI of the physical card A 10 transmits a link establishment request, which can be a transmission control protocol (TCP) request, and the link establishment request can carry an IP address or a port address of the physical card A 10 , link information of a link to be established, and the like.

[0113] The physical card A 10 When starting a user application process, a local IP address or a port address can be transmitted to the HcclRawAPI of the physical card A 10 through a binding function HcclBind(), and a listening function HcclRawListen() of the HcclRawAPI of the physical card A 10 is started. Therefore, the HcclRawAPI of the physical card A 10 can listen to a link establishment request sent by the HcclRawAPI of the physical card B 40 , determine that the IP address or the port address carried in the link establishment request is the IP address or the port address of the physical card A 10 , and the physical card A 10 can establish a point-to-point communication link with the physical card B 40 based on the link information carried in the link establishment request.

[0114] The process of establishing a point-to-point communication link between other physical cards can be performed with reference to the process of establishing a point-to-point communication link between the physical card B 40 and the physical card A 10 , and will not be described here. After the computing device B4 establishes a communication connection with the computing device A1, the computing device C can save the matching relationship between the computing device B4 and the computing device A1.

[0115] After the communication link is established, the physical card of the computing device B4 can receive a processing result of a pre-population stage transmitted by the corresponding physical card of the computing device A1 through the RDMA link. In an optional embodiment, for a multi-round dialogue scenario, in the process of reasoning the second round of dialogue, the physical card of the computing device B4 can transmit a processing result of a decoding stage in the reasoning process of the first round of dialogue to the corresponding physical card of the computing device A1 through the RDMA link, so that the physical card of the computing device A1 can perform a pre-population stage corresponding to the second round of dialogue based on the processing result of the decoding stage of the first round of dialogue.

[0116] Exemplarily, the physical card B 40 receives the physical card A 10Taking the pre-filling stage processing result as an example, this describes the process by which the physical card of computing device B4 receives the pre-filling stage processing result transmitted from the corresponding physical card of computing device A1. Physical card A 10 The user application process executes the pre-filling task of the LLM model and obtains the processing results of the pre-filling stage. (Physical card B) 40 From physical card A 10 Retrieve the processing results from the pre-filling stage. For example, the processing results from the pre-filling stage may include key-value pairs. Physical Card A 10 The user application process transmits the key value (count) to the physical card A via the transfer function HcclRawSend(). 10 HcclRawAPI, Physical Card A 10 The HcclRawAPI encapsulates the count into an envelope, and then transmits the envelope to the physical card B via RDMA asynchronous transfer. 40 Physical Card B 40 The HcclRawAPI can obtain the count by pulling the envelope and setting the information flag to 1. Physical Card B 40 When the user application process detects that the flag is 1, it retrieves the count from the physical card B using the function HcclRawGetCount(). 40 The HcclRawAPI is used to obtain the count, and a read function HcclRawmrecv() is generated based on the count. The physical card B is then called through the read function HcclRawmrecv(). 40 The HcclRawAPI enables physical card B 40 HcclRawAPI to physical card A 10 Sending an RDMA read data request. Physical Card A 10 The HcclRawAPI receives an RDMA read data request and determines the value corresponding to the key based on the count carried in the RDMA read data request. (Physical card B) 40 HcclRawAPI from physical card A 10 HcclRawAPI retrieves the value, physical card B 40 User application processes from physical card B 40 The value is obtained from the HcclRawAPI; at this point, physical card B... 40 Obtain Physical Card A 10 The processing results of the pre-filling stage obtained from performing the pre-filling task. Physical Card B 40 The decoding task of the LLM model can be performed based on the processing results of the pre-filling stage.

[0117] With physical card B40 to the physical card A 10 The process of transmitting the processing result of the decoding stage of the first round of dialog by the physical card of the computing device B4 to the corresponding physical card of the computing device A1 is illustrated by taking the processing result of the decoding stage of the first round of dialog as an example. The user application process of the physical card B 40 performs the decoding task of the LLM model for the first round of dialog to obtain the processing result of the decoding stage of the first round of dialog, and sends the processing result of the decoding stage of the first round of dialog to the physical card A 10 . Illustratively, the user application process of the physical card B 40 transmits the processing result of the decoding stage of the first round of dialog to the HcclRaw API of the physical card B 40 by using the transmission function HcclRawSend(). The HcclRaw API of the physical card B 40 may transmit the processing result of the decoding stage of the first round of dialog to the HcclRaw API of the physical card A 10 by using the RDMA asynchronous transmission. The user application process of the physical card A 10 may obtain the processing result of the decoding stage of the first round of dialog from the HcclRaw API of the physical card A 10 . The physical card A 10 may perform the pre-filling task of the LLM model for the second round of dialog based on the obtained processing result of the decoding stage of the first round of dialog.

[0118] The above describes the process of establishing a communication connection between the computing device B4 and the computing device A1. The process of establishing a communication connection between the computing device B4 and the computing device A2 can be performed by referring to the process of establishing a communication connection between the computing device B4 and the computing device A1, which is not described herein again. Through the above process, the number of computing devices in the device cluster B is increased. As shown in FIG. 4, compared with the device cluster B shown in FIG. 1, the device cluster B includes four computing devices, i.e., the computing device B1, the computing device B2, the computing device B3 and the computing device B4. The computing device B4 can share the computing task of the decoding stage performed by the computing device B1, the computing device B2 and the computing device B3, thereby reducing the load of each computing device in the device cluster B.

[0119] The embodiment shown in FIG. 3 describes the interaction process between the computing device C and each computing device in the process of adding an incremental node. FIG. 5 illustrates an interaction flow diagram between the computing device C and each computing device in the process of adding a full node. As shown in FIG. 5, the process can include the following steps:

[0120] S501, the computing device C determines to increase the number of computing devices in the device cluster A.

[0121] During the running of the device cluster A and the device cluster B, the computing device C can monitor the load of each computing device in the device cluster A and the device cluster B in real time.

[0122] For example, the computing device C can monitor the load of each computing device in the device cluster A in real time, and when it is monitored that the load of the device cluster A exceeds the second load threshold, it can be determined that the number of computing devices in the device cluster A needs to be increased. The load of the device cluster A can be the average of the loads of the plurality of computing devices included in the device cluster A, or the load of the device cluster A can be the minimum value of the loads of the plurality of computing devices included in the device cluster A. The second load threshold can be a preset value, and the second load threshold can be the same as the first load threshold or different from the first load threshold.

[0123] S502, the computing device C sends a start notification to the computing device A3.

[0124] When the computing device C determines that the number of computing devices in the device cluster A needs to be increased, it can notify the computing device A3 to join the device cluster A. The computing device A3 can be a randomly selected idle device from the idle devices of the computer cluster by the computing device C. The computing device A3 is deployed with an LLM model, and the computing device C can send a start notification to the computing device A3.

[0125] S503, the computing device A3 starts the listening function of the physical card based on the received start notification.

[0126] In some embodiments, if the LLM model deployed in the computing device A3 is not running, the computing device A3 can start the running of the LLM model based on the received start notification, and start the listening function of the physical card of the computing device A3, and wait to receive the link establishment request sent by the physical card of the computing device in the device cluster B.

[0127] In other embodiments, if the LLM model deployed in the computing device A3 is in a running state, the computing device A3 can start the listening function of the physical card of the computing device A3 based on the received start notification, and wait to receive the link establishment request sent by the physical card of the computing device in the device cluster B.

[0128] S504, the computing device C sends a link establishment notification to the computing device B1.

[0129] The computing device C matches the corresponding computing device for the computing device A3 in the device cluster B. For example, assuming that the device cluster B includes three computing devices, the computing device B1, the computing device B2 and the computing device B3, the computing device C determines that the computing device A3 matches the computing device B1, and sends a link establishment notification to the computing device B1. The link establishment notification is used to instruct the computing device B1 to establish a communication connection with the computing device A3. In other embodiments, if the computing device C determines that the computing device B1, the computing device B2 and the computing device B3 all match the computing device A3, the link establishment notification can be sent to the computing device B1, the computing device B2 and the computing device B3 respectively, to instruct the corresponding computing device to establish a communication connection with the computing device A3.

[0130] This embodiment is described by taking the example that the computing device C determines that the computing device A3 matches the computing device B1. Assuming that the computing device B1 and the computing device A3 each include eight physical cards. The computing device C sends a link establishment notification to the computing device B1, and the link establishment notification can carry the correspondence between the eight physical cards of the computing device B1 and the eight physical cards of the computing device A3. For example, the computing device B1 includes eight physical cards, which are physical card B1, physical card B2, physical card B3, physical card B4, physical card B5, physical card B6, physical card B7 and physical card B8 respectively. The computing device A3 includes eight physical cards, which are physical card A1, physical card A2, physical card A3, physical card A4, physical card A5, physical card A6, physical card A7 and physical card A8 respectively. The link establishment notification can carry the information of the eight physical cards of the computing device B1 and the information of the eight physical cards of the computing device A3, wherein the information of the physical card B1 corresponds to the information of the physical card A1, the information of the physical card B2 corresponds to the information of the physical card A2, the information of the physical card B3 corresponds to the information of the physical card A3, the information of the physical card B4 corresponds to the information of the physical card A4, the information of the physical card B5 corresponds to the information of the physical card A5, the information of the physical card B6 corresponds to the information of the physical card A6, the information of the physical card B7 corresponds to the information of the physical card A7, and the information of the physical card B8 corresponds to the information of the physical card A8. 10 17 30 37 10 30 11 31 17 37

[0131] S505, the computing device B1 establishes a communication connection with the computing device A3.

[0132] Based on the link establishment notification received by the computing device B1, the physical cards of the computing device B1 and the physical cards of the computing device A3 correspondingly establish point-to-point communication links. The process of establishing a communication connection between the computing device B1 and the computing device A3 can be performed by referring to the process of establishing a communication connection between the computing device B4 and the computing device A1 in the above embodiment, which will not be described herein. After the computing device B1 establishes a communication connection with the computing device A3, the computing device C can save the matching relationship between the computing device B4 and the computing device A1.

[0133] ​​​​​​​​​​After the communication link is established, the physical card of the computing device B1 can receive the processing result of the pre-population stage transmitted by the corresponding physical card of the computing device A3 through the RDMA link. In an optional embodiment, for the scenario of multiple rounds of dialog, during the inference of the second round of dialog, the physical card of the computing device B1 can transmit the processing result of the decoding stage in the inference process of the first round of dialog to the corresponding physical card of the computing device A3 through the RDMA link, so that the physical card of the computing device A3 can perform the pre-population stage corresponding to the second round of dialog based on the processing result of the decoding stage of the first round of dialog. The process in which the physical card of the computing device B1 receives the processing result of the pre-population stage transmitted by the corresponding physical card of the computing device A3 can refer to the process of the physical card B 40 The physical card A 10 transmits the processing result of the pre-population stage. The process in which the physical card of the computing device B1 transmits the processing result of the decoding stage in the inference process of the first round of dialog to the corresponding physical card of the computing device A3 can refer to the process of the physical card B 40 The physical card A 10 transmits the processing result of the decoding stage of the first round of dialog, which will not be repeated here.

[0134] The above describes the process of establishing the communication connection between the computing device A3 and the computing device B1. The process of establishing the communication connection between the computing device A3 and the computing device B2 and the computing device B3 can refer to the process of establishing the communication connection between the computing device A3 and the computing device B1, which will not be repeated here. Through the above process, the number of computing devices in the device cluster A is increased. As shown in FIG. 6, compared with the device cluster A shown in FIG. 1, the device cluster A includes two computing devices, which becomes three computing devices including the computing device A1, the computing device A2 and the computing device A3. The computing device A3 can share the computing task of the pre-population stage performed by the computing device A1 and the computing device A2, thereby reducing the load of each computing device in the device cluster A.

[0135] In the embodiments of the present application, when the user request volume decreases, the number of computing devices in the device cluster A or the device cluster B can also be reduced according to the load of each computing device in the device cluster A and the device cluster B.

[0136] FIG. 7 exemplarily shows an interaction flow diagram between the computing device C and each computing device in the process of reducing the incremental node. As shown in FIG. 7, the process can include the following steps:

[0137] S701, the computing device C determines to reduce the number of computing devices in the device cluster B.

[0138] During the running of the device cluster A and the device cluster B, the computing device C can monitor the load of each computing device in the device cluster A and the device cluster B in real time, and determine whether the number of computing devices in the device cluster A needs to be reduced or whether the number of computing devices in the device cluster B needs to be reduced according to the load of each computing device when the user request volume decreases.

[0139] For example, the computing device C can monitor the load of each computing device in the device cluster B in real time, and when it is monitored that the load of the device cluster B is less than a third load threshold, it can be determined that the number of computing devices in the device cluster B needs to be reduced. The load of the device cluster B can be an average of the loads of the plurality of computing devices included in the device cluster B, or the load of the device cluster B can be the maximum value in the loads of the plurality of computing devices included in the device cluster B. The third load threshold can be a preset value, and the third load threshold is less than the first load threshold described above. For example, the third load threshold can be 60% or 50%.

[0140] S702, the computing device C sends a disconnection notification to the computing device B4.

[0141] When the computing device C determines that the number of computing devices in the device cluster B needs to be reduced, it can select one or more computing devices from the computing devices included in the device cluster B and send a disconnection notification to the selected computing device. For example, the computing device C selects the computing device B4 from the computing devices included in the device cluster B and sends a disconnection notification to the computing device B4 to inform the computing device B4 to exit from the device cluster B. The computing device B4 can be any computing device in the device cluster B.

[0142] The computing device C can determine the computing device in the device cluster A that matches the computing device B4. For example, if the computing device in the device cluster A that matches the computing device B4 is the computing device A1, the computing device C can carry the information of the physical card of the computing device A1 in the disconnection notification sent to the computing device B4 to instruct the computing device B4 to disconnect the communication connection with the computing device A1. If the computing device in the device cluster A that matches the computing device B4 is the computing device A1 and the computing device A2, the computing device C can send a disconnection notification to the computing device B4, and the disconnection notification is used to instruct the computing device B4 to disconnect the communication connection with the computing device A1 and the computing device A2.

[0143] The embodiment takes the case that the disconnection notification is used to indicate that the computing device B4 is disconnected from the computing device A1 as an example. It is assumed that the computing device B4 and the computing device A1 each include 8 physical cards. The disconnection notification can carry the correspondence between the 8 physical cards of the computing device B4 and the 8 physical cards of the computing device A1. The correspondence between the physical cards of the computing device B4 and the physical cards of the computing device A1 has been described above, and will not be described here again.

[0144] In other embodiments, if the disconnection notification is used to indicate that the computing device B4 is disconnected from the computing device A1 and the computing device A2, the computing device A2 also includes 8 physical cards. The disconnection notification can carry two sets of correspondence. One set of correspondence is the correspondence between the 8 physical cards of the computing device B4 and the 8 physical cards of the computing device A1, and the other set of correspondence is the correspondence between the 8 physical cards of the computing device B4 and the 8 physical cards of the computing device A2.

[0145] S703, the computing device B4 is disconnected from the computing device A1.

[0146] The embodiment takes the case that the disconnection notification is used to indicate that the computing device B4 is disconnected from the computing device A1 as an example. The computing device B4 receives the disconnection notification sent by the computing device C, and disconnects the communication link between the physical card of the computing device B4 and the physical card of the computing device A1 based on the disconnection notification. For example, the computing device B4 receives the disconnection notification sent by the computing device C. The host of the computing device B4 can send a disconnection instruction to each of the 8 physical cards of the computing device B4 according to the correspondence carried by the disconnection notification. The disconnection instruction sent to any physical card carries the information of the physical card of the computing device A1 corresponding to the physical card. 40 The disconnection instruction sent by the host of the computing device B4 to the physical card B 10 may carry the information of the physical card A 40 , to indicate that the physical card B 10 is disconnected from the physical card A 41 . The disconnection instruction sent by the host of the computing device B4 to the physical card B 11 may carry the information of the physical card A 41 , to indicate that the physical card B 11 is disconnected from the physical card A 40 , and so on.

[0147] The following takes the case that the physical card B 40 is disconnected from the physical card A 10 as an example to describe the process of disconnection.

[0148] The process can include the following steps: the physical card B 40Upon receiving the disconnection instruction sent by the host of the computing device B4, the physical card B sends a disconnection request to the physical card A based on the information carried in the disconnection instruction. 10 Upon receiving the disconnection request sent by the physical card B, the physical card A disconnects the point-to-point communication link with the physical card B based on the link information carried in the disconnection request. 10 Upon receiving the disconnection instruction sent by the host of the computing device B4, the physical card B sends a disconnection request to the physical card A based on the information carried in the disconnection instruction. 10 Upon receiving the disconnection request sent by the physical card B, the physical card A disconnects the point-to-point communication link with the physical card B based on the link information carried in the disconnection request. 10 Upon receiving the disconnection request sent by the physical card B, the physical card A disconnects the point-to-point communication link with the physical card B based on the link information carried in the disconnection request. 40 Upon receiving the disconnection request sent by the physical card B, the physical card A disconnects the point-to-point communication link with the physical card B based on the link information carried in the disconnection request. 40 Upon receiving the disconnection request sent by the physical card B, the physical card A disconnects the point-to-point communication link with the physical card B based on the link information carried in the disconnection request. 40 Upon receiving the disconnection request sent by the physical card B, the physical card A disconnects the point-to-point communication link with the physical card B based on the link information carried in the disconnection request. 10 Upon receiving the disconnection request sent by the physical card B, the physical card A disconnects the point-to-point communication link with the physical card B based on the link information carried in the disconnection request. 40 Upon receiving the disconnection request sent by the physical card B, the physical card A disconnects the point-to-point communication link with the physical card B based on the link information carried in the disconnection request.

[0149] The process of disconnecting the point-to-point communication link between the other physical cards can be performed by referring to the process of disconnecting the point-to-point communication link between the physical card B and the physical card A, which will not be described herein again. 40 Upon receiving the disconnection request sent by the physical card B, the physical card A disconnects the point-to-point communication link with the physical card B based on the link information carried in the disconnection request. 10 Upon receiving the disconnection request sent by the physical card B, the physical card A disconnects the point-to-point communication link with the physical card B based on the link information carried in the disconnection request.

[0150] In other embodiments, if the disconnection notification is used to indicate that the computing device B4 disconnects the communication connection with the computing device A1 and the computing device A2, the process of disconnecting the communication connection between the computing device B4 and the computing device A2 can be performed by referring to the process of disconnecting the communication connection between the computing device B4 and the computing device A1.

[0151] Through the above process, the number of computing devices in the device cluster B is reduced from four to three. The computing device B4 that exits the device cluster B does not need to perform the decoding task of the LLM model, and can be used to process other services, so that the computing resources in the cloud computing system can be effectively utilized and resource waste can be reduced.

[0152] The embodiment shown in FIG. 7 describes the interaction process between the computing device C and each computing device in the process of reducing the incremental node. FIG. 8 exemplarily shows a flowchart of the interaction process between the computing device C and each computing device in the process of reducing the full node. As shown in FIG. 8, the process can include the following steps:

[0153] S801, the computing device C determines to reduce the number of computing devices in the device cluster A.

[0154] During the running of the device cluster A and the device cluster B, the computing device C can monitor the load of each computing device in the device cluster A and the device cluster B in real time.

[0155] For example, the computing device C can monitor the load of each computing device in the device cluster A in real time, and when it is monitored that the load of the device cluster A is less than the fourth load threshold, it can be determined to reduce the number of computing devices in the device cluster A. The load of the device cluster A can be the average of the loads of the plurality of computing devices included in the device cluster A, or the load of the device cluster A can be the maximum value in the loads of the plurality of computing devices included in the device cluster A. The fourth load threshold can be a preset value, the fourth load threshold is less than the second load threshold, and the fourth load threshold can be the same as the third load threshold or different from the third load threshold.

[0156] S802, the computing device C determines that the computing device matched with the computing device A3 in the device cluster B includes the computing device B1.

[0157] When the computing device C determines to reduce the number of computing devices in the device cluster A, it can select one or more computing devices from the computing devices included in the device cluster A. For example, the computing device C selects the computing device A3 from the computing devices included in the device cluster A, and the computing device A3 can be understood as any one of the computing devices in the device cluster A. The computing device C determines the computing device matched with the computing device A3 in the device cluster B. In this embodiment, the computing device matched with the computing device A3 in the device cluster B includes the computing device B1, and in other embodiments, the computing device matched with the computing device A3 in the device cluster B can include one or more of the computing device B1, the computing device B2 and the computing device B3.

[0158] S803, the computing device C sends a disconnection notification to the computing device B1.

[0159] In this embodiment, the computing device matched with the computing device A3 in the device cluster B includes the computing device B1. The computing device C sends a disconnection notification to the computing device B1, and the disconnection notification carries the information of the physical card of the computing device A3, which is used to instruct the computing device B1 to disconnect the communication connection with the computing device A3. Assuming that the computing device B1 and the computing device A3 each include 8 physical cards. The disconnection notification can carry the correspondence between the 8 physical cards of the computing device B1 and the 8 physical cards of the computing device A3.

[0160] If the computing devices in the device cluster B that match the computing device A3 include the computing device B1, the computing device B2 and the computing device B3, the computing device C can send a disconnection notification to the computing device B1, the computing device B2 and the computing device B3 respectively; the disconnection notification sent to the computing device B1 is used to indicate that the computing device B1 disconnects the communication connection with the computing device A3, the disconnection notification sent to the computing device B2 is used to indicate that the computing device B2 disconnects the communication connection with the computing device A3, and the disconnection notification sent to the computing device B3 is used to indicate that the computing device B3 disconnects the communication connection with the computing device A3.

[0161] S804, the computing device B1 disconnects the communication connection with the computing device A3.

[0162] The computing device B1 receives the disconnection notification sent by the computing device C, and disconnects the communication link between the physical card of the computing device B1 and the physical card of the computing device A3 based on the disconnection notification. For example, the computing device B1 receives the disconnection notification sent by the computing device C, and the host of the computing device B1 can send a disconnection instruction to each of the eight physical cards of the computing device B1 according to the corresponding relationship carried in the disconnection notification. The disconnection instruction sent to any physical card carries the information of the physical card of the computing device A3 corresponding to the physical card. For example, the host of the computing device B1 sends the disconnection instruction to the physical card B 10 , which carries the information of the physical card A 30 , to indicate that the physical card B 10 disconnects the point-to-point communication link with the physical card A 30 . The host of the computing device B1 sends the disconnection instruction to the physical card B 11 , which carries the information of the physical card A 31 , to indicate that the physical card B 11 disconnects the point-to-point communication link with the physical card A 31 , and so on.

[0163] The process of disconnecting the point-to-point communication link between each group of physical cards can be performed according to the process of disconnecting the point-to-point communication link between the physical card B 40 and the physical card A 10 , which is not repeated here. After the computing device B1 disconnects the communication connection with the computing device A3, the computing device C can delete the matching relationship between the computing device B1 and the computing device A3 previously saved.

[0164] In some embodiments, after the computing device A3 monitors that the communication connection with each computing device in the device cluster B is disconnected, the computing device A3 can stop running the LLM model, or can keep the process of running the LLM model, but the process is idle and no longer performs model inference business, and almost does not occupy computing resources.

[0165] Through the foregoing process, the number of computing devices in the device cluster A is reduced from three to two. The computing device A3 that exits the device cluster A no longer needs to perform the model inference service and can be used to process other services, so that computing resources in the cloud computing system can be effectively utilized and resource waste is reduced.

[0166] In the embodiment of the present application, when a computing device in the device cluster A or the device cluster B sends a fault, only the computing device that sends the fault can be isolated.

[0167] As shown in FIG. 9, when a certain incremental node in the device cluster B sends a fault, in the process of isolating the incremental node, the interaction process between the computing device C and each computing device can include the following steps:

[0168] S901, the computing device C monitors that the computing device B3 in the device cluster B sends a fault, and determines that the computing device A1 in the device cluster A matches the computing device B3.

[0169] During the running of the device cluster A and the device cluster B, the computing device C can monitor the running of each computing device in the device cluster A and the device cluster B in real time. When the computing device C monitors that the computing device B3 in the device cluster B sends a fault, the computing device C can determine the computing device in the device cluster A that matches the computing device B3. In the embodiment, the computing device A1 in the device cluster A matches the computing device B3. In other embodiments, the computing device A1, the computing device A2, and the computing device A3 in the device cluster A can match the computing device B3.

[0170] S902, the computing device C sends a device fault notification to the computing device A1.

[0171] In the embodiment, the computing device A1 in the device cluster A that matches the computing device B3 is taken as an example for description. The computing device C sends a device fault notification to the computing device A1. The device fault notification carries information of the faulty device, that is, information of the computing device B3. The information of the computing device B3 can include information of the physical card of the computing device B3, which is used to instruct the computing device A1 to disconnect the communication connection with the computing device B3. Assuming that the computing device B3 and the computing device A1 each include eight physical cards. The device fault notification can carry a correspondence between the eight physical cards of the computing device B3 and the eight physical cards of the computing device A1.

[0172] S903, the computing device A1 disconnects the communication connection with the computing device B3 based on the received device fault notification.

[0173] When computing device A1 receives a device fault notification from computing device C, it retrieves the physical card information of computing device B3 carried in the notification and disconnects the point-to-point communication link between the physical cards of computing device A1 and computing device B3. For example, upon receiving the device fault notification from computing device C, the host of computing device A1 can, based on the correspondence between the physical cards of computing device B3 and computing device A1 carried in the notification, send disconnection commands to each of the eight physical cards of computing device A1. The disconnection command sent to any physical card carries the information of the corresponding physical card of computing device B3. For instance, the host of computing device A1 sends a disconnection command to physical card A... 10 The sent disconnect command can carry physical card B. 30 Information to instruct physical card A 10 Disconnect from physical card B 30 A point-to-point communication link between them. The host of computing device A1 communicates with physical card A. 11 The sent disconnect command can carry physical card B. 31 Information to instruct physical card A 11 Disconnect from physical card B 31 Point-to-point communication links between them, and so on.

[0174] With physical card A 10 Taking physical card A as an example, let's illustrate this. 10 Upon receiving a disconnect command from the host computer of computing device A1, based on the physical card B carried in the disconnect command... 30 Information found related to physical card B 30 The link information between them is deleted, and the saved link information is removed to disconnect physical card A. 10 With physical card B 30 Point-to-point communication links between them.

[0175] For the process of disconnecting the point-to-point communication link for other physical cards, please refer to Physical Card A. 10 Disconnect from physical card B 30 The execution process of the point-to-point communication link between them will not be described in detail here. At this point, computing device A1 disconnects from computing device B3. After computing device A1 disconnects from computing device B3, computing device C can delete the previously saved matching relationship between computing device A1 and computing device B3.

[0176] In other embodiments, if the computing device matched with the computing device B3 in the device cluster A can include the computing device A2 and the computing device A3 in addition to the computing device A1. The computing device C can send the device failure notification to the computing device A2, and the computing device A2 can disconnect the communication connection with the computing device B3 based on the received device failure notification. The computing device C can send the device failure notification to the computing device A3, and the computing device A3 can disconnect the communication connection with the computing device B3 based on the received device failure notification. The process of disconnecting the communication connection with the computing device B3 by the computing device A2 and the computing device A3 can be performed by referring to the process of disconnecting the communication connection with the computing device B3 by the computing device A1, which will not be described herein again. Through the above process, the faulty computing device B3 can be isolated.

[0177] As shown in FIG. 10, when a certain full node in the device cluster A fails, the interaction process between the computing device C and each computing device can include the following steps:

[0178] S1001, the computing device C monitors that the computing device A1 in the device cluster A fails, and determines that the computing device matched with the computing device A1 in the device cluster B includes the computing device B3.

[0179] During the running of the device cluster A and the device cluster B, the computing device C can monitor the running of each computing device in the device cluster A and the device cluster B in real time. When the computing device C monitors that the computing device A1 in the device cluster A fails, the computing device C can determine the computing device matched with the computing device A1 in the device cluster B. In this embodiment, the computing device matched with the computing device A1 in the device cluster B includes the computing device B3. In other embodiments, the computing device matched with the computing device A1 in the device cluster B can include one or more of the computing device B1, the computing device B2 and the computing device B3.

[0180] S1002, the computing device C sends a device failure notification to the computing device B3.

[0181] In this embodiment, the computing device matched with the computing device A1 in the device cluster B includes the computing device B3. The computing device C sends a device failure notification to the computing device B3, and the device failure notification carries the information of the faulty device, i.e., the information of the computing device A1. The information of the computing device A1 can include the information of the physical card of the computing device A1, which is used to instruct the computing device B3 to disconnect the communication connection with the computing device A1. Assuming that the computing device B3 and the computing device A1 each include 8 physical cards. The device failure notification can carry the correspondence between the 8 physical cards of the computing device B3 and the 8 physical cards of the computing device A1.

[0182] S1003, based on the received device failure notification, the computing device B3 disconnects the communication connection with the computing device A1.

[0183] The computing device B3 receives the device failure notification sent by the computing device C, obtains the information of the physical card of the computing device A1 carried in the device failure notification, and disconnects the point-to-point communication link between the physical card of the computing device B3 and the physical card of the computing device A1. Illustratively, the computing device B3 receives the device failure notification sent by the computing device C, and the host of the computing device B3 can send a disconnection instruction to each of the eight physical cards of the computing device B3 according to the correspondence between the physical card of the computing device A1 and the physical card of the computing device B3 carried in the device failure notification. For example, the host of the computing device B3 sends a disconnection instruction to the physical card B 30 , which carries the information of the physical card A 10 , to instruct the physical card B 30 to disconnect the point-to-point communication link with the physical card A 10 . The host of the computing device B3 sends a disconnection instruction to the physical card B 31 , which carries the information of the physical card A 11 , to instruct the physical card B 31 to disconnect the point-to-point communication link with the physical card A 11 , and so on.

[0184] Taking the physical card B 30 as an example, the physical card B 30 receives the disconnection instruction sent by the host of the computing device B3, finds the link information between the physical card A 10 based on the information of the physical card A 10 carried in the disconnection instruction, and deletes the saved link information to disconnect the point-to-point communication link between the physical card B 30 and the physical card A 10 .

[0185] The process of disconnecting the point-to-point communication link by other physical cards can be performed with reference to the process of disconnecting the point-to-point communication link between the physical card B 30 and the physical card A 10 , and will not be described here. At this point, the computing device B3 disconnects the communication connection with the computing device A1. After the computing device B3 disconnects the communication connection with the computing device A1, the computing device C can delete the matching relationship between the computing device B3 and the computing device A1 previously saved.

[0186] In other embodiments, if the computing devices in the device cluster B that match the computing device A1 can include the computing device B1 and the computing device B2 in addition to the computing device B3. The computing device C can send the device failure notification to the computing device B1, and the computing device B1 can disconnect the communication connection with the computing device A3 based on the received device failure notification. The computing device C can send the device failure notification to the computing device B2, and the computing device B2 can disconnect the communication connection with the computing device A3 based on the received device failure notification. The process of disconnecting the communication connection with the computing device A3 by the computing device B1 and the computing device B2 can be performed with reference to the process of disconnecting the communication connection with the computing device A1 by the computing device B3, and will not be described here. Through the above process, the faulty computing device A1 can be isolated.

[0187] In the related art, when a computing device in a computing device cluster fails, the entire computing device cluster needs to be restarted, and therefore, a single computing device failure has a high impact on the overall business of the computing device cluster. In the embodiments of the present application, through the above process, when a computing device in a computing device cluster fails, only the faulty computing device can be isolated, and the business processing of other computing devices in the computing device cluster is not affected.

[0188] The embodiments of the present application can also support smooth switching between full nodes and incremental nodes. FIG. 11 exemplarily shows an interaction flow diagram between the computing device C and each computing device in the process of converting an incremental node into a full node. As shown in FIG. 11, the process can include the following steps:

[0189] S1101, the computing device C determines to reduce the number of computing devices in the device cluster B and increase the number of computing devices in the device cluster A.

[0190] During the operation of the device cluster A and the device cluster B, the computing device C can monitor the load of each computing device in the device cluster A and the device cluster B in real time. If the computing device C monitors that the load of the device cluster B is less than a third load threshold and the load of the device cluster A exceeds a second load threshold, it can be determined that the number of computing devices in the device cluster B needs to be reduced and the number of computing devices in the device cluster A needs to be increased.

[0191] S1102, the computing device C sends a disconnection notification to the computing device B4.

[0192] The computing device C can select one or more computing devices from the computing devices included in the device cluster B, and send a disconnection notification to the selected computing devices, when determining to reduce the number of computing devices in the device cluster B and increase the number of computing devices in the device cluster A. For example, the computing device C selects the computing device B4 from the computing devices included in the device cluster B, and sends a disconnection notification to the computing device B4. The computing device B4 can be any computing device in the device cluster B.

[0193] The computing device C can determine the computing device in the device cluster A that matches the computing device B4. For example, if the computing device in the device cluster A that matches the computing device B4 is the computing device A1, the computing device C can carry the information of the physical card of the computing device A1 in the disconnection notification sent to the computing device B4, to instruct the computing device B4 to disconnect the communication connection with the computing device A1. If the computing device in the device cluster A that matches the computing device B4 is the computing device A1 and the computing device A2, the disconnection notification sent by the computing device C to the computing device B4 can be used to instruct the computing device B4 to disconnect the communication connection with the computing device A1 and the computing device A2.

[0194] In this embodiment, the disconnection notification is used to instruct the computing device B4 to disconnect the communication connection with the computing device A1. It is assumed that the computing device B4 and the computing device A1 each include 8 physical cards. The disconnection notification can carry the correspondence between the 8 physical cards of the computing device B4 and the 8 physical cards of the computing device A1. The correspondence between the physical cards of the computing device B4 and the physical cards of the computing device A1 has been described above, and will not be described here.

[0195] In other embodiments, if the disconnection notification is used to instruct the computing device B4 to disconnect the communication connection with the computing device A1 and the computing device A2, the computing device A2 also includes 8 physical cards. The disconnection notification can carry two sets of correspondence. One set of correspondence is the correspondence between the 8 physical cards of the computing device B4 and the 8 physical cards of the computing device A1, and the other set of correspondence is the correspondence between the 8 physical cards of the computing device B4 and the 8 physical cards of the computing device A2.

[0196] S1103, the computing device B4 disconnects the communication connection with the computing device A1.

[0197] The process of the computing device B4 disconnecting the communication connection with the computing device A1 can be performed by referring to step S703, and will not be described here.

[0198] In other embodiments, if the chain breakage notification is used to indicate that the computing device B4 is disconnected from the computing device A1 and the computing device A2, the process in which the computing device B4 is disconnected from the computing device A2 can be performed with reference to the process in which the computing device B4 is disconnected from the computing device A1.

[0199] S1104, the computing device C sends a role conversion notification to the computing device B4.

[0200] The computing device C can send a role conversion notification to the computing device B4 to inform the computing device B4 to switch from the device cluster B to the device cluster A. The role conversion notification can carry an indication field, and when the indication field is set to a first value, it is used to indicate that the computing device is converted from an incremental node to a full node.

[0201] The computing device C matches the corresponding computing device for the computing device B4 in the device cluster B. For example, assuming that the device cluster B currently includes three computing devices: the computing device B1, the computing device B2, and the computing device B3, when the computing device C determines that the computing device B4 is matched with the computing device B1, the role conversion notification can also carry the correspondence between the physical card of the computing device B4 and the physical card of the computing device B1.

[0202] S1105, the computing device B4 starts the listening function of the physical card based on the received role conversion notification.

[0203] The computing device B4 receives the role conversion notification sent by the computing device C, and can start the listening function of the physical card of the computing device B4 according to the correspondence between the physical card of the computing device B4 and the physical card of the computing device B1 carried in the role conversion notification, and wait to receive the link establishment request sent by the physical card of the computing device B1 in the device cluster B. For example, assuming that the physical card B 40 of the computing device B4 corresponds to the physical card B 10 of the computing device B1, after the computing device B4 starts the listening function, it is used to listen to the link establishment request sent by the physical card B 40 of the computing device B1; the physical card B 10 of the computing device B4 corresponds to the physical card B 41 of the computing device B1, after the computing device B4 starts the listening function, it is used to listen to the link establishment request sent by the physical card B 11 of the computing device B1; the physical card B 41 of the computing device B4 corresponds to the physical card B 11 of the computing device B1, after the computing device B4 starts the listening function, it is used to listen to the link establishment request sent by the physical card B 11 of the computing device B1; and so on.

[0204] S1106, the computing device C sends a link establishment notification to the computing device B1.

[0205] When the computing device C determines that the computing device B4 matches the computing device B1, the computing device C sends a link establishment notification to the computing device B1. The link establishment notification is used to instruct the computing device B1 to establish a communication connection with the computing device B4. In other embodiments, if the computing device C determines that the computing device B1, the computing device B2 and the computing device B3 all match the computing device B4, the computing device C can send a link establishment notification to the computing device B1, the computing device B2 and the computing device B3 respectively to instruct the corresponding computing device to establish a communication connection with the computing device B4.

[0206] This embodiment is described by taking the case that the computing device C determines that the computing device B4 matches the computing device B1. It is assumed that the computing device B1 and the computing device B4 each include 8 physical cards. The computing device C sends a link establishment notification to the computing device B1, and the link establishment notification can carry the correspondence between the 8 physical cards of the computing device B1 and the 8 physical cards of the computing device B4. For example, the computing device B1 includes 8 physical cards, which are physical card B1-1, physical card B1-2, physical card B1-3, physical card B1-4, physical card B1-5, physical card B1-6, physical card B1-7 and physical card B1-8 respectively. The computing device B4 includes 8 physical cards, which are physical card B4-1, physical card B4-2, physical card B4-3, physical card B4-4, physical card B4-5, physical card B4-6, physical card B4-7 and physical card B4-8 respectively. The link establishment notification can carry the information of the 8 physical cards of the computing device B1 and the information of the 8 physical cards of the computing device B4, wherein the information of the physical card B1-1 corresponds to the information of the physical card B4-1, the information of the physical card B1-2 corresponds to the information of the physical card B4-2, the information of the physical card B1-3 corresponds to the information of the physical card B4-3, the information of the physical card B1-4 corresponds to the information of the physical card B4-4, the information of the physical card B1-5 corresponds to the information of the physical card B4-5, the information of the physical card B1-6 corresponds to the information of the physical card B4-6, the information of the physical card B1-7 corresponds to the information of the physical card B4-7, and the information of the physical card B1-8 corresponds to the information of the physical card B4-8. 10 17 40 47 10 40 11 41 17 47

[0207] S1107, the computing device B1 establishes a communication connection with the computing device B4.

[0208] Based on the link establishment notification received by the computing device B1, the physical cards of the computing device B1 and the physical cards of the computing device B4 correspondingly establish point-to-point communication links. The process of establishing a communication connection between the computing device B1 and the computing device B4 can be performed by referring to the process of establishing a communication connection between the computing device B4 and the computing device A1 in the above embodiment, which will not be described herein again. After the computing device B1 establishes a communication connection with the computing device B4, the computing device C can save the matching relationship between the computing device B4 and the computing device B4.

[0209] ​​​​​​​​​​After the communication link is established, the physical card of the computing device B1 can receive the processing result of the pre-population stage transmitted by the corresponding physical card of the computing device B4 through the RDMA link. At this point, the computing device B4 is converted from the delta node in the device cluster B to the full node in the device cluster A. As shown in FIG. 12, compared with the device cluster A and the device cluster B shown in FIG. 4, the device cluster A includes 2 computing devices, which is changed to include three computing devices, i.e., the computing device A1, the computing device A2 and the computing device B4. The device cluster B includes 4 computing devices, which is changed to include three computing devices, i.e., the computing device B1, the computing device B2 and the computing device B3. The computing device B4 can share the computing task of the pre-population stage performed by the computing device A1 and the computing device A2, thereby reducing the load of each computing device in the device cluster A.

[0210] FIG. 13 exemplarily shows an interaction flow diagram between the computing device C and each computing device in the process of converting the full node to the delta node. As shown in FIG. 13, the process can include the following steps:

[0211] S1301, the computing device C determines to reduce the number of computing devices in the device cluster A and increase the number of computing devices in the device cluster B.

[0212] During the running of the device cluster A and the device cluster B, the computing device C can monitor the load of each computing device in the device cluster A and the device cluster B in real time. If the computing device C monitors that the load of the device cluster B exceeds the first load threshold and the load of the device cluster A is less than the fourth load threshold, it can be determined that the number of computing devices in the device cluster A needs to be reduced and the number of computing devices in the device cluster B needs to be increased.

[0213] When the computing device C determines to reduce the number of computing devices in the device cluster A and increase the number of computing devices in the device cluster B, it can select one or more computing devices from the computing devices included in the device cluster A. For example, the computing device C selects the computing device A3 from the computing devices included in the device cluster A, and can switch the computing device A3 from the device cluster A to the device cluster B. The computing device A3 can be understood as any computing device in the device cluster A.

[0214] S1302, the computing device C determines that the computing device matched with the computing device A3 in the device cluster B includes the computing device B1.

[0215] The computing device C determines the computing device currently matched with the computing device A3 in the device cluster B. In this embodiment, the computing device matched with the computing device A3 in the device cluster B includes the computing device B1. In other embodiments, the computing device matched with the computing device A3 in the device cluster B can include one or more of the computing device B1, the computing device B2 and the computing device B3.

[0216] In step S1303, the computing device C sends a disconnection notification to the computing device B1.

[0217] In this embodiment, the computing device matched with the computing device A3 in the device cluster B includes the computing device B1. The computing device C sends the disconnection notification to the computing device B1, and the disconnection notification carries the information of the physical card of the computing device A3, which is used to instruct the computing device B1 to disconnect the communication connection with the computing device A3. It is assumed that the computing device B1 and the computing device A3 each include 8 physical cards. The disconnection notification can carry the correspondence between the 8 physical cards of the computing device B1 and the 8 physical cards of the computing device A3.

[0218] If the computing device matched with the computing device A3 in the device cluster B includes the computing device B1, the computing device B2 and the computing device B3, the computing device C can send the disconnection notification to the computing device B1, the computing device B2 and the computing device B3 respectively. The disconnection notification sent to the computing device B1 is used to instruct the computing device B1 to disconnect the communication connection with the computing device A3, the disconnection notification sent to the computing device B2 is used to instruct the computing device B2 to disconnect the communication connection with the computing device A3, and the disconnection notification sent to the computing device B3 is used to instruct the computing device B3 to disconnect the communication connection with the computing device A3.

[0219] In step S1304, the computing device B1 disconnects the communication connection with the computing device A3.

[0220] The process that the computing device B1 disconnects the communication connection with the computing device A3 can be performed by referring to step S804, which will not be repeated here. After the computing device B1 disconnects the communication connection with the computing device A3, the computing device C can delete the previously saved matching relationship between the computing device B1 and the computing device A3.

[0221] If the computing device matched with the computing device A3 in the device cluster B includes the computing device B2 and the computing device B3 in addition to the computing device B1, the process that the computing device B2 and the computing device B3 disconnect the communication connection with the computing device A3 can be performed by referring to the process that the computing device B1 disconnects the communication connection with the computing device A3, which will not be repeated here.

[0222] In step S1305, the computing device C sends a role conversion notification to the computing device A3.

[0223] The computing device C can send a role conversion notification to the computing device A3 to inform the computing device A3 to switch from the device cluster A to the device cluster B. The role conversion notification can carry an indication field, and when the indication field is set to the second value, it is used to indicate that the computing device is converted from a full node to a delta node.

[0224] In S1306, the computing device A3 closes the listening function of the physical card based on the received role conversion notification.

[0225] When the computing device A3 receives the role conversion notification sent by the computing device C and determines that the indication field in the role conversion notification is the second value, the computing device A3 can close the listening function of the physical card of the computing device A3.

[0226] In S1307, the computing device C sends a link establishment notification to the computing device A3.

[0227] The computing device C matches the corresponding computing device for the computing device A3 in the device cluster A and sends a link establishment notification to the computing device A3. The link establishment notification is used to instruct the computing device A3 to establish a communication connection with the computing device A1 and / or the computing device A2.

[0228] In this embodiment, the link establishment notification is used to instruct the computing device A3 to establish a communication connection with the computing device A1. It is assumed that the computing device A3 and the computing device A1 each include 8 physical cards. The link establishment notification can carry the correspondence between the 8 physical cards of the computing device A3 and the 8 physical cards of the computing device A1. For example, the computing device A3 includes 8 physical cards, which are physical card A 30 ~ physical card A 37 ; the computing device A1 includes 8 physical cards, which are physical card A 10 ~ physical card A 17 ; the link establishment notification can carry the information of the 8 physical cards of the computing device A3 and the information of the 8 physical cards of the computing device A1, wherein the information of the physical card A 30 corresponds to the information of the physical card A 10 , the information of the physical card A 31 corresponds to the information of the physical card A 11 , and so on, and the information of the physical card A 37 corresponds to the information of the physical card A 17 .

[0229] In other embodiments, if the link establishment notification is used to instruct the computing device A3 to establish a communication connection with the computing device A1 and the computing device A2, the computing device A2 includes 8 physical cards, which are physical card A 20 ~ physical card A 27The link establishment notification can carry two sets of correspondence. One set of correspondence is the correspondence between the eight physical cards of the computing device A3 and the eight physical cards of the computing device Al. The other set of correspondence is the correspondence between the eight physical cards of the computing device A3 and the eight physical cards of the computing device A2. The correspondence between the eight physical cards of the computing device A3 and the eight physical cards of the computing device A2 can include: the information of the physical card A 30 corresponds to the information of the physical card A 20 , the information of the physical card A 31 corresponds to the information of the physical card A 21 , and so on, the information of the physical card A 37 corresponds to the information of the physical card A 27 .

[0230] S1308, the computing device A3 establishes a communication connection with the computing device Al.

[0231] The process of the computing device A3 establishing a communication connection with the computing device Al can be performed by referring to the process of the computing device B4 establishing a communication connection with the computing device Al, which will not be described herein again. After the computing device A3 establishes a communication connection with the computing device Al, the computing device C can save the matching relationship between the computing device A3 and the computing device Al.

[0232] After the communication link is established, the physical card of the computing device A3 can receive the processing result of the pre-population stage transmitted by the corresponding physical card of the computing device Al through the RDMA link. In an optional embodiment, for the scenario of multiple rounds of dialog, in the process of reasoning the second round of dialog, the physical card of the computing device A3 can transmit the processing result of the decoding stage in the reasoning process of the first round of dialog to the corresponding physical card of the computing device Al through the RDMA link, so that the physical card of the computing device Al can perform the pre-population stage corresponding to the second round of dialog based on the processing result of the decoding stage of the first round of dialog.

[0233] The process of the physical card of the computing device A3 receiving the processing result of the pre-population stage transmitted by the corresponding physical card of the computing device Al can be performed by referring to the process of the physical card B 40 receiving the processing result of the pre-population stage transmitted by the physical card A 10 . The process of the physical card of the computing device A3 transmitting the processing result of the decoding stage in the reasoning process of the first round of dialog to the corresponding physical card of the computing device Al can be performed by referring to the process of the physical card B 40 transmitting the processing result of the decoding stage of the first round of dialog to the physical card A 10 , which will not be described herein again.

[0234] The process of establishing a communication connection between the computing device A3 and the computing device A2 can be performed by referring to the process of establishing a communication connection between the computing device A3 and the computing device A1, and details are not repeated here. At this point, the computing device A3 is converted from a full node in the device cluster A to a delta node in the device cluster B. As shown in FIG. 14, compared with the device cluster A and the device cluster B shown in FIG. 6, the device cluster A includes three computing devices, which is changed to two computing devices including the computing device A1 and the computing device A2. The device cluster B includes three computing devices, which is changed to four computing devices including the computing device B1, the computing device B2, the computing device B3, and the computing device A3. The computing device A3 can share the computing task of the decoding stage performed by the computing device B1, the computing device B2, and the computing device B3, thereby reducing the load of each computing device in the device cluster B.

[0235] In the embodiments of the present application, the full node and the delta node in the computing device cluster can directly switch roles without restarting the computing device. If the computing device is restarted, the system memory size available for storing the related data of the model inference process needs to be recalculated in the device warm-up stage, and the system memory needs to be applied in the initialization stage. The time consumption of the device warm-up stage and the initialization stage will be longer, usually several minutes or tens of minutes, which has a greater impact on the model inference business. By using the scheme provided in the embodiments of the present application, the computing device does not need to be restarted, and the impact on the model inference business can be almost zero.

[0236] In the embodiments of the present application, the point-to-point communication link is established between the physical card of the full node and the physical card of the delta node, the data transmission performance is high, the number of full nodes and delta nodes can be dynamically adjusted according to the size of the business demand, the ratio of full nodes to delta nodes can be flexibly adjusted, the role switching between full nodes and delta nodes can be supported, so that the full nodes and the delta nodes are quickly balanced, the business redundancy is reduced from 40% to 20%, the occupation of the computing resources by the model inference business is reduced, the use rate of the physical card is increased by 5% to 10%, and the management complexity of the computing device cluster is reduced. Moreover, the computing device cluster can only isolate the computing device that fails, minimize the fault isolation, and reduce the impact on the model inference business.

[0237] In other embodiments, the full node in the device cluster A can act as a client, and the delta node in the device cluster B can act as a server. The full node sends a request to the delta node.

[0238] In this embodiment, the computing device C determines that the number of full nodes in the device cluster A needs to be increased when the load of the device cluster A exceeds the second load threshold. The process of adding a full node can be performed with reference to the embodiment of adding an incremental node shown in FIG. 3. For example, assume that the newly added full node is the computing device A4, the matching incremental node of the computing device A4 in the device cluster B is the computing device B1, the computing device A4 corresponds to the computing device B4 in the embodiment shown in FIG. 3, and the computing device B1 corresponds to the computing device A1 in the embodiment shown in FIG. 3. The difference is that the computing device A4 performs the pre-filling task of the LLM model, and the computing device B1 performs the decoding task of the LLM model. The process of adding the computing device A4 to the device cluster A will not be repeated here. After adding the computing device A4 to the device cluster A, the physical card of the computing device A4 can send the processing result of the pre-filling stage of the LLM model to the physical card of the computing device B1, and the physical card of the computing device A4 can also pull the processing result of the decoding stage of the first round of conversations from the physical card of the computing device B1, and perform the pre-filling stage of the second round of conversations based on the processing result of the decoding stage of the first round of conversations.

[0239] The computing device C determines that the number of incremental nodes in the device cluster B needs to be increased when the load of the device cluster B exceeds the first load threshold. The process of adding an incremental node can be performed with reference to the embodiment of adding a full node shown in FIG. 5. For example, assume that the newly added incremental node is the computing device B5, the matching full node of the computing device B5 in the device cluster A is the computing device A1, the computing device B5 corresponds to the computing device A3 in the embodiment shown in FIG. 5, and the computing device A1 corresponds to the computing device B1 in the embodiment shown in FIG. 5. The difference is that the computing device B5 performs the decoding task of the LLM model, and the computing device A1 performs the pre-filling task of the LLM model. The process of adding the computing device B5 to the device cluster B will not be repeated here. After adding the computing device B5 to the device cluster B, the physical card of the computing device B5 can pull the processing result of the pre-filling stage of the LLM model from the physical card of the computing device A1, and perform the decoding stage of the LLM model based on the processing result of the pre-filling stage. The physical card of the computing device B5 can also send the processing result of the decoding stage of the first round of conversations to the physical card of the computing device A1, so that the physical card of the computing device A1 performs the pre-filling stage of the second round of conversations based on the processing result of the decoding stage of the first round of conversations.

[0240] When the computing device C monitors that the load of the device cluster A is less than the fourth load threshold, it is determined that the number of full nodes in the device cluster A needs to be reduced. The process of reducing the full nodes can be performed with reference to the embodiment of reducing the incremental nodes shown in FIG. 7. For example, it is assumed that the full node to be exited from the device cluster A is the computing device A3, the incremental node matched by the computing device A3 in the device cluster B is the computing device B1, the computing device A3 corresponds to the computing device B4 in the embodiment shown in FIG. 7, and the computing device B1 corresponds to the computing device A1 in the embodiment shown in FIG. 7. The difference is that the computing device A3 performs the pre-padding task of the LLM model, and the computing device B1 performs the decoding task of the LLM model. The process of exiting the computing device A3 from the device cluster A will not be repeated here.

[0241] When the computing device C monitors that the load of the device cluster B is less than the third load threshold, it is determined that the number of incremental nodes in the device cluster B needs to be reduced. The process of reducing the incremental nodes can be performed with reference to the embodiment of reducing the full nodes shown in FIG. 8. For example, it is assumed that the incremental node to be exited from the device cluster B is the computing device B4, the incremental node matched by the computing device B4 in the device cluster A is the computing device A1, the computing device B4 corresponds to the computing device A3 in the embodiment shown in FIG. 8, and the computing device A1 corresponds to the computing device B1 in the embodiment shown in FIG. 8. The difference is that the computing device B4 performs the decoding task of the LLM model, and the computing device A1 performs the pre-padding task of the LLM model. The process of exiting the computing device B4 from the device cluster B will not be repeated here.

[0242] When the computing device C monitors that a certain full node in the device cluster A fails, the full node can be isolated. The process of isolating the full node can be performed with reference to the embodiment of isolating the incremental node shown in FIG. 9. For example, it is assumed that the computing device A1 in the device cluster A fails, and the incremental node matched by the computing device A1 in the device cluster B is the computing device B3. The computing device A1 corresponds to the computing device B3 in the embodiment shown in FIG. 9, and the computing device B3 corresponds to the computing device A1 in the embodiment shown in FIG. 9. The difference is that the computing device A1 performs the pre-padding task of the LLM model, and the computing device B3 performs the decoding task of the LLM model. The process of isolating the failed computing device A1 will not be repeated here.

[0243] When the computing device C monitors that a certain delta node in the device cluster B fails, the delta node can be isolated. The process of isolating the delta node can be performed with reference to the embodiment of isolating the full node shown in FIG. 10. For example, it is assumed that the computing device B3 in the device cluster B fails, and the matching delta node of the computing device B3 in the device cluster A is the computing device A1. The computing device B3 corresponds to the computing device A1 in the embodiment shown in FIG. 10, and the computing device A1 corresponds to the computing device B3 in the embodiment shown in FIG. 10. The difference is that the computing device B3 performs the decoding task of the LLM model, and the computing device A1 performs the pre-filling task of the LLM model. The process of isolating the failed computing device B3 is not repeated here.

[0244] When the computing device C monitors that the load of the device cluster B exceeds the first load threshold and the load of the device cluster A is less than the fourth load threshold, it can be determined that the number of computing devices in the device cluster A needs to be reduced and the number of computing devices in the device cluster B needs to be increased, at which time one or more full nodes in the device cluster A need to be converted into delta nodes. The process of converting the full node into the delta node can be performed with reference to the embodiment of converting the delta node into the full node shown in FIG. 11. For example, it is assumed that the full node to be converted is the computing device A3, and the matching delta node of the computing device A3 in the device cluster B is the computing device B1. The computing device A3 corresponds to the computing device B4 in the embodiment shown in FIG. 11, and the computing device B1 corresponds to the computing device A1 in the embodiment shown in FIG. 11. The process of converting the computing device A3 from the full node to the delta node is not repeated here.

[0245] When the computing device C monitors that the load of the device cluster B is less than the third load threshold and the load of the device cluster A exceeds the second load threshold, it can be determined that the number of computing devices in the device cluster B needs to be reduced and the number of computing devices in the device cluster A needs to be increased, at which time one or more delta nodes in the device cluster B need to be converted into full nodes. The process of converting the delta node into the full node can be performed with reference to the embodiment of converting the delta node into the full node shown in FIG. 13. For example, it is assumed that the full node to be converted is the computing device B4, and the matching delta node of the computing device B4 in the device cluster A is the computing device A1. The computing device B4 corresponds to the computing device A3 in the embodiment shown in FIG. 13, and the computing device A1 corresponds to the computing device B1 in the embodiment shown in FIG. 13. The process of converting the computing device B4 from the full node to the delta node is not repeated here.

[0246] The embodiment of the present application further provides a device cluster running apparatus, which can be applied to a first device in a device cluster. In some embodiments, as shown in FIG. 15, the device cluster running apparatus 1500 can include a notification receiving module 1501 and a link adjusting module 1502. The device cluster running apparatus 1500 can be used to implement the functions of the first device in the above-mentioned method embodiments, and thus can achieve the beneficial effects possessed by the above-mentioned method embodiments.

[0247] The notification receiving module 1501 can be used to receive a first notification, wherein the first notification is sent by a computing device C when it is determined that the number of devices in the first device cluster needs to be adjusted, and the first notification is used to instruct the first device and a second device to adjust a communication connection relationship. The first device belongs to the first device cluster, and the second device belongs to a second device cluster. The devices in the first device cluster are used to execute a first stage of a model inference process, and the devices in the second device cluster are used to execute a second stage of the model inference process.

[0248] The link adjusting module 1502 can be used to establish or disconnect a communication link between a physical card of the first device and a physical card of the second device based on the first notification.

[0249] The notification receiving module 1501 and the link adjusting module 1502 can be implemented by software or by hardware. By way of example, the implementation of the notification receiving module 1501 is described below. Similarly, the implementation of the link adjusting module 1502 can refer to the implementation of the notification receiving module 1501.

[0250] As an example of a software function module, the notification receiving module 1501 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the A module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region, or can be distributed in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ), or can be distributed in different AZs, and each AZ includes one data center or multiple data centers with similar geographical locations. Generally, one region can include multiple AZs.

[0251] Likewise, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Among them, usually one VPC is set in one region, and a communication gateway needs to be set in each VPC for cross-region communication between two VPCs in the same region and between VPCs in different regions, and the interconnection between VPCs is realized through the communication gateway.

[0252] As an example of a hardware functional unit, the notification receiving module 1501 can include at least one computing device, such as a server, etc. Alternatively, the notification receiving module 1501 can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. Among them, the above-mentioned PLD can be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0253] The multiple computing devices included in the notification receiving module 1501 can be distributed in the same region or in different regions. The multiple computing devices included in the notification receiving module 1501 can be distributed in the same AZ or in different AZs. Likewise, the multiple computing devices included in the notification receiving module 1501 can be distributed in the same VPC or in multiple VPCs. Among them, the multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs, etc.

[0254] It should be noted that in some embodiments, the notification receiving module 1501 can be used to perform any step in the device cluster running method, and the link adjustment module 1502 can also be used to perform any step in the device cluster running method. The steps responsible for the notification receiving module 1501 and the link adjustment module 1502 can be specified as needed, and the notification receiving module 1501 and the link adjustment module 1502 respectively implement different steps in the device cluster running method to realize the full function of the device cluster running device. The device cluster running device 1500 can also use more or fewer functional modules to realize the function of the device cluster running device 1500.

[0255] The embodiment of the present application further provides a device cluster running apparatus, which can be applied to a computing device C in a computing device cluster. In some embodiments, as shown in FIG. 16, the device cluster running apparatus 1600 can include a notification generation module 1601 and a notification sending module 1602. The device cluster running apparatus 1600 can be used to implement the functions of the computing device C in the above-mentioned method embodiments, and thus can achieve the beneficial effects possessed by the above-mentioned method embodiments.

[0256] The notification generation module 1601 can generate a first notification when it is determined that the number of devices in the first device cluster needs to be adjusted.

[0257] The notification sending module 1602 can send the first notification to the first device, and the first notification is used to instruct the first device to adjust the communication connection relationship with the second device. The first device belongs to the first device cluster, and the second device belongs to the second device cluster. The devices in the first device cluster are used to execute the first stage of the model inference process, and the devices in the second device cluster are used to execute the second stage of the model inference process.

[0258] The notification generation module 1601 and the notification sending module 1602 can be implemented by software or by hardware. For example, the implementation of the notification generation module 1601 is described below. Similarly, the implementation of the notification sending module 1602 can refer to the implementation of the notification generation module 1601.

[0259] As an example of a software function module, the notification generation module 1601 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the A module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region (region), or can be distributed in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ), or can be distributed in different AZs, and each AZ includes a data center or multiple data centers with similar geographical locations. Generally, one region can include multiple AZs.

[0260] Likewise, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Among them, usually one VPC is set in one region, and a communication gateway needs to be set in each VPC for cross-region communication between two VPCs in the same region and between VPCs in different regions, and the interconnection between VPCs is realized through the communication gateway.

[0261] As an example of a hardware functional unit, the notification generation module 1601 can include at least one computing device, such as a server, etc. Alternatively, the notification generation module 1601 can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. Among them, the above-mentioned PLD can be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0262] The multiple computing devices included in the notification generation module 1601 can be distributed in the same region or in different regions. The multiple computing devices included in the notification generation module 1601 can be distributed in the same AZ or in different AZs. Likewise, the multiple computing devices included in the notification generation module 1601 can be distributed in the same VPC or in multiple VPCs. Among them, the multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs, etc.

[0263] It should be noted that in some embodiments, the notification generation module 1601 can be used to perform any step in the device cluster running method, and the notification sending module 1602 can also be used to perform any step in the device cluster running method. The steps implemented by the notification generation module 1601 and the notification sending module 1602 can be specified as needed, and the notification generation module 1601 and the notification sending module 1602 respectively implement different steps in the device cluster running method to realize the entire function of the device cluster running apparatus. The device cluster running apparatus 1600 can also use more or fewer functional modules to realize the function of the device cluster running apparatus 1600.

[0264] The device cluster running apparatus 1700 can be used to implement the functions of the second device in the method embodiments, and thus can achieve the beneficial effects of the method embodiments.

[0265] The request receiving module 1701 can be configured to receive a first request sent by a physical card of a first device, where the first request is sent by the physical card of the first device based on a first notification received by the first device, the first notification is sent by a computing device C when determining that the number of devices in a first device cluster needs to be adjusted, and the first notification is used to instruct the first device and a second device to adjust a communication connection relationship.

[0266] The link adjusting module 1702 can be configured to, in response to the first request sent by the physical card of the first device, establish or disconnect a communication link between the physical card of the second device and the physical card of the first device.

[0267] The various functional modules in the embodiments of the present application can be integrated in one processor, or each module can exist physically, or two or more modules can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module.

[0268] The computing device can be used to implement the functions of the device cluster running apparatus in the above embodiments, and can be a first device in a first device cluster. In some embodiments, the structure of the computing device can be as shown in the computing device 120 or the computing device 130 in FIG. 2, and can include a host and one or more physical cards connected to the host. The host can be configured to receive a first notification, where the first notification is sent by a computing device C when determining that the number of devices in a first device cluster needs to be adjusted according to real-time load conditions of the first device cluster, and the first notification is used to instruct a first device and a second device to adjust a communication connection relationship. The first device belongs to the first device cluster, the second device belongs to a second device cluster, the devices in the first device cluster are used to perform a first stage of a model inference process, and the devices in the second device cluster are used to perform a second stage of the model inference process. The physical card can be configured to, based on the first notification, establish or disconnect a communication link with a physical card of the second device.

[0269] The embodiment of the present application further provides a computing device, which can be used to implement the functions of the device cluster running apparatus in the above-mentioned embodiments. The computing device can be a second device in a second device cluster. In some embodiments, the structure of the computing device can be as shown in the computing device 120 or the computing device 130 in FIG. 2, and can include a host and one or more physical cards connected to the host. The host can be configured to communicate with the computing device C. The physical card can be configured to, in response to a first request sent by a physical card of a first device, establish or disconnect a communication link between the physical card of the second device and the physical card of the first device; the first request is sent by the physical card of the first device based on a first notification received by the first device; the first notification is sent by the computing device C when it is determined that the number of devices in the first device cluster needs to be adjusted; the first notification is used to indicate that the first device and the second device adjust the communication connection relationship; the first device belongs to the first device cluster, and the second device belongs to the second device cluster; the devices in the first device cluster are used to execute a first stage of a model inference process; and the devices in the second device cluster are used to execute a second stage of the model inference process.

[0270] The embodiment of the present application further provides a computing device, which can be the computing device C described above and be used to implement the functions of the device cluster running apparatus in the embodiment shown in FIG. 16. As shown in FIG. 18, the computing device 1800 can include a bus 1801, a processor 1802, a memory 1803 and a communication interface 1804. The processor 1802, the memory 1803 and the communication interface 1804 communicate with each other through the bus 1801. The computing device 1800 can be a server. It should be understood that the number of processors and memories in the computing device 1800 is not limited in the present application.

[0271] The bus 1801 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one line is shown in FIG. 18, but it does not mean that there is only one bus or only one type of bus. The bus 1801 can include a path for transmitting information between various components (for example, the memory 1803, the processor 1802, the communication interface 1804) of the computing device 1800.

[0272] The processor 1802 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), among other processors.

[0273] The memory 1803 can include volatile memory, such as random access memory (RAM) comprising memory elements. The processor 1802 can also include non-volatile memory, such as read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or a mechanical hard disk (HDD), among other memory elements.

[0274] The memory 1803 stores executable program code that, when executed by the processor 1802, implements the functions of the notification generation module 1601 and the notification sending module 1602 described above, thereby implementing the device cluster operation method. That is, the memory 1803 stores instructions for executing the device cluster operation method.

[0275] The communication interface 1804 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to enable communication between the computing device 1800 and other devices or communication networks.

[0276] Embodiments of the present application also provide a computing device cluster. The structure of the computing device cluster can be as shown in FIG. 1 or FIG. 2. The computing device cluster can include a computing device C, at least one first device, and at least one second device. The at least one first device constitutes a first device cluster, and the at least one second device constitutes a second device cluster. The structure of the computing device C can be as shown in FIG. 18, and the structure of the first device and the second device can refer to the computing device 120 or the computing device 130 shown in FIG. 2, which will not be described here. A point-to-point communication link can be established between the physical card of the first device and the physical card of the second device. The computing device cluster can dynamically adjust the number of devices in the first device cluster or the second device cluster according to the change in the user request volume, and can also support isolation of only the computing device that fails, and role switching of the computing devices in the first device cluster and the second device cluster without affecting the service.

[0277] The embodiments of the present application further provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, the at least one computing device is caused to perform the device cluster running method.

[0278] The embodiments of the present application further provide a computer readable storage medium. The computer readable storage medium can be any available medium that the computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium contains instructions, which instruct the computing device to perform the service running method or instruct the computing device to perform the service running method.

[0279] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for operating a cluster of devices, characterized in that, Applied to a first device; the method includes: Receive a first notification; the first notification is sent by the fifth device when it determines that the number of devices in the first device cluster needs to be adjusted; the first notification is used to instruct the first device and the second device to adjust their communication connection relationship; the first device belongs to the first device cluster, the second device belongs to the second device cluster, the devices in the first device cluster are used to execute the first stage of the model inference process; the devices in the second device cluster are used to execute the second stage of the model inference process; Based on the first notification, establish or disconnect the communication link between the physical card of the first device and the physical card of the second device.

2. The method according to claim 1, characterized in that, The first device is used to execute the first stage of the model inference process using N physical cards; the second device is used to execute the second stage of the model inference process using N physical cards; the N physical cards in the first device and the N physical cards in the second device have a corresponding relationship; The step of establishing a communication link between the physical card of the first device and the physical card of the second device based on the first notification includes: Based on the first notification, each of the N physical cards in the first device sends a link establishment request to the corresponding physical card in the second device to establish a communication link with the corresponding physical card; the link establishment request sent by each of the N physical cards contains information about the corresponding physical card in the second device.

3. The method according to claim 1, characterized in that, The first device is used to execute the first stage of the model inference process using N physical cards; the second device is used to execute the second stage of the model inference process using N physical cards; the N physical cards in the first device and the N physical cards in the second device have a corresponding relationship; The step of disconnecting the communication link between the physical card of the first device and the physical card of the second device based on the first notification includes: Based on the first notification, each of the N physical cards in the first device sends a disconnection request to the corresponding physical card in the second device; the disconnection request sent by each of the N physical cards contains information about the corresponding physical card in the second device.

4. The method according to claim 1 or 2, characterized in that, The communication link is a Remote Direct Data Access (RDMA) link; the first physical card of the first device is used to transmit the processing result of the first stage to the corresponding physical card in the second device through the RDMA link; or, the first physical card of the first device is used to receive the processing result of the second stage transmitted by the corresponding physical card in the second device through the RDMA link.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: When the second notification is received, the communication link between the physical card of the first device and the physical card of the second device is disconnected; the second notification is sent by the fifth device when it determines that the second device is faulty.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Upon receiving the third notification, the physical card of the first device sends a disconnection request to the physical card of the second device; the third notification is sent by the fifth device when it determines that the number of devices contained in the first device cluster and the second device cluster needs to be adjusted. In response to a link establishment request sent by the physical card of the third device, a communication link is established between the physical card of the first device and the physical card of the third device; the third device belongs to the first device cluster.

7. A method for operating a cluster of devices, characterized in that, Applied to a fifth device, the method includes: When it is determined that the number of devices in the first device cluster needs to be adjusted, a first notification is sent to the first device; the first notification is used to instruct the first device and the second device to adjust their communication connection relationship; the first device belongs to the first device cluster, the second device belongs to the second device cluster, the devices in the first device cluster are used to execute the first stage of the model inference process; the devices in the second device cluster are used to execute the second stage of the model inference process.

8. The method according to claim 7, characterized in that, The method further includes: Monitor the load of the first device cluster; Based on the load of the first device cluster, determine whether the number of devices in the first device cluster needs to be adjusted.

9. The method according to claim 7 or 8, characterized in that, The first notification carries the correspondence between the N physical cards of the first device and the N physical cards of the second device.

10. The method according to any one of claims 7 to 9, characterized in that, The method further includes: When the second device is determined to be faulty, a second notification is sent to the first device; the second notification is used to instruct the communication link between the physical card of the first device and the physical card of the second device to be disconnected.

11. The method according to any one of claims 7 to 10, characterized in that, The method further includes: When it is determined that the number of devices in the first device cluster and the second device cluster needs to be adjusted, a third notification is sent to the first device; the third notification is used to instruct the first device to change from the first device cluster to the second device cluster; or... When it is determined that the number of devices contained in the first device cluster and the second device cluster needs to be adjusted, a fourth notification is sent to the second device; the fourth notification is used to instruct the second device to change from the second device cluster to the first device cluster.

12. A method for operating a cluster of devices, characterized in that, Applied to a second device, the method includes: In response to a first request sent by the physical card of the first device, a communication link is established or disconnected between the physical card of the second device and the physical card of the first device; the first request is sent by the physical card of the first device based on a first notification received by the first device; The first notification is sent by the fifth device when it determines that the number of devices in the first device cluster needs to be adjusted; the first notification is used to instruct the first device and the second device to adjust their communication connection relationship; the first device belongs to the first device cluster, the second device belongs to the second device cluster, the devices in the first device cluster are used to perform the first stage of the model inference process; the devices in the second device cluster are used to perform the second stage of the model inference process.

13. The method according to claim 12, characterized in that, The method further includes: When the fourth notification is received, the communication link between the physical card of the second device and the physical card of the first device is disconnected; the fourth notification is sent by the fifth device when it determines that the number of devices contained in the first device cluster and the second device cluster needs to be adjusted. The second device sends a link establishment request to the fourth device's physical card to establish a communication link between the physical cards of the second and fourth devices; the fourth device belongs to the second device cluster.

14. A device for operating a cluster of equipment, characterized in that, Applied to a first device, the device includes: A notification receiving module is used to receive a first notification; the first notification is sent by the fifth device when it determines that the number of devices in the first device cluster needs to be adjusted; the first notification is used to instruct the first device and the second device to adjust their communication connection relationship; the first device belongs to the first device cluster, the second device belongs to the second device cluster, the devices in the first device cluster are used to execute the first stage of the model inference process; the devices in the second device cluster are used to execute the second stage of the model inference process. The link adjustment module is used to establish or disconnect the communication link between the physical card of the first device and the physical card of the second device based on the first notification.

15. A device for operating a cluster of equipment, characterized in that, Applied to a fifth device, the device includes: The notification sending module is used to send a first notification to the first device when it is determined that the number of devices in the first device cluster needs to be adjusted; the first notification is used to instruct the first device and the second device to adjust their communication connection relationship; the first device belongs to the first device cluster, the second device belongs to the second device cluster, the devices in the first device cluster are used to perform the first stage of the model inference process; the devices in the second device cluster are used to perform the second stage of the model inference process.

16. A device for operating a cluster of equipment, characterized in that, Applied to a second device, the device includes: The link adjustment module is used to establish or disconnect a communication link between the physical card of the second device and the physical card of the first device in response to a first request sent by the physical card of the first device; the first request is sent by the physical card of the first device based on a first notification received by the first device; the first notification is sent by the fifth device when it determines that the number of devices in the first device cluster needs to be adjusted; the first notification is used to instruct the first device and the second device to adjust their communication connection relationship; the first device belongs to the first device cluster, the second device belongs to the second device cluster, the devices in the first device cluster are used to perform the first stage of the model inference process, and the devices in the second device cluster are used to perform the second stage of the model inference process.

17. A computing device, characterized in that, The computing device is a first device, which includes a host and a physical card connected to the host. The host is used to receive a first notification; the first notification is sent by the fifth device when it determines that the number of devices in the first device cluster needs to be adjusted based on the real-time load of the first device cluster, and the first notification is used to instruct the first device and the second device to adjust their communication connection relationship. The first device belongs to a first device cluster, and the second device belongs to a second device cluster. The devices in the first device cluster are used to execute the first stage of the model inference process, and the devices in the second device cluster are used to execute the second stage of the model inference process. The physical card is used to establish or disconnect a communication link with the physical card of the second device based on a first notification.

18. A computing device, characterized in that, It includes a processor and a memory; the memory stores computer-executable instructions; the processor is configured to execute the computer-executable instructions stored in the memory so that the computing device performs the method as described in any one of claims 7 to 11.

19. A computing device, characterized in that, The computing device is a second device, which includes a host and a physical card connected to the host. The host is used to communicate with the fifth device; The physical card is used to establish or disconnect a communication link between the physical card of the second device and the physical card of the first device in response to a first request sent by the physical card of the first device; the first request is sent by the physical card of the first device based on a first notification received by the first device; The first notification is sent by the fifth device when it determines that the number of devices in the first device cluster needs to be adjusted; the first notification is used to instruct the first device and the second device to adjust their communication connection relationship. The first device belongs to a first device cluster, and the second device belongs to a second device cluster. The devices in the first device cluster are used to execute the first stage of the model inference process. The devices in the second device cluster are used to perform the second phase of the model inference process.

20. A computing device cluster, characterized in that, It includes a fifth device, at least one first device, and at least one second device; the fifth device is the computing device of claim 18, the first device is the computing device of claim 17, and the second device is the computing device of claim 19.

21. A computer-readable storage medium, characterized in that, The device stores computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 6; or, to perform the method as described in any one of claims 7 to 11; or, to perform the method as described in claim 12 or 13.

22. A computer program product, characterized in that, It includes computer-executable instructions for causing a computer to perform the method as claimed in any one of claims 1 to 6; or, to perform the method as claimed in any one of claims 7 to 11; or, to perform the method as claimed in claim 12 or 13.

Images