A model management method based on a cloud management platform and a cloud management platform

Abstract

Embodiments of the present application provide a model management method based on a cloud management platform and the cloud management platform, which can avoid resource waste. The method of the present application comprises: a plurality of devices managed by the cloud management platform can run target models, the target models can include a plurality of routing experts, a plurality of shared experts and a plurality of redundant experts. If a first device in the plurality of devices has a fault, the first device runs a first routing expert, the cloud management platform can detect whether a first redundant expert corresponding to the first routing expert exists in the plurality of redundant experts, if not, the cloud management platform can select a second device from the plurality of devices, and eliminate the first device from the plurality of devices, wherein the second device runs at least one of a first shared expert and a second redundant expert. Then, the cloud management platform notifies the second device to delete at least one of the first shared expert and the second redundant expert running thereon, and notifies the second device to run the first routing expert.

Description

[0001] This application claims priority to Chinese patent application filed on April 28, 2025, with application number 202510549455.7 and entitled "A Model Management Method and Cloud Management Platform Based on a Cloud Management Platform", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of cloud technology, and in particular to a model management method based on a cloud management platform and a cloud management platform. Background Technology

[0003] With the rapid development of cloud technology, more and more tenants are choosing cloud vendors' cloud-based inference systems to deploy their inference models, in order to reduce the cost of local deployment and improve the inference efficiency and accuracy of the models.

[0004] In related technologies, the inference system provided by cloud vendors can include a cloud management platform and multiple inference instances. Each inference instance can contain multiple physical servers, and each physical server can contain multiple devices. The cloud management platform can deploy a tenant's inference model across these inference instances. When a tenant has pending business operations, the cloud management platform can run the inference model through any one of these inference instances to complete the tenant's business.

[0005] In the process described above, when a particular inference instance fails, the cloud management platform often removes that instance from the pool of inference instances and delegates the tenant's business to the remaining inference instances. However, since that inference instance contains a large number of devices, directly discarding the inference instance would result in a significant waste of resources as a large number of devices would be abandoned. Summary of the Invention

[0006] This application provides a model management method and a cloud management platform based on a cloud management platform, which can help to continuously and effectively utilize existing equipment resources and avoid resource waste in the event of equipment failure.

[0007] The first aspect of this application provides a model management method based on a cloud management platform. The cloud management platform implementing this method can manage inference instances. Each inference instance includes multiple devices running a target model. The target model includes multiple routing experts, multiple shared experts, and multiple redundant experts. Each redundant expert corresponds to at least one of the multiple routing experts. Any one of the multiple devices runs any one or any combination of at least one routing expert, at least one shared expert, and at least one redundant expert. The method includes:

[0008] When the first device among these multiple devices fails, since the first device is running a first routing expert, the cloud management platform can detect whether there is a first redundant expert corresponding to the first routing expert among these multiple redundancies.

[0009] If the multiple redundant experts do not include the first redundant expert, since some of these devices are running both a redundant expert and a shared expert, the cloud management platform can select a second device from these devices to replace the first device and remove the first device from these multiple devices. The second device is running at least one of the first shared expert and the second redundant expert.

[0010] Once the second device is identified, the cloud management platform can instruct it to delete at least one of the first shared expert and the second redundant expert, and then instruct it to run the first routing expert. In this way, the cloud management platform can successfully allow the second device to run the first routing expert in place of the first device, ensuring the normal operation of the target model.

[0011] As can be seen from the above method, when the cloud management platform detects a fault in the first device of the inference instance, it can hand over the first routing expert it is running to the second device running the second redundant expert and the first shared expert. In this way, even if only the first device is removed, the target model can still run normally on the second device. This does not require abandoning the entire inference instance, which is conducive to the continuous and effective use of existing device resources and avoids resource waste.

[0012] In one possible implementation, the method further includes: the cloud management platform determining that multiple redundant experts include a first redundant expert; selecting a third device running the first redundant expert from multiple devices; and removing the first device from the multiple devices; the cloud management platform instructs the third device to replace the first redundant expert with a first routing expert, so that the first routing expert can run through the third device. In the aforementioned implementation, if these multiple redundant experts include a first redundant expert, the cloud management platform can select a third device running the first redundant expert from these multiple devices and remove the first device from the multiple devices. Then, the cloud management platform can replace the first redundant expert with the first routing expert in the third device in some way. In this way, the cloud management platform can successfully make the third device run the first routing expert, thereby ensuring the normal operation of the target model. Therefore, when the cloud management platform discovers a fault in the first device in the inference instance, it can hand over the first routing expert it is running to the third device running the first redundant expert. This ensures that the target model continues to run normally on the third device even when only the faulty first device is removed, without abandoning the entire inference instance. This is beneficial for the continuous and effective use of existing device resources and further avoids resource waste.

[0013] In one possible implementation, the method further includes: a cloud management platform determining that a fourth device among multiple devices is faulty, wherein the fourth device is running a second shared expert; the cloud management platform selecting a fifth device from the multiple devices and removing the fourth device from the multiple devices, wherein the fifth device is running a third redundant expert; the cloud management platform notifying the fifth device to delete the third redundant expert and notifying the fifth device to run the second shared expert. In the aforementioned implementation, if the fourth device among these multiple devices is faulty, since the fourth device is running a second shared expert, the cloud management platform can select the fifth device from the multiple devices that have deployed redundant experts and remove the fourth device from the multiple devices, and the fifth device will run a third redundant expert. Then, the cloud management platform can notify the fifth device to delete the third redundant expert and notify the fifth device to run the second shared expert. In this way, the cloud management platform can successfully enable the fifth device to run the second shared expert, ensuring the normal operation of the target model. Therefore, when the cloud management platform detects a fault in the fourth device in the inference instance, it can transfer the second shared expert running in the instance to the fifth device running the third redundant expert. This ensures that the target model continues to run normally on the fifth device, even if only the fourth device is removed. This avoids abandoning the entire inference instance, which helps to make continuous and effective use of existing device resources and further avoids resource waste.

[0014] In one possible implementation, the method further includes: a cloud management platform determining that a sixth device among multiple devices is faulty, wherein the sixth device is running a fourth redundant expert; and the cloud management platform removing the sixth device from the multiple devices. In the aforementioned implementation, if the sixth device among these multiple devices is faulty, since the sixth device is running a fourth redundant expert, the cloud management platform does not need to perform an expert replacement operation; it can directly remove the sixth device from the multiple devices, thus not affecting the normal operation of the target model. In this way, even if the sixth device fails, the cloud management platform does not need to discard the entire inference instance, which is beneficial for the continuous and effective use of existing device resources and further avoids resource waste.

[0015] In one possible implementation, multiple devices are divided into multiple device groups based on a tensor parallelism strategy. The number of device groups is related to the first degree of parallelism of the tensor parallelism strategy, and the first device and the second device belong to different device groups. In the aforementioned implementation, since the cloud management platform can divide these multiple devices into multiple device groups in this inference instance, the first device and the second device used to replace the first device are usually located in different device groups, which is beneficial for subsequent expert replacement.

[0016] In one possible implementation, the target model further includes models other than multiple routing experts, multiple shared experts, and multiple redundant experts. The model is partitioned into multiple first sub-models based on a tensor parallelism strategy. The number of these first sub-models is associated with the first degree of parallelism of the tensor parallelism strategy. Any one of the multiple devices runs one of the first sub-models. The method further includes: a cloud management platform determining a second degree of parallelism for the tensor parallelism strategy based on the number of devices other than the first devices; the cloud management platform partitioning the model based on the second degree of parallelism to obtain multiple second sub-models, wherein the number of second sub-models is less than the number of first sub-models; and the cloud management platform notifying the remaining devices to run the multiple second sub-models, wherein any one of the remaining devices runs one of the second sub-models. In the aforementioned implementation, since the target model typically includes models other than multiple routing experts, multiple shared experts, and multiple redundant experts, this model can be partitioned into multiple first sub-models based on a tensor parallelism strategy. The number of these first sub-models is typically associated with the first degree of parallelism of the tensor parallelism strategy, and the multiple devices run multiple first sub-models. After removing the first device from the set of devices, the cloud management platform can determine the second degree of parallelism for the remaining devices (excluding the first device) based on their data. It then re-partitions the model based on this second degree of parallelism, resulting in multiple second sub-models. The cloud management platform can then instruct the remaining devices to run these second sub-models, replacing the multiple first sub-models they were previously running. This ensures that the model continues to function correctly on the remaining devices and allows inference to continue. Therefore, the cloud management platform can re-partition multiple models (excluding those of the expert) based on the number of remaining devices and the changes in the parallelism of the tensor parallelism strategy. This ensures that the remaining devices continue to run the target model correctly, preventing the inference process from stalling.

[0017] In one possible implementation, the sum of the number of routing experts, the number of shared experts, and the number of redundant experts is associated with the third degree of parallelism of the expert parallelism strategy. The method further includes: the cloud management platform determining the fourth degree of parallelism of the expert parallelism strategy based on the number of remaining devices; and the cloud management platform obtaining the communication relationships between the remaining devices based on the number of remaining devices, the second degree of parallelism, and the fourth degree of parallelism. These communication relationships are used to instruct the devices to communicate with each other to run the target model and process the target task. In the aforementioned implementation, the cloud management platform can use the number of remaining devices to determine the fourth degree of parallelism of the expert parallelism strategy, and then use the number of remaining devices, the second degree of parallelism of the tensor parallelism strategy, and the fourth degree of parallelism of the expert parallelism strategy to obtain the communication relationships between the remaining devices, guiding them to communicate with each other to run the target model and process the target task from the tenant. Therefore, the cloud management platform can also rebuild the communication relationships for the remaining devices. In the newly established communication relationships, the communication relationships of some devices may change, while the communication relationships of others may remain consistent. Therefore, the cloud management platform only needs to dynamically replace the changed communication relationships, which can alleviate the communication relationship rebuilding operation to some extent.

[0018] In one possible implementation, the target task comprises multiple subtasks pointing to multiple routing experts, and each subtask includes a first subtask pointing to a first routing expert. The method further includes: the cloud management platform notifying a second or third device to process the first subtask through the first routing expert. In the aforementioned implementation, after expert replacement, the cloud management platform can notify the remaining devices to continue inference to complete the tenant's target task. The target task comprises multiple subtasks pointing to multiple routing experts, and the cloud management platform can distribute these multiple subtasks to the remaining devices. Since these multiple subtasks include a first subtask pointing to the first routing expert, the second or third device, which has the first routing expert deployed, can select the first subtask from these multiple subtasks and process it through the first routing expert. Therefore, the cloud management platform can avoid erroneous expert activation and correctly complete the tenant's task by promptly routing subsequent unprocessed subtasks to the routing expert whose location has changed.

[0019] A second aspect of this application provides a cloud management platform for managing multiple devices that run a target model. The target model includes multiple routing experts, multiple shared experts, and multiple redundant experts. Each redundant expert corresponds to at least one of the multiple routing experts. Any one of the multiple devices runs any one or any combination of at least one routing expert, at least one shared expert, and at least one redundant expert. The cloud management platform includes: a first determining module for determining that a first device among the multiple devices is faulty, wherein the first device runs a first routing expert; a second determining module for determining that the multiple redundant experts do not include a first redundant expert corresponding to the first routing expert, selecting a second device from the multiple devices to replace the first device, and removing the first device from the multiple devices, wherein the second device runs at least one of a first shared expert and a second redundant expert; and a first notification module for notifying the second device to delete at least one of the first shared expert and the second redundant expert, and notifying the second device to run the first routing expert.

[0020] In one possible implementation, the cloud management platform further includes: a third determining module, used to determine that the multiple redundant experts include the first redundant expert, select a third device running the first redundant expert from the multiple devices, and remove the first device from the multiple devices; and a second notification module, used to notify the third device to replace the first redundant expert with the first routing expert so that the first routing expert can be run through the third device.

[0021] In one possible implementation, the cloud management platform further includes: a fourth determination module for determining that a fourth device among multiple devices is faulty, wherein the fourth device is running a second shared expert; a fourth determination module for selecting a fifth device from the multiple devices and removing the fourth device from the multiple devices, wherein the fifth device is running a third redundancy expert; and a third notification module for notifying the fifth device to delete the third redundancy expert and notifying the fifth device to run the second shared expert.

[0022] In one possible implementation, the cloud management platform further includes: a fifth determination module for determining that a sixth device among multiple devices is faulty, wherein the sixth device operates a fourth redundancy expert; and a removal module for removing the sixth device from the multiple devices.

[0023] In one possible implementation, multiple devices are divided into multiple device groups based on a tensor parallelism strategy. The number of device groups is related to the first degree of parallelism of the tensor parallelism strategy, and the first device and the second device belong to different device groups.

[0024] In one possible implementation, the target model further includes models other than multiple routing experts, multiple shared experts, and multiple redundant experts. The model is partitioned into multiple first sub-models based on a tensor parallel strategy. The number of the multiple first sub-models is related to the first degree of parallelism of the tensor parallel strategy. Any one of the multiple devices runs one of the multiple first sub-models. The cloud management platform also includes: a sixth determining module, used to determine the second degree of parallelism of the tensor parallel strategy based on the number of devices other than the first device; a partitioning module, used to partition the model based on the second degree of parallelism to obtain multiple second sub-models, wherein the number of the multiple second sub-models is less than the number of the multiple first sub-models; and a fourth notification module, used to notify the other devices to run the multiple second sub-models, wherein any one of the other devices runs one of the multiple second sub-models.

[0025] In one possible implementation, the sum of the number of multiple routing experts, the number of multiple shared experts, and the number of multiple redundant experts is associated with the third degree of parallelism of the expert parallelism strategy. The cloud management platform also includes: a seventh determination module, used to determine the fourth degree of parallelism of the expert parallelism strategy based on the number of remaining devices; and an acquisition module, used to acquire the communication relationships between the remaining devices based on the number of remaining devices, the second degree of parallelism, and the fourth degree of parallelism, wherein the communication relationships are used to instruct the devices to communicate with each other to run the target model to process the target task.

[0026] In one possible implementation, the target task includes multiple subtasks pointing to multiple routing experts, and the multiple subtasks include a first subtask pointing to a first routing expert. The cloud management platform also includes a fifth notification module for notifying a second or third device to process the first subtask through the first routing expert.

[0027] A third aspect of this application provides a computing device cluster, which includes at least one computing device, each computing device including a processor and a memory: the memory is used to store instructions; the processor is used to cause the computing device cluster to perform the method described in the first aspect or any possible implementation of the first aspect according to the instructions.

[0028] A fourth aspect of this application provides a computer storage medium storing one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method described in the first aspect or any possible implementation of the first aspect.

[0029] A fifth aspect of this application provides a computer program product storing instructions that, when executed by a computer, cause the computer to perform the method described in the first aspect or any possible implementation of the first aspect.

[0030] In this embodiment, a certain inference instance managed by the cloud management platform may include multiple devices. These devices may run a target model, which may include multiple routing experts, multiple shared experts, and multiple redundant experts. Each redundant expert corresponds to at least one of the routing experts. For any one of these devices, it may run any one or any combination of at least one routing expert, at least one shared expert, and at least one redundant expert. Therefore, if the first device among these devices malfunctions, since it runs a first routing expert, the cloud management platform can detect whether a first redundant expert corresponding to the first routing expert exists among the redundant experts. If not, the cloud management platform can select a second device from among the multiple devices to replace the first device and remove the first device from the multiple devices. The second device runs at least one of a first shared expert and a second redundant expert. Then, the cloud management platform can notify the second device to delete at least one of its running first shared expert and second redundant expert, and notify the second device to run the first routing expert. In this way, the cloud management platform can successfully allow the second device to replace the first device in running the first routing expert, ensuring the normal operation of the target model. In the aforementioned process, when the cloud management platform detects a fault in the first device of the inference instance, it can transfer the first routing expert it is running to the second device running the second redundant expert and the first shared expert. This ensures that the target model continues to run normally on the second device even if only the first device is removed from the list. This avoids abandoning the entire inference instance, which is conducive to the continuous and effective use of existing device resources and avoids resource waste. Attached Figure Description

[0031] Figure 1 A schematic diagram of the structure of the cloud service system provided in the embodiments of this application;

[0032] Figure 2 A schematic diagram of a reasoning example provided in this application;

[0033] Figure 3 A schematic diagram of the cloud management platform provided in the embodiments of this application;

[0034] Figure 4 A flowchart illustrating a model management method based on a cloud management platform provided in an embodiment of this application;

[0035] Figure 5Another structural schematic diagram of the reasoning example provided in the embodiments of this application;

[0036] Figure 6 Another structural schematic diagram of the reasoning example provided in the embodiments of this application;

[0037] Figure 7 Another structural schematic diagram of the reasoning example provided in the embodiments of this application;

[0038] Figure 8 Another structural schematic diagram of the reasoning example provided in the embodiments of this application;

[0039] Figure 9 A schematic diagram of the structure of the cloud management platform provided in the embodiments of this application;

[0040] Figure 10 A schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0041] Figure 11 A schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0042] Figure 12 This is a schematic diagram illustrating the network connection of computer devices in a computer cluster provided in an embodiment of this application. Detailed Implementation

[0043] This application provides a model management method and a cloud management platform based on a cloud management platform, which can help to continuously and effectively utilize existing equipment resources and avoid resource waste in the event of equipment failure.

[0044] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0045] With the rapid development of cloud technology, more and more tenants are choosing cloud vendors' cloud-based inference systems to deploy their inference models, in order to reduce the cost of local deployment and improve the inference efficiency and accuracy of the models.

[0046] In related technologies, cloud vendor-provided inference systems can include a cloud management platform and multiple inference instances. Each inference instance can contain multiple physical servers, and each physical server can contain multiple devices. The cloud management platform can deploy a tenant's inference model across these inference instances. Generally, although different inference instances all deploy the same model for the tenant, different inference instances can use this model to complete different business operations or functions for the tenant. Based on this, when a tenant has a pending business operation, the cloud management platform can run the inference model through any one of these multiple inference instances to complete the business operation.

[0047] In the process described above, when a device in an inference instance fails, the cloud management platform often removes that instance from the pool of inference instances and delegates the tenant's business, which was originally handled by that inference instance, to other inference instances. However, this inference instance also contains other devices besides the failed one. Directly discarding the inference instance would result in a large number of normal devices being abandoned, causing significant resource waste.

[0048] To address the aforementioned issues, this application provides a model management method based on a cloud management platform, which can be implemented through a cloud service system. Figure 1 A schematic diagram of the cloud service system provided in the embodiments of this application is shown below. Figure 1 As shown, a cloud service system includes the infrastructure that provides cloud services and a cloud management platform that manages this infrastructure. The cloud management platform and the infrastructure are described separately below:

[0049] A cloud management platform can centrally manage the infrastructure across the entire cloud service system. (For example, following a tenant's instructions, multiple inference instances can be created within the infrastructure. These inference instances can run the target models specified by the tenant. These target models, during runtime, can process the target tasks from the tenant and return the processing results to the tenant, thereby meeting the tenant's business needs, etc.) The cloud management platform can also be open to tenants outside the cloud service system and respond to their requests. For example, the cloud management platform can provide various interfaces, such as login and processing interfaces, for tenant clients (e.g., the terminal devices used by the tenant or the browsers on those devices). Specifically, the cloud management platform can authenticate a tenant's client through the login interface, allowing the tenant's client to log in to the cloud management platform upon successful authentication. For example, the cloud management platform can also allow the tenant's client to send a business processing request to the cloud management platform through a processing interface. The cloud management platform can forward the business processing request to an inference instance running the target model, so that the inference instance can process the target task indicated by the business processing request through its running target model, thereby obtaining the processing result of the target task and returning the processing result of the target task to the cloud management platform, so that the cloud management platform can return the processing result of the target task to the tenant, thereby meeting the tenant's business needs.

[0050] The infrastructure comprises multiple inference instances created by the cloud management platform. Each of these inference instances can contain multiple compute nodes, each possessing a certain amount of computing resources (e.g., various processors), storage resources (e.g., memory and disks), and network resources (e.g., network interface cards). Therefore, this inference instance, composed of multiple compute nodes, possesses a large number of resources. These compute nodes can then collaborate to provide high-quality remote services to tenants, such as running the tenant's target model (e.g., large language models (LLM) and other neural network models) to fulfill the tenant's business requirements.

[0051] Furthermore, such as Figure 2 As shown ( Figure 2(A schematic diagram of a structure of an inference instance provided in the embodiments of this application) For any one of these multiple inference instances, the inference instance may contain multiple computing nodes, and the computing resources contained in each computing node may be presented as multiple (physical) devices for deploying the target model. These multiple devices may include at least one of the following: a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), and a data processing unit (DPU), etc., and other types of processors. It should be noted that the inference instance can run the target model according to a certain parallel strategy. The target model can at least include a mixture-of-experts (MoE) layer and other model layers (e.g., a multi-layer perceptron (MLP) layer, an attention mechanism layer, and a language modeling head (LM head), etc.). The parallel strategy can include expert parallelism (EP), data parallelism (DP), and tensor parallelism (TP). Since the MoE layer and other model layers can be logically divided into multiple sub-models—for example, the multiple sub-models obtained from the MoE layer can include multiple routing experts, multiple redundant experts, and multiple shared experts; the multiple sub-models obtained from the MLP layer can be represented as multiple sub-MLPs; the multiple sub-models obtained from the attention layer can be represented as multiple sub-attentions; and the multiple sub-models obtained from the LM head layer can be represented as multiple sub-LM heads.

[0052] It is worth noting that the number of devices included in this inference instance is set by the tenant. This number often determines the parallelism of the EP (Electronic Programming) and TP (TP) policies. These parallelisms are related to the number of routing experts, redundant experts, shared experts, sub-MLPs, sub-attentions, and sub-LM heads. For example, assuming a tenant's inference instance contains 8 compute nodes, and each compute node can contain 16 devices, then this inference instance can contain 128 devices. The MoE layer of the target model can be divided into 256 experts (e.g., 128 routing experts and 128 redundant experts, or 64 routing experts, 64 redundant experts, and 128 shared experts, etc.). Therefore, the parallelism of the EP policy can be determined to be 128, that is, each of these 128 devices can be deployed with 2 experts (e.g., each device runs 1 routing expert and 1 redundant expert, or 64 of these 128 devices are each deployed with 1 routing expert and 1 shared expert, and the other 64 devices are each deployed with 1 redundant expert and 1 shared expert). Furthermore, since the parallelism of the TP strategy for the attention layer is 4, and the parallelism of the TP strategy for the MLP layer and LM head layer is 8 (of course, this parallelism can also be 4 or 16, etc.), based on these parallelisms, the 16 devices of each computing node can be divided into 4 device groups, and each device group can contain 4 devices. Therefore, the attention layer can be divided into 128 sub-attentions (every 4 sub-attentions form a group, and the 4 sub-attentions in the same group can form a complete attention layer). Thus, the 4 devices in each device group can each deploy the 4 sub-attentions of the same group. Correspondingly, the MLP layer can be divided into 128 sub-MLPs (every 8 sub-MLPs form a group, and the 8 sub-MLPs in the same group can form a complete MLP layer). Thus, the 8 devices in every 2 device groups can each deploy the 8 sub-MLPs of the same group. Similarly, the LM head layer is also divided in this way, which will not be elaborated here.

[0053] Furthermore, for the MoE layer of the target model, there is a correspondence between the multiple routing experts and multiple redundant experts obtained from its partitioning. That is, these multiple redundant experts can be regarded as copies of at least one of the multiple routing experts. For example, when the MoE layer includes routing experts 1 to 4 and redundant experts 1 to 4, redundant expert 1 can be a copy of routing expert 1, redundant expert 2 can be a copy of routing expert 2, redundant expert 3 can be a copy of routing expert 3, and redundant expert 4 can be a copy of routing expert 4. Similarly, when the MoE layer includes routing experts 1 to 4 and redundant experts 1 to 4, redundant experts 1 and 2 can be copies of routing expert 1, redundant experts 3 and 4 can be copies of routing expert 2, and routing experts 3 and 4 do not have copies, and so on.

[0054] Furthermore, such as Figure 3 As shown ( Figure 3 (This is a schematic diagram of a cloud management platform provided in an embodiment of this application). The cloud management platform may include multiple functional modules such as a fault decision module, an expert management module, a device group management module, and a request management module. When a device of a certain inference instance in the inference framework fails, the inference instance can notify the fault decision module that the device has failed. After confirming the situation, the fault decision module can first notify each inference instance to suspend inference (i.e., suspend the processing of tenant tasks), and determine the type of expert running on the failed device in the mapping table provided by the expert management module, and determine whether an expert replacement operation is needed for the device based on the type. If necessary, the fault decision module can determine the fault card that can be used to replace the fault card based on the load and mapping table of each device provided by the device group management module, and perform various operations such as expert replacement, model parameter reallocation, computation graph reallocation, communication domain reconstruction, and routing adaptation for these normal devices, so that the failed device is removed from the inference instance, and the expert deployed on the failed device is run by the normal device used to replace the device. In this way, the fault decision module can notify the inference instance and other inference instances to continue inference, i.e., continue processing tenant tasks, through the request processing module.

[0055] Furthermore, Figure 3 The example shown is only a illustrative illustration of the centralized deployment of multiple functional modules, such as the fault decision module, expert management module, device group management module, and request management module, on the cloud management platform. In practical applications, these multiple functional modules can also be deployed remotely in various inference instances, but they are still considered part of the cloud management platform. In this case, the cloud management platform can be regarded as a distributed platform.

[0056] Furthermore, for any one of the multiple inference instances in the infrastructure, the inference instance may contain multiple compute nodes. These compute nodes can be presented in various ways. For example, these compute nodes may be physical servers selected by the cloud management platform in the infrastructure; they may also be bare metal servers (BMS) selected by the cloud management platform in the infrastructure; they may also be virtual machines (VMs) created by the cloud management platform on the physical server using virtualization technology; they may also be containers created by the cloud management platform on the physical server using virtualization technology; they may also be microvirtual machines (microVMs) created by the cloud management platform on the physical server using virtualization technology, and so on.

[0057] Furthermore, for any one of the multiple inference instances in the infrastructure, the inference instance may contain multiple compute nodes. These compute nodes may be deployed in the same site or different sites. These sites can be presented in various forms, such as one or more regions in the infrastructure, one or more availability zones (AZs) in the infrastructure, one or more data centers (DCs) in the infrastructure, one or more server rooms in the infrastructure, or one or more racks in the infrastructure.

[0058] Based on the aforementioned cloud service system, a single inference instance managed by the cloud management platform can contain multiple devices. These devices can run a target model, and the MoE layer of the target model can contain multiple routing experts, multiple shared experts, and multiple redundant experts. Each redundant expert corresponds to at least one of the routing experts. For any one of these devices, it can run any one or any combination of at least one routing expert, at least one shared expert, and at least one redundant expert. Therefore, if one of these devices fails, since the failed device runs a routing expert, the cloud management platform can check if a redundant expert corresponding to that routing expert exists among the redundant experts. If not, the cloud management platform can select a normal device from among the multiple devices to replace the failed device and remove the failed device from the multiple devices. The selected normal device runs at least one of a shared expert and a redundant expert. Then, the cloud management platform can notify the selected normal device to delete at least one of its running shared expert and redundant expert, and notify the selected normal device to run the routing expert. In this way, the cloud management platform can successfully assign a selected normal device to run the routing expert in place of the faulty device, ensuring the normal operation of the target model. In the aforementioned process, when the cloud management platform detects a fault in a device within the inference instance, it can transfer the routing expert it is running to a normal device running redundant and shared experts. This ensures that the target model continues to operate normally on other devices while only removing the faulty device, without abandoning the entire inference instance. This facilitates the continuous and effective use of existing device resources and avoids resource waste. To further understand the workflow of the cloud management platform, the following section combines... Figure 4 To further explain this workflow, Figure 4 A flowchart illustrating a model management method based on a cloud management platform provided in this application embodiment is shown below. Figure 4 As shown, this method can be achieved through, as Figure 1 The cloud service system implementation shown may include infrastructure providing cloud services to tenants and a cloud management platform for managing this infrastructure. The cloud management platform manages multiple inference instances. For any one of these inference instances, the inference instance may include multiple devices, which may run a target model. The target model includes multiple routing experts, multiple shared experts, and multiple redundancy experts. The multiple redundancy experts correspond to at least one of the multiple routing experts. Any one of the multiple devices may run any one or any combination of at least one routing expert, at least one shared expert, and at least one redundancy expert. The method includes:

[0059] 401. The cloud management platform has determined that the first device among multiple devices is faulty. The first device is running the First Router Expert.

[0060] In this embodiment, when the first device among the multiple devices malfunctions, since the first device runs a first routing expert, the cloud management platform can utilize a pre-set mapping table. This mapping table records the locations of the multiple routing experts, the multiple shared experts, and the multiple redundant experts (e.g., a routing expert is deployed on a specific device among these multiple devices), as well as the routing experts corresponding to each of the multiple redundant experts (e.g., a redundant expert corresponds to one or more routing experts). Then, the cloud management platform can use this mapping table to detect whether there exists a first redundant expert corresponding to the first routing expert among these multiple redundant experts; that is, the cloud management platform can detect whether there is a first redundant expert that serves as a copy of the first routing expert among these multiple redundant experts.

[0061] For example, such as Figure 5 As shown ( Figure 5 (This is another structural diagram of an inference instance provided in an embodiment of this application). Assume that an inference instance running a target model includes computing nodes 1 to 8. Computing node 1 may include devices 1 to 16, ..., and computing node 8 may include devices 113 to 128. The 16 devices of computing node 1 can be divided into device groups 1 to 4, ..., and the 16 devices of computing node 8 can be divided into device groups 29 to 32. Since the MoE layer of the target model includes routing experts 1 to 64, redundancy experts 1 to 32, and sharing experts 1 to 32, the routing experts 1 to 64 run in devices 1 to 64 included in computing nodes 1 to 4, the redundancy experts 1 to 32 run in devices 65 to 96 included in computing nodes 5 to 6, and the sharing experts 1 to 32 run in devices 97 to 128 included in computing nodes 7 to 8.

[0062] If device 3 malfunctions, since device 3 (i.e., the aforementioned first device) runs routing expert 3 (i.e., the aforementioned first routing expert), the cloud management platform can obtain a pre-set mapping table. This mapping table records the locations of these 128 experts (for example, routing expert 1 is located at device 1 of computing node 1, ..., routing expert 64 is located at device 64 of computing node 4, redundancy expert 1 is located at device 65 of computing node 5, ..., redundancy expert 32 is located at device 96 of computing node 6, ..., shared expert 32 is located at device 128 of computing node 8, etc.) and the routing experts corresponding to these 32 redundant experts.

[0063] Check whether there is a redundant expert (i.e. the aforementioned first redundant expert) among redundant experts 1 to 32 that corresponds to routing expert 3.

[0064] 402. The cloud management platform determines that multiple redundant experts do not include the first redundant expert corresponding to the first routing expert, selects a second device from multiple devices to replace the first device, and removes the first device from the multiple devices, wherein the second device runs at least one of the first shared expert and the second redundant expert.

[0065] If the multiple redundant experts do not include the first redundant expert, the cloud management platform can determine that a device needs to be removed from the inference instance. Therefore, the cloud management platform can first notify the inference instance to suspend the target model. Since some of these devices are running both redundant experts and shared experts, the cloud management platform can select the device with the lightest load from these devices as the second device to replace the first device, and remove the first device from the multiple devices. The second device runs at least one of the first shared expert and the second redundant expert; for example, the second device may only run the first shared expert, or it may only run the second redundant expert, or it may run both the first shared expert and the second redundant expert.

[0066] As in the example above, since there is no redundant expert corresponding to routing expert 3 among redundant experts 1 to 32, the cloud management platform can determine that the load of device 65 is the lightest based on the load of device 65 to device 96. Therefore, the cloud management platform can determine that device 65 (i.e. the aforementioned second device) is used to replace device 3.

[0067] 403. The cloud management platform notifies the second device to delete at least one of the first shared expert and the second redundant expert, and notifies the second device to run the first routing expert.

[0068] Once the second device is identified, the cloud management platform can instruct the second device to delete at least one of the first shared expert and the second redundant expert it is running, and then instruct the second device to run the first routing expert. In this way, the cloud management platform can successfully enable the second device to run the first routing expert in place of the first device, thereby ensuring the normal operation of the target model.

[0069] Specifically, since these multiple devices can be divided into multiple device groups based on the TP policy targeting the target model, and the number of these device groups is related to the first parallelism of the TP policy, the first device and the second device are usually located in different device groups. It should be noted that since the devices in the device group where the first device is located usually have routing experts deployed, and the devices in the device group where the second device is located usually have redundant experts deployed, the cloud management platform can delete the redundant experts (including at least one of the first shared expert and the second redundant expert on the second device) on each device in the device group where the second device is located, and migrate the routing experts (including the first routing expert on the first device) of each device in the device group where the first device is located to the corresponding devices (including the second device) in the device group where the second device is located.

[0070] In addition, the cloud management platform can also remove the device group containing the first device from among these multiple devices.

[0071] In addition, the cloud management platform can also modify the location of the routing expert on each device in the device group where the first device is located to the corresponding device in the device group where the second device is located in the mapping table, thereby obtaining an updated mapping table.

[0072] Continuing with the example above, after identifying device 65, since device 3 is located in device group 1 and device 65 is located in device group 17, the cloud management platform can remove redundant experts 1 to 4 from devices 65 to 68 in device group 17, and migrate routing experts 1 to 4 from devices 1 to 4 in device group 1 to devices 65 to 68. Then, these four devices from device group 1 are removed from the inference instance. In this way, routing experts 1 to 4 can be replaced by devices 65 to 68, thus ensuring the normal operation of the entire target model. Therefore, the cloud management platform can continue to use the target model for inference through this inference instance.

[0073] In addition, the cloud management platform can also change the location of routing expert 1 to routing expert 4 in the mapping table from device 1 to device 4 to device 65 to device 68, thereby obtaining an updated mapping table.

[0074] More specifically, the cloud management platform can also perform the following operations:

[0075] If these multiple redundancy experts include the first redundancy expert, the cloud management platform can, based on the mapping table, select a third device running the first redundancy expert from among these devices and remove the first device from the list. Then, the cloud management platform can modify the location of the first routing expert in the mapping table from the first device to the third device (because the parameters of the first redundancy expert are usually consistent with those of the first routing expert), thus obtaining an updated mapping table. Therefore, the cloud management platform indirectly replaces the first redundancy expert with the first routing expert in the third device. In this way, the cloud management platform can successfully enable the third device to run the first routing expert, thereby ensuring the normal operation of the target model.

[0076] More specifically, since the first device and the third device are typically located in different device groups, it should be noted that devices in the device group where the first device is located usually have routing experts deployed, and devices in the device group where the third device is located usually have redundancy experts deployed. Furthermore, these routing experts and redundancy experts are usually corresponding; therefore, the cloud management platform can also modify the location of the routing experts on each device in the device group where the first device is located to match the corresponding devices in the device group where the third device is located, thus obtaining an updated mapping table.

[0077] In addition, the cloud management platform can also remove the device group containing the first device from among these multiple devices.

[0078] As in the example above, such as Figure 6 As shown ( Figure 6 (This is another structural diagram of the reasoning example provided in the embodiments of this application). Since there is a redundant expert 5 (i.e. the aforementioned first redundant expert) corresponding to the routing expert 3 among the redundant experts 1 to 32, the cloud management platform can determine that the redundant expert 5 is located in device 69 by the mapping table. Therefore, the cloud management platform can determine that device 69 (i.e. the aforementioned third device) is used to replace device 3.

[0079] After identifying device 69, since device 3 is located in device group 1 and device 65 is located in device group 18, the cloud management platform can determine, based on this mapping table, that redundant experts 5 to 8 on devices 69 to 72 in device group 18 correspond to routing experts 1 to 4 on devices 1 to 4 in device group 1. Therefore, the cloud management platform can modify the location of routing experts 1 to 4 in the mapping table from devices 1 to 4 to devices 69 to 72, thus obtaining an updated mapping table. In this way, the cloud management platform can successfully allow routing experts 1 to 4 to be replaced by devices 69 to 72, thereby ensuring the normal operation of the entire target model. Therefore, the cloud management platform can continue to use the target model for inference through this inference instance.

[0080] More specifically, the cloud management platform can also perform the following operations:

[0081] If the fourth device among these multiple devices fails, since the fourth device operates with a second shared expert, the cloud management platform can select the device with the lightest load from the devices with redundant experts deployed among these multiple devices as the fifth device, and remove the fourth device from the multiple devices. The fifth device operates with a third redundant expert.

[0082] Once the fifth device is identified, the cloud management platform can instruct it to remove the third redundant expert it is running and to start running the second shared expert. In this way, the cloud management platform can successfully allow the fifth device to replace the fourth device in running the second shared expert, thus ensuring the normal operation of the target model.

[0083] More specifically, since the fourth and fifth devices are typically located in different device groups, it should be noted that devices in the device group containing the fourth device usually have shared experts deployed, while devices in the device group containing the fifth device usually have redundant experts deployed. Therefore, the cloud management platform can delete the redundant experts (including the third redundant expert on the fifth device) from each device in the device group containing the fifth device, and migrate the shared experts (including the second shared expert on the fourth device) from each device in the device group containing the fourth device to the corresponding devices in the device group containing the fifth device (including the fifth device).

[0084] In addition, the cloud management platform can also remove the device group containing the fourth device from among these multiple devices.

[0085] In addition, the cloud management platform can also modify the location of the shared expert on each device in the device group where the fourth device is located to the corresponding device in the device group where the fifth device is located in the mapping table, thereby obtaining an updated mapping table.

[0086] As in the example above, such as Figure 7 As shown ( Figure 7 (This is another structural diagram of the reasoning example provided in the embodiments of this application). Assuming that the shared expert 1 is located in device 97 (the aforementioned fourth device) is faulty, the cloud management platform can determine that the load of device 73 is the lightest based on the load of device 65 to device 96. Therefore, the cloud management platform can determine that device 73 (i.e. the aforementioned fifth device) is used to replace device 97.

[0087] After identifying device 73, since device 97 is located in device group 25 and device 73 is located in device group 19, the cloud management platform can remove redundant experts 9 to 12 from devices 73 to 76 in device group 19, and migrate shared experts 1 to 4 from devices 97 to 100 in device group 25 to devices 73 to 76. Then, these four devices from device group 25 are removed from this inference instance. In this way, shared experts 1 to 4 can be replaced by devices 73 to 76, thus ensuring the normal operation of the entire target model. Therefore, the cloud management platform can continue to use the target model for inference through this inference instance.

[0088] In addition, the cloud management platform can also change the location of shared experts 1 to 4 in the mapping table from device 97 to device 100 to device 73 to device 76, thereby obtaining an updated mapping table.

[0089] More specifically, the cloud management platform can also perform the following operations:

[0090] If the sixth device among these multiple devices malfunctions, since it operates a fourth redundant expert, the cloud management platform does not need to perform an expert replacement operation. It can simply remove the sixth device from the pool of devices, thus not affecting the normal operation of the target model. Of course, the cloud management platform can also delete entries related to the fourth redundant expert from the mapping table to obtain an updated mapping table.

[0091] As in the example above, if device 77 (i.e. the aforementioned sixth device) is faulty, the cloud management platform can determine that device 77 is running a redundancy expert 13 based on the mapping table. Therefore, the cloud management platform can directly remove device 77 from the inference instance and delete any content related to redundancy expert 13 from the mapping table.

[0092] More specifically, the cloud management platform can also perform the following operations:

[0093] Since the target model typically includes models other than multiple routing experts, multiple shared experts, and multiple redundant experts, this model can usually refer to the remaining model layers (e.g., at least one of MLP layers, attention layers, and LMhead layers) besides the MoE layer. Because the model can be partitioned into multiple first sub-models based on a tensor parallel strategy (e.g., at least one of multiple sub-MLPs, multiple sub-attentions, and multiple sub-LMheads), the number of these multiple first sub-models is usually associated with the first degree of parallelism of the tensor parallel strategy. Any one of these multiple devices runs one of the multiple first sub-models.

[0094] After removing the device group containing the first device from the list of devices, the cloud management platform determines the number of the remaining devices (also referred to as the residual devices) and redetermines the second degree of parallelism for the tensor parallelism strategy based on this number. Then, the cloud management platform can repartition the model based on the second degree of parallelism, resulting in multiple second sub-models. Generally, the number of these second sub-models is less than the number of the first sub-models.

[0095] After obtaining these multiple second sub-models, the cloud management platform can instruct the remaining devices to stop running their original first sub-models and instead instruct them to run these multiple second sub-models. This ensures that the MoE layer of the entire target model, as well as the other model layers, run normally on the remaining devices to continue inference. Each of the remaining devices runs one of these multiple second sub-models.

[0096] It should be noted that since this inference instance contains multiple computing nodes, assuming the first device is located in the first computing node among these multiple computing nodes, the first computing node includes not only the device group to which the first device belongs, but also devices outside of that device group. After removing the device group to which the first device belongs from the first computing node, the first computing node only contains devices outside of that device group. Therefore, it can be seen that the remaining devices in this inference instance, excluding the device group to which the first device belongs, comprise two parts: the first part consists of devices within the first computing node excluding the device group to which the first device belongs, and the second part consists of devices within the remaining computing nodes excluding the first computing node. For these multiple first sub-models and these multiple second sub-models, the first sub-models and second sub-models deployed on the first part of the devices are sub-models of different sizes, while the first sub-models and second sub-models deployed on the second part of the devices are sub-models of the same size.

[0097] Still as Figure 8 The example ( Figure 8(This is another structural diagram of the inference example provided in the embodiments of this application). Since the target model includes not only the MoE layer, but also the MLP layer, the attention layer, and the LM head layer, assuming the parallelism of the TP strategy for the attention layer is 4, and the parallelism of the TP strategy for the MLP layer and the LM head layer is 16 (the aforementioned first parallelism), the MLP layer, the attention layer, and the LM head layer can include 128 sub-MLPs (each group of 16 sub-MLPs, and the 8 sub-MLPs in the same group can form a complete MLP layer, these 128 sub-MLPs are the aforementioned multiple first sub-models), 128 sub-attentions (each group of 4 sub-attentions, and the 4 sub-attentions in the same group can form a complete attention layer), and 128 sub-LM heads (each group of 16 sub-LM heads, and the 8 sub-LM heads in the same group can form a complete LM head layer, these 128 sub-LM heads are the aforementioned multiple first sub-models). In devices 1 to 128, device 1 deploys sub-MLP1, sub-attention1, and sub-LM. head1, ..., device 128 is equipped with sub-MLP128, sub-attention128 and sub-LM head128.

[0098] Therefore, it can be seen that the four devices in each device group can each deploy four sub-attentions of the same group, the 16 devices in the four device groups in each computing node can each deploy 16 sub-MLPs of the same group, and the 16 devices in the four device groups in each computing node can each deploy 16 sub-LM heads of the same group.

[0099] After removing device group 1 from compute node 1, compute node 1 (the aforementioned first compute node) will only have device groups 2 to 3 remaining (i.e., devices 5 to 16, which are the aforementioned first part of the devices), while the device groups of compute nodes 2 to 4 will remain unchanged (i.e., devices 17 to 128, which are the aforementioned second part of the devices). For compute node 1, since the number of devices has been reduced from 16 to 12, the cloud management platform can determine the parallelism of the TP strategy for the MLP layer and LM head layer to be 12. For compute nodes 2 to 4, the parallelism of the TP strategy for the MLP layer and LM head layer remains 16 (these two parallelisms are the aforementioned second parallelism). Therefore, the cloud management platform can re-divide the MLP layer and LM head layer based on two degrees of parallelism, thereby obtaining 124 new sub-MLPs (i.e., the aforementioned multiple second sub-models) and 124 new sub-LM heads (i.e., the aforementioned multiple second sub-models). Among them, new sub-MLP1 to new sub-MLP12 are deployed on devices 5 to 16, new sub-MLP13 to new sub-MLP124 are deployed on devices 17 to 128, new sub-LM head1 to new sub-LM head12 are deployed on devices 5 to 16, and new sub-LM head13 to new sub-LM head124 are deployed on devices 17 to 128.

[0100] Among them, the size of the new sub-MLP1 is different from the size of the original sub-MLP5, the size of the new sub-MLP2 is different from the size of the original sub-MLP6, ..., the size of the new sub-MLP12 is different from the size of the original sub-MLP16. The size of the new sub-MLP13 is the same as the size of the original sub-MLP17, the size of the new sub-MLP14 is the same as the size of the original sub-MLP18, ..., the size of the new sub-MLP124 is the same as the size of the original sub-MLP128. Similarly, the sizes of the 124 new sub-LM heads are the same as the sizes of the corresponding original 124 sub-LM heads, which will not be elaborated here.

[0101] It should be noted that the parallelism of the TP strategy for the attention layer remains at 4, so there is no need to re-divide the attention layer. These 124 devices can retain the original sub-attention.

[0102] More specifically, the cloud management platform can also perform the following operations:

[0103] Since the sum of the number of routing experts, shared experts, and redundant experts contained in the MoE layer of the target model is related to the third degree of parallelism of the expert parallelism strategy, after removing the device group containing the first device from these multiple devices, the experts and sub-models on the remaining devices will inevitably change, leading to changes in the communication relationships between the remaining devices. Therefore, the cloud management platform can use the number of remaining devices to determine the fourth degree of parallelism of the expert parallelism strategy, and then use the number of remaining devices, the second degree of parallelism of the tensor parallelism strategy, and the fourth degree of parallelism of the expert parallelism strategy to obtain the (new) communication relationships (also known as the (new) communication domains covering the remaining devices) between each device. This communication relationship includes the communication objects and their communication order of each device in the remaining devices. Therefore, this communication relationship can be used to instruct the devices in the remaining devices to communicate in order to run the target model to process the target task from the tenant.

[0104] It is worth noting that after establishing (new) communication relationships between the remaining devices, the cloud management platform can retrieve the previously established communication relationships between the remaining devices, compare the differences between the (new) communication relationships and the original communication relationships, and directly apply the changed parts of these two communication relationships to the corresponding devices, while leaving the unchanged parts unchanged. Specifically, among the remaining devices, the (new) communication relationships between the first group of devices are different from their original communication relationships, the (new) communication relationships between the first and second group of devices are also different from their original communication relationships, while the (new) communication relationships between the second group of devices are unchanged. Therefore, the cloud management platform can apply the (new) communication relationships between the first group of devices only to the first group of devices, and apply the (new) communication relationships between the first and second group of devices to both the first and second group of devices.

[0105] Continuing with the example above, for devices 1 to 128 of this inference instance, since each device has an expert deployed, the parallelism of the EP strategy is 128 (the aforementioned third parallelism). After removing device group 1 from the inference instance, only devices 5 to 128 remain. Therefore, the cloud management platform can determine the parallelism of the EP strategy to be 124 (the aforementioned fourth parallelism). Thus, the cloud management platform can determine the parallelism of the EP strategy based on the number of these 124 devices, the parallelism of the TP strategy for the attention layer (4), the parallelism of the TP strategy for the MLP layer (12 and 16), and the parallelism of the LM strategy. The parallelism of the TP strategy in the head layer (12 and 16) and the parallelism of the EP strategy (124) determine the (new) communication objects and (new) communication order (i.e. (new communication relationship) of device 5 among these 124 devices, ... the communication objects and communication order of device 124 among these 124 devices, thus obtaining a new communication domain covering these 124 devices. This new communication domain can be used to instruct these 124 devices to communicate in order to successfully run the target model and complete the target task.

[0106] It should be noted that the (new) communication relationships within devices 5 to 16 are different from the original communication relationships. Therefore, the cloud management platform only needs to apply these new communication relationships to devices 5 to 16. Similarly, the (new) communication relationships between devices 5 to 16 and between devices 17 to 128 are different from the original communication relationships. Therefore, the cloud management platform only needs to apply these new communication relationships to the relationships between devices 5 to 16 and between devices 17 to 128, while the communication relationships within devices 17 to 128 do not need to be changed.

[0107] More specifically, the cloud management platform can also perform the following operations:

[0108] After completing the above operations, the cloud management platform can notify the remaining devices to continue inference to complete the tenant's target task. Since the target task includes multiple subtasks pointing to multiple routing experts, multiple subtasks pointing to multiple redundant experts, and multiple subtasks pointing to multiple shared experts, the cloud management platform can distribute these subtasks and the updated mapping table to the remaining devices. Then, since these subtasks include a first subtask pointing to the first routing expert and the remaining subtasks pointing to the other experts, the second or third device, which has the first routing expert deployed, can independently select the first subtask from these subtasks based on the updated mapping table and call the first routing expert to process the first subtask (equivalent to routing the first subtask to the first routing expert). Similarly, the remaining subtasks within these subtasks are handled in the same way, which will not be elaborated here.

[0109] As in the example above, gating networks are deployed in devices 5 through 128, and these gating networks can be considered part of the MoE layer. After the cloud management platform instructs devices 5 through 128 to run the target model and continue inference, assuming the target task to be completed by the target model includes tokens 1 (the aforementioned first subtask) to 124 to be processed, the cloud management platform can distribute the updated mapping table and these 124 tokens to the gating networks of devices 5 through 128. Assuming device 65 runs routing expert 3, and token 3 is a token pointing to routing expert 3, the gating network of device 65 can automatically select token 3 from these 124 tokens and hand it over to routing expert 3 for processing. Similarly, the other tokens (i.e., the other subtasks) are handled in the same way, and will not be elaborated here.

[0110] More specifically, the cloud management platform can also perform the following operations:

[0111] The cloud management platform can pre-compile multiple computation graphs. Different computation graphs can contain different operators, and each graph corresponds to a different degree of parallelism for the EP (Extended Processing) strategy, a different degree of parallelism for the TP (Transaction Processing) strategy, and a different number of devices. In other words, for a given inference instance, the cloud management platform can determine a suitable computation graph from these multiple graphs based on the number of devices in the inference instance, the first degree of parallelism for the TP strategy, and the third degree of parallelism for the EP strategy. Then, it can guide these devices to run the target model according to the operators of this computation graph, thereby completing the tenant's business. After removing the device group containing the first device from these multiple devices, the cloud management platform can determine another suitable computation graph from these multiple graphs based on the number of remaining devices, the second degree of parallelism for the TP strategy, and the third degree of parallelism for the EP strategy. Then, it can guide the remaining devices to continue running the target model according to the operators of this other computation graph, thereby completing the tenant's business.

[0112] Of course, the cloud management platform can also incrementally compile the original computation graph based on the number of other devices, the second degree of parallelism of the TP strategy, and the third degree of parallelism of the EP strategy, so as to obtain another computation graph suitable for the other devices.

[0113] It should be understood that this embodiment is illustrated illustratively assuming that the first device does not have a backup device. In this case, the cloud management platform needs to select a second or third device to replace the first device in running the first routing device. In other possible cases, if the first device has a backup device, the cloud management platform can directly copy the first routing expert running on the first device and all relevant information of the first device to the backup device. In this way, the backup device is directly added as a new device to these multiple devices, and the first device is removed from these multiple devices. However, the number of these multiple devices remains unchanged as before. Therefore, there is no need for various operations such as expert replacement, model parameter reallocation, computation graph reallocation, communication domain reconstruction, and route adaptation.

[0114] In this embodiment, a certain inference instance managed by the cloud management platform may include multiple devices. These devices may run a target model, which may include multiple routing experts, multiple shared experts, and multiple redundant experts. Each redundant expert corresponds to at least one of the routing experts. For any one of these devices, it may run any one or any combination of at least one routing expert, at least one shared expert, and at least one redundant expert. Therefore, if the first device among these devices malfunctions, since it runs a first routing expert, the cloud management platform can detect whether a first redundant expert corresponding to the first routing expert exists among the redundant experts. If not, the cloud management platform can select a second device from among the multiple devices to replace the first device and remove the first device from the multiple devices. The second device runs at least one of a first shared expert and a second redundant expert. Then, the cloud management platform can notify the second device to delete at least one of its running first shared expert and second redundant expert, and notify the second device to run the first routing expert. In this way, the cloud management platform can successfully allow the second device to replace the first device in running the first routing expert, ensuring the normal operation of the target model. In the aforementioned process, when the cloud management platform detects a fault in the first device of the inference instance, it can transfer the first routing expert it is running to the second device running the second redundant expert and the first shared expert. This ensures that the target model continues to run normally on the second device even if only the first device is removed from the list. This avoids abandoning the entire inference instance, which is conducive to the continuous and effective use of existing device resources and avoids resource waste.

[0115] Furthermore, in this embodiment of the application, the cloud management platform implements expert replacement on the device (i.e., in the second device, at least one of the original second redundant expert and the first shared expert is replaced with the first routing expert on the first device). This allows the inference instance to not exit the inference process related to the target model, ensuring that the routing expert is intact, so as to guarantee the subsequent normal operation of the target model and the inference accuracy.

[0116] Furthermore, in this embodiment of the application, the cloud management platform can perform a re-partition of the model layer other than the MoE layer based on the number of remaining devices and the change in the parallelism of the TP strategy (i.e., re-partitioning based on the number of remaining devices and the second parallelism of the TP strategy to obtain multiple second sub-models). This can ensure that the remaining devices after the adjustment continue to run the target model normally, so as to ensure that the inference process in the remaining devices does not exit.

[0117] Furthermore, in this embodiment of the application, the cloud management platform can also rebuild the communication domain for the remaining devices. In the newly built communication domain, the communication relationships of some of the remaining devices (i.e., the communication relationships between the first group of devices and the communication relationships between the first group of devices and the second group of devices) may change, while the communication relationships of another group of devices (i.e., the communication relationships between the second group of devices) may remain consistent with the original. Therefore, the cloud management platform only needs to dynamically replace the changed communication relationships, which can reduce the communication domain reconstruction operation to a certain extent.

[0118] Furthermore, in this embodiment of the application, the cloud management platform can avoid erroneous expert activation by promptly routing subsequent unprocessed subtasks (e.g., the first subtask) to the routing expert whose location has changed (e.g., the first routing expert), thereby correctly completing the tenant's task.

[0119] The above is a detailed description of the model management method based on a cloud management platform provided in the embodiments of this application. The cloud management platform provided in the embodiments of this application will be introduced below. Figure 9 A schematic diagram of the structure of the cloud management platform provided in the embodiments of this application is shown below. Figure 9 As shown, the cloud management platform is used to manage multiple devices, which run a target model. The target model includes multiple routing experts, multiple shared experts, and multiple redundancy experts. Each redundant expert corresponds to at least one of the routing experts. Any one of the multiple devices runs any one or any combination of at least one routing expert, at least one shared expert, and at least one redundancy expert. The cloud management platform includes:

[0120] The first determining module 901 is used to determine that a first device among multiple devices is faulty, wherein the first device is running a first routing expert; for example, the first determining module 901 can be used to implement Figure 4 Step 401 of the illustrated embodiment.

[0121] The second determining module 902 is used to determine that the multiple redundant experts do not include the first redundant expert corresponding to the first routing expert, select a second device from the multiple devices to replace the first device, and remove the first device from the multiple devices, wherein the second device operates at least one of the first shared expert and the second redundant expert; for example, the second determining module 902 can be used to implement Figure 4 Step 402 of the illustrated embodiment.

[0122] The first notification module 903 is used to notify the second device to delete at least one of the first shared expert and the second redundant expert, and to notify the second device to run the first routing expert. For example, the first notification module 903 can be used to implement... Figure 4 Step 403 of the illustrated embodiment.

[0123] In one possible implementation, the cloud management platform further includes: a third determining module, used to determine that the multiple redundant experts include the first redundant expert, select a third device running the first redundant expert from the multiple devices, and remove the first device from the multiple devices; and a second notification module, used to notify the third device to replace the first redundant expert with the first routing expert so that the first routing expert can be run through the third device.

[0124] In one possible implementation, the cloud management platform further includes: a fourth determination module for determining that a fourth device among multiple devices is faulty, wherein the fourth device is running a second shared expert; a fourth determination module for selecting a fifth device from the multiple devices and removing the fourth device from the multiple devices, wherein the fifth device is running a third redundancy expert; and a third notification module for notifying the fifth device to delete the third redundancy expert and notifying the fifth device to run the second shared expert.

[0125] In one possible implementation, the cloud management platform further includes: a fifth determination module for determining that a sixth device among multiple devices is faulty, wherein the sixth device operates a fourth redundancy expert; and a removal module for removing the sixth device from the multiple devices.

[0126] In one possible implementation, multiple devices are divided into multiple device groups based on a tensor parallelism strategy. The number of device groups is related to the first degree of parallelism of the tensor parallelism strategy, and the first device and the second device belong to different device groups.

[0127] In one possible implementation, the target model further includes models other than multiple routing experts, multiple shared experts, and multiple redundant experts. The model is partitioned into multiple first sub-models based on a tensor parallel strategy. The number of the multiple first sub-models is related to the first degree of parallelism of the tensor parallel strategy. Any one of the multiple devices runs one of the multiple first sub-models. The cloud management platform also includes: a sixth determining module, used to determine the second degree of parallelism of the tensor parallel strategy based on the number of devices other than the first device; a partitioning module, used to partition the model based on the second degree of parallelism to obtain multiple second sub-models, wherein the number of the multiple second sub-models is less than the number of the multiple first sub-models; and a fourth notification module, used to notify the other devices to run the multiple second sub-models, wherein any one of the other devices runs one of the multiple second sub-models.

[0128] In one possible implementation, the sum of the number of multiple routing experts, the number of multiple shared experts, and the number of multiple redundant experts is associated with the third degree of parallelism of the expert parallelism strategy. The cloud management platform also includes: a seventh determination module, used to determine the fourth degree of parallelism of the expert parallelism strategy based on the number of remaining devices; and an acquisition module, used to acquire the communication relationships between the remaining devices based on the number of remaining devices, the second degree of parallelism, and the fourth degree of parallelism, wherein the communication relationships are used to instruct the devices to communicate with each other to run the target model to process the target task.

[0129] In one possible implementation, the target task includes multiple subtasks pointing to multiple routing experts, and the multiple subtasks include a first subtask pointing to a first routing expert. The cloud management platform also includes a fifth notification module for notifying a second or third device to process the first subtask through the first routing expert.

[0130] It should be noted that the information interaction and implementation process between the modules / units of the above-mentioned device are based on the same concept as the method embodiments of this application, and the resulting technical effects are the same as those of the method embodiments of this application. For details, please refer to the description in the method embodiments shown above in the embodiments of this application, and will not be repeated here.

[0131] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 10As shown, the computing device 1000 (which can be used to present the aforementioned cloud management platform) includes: a processor 1001, a memory 1002, a communication interface 1003, and a bus 1004. The processor 1001, memory 1002, and communication interface 1003 are coupled via the bus. The memory 1002 stores instructions. When the execution instructions in the memory 1002 are executed, the computing device 1000 executes the method performed by the cloud management platform in the above method embodiment.

[0132] The computing device 1000 may be one or more integrated circuits configured to implement the methods described above, such as: one or more application-specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs), or a combination of at least two of these forms of integrated circuits. Furthermore, when the units in the device can be implemented in the form of a processing element scheduler, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling programs. Alternatively, these units may be integrated together to implement a system-on-a-chip (SOC).

[0133] The processor 1001 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0134] The memory 1002 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0135] The memory 1002 stores executable program code, and the processor 1001 executes the executable program code to implement the functions of the aforementioned first determining module, second determining module, and first notification module, thereby realizing the aforementioned model management method based on the cloud management platform. That is, the memory 1002 stores instructions for executing the aforementioned model management method based on the cloud management platform.

[0136] The communication interface 1003 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.

[0137] In addition to the data bus, the 1004 bus can also include a power bus, a control bus, and a status signal bus. The bus can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The bus can be divided into address bus, data bus, and control bus.

[0138] Please see Figure 11 , Figure 11 This is a schematic diagram of a computing device cluster provided in an embodiment of this application. Figure 11 As shown, the computing device cluster 1100 includes at least one computing device 1000.

[0139] like Figure 11 As shown, the computing device cluster 1100 includes at least one computing device 1000. The memory 1002 of one or more computing devices 1000 in the computing device cluster 1100 may store the same instructions for executing the above-described model management method based on the cloud management platform.

[0140] In some possible implementations, the memory 1002 of one or more computing devices 1000 in the computing device cluster 1100 may also store partial instructions for executing the aforementioned model management method based on the cloud management platform. In other words, a combination of one or more computing devices 1000 can jointly execute the aforementioned model management method based on the cloud management platform.

[0141] It should be noted that the memory 1002 in different computing devices 1000 within the computing device cluster 1100 can store different instructions, which are used to execute certain functions of the aforementioned cloud management platform. That is, the instructions stored in the memory 1002 of different computing devices 1000 can implement the functions of one or more modules, such as the first determining module, the second determining module, and the first notification module.

[0142] In some possible implementations, one or more computing devices 1000 in the computing device cluster 1100 can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0143] Please see Figure 12 , Figure 12 This is a schematic diagram illustrating the network connection of computer devices in a computer cluster 1200 provided in an embodiment of this application. Figure 12 As shown, the two computing devices 1000A and 1000B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0144] In one possible implementation, the memory in computing device 1000A stores instructions for performing the functions of modules such as the first determining module. Meanwhile, the memory in computing device 1000B stores instructions for performing the functions of modules such as the second determining module and the first notification module.

[0145] It should be understood that Figure 12 The functions of computing device 1000A shown can also be performed by multiple computing devices. Similarly, the functions of computing device 1000B can also be performed by multiple computing devices.

[0146] This application also relates to a computer storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform actions such as... Figure 4 The steps performed by the cloud management platform in the illustrated embodiment.

[0147] This application also relates to a computer program product that stores instructions that, when executed by a computer, cause the computer to perform actions such as... Figure 4 The steps performed by the cloud management platform in the illustrated embodiment.

[0148] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0149] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0150] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0151] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0152] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A model management method based on a cloud management platform, characterized in that, The cloud management platform is used to manage multiple devices, which run a target model. The target model includes multiple routing experts, multiple shared experts, and multiple redundant experts. Each redundant expert corresponds to at least one of the multiple routing experts. Any one of the multiple devices runs any one or any combination of at least one routing expert, at least one shared expert, and at least one redundant expert. The method includes: The cloud management platform determines that the first device among the plurality of devices is faulty, wherein the first device is running a first routing expert; The cloud management platform determines that the plurality of redundant experts does not include the first redundant expert corresponding to the first routing expert, selects a second device from the plurality of devices to replace the first device, and removes the first device from the plurality of devices, wherein the second device runs at least one of the first shared expert and the second redundant expert; The cloud management platform notifies the second device to delete at least one of the first shared expert and the second redundant expert, and notifies the second device to run the first routing expert.

2. The method according to claim 1, characterized in that, The method further includes: The cloud management platform determines that the plurality of redundancy experts includes the first redundancy expert, selects a third device running the first redundancy expert from the plurality of devices, and removes the first device from the plurality of devices; The cloud management platform notifies the third device to replace the first redundant expert with the first routing expert, so that the first routing expert can be run through the third device.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The cloud management platform determines that the fourth device among the plurality of devices is faulty, wherein the fourth device is running a second shared expert; The cloud management platform selects a fifth device from the plurality of devices and removes the fourth device from the plurality of devices, wherein the fifth device runs a third redundancy expert; The cloud management platform notifies the fifth device to delete the third redundant expert and notifies the fifth device to run the second shared expert.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The cloud management platform determines that the sixth device among the plurality of devices is faulty, wherein the sixth device is running a fourth redundancy expert; The cloud management platform removes the sixth device from the plurality of devices.

5. The method according to any one of claims 1 to 4, characterized in that, The multiple devices are divided into multiple device groups based on a tensor parallelism strategy. The number of the multiple device groups is related to the first parallelism of the tensor parallelism strategy. The first device and the second device belong to different device groups.

6. The method according to any one of claims 1 to 5, characterized in that, The target model further includes models other than the plurality of routing experts, the plurality of shared experts, and the plurality of redundant experts. The model is partitioned into multiple first sub-models based on a tensor parallel strategy. The number of the multiple first sub-models is related to the first parallelism of the tensor parallel strategy. Any one of the plurality of devices runs one of the multiple first sub-models. The method further includes: The cloud management platform determines the second degree of parallelism of the tensor parallel strategy based on the number of devices other than the first device among the multiple devices. The cloud management platform divides the model based on the second degree of parallelism to obtain multiple second sub-models, wherein the number of the multiple second sub-models is less than the number of the multiple first sub-models; The cloud management platform notifies the remaining devices to run the plurality of second sub-models, wherein any one of the remaining devices runs one of the plurality of second sub-models.

7. The method according to claim 6, characterized in that, The sum of the number of multiple routing experts, the number of multiple shared experts, and the number of multiple redundant experts is associated with the third degree of parallelism of the expert parallelism strategy. The method further includes: The cloud management platform determines the fourth degree of parallelism of the expert parallel strategy based on the number of the remaining devices; The cloud management platform obtains the communication relationship between each of the remaining devices based on the number of remaining devices, the second degree of parallelism, and the fourth degree of parallelism. The communication relationship is used to instruct the various devices to communicate with each other in order to run the target model to process the target task.

8. The method according to claim 7, characterized in that, The target task includes multiple subtasks pointing to the plurality of routing experts, and the plurality of subtasks includes a first subtask pointing to the first routing expert. The method further includes: The cloud management platform notifies the second or third device to process the first sub-task through the first routing expert.

9. A cloud management platform, characterized in that, The cloud management platform is used to manage multiple devices, which run a target model. The target model includes multiple routing experts, multiple shared experts, and multiple redundant experts. Each redundant expert corresponds to at least one of the multiple routing experts. Each of the multiple devices runs any one or any combination of at least one routing expert, at least one shared expert, and at least one redundant expert. The cloud management platform includes: A first determining module is used to determine that a first device among the plurality of devices is faulty, wherein the first device is running a first routing expert; The second determining module is used to determine that the plurality of redundant experts does not include the first redundant expert corresponding to the first routing expert, select a second device from the plurality of devices to replace the first device, and remove the first device from the plurality of devices, wherein the second device operates at least one of a first shared expert and a second redundant expert; The first notification module is used to notify the second device to delete at least one of the first shared expert and the second redundant expert, and to notify the second device to run the first routing expert.

10. The cloud management platform according to claim 9, characterized in that, The cloud management platform also includes: The third determining module is used to determine that the plurality of redundancy experts includes the first redundancy expert, select a third device running the first redundancy expert from the plurality of devices, and remove the first device from the plurality of devices; The second notification module is used to notify the third device to replace the first redundant expert with the first routing expert, so that the first routing expert can be run through the third device.

11. The cloud management platform according to claim 9 or 10, characterized in that, The cloud management platform also includes: The fourth determining module is used to determine that a fourth device among the plurality of devices is faulty, wherein the fourth device is running a second shared expert; The fourth determining module is used to select a fifth device from the plurality of devices and remove the fourth device from the plurality of devices, wherein the fifth device is running a third redundancy expert; The third notification module is used to notify the fifth device to delete the third redundant expert and to notify the fifth device to run the second shared expert.

12. The cloud management platform according to any one of claims 9 to 11, characterized in that, The cloud management platform also includes: The fifth determining module is used to determine that the sixth device among the plurality of devices is faulty, wherein the sixth device is running a fourth redundancy expert; The rejection module is used to reject the sixth device from the plurality of devices.

13. The cloud management platform according to any one of claims 9 to 12, characterized in that, The multiple devices are divided into multiple device groups based on a tensor parallelism strategy. The number of the multiple device groups is related to the first parallelism of the tensor parallelism strategy. The first device and the second device belong to different device groups.

14. The cloud management platform according to any one of claims 9 to 13, characterized in that, The target model also includes models other than the plurality of routing experts, the plurality of shared experts, and the plurality of redundant experts. The model is partitioned into multiple first sub-models based on a tensor parallel strategy. The number of the multiple first sub-models is related to the first parallelism of the tensor parallel strategy. Any one of the plurality of devices runs one of the multiple first sub-models. The cloud management platform also includes: The sixth determining module is used to determine the second parallelism of the tensor parallelism strategy based on the number of the remaining devices among the plurality of devices excluding the first device; A partitioning module is used to partition the model based on the second parallelism to obtain multiple second sub-models, wherein the number of the multiple second sub-models is less than the number of the multiple first sub-models; The fourth notification module is used to notify the remaining devices to run the plurality of second sub-models, wherein any one of the remaining devices runs one of the plurality of second sub-models.

15. The cloud management platform according to claim 14, characterized in that, The sum of the number of multiple routing experts, the number of multiple shared experts, and the number of multiple redundant experts is related to the third degree of parallelism of the expert parallel strategy. The cloud management platform also includes: The seventh determining module is used to determine the fourth degree of parallelism of the expert parallel strategy based on the number of the remaining devices; The acquisition module is used to acquire the communication relationship between each of the remaining devices based on the number of remaining devices, the second parallelism, and the fourth parallelism, wherein the communication relationship is used to instruct the various devices to communicate with each other to run the target model to process the target task.

16. The cloud management platform according to claim 15, characterized in that, The target task includes multiple subtasks pointing to the plurality of routing experts, and the plurality of subtasks includes a first subtask pointing to the first routing expert. The cloud management platform further includes: The fifth notification module is used to notify the second or third device to process the first subtask through the first routing expert.

17. A computing device cluster, characterized in that, The computing device cluster includes at least one computing device, each computing device including a processor and memory: The memory is used to store instructions; The processor is configured to, according to the instructions, cause the computing device cluster to perform the method of any one of claims 1 to 8.

18. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 8.

19. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 8.

Images