Inference method and related device

Abstract

The present application relates to the field of artificial intelligence, and in particular to an inference method and a related device. The method comprises: a first computing device acquiring an inference request statement of a user, and splitting the inference request statement so as to obtain a plurality of tokens; on the basis of a mapping table of parts of speech and devices, dividing the plurality of tokens into K groups of tokens, and determining the correspondence between the K groups of tokens and K second computing devices, wherein in the mapping table of parts of speech and devices, the parts of speech corresponding each of the K second computing devices comprise parts of speech of tokens in a token group corresponding to said second computing device; the first computing device respectively sending the K groups of tokens to the K second computing devices, such that each second computing device processes the received tokens on the basis of deployed experts, so as to obtain an inference response statement for responding to the inference request statement; and the first computing device receiving K inference response statements respectively sent by the K second computing devices. The use of the solution of the present application facilitates reduction of communication overhead.

Description

Reasoning methods and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on March 20, 2024, with application number 202410334703.1 and invention name “Inference Method and Related Equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence, and in particular to a reasoning method and related equipment. Background Art

[0003] The mixture of experts (MOE) architecture is a common method for model sparsification and is often used for models with greatly expanded scale. Figure 1 describes the dense architecture (left) and MoE architecture (right) of a Transformer model with a decoder-only architecture. Generally speaking, a dense model is composed of multiple TransformerBlocks stacked together, and each TransformerBlock contains a Self-Attention module and a feedforward neural network (FFN) module. In the MoE architecture, a single feedforward neural network is replaced by multiple FFNs (i.e., the E in the right figure). 21 ,…,E 2k ), and a gating unit (i.e. G2 in the right figure) controls which FFN is activated. In the MoE architecture, each FFN is called an expert.

[0004] Figure 2 illustrates an example of deploying a six-layer MoE model with eight experts per layer on two devices using expert parallelism. In this example, the model receives a total of 16 tokens, with each device processing eight tokens. In each layer, the token is computed by the gating unit G, sent to the target device for computation in an all-to-all manner, and then transmitted back to the local device. Therefore, the computational process of expert parallelism can be summarized as follows: local token computation (attention operator) → token distribution via remote communication (all-to-all) → local token computation (expert operator) → token collection via remote communication (all-to-all).

[0005] However, there is a problem with using experts to deploy large models of the MoE architecture in parallel: after the token is calculated by the gating unit, it often needs to be sent to experts in other locations for calculation, which brings all-to-all communication overhead. Summary of the Invention

[0006] The embodiments of the present application provide an inference method and related devices, which are conducive to improving the affinity between word units and experts, thereby helping to reduce communication overhead.

[0007] In a first aspect, embodiments of the present application provide an inference method. The method utilizes a first computing device, which is applied to an inference system. The inference system is applied to an inference model, wherein the inference model includes L MOE layers, each MOE layer includes multiple experts, and the inference system further includes K second computing devices, each of which is equipped with an expert from each MOE layer, where L and K are both integers greater than 1. The method includes:

[0008] The first computing device obtains the user's inference request statement and splits the inference request statement to obtain multiple word-grams; the first computing device divides the multiple word-grams into K groups of word-grams based on the part-of-speech-device mapping table, and determines the correspondence between the K groups of word-grams and the K second computing devices; the part of speech corresponding to each second computing device in the part-of-speech-device mapping table includes the part of speech of the word-grams in the word-gram group corresponding to each second computing device; the first computing device sends K groups of word-grams to the K second computing devices respectively, so that each second computing device processes the received word-grams based on the deployed experts to obtain an inference response statement for responding to the inference request statement; the first computing device receives the K inference response statements sent by the K second computing devices respectively.

[0009] The first computing device and the second computing device are modules in the same server, for example, the first computing device is a CPU in the server, and the second computing device is a GPU in the server, or the first computing device and the second computing device are different servers.

[0010] It can be seen that grouping the word-grams in the inference request statement based on the part-of-speech-device mapping table, grouping the word-grams of the part-of-speech processed by the expert deployed on a second computing device, and assigning the group of word-grams to the expert on the second computing device for processing is beneficial to improving the affinity between the word-grams and the expert on the second computing device, so that when the second computing device calls the expert deployed on it to process the word-grams, the number of times the expert is activated remotely is reduced, which is beneficial to reducing the communication overhead between the second computing devices.

[0011] In conjunction with the first aspect, in one possible implementation, the method of this embodiment further includes:

[0012] The first computing device obtains the serial numbers of the experts called by the K second computing devices to process the received word-grams; the first computing device determines a word-gram activation path table based on the serial numbers of the experts called by the K second computing devices to process the received word-grams, the word-gram activation path table including the activation path of each word-gram in a plurality of word-grams, the activation path of each word-gram being composed of the serial number of the expert called when each word-gram is processed; the first computing device updates the expert placement table based on the word-gram activation path table and the word-gram part-of-speech table, wherein the expert placement table includes the identifiers of the K second computing devices and the identifiers of the experts deployed on each of the K second computing devices; the first computing device generates K expert adjustment instructions based on the updated expert placement table, and sends the K expert adjustment instructions to the K second computing devices respectively, the K expert adjustment instructions being used to instruct the K second computing devices to adjust the experts deployed on the K second computing devices respectively, and the expert placement table includes the expert serial number and the identifier of the second computing device to which the expert is deployed.

[0013] The expert placement table is updated using the expert information called when the second computing device processes word elements, and the experts deployed on the second computing device are adjusted based on the updated expert placement table. This is beneficial to improving the affinity between the experts deployed on the second computing device and the received word elements, and is beneficial to reducing the number of times the experts are activated remotely, thereby reducing the communication overhead between the second computing devices; at the same time, it is beneficial to ensure load balancing among different second computing devices.

[0014] In conjunction with the first aspect, in one possible implementation, the first computing device updates the expert placement table based on the word-unit activation path table and the word-unit part-of-speech table, including:

[0015] The first computing device converts the word unit activation path table into a first matrix; processes the first matrix based on one-hot encoding to obtain a second matrix; the first computing device clusters the second matrix to obtain a clustered matrix, wherein each row of word units in the clustered matrix corresponds to a cluster label; the first computing device processes the clustered matrix based on the cluster labels to obtain multiple sets; wherein the cluster labels of the word units included in each set are the same; the first computing device distributes the word units in the multiple sets into K buckets based on the sets; the difference in the number of word units in any two buckets of the K buckets is less than a preset threshold; the first computing device updates the expert placement table based on the K buckets, and updates the word unit-device table based on the K buckets and the word unit part-of-speech table.

[0016] In conjunction with the first aspect, in one possible implementation, K buckets correspond to K second computing devices, and the first computing device determines an expert placement table based on the K buckets, including:

[0017] For each MoE layer, the first computing device allocates T+s slots to each of the K second computing devices; T is the ratio of the number of types of experts in a single MOE layer to K; s is a positive integer; the first computing device determines the expert called when each word in each bucket of the K buckets is processed, and determines the number of times each expert in the called experts is called; the T experts with the largest number of calls are allocated to the T slots of the second computing device corresponding to each bucket to obtain the first allocation result; when the first computing device processes the word in the first bucket and there are still experts with unallocated resources, if they have not been allocated, If the number of experts who have not yet been allocated resources is not greater than s, the first computing device will allocate the experts who have not yet been allocated resources to the remaining slots of the second computing device corresponding to the first bucket; if the number of experts who have not yet been allocated resources is greater than s, the first computing device will allocate s experts among the experts who have not yet been allocated resources to the remaining s slots of the second computing device corresponding to the first bucket, and allocate the experts who have not yet been allocated resources except the s experts to the slots occupied by experts whose number of dedicated slots is greater than the first preset number, so as to obtain a second allocation result; the first computing device updates the expert placement table based on the first allocation result and the second allocation result.

[0018] Assigning the experts in the inference model to different second computing devices in this manner helps maintain load balancing across the K second computing devices, thereby improving the inference efficiency of the K second computing devices. When assigning experts, experts who are frequently called during word processing are prioritized for assignment to the same second computing device. This also helps reduce the number of remote expert activations, thereby reducing communication overhead between the second computing devices.

[0019] In conjunction with the first aspect, in one possible implementation, the first computing device updates the word-unit-device table based on the K buckets and the word-unit part-of-speech table, including:

[0020] The first computing device obtains the proportion of each part of speech in the word-grams included in each bin of the K bins corresponding to the K second computing devices based on the word-gram part-of-speech table; the first computing device determines the part of speech with the largest proportion among the proportion of each part of speech in the word-grams included in each bin as the part of speech of the word-gram processed by the second computing device corresponding to each bin; the first computing device updates the part-of-speech-device mapping table based on the identifier of the second computing device corresponding to each bin and the part of speech of the word-gram processed by each second computing device.

[0021] By updating the part-of-speech-device mapping table in the above manner, the first computing device can improve the accuracy of allocating word units in subsequent inference request statements to the second computing device that processes the corresponding part-of-speech when processing subsequent inference request statements, which is beneficial to improving the affinity between word units and experts on the second computing device, and is beneficial to reducing the number of times experts are activated remotely, thereby helping to reduce the communication overhead between the second computing devices.

[0022] In the second aspect, an embodiment of the present application provides another reasoning method, which is applied to a target computing device, the target computing device is applied to an reasoning system, the reasoning system includes a first computing device and K second computing devices, the reasoning system is applied to an reasoning model, the reasoning model includes L MOE layers, each MOE layer includes multiple experts, and the experts in each MOE layer are deployed on the K second computing devices, K is an integer greater than 1; the target computing device is one of the K second computing devices.

[0023] The target computing device receives a target word tuple from the first computing device, where the target word tuple is one of K word tuples, where the K word tuples are obtained by the first computing device grouping a plurality of word segments based on a part-of-speech-device mapping table, where the plurality of word segments are obtained by segmenting an inference request statement, and the K word tuples correspond to K second computing devices. In the part-of-speech-device mapping table, the part-of-speech corresponding to each of the K second computing devices includes the part-of-speech of the word tuple corresponding to each second computing device. The target computing device calls an expert deployed on the target computing device to process the target word tuple to obtain a first processing result. The target computing device obtains K-1 second processing results from K-1 second computing devices other than the target computing device among the K second computing devices, where the K-1 processing results are obtained by the K-1 second computing devices processing the received word tuple based on the deployed expert. The target computing device obtains an inference response statement for responding to the inference request statement based on the first processing result and the K-1 second processing results. The target computing device sends the inference response statement to the first computing device.

[0024] It can be seen that the word elements in the inference request statement are grouped based on the part-of-speech-device mapping table. For the target computing device, the word elements of the word elements processed by the experts on the target computing device are grouped together, and the group of word elements are assigned to the experts on the target computing device for processing. This is beneficial to improving the affinity between the word elements and the experts on the target computing device, so that when the target computing device calls the experts to process the word elements, the number of times the experts are activated remotely is reduced, which is beneficial to reducing the communication overhead between the target computing device and the other second computing devices.

[0025] In conjunction with the second aspect, in one possible implementation, the method of this embodiment further includes:

[0026] The target computing device receives an expert adjustment instruction from the first computing device; the target computing device adjusts the experts deployed on the target computing device based on the expert adjustment instruction, and the experts deployed on the adjusted target computing device include the T experts that are called the most times among the experts called when the target computing device processes the received word tuples, where T is the ratio of the number of expert types in a single MOE layer to K.

[0027] It can be seen that by deploying the T experts with the most calls on the target computing device, the target computing device is able to avoid calling experts on other second computing devices as much as possible when processing word units, which is beneficial to reducing the probability of remote activation, and further beneficial to reducing the communication overhead between the target computing device and other second computing devices.

[0028] In a third aspect, embodiments of the present application provide a first computing device. The first computing device is applied to an inference system, which is applied to an inference model. The inference model includes L MOE layers, each MOE layer includes multiple experts. The inference system also includes K second computing devices, each of which is equipped with an expert from each MOE layer, where L and K are both integers greater than 1. The first computing device includes an acquisition unit, a processing unit, and a transceiver unit.

[0029] An acquisition unit, configured to acquire a user's inference request statement and split the inference request statement to obtain multiple tokens;

[0030] a processing unit configured to divide the plurality of word-grams into K groups of word-grams based on a part-of-speech-device mapping table, and determine a correspondence between the K groups of word-grams and the K second computing devices; wherein the part-of-speech corresponding to each of the K second computing devices in the part-of-speech-device mapping table includes the part-of-speech of the word-grams in the word-gram group corresponding to each second computing device;

[0031] The transceiver unit is used to send K groups of word elements to K second computing devices respectively, so that each second computing device processes the received word elements based on the deployed experts to obtain an inference response statement for responding to the inference request statement; and receive the K inference response statements sent by the K second computing devices respectively.

[0032] In conjunction with the third aspect, in one possible implementation, the acquiring unit is further configured to acquire serial numbers of experts called by the K second computing devices to process received word-grams;

[0033] The processing unit is further used to determine a word-gram activation path table based on the serial numbers of the experts called by the K second computing devices to process the received word-grams, the word-gram activation path table including the activation path of each word-gram in the multiple word-grams, and the activation path of each word-gram is composed of the serial number of the expert called when each word-gram is processed; update the expert placement table based on the word-gram activation path table and the word-gram part-of-speech table, wherein the expert placement table includes the identifiers of the K second computing devices and the identifiers of the experts deployed on each of the K second computing devices; generate K expert adjustment instructions based on the updated expert placement table, and send the K expert adjustment instructions to the K second computing devices respectively, the K expert adjustment instructions are respectively used to instruct the K second computing devices to adjust the experts deployed on the K second computing devices, and the expert placement table includes the expert serial number and the identifier of the second computing device to which the expert is deployed.

[0034] In conjunction with the third aspect, in one possible implementation, in terms of updating the expert placement table based on the word-unit activation path table and the word-unit part-of-speech table, the processing unit is specifically configured to:

[0035] The word unit activation path table is converted into a first matrix; the first matrix is ​​processed based on one-hot encoding to obtain a second matrix; the second matrix is ​​clustered to obtain a clustered matrix, wherein each row of word units in the clustered matrix corresponds to a cluster label; the clustered matrix is ​​processed based on the cluster labels to obtain multiple sets; wherein the cluster labels of the word units included in each set are the same; the word units in the multiple sets are allocated to K buckets based on the sets; the difference in the number of word units in any two buckets of the K buckets is less than a preset threshold; the expert placement table is updated based on the K buckets, and the word unit-device table is updated based on the K buckets and the word unit part-of-speech table.

[0036] In conjunction with the third aspect, in one possible implementation, the K buckets correspond to the K second computing devices. In determining the expert placement table based on the K buckets, the processing unit is specifically configured to:

[0037] For each MoE layer, T+s slots are allocated to each of the K second computing devices; T is the ratio of the number of types of experts in a single MOE layer to K; s is a positive integer; determine the expert called when each word in each bucket of the K buckets is processed, and determine the number of times each expert in the called experts is called; allocate the T experts with the largest number of calls to the T slots of the second computing device corresponding to each bucket to obtain a first allocation result; when there are still experts with unallocated resources among the experts called to process the word in the first bucket, if they have not yet been allocated, If the number of experts to whom resources are allocated is not greater than s, the experts to whom resources have not been allocated will be allocated to the remaining slots of the second computing device corresponding to the first bucket; if the number of experts to whom resources have not been allocated is greater than s, s experts among the experts to whom resources have not been allocated will be allocated to the remaining s slots of the second computing device corresponding to the first bucket, and the experts other than s experts among the experts to whom resources have not been allocated will be allocated to the slots occupied by experts whose number of dedicated slots is greater than the first preset number, so as to obtain a second allocation result; the expert placement table is updated based on the first allocation result and the second allocation result.

[0038] In conjunction with the third aspect, in one possible implementation, in terms of updating the word-unit-device table based on the K buckets and the word-unit part-of-speech table, the processing unit is specifically configured to:

[0039] Based on the word-gram part-of-speech table, the proportion of word-grams of each part of speech in the word-grams included in each bin of the K bins corresponding to the K second computing devices is obtained; among the proportion of word-grams of each part of speech in the word-grams included in each bin, the part-of-speech with the largest proportion is determined as the part-of-speech of the word-gram processed by the second computing device corresponding to each bin; based on the identification of the second computing device corresponding to each bin and the part-of-speech of the word-gram processed by each second computing device, the part-of-speech-device mapping table is updated.

[0040] In a fourth aspect, embodiments of the present application provide a target computing device, which is applied to an inference system. The inference system includes a first computing device and K second computing devices. The inference system is applied to an inference model. The inference model includes L MOE layers, each MOE layer includes multiple experts, and the K second computing devices are each deployed with an expert from each MOE layer, where K is an integer greater than 1. The target computing device is one of the K second computing devices. The target computing device includes a transceiver unit, a processing unit, and an acquisition unit.

[0041] a transceiver unit configured to receive a target word tuple from a first computing device, the target word tuple being one of K word tuples, the K word tuples being obtained by the first computing device by grouping a plurality of word segments based on a part-of-speech-device mapping table, the plurality of word segments being obtained by segmenting an inference request statement, the K word tuples corresponding to K second computing devices, the part of speech corresponding to each of the K second computing devices in the part-of-speech-device mapping table including the part of speech of the word tuple corresponding to each second computing device;

[0042] a processing unit, configured to call an expert deployed on a target computing device to process the target word tuple to obtain a first processing result;

[0043] an acquiring unit, configured to acquire K-1 second processing results from K-1 second computing devices excluding the target computing device among the K second computing devices, where the K-1 processing results are obtained by the K-1 second computing devices processing the received word tuples based on the deployed experts;

[0044] The processing unit is further configured to obtain an inference response statement for responding to the inference request statement based on the first processing result and the K-1 second processing results;

[0045] The transceiver unit is further configured to send an inference response statement to the first computing device.

[0046] In conjunction with the fourth aspect, in one possible implementation, the transceiver unit is further configured to receive an expert adjustment instruction from the first computing device;

[0047] The processing unit is further used to adjust the experts deployed on the target computing device based on the expert adjustment instruction. The experts deployed on the adjusted target computing device include the T experts that are called the most times among the experts called when the target computing device processes the received word tuples, where T is the ratio of the number of expert types in a single MOE layer to K.

[0048] In a fifth aspect, an embodiment of the present application provides a first computing device comprising a processor and a memory. The memory is configured to store program code. The processor is configured to invoke the program code stored in the memory to execute the method provided in the first aspect or any possible implementation of the first aspect.

[0049] In a sixth aspect, an embodiment of the present application provides a target computing device comprising a processor and a memory. The memory is configured to store program code. The processor is configured to invoke the program code stored in the memory to execute the method provided in the second aspect or any possible implementation of the second aspect.

[0050] In the seventh aspect, an embodiment of the present application provides a computer storage medium, including computer instructions. When the computer instructions are run on an electronic device, the electronic device executes the method provided in any possible implementation of the first aspect, or the method provided in any possible implementation of the second aspect.

[0051] In an eighth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the method provided in any possible implementation of the first aspect, or the method provided in any possible implementation of the second aspect.

[0052] It can be understood that the beneficial effects of the embodiments described in the third to eighth aspects can refer to the beneficial effects of the methods described in the first to second aspects, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a schematic diagram of a Transformer model based on a decoder-only architecture;

[0054] FIG2 is a schematic diagram of an MOE model;

[0055] FIG3 is a schematic diagram of a system architecture provided in an embodiment of the present application;

[0056] FIG4 is a flow chart of an inference method provided in an embodiment of the present application;

[0057] Figure 5 is a schematic diagram of expert deployment and binary clustering;

[0058] FIG6 is a flow chart of another reasoning method provided in an embodiment of the present application;

[0059] FIG7 is a schematic diagram of the architecture of an inference model provided in an embodiment of the present application;

[0060] FIG8 is a schematic structural diagram of a first computing device provided in an embodiment of the present application;

[0061] FIG9 is a schematic diagram of the structure of a target computing device provided in an embodiment of the present application;

[0062] FIG10 is a schematic structural diagram of another first computing device provided in an embodiment of the present application;

[0063] FIG11 is a schematic diagram of the structure of another target computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The terms "first", "second", "third" and "fourth" in the specification, claims and drawings of this application are used to distinguish different objects rather than to describe a specific order.

[0065] "Multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating the existence of three relationships. For example, "A and / or B" means: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0066] The embodiments of the present application are described below with reference to the accompanying drawings.

[0067] Referring to Figure 3, which is a schematic diagram of a system architecture provided by an embodiment of the present application, the system architecture includes a first computing device 301 and K second computing devices 302, where K is an integer greater than 1.

[0068] The first computing device 301 and the second computing device 302 are both devices with computing or data processing functions. The first computing device 301 and the second computing device 302 are different computing devices, or the first computing device 301 and the second computing device 302 are different modules in the same server. For example, the first computing device 301 and the second computing device 302 are respectively a central processing unit (CPU) and a graphics processing unit (GPU) in the server.

[0069] The system architecture shown in Figure 3 is applied to an inference model based on a MOE architecture. The inference model includes L transformer blocks, each of which includes a self-attention layer and a MOE layer. Each MOE layer includes a gating unit and multiple experts. Each of the multiple second computing devices 302 is deployed with an expert from each MOE. In other words, multiple second computing devices 302 jointly deploy multiple experts from a single MOE layer. Of course, the inference model can also include other layers, which are not limited here.

[0070] When the first computing device 301 receives a user's inference request statement, it segments the inference request statement to obtain multiple word-grams. The first computing device 301 determines the part of speech of each word-gram in the multiple word-grams, groups the multiple word-grams based on the part of speech of each word-gram and the part-of-speech-device mapping table to obtain K groups of word-grams, and determines a correspondence between the K groups of word-grams and the K second computing devices 302. The part of speech corresponding to each second computing device 302 in the part-of-speech-device mapping table includes the part of speech of the word-grams in the word-gram group corresponding to the second computing device 302. Based on the correspondence between the K groups of word-grams and the K second computing devices 302, the first computing device 301 sends the K groups of word-grams to each of the K second computing devices 302. Each second computing device 302 processes the word-grams in the received word-gram group based on the experts deployed by it to obtain a processing result, and each second computing device 302 obtains the results obtained by the experts deployed by other second computing devices 302 processing the word-grams in the received word-gram group; each second computing device 302 obtains an inference response statement for the inference request statement based on the obtained processing result, and feeds back the inference response statement to the first computing device 301 to feed back the inference response statement to the user.

[0071] It can be seen that the first computing device 301 groups the word-grams in the inference request statement based on the part-of-speech-device mapping table, groups the word-grams of the part-of-speech processed by the expert deployed on a second computing device 302 into a group, and assigns the group of word-grams to the expert on the second computing device 302 for processing, which is beneficial to improving the affinity between the word-grams and the expert on the second computing device 302, so that when the second computing device 302 calls the expert to process the word-grams, the number of times the expert is activated remotely is reduced, which is beneficial to reducing the communication overhead between the second computing devices 302.

[0072] Referring to FIG4 , FIG4 is a flow chart of an inference method provided in an embodiment of the present application. The method is applied to the first computing device 301 in the system architecture shown in FIG1 . As shown in FIG4 , the method includes:

[0073] S401: A first computing device obtains a user's inference request statement and performs word segmentation on the inference request statement to obtain a plurality of word-grams.

[0074] It should be pointed out here that the method of the present application can be applied to human-computer dialogue scenarios, voice assistant scenarios, etc.

[0075] In an example, the inference request sentence is "What is the weather like in Shenzhen today?"

[0076] S402: The first computing device divides the multiple word segments into K word groups based on the part-of-speech-device mapping table, and determines the corresponding relationship between the K word groups and the K second computing devices.

[0077] The part of speech corresponding to each of the K second computing devices in the part-of-speech-device mapping table includes the part of speech of the word-tuple in the word-tuple group corresponding to each second computing device.

[0078] Specifically, the part-of-speech-device mapping table includes the second computing device identifier and the part-of-speech of the word-grams processed by the expert deployed on the second computing device; when the first computing device groups multiple word segments, the word-grams with the same part-of-speech as the word-grams processed by the expert on a second computing device are grouped into the same group based on the part-of-speech-device mapping table. In other words, the first computing device groups the word-grams that can be processed by the expert on the second computing device among the multiple word-grams into the same group based on the part-of-speech-device mapping table. K groups of word-grams may be obtained in the above manner. The first computing device determines the correspondence between the K groups of word-grams and the K second computing devices. The part-of-speech corresponding to each of the K second computing devices in the part-of-speech-device mapping table includes the part-of-speech of the word-grams in the word-gram group corresponding to each second computing device. In other words, the word-grams in the word-gram group corresponding to the second computing device belong to the word-grams that can be processed by the expert deployed on the second computing device.

[0079] S403: The first computing device sends K groups of word-grams to the K second computing devices respectively based on the correspondence between the K groups of word-grams and the K second computing devices.

[0080] It should be noted that when the first computing device sends K groups of word grammars to K second computing devices, the word gram group corresponding to each second computing device is sent to the second computing device.

[0081] S404: The first computing device receives K inference response statements fed back by the second computing devices in response to the inference request statements.

[0082] In one example, the inference response statement is "The weather in Shenzhen is sunny today." After the first computing device receives the inference response statement, it feeds back the inference response statement to the user, such as displaying the inference response statement on the display screen of the first computing device, or playing the inference response statement through the microphone of the first computing device.

[0083] In one possible implementation, the method of this embodiment further includes:

[0084] The first computing device obtains the serial numbers of the experts called by the K second computing devices to process the received word-grams; the first computing device determines a word-gram activation path table based on the serial numbers of the experts called by the K second computing devices to process the received word-grams, the word-gram activation path table including the activation path of each word-gram in a plurality of word-grams, the activation path of each word-gram being composed of the serial number of the expert called when each word-gram is processed; the first computing device updates the expert placement table based on the word-gram activation path table and the word-gram part-of-speech table, wherein the expert placement table includes the identifiers of the K second computing devices and the identifiers of the experts deployed on each of the K second computing devices; the first computing device generates K expert adjustment instructions based on the updated expert placement table, and sends the K expert adjustment instructions to the K second computing devices respectively, the K expert adjustment instructions being used to instruct the K second computing devices to adjust the experts deployed on the K second computing devices respectively, and the expert placement table includes the expert serial number and the identifier of the second computing device to which the expert is deployed.

[0085] Specifically, the first computing device periodically obtains from the second computing device the word-grams processed by the second computing device, the part of speech of the word-gram, and the serial number of the expert called when processing the word-gram. It should be noted that after processing a word-gram, the second computing device saves the word-gram, the part of speech of the word-gram, and the serial number of the expert called when processing the word-gram. The first computing device periodically obtains from the second computing device the word-grams processed by the second computing device, the part of speech of the word-gram, and the serial number of the expert called when processing the word-gram. The word-grams, the part of speech of the word-gram, and the serial number of the expert called when processing the word-gram that the first computing device caches from the second computing device at preset time intervals, or when the second computing device saves a preset number of word-grams, the part of speech of the word-gram, and the serial number of the expert called when processing the word-gram, the second computing device sends the saved preset number of word-grams, the part of speech of the word-gram, and the serial number of the expert called when processing the word-gram to the first computing device.

[0086] The first computing device updates a lemma activation path table and a lemma part-of-speech table based on the lemma processed by the second computing device, the part-of-speech of the lemma, and the sequence number of the expert invoked when processing the lemma. The lemma activation path table includes multiple lemma activation paths, each of which includes the sequence number of the expert invoked to process the lemma. The lemma part-of-speech table includes a correspondence between lemmas and parts of speech.

[0087] In one example, the word unit activation path can be expressed as: Among them, t i is the i-th word, e ij is the number of the expert called by the i-th word in the j-th layer, L is the number of MOE layers, and n is the number of words in the word activation path table. The word part-of-speech table can be expressed as Q={t i :pos i},posi is the part of speech of the i-th word.

[0088] The first computing device updates the expert placement table based on the word unit activation path table and the updated word unit part-of-speech table. Specifically, the first computing device converts the word unit activation path table P into a first matrix M∈R n×L , the i-th row M of the first matrix M i t i The first computing device processes the first matrix based on one-hot encoding to obtain the second matrix For example, the 0th word selects the second expert in the first layer, and there are 4 experts in total, then M 0,1 =2, The first computing device performs binary clustering on the second matrix to obtain a clustered matrix. The clustered matrix has a diagonal distribution. Each row of tokens in the clustered matrix corresponds to a cluster label.

[0089] In one example, the first computing device performs binary clustering on the second matrix based on the cluster.SpectralCoClustering algorithm in the sklearn package to obtain a clustered matrix, wherein the parameter n_clusters=n_expert in the cluster.SpectralCoClustering algorithm means that the number of cluster labels is the same as the number of experts, and other parameters are default values.

[0090] The first computing device classifies the word-units in the clustered matrix based on the cluster labels to obtain multiple sets. Word-units in the same set have the same cluster label. In one example, the set R can be expressed as:

[0091] In a specific example, as shown in the left figure of Figure 5, experts 0 and 1 are deployed on second computing device A, while experts 2 and 3 are deployed on second computing device B. The word-grams processed by second computing device A include word-gram T0, word-gram T1, word-gram T2, and word-gram T3; the word-grams processed by second computing device B include word-gram T4, word-gram T5, word-gram T6, and word-gram T7. Word-grams T0 and T1 are processed by experts 1 and 3, word-grams T2 and T3 are processed by experts 0 and 2, word-grams T4 and T5 are processed by experts 1 and 3, and word-grams T6 and T7 are processed by expert 0. It can be seen that the second computing device A needs to activate expert 3 remotely to process word-grams T0 and T1, the second computing device A needs to activate expert 2 remotely to process word-grams T2 and T3, the second computing device B needs to activate expert 1 remotely to process word-grams T4 and T5, and the second computing device B needs to activate expert 0 remotely to process word-grams T6 and T7. When the above clustering method is used for binary clustering, when placing experts based on the clustering results, the first computing device deploys experts 3 and 1 on the second computing device A, and experts 0 and 2 on the second computing device B, as shown in the right figure in Figure 5. This method reduces the probability of remotely activating experts when the second computing devices A and B process word-grams, thereby reducing the communication overhead between the second computing devices.

[0092] The first computing device allocates the word elements in the multiple sets into K buckets; the difference in the number of word elements in any two buckets of the K buckets is less than a preset threshold. In a specific example, the first computing device packs the word elements in the multiple sets into K buckets (bins) based on the best fitting algorithm so that the difference in the number of word elements allocated to different bins is minimized. The set consisting of K bins can be called Among them, K bins correspond to K second computing devices.

[0093] In one possible implementation, the K buckets correspond to the K second computing devices, and the first computing device determines an expert placement table based on the K buckets, including:

[0094] For each MOE layer, the first computing device allocates T+s slots to each of the K second computing devices; T is the ratio of the number of types of experts in a single MOE layer to K; s is a positive integer; the first computing device determines the expert called when each word in each bucket of the K buckets is processed, and determines the number of times each expert among the called experts is called; the T experts with the largest number of calls are allocated to the T slots of the second computing device corresponding to each bucket to obtain a first allocation result.

[0095] Specifically, for each second computing device, M slots are allocated to each second computing device, where M is equal to T+s, where T is the ratio of the total number of experts in the inference model to K, and s is a positive integer. In one example, s is equal to 2. It should be noted that the slots here refer to the hardware resources that support the experts in performing computing and processing. For each of the K bins, the first computing device obtains the experts called when all word elements in each bin are processed, and determines the number of times each of the called experts is called, and allocates the T experts with a larger number of calls to T slots of the M slots in the second computing device corresponding to the bin, so as to allocate resources to the T experts with a larger number of calls; the first allocation result is obtained according to the above allocation method.

[0096] When the first computing device processes the word-grams in the first bucket and there are still unallocated experts among the experts called by the first computing device, if the number of unallocated experts is not greater than s, the first computing device allocates the unallocated experts to the remaining slots of the second computing device corresponding to the first bucket; if the number of unallocated experts is greater than s, the first computing device allocates s of the unallocated experts to the remaining s slots of the second computing device corresponding to the first bucket, and allocates the experts other than the s unallocated experts to slots occupied by experts whose number of dedicated slots is greater than a first preset number, where the first preset number is an integer greater than 1.

[0097] Specifically, for the experts called when all word units in the first bin are processed, if there are still experts with unallocated resources, the first computing device will allocate the experts with unallocated resources to the remaining s slots among the M slots in the second computing device corresponding to the first bin, and the first bin is one of the K bins. After processing the experts called when the word units in the K bins are processed in the above manner, the M slots of each second computing device in the K second computing devices are allocated experts. For two slots belonging to different second computing devices, the same expert may be allocated, that is, the same expert may be deployed on multiple second computing devices, or the same expert may occupy multiple slots. The number of slots occupied by an expert can be called the number of copies of the expert. For example, if the target expert is deployed on two second computing devices, the number of copies of the target expert is 2. If there are still unallocated experts among the experts called when all word-grams in the first bin are processed, the first computing device will assign replicas of the experts with unallocated resources to slots occupied by experts with a first number of replicas. For example, if there are two experts with unallocated resources and the first number is 2, the first computing device will assign the two experts with unallocated resources to slots occupied by experts with a replica number of 2. After processing in this manner, if there are still unallocated experts among the experts called when all word-grams in the first bin are processed, the first computing device will assign the experts with unallocated resources to slots occupied by experts with a second number of replicas, where the first number is less than the second number. This process continues in this manner until resources are allocated to the experts called when all word-grams in K bins are processed, thereby obtaining a second allocation result.

[0098] The first computing device updates the expert placement table based on the first allocation result and the second allocation result, wherein the expert placement table includes the serial number of the expert and the identifier of the device allocated by the expert.

[0099] For example, the number of experts in the inference model is 8, and the second computing device has 2, namely the second computing device A and the second computing device B. The second computing device A has 6 slots, and the second computing device B has 6 slots. When processing the word in the bin corresponding to the second computing device A, the experts called include the first expert, the second expert, the third expert, the fourth expert, the fifth expert, the sixth expert and the seventh expert. Among them, the four experts called the most are the first expert, the third expert, the fourth expert and the seventh expert; the first computing device allocates the first expert, the third expert, the fourth expert and the seventh expert to 4 of the 6 slots in the second computing device A, and the second expert, the fifth expert and the sixth expert are left unallocated. Source; the first computing device assigns two of the second, fifth, and sixth experts, for example, the second and fifth experts, to the two remaining slots in the second computing device A, leaving the sixth expert unallocated. The experts called when processing the word in the bin corresponding to the second computing device B include the first, second, fourth, fifth, sixth, seventh, and eighth experts, among which the four experts called the most are the first, second, sixth, and eighth experts. The first computing device assigns the first, second, sixth, and eighth experts to four of the six slots in the second computing device B, leaving the fourth, fifth, and seventh experts unallocated. The first computing device assigns two of the fourth, fifth, and seventh experts, for example, the fourth and fifth experts, to four of the six slots in the second computing device B. At this point, among the experts called when processing the word-grams in the bin corresponding to second computing device A, the sixth expert still has unallocated resources, and among the experts called when processing the word-grams in the bin corresponding to second computing device B, the seventh expert still has unallocated resources. The first computing device allocates the sixth expert, which has not yet been allocated resources, among the experts called when processing the word-grams in the bin corresponding to second computing device A, to one of the two slots occupied by the first expert with a replica number of 2. The first computing device allocates the seventh expert, which has not yet been allocated resources, among the experts called when processing the word-grams in the bin corresponding to second computing device B, to one of the two slots occupied by the fourth expert with a replica number of 2. At this point, resources are allocated to both the experts called by second computing device A to process the received word-grams and the experts called by second computing device B to process the received word-grams.

[0100] In one possible implementation, the first computing device updates the word-unit-device table based on the K bins and the word-unit part-of-speech table, including:

[0101] The first computing device obtains the proportion of each part of speech in the word-grams included in each bin of the K bins corresponding to the K second computing devices based on the word-gram part-of-speech table; the first computing device determines the part of speech with the largest proportion among the proportion of each part of speech in the word-grams included in each bin as the proportion corresponding to the second computing device corresponding to each bin; the first computing device updates the part-of-speech-device mapping table based on the identification of each second computing device and the part of speech corresponding to each second computing device.

[0102] Specifically, for the first bin of the K bins, the first computing device groups all word-grams in the first bin according to their part-of-speech based on the word-gram part-of-speech table, so that word-grams belonging to the same group have the same part-of-speech; the first computing device obtains the proportion of word-grams of each part-of-speech in the first bin based on the grouping result, and determines the part-of-speech with the largest proportion as the part-of-speech of the word-gram processed by the second computing device corresponding to the first bin; the first computing device updates the part-of-speech-to-device mapping table based on the identifiers of the first and second computing devices corresponding to the first bin and the part-of-speech of the word-gram processed by the second computing device. In one example, the first computing device adds the identifier of the second computing device corresponding to the first bin and the part-of-speech of the word-gram processed by the second computing device to the part-of-speech-to-device mapping table.

[0103] In another example, the first computing device determines whether there is an identifier of the second computing device corresponding to the first bin in the part-of-speech-device mapping table; if so, the first computing device updates the part-of-speech corresponding to the second computing device corresponding to the first bin in the part-of-speech-device mapping table to the part-of-speech of the word processed by the second computing device corresponding to the first bin obtained in the above manner; if not, the first computing device adds the identifier of the second computing device corresponding to the first bin and the part-of-speech of the word processed by the second computing device to the part-of-speech-device mapping table.

[0104] By updating the part-of-speech-device mapping table in the above manner, the first computing device can improve the accuracy of allocating word units in subsequent inference request statements to the second computing device that processes the corresponding part-of-speech when processing subsequent inference request statements, which is beneficial to improving the affinity between word units and experts on the second computing device, and is beneficial to reducing the number of times experts are activated remotely, thereby helping to reduce the communication overhead between the second computing devices.

[0105] As can be seen, in the embodiment, by grouping the word-grams in the inference request statement according to part of speech and part-of-speech-device mapping table to obtain multiple groups of word-grams, word-grams that can be processed by an expert deployed on a second computing device are grouped together. Furthermore, when sending word-gram groups to the second computing device, the word-gram groups are sent to the second computing device where the deployed expert can process them. This helps improve the affinity between the word-grams and the experts deployed on the second computing device, reducing the number of expert reactivations when the second computing device calls an expert to process the word-grams, thereby reducing communication overhead between the second computing devices. When updating the expert placement table, experts that have been called more frequently when processing word-grams are preferentially assigned to the same second computing device, which also helps reduce the number of expert reactivations and communication overhead between the second computing devices. Furthermore, for experts that have not yet been allocated resources, they are assigned to slots occupied by experts with a replica number greater than 1, and experts that have not yet been allocated resources are preferentially assigned to slots occupied by experts with a smaller replica number, thereby facilitating load balancing between the second computing devices.

[0106] See FIG6 , which is a flowchart of an inference method provided in an embodiment of the present application. The method is applied to a target computing device, which is one of the K second computing devices 302 in the system architecture shown in FIG1 . The method includes:

[0107] S601: A target computing device receives a target word tuple from a first computing device.

[0108] Among them, the target word tuple is one group of multiple groups of word grammars, the multiple groups of word grammars are obtained by grouping multiple word segmentations by the first computing device based on the part-of-speech-device mapping table, the multiple word segmentations are obtained by segmenting the inference request statement, K groups of word grammars correspond to K second computing devices, and the part of speech corresponding to each second computing device in the K second computing devices in the part-of-speech-device mapping table includes the part of speech of the word gram in the word tuple corresponding to each second computing device.

[0109] S602: The target computing device calls an expert deployed on the target computing device to process the target word tuple to obtain a first processing result.

[0110] It should be noted that the inference model includes L transformer blocks, each of which includes a self-attention layer and a MOE layer. Each MOE layer includes a gating unit and multiple experts. Each of the multiple second computing devices 302 is deployed with an expert from each MOE. In other words, multiple second computing devices 302 jointly deploy multiple experts from a single MOE layer. A word is input from its self-attention layer, processed by an expert selected by the gating unit, and the result is output, which is the first output result described above. If the expert selected by the gating unit is not an expert deployed on the target computing device, the target computing device forwards the received word to the second computing device deployed with the expert selected by the gating unit.

[0111] S603: The target computing device obtains K-1 second processing results from K-1 second computing devices excluding the target computing device among the K second computing devices.

[0112] Among them, the K-1 processing results are obtained by the K-1 second computing devices processing the word tuples received by the K-1 second computing devices based on the experts in the MOE layers respectively deployed on the K-1 second computing devices.

[0113] S604: The target computing device obtains an inference response statement for responding to the inference request statement based on the first processing result and the K-1 second processing results; and sends the inference response statement to the first computing device.

[0114] In one possible implementation, the method of this embodiment further includes:

[0115] The target computing device receives an expert adjustment instruction from the first computing device; the target computing device adjusts the experts deployed on the target computing device based on the expert adjustment instruction, and the experts deployed on the target computing device after the adjustment include the T experts that are called the most times among the experts called when the target computing device processes the received word tuples, where T is the ratio of the number of expert types in a single MOE layer to K.

[0116] As can be seen, in the embodiment, the part of speech of the word-gram received by the target computing device is the same as the part of speech of the word-gram that the expert on the target computing device can process. This helps improve the affinity between the word-gram and the expert deployed on the second computing device, reduces the number of remote expert activations, and thus helps reduce communication overhead between the second computing devices. By deploying the T experts with the highest number of calls on the target computing device, the target computing device is able to minimize calling experts on other second computing devices when processing word-grams, which helps reduce the probability of remote activations and thus helps reduce communication overhead between the second computing devices.

[0117] It should be pointed out here that the solution of this application is based on expert parallelism, that is, the experts deployed on different devices in the inference model can be executed in parallel. In one example, this application can also combine data parallelism (DP) and tensor parallelism (TP) to give an expert placement strategy, that is, an expert placement table, under different hardware configurations and model scales. In another example, combined with the parallel methods of other layers (such as the attention layer), the optimal parallel strategy for the Transformer architecture model as a whole and the scheduling scheme for experts / words are given.

[0118] Refer to Figure 7, which is a schematic diagram of the architecture of an inference model provided in an embodiment of the present application. As shown in Figure 7, the inference model includes a batching module, L MOE layers, a hotness-aware expert asymmetric parallelism (HEAP) decision maker and a speculative token-expert route orchestration (STEREO) decision maker. Of course, the inference model can also include other layers, which are not limited here. Among them, each MOE layer includes multiple experts, and the inference system also includes K second computing devices, and the experts in each MOE layer are deployed on the K second computing devices, and the batching module, HEAP decision maker and STEREO decision maker are deployed on the first computing device.

[0119] The first computing module receives the user's reasoning request statement and performs word segmentation on the reasoning request statement to obtain n word segments. At this time, the first computing module obtains a word sequence consisting of n word units. The word sequence can be expressed as T = [t0, t1, ..., t n-1 ], t i ∈R d The first computing device calls the batching module to divide the word sequence into K groups of words based on the word-word part of the word sequence and the word-word-device mapping table, namely T0, T1, ..., T K-1 , and determine the correspondence between K groups of word elements and K second computing devices; wherein, in the part-of-speech-device mapping table, the part-of-speech corresponding to each second computing device in the K second computing devices includes the part-of-speech of the word element in the word element group corresponding to each second computing device.

[0120] The first computing module converts the K groups of word elements T0, T1, ..., T into K groups of word elements based on the correspondence between the K groups of word elements and the K second computing devices. K-1 The data are transmitted to K second computing devices respectively, and the K second computing devices process K groups of word-grams respectively. In other words, the K second computing devices process word-grams of different parts of speech.

[0121] The MOE layer deployed on the second computing device includes a gating unit and multiple experts. The gating module can be called a word unit router and a word unit dispatcher; the second computing device includes a local expert cache, which is divided into multiple parts. Each part of the cache can be called a slot of the second computing device. Multiple experts of an MOE layer are deployed on K slots of the second computing device. The MOE layer can be called a distributed expert-as-a-service (EaaS) module. The second computing device also has deployed therein parts of the inference model other than the MOE layer, batching module, HEAP decision maker, and STEREO decision maker. From a holistic perspective, the second computing device can be regarded as an inference engine. The second computing device processes the received word units. Specifically, the second computing device calls the gating unit of the deployed MOE layer to select an expert. After determining the expert, the word units are transmitted to the selected expert for processing. After the expert processes the word units, the processing result is output. Each of the K second computing devices obtains K-1 processing results output by experts on K-1 second computing devices other than itself among the K second computing devices; each second computing device obtains an inference response statement for responding to the inference request statement based on the processing results output by the experts and the K-1 processing results, and feeds back the inference response statement to the first computing device.

[0122] The above process is the online reasoning process. The following describes the offline update process:

[0123] The first computing device periodically obtains K word-grams processed by the second computing device, the part of speech of the word-gram, and the serial number of the expert called to process the word-gram; and obtains a word-gram activation path table and a word-gram part-of-speech table based on the periodically obtained information; the first computing device calls a STEREO decider to convert the word-gram activation path table into a first matrix; processes the first matrix based on one-hot encoding to obtain a second matrix; the first computing device clusters the second matrix to obtain a clustered matrix; processes the clustered matrix based on cluster labels to obtain multiple sets; wherein, the cluster labels of the word-grams included in each set are the same; the word-grams in the multiple sets are distributed into K buckets based on the set; the difference in the number of word-grams in any two buckets of the K buckets is less than a preset threshold.

[0124] The first computing device invokes the HEAP decider to update the expert placement table based on the K buckets and to update the lemma-device table based on the K buckets and the lemma part-of-speech table. The specific process is described in S404 and will not be repeated here. The expert placement table and lemma-device table are collectively referred to as metadata.

[0125] After the first computing device obtains the updated expert placement table, it generates K expert adjustment instructions based on the expert placement table and sends the K expert adjustment instructions to the K second computing devices respectively. The K expert adjustment instructions are used to instruct the K second computing devices to adjust the experts deployed on the K second computing devices respectively.

[0126] In the solution of the present application, the calculation of the MOE layer is decoupled from other layers of the reasoning model. This decoupling is achieved through the EaaS module. When calculation of the MOE layer is required, the EaaS module can be called.

[0127] Referring to FIG8 , FIG8 is a schematic diagram of the structure of a first computing device provided in an embodiment of the present application. The first computing device 800 is the first computing device 301 in FIG3 . The first computing device 800 is applied to an inference system, which is applied to an inference model. The inference model includes L MOE layers, each MOE layer includes multiple experts, and the inference system also includes K second computing devices, each of which is deployed with an expert in each MOE layer. The first computing device 800 includes:

[0128] An acquisition unit 801 is configured to acquire a user's inference request statement and split the inference request statement to obtain multiple tokens.

[0129] Processing unit 802 is configured to divide the plurality of word-grams into K groups of word-grams based on a part-of-speech-device mapping table, and determine a correspondence between the K groups of word-grams and the K second computing devices; the part-of-speech corresponding to each of the K second computing devices in the part-of-speech-device mapping table includes the part-of-speech of the word-grams in the word-gram group corresponding to each second computing device;

[0130] The transceiver unit 803 is used to send K groups of word elements to K second computing devices respectively, so that each second computing device processes the received word elements based on the deployed experts to obtain an inference response statement for responding to the inference request statement; and receive the K inference response statements sent by the K second computing devices respectively.

[0131] In a possible implementation, the acquiring unit 801 is further configured to acquire serial numbers of experts called by the K second computing devices to process the received word-grams;

[0132] The processing unit 802 is further used to determine a word-gram activation path table based on the serial numbers of the experts called by the K second computing devices to process the received word-grams, the word-gram activation path table including the activation path of each word-gram in the multiple word-grams, and the activation path of each word-gram is composed of the serial number of the expert called when each word-gram is processed; update the expert placement table based on the word-gram activation path table and the word-gram part-of-speech table, wherein the expert placement table includes the identifiers of the K second computing devices and the identifiers of the experts deployed on each of the K second computing devices; generate K expert adjustment instructions based on the updated expert placement table, and send the K expert adjustment instructions to the K second computing devices respectively, the K expert adjustment instructions are respectively used to instruct the K second computing devices to adjust the experts deployed on the K second computing devices, and the expert placement table includes the expert serial number and the identifier of the second computing device to which the expert is deployed.

[0133] In one possible implementation, in terms of updating the expert placement table based on the word-unit activation path table and the word-unit part-of-speech table, the processing unit 802 is specifically configured to:

[0134] The word unit activation path table is converted into a first matrix; the first matrix is ​​processed based on one-hot encoding to obtain a second matrix; the second matrix is ​​clustered to obtain a clustered matrix, wherein each row of word units in the clustered matrix corresponds to a cluster label; the clustered matrix is ​​processed based on the cluster labels to obtain multiple sets; wherein the cluster labels of the word units included in each set are the same; the word units in the multiple sets are allocated to K buckets based on the sets; the difference in the number of word units in any two buckets of the K buckets is less than a preset threshold; the expert placement table is updated based on the K buckets, and the word unit-device table is updated based on the K buckets and the word unit part-of-speech table.

[0135] In one possible implementation, the K buckets correspond to the K second computing devices. In determining the expert placement table based on the K buckets, the processing unit 802 is specifically configured to:

[0136] For each MoE layer, T+s slots are allocated to each of the K second computing devices; T is the ratio of the number of types of experts in a single MOE layer to K; s is a positive integer; determine the expert called when each word in each bucket of the K buckets is processed, and determine the number of times each expert in the called experts is called; allocate the T experts with the largest number of calls to the T slots of the second computing device corresponding to each bucket to obtain a first allocation result; when there are still experts with unallocated resources among the experts called to process the word in the first bucket, if they have not yet been allocated, If the number of experts to whom resources are allocated is not greater than s, the experts to whom resources have not been allocated will be allocated to the remaining slots of the second computing device corresponding to the first bucket; if the number of experts to whom resources have not been allocated is greater than s, s experts among the experts to whom resources have not been allocated will be allocated to the remaining s slots of the second computing device corresponding to the first bucket, and the experts other than s experts among the experts to whom resources have not been allocated will be allocated to the slots occupied by experts whose number of dedicated slots is greater than the first preset number, so as to obtain a second allocation result; the expert placement table is updated based on the first allocation result and the second allocation result.

[0137] In one possible implementation, in terms of updating the word-unit-device table based on the K buckets and the word-unit part-of-speech table, the processing unit 802 is specifically configured to:

[0138] Based on the word-gram part-of-speech table, the proportion of word-grams of each part of speech in the word-grams included in each bin of the K bins corresponding to the K second computing devices is obtained; among the proportion of word-grams of each part of speech in the word-grams included in each bin, the part-of-speech with the largest proportion is determined as the part-of-speech of the word-gram processed by the second computing device corresponding to each bin; based on the identification of the second computing device corresponding to each bin and the part-of-speech of the word-gram processed by each second computing device, the part-of-speech-device mapping table is updated.

[0139] It is worth noting that, for the specific functional implementation of the first computing device 800, please refer to the specific description of the embodiment shown in Figure 4 above, such as the acquisition unit 801 is used to execute the relevant content of S401, and the processing unit 802 is used to execute the relevant content of S402. The transceiver unit 803 is used to execute the relevant content of S403 and S404. The various units or modules in the first computing device 800 can be individually or completely merged into one or several other units or modules to form a structure, or one (or some) of the units or modules can be further divided into multiple functionally smaller units or modules to form a structure, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above-mentioned units or modules are divided based on logical functions. In actual applications, the functions of one unit (or module) are implemented by multiple units (or modules), or the functions of multiple units (or modules) are implemented by one unit (or module).

[0140] Referring to Figure 9 , Figure 9 is a schematic diagram of the structure of a target computing device provided in an embodiment of the present application. Target computing device 900 is one of the K second computing devices 302 in Figure 3 . Target computing device 900 is used in an inference system, which is used in an inference model. The inference model includes L MOE layers, each of which includes multiple experts. Experts from each MOE layer are deployed on each of the K second computing devices. The target computing device includes a transceiver unit, a processing unit, and an acquisition unit.

[0141] A transceiver unit 901 is configured to receive a target word tuple from a first computing device, where the target word tuple is one of K word tuples, where the K word tuples are obtained by the first computing device by grouping a plurality of word segments based on a part-of-speech-device mapping table, where the plurality of word segments are obtained by segmenting an inference request statement, where the K word tuples correspond to K second computing devices, and where the part-of-speech corresponding to each of the K second computing devices in the part-of-speech-device mapping table includes the part-of-speech of the word tuple corresponding to each second computing device.

[0142] The processing unit 902 is configured to call an expert deployed on the target computing device to process the target word tuple to obtain a first processing result;

[0143] An acquiring unit 903 is configured to acquire K-1 second processing results from K-1 second computing devices excluding the target computing device among the K second computing devices, where the K-1 processing results are obtained by the K-1 second computing devices processing the received word tuples based on the deployed experts.

[0144] The processing unit 902 is further configured to obtain an inference response statement for responding to the inference request statement based on the first processing result and the K-1 second processing results;

[0145] The transceiver unit 901 is further configured to send an inference response statement to the first computing device.

[0146] In one possible implementation, the transceiver unit 901 is further configured to receive an expert adjustment instruction from the first computing device;

[0147] The processing unit 902 is further used to adjust the experts deployed on the target computing device based on the expert adjustment instruction. The experts deployed on the adjusted target computing device include the T experts that are called the most times among the experts called when the target computing device processes the received word tuples, where T is the ratio of the number of expert types in a single MOE layer to K.

[0148] It is worth noting that, for the specific functional implementation of the target computing device 900, please refer to the specific description of the embodiment shown in Figure 6 above. For example, the transceiver unit 901 is used to execute the relevant content of S601 and S604, and the processing unit 902 is used to execute the relevant content of S602 and S604. The acquisition unit 903 is used to execute the relevant content of S603. The various units or modules in the target computing device 900 can be individually or completely merged into one or several other units or modules to form a structure, or some (some) of the units or modules can be further divided into multiple functionally smaller units or modules to form a structure, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above-mentioned units or modules are divided based on logical functions. In actual applications, the functions of one unit (or module) are implemented by multiple units (or modules), or the functions of multiple units (or modules) are implemented by one unit (or module).

[0149] Based on the description of the above method embodiments and related device embodiments, please refer to Figure 10 , which provides a schematic structural diagram of a first computing device 1000 according to an embodiment of the present invention. The first computing device 1000 shown in Figure 10 includes a memory 1001, a processor 1002, a communication interface 1003, and a bus 1004. The memory 1001, processor 1002, and communication interface 1003 are interconnected via bus 1004.

[0150] Optionally, the memory 1001 is a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM).

[0151] The memory 1001 can store programs. When the program stored in the memory 1001 is executed by the processor 1002, the processor 1002 and the communication interface 1003 are used to execute the various steps of the reasoning method of the embodiment shown in FIG. 4 .

[0152] The processor 1002 uses a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), a GPU or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the first computing device 800 of the embodiment of the present application, or to execute the reasoning method of the embodiment shown in Figure 4 of the present application.

[0153] Processor 1002 can also be an integrated circuit chip with signal processing capabilities. During implementation, each step of the inference method shown in FIG. 4 of the present application can be completed by hardware integrated logic circuits or software instructions in processor 1002. Optionally, processor 1002 is a general-purpose processor, a digital signal processor (DSP), an ASIC, a field programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. Processor 1002 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor is a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of this application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Optionally, the software module is located in random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 1001, and the processor 1002 reads the information in the memory 1001 and combines its hardware to complete the functions required to be performed by the units included in the first computing device 800 of the embodiment of the present application, or executes the reasoning method of the embodiment shown in Figure 4.

[0154] The communication interface 1003 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the first computing device 1000 and other devices (such as the second computing device shown in FIG. 1 ) or a communication network.

[0155] The bus 1004 may include a path for transmitting information between various components of the first computing device 1000 (eg, the memory 1001 , the processor 1002 , and the communication interface 1003 ).

[0156] It should be noted that although the first computing device 1000 shown in FIG10 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art will understand that the first computing device 1000 also includes other components necessary for normal operation. At the same time, according to specific needs, those skilled in the art will understand that the first computing device 1000 may also include hardware components that implement other additional functions. In addition, those skilled in the art will understand that the first computing device 1000 may also include only the components necessary to implement the embodiments of the present application, and does not necessarily include all the components shown in FIG10.

[0157] Based on the description of the above method embodiments and related device embodiments, please refer to Figure 11 , which shows a schematic diagram of the structure of a target computing device 1100 provided in an embodiment of the present invention. The target computing device 1100 shown in Figure 11 includes a memory 1101, a processor 1102, a communication interface 1103, and a bus 1104. The memory 1101, processor 1102, and communication interface 1103 are interconnected via bus 1104.

[0158] Optionally, the memory 1101 is a ROM, a static storage device, a dynamic storage device or a RAM.

[0159] The memory 1101 can store programs. When the program stored in the memory 1101 is executed by the processor 1102, the processor 1102 and the communication interface 1103 are used to execute the various steps of the reasoning method of the embodiment shown in FIG6 .

[0160] The processor 1002 uses a general-purpose CPU, microprocessor, ASIC, GPU or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the target computing device 900 of the embodiment of the present application, or to execute the reasoning method of the embodiment shown in Figure 6 of the present application.

[0161] Processor 1102 can also be an integrated circuit chip with signal processing capabilities. During implementation, the various steps of the inference method shown in FIG. 4 of the present application can be completed by hardware integrated logic circuits or software instructions in processor 1102. Optionally, processor 1102 is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. Processor 1102 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module is located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 1101, and the processor 1102 reads the information in the memory 1101 and combines its hardware to complete the functions required to be performed by the units included in the target computing device 900 of the embodiment of the present application, or executes the reasoning method of the embodiment shown in Figure 6.

[0162] The communication interface 1103 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the target computing device 1100 and other devices (such as the first computing device shown in FIG. 1 ) or a communication network.

[0163] The bus 1104 may include a path for transmitting information between various components of the target computing device 1100 (eg, the memory 1101 , the processor 1102 , and the communication interface 1103 ).

[0164] It should be noted that although the target computing device 1100 shown in FIG11 only shows a memory, a processor, and a communication interface, during the specific implementation process, those skilled in the art will understand that the target computing device 1100 also includes other components necessary for normal operation. At the same time, according to specific needs, those skilled in the art will understand that the target computing device 1100 may also include hardware components that implement other additional functions. In addition, those skilled in the art will understand that the target computing device 1100 may also include only the components necessary to implement the embodiments of the present application, and does not necessarily include all the components shown in FIG11.

[0165] An embodiment of the present application further provides a chip, which includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface to implement the reasoning method of the embodiment of the present application.

[0166] Optionally, as an implementation, the chip further includes a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the reasoning method.

[0167] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computer or a processor, the computer or processor executes one or more steps in any of the above methods.

[0168] The present application also provides a computer program product comprising instructions, which, when executed on a computer or processor, causes the computer or processor to execute one or more steps in any of the above methods.

[0169] Those skilled in the art will appreciate that the functions described in conjunction with the various illustrative logic blocks, modules, and algorithm steps disclosed herein can be implemented by hardware, software, firmware, or any combination thereof. If implemented in software, the functions described by the various illustrative logic blocks, modules, and steps may be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media, such as data storage media, or communication media including any media that facilitates the transfer of computer programs from one place to another (e.g., based on a communication protocol). In this manner, computer-readable media may generally correspond to (1) non-transitory tangible computer-readable storage media, or (2) communication media, such as signals or carrier waves. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in this application. A computer program product may include computer-readable media.

[0170] By way of example, and not limitation, such computer-readable storage media include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of media. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are actually directed to non-transitory tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), and Blu-ray disc. Disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above are also intended to be included within the scope of computer-readable media.

[0171] Instructions may be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Thus, the term "processor," as used herein, may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described by the various illustrative logical blocks, modules, and steps described herein is provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Furthermore, the techniques may be fully implemented in one or more circuits or logic elements.

[0172] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the division of the unit is only a logical function division, and there are other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Optionally, the mutual coupling, direct coupling, or communication connection shown or discussed is an indirect coupling or communication connection through some interfaces, devices or units, such as electrical, mechanical or other forms.

[0173] Optionally, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., located in one place, or distributed across multiple network units. Some or all of the units may be selected based on actual needs to achieve the purpose of the solution of this embodiment.

[0174] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are generated in whole or in part.

[0175] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A reasoning method, characterized in that: The method is applied to a first computing device, the first computing device is applied to an inference system, the inference system is applied to an inference model, the inference model includes L hybrid expert MOE layers, each of the MOE layers includes multiple MOEs, the inference system also includes K second computing devices, each of the K second computing devices is deployed with an expert in each MOE layer, and L and K are both integers greater than 1; the method includes: Obtaining a user's reasoning request sentence, and splitting the reasoning request sentence to obtain multiple word units; Dividing the plurality of word-grams into K groups of word-grams based on a part-of-speech-device mapping table, and determining a correspondence between the K groups of word-grams and the K second computing devices, wherein the part-of-speech corresponding to each of the K second computing devices in the part-of-speech-device mapping table includes the part-of-speech of the word-grams in the word-gram group corresponding to each second computing device; sending the K groups of word-grams to the K second computing devices respectively based on the correspondence between the K groups of word-grams and the K second computing devices, so that each of the second computing devices processes the received word-grams based on the deployed experts to obtain an inference response statement for responding to the inference request statement; Receive the K inference response statements respectively sent by the K second computing devices.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining serial numbers of experts called by the K second computing devices to process received word-grams; determining a word-gram activation path table based on the sequence numbers of the experts called when the K second computing devices process the received word-grams, wherein the word-gram activation path table includes an activation path for each word-gram in the plurality of word-grams, and the activation path of each word-gram is composed of the sequence number of the expert called when the each word-gram is processed; updating an expert placement table and the part-of-speech-device mapping table based on the word-gram activation path table and the word-gram part-of-speech table, wherein the expert placement table includes identifiers of the K second computing devices and identifiers of experts deployed on each of the K second computing devices; Based on the updated expert placement table, K expert adjustment instructions are generated and sent to the K second computing devices respectively. The K expert adjustment instructions are used to instruct the K second computing devices to adjust the experts deployed on the K second computing devices respectively. The expert placement table includes the expert serial number and the identifier of the second computing device to which the expert is deployed.

3. The method according to claim 2, characterized in that The updating of the expert placement table based on the word unit activation path table and the word unit part-of-speech table includes: Converting the word unit activation path table into a first matrix; processing the first matrix based on a one-hot encoding method to obtain a second matrix; Clustering the second matrix to obtain a clustered matrix, wherein each row of tokens in the clustered matrix corresponds to a cluster label; Processing the clustered matrix based on the cluster labels to obtain a plurality of sets; wherein each set includes word units having the same cluster label; Taking the sets as units, allocating the word units in the multiple sets into K buckets; the difference in the number of word units in any two buckets of the K buckets is less than a preset threshold; The expert placement table is updated based on the K buckets, and the word-gram-device table is updated based on the K buckets and the word-gram part-of-speech table.

4. The method according to claim 3, characterized in that The K buckets correspond to the K second computing devices, and determining the expert placement table based on the K buckets includes: For each MOE layer, T+s slots are allocated to each of the K second computing devices; T is the ratio of the number of types of experts in the MOE layer to K; s is a positive integer; Determining an expert to be called when each word in each bucket of the K buckets is processed, and determining the number of times each of the called experts has been called; assigning T experts with the largest number of calls to T slots of the second computing device corresponding to each bucket to obtain a first assignment result; When there are still experts with unallocated resources among the experts called for processing the word-grams in the first bucket, if the number of the experts with unallocated resources is not greater than s, the experts with unallocated resources are allocated to the remaining slots of the second computing device corresponding to the first bucket; if the number of the experts with unallocated resources is greater than s, s experts among the experts with unallocated resources are allocated to the remaining s slots of the second computing device corresponding to the first bucket, and the experts other than the s experts among the experts with unallocated resources are allocated to slots occupied by experts whose number of dedicated slots is greater than a first preset number, so as to obtain a second allocation result; wherein the first bucket is one of the multiple buckets; The expert placement table is updated based on the first allocation result and the second allocation result.

5. The method according to claim 3 or 4, characterized in that The updating of the word-element-device table based on the K buckets and the word-element part-of-speech table includes: Obtaining, based on the word-gram part-of-speech table, a word-gram proportion of each part of speech in the word-grams included in each bin of the K bins corresponding to the K second computing devices; Determine the part of speech with the largest proportion among the word-gram proportions of each part of speech in the word-grams included in each bin as the part of speech of the word-gram processed by the second computing device corresponding to each bin; The part-of-speech-device mapping table is updated based on the identifier of the second computing device corresponding to each bin and the part-of-speech of the word-gram processed by each second computing device.

6. A reasoning method, characterized in that The method is applied to a target computing device, the target computing device is applied to an inference system, the inference system includes a first computing device and K second computing devices, the inference system is applied to an inference model, the inference model includes L hybrid expert MOE layers, each MOE layer includes multiple MOEs, and experts in each MOE layer are deployed on the K second computing devices, where L and K are integers greater than 1; The target computing device is one of the K second computing devices, and the method includes: Receive a target word tuple from the first computing device, the target word tuple being one of K groups of word tuples, the K groups of word tuples being obtained by the first computing device by grouping a plurality of word segments based on a part-of-speech-device mapping table, the plurality of word segments being obtained by segmenting an inference request statement, the K groups of word tuples corresponding to the K second computing devices, and the part of speech corresponding to each of the K second computing devices in the part-of-speech-device mapping table including the part of speech of the word tuple corresponding to each second computing device; Invoking an expert deployed on a target computing device to process the target word tuple to obtain a first processing result; Obtaining K-1 second processing results from K-1 second computing devices other than the target computing device among the K second computing devices, wherein the K-1 processing results are obtained by the K-1 second computing devices processing the received word tuples based on the deployed experts; Obtaining an inference response statement for responding to an inference request statement based on the first processing result and the K-1 second processing results; The inference response statement is sent to the first computing device.

7. The method according to claim 6, characterized in that The method further comprises: receiving an expert adjustment instruction from the first computing device; adjusting the experts deployed on the target computing device based on the expert adjustment instruction, where after the adjustment, the experts deployed on the target computing device include T experts that are called the most times when the target computing device processes the received word tuples; Here, T is the ratio of the number of types of experts in the MOE layer to K.

8. A first computing device, characterized in that: The first computing device is applied to an inference system, and the inference system is applied to an inference model. The inference model includes L hybrid expert MOE layers, each of which includes multiple MOEs. The inference system also includes K second computing devices, and experts in each MOE layer are deployed on the K second computing devices. Both L and K are integers greater than 1. The first computing device includes: an acquisition unit, configured to acquire a user's inference request statement and split the inference request statement to obtain a plurality of word units; a processing unit configured to divide the plurality of word-grams into K groups of word-grams based on a part-of-speech-device mapping table, and determine a correspondence between the K groups of word-grams and the K second computing devices, wherein the part-of-speech corresponding to each of the K second computing devices in the part-of-speech-device mapping table includes the part-of-speech of the word-grams in the word-gram group corresponding to each second computing device; A transceiver unit is configured to send the K groups of word elements to K second computing devices respectively based on the correspondence between the K groups of word elements and the K second computing devices, so that each second computing device processes the received word elements based on the deployed experts to obtain an inference response statement for responding to the inference request statement; and receive the K inference response statements respectively sent by the K second computing devices.

9. The first computing device according to claim 8, wherein: The acquisition unit is further configured to acquire serial numbers of experts called by the K second computing devices to process the received word-grams; The processing unit is further used to determine a word-gram activation path table based on the serial numbers of the experts called when the K second computing devices process the received word-grams, the word-gram activation path table including the activation path of each word-gram in the multiple word-grams, and the activation path of each word-gram is composed of the serial number of the expert called when each word-gram is processed; update the expert placement table and the part-of-speech-device mapping table based on the word-gram activation path table and the word-gram part-of-speech table, wherein the expert placement table includes the identifiers of the K second computing devices and the identifiers of the experts deployed on each of the K second computing devices; generate K expert adjustment instructions based on the updated expert placement table, and send the K expert adjustment instructions to the K second computing devices respectively, the K expert adjustment instructions being used to instruct the K second computing devices to adjust the experts deployed on the K second computing devices respectively, and the expert placement table includes the expert serial number and the identifier of the second computing device to which the expert is deployed.

10. The first computing device according to claim 9, wherein: In the aspect of updating the expert placement table based on the word-unit activation path table and the word-unit part-of-speech table, the processing unit is specifically configured to: Converting the word unit activation path table into a first matrix; processing the first matrix based on a one-hot encoding method to obtain a second matrix; Clustering the second matrix to obtain a clustered matrix, wherein each row of tokens in the clustered matrix corresponds to a cluster label; Processing the clustered matrix based on the cluster labels to obtain a plurality of sets; wherein each set includes word units having the same cluster label; Taking the sets as units, allocating the word units in the multiple sets into K buckets; the difference in the number of word units in any two buckets of the K buckets is less than a preset threshold; The expert placement table is updated based on the K buckets, and the word-gram-device table is updated based on the K buckets and the word-gram part-of-speech table.

11. The first computing device according to claim 10, wherein: The K buckets correspond to the K second computing devices, and in terms of updating the expert placement table based on the K buckets, the processing unit is specifically configured to: For each MOE layer, T+s slots are allocated to each of the K second computing devices; T is the ratio of the number of types of experts in the MOE layer to K; s is a positive integer; Determine an expert to be called when each word in each bucket of the K buckets is processed, and determine the number of times each of the called experts is called; Allocating the T experts with the largest number of calls to the T slots of the second computing device corresponding to each bucket to obtain a first allocation result; When there are still experts with unallocated resources among the experts called for processing the word-grams in the first bucket, if the number of the experts with unallocated resources is not greater than s, the experts with unallocated resources are allocated to the remaining slots of the second computing device corresponding to the first bucket; if the number of the experts with unallocated resources is greater than s, s experts among the experts with unallocated resources are allocated to the remaining s slots of the second computing device corresponding to the first bucket, and the experts other than the s experts among the experts with unallocated resources are allocated to slots occupied by experts whose number of dedicated slots is greater than a first preset number, so as to obtain a second allocation result; wherein the first bucket is one of the multiple buckets; The expert placement table is updated based on the first allocation result and the second allocation result.

12. The first computing device according to claim 10, wherein: In the aspect of updating the word-element-device table based on the K buckets and the word-element part-of-speech table, the processing unit is specifically configured to: Obtaining, based on the word-gram part-of-speech table, a word-gram proportion of each part of speech in the word-grams included in each bin of the K bins corresponding to the K second computing devices; Determine the part of speech with the largest proportion among the word-gram proportions of each part of speech in the word-grams included in each bin as the part of speech of the word-gram processed by the second computing device corresponding to each bin; The part-of-speech-device mapping table is updated based on the identifier of the second computing device corresponding to each bin and the part-of-speech of the word-gram processed by each second computing device.

13. A target computing device, characterized in that The target computing device is applied to an inference system, the inference system includes a first computing device and K second computing devices, the inference system is applied to an inference model, the inference model includes L hybrid expert MOE layers, each MOE layer includes multiple MOEs, and experts in each MOE layer are deployed on the K second computing devices, where L and K are integers greater than 1; The target computing device is one of the K second computing devices, and the target computing device includes: a transceiver unit configured to receive a target word tuple from the first computing device, the target word tuple being one of K word tuples, the K word tuples being obtained by grouping a plurality of word segments based on a part-of-speech-device mapping table, the plurality of word segments being obtained by segmenting an inference request statement, the K word tuples corresponding to the K second computing devices, the part of speech corresponding to each of the K second computing devices in the part-of-speech-device mapping table including the part of speech of the word tuple corresponding to each second computing device; a processing unit, configured to call an expert deployed on a target computing device to process the target word tuple to obtain a first processing result; an acquiring unit, configured to acquire K-1 second processing results from K-1 second computing devices other than the target computing device among the K second computing devices, the K-1 processing results being obtained by the K-1 second computing devices processing the received word tuples based on the deployed experts; The processing unit is further configured to obtain an inference response statement for responding to the inference request statement based on the first processing result and the K-1 second processing results; The transceiver unit is further configured to send the inference response statement to the first computing device.

14. The target computing device according to claim 13, wherein: The transceiver unit is further configured to receive an expert adjustment instruction from the first computing device; The processing unit is further configured to adjust the experts deployed on the target computing device based on the expert adjustment instruction, where after the adjustment, the experts deployed on the target computing device include T experts that are called the most times when the target computing device processes the received word tuples. Here, T is the ratio of the number of types of experts in the MOE layer to K.

15. A first computing device, characterized in that: The system comprises a processor and a memory, wherein the memory is used to store program codes, and the processor is used to execute the program codes to implement the method according to any one of claims 1 to 5.

16. A target computing device, characterized in that The system comprises a processor and a memory, wherein the memory is used to store program codes, and the processor is used to execute the program codes to implement the method according to any one of claims 6 to 7.

17. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

18. A computer program product, characterized in that The computer program product comprises instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is implemented.

Images