Task reasoning method and content recommendation method

Abstract

Embodiments of the present specification provide a task reasoning method and a content recommendation method, wherein the task reasoning method comprises: obtaining routing information of target task data in a reasoning process of the target task data, wherein the routing information is used to indicate a plurality of candidate networks participating in the reasoning process in a mixed expert model; dividing the plurality of candidate networks into reserved networks and to-be-mapped networks based on the routing information; determining a target reserved network corresponding to a to-be-mapped network from the reserved networks according to a mapping loss of the to-be-mapped network to the reserved networks, and determining a mapping parameter of the to-be-mapped network mapping to the target reserved network; and based on the mapping parameter, calling the reserved networks to continue reasoning on the target task data. Thus, while reducing the network calling and the corresponding computing overhead in the reasoning process of the mixed expert model, the influence of the reduction of the actual number of networks on the reasoning result is reduced, so as to balance the reasoning efficiency and the accuracy of the reasoning result of the mixed expert model.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to task reasoning methods and content recommendation methods. Background Technology

[0002] With the development of large-scale neural network models, hybrid expert models have been widely applied in tasks such as natural language processing, content recommendation, and intelligent question answering. Hybrid expert models typically include multiple expert networks and routers, with the routers selecting some expert networks to participate in the computation based on the input data. To reduce the computational, storage, and communication overhead during inference, related technologies often employ methods such as reducing the number of experts involved in the computation, compressing the expert set, or rerouting experts to accelerate inference.

[0003] However, existing methods for accelerating hybrid expert models typically reduce inference overhead by reducing or replacing parts of the expert network. This can easily lead to the loss of output contributions or output bias corresponding to the reduced or replaced expert networks, thus affecting the model's inference results. Furthermore, the resource bottlenecks faced at different inference stages vary, and existing acceleration methods struggle to balance resource constraints at different inference stages with the accuracy of the inference results. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a task reasoning method. One or more embodiments of this specification also relate to a content recommendation method, a task reasoning apparatus, a content recommendation device, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a task reasoning method is provided, comprising: During the reasoning process of the target task data, the routing information of the target task data is obtained, wherein the routing information is used to indicate multiple candidate networks participating in the reasoning process in the hybrid expert model; Based on routing information, multiple candidate networks are divided into reserved networks and networks to be mapped; Based on the mapping loss of the network to be mapped to the retained network, the target retained network corresponding to the network to be mapped is determined from the retained network, and the mapping parameters of the network to be mapped to the target retained network are determined. Based on the mapping parameters, the retained network is invoked to continue inference on the target task data.

[0006] According to a second aspect of the embodiments of this specification, a content recommendation method is provided, comprising: During the recommendation process of the content to be recommended, the routing information of the content to be recommended is obtained. The routing information is used to indicate the multiple candidate networks participating in the recommendation process in the hybrid expert model. Based on routing information, multiple candidate networks are divided into reserved networks and networks to be mapped; Based on the mapping loss of the network to be mapped to the retained network, the target retained network corresponding to the network to be mapped is determined from the retained network, and the mapping parameters of the network to be mapped to the target retained network are determined. Based on the mapping parameters, the retention network is invoked to continue recommending the content to be recommended.

[0007] According to a third aspect of the embodiments of this specification, a task reasoning apparatus is provided, comprising: The first acquisition module is configured to acquire routing information of the target task data during the inference process of the target task data, wherein the routing information is used to indicate multiple candidate networks participating in the inference process in the hybrid expert model. The first partitioning module is configured to divide multiple candidate networks into reserved networks and networks to be mapped based on routing information; The first determining module is configured to determine the target retained network corresponding to the network to be mapped from the retained network based on the mapping loss of the network to be mapped to the retained network, and to determine the mapping parameters of the network to be mapped to the target retained network. The inference module is configured to invoke the retained network based on the mapping parameters to continue inference on the target task data.

[0008] According to a fourth aspect of the embodiments of this specification, a content recommendation apparatus is provided, comprising: The second acquisition module is configured to acquire routing information of the content to be recommended during the recommendation process, wherein the routing information is used to indicate multiple candidate networks participating in the recommendation process in the hybrid expert model; The second partitioning module is configured to divide multiple candidate networks into reserved networks and networks to be mapped based on routing information; The second determining module is configured to determine the target retained network corresponding to the network to be mapped from the retained network based on the mapping loss of the network to be mapped to the retained network, and to determine the mapping parameters of the network to be mapped to the target retained network. The recommendation module is configured to call the retention network based on the mapping parameters to continue recommending the content to be recommended.

[0009] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the methods provided in the first or second aspect above.

[0010] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the method provided in the first or second aspect described above.

[0011] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method provided in the first or second aspect described above.

[0012] This specification provides a task inference method in one embodiment. By acquiring routing information during the inference process of target task data, it can identify multiple candidate networks involved in the current inference process, providing a data foundation for subsequent selection and processing of the candidate networks currently involved in inference. By dividing multiple candidate networks into reserved networks and networks to be mapped based on routing information, it can distinguish between networks that will be actually called later and networks that do not need to be directly called, so that subsequent inference does not need to perform corresponding calculations on multiple candidate networks, thereby reducing the computational resource overhead caused by network calls. By determining the target reserved network corresponding to the network to be mapped from the reserved networks based on the mapping loss of the network to be mapped to the reserved network, it can utilize... The mapping loss characterizes the degree of approximation of the contribution of the receiving network when different retained networks undertake the corresponding contribution of the receiving network. This ensures a relatively suitable mapping relationship between the determined target retained network and the network to be mapped. By determining the mapping parameters from the network to be mapped to the target retained network, the mapping relationship between the two networks can be characterized, providing a basis for approximating the output contribution of the receiving network using the target retained network. By calling the retained network based on the mapping parameters to continue inference on the target task data, the relevant contribution of the network to be mapped can be approximated using the retained network and its corresponding mapping parameters without calling the network to be mapped, reducing the output bias caused by directly omitting or replacing the network to be mapped. Therefore, while reducing network calls and corresponding computational overhead during hybrid expert model inference, the impact of reducing the number of actually executed networks on the inference results can be mitigated, thus balancing the inference efficiency and accuracy of the hybrid expert model. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the inference latency of a hybrid expert model in the pre-filling and decoding stages provided in this specification; Figure 2This is a flowchart illustrating a task reasoning method provided in one embodiment of this specification; Figure 3 This is a flowchart illustrating a content recommendation method provided in one embodiment of this specification; Figure 4 This is a schematic diagram illustrating the principle of a task reasoning method based on a hybrid expert model, provided in one embodiment of this specification. Figure 5 This is an overall flowchart of a task reasoning method based on a hybrid expert model provided in one embodiment of this specification; Figure 6 This is a flowchart illustrating a method for determining mapping parameters and mapping loss according to an embodiment of this specification; Figure 7 This is a flowchart of a network partitioning and mapping method in the pre-filling stage provided by one embodiment of this specification; Figure 8 This is a flowchart of a network partitioning and mapping method for the decoding stage provided in one embodiment of this specification; Figure 9 This is a flowchart illustrating the deployment and exception rollback of a task reasoning method based on a hybrid expert model, provided in one embodiment of this specification. Figure 10 This is a flowchart of a task reasoning method provided in one embodiment of this specification; Figure 11 This is a schematic diagram of the structure of a task reasoning device provided in one embodiment of this specification; Figure 12 This is a schematic diagram of the structure of a content recommendation device provided in one embodiment of this specification; Figure 13 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0014] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0015] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The term “at least one” as used in one or more embodiments of this specification means “one or more,” and “a plurality of” means “two or more.” The term “comprising” is an open-ended description and should be understood as “including but not limiting,” and may include other content in addition to what has been described.

[0016] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0017] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in one or more embodiments of this specification are obtained through open-source datasets or public datasets that comply with their license agreements, or are obtained with full authorization from the relevant parties. Moreover, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0018] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0019] Mixture-of-Experts (MoE) is a neural network model that includes multiple expert networks and routers. The router selects some expert networks from multiple expert networks to participate in the calculation based on the input data, and combines the outputs of the selected expert networks according to the corresponding routing weights to obtain the output of the mixed expert layer.

[0020] Top-K routing refers to a routing mechanism in which a router selects the top K expert networks from multiple expert networks to participate in the processing of the current data unit, based on the routing results obtained from multiple expert networks. Here, K is the preset number of routing networks.

[0021] For example, for a target data unit input to the hybrid expert layer, the output of the hybrid expert layer can be determined based on the multiple candidate networks corresponding to the target data unit and their routing weights. The output of the hybrid expert layer can be expressed as formula (1): (1) in, Indicates the input data unit; Indicates that the hybrid expert model is for The set of candidate networks determined by Top-K routing; Represent the candidate network; Representing candidate networks Corresponding to Routing weights; Representing candidate networks against The output results obtained; This represents the output of the corresponding hybrid expert layer.

[0022] Key-Value Cache (KV Cache): This refers to a data storage structure that caches the key and value data required for attention computation during model inference, allowing for reuse in subsequent decoding processes. The pre-filling stage can establish a key-value cache based on multiple input data units, and the decoding stage can progressively generate output data units based on this key-value cache.

[0023] Fused Hybrid Expert Operator (Fused MoE): This refers to a computational operator used to integrate and execute hybrid expert processing procedures such as data distribution, expert network computation, and statistical analysis of computation results. It can run on graphics processors or other computing acceleration devices.

[0024] Time to First Token (TTFT): This refers to the time elapsed from the arrival of the inference request to the generation of the first output token by the model. It can characterize the inference latency of the model during the pre-filling stage and when generating the first output result.

[0025] Time per Output Token (TPOT) refers to the average time required to generate one output token during the decoding phase. It can characterize the inference latency of the model when it continuously generates output data during the decoding phase.

[0026] Requests per second (QPS): This refers to the number of requests received or processed by the inference service per unit of time, and can characterize the request load of the inference service. In inference performance testing, inference latency or throughput performance can be evaluated at different QPS levels.

[0027] L2 Norm: refers to the norm used to characterize the magnitude of a vector.

[0028] In the current field of hybrid expert model inference acceleration, the mainstream technical solutions can be mainly divided into the following three categories.

[0029] The first type involves reducing the number of expert networks invoked for each input data point. This approach typically reduces the number of expert networks actually executed for each input data point, based on the original routing results, to lower the computational overhead of expert network calculations. However, the omitted expert networks no longer participate in the computation, and their original output contributions may be lost, thus biasing the model output and affecting the inference results.

[0030] The second category involves reducing the number of expert networks needed during inference. This type of approach typically reduces the amount of expert networks that need to be computed or loaded during inference by pruning or merging some expert networks, or limiting the number of expert networks actually accessed within a decoding batch. See also... Figure 1 , Figure 1 This diagram illustrates the inference latency of a hybrid expert model provided in this specification during the pre-filling and decoding phases. Figure 1 The horizontal axis represents the number of expert networks activated in the batch, including 8, 16, 24, 32, 48, 64, 80, 96, 112, and 128 expert networks; the vertical axis represents the inference latency of the hybrid expert model in milliseconds, expressed on a logarithmic scale; the different curves correspond to the cases where the number of data units in each hybrid expert layer is 8, 128, 1024, 4096, 8192, and 16384, respectively. Figure 1 The results show that during the pre-filling stage, increasing the number of active expert networks in a batch from 8 to 128 only changes the inference latency by about 2.9%; however, during the decoding stage, increasing the number of active expert networks in a batch from 8 to 64 increases the inference latency by about 3.7 times. This indicates that the pre-filling and decoding stages have different inference bottlenecks: the pre-filling stage is more susceptible to the computational load of the expert network corresponding to each data unit, while the decoding stage is more susceptible to the number of expert networks actually accessed in a batch and the corresponding expert network weights read.

[0031] The third category involves replacing the unexecuted expert network with another expert network. This type of scheme typically redistributes data originally processed by the unexecuted expert network to other expert networks that are more similar to it, based on the similarity between the outputs of different expert networks, thus approximating the unexecuted expert network using the other expert network. However, even if the outputs of different expert networks have high directional similarity, their output magnitudes may still differ. Therefore, directly using another expert network for replacement may lead to a deviation between the replaced output and the original expert network's output, thereby affecting the inference results.

[0032] Based on this, this specification provides a task reasoning method, a content recommendation method, a task reasoning device, a content recommendation device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0033] The task reasoning method provided in the embodiments of this specification can be applied to various reasoning scenarios that use hybrid expert models for task processing. The specific choice can be made based on actual business needs, model type, and deployment environment; this specification does not impose any limitations on this. For example, this task reasoning method can be applied to scenarios that require calling hybrid expert models for online reasoning, such as online dialogue, intelligent question answering, information search, content recommendation, code generation, mathematical reasoning, and intelligent agent services; it can also be applied to scenarios with high requirements for model reasoning latency and throughput, such as long context processing and high-concurrency request processing.

[0034] In some embodiments, this task inference method can also be applied to batch inference scenarios such as offline batch processing, data generation, knowledge extraction, and model evaluation. In other embodiments, this task inference method can be applied to model deployment scenarios with limited computing resources, such as deployment on graphics processors, artificial intelligence accelerators, or heterogeneous computing environments composed of multiple computing devices, and is particularly suitable for scenarios where memory capacity, memory bandwidth, or cross-device communication resources are limited. Therefore, the task inference method provided in the embodiments of this specification is not limited to specific business tasks or specific computing devices. As long as a hybrid expert model is used for inference during task processing, the task inference method provided in the embodiments of this specification can be applied according to the actual situation.

[0035] For example, the task inference method provided in the embodiments of this specification can run as a pluggable optimization module in the inference engine. When applying this task inference method, it is not necessary to retrain the original hybrid expert model, modify the network parameters of each network in the hybrid expert model, or rely on the labeled data corresponding to a specific task. Whether to enable the corresponding network partitioning and mapping functions can be controlled by the startup parameters of the inference engine.

[0036] See Figure 2 , Figure 2 This specification shows a flowchart of a task reasoning method according to an embodiment, which specifically includes the following steps: Step 202: During the reasoning process of the target task data, obtain the routing information of the target task data, wherein the routing information is used to indicate multiple candidate networks participating in the reasoning process in the hybrid expert model.

[0037] It should be noted that the target task data refers to the data to be processed by the hybrid expert model for inference, which can be text data, image data, voice data, code data, or other data that can be input into the hybrid expert model for processing.

[0038] Hybrid expert models refer to neural networks that include multiple networks and routers. Routers can select some networks from multiple networks to participate in the current inference process based on input data.

[0039] Candidate networks are networks determined based on the routing results corresponding to the target task data and that participate in the current inference process. Hybrid expert models typically involve a router selecting a subset of networks from multiple networks to participate in the computation and combining the outputs of the selected networks.

[0040] Routing information refers to information that can characterize the routing relationship between the target task data and multiple candidate networks. Through routing information, it can be determined which candidate networks the target task data is routed to during the current inference process.

[0041] In practical applications, there are multiple ways to obtain routing information for target task data during the inference process. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on these methods. One possible implementation involves calling the router in the hybrid expert model to perform routing processing on the target task data. The routing information of the target task data is obtained based on the routing results output by the router, thereby determining the multiple candidate networks involved in the current inference process. Another possible implementation involves the inference engine obtaining and reading the corresponding original routing results after the hybrid expert model has completed the routing processing for the target task data, and using these original routing results as the routing information for the target task data. The inference engine can obtain the corresponding original routing results at different inference stages.

[0042] For example, after the target task data is input into the hybrid expert model, the router in the hybrid expert model can select network A, network B and network C from multiple networks to participate in the current inference process. The obtained routing information can be used to indicate network A, network B and network C, and determine network A, network B and network C as multiple candidate networks corresponding to the current inference process.

[0043] Step 204: Based on routing information, divide multiple candidate networks into reserved networks and networks to be mapped.

[0044] It should be noted that "reserved networks" refer to those selected from multiple candidate networks that will continue to participate in data inference for the target task; "networks to be mapped" refer to those candidate networks that were not selected as reserved networks but will subsequently inherit their contributions through the corresponding reserved networks. Multiple candidate networks can be divided into reserved networks and networks to be mapped. Reserved networks will actually participate in network computation, while networks to be mapped do not need to be directly executed.

[0045] In practical applications, there are various ways to divide multiple candidate networks into reserved networks and networks to be mapped based on routing information. The specific method chosen depends on the actual situation, and this specification does not limit this approach. In one possible implementation, multiple candidate networks can be determined based on routing information, and a portion of these networks can be selected as reserved networks according to preset retention criteria, while the remaining candidate networks are designated as networks to be mapped. In another possible implementation, the number of candidate networks to be retained can be determined based on the resource budget corresponding to the current inference process, and a corresponding number of reserved networks can be determined from the multiple candidate networks based on routing information. The unretained candidate networks are designated as networks to be mapped. The specific granularity of network division and the criteria for network selection can be set according to actual inference requirements.

[0046] In one optional embodiment of this specification, the process of dividing multiple candidate networks into reserved networks and networks to be mapped based on routing information may include the following steps: For the first candidate network, obtain the statistical indicators of the first candidate network. The statistical indicators are obtained based on the output results of the first candidate network for multiple auxiliary data units. The first candidate network is any one of the multiple candidate networks. Based on the routing weights and statistical metrics of the first candidate network, the importance metrics of the first candidate network are determined. Based on the importance index of the first candidate network, the first candidate network is divided into a reserved network or a network to be mapped.

[0047] It should be noted that auxiliary data units refer to data units used as input to the candidate network for processing to obtain the corresponding output results of the candidate network. Multiple auxiliary data units can originate from pre-acquired auxiliary data. For example, when the auxiliary data is text data, the auxiliary data units can be data units obtained by word segmentation of the text data.

[0048] Statistical metrics are indicators derived from the output results of a candidate network across multiple auxiliary data units. For example, a statistical metric could be the output norm of the candidate network, reflecting the magnitude of its output. Statistical metrics can be pre-generated and cached, and then directly accessed during the inference process of the target task data.

[0049] Routing weight refers to the weight determined by the router for a candidate network, used to weight the output of the candidate network. It can be used to characterize the relative weight of the candidate network in the current routing result. The larger the routing weight, the greater the weight of the candidate network's output in the corresponding hybrid expert layer output.

[0050] Importance metrics are used to characterize the relative contribution of the candidate network's corresponding route to the current hybrid expert layer output.

[0051] In practical applications, there are various ways to obtain the statistical indicators of the first candidate network, and the specific method should be selected according to the actual situation. This specification does not impose any limitations on these methods in the embodiments. In one possible implementation, the first candidate network can be pre-processed to obtain the output results corresponding to each auxiliary data unit, and statistical indicators of the first candidate network can be pre-generated based on these output results. These statistical indicators are then read during the inference process of the target task data. In another possible implementation, during the forward processing of multiple auxiliary data units using a hybrid expert model, the output results of the first candidate network for each auxiliary data unit can be continuously statistically analyzed, and the statistical indicators of the first candidate network can be updated based on the newly added output results. The updated statistical indicators are then obtained during the inference process of the target task data.

[0052] Furthermore, there are multiple ways to determine the importance index of the first candidate network based on its routing weights and statistical metrics. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In one possible implementation, the routing weights of the first candidate network can be multiplied by the statistical metrics, and the resulting product can be used as the importance index of the first candidate network. For example, when the statistical metric is the output norm statistic, the routing weights can be multiplied by the output norm statistic to estimate the contribution magnitude of the route corresponding to the first candidate network to the current hybrid expert layer output. In another possible implementation, the importance index of the first candidate network can be further determined by combining at least one of historical access frequency, weight loading cost, or communication cost with the product of the routing weights and statistical metrics, to adapt to different model deployment environments or resource constraints.

[0053] Furthermore, there are multiple ways to classify the first candidate network into a retained network or a network to be mapped based on its importance index. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In one possible implementation, multiple candidate networks can be ranked according to their importance indices, and the candidate networks ranked higher are determined as retained networks, while the remaining candidate networks are determined as networks to be mapped, based on a preset number of retained networks. In another possible implementation, the number of retained networks can be determined based on the network retention budget corresponding to the current inference process, and the first candidate network can be determined to be either a retained network or a network to be mapped by combining the relative ranking of the importance indices of multiple candidate networks. The network retention budget can be configured based on the model layer, request, batch, or device load. The specific implementation in this scheme also adopts the method of ranking by importance and retaining the top-ranked networks.

[0054] By applying the scheme of the embodiments of this specification, statistical indicators obtained in advance by candidate networks based on multiple auxiliary data units are used, and importance indicators are determined by combining the current routing weights. The relative contribution of the corresponding routes of candidate networks to the output of the hybrid expert layer can be estimated by comprehensively considering the current routing weights and the statistical situation of the output amplitude of candidate networks. Based on this contribution, candidate networks can be retained or mapped, so that relatively important candidate networks are retained first, providing a basis for reducing the impact on the inference results while reducing the number of actual execution networks.

[0055] In one optional embodiment of this specification, obtaining the statistical metrics of the first candidate network may include the following steps: The first candidate network is invoked to process multiple auxiliary data units, and the output results corresponding to each of the multiple auxiliary data units are obtained. The output results are statistically analyzed to obtain the statistical indicators of the first candidate network.

[0056] It should be noted that the output of the first candidate network for multiple auxiliary data units refers to the data generated after the first candidate network processes the auxiliary data units, which can be an output vector with multiple dimensions. For example, the output norm statistics can be obtained using the output of the first candidate network for multiple auxiliary data units, and this output norm statistics can be used as a statistical indicator of the first candidate network, where the output norm is used to characterize the magnitude of the vector corresponding to the output result. The obtained statistical indicator can be cached for use in the inference process of the target task data.

[0057] In practical applications, there are multiple ways to call the first candidate network to process multiple auxiliary data units and obtain the output results corresponding to each auxiliary data unit. The specific method is selected according to the actual situation, and this specification does not limit this approach. In one possible implementation, unlabeled auxiliary data can be pre-acquired, and the multiple auxiliary data units included in the auxiliary data can be input into a hybrid expert model. Forward processing is performed while keeping the model parameters of the hybrid expert model unchanged, and the output results generated by the first candidate network for each of the multiple auxiliary data units routed to the first candidate network are obtained. In another possible implementation, multiple sets of auxiliary data can be acquired, and the hybrid expert model can be called to perform forward processing on the auxiliary data units included in each set of auxiliary data. The output results generated by the first candidate network for the multiple auxiliary data units are then summarized. The auxiliary data can be unlabeled data from code, scientific, dialogue, or other domains. In a specific example, 32 unlabeled texts can be used, with each text containing no more than 4096 data units. This number is merely an example and does not constitute a limitation.

[0058] Furthermore, there are various ways to statistically analyze multiple output results to obtain the statistical indicators of the first candidate network. The specific method chosen depends on the actual situation, and this specification does not limit this approach. In one possible implementation, the output norms corresponding to the output results obtained by the first candidate network for multiple auxiliary data units can be determined separately, and statistics can be performed based on these multiple output norms to obtain the output norm statistics of the first candidate network. These output norm statistics are then determined as the statistical indicators of the first candidate network. In another possible implementation, statistics can be performed on multiple output results of the corresponding network according to different hybrid expert layers in the hybrid expert model, thereby obtaining the output norm statistics of the corresponding network in each hybrid expert layer. The output norm statistics corresponding to the hybrid expert layer where the first candidate network is located are then determined as the statistical indicators of the first candidate network. Related schemes can read pre-cached output norm estimates according to the expert network during inference.

[0059] In one possible embodiment of this specification, when dividing the reserved network and the network to be mapped based on routing information, the network score corresponding to the candidate network can be determined according to at least one of the following: the routing weight of the candidate network, the output norm statistics, the combination of the routing weight and the output norm statistics, the historical access frequency, the weight loading cost, and the communication cost. The reserved network and the network to be mapped can then be determined based on the network scores corresponding to multiple candidate networks.

[0060] By applying the scheme of the embodiments of this specification, by calling the first candidate network to process multiple auxiliary data units and obtaining statistical indicators based on the corresponding multiple output results, the statistical situation of the output size of the first candidate network can be reflected by multiple auxiliary data units. This allows the estimated information of its output amplitude to be obtained without actually executing the first candidate network in the current inference process of the target task data, thus providing a basis for determining the importance index of the first candidate network in combination with routing weights.

[0061] In one optional embodiment of this specification, before dividing multiple candidate networks into reserved networks and networks to be mapped based on routing information, the following steps may be included: Based on the current inference stage of the target task data, a target partitioning strategy is determined, where different current inference stages correspond to different target partitioning strategies; Based on routing information, multiple candidate networks are divided into reserved networks and networks to be mapped, including: Based on the target partitioning strategy and routing information, multiple candidate networks are divided into reserved networks and networks to be mapped.

[0062] It should be noted that the current inference stage refers to the inference stage in which the hybrid expert model is currently processing the target task data. Since different inference stages involve different network call methods and resource constraints, different target partitioning strategies can be set for different inference stages.

[0063] The target partitioning strategy refers to the partitioning granularity and network retention budget used in the current inference stage to determine the retained network and the network to be mapped from multiple candidate networks.

[0064] In practical applications, there are multiple ways to determine the target partitioning strategy based on the current inference stage of the target task data. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. One possible implementation of this specification is to pre-establish a correspondence between different inference stages and different partitioning strategies. After determining the current inference stage, the target partitioning strategy corresponding to the current inference stage is determined based on this correspondence. Another possible implementation of this specification is to configure corresponding network retention budgets for different inference stages. After determining the current inference stage, the network retention budget corresponding to the current inference stage is read, and the target partitioning strategy is determined based on the network retention budget. The network retention budgets for different inference stages can be set separately and adjusted according to the model layer, requests, batches, or device load.

[0065] Furthermore, based on the target partitioning strategy and routing information, there are various ways to divide multiple candidate networks into reserved networks and networks to be mapped. The specific method chosen depends on the actual situation, and this specification does not limit this approach. In one possible implementation, when the target partitioning strategy indicates that the partitioning granularity is the target data unit, multiple candidate networks corresponding to the target data unit can be determined based on routing information. Then, combining the importance indicators of the multiple candidate networks and the network retention budget corresponding to the target partitioning strategy, a reserved network is determined from the multiple candidate networks, and the remaining candidate networks are determined as networks to be mapped. In another possible implementation, when the target partitioning strategy indicates that the partitioning granularity is a set of data units composed of multiple target data units, multiple candidate networks corresponding to the set of data units can be determined based on routing information. Then, combining the importance of the multiple candidate networks in the set of data units and the network retention budget corresponding to the target partitioning strategy, a reserved network is determined from the multiple candidate networks, and the remaining candidate networks are determined as networks to be mapped.

[0066] Furthermore, to uniformly describe the network partitioning and mapping processes corresponding to different current inference stages, the pre-filling stage and the decoding stage can be uniformly represented as an approximate processing process of the hybrid expert layer's output under the network retention budget constraint corresponding to the current inference stage. Specifically, the pre-filling stage can determine the retention network at the target data unit granularity, and the decoding stage can determine a shared retention network at the target data unit set granularity. Different current inference stages can use the same mapping parameters, mapping loss, and network mapping method, but different network partitioning granularities and network retention budgets. For example, to uniformly describe the network partitioning and mapping processes of different inference stages, the current inference stage can be denoted as... Let the original routing network set corresponding to the target data unit be denoted as Let the set of retained networks corresponding to the current inference stage be denoted as Then the target data unit In the current reasoning stage The corresponding set of networks to be mapped can be expressed as formula (2): (2) in Represents the target data unit In the current reasoning stage The corresponding set of networks to be mapped; Represents the target data unit The corresponding original candidate network set; Represents the target data unit In the current reasoning stage The corresponding set of retained networks. In other words, the set of networks to be mapped consists of candidate networks from the original candidate network set excluding the retained network set.

[0067] For each network to be mapped in the set of networks to be mapped, the corresponding target retention network can be determined from the currently corresponding retention network, and the approximate relationship between the output of the network to be mapped and the output of the target retention network can be characterized using pre-determined mapping parameters. For example, the directional approximation relationship between the output of the network to be mapped and the output of the target retention network can be expressed as formula (3): (3) in, Indicates the network to be mapped. Represented as the network to be mapped The defined target retains the network. Represents the network to be mapped Mapped to the target retention network The mapping parameter allows the output of the target network to be approximated using the output of the target network, without needing to invoke the target network during the actual inference process.

[0068] Furthermore, after determining the set of retained networks and the mapping relationship between the network to be mapped and the target retained network, the output of the hybrid expert layer after network partitioning and mapping can be determined based on the output of the retained networks and the approximate results of the target retained network on the output of the network to be mapped. For example, in the current inference stage... The corresponding output of the hybrid expert layer can be expressed as formula (4): (4) in, Indicates the current stage of reasoning The target data unit obtained after network partitioning and mapping The corresponding hybrid expert layer output; t represents the reserved network; Represents the target data unit The intersection of the original candidate network set and the currently retained network set; Represents the target data unit Corresponding to the original routing weights of the preserved network t; Indicates network reservation For target data unit The output; Indicates the network to be mapped; Represents the target data unit Corresponding to the network to be mapped The original routing weights; Represents the network to be mapped Mapped to the target retention network Mapping parameters; Indicates the target-preserving network For target data unit The output of .

[0069] As can be seen from formula (4), for a reserved network, the corresponding output contribution can be calculated according to its original routing weight; for a network to be mapped, there is no need to actually call the network to be mapped, but the original output contribution of the network to be mapped can be approximated by the routing weight, mapping parameters and output of the target reserved network.

[0070] Furthermore, the corresponding network retention budget can be determined based on the resource constraints of the current inference stage, and the set of retained networks and the mapping relationship between the network to be mapped and the target retained network can be determined under the network retention budget constraints. For example, the network partitioning and mapping target corresponding to the current inference stage can be expressed as formula (5): (5) in, Indicates the current inference stage The corresponding set of retained networks; Indicates the current inference stage The mapping relationship between the network to be mapped and the target retention network; Indicates the current inference stage The corresponding set of target data units; express Target data unit in This represents the hybrid expert layer output obtained by calling multiple candidate networks according to the original routing information; This represents the output of the hybrid expert layer obtained after network partitioning and mapping. Indicates the current inference stage Using a reserved network set The corresponding resource overhead; Indicates the current inference stage The corresponding resource budget; This indicates a constraint.

[0071] Specifically, when the current inference stage is the pre-filling stage, the network retention budget can be used to constrain the number of candidate networks actually invoked for each target data unit; when the current inference stage is the decoding stage, the network retention budget can be used to constrain the number of different candidate networks actually accessed by the target data unit set. Therefore, appropriate network partitioning granularity and network retention budgets can be adopted at different current inference stages, and the difference between the hybrid expert layer output obtained after network partitioning and mapping and the original hybrid expert layer output can be reduced under the corresponding budget constraints.

[0072] By applying the scheme of the embodiments in this specification, and by determining the corresponding target partitioning strategy according to the current inference stage, different network partitioning granularity and network retention budget can be adopted for different inference stages. This enables the pre-filling stage to control the number of networks actually executed for each data unit, and the decoding stage to control the number of different networks actually accessed for the current batch. As a result, the same network mapping and folding mechanism can adapt to the resource constraints corresponding to different inference stages.

[0073] In one optional embodiment of this specification, the current inference stage includes a pre-filling stage, the target task data includes multiple target data units, and the target partitioning strategy corresponding to the pre-filling stage is used to indicate partitioning based on the target data units respectively. The above-mentioned division of multiple candidate networks into reserved networks and networks to be mapped based on target partitioning strategies and routing information may include the following steps: For a first candidate network, a first target data unit is determined from multiple target data units and routed to the first candidate network, wherein the first candidate network is any one of the multiple candidate networks, and the first target data unit is any one of the target data units routed to the first candidate network; Based on routing information, a first candidate network set corresponding to the first target data unit is determined from multiple candidate networks, wherein the first candidate network set includes the candidate networks to which the first target data unit is routed; Based on the importance index of multiple candidate networks in the first candidate network set, the first candidate networks are divided into reserved networks or networks to be mapped.

[0074] It should be noted that the pre-filling stage refers to the inference stage where the hybrid expert model processes multiple data units of the input to generate the information needed for subsequent decoding. For example, the pre-filling stage can process multiple data units from the input prompt information at once and establish corresponding key-value caches. Compared to the decoding stage, the pre-filling stage is mainly affected by the computational cost of the candidate networks corresponding to each data unit.

[0075] A target data unit refers to a data unit within the target task data that is routed and processed in a hybrid expert model. The target task data may include multiple target data units. During the pre-filling stage, routing can be performed on each target data unit separately to determine multiple candidate networks corresponding to each target data unit. When the target task data is text data, the target data unit can be a tag obtained after word segmentation of the text data.

[0076] The first candidate network refers to any one of multiple candidate networks, used as a candidate network for determining whether it belongs to the reserved network or the network to be mapped.

[0077] The first target data unit refers to any target data unit among multiple target data units that is routed to the first candidate network.

[0078] The first candidate network set refers to the set of candidate networks corresponding to the first target data unit, including multiple candidate networks corresponding to the first target data unit after routing. Since the first target data unit is routed to the first candidate network, the first candidate network also belongs to the first candidate network set.

[0079] In practical applications, there are multiple ways to determine the first target data unit routed to the first candidate network from multiple target data units, depending on the actual situation. This specification does not impose any limitations on this method. In one possible implementation, the routing results corresponding to multiple target data units can be obtained sequentially, and it can be determined whether each routing result includes the first candidate network. The target data unit whose routing result includes the first candidate network is then determined as the first target data unit. In another possible implementation, one or more target data units routed to the first candidate network can be determined based on the routing record corresponding to the first candidate network in the routing information, and any one of these target data units can be selected as the first target data unit.

[0080] Furthermore, there are multiple ways to determine the first candidate network set corresponding to the first target data unit from multiple candidate networks based on routing information. The specific method chosen depends on the actual situation, and this specification does not limit this approach in any way. In one possible implementation, the routing result corresponding to the first target data unit can be read, and the multiple candidate networks indicated by the routing result can be determined as the first candidate network set. In another possible implementation, network routing records corresponding to the first target data unit can be filtered from multiple network routing records included in the routing information, and the candidate networks indicated by the filtered network routing records can be combined to form the first candidate network set.

[0081] Furthermore, there are multiple ways to classify the first candidate networks into retained networks or networks to be mapped based on the importance indices of multiple candidate networks in the first candidate network set. The specific method chosen depends on the actual situation, and this specification does not limit this approach. In one possible implementation, the candidate networks in the first candidate network set can be sorted from largest to smallest according to their importance indices, and a predetermined number of candidate networks at the top of the sorting can be classified as retained networks, while the remaining candidate networks can be classified as networks to be mapped. In another possible implementation, the number of retained networks can be determined based on the network retention budget corresponding to the pre-filling stage, and candidate networks corresponding to this number of retained networks can be selected from the first candidate network set according to their importance indices, while the unretained candidate networks can be designated as networks to be mapped.

[0082] For example, for the first target data unit The corresponding set of retained networks can be expressed as formula (6): (6) in, Indicates the first target data unit The corresponding set of retained networks; Indicates the first target data unit The corresponding first candidate network set; This represents the candidate networks in the first candidate network set; Indicates the first target data unit Corresponding to candidate network Routing weights; Representing candidate networks The corresponding statistical indicators, for example, It can be used to estimate the norm of pre-cached outputs; Representing candidate networks Corresponding to the first target data unit Importance indicators; This indicates the number of networks retained for a target data unit during the pre-filling stage; This indicates that a corresponding number of candidate networks are selected from the first candidate network set according to the importance index.

[0083] Therefore, in the first candidate network, it belongs to In the case where the first candidate network is not a reserved network corresponding to the first target data unit, the first candidate network can be divided into a reserved network. In this case, the first candidate network can be divided into the network to be mapped corresponding to the first target data unit.

[0084] For example, the first candidate network set corresponding to the first target data unit includes network A, network B, network C, and network D, with importance indices of 0.8, 0.6, 0.3, and 0.2, respectively. If two candidate networks are reserved for each target data unit during the pre-filling stage, then networks A and B, which rank higher in importance indices, can be determined as the reserved networks corresponding to the first target data unit, and networks C and D can be determined as the networks to be mapped. For other target data units in the target task data, their respective reserved networks and networks to be mapped can be determined in the same way, so the final reserved candidate networks for different target data units can be different.

[0085] By applying the scheme of the embodiments of this specification, by determining the set of candidate networks to which each target data unit is routed during the pre-filling stage, and determining the retained network and the network to be mapped based on the importance index of each candidate network in the candidate network set, each target data unit can retain relatively important candidate networks, thereby reducing the number of networks that each target data unit actually needs to execute and reducing the network computation load during the pre-filling stage.

[0086] In one optional embodiment of this specification, the current inference stage includes a decoding stage, the target task data includes multiple sets of target data units, and the target partitioning strategy corresponding to the decoding stage is used to indicate partitioning based on the sets of target data units respectively; The above-mentioned division of multiple candidate networks into reserved networks and networks to be mapped based on target partitioning strategies and routing information may include the following steps: For a first candidate network, a first target data unit set is determined from multiple target data unit sets, wherein the first candidate network is any one of the multiple candidate networks, and the first target data unit set includes multiple target data units routed to the first candidate network; Based on routing information, a second candidate network set corresponding to the first target data unit set is determined from multiple candidate networks, wherein the second candidate network set includes the candidate networks to which the first target data unit set is routed; Based on the importance index of multiple candidate networks in the second candidate network set, the first candidate network is divided into reserved networks or networks to be mapped.

[0087] It should be noted that the decoding phase refers to the inference phase where the hybrid expert model progressively generates output data units based on information obtained from preceding inference. For example, the hybrid expert model can progressively generate output data units based on the key-value cache established in the pre-filling phase. During the decoding phase, the inference engine can group the target data units currently being processed by multiple inference requests into the same scheduling batch for processing. The decoding phase is primarily affected by the number of different candidate networks actually accessed in the current batch and the overhead of reading the corresponding network parameters.

[0088] A target data unit set refers to a collection of multiple target data units that are jointly partitioned into candidate networks during the decoding phase. For example, a target data unit set can correspond to a decoding batch formed by the inference engine, which may include target data units corresponding to multiple inference requests in the current decoding process. Different decoding batches can correspond to different target data unit sets.

[0089] The first candidate network refers to any one of a plurality of candidate networks, used as a candidate network for determining whether it belongs to the reserved network or the network to be mapped. The first candidate network can be routed to by one or more target data units in the first target data unit set.

[0090] The first target data unit set refers to a set of target data units that includes multiple target data units routed to the first candidate network. For example, the first target data unit set may correspond to the current decoding batch; the first candidate network may be the original routing candidate network corresponding to at least some of the target data units in the decoding batch. It should be noted that the first target data unit set may also include target data units routed to other candidate networks.

[0091] The second candidate network set refers to the set of candidate networks corresponding to the entire first target data unit set, including the candidate networks to which each target data unit in the first target data unit set is routed. For example, the second candidate network set can be obtained by taking the union of the candidate networks corresponding to each target data unit in the first target data unit set. Therefore, the second candidate network set can include the first candidate network and the candidate networks to which other target data units in the first target data unit set are routed.

[0092] In practical applications, there are multiple ways to determine the first target data unit set from multiple target data unit sets for a first candidate network. The specific method is selected based on the actual situation, and this specification does not limit this approach. In one possible implementation, routing information corresponding to multiple target data unit sets can be obtained, and the target data unit set comprising the target data units routed to the first candidate network can be determined. This determined target data unit set is then designated as the first target data unit set. In another possible implementation, the decoding batch currently corresponding to the first candidate network can be determined based on the batch scheduling information of the inference engine, and the target data unit set comprising the multiple target data units included in this decoding batch can be designated as the first target data unit set.

[0093] Furthermore, based on routing information, there are multiple ways to determine the second candidate network set corresponding to the first target data unit set from multiple candidate networks. The specific method chosen depends on the actual situation, and this specification does not limit this approach. In one possible implementation, the candidate network set corresponding to each target data unit in the first target data unit set can be determined separately, and the multiple candidate network sets can be merged to obtain the second candidate network set. In another possible implementation, multiple routing records corresponding to the first target data unit set can be traversed, network identifiers of candidate networks can be extracted from the multiple routing records, and the extracted network identifiers can be deduplicated. The second candidate network set can then be determined based on the deduplicated network identifiers.

[0094] Furthermore, based on the importance indicators of multiple candidate networks in the second candidate network set, there are various ways to classify the first candidate network into reserved networks or networks to be mapped, and the specific method can be selected according to the actual situation. This specification does not limit this approach in any way. In one possible implementation, for each candidate network in the second candidate network set, the importance indicators pointing to that candidate network in the first target data unit set can be statistically analyzed. The candidate networks can be sorted according to their respective statistical importance indicators, and a predetermined number of candidate networks at the top of the sorting can be classified as reserved networks, while the remaining candidate networks can be classified as networks to be mapped. For example, the importance indicators corresponding to multiple target data units routed to the same candidate network can be summed to obtain the statistical importance indicator of that candidate network. In another possible implementation, based on the importance indicators of multiple candidate networks, at least one of the following information can be combined to determine the network score corresponding to each candidate network in the second candidate network set: historical access frequency, weight loading cost, or communication cost. The reserved networks and networks to be mapped can then be determined based on the network scores of the multiple candidate networks. For example, candidate networks with higher importance and lower weight loading cost or communication cost can be preferentially retained.

[0095] For example, the set of retained networks corresponding to the first set of target data units can be expressed as formula (7): (7) in, This represents the first set of target data units, which in one specific implementation can be the current decoding batch; Represents the target data unit The corresponding set of candidate networks; This represents the union of candidate networks corresponding to multiple target data units in the first target data unit set, i.e., the second candidate network set; This represents the candidate networks in the second candidate network set; Represents the target data unit Corresponding to candidate network Routing weights; Representing candidate networks The corresponding statistical metric, in a specific implementation, can be the pre-cached output norm estimate; Representing candidate networks Corresponding to the target data unit Importance indicators; This indicates a pointer to a candidate network in the first target data unit set. The statistical importance index is obtained by statistically analyzing the importance indicators. This indicates the number of candidate networks retained during the decoding stage. For networks not routed to a candidate network... The target data unit, which corresponds to the candidate network. The importance of is not included in the above statistics.

[0096] For example, the first target data unit set includes target data unit 1, target data unit 2, and target data unit 3. These three target data units are routed to multiple candidate networks, and the union of these candidate networks yields the second candidate network set. For network A, the importance indices of the data units routed to network A from the three target data units can be summed to obtain the statistical importance index of network A; similarly, the statistical importance indices of other candidate networks in the second candidate network set can be obtained. If the number of networks retained during the decoding stage... If the value is 2, then the two candidate networks ranked first in terms of statistical importance can be identified as the reserved networks shared by the first target data unit set, and the remaining candidate networks can be identified as the networks to be mapped.

[0097] By applying the scheme of the embodiments of this specification, by summarizing the candidate networks corresponding to multiple target data units at the granularity of the target data unit set during the decoding stage, and statistically analyzing the importance index corresponding to the same candidate network, the shared reserved network can be determined from the multiple candidate networks involved in the current target data unit set. This limits the number of different candidate networks actually accessed by the current target data unit set to the network reservation budget, improves the reuse of candidate network parameters among multiple target data units, and reduces network parameter reading and cross-device communication overhead.

[0098] Furthermore, in one possible implementation of this specification, the pre-filling stage and the decoding stage can be unified into an approximate processing procedure for the hybrid expert layer output under the network retention budget constraint corresponding to the current inference stage. Specifically, the network retention budget corresponding to the pre-filling stage can be used to limit the number of candidate networks actually invoked for each target data unit, and the network retention budget corresponding to the decoding stage can be used to limit the number of different candidate networks actually accessed by the target data unit set. Under the corresponding network retention budget constraint, the goal can be to reduce the difference between the original hybrid expert layer output and the hybrid expert layer output obtained based on the retained networks; for example, the goal can be to reduce the squared error between the two. The pre-filling stage and the decoding stage can use the same mapping parameters, mapping loss, and network mapping method, but employ different network partitioning granularities and network retention budgets.

[0099] Step 206: Based on the mapping loss of the network to be mapped to the retained network, determine the target retained network corresponding to the network to be mapped from the retained network, and determine the mapping parameters of the network to be mapped to the target retained network.

[0100] It's important to note that mapping loss is a metric used to measure the difference in output when the output of a retained network is used to approximate the output of the network to be mapped. A smaller mapping loss indicates that the output of the retained network, after adjustment of the mapping parameters, is more suitable for approximating the output of the network to be mapped. Mapping loss is directional; the same pair of networks can have different mapping losses depending on the mapping direction.

[0101] A target reserved network is a reserved network selected from multiple reserved networks to receive the contributions of the network to be mapped. One network to be mapped corresponds to one target reserved network, and the same target reserved network can correspond to one or more networks to be mapped.

[0102] Mapping parameters are parameters used to characterize the approximate output relationship between the network to be mapped and the target retention network. They can be used to adjust the output contribution of the target retention network when approximating the output of the network to be mapped using the output of the target retention network. Mapping parameters can be directional, meaning that the mapping parameters from the first network to the second network can be different from the mapping parameters from the second network to the first network.

[0103] In practical applications, there are multiple ways to determine the target retained network from the retained networks based on the mapping loss of the network to be mapped to the retained networks. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. One possible implementation of this specification involves obtaining the mapping loss of the network to be mapped to multiple retained networks, comparing these losses, and determining the retained network with the smallest mapping loss as the target retained network. Another possible implementation involves further considering the mapping loss of the network to be mapped to multiple retained networks, and then determining a comprehensive selection result for each retained network based on at least one of the following: the device location, communication cost, capacity limitation, or load balancing cost. The target retained network is then determined based on this comprehensive selection result. For example, if the mapping losses of two retained networks are similar, the retained network located on the same device as the network to be mapped or with lower communication cost can be preferentially selected as the target retained network.

[0104] Furthermore, there are multiple ways to determine the mapping parameters from the network to be mapped to the target retained network, and the specific method should be selected according to the actual situation. This specification does not impose any limitations on these methods in the embodiments. In one possible implementation, mapping parameters can be pre-generated for different ordered network pairs and stored according to the correspondence between the network pairs. After determining the network to be mapped and the target retained network, the corresponding mapping parameters are read from the pre-stored mapping parameters based on the ordered network pair formed by the network to be mapped and the target retained network. For example, directional mapping parameter tables can be stored separately for each hybrid expert layer of the hybrid expert model, and scalar-form directional projection coefficients can be read from the mapping parameter tables of the corresponding hybrid expert layer according to the network identifiers of the network to be mapped and the target retained network. In another possible implementation, a non-scalar form of mapping relationship can be used to determine the corresponding mapping parameters based on the network to be mapped and the target retained network. For example, the mapping parameters may include at least one of the following: channel-wise diagonal projection parameters, low-rank matrix parameters, grouped scalar parameters, or piecewise linear function parameters, to characterize the approximate relationship between the output of the target retained network and the output of the network to be mapped through different forms of mapping relationships.

[0105] For example, the network to be mapped is network A, and the currently retained networks include networks B, C, and D. The mapping losses of network A to networks B, C, and D are 0.10, 0.26, and 0.31, respectively. Therefore, network B, which has the smallest mapping loss, can be determined as the target retained network corresponding to network A. Furthermore, based on the ordered network pair formed by networks A and B, the mapping parameters from network A to network B can be read from a pre-generated mapping parameter table. Subsequently, based on these mapping parameters, the corresponding contributions originally belonging to network A can be collapsed to network B without actually executing network A.

[0106] In one optional embodiment of this specification, the hybrid expert model includes multiple original networks; Before determining the target retained network corresponding to the network to be mapped from the retained network based on the mapping loss of the network to be mapped to the retained network, and before determining the mapping parameters from the network to be mapped to the target retained network, the above steps may further include: Based on the auxiliary data units processed by multiple original networks, the network set corresponding to the auxiliary data unit is determined, wherein the same network set includes the original networks that process the same auxiliary data unit. For the first original network and the second original network in the first network set, determine the first output result of the first original network and the second output result of the second original network, wherein the first network set is any one of the network sets, and the first original network and the second original network are any two original networks in the first network set; Based on the difference between the first output result and the second output result, determine the mapping loss of the first original network to the second original network and the mapping parameters from the first original network to the second original network.

[0107] It should be noted that the original network refers to the network that was originally included in the hybrid expert model before the network partitioning and mapping were performed. For example, the original network can be the expert network selected by the router and participating in the computation of the corresponding hybrid expert layer in the hybrid expert model.

[0108] A network set refers to a collection of multiple original networks corresponding to an auxiliary data unit. For example, for each hybrid expert layer in a hybrid expert model, based on the original routing results corresponding to the auxiliary data unit at that hybrid expert layer, multiple original networks that the same auxiliary data unit is routed to can be combined to form the network set corresponding to that auxiliary data unit.

[0109] The first network set refers to any one of multiple network sets used as the basis for determining the mapping relationship between networks. Multiple original networks in the first network set correspond to the same auxiliary data unit.

[0110] The first and second original networks refer to any two different original networks in the first network set. When determining the mapping relationship from the first original network to the second original network, the first original network can be used as the source network, and the second original network can be used as the target network. The order of the first and second original networks is directional; that is, the mapping from the first original network to the second original network and the mapping from the second original network to the first original network are determined separately, and the corresponding mapping parameters and mapping losses can be different.

[0111] The first output result refers to the output result generated by the first original network after processing the corresponding auxiliary data unit; the second output result refers to the output result generated by the second original network after processing the same auxiliary data unit. For example, the first output result and the second output result can be vectors output by the corresponding original networks, respectively.

[0112] The result difference refers to the difference between the output of the first original network and the output of the first original network when approximated by the output of the second original network. For example, the second output result can be adjusted using mapping parameters, and the result difference can be characterized by the vector difference between the first output result and the adjusted second output result.

[0113] The mapping parameter is used to characterize the directional approximation relationship between the outputs of the first original network and the second original network when the first original network is mapped to the second original network; the mapping loss is used to characterize the reconstruction error generated when the output of the second original network and the corresponding mapping parameter are used to approximate the output of the first original network.

[0114] In practical applications, there are multiple ways to determine the network set corresponding to an auxiliary data unit based on the auxiliary data units processed by multiple original networks. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. One possible implementation of this specification involves calling a hybrid expert model to perform forward processing on multiple auxiliary data units. For each auxiliary data unit, the original routing results generated by the router are read, and based on these results, the multiple original networks that each auxiliary data unit commonly routes to in the current hybrid expert layer are determined. These original networks are then grouped into the network set corresponding to the respective auxiliary data unit. Another possible implementation of this specification involves recording the network identifiers corresponding to multiple original networks and the auxiliary data units actually processed by each original network during the execution of multiple auxiliary data units by the hybrid expert model. Based on the recorded correspondence between network identifiers and auxiliary data units, multiple original networks processing the same auxiliary data unit are grouped together to obtain the network set corresponding to the respective auxiliary data unit.

[0115] Furthermore, for the first original network and the second original network in the first network set, there are multiple ways to determine the first output result of the first original network and the second output result of the second original network. The specific method is selected based on the actual situation, and this specification does not limit this approach. In one possible implementation, during the forward processing of the hybrid expert model while keeping the model parameters unchanged, the output vectors generated by the first original network and the second original network for the same auxiliary data unit are recorded respectively. The output vector corresponding to the first original network is determined as the first output result, and the output vector corresponding to the second original network is determined as the second output result. In another possible implementation, for multiple auxiliary data units, a network set including both the first and second original networks can be repeatedly determined, and the output results obtained by the first and second original networks for these auxiliary data units can be obtained respectively, thereby forming multiple sets of output results corresponding to the first and second original networks for subsequent joint determination of the mapping relationship between them.

[0116] Furthermore, based on the difference between the first output result and the second output result, there are multiple ways to determine the mapping loss of the first original network to the second original network and the mapping parameters from the first original network to the second original network. The specific method should be selected according to the actual situation, and this specification does not limit this approach. In one possible implementation of this specification, scalar mapping parameters can be used to characterize the output amplitude relationship between the first and second original networks. For multiple sets of output results corresponding to the first and second original networks, the L2 norm of the first output result can be used as the sample weights corresponding to each set of output results, so that the first output result with a larger amplitude has a greater impact on the determination of the mapping parameters. For example, the sample weights can be expressed as formula (8): (8) In other possible implementations of this specification, the sample weights can also be determined based on the square of the L2 norm of the first output result or the cutoff value, or further determined based on the L2 norm of the first output result in combination with at least one of the corresponding routing weights, service importance, or confidence levels corresponding to auxiliary data units. The specific choice can be made according to the actual calibration requirements.

[0117] Furthermore, the mapping parameters from the first original network to the second original network can be determined by minimizing the weighted reconstruction error between multiple first outputs and the mapped second output. When the mapping parameters use scalars... In this case, it can be expressed as formula (9): (9) in, Represents the first primitive network Mapped to the second original network Mapping parameters; Indicates the first The sample weights corresponding to the group output results; This represents the inner product of the second output and the first output. This represents the regularization coefficient.

[0118] After obtaining the mapping parameters, the mapping loss of the first original network for the second original network can be determined based on the differences between multiple first output results and the second output results adjusted by the mapping parameters. For example, the weighted reconstruction error normalized by the energy of the first output results can be used as the mapping loss, see formula (10): (10) in, This represents the mapping loss of the first original network to the second original network. By normalizing the output energy corresponding to the first output result, the mapping losses of different ordered networks can be compared more effectively.

[0119] In another possible implementation of this specification, the mapping parameters can be in non-scalar form, such as channel-wise diagonal mapping parameters, low-rank matrix mapping parameters, grouped scalar mapping parameters, or piecewise linear mapping parameters. In this case, an output reconstruction target can be constructed based on the difference between the first output result and the second output result processed using the corresponding mapping parameters. The corresponding mapping parameters can be determined by solving the output reconstruction target, and then the mapping loss of the first original network for the second original network can be calculated based on the determined mapping parameters.

[0120] For example, the auxiliary data can consist of 32 unlabeled texts, each containing a maximum of 4096 data units, and only forward processing is performed while keeping the parameters of the hybrid expert model constant; regularization coefficients It can be set to 10 3 This value is merely one specific implementation method and does not constitute a limitation. Furthermore, when generating the mapping parameter table and the mapping loss table, for network pairs consisting of the same original network, the corresponding mapping parameter can be set to 1; for original network pairs that are not commonly routed to during the processing of multiple auxiliary data units, the mapping loss corresponding to the original network pair can be set to infinity or a preset large value, in order to reduce the possibility of selecting an unobserved original network pair as the mapping relationship when there is an observed original network pair.

[0121] In one possible implementation of this specification, the mapping parameters and mapping loss can be determined offline in advance, or new auxiliary data units and corresponding original network output results can be obtained through online sampling while keeping the model parameters of the hybrid expert model unchanged, and the existing mapping parameters and mapping loss can be incrementally updated based on the newly added output results.

[0122] By applying the scheme of the embodiments in this specification, the output results of multiple original networks that are commonly routed to by the same auxiliary data unit are collected, and the directional mapping parameters and mapping losses are determined on an ordered network pair basis. This allows for the calibration of the output amplitude relationship between different original networks. Furthermore, by weighting multiple sets of calibration results using the output amplitude of the source network and normalizing the mapping loss, the comparability of the mapping relationship between different network pairs can be improved. This provides a basis for subsequently selecting the target retention network based on the mapping loss and recovering the corresponding contribution of the network to be mapped using the mapping parameters.

[0123] Step 208: Based on the mapping parameters, invoke the retention network to continue inference on the target task data.

[0124] It should be noted that, based on the mapping parameters, calling the reserved network means, without actually calling the network to be mapped, transforming the original routing contribution of the network to be mapped to the target reserved network according to the mapping parameters between the network to be mapped and the target reserved network, and then calling the reserved network to complete the corresponding network calculations. The network to be mapped does not need to generate a corresponding network output during this inference process; its original contribution can be approximated through the target reserved network and the corresponding mapping parameters.

[0125] Continuing to infer from the target task data means that after completing the reserved network call in the current hybrid expert layer, the subsequent calculations in the model are continued based on the output obtained from the hybrid expert layer.

[0126] In practical applications, there are multiple ways to call the reserved network based on mapping parameters to continue inference on the target task data. The specific method is selected according to the actual situation, and the embodiments in this specification do not limit this. In one possible implementation of this specification, the mapping route weight corresponding to the network to be mapped can be determined according to the routing weight corresponding to the network to be mapped and the mapping parameters of the network to be mapped to the target reserved network. The mapping route weight is then merged with the original routing weight corresponding to the target reserved network. The target reserved network is then called according to the merged route weight, and the output of the reserved network is combined based on the routing weights corresponding to each reserved network to obtain the output of the current hybrid expert layer. When multiple networks to be mapped are mapped to the same target reserved network, the mapping route weights corresponding to multiple networks to be mapped can be merged before calling the target reserved network, so that the target reserved network is executed only once. In another possible implementation of this specification, the routing metadata corresponding to the network to be mapped can be remapped based on the mapping parameters. The network identifier corresponding to the network to be mapped can be modified to the network identifier of the target reserved network, and the corresponding routing weight can be adjusted according to the mapping parameters. The remapped routing metadata is then input into the hybrid expert calculation operator, which calls the corresponding reserved network to complete the calculation. This approach maintains the original organization of routing data while modifying only network identifiers and route weights, without altering network parameters in the hybrid expert model. For cases where multiple networks retain the same target after remapping, the hybrid expert calculation operator can handle duplicate targets, or the routing information corresponding to duplicate targets can be merged before invoking the retained network.

[0127] For example, during the decoding stage, the network identifier and its routing weight corresponding to the network to be mapped can be remapped to the network identifier and adjusted routing weight corresponding to the target retained network. This remapping relationship can be expressed as formula (11): (11) in, Indicates the network to be mapped; This represents the original routing weight corresponding to the network to be mapped; This indicates the network to be mapped during the decoding stage. The defined target retains the network; This represents the corresponding mapping parameters.

[0128] Furthermore, the mapping parameters can be in scalar form and absorbed into the routing weights of the network to be mapped, thus eliminating the need for additional vector transformation of the target retained network's output during inference. When using scalar mapping parameters, the transfer of the network's contribution can be achieved simply by adjusting the expert identifier and routing weights, thereby reusing existing fusion hybrid expert computation operators.

[0129] By applying the scheme of the embodiments in this specification, by obtaining routing information during the inference process of the target task data, multiple candidate networks participating in the current inference process can be identified, providing a data foundation for subsequent selection and processing of candidate networks actually participating in the inference. By dividing multiple candidate networks into reserved networks and networks to be mapped based on routing information, it is possible to distinguish between networks that will be actually called later and networks that do not need to be called directly, so that subsequent inference does not need to perform corresponding calculations on multiple candidate networks, thereby reducing the computational resource overhead caused by network calls. By determining the target reserved network corresponding to the network to be mapped from the reserved networks according to the mapping loss of the network to be mapped to the reserved network, the mapping loss can be utilized. The loss characterization method represents the degree of approximation of the contributions of different retained networks to the corresponding mapped networks, ensuring a more suitable mapping relationship between the determined target retained network and the network to be mapped. By determining the mapping parameters from the network to be mapped to the target retained network, the mapping relationship between the two networks can be characterized, providing a basis for approximating the output contribution of the corresponding mapped network using the target retained network. By calling the retained network based on the mapping parameters to continue inference on the target task data, the relevant contributions of the network to be mapped can be approximated using the retained network and its corresponding mapping parameters without calling the network to be mapped, reducing the output bias caused by directly omitting or replacing the network to be mapped. Thus, while reducing network calls and corresponding computational overhead during hybrid expert model inference, the impact of reducing the number of actually executed networks on the inference results can be reduced, thereby balancing the inference efficiency and accuracy of the hybrid expert model.

[0130] In one optional embodiment of this specification, the above-mentioned invocation of the retention network based on mapping parameters to continue inference on the target task data may include the following steps: For each target retention network in the retention network, the target retention network is invoked, and the target task data is further inferred based on the routing weight of the target retention network and the corresponding mapping parameters. For each non-target reserved network in the reserved network, the non-target reserved network is invoked, and the target task data is further inferred based on the routing weight of the non-target reserved network.

[0131] It should be noted that non-target reserved networks refer to reserved networks that have not been identified as mapping targets by the networks to be mapped. Target reserved networks, in addition to carrying their own original routing contributions, can also carry the routing contributions of their corresponding networks to be mapped, adjusted by mapping parameters. Non-target reserved networks, since they do not carry the routing contributions of the networks to be mapped, can participate in the calculation of the current hybrid expert layer according to their original routing weights.

[0132] In practical applications, for each target reserved network in the reserved network, there are multiple ways to invoke the target reserved network and continue reasoning on the target task data based on the routing weight and corresponding mapping parameters of the target reserved network. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In one possible implementation, before invoking the target reserved network, the corresponding mapping route contribution can be determined based on the routing weights and mapping parameters of one or more networks to be mapped to the target reserved network. These mapping route contributions are then merged with the original routing contribution of the target reserved network to obtain the adjusted routing information corresponding to the target reserved network. Subsequently, the target reserved network is invoked based on the adjusted routing information. Using this method, even if multiple networks to be mapped are mapped to the same target reserved network, the merging of corresponding routing contributions can be completed before the target reserved network is executed, ensuring that the target reserved network is executed only once. In another possible implementation of this specification, while maintaining the original routing data organization, the network identifiers corresponding to the networks to be mapped are remapped to the network identifiers of the corresponding target reserved networks. The corresponding routing weights are then adjusted according to the mapping parameters of the networks to be mapped. The adjusted routing information is then input into the hybrid expert calculation operator, which calls the target reserved network based on the adjusted network identifiers and routing weights. When multiple networks to be mapped are mapped to the same target reserved network, the hybrid expert calculation operator can handle duplicate target reserved networks, or the routing information corresponding to duplicate target reserved networks can be merged before being input into the hybrid expert calculation operator.

[0133] Furthermore, for each non-target reserved network in the reserved network, there are multiple ways to invoke the non-target reserved network and continue reasoning on the target task data based on the routing weight of the non-target reserved network. The specific method is selected according to the actual situation, and this specification does not limit this approach. In one possible implementation, the network identifier and routing weight corresponding to the non-target reserved network can be kept unchanged, and the original routing information of the non-target reserved network can be directly input into the hybrid expert calculation operator, which then invokes the non-target reserved network according to its original routing weight. In another possible implementation, when generating target routing information for invoking multiple reserved networks, the original routing information corresponding to the non-target reserved network and the adjusted routing information corresponding to the target reserved network can be jointly written into the target routing information, and multiple reserved networks can be invoked uniformly based on this target routing information. In this case, the routing weight corresponding to the non-target reserved network does not need to be adjusted according to the mapping parameters.

[0134] For example, candidate networks include network A, network B, network C, and network D, where network A and network B are reserved networks, and network C and network D are networks to be mapped. If both network C and network D are mapped to network A, then network A is the target reserved network, and network B is a non-target reserved network. Based on the mapping parameters from network C to network A and from network D to network A, the routing contributions corresponding to networks C and D can be adjusted and transferred to network A, allowing network A to participate in the calculation based on its original routing contribution and the transferred routing contribution; network B continues to participate in the calculation according to its original routing weight. Therefore, only networks A and B need to be called, without needing to call networks C and network D.

[0135] By applying the scheme of the embodiments in this specification, the mapping parameters are used to accept the routing contribution of the target reserved network, while the non-target reserved networks participate in the calculation according to their original routing weights. This can preserve the approximate contribution of the network to be mapped when only the reserved network is called, and reduce the number of networks actually executed.

[0136] In one optional embodiment of this specification, the above-mentioned invocation of the target reserved network, based on the routing weight of the target reserved network and the corresponding mapping parameters, to continue inference on the target task data may include the following steps: Based on the routing weight of the target network to be mapped and the mapping parameters from the target network to the target reserved network, determine the mapping routing weight from the target network to the target reserved network. Here, the target network to be mapped is the network to be mapped corresponding to the target reserved network among the networks to be mapped. Determine the target route weight of the target reserved network based on the mapped route weight and the route weight of the target reserved network; The target retention network is invoked, and further inference is performed on the target task data based on the target routing weight.

[0137] It should be noted that the target network to be mapped refers to the network to be mapped that is mapped to the current target retained network. One target retained network can correspond to one or more target networks to be mapped.

[0138] Mapped route weights are route weights determined based on the original route weights of the target network to be mapped and the mapping parameters from the target network to the target retained network. They characterize the weights corresponding to the route contribution of the target network to be mapped after conversion to the target retained network. For example, the mapping parameters can be scalar-form directional projection coefficients, which can be directly incorporated into the route weights of the target network to be mapped without actually calculating the output of the target network to be mapped.

[0139] Target route weight refers to the route weight used by a target reserved network to invoke a target reserved network after it has received the mapping route contribution corresponding to a target unmapped network. Target route weight can include the original route weight of the target reserved network and the mapping route weight corresponding to the target unmapped network mapped to it. When multiple target unmapped networks are mapped to the same target reserved network, the multiple mapping route weights can be merged into the original route weight of that target reserved network.

[0140] In practical applications, there are multiple ways to determine the mapping route weight from the target network to the target reserved network, based on the routing weight of the target network to be mapped and the mapping parameters from the target network to the target reserved network. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In one possible implementation, when the mapping parameter is in scalar form, the routing weight of the target network to be mapped can be multiplied by the mapping parameter to obtain the corresponding mapping route weight. In another possible implementation, the mapping parameter can be numerically restricted according to a preset parameter range, and then the mapping route weight can be determined using the numerically restricted mapping parameter and the routing weight of the target network to be mapped. For example, the directional scalar mapping parameter can be truncated to avoid large changes in the mapping route weight due to excessively large numerical values. For example, the off-diagonal directional projection coefficient can be truncated to -4 to 4; this specific value is not a limitation.

[0141] Furthermore, there are multiple ways to determine the target route weight of the target reserved network based on the mapped route weight and the route weight of the target reserved network. The specific method should be chosen according to the actual situation, and this specification does not impose any limitations on this method. In one possible implementation of this specification, when a target network to be mapped is mapped to a target reserved network, the mapped route weight corresponding to the target network to be mapped can be added to the original route weight of the target reserved network to obtain the target route weight corresponding to the target reserved network. In another possible implementation of this specification, when multiple target networks to be mapped are mapped to the same target reserved network, the mapped route weights corresponding to the multiple target networks to be mapped can be determined separately, and then the multiple mapped route weights can be statistically analyzed. The statistical result can then be merged with the original route weight of the target reserved network to obtain the target route weight corresponding to the target reserved network. Referring to formula (12), the target route weight can be expressed as: (12) in, Indicates the target-preserving network after folding Routing weights; Represents the target data unit The original set of Top-K routing networks; Represents the target data unit The set of retained networks corresponding to the pre-filling stage; This indicates networks that were not reserved. Represents the network to be mapped Mapping to the target retention network ; Represents the network to be mapped Existing route weights; Represents network Oriented mapping parameters mapped to network t.

[0142] Furthermore, there are multiple ways to invoke the target reserved network and continue inference on the target task data based on the target route weights. The specific method chosen depends on the actual situation, and this specification does not limit this approach. In one possible implementation, the mapped route weights and the original route weights of the target reserved network can be merged before the target reserved network is executed. The network identifier of the target reserved network and the target route weights are then input into the hybrid expert computation operator, allowing the hybrid expert computation operator to invoke the target reserved network only once and weight the output of the target reserved network using the target route weights. In another possible implementation, multiple routing records mapped to the same target reserved network can be retained. The network identifiers corresponding to these multiple routing records are uniformly pointed to the target reserved network, and each record carries its corresponding route weight. The hybrid expert computation operator processes the route weights pointing to the same target reserved network during execution. This method maintains the original data organization of the routing tensor, adjusting only the network identifiers and route weights.

[0143] For example, the target retained network is network A, with an original routing weight of 0.4; the target network to be mapped is network C, with an original routing weight of 0.2, and the mapping parameter from network C to network A is 1.5. Therefore, the mapping routing weight corresponding to network C can be determined as 0.2 × 1.5 = 0.3, and the target routing weight of network A can be further determined as 0.4 + 0.3 = 0.7. Subsequently, only network A is called, and its output is used in the current hybrid expert layer output calculation according to the target routing weight of 0.7, without actually calling network C.

[0144] By applying the scheme of the embodiments of this specification, the mapping parameters of the target network to be mapped are absorbed into its routing weight to form the mapping routing weight, and the mapping routing weight is further merged into the routing weight of the target reserved network. The transfer of the corresponding contribution of the target network to be mapped to the target reserved network can be achieved by simply adjusting the routing metadata, without the need to calculate and store the output of the target network to be mapped.

[0145] In one optional embodiment of this specification, the above task reasoning method further includes the following steps: If the candidate network partitioning fails, multiple candidate networks are invoked based on the routing information to continue inference on the target task data.

[0146] It should be noted that candidate network partitioning failure refers to the failure to obtain a network partitioning result that meets the current inference requirements during the process of partitioning multiple candidate networks into reserved networks and networks to be mapped based on routing information. In the case of candidate network partitioning failure, the network mapping and folding process based on the partitioning result can be discontinued, and the original routing result corresponding to the target task data can be used to continue inference.

[0147] Calling multiple candidate networks based on routing information means calling multiple candidate networks indicated by the routing information according to the routing results originally determined by the hybrid expert model for the target task data, instead of reducing the number of networks actually called based on the partitioning results of reserved networks and networks to be mapped. For example, it can revert to the original execution method of the top K networks of the hybrid expert model.

[0148] In practical applications, when candidate network partitioning fails, there are multiple ways to continue inference on the target task data by invoking multiple candidate networks based on routing information. The specific method chosen depends on the actual situation, and this specification does not limit this approach. In one possible implementation, after performing candidate network partitioning, the validity of the obtained network partitioning result can be checked. If it is determined that the current network partitioning result cannot be used for subsequent inference, the folding process based on the retained network and the network to be mapped is stopped, and the original candidate networks and their routing weights corresponding to the target task data are read. Multiple candidate networks are then invoked based on the original routing information to continue inference. In another possible implementation, during the candidate network partitioning process, the corresponding calculation results can be numerically checked. When the numerical check fails and a usable network partitioning result cannot be obtained, a rollback process is triggered. The original candidate network identifiers and routing weights corresponding to the current target task data are retained, and multiple candidate networks are invoked based on this original routing information to continue inference.

[0149] Furthermore, in other possible embodiments of this specification, in addition to candidate network partitioning failure, if other abnormal situations occur during the inference process, the network folding process based on the reserved network and the network to be mapped can be stopped, and multiple candidate networks can be invoked to continue inference based on the original routing information corresponding to the target task data. These other abnormal situations may include at least one of the following: missing calibration information, failure to determine an available target reserved network, numerical check failure, or quality monitoring triggering.

[0150] Missing calibration information may mean that the mapping loss, mapping parameters, or corresponding lookup table required for the current inference are not loaded, or that there is a lack of available mapping information between the network to be mapped and the reserved network. In this case, since the information required for subsequent network mapping cannot be obtained, the corresponding network mapping process can be stopped, and candidate networks can be called according to the original routing information.

[0151] An undetermined target retained network can be a pointer to the network to be mapped, where no retained network currently exists that satisfies the mapping conditions. For example, if the network to be mapped has no available mapping loss or mapping parameters for any of the current retained networks, it can be determined that no target retained network exists, and the process can fall back to the original network calling method.

[0152] Numerical check failure can mean that the intermediate calculation results obtained during candidate network selection, network mapping, or route weight adjustment do not meet the preset numerical conditions. For example, if the mapping parameters or adjusted route weights do not meet the preset numerical range, the current network folding process can be stopped, and inference can continue according to the original routing information.

[0153] Quality monitoring triggering occurs when, during or after network folding, the model output's corresponding quality metric fails to meet preset quality conditions. In the event of a quality monitoring trigger, the current network folding method can be stopped, and multiple candidate networks can be invoked based on the original routing information to continue subsequent inference.

[0154] In any of the above-mentioned abnormal situations, the original candidate network identifiers and routing weights corresponding to the target task data can be retained or restored, and inference can continue according to the original top K candidate network execution method of the hybrid expert model, without reloading or modifying the network parameters of the hybrid expert model.

[0155] By applying the solution in the embodiments of this specification, and by falling back to the network call method based on the original routing information when the candidate network partitioning fails, the original inference path of the hybrid expert model can be maintained even if the network folding process cannot continue, without the need to reload or modify the model parameters.

[0156] See Figure 3 , Figure 3 This specification illustrates a flowchart of a content recommendation method according to an embodiment, which specifically includes the following steps: Step 302: During the recommendation process of the content to be recommended, obtain the routing information of the content to be recommended, wherein the routing information is used to indicate multiple candidate networks participating in the recommendation process in the hybrid expert model.

[0157] Step 304: Based on routing information, divide multiple candidate networks into reserved networks and networks to be mapped.

[0158] Step 306: Based on the mapping loss of the network to be mapped to the retained network, determine the target retained network corresponding to the network to be mapped from the retained network, and determine the mapping parameters of the network to be mapped to the target retained network.

[0159] Step 308: Based on the mapping parameters, call the retention network to continue recommending the content to be recommended.

[0160] It should be noted that the content to be recommended refers to the data that needs to be processed for recommendation using a hybrid expert model. The content to be recommended can include content data used to determine the recommendation result or data related to content recommendation. For example, the content to be recommended can correspond to text content, product content, media content, or other data that can be processed by a hybrid expert model.

[0161] The recommendation process refers to the process of using a hybrid expert model to process the content to be recommended in order to obtain processing results related to content recommendation. During the recommendation process, the routers in the hybrid expert model can determine multiple candidate networks for the content to be recommended, and then use these candidate networks to process the content.

[0162] In the embodiments of this specification, the technical solutions of the content recommendation method and the aforementioned task reasoning method belong to the same inventive concept. The content to be recommended can correspond to the target task data in the aforementioned task reasoning method, and the recommendation process of the content to be recommended can correspond to the reasoning process of the target task data. Since the technical solutions of this content recommendation method and the aforementioned task reasoning method belong to the same concept, any details not described in detail in the technical solution of the content recommendation method can be found in the description of the technical solution of the aforementioned task reasoning method.

[0163] The solution implemented in this specification involves dividing multiple candidate networks during the content recommendation process and using mapping parameters to map the contributions of the network to be mapped to the target retained network. This reduces the number of candidate networks actually invoked while allowing the retained network to bear the approximate contributions of the network to be mapped.

[0164] See Figure 4 , Figure 4 This diagram illustrates the principle of a task reasoning method based on a hybrid expert model, as provided in one embodiment of this specification. Figure 4As shown in the diagram, this principle diagram may include a part for determining mapping parameters and mapping loss, a part for network partitioning and mapping in the pre-filling stage, and a part for network partitioning and mapping in the decoding stage. The pre-determined mapping parameters and mapping loss can be used together in the pre-filling and decoding stages to determine the retained network and the network to be mapped at the appropriate network partitioning granularity in both inference stages, and to map the contribution of the network to be mapped to the retained network.

[0165] like Figure 4 As shown, multiple auxiliary data units can be obtained during the determination of mapping parameters and mapping loss. The multiple document graphics in the upper left corner of the figure represent these auxiliary data units, and the single square below them represents any one of these auxiliary data units. A hybrid expert model can be invoked to process these multiple auxiliary data units. The hybrid expert model can include multiple original networks and a routing mechanism for determining the original network corresponding to each auxiliary data unit. The routing information represents the routing result corresponding to the auxiliary data unit, and based on this routing information, multiple original networks that process the same auxiliary data unit can be determined. Multiple original networks and the adjacent boxes representing multiple original networks represent multiple original networks participating in the corresponding processing in the hybrid expert model. The ellipses between the multiple original networks indicate that other original networks not shown individually in the figure may also be included.

[0166] For multiple original networks processing the same auxiliary data unit, a first original network and a second original network can be identified. In the figure, the two original networks shown in dashed boxes are designated as the first and second original networks, illustrating the pair of original networks with the currently defined mapping relationship. The other original networks shown in solid boxes are used to illustrate other original networks besides the current first and second original networks. It should be noted that the use of solid or dashed lines is primarily to distinguish the roles of different networks in the current example and does not limit the network structure of the original networks themselves.

[0167] Calling the first original network to process the corresponding auxiliary data unit yields a first output result; calling the second original network to process the same auxiliary data unit yields a second output result. The first and second output results can be vectors output by the corresponding original networks. The selection of the first and second original networks is directional; that is, under the current mapping direction, the first original network serves as the source network to be approximated, and the second original network serves as the target network for approximating the first original network.

[0168] Furthermore, sample weights can be determined based on the first output result. For example, the sample weights corresponding to the auxiliary data units can be determined based on the L2 norm of the first output result, so that the first output results corresponding to different auxiliary data units have corresponding weights in the mapping relationship determination process. After obtaining the first output results, second output results, and sample weights of the first and second original networks for multiple auxiliary data units, the mapping parameters from the first original network to the second original network can be determined based on the difference between the first and second output results. The mapping parameters are used to characterize the directional output approximation relationship between the first and second original networks, and can be, for example, directional scalar mapping parameters from the first original network to the second original network.

[0169] Furthermore, based on the first output, the second output, and the determined mapping parameters, the mapping loss of the first original network for the second original network can be determined. The mapping loss characterizes the difference in results when the output of the second original network and its corresponding mapping parameters are used to approximate the output of the first original network. By repeating the above process for multiple ordered original network pairs, the mapping loss can be obtained. Figure 4 The diagram shows a mapping loss table and a mapping parameter table. The mapping loss table stores the mapping losses corresponding to different ordered pairs of original networks, and the mapping parameter table stores the mapping parameters corresponding to different ordered pairs of original networks. The numbers e1, e2, ..., em surrounding the matrices in the diagram represent different original networks in the hybrid expert model. Here, e1 to em represent the network identifiers of different original networks, m represents the number of original networks in the corresponding hybrid expert layer, and the ellipsis "..." indicates that other original networks not shown individually may also be included. One dimension of the matrix can be used to represent the first original network, and the other dimension can be used to represent the second original network, so that one cell in the matrix corresponds to an original network pair with a defined mapping direction. The matrix cells in the mapping loss table with different fill patterns are used to indicate that the mapping losses corresponding to different pairs of original networks can be different, and the matrix cells in the mapping parameter table with different fill patterns are used to indicate that the mapping parameters corresponding to different pairs of original networks can be different. The fill patterns in the diagram are only used to distinguish the values ​​corresponding to different matrix cells and do not indicate a specific relationship between the values, nor do they limit the specific values ​​of the mapping loss or mapping parameters. Because the mapping relationship is directional, the matrix units corresponding to the mapping from the first original network to the second original network can have different values ​​than the matrix units corresponding to the mapping from the second original network to the first original network.

[0170] exist Figure 4The pre-filling stage, shown in the lower left corner, allows for network partitioning and mapping based on target data units. A single square in the diagram represents any one of multiple target data units. For each target data unit, routing information as shown in the diagram can be obtained. Based on this routing information, multiple candidate networks and their respective routing weights can be determined. The bar chart below the routing weights illustrates the routing weights for different candidate networks; different bar heights indicate that the routing weights for different candidate networks can be different. The bar chart below "Statistical Indicators" illustrates the statistical indicators for different candidate networks; different bar heights indicate that the statistical indicators for different candidate networks can be different. These statistical indicators can be pre-calculated based on the output results of the candidate networks for multiple auxiliary data units.

[0171] Furthermore, importance indicators can be determined based on routing weights and statistical metrics. The importance indicators of candidate networks can be determined according to their corresponding routing weights and statistical metrics. Based on the importance indicators of multiple candidate networks, they can be divided into reserved networks and networks to be mapped. In the figure, networks indicated by solid boxes represent reserved networks, and networks indicated by dashed boxes represent networks to be mapped. The figure exemplarily shows two reserved networks and two networks to be mapped; this number is only used to illustrate the relationship between reserved networks and networks to be mapped, and does not limit the actual number of reserved or mapped networks. For each network to be mapped, the target reserved network corresponding to that network can be determined. Specifically, the target reserved network corresponding to the network to be mapped can be determined from the multiple reserved networks based on the mapping loss of the network to be mapped to multiple reserved networks. For example, the aforementioned mapping loss table can be consulted to determine the reserved network with the smaller mapping loss as the target reserved network corresponding to the network to be mapped, and the mapping parameters from the network to be mapped to the target reserved network can be obtained from the mapping parameter table. The arrows pointing from the network to be mapped to the reserved networks below in the figure illustrate the mapping relationship between the network to be mapped and the corresponding target reserved network; one target reserved network can correspond to one or more networks to be mapped.

[0172] After mapping between the network to be mapped and the target reserved network is completed, the routing contribution of the network to be mapped can be adjusted based on the corresponding mapping parameters, and the target reserved network will then take over the corresponding routing contribution. Inference is then performed using the reserved network; that is, the target network is not invoked, but the reserved network is invoked to continue processing the target data unit. The solid-lined square network shown again in the lower part of the diagram represents the reserved network actually invoked after network mapping is completed.

[0173] exist Figure 4The decoding stage shown in the lower right corner can perform network partitioning and mapping based on the target data unit set. The vertical matrix in the diagram, x1, x2, x3, x4, ..., xB, represents multiple target data units in the first target data unit set. Here, x1 represents the first target data unit, x2 represents the second target data unit, and so on, with xB representing the Bth target data unit. B represents the number of target data units in the first target data unit set. The ellipsis indicates that other target data units not shown individually may also be included. The horizontal matrix, e1, e2, e3, e4, e5, e6, e7, e8, illustrates multiple candidate networks in the second candidate network set corresponding to the first target data unit set. The diagram divides e1 to e4 and e5 to e8 into two rectangular areas only for illustration purposes and does not imply that they belong to two independent candidate network sets; e1 to e8 can all belong to the second candidate network set. The number of candidate networks shown as 8 is merely illustrative; the actual number of candidate networks can be determined based on the routing results.

[0174] The circles at the intersection of target data units and candidate networks illustrate the routing correspondence between each target data unit and each candidate network. One circle corresponds to one combination of a target data unit and a candidate network. Circles with different fill patterns in the diagram distinguish different routing states or information between target data units and candidate networks. For example, a circle filled with a preset pattern can indicate that the corresponding target data unit is routed to the corresponding candidate network. Different fill patterns can also distinguish routing weights, importance indicators, or other routing information corresponding to different routes. Unfilled circles indicate that the corresponding target data unit is not routed to the corresponding candidate network. It should be noted that the fill patterns and distribution of the circles in the diagram are only used to illustrate the situation where multiple target data units correspond to different candidate networks, and do not limit the specific routing results, routing weights, importance indicators, or the magnitude relationships between these indicators.

[0175] For each candidate network in the second candidate network set, the importance of that candidate network to the first target data unit set can be determined based on the importance index corresponding to the target data units routed to that candidate network in the first target data unit set. Then, based on the importance indices of multiple candidate networks, they are divided into reserved networks and networks to be mapped. In the figure, the solid squares below the text box indicate candidate networks classified as reserved networks, and the dashed squares indicate candidate networks classified as networks to be mapped. Multiple solid squares and multiple dashed squares respectively represent the inclusion of multiple reserved networks and multiple networks to be mapped.

[0176] Furthermore, for each network to be mapped, the target retained network can be determined based on the mapping loss. Specifically, the mapping loss table can be queried to compare the mapping losses of the network to be mapped with respect to multiple retained networks, and the target retained network corresponding to the network to be mapped can be determined from among the multiple retained networks. The arrows between the network to be mapped and the retained networks in the diagram are used to indicate the mapping direction between the corresponding networks, that is, from the network to be mapped to its corresponding target retained network. After determining the target retained network, the mapping route weight can also be determined based on the route weight and mapping parameters. Specifically, the mapping parameters from the network to be mapped to the target retained network can be obtained from the mapping parameter table, and the mapping route weight can be determined based on the original route weight of the network to be mapped and the corresponding mapping parameters, so as to transfer the original route contribution of the network to be mapped to the target retained network. Multiple networks to be mapped can be mapped to their respective target retained networks, or multiple networks to be mapped to the same target retained network.

[0177] The small matrix labeled e1, e2, e3, and e4 in the diagram illustrates that after network partitioning and mapping, the routes corresponding to the first target data unit set are centralized in the reserved networks. The square cells in the matrix represent the correspondence between different target data units or routes and the reserved networks. Square cells with different filling patterns distinguish different route states or route information; these filling patterns are illustrative and do not indicate a specific numerical relationship between the corresponding route information. The network identifiers e1 to e4 in the diagram are merely examples and do not imply that networks e1 to e4 will actually be reserved. The statement that the number of reserved networks does not exceed a preset network reservation limit indicates that during the decoding stage, a corresponding network reservation budget can be set for the number of reserved networks shared by the first target data unit set, ensuring that the number of different reserved networks ultimately called by the first target data unit set does not exceed the preset network reservation limit. This allows multiple target data units to share the corresponding set of reserved networks.

[0178] Finally, the reserved network can be invoked for inference. That is, based on the routing information after mapping, the corresponding reserved network can be invoked to continue inference on multiple target data units in the first target data unit set, without actually invoking the network to be mapped.

[0179] also, Figure 4Solid arrows in the figure indicate the direction of data, routing information, or processing results transmission between processing stages. Branch arrows extending from the mapping loss table and mapping parameter table to the pre-filling and decoding stages indicate that the same mapping loss and mapping parameters can be queried and used in both stages. Rounded solid outlines in the figure distinguish different processing stages or functional areas, while dashed outlines group multiple objects belonging to the same processing stage, model, or candidate network set. Solid outlines, dashed outlines, and different fill patterns in the figure can be used to distinguish different types of networks, routing states, parameters, or processing results; each fill pattern is only used to improve the distinguishability of different objects in the figure and does not indicate a specific numerical relationship between corresponding objects. Squares, circles, columns, matrices, fill patterns, number of objects, and arrangement in the figure are all schematic and do not constitute a limitation on the embodiments of this specification.

[0180] See Figure 5 , Figure 5 This specification illustrates an overall flowchart of a task reasoning method based on a hybrid expert model, provided in one embodiment. Figure 5 As shown, the process of determining mapping parameters and mapping loss can be pre-executed. Specifically, multiple auxiliary data units can be acquired, and a hybrid expert model can be used to determine the network set corresponding to the auxiliary data units and the output results of the original network, thereby determining the mapping parameters and mapping loss. After determining the mapping parameters and mapping loss, the mapping parameters and mapping loss can be loaded, and the inference process of the target task data can begin. First, the target task data is acquired, and the target partitioning strategy is determined according to the current inference stage. Specifically, when the current inference stage is the pre-filling stage, the retained network and the network to be mapped can be partitioned based on the target data units; when the current inference stage is the decoding stage, the retained network and the network to be mapped can be partitioned based on the set of target data units. Further, the target retained network corresponding to the network to be mapped can be determined according to the mapping loss, and the routing information can be adjusted based on the mapping parameters. The retained network can be called for inference to obtain the current inference result and continue subsequent inference.

[0181] See Figure 6 , Figure 6 A flowchart illustrating a method for determining mapping parameters and mapping loss according to an embodiment of this specification is shown. Figure 6 As shown, firstly, multiple auxiliary data units are acquired, and a hybrid expert model is invoked to process these auxiliary data units to determine the network set and the first and second output results. Further, sample weights are determined and weighted statistics are performed. Based on the weighted statistical results, mapping parameters and corresponding mapping losses are determined. Subsequently, mapping parameter tables and mapping loss tables are generated, numerical processing is performed on the mapping parameters and mapping losses, and the processed mapping parameter tables and mapping loss tables are saved.

[0182] See Figure 7 , Figure 7 A flowchart illustrating a network partitioning and mapping method in a pre-filling stage according to an embodiment of this specification is shown. Figure 7 As shown, firstly, a first set of candidate networks is determined, and the importance index of the candidate networks is determined. Based on the importance index, the candidate networks are divided into reserved networks and networks to be mapped. Further, the target reserved network corresponding to the network to be mapped is determined, and the mapping route weight corresponding to the network to be mapped is determined. Based on the mapping route weight and the route weight of the target reserved network, the target route weight is determined. Subsequently, inference is performed by calling the reserved network based on the target route weight, without calling the network to be mapped during the inference process.

[0183] See Figure 8 , Figure 8 A flowchart illustrating a network partitioning and mapping method for the decoding stage according to an embodiment of this specification is shown. Figure 8 As shown, firstly, a second set of candidate networks is determined, and the importance indicators of the candidate networks are calculated. Based on these importance indicators, the candidate networks are divided into reserved networks and networks to be mapped. Further, the target reserved networks corresponding to the networks to be mapped are determined, and the routing information is adjusted based on the mapping parameters while maintaining the original data organization of the routing information. Subsequently, the reserved networks are invoked for inference, and the networks to be mapped are not invoked during the inference process.

[0184] See Figure 9 , Figure 9 This document illustrates a flowchart of the deployment and exception rollback of a task reasoning method based on a hybrid expert model, as provided in one embodiment of this specification. Figure 9 As shown, when the inference service starts, the startup parameters can be obtained, and the decision on whether to enable network partitioning and mapping can be made based on these parameters. Without network partitioning and mapping enabled, multiple candidate networks can be directly invoked based on the original routing information, and the inference of the target task data can be completed according to the original routing method of the hybrid expert model. With network partitioning and mapping enabled, the hybrid expert model parameters, mapping parameters, mapping loss, and statistical indicators can be loaded, and the target task data and the current inference stage can be obtained.

[0185] Furthermore, when the current inference stage is the pre-filling stage, network partitioning and mapping can be performed during the pre-filling stage; when the current inference stage is the decoding stage, network partitioning and mapping can be performed during the decoding stage. Different network partitioning strategies can be used in the pre-filling and decoding stages, and after completing network partitioning and mapping, the reserved network is called for inference. During the process of calling the reserved network for inference, corresponding inference state information can also be obtained, and inference state monitoring can be performed based on this information. Inference state information may include, for example, at least one of model output quality information, inference latency information, and numerical state information. When inference state monitoring determines that a preset rollback condition is met, a rollback can be triggered, stopping the current network partitioning and mapping method, and re-calling multiple candidate networks for inference based on the original routing information. For example, the above rollback process can be performed in cases where calibration information is missing, an unavailable target reserved network is not determined, numerical checks fail, or quality monitoring is triggered.

[0186] Regardless of whether a network-based inference method using network partitioning and mapping is employed, or an inference method using original routing information to call multiple candidate networks is used, the corresponding inference results can be output through the same model service interface, and the shared downstream processing program can perform subsequent processing on the inference results. Therefore, without changing the model service interface and downstream processing program, the appropriate inference method can be selected based on the inference service startup parameters, and a fallback to the original routing method can be implemented if an anomaly occurs during network partitioning and mapping.

[0187] For example, during the inference service startup process, the model parameters, word segmenter, and inference engine of the hybrid expert model can be loaded. When network partitioning and mapping are enabled, the mapping parameters, mapping loss, and statistical metrics corresponding to each hybrid expert layer can also be loaded. The inference service startup parameters can be used to configure the number of networks retained in the pre-filling stage, the number of networks retained in the decoding stage, numerical limits for mapping parameters, and anomaly fallback conditions. After completing the corresponding network calls in the pre-filling stage, attention calculations can continue and a key-value cache can be established. In the decoding stage, the scheduler can form the current decoding batch and perform network partitioning and mapping for that batch. Hybrid expert model parameters, request scheduling methods, output protocols, and evaluation programs can be shared when using the original routing method and when using the network partitioning and mapping method.

[0188] See Figure 10 , Figure 10 The flowchart of a task reasoning method provided in one embodiment of this specification is shown. During the task reasoning process, the server and the client interact with each other. The client is used to send target task data to the server. The server receives target task data sent by the client, calls the hybrid expert model to perform inference on the target task data, and obtains routing information of the target task data during the inference process. The routing information is used to indicate multiple candidate networks participating in the inference process in the hybrid expert model. Based on the routing information, the multiple candidate networks are divided into reserved networks and unmapped networks.

[0189] The server is also used to determine the target retained network corresponding to the network to be mapped from the retained network based on the mapping loss of the network to be mapped to the retained network, and to determine the mapping parameters of the network to be mapped to the target retained network; based on the mapping parameters, the retained network is invoked to continue inference on the target task data and obtain the inference result corresponding to the target task data.

[0190] The server-side component is also used to send inference results to the client.

[0191] The client is also used to receive inference results sent by the server.

[0192] For example, to verify the technical effectiveness of the task inference method provided in the embodiments of this specification for inference of a hybrid expert model, tests can be performed on both the task inference method provided in the embodiments of this specification and the hybrid expert model using the original inference method, and the model output quality and inference performance of the two can be compared. In the performance evaluation process, at least one of the following indicators can be used for evaluation: multi-task average quality, first output label time, time per output label, offline throughput, and output reconstruction proxy error.

[0193] During the pre-filling stage verification, the number of retained networks corresponding to each target data unit can be set to 4. Experimental results show that the average multi-task quality of the original inference method is 69.04, while the average multi-task quality of the task inference method provided in this specification is 69.26. Even with half the number of candidate networks actually called during the pre-filling stage, the average multi-task quality did not decrease, indicating that by mapping the contributions of the networks to be mapped to the target retained network, the impact of the reduced number of networks on the model output quality can be minimized while reducing the number of network calls during the pre-filling stage. During the decoding stage verification, the number of retained networks corresponding to the target data unit set can be set to 32. Experimental results show that the average multi-task quality of the original inference method is 69.04, while the average multi-task quality of the task inference method provided in this specification is 68.97, only 0.07 different from the original inference method. This demonstrates that by limiting the number of different candidate networks actually called for the target data unit set during the decoding stage and utilizing the target retained network to bear the contributions of the networks to be mapped, a high model output quality can be maintained while reducing the actual number of networks accessed.

[0194] Furthermore, the inference efficiency of the task inference method provided in the embodiments of this specification can be verified. With an input comprising approximately 8,000 data units and a request rate of 8 requests per second, the time to the first output marker corresponding to the original inference method is 397 milliseconds, while the time to the first output marker after using the task inference method provided in the embodiments of this specification is 283 milliseconds, corresponding to an acceleration effect of approximately 1.41 times. With a request rate of 8 requests per second, the time for each output marker corresponding to the original inference method is 23.5 milliseconds, while the time for each output marker after using the task inference method provided in the embodiments of this specification is 9.58 milliseconds, corresponding to an acceleration effect of approximately 2.45 times.

[0195] In offline inference scenarios, the offline throughput of the original inference method is 2.07 requests per second, while the offline throughput after adopting the task inference method provided in the embodiments of this specification is 2.48 requests per second, an increase of approximately 1.20 times. This demonstrates that the task inference method provided in the embodiments of this specification can not only reduce the inference time in the pre-filling and decoding stages, but also improve the overall processing efficiency in batch inference scenarios.

[0196] Furthermore, the output reconstruction surrogate error evaluation can be used to reduce the difference between the output obtained after candidate networks and the original model output. In the control method of directly retaining four candidate networks, the output reconstruction surrogate error is 1.83; after using the task inference method provided in the embodiments of this specification, the output reconstruction surrogate error is reduced to 0.07. This shows that by determining the target retention network for the network to be mapped based on the mapping loss, and using the mapping parameters to transfer the contribution corresponding to the network to be mapped to the target retention network, the output approximation error caused by not calling the network to be mapped can be reduced.

[0197] The above experimental results are only used to illustrate a specific technical effect of the task reasoning method provided in the embodiments of this specification. The number of networks retained, the request rate, the amount of input data, and various test parameters used in the experiment can be adjusted according to the actual application scenario. The embodiments of this specification do not impose any limitations on this.

[0198] Corresponding to the above method embodiments, this specification also provides embodiments of a task reasoning device. Figure 11 A schematic diagram of a task reasoning device according to one embodiment of this specification is shown. Figure 11 As shown, the device includes: The first acquisition module 1102 is configured to acquire routing information of the target task data during the inference process of the target task data, wherein the routing information is used to indicate multiple candidate networks participating in the inference process in the hybrid expert model. The first partitioning module 1104 is configured to partition multiple candidate networks into reserved networks and networks to be mapped based on routing information; The first determining module 1106 is configured to determine the target retention network corresponding to the network to be mapped from the retention network based on the mapping loss of the network to be mapped to the retention network, and to determine the mapping parameters of the network to be mapped to the target retention network. The inference module 1108 is configured to call the retained network based on the mapping parameters to continue inference on the target task data.

[0199] Optionally, the first partitioning module 1104 is further configured to obtain statistical indicators of the first candidate network for the first candidate network, wherein the statistical indicators are obtained based on the output results of the first candidate network for multiple auxiliary data units, and the first candidate network is any one of the multiple candidate networks; determine the importance index of the first candidate network based on the routing weight and statistical indicators of the first candidate network; and partition the first candidate network into a reserved network or a network to be mapped according to the importance index of the first candidate network.

[0200] Optionally, the first partitioning module 1104 is further configured to call the first candidate network to process multiple auxiliary data units and obtain the output results corresponding to the multiple auxiliary data units respectively; and to perform statistics on the output results to obtain the statistical indicators of the first candidate network.

[0201] Optionally, the inference module 1108 is further configured to, for each target reserved network in the reserved network, invoke the target reserved network and continue inference on the target task data based on the routing weight of the target reserved network and the corresponding mapping parameters; for each non-target reserved network in the reserved network, invoke the non-target reserved network and continue inference on the target task data based on the routing weight of the non-target reserved network.

[0202] Optionally, the inference module 1108 is further configured to: determine the mapping route weight from the target network to the target reserved network based on the routing weight of the target network to be mapped and the mapping parameters from the target network to the target reserved network, wherein the target network to be mapped is the network to be mapped corresponding to the target reserved network among the networks to be mapped; determine the target routing weight of the target reserved network based on the mapping route weight and the routing weight of the target reserved network; and invoke the target reserved network to continue inference on the target task data based on the target routing weight.

[0203] Optionally, the hybrid expert model includes multiple original networks; the device further includes a third determining module configured to determine a network set corresponding to an auxiliary data unit based on the auxiliary data units processed by the multiple original networks respectively, wherein the same network set includes the original networks that process the same auxiliary data unit; a fourth determining module configured to determine a first output result of the first original network and a second output result of the second original network for the first original network and the second original network in the first network set, wherein the first network set is any one of the network sets, and the first original network and the second original network are any two original networks in the first network set; a fifth determining module configured to determine the mapping loss of the first original network to the second original network and the mapping parameters from the first original network to the second original network based on the difference between the first output result and the second output result.

[0204] Optionally, the strategy determination module is configured to determine a target partitioning strategy based on the current inference stage of the target task data, wherein different current inference stages correspond to different target partitioning strategies; the first partitioning module 1104 is further configured to partition multiple candidate networks into reserved networks and networks to be mapped based on the target partitioning strategy and routing information.

[0205] Optionally, the current inference phase includes a pre-filling phase, the target task data includes multiple target data units, and the target partitioning strategy corresponding to the pre-filling phase is used to indicate partitioning based on the target data units respectively; the first partitioning module 1104 is further configured to, for the first candidate network, determine a first target data unit routed to the first candidate network from the multiple target data units, wherein the first candidate network is any one of the multiple candidate networks, and the first target data unit is any one of the target data units routed to the first candidate network; determine a first candidate network set corresponding to the first target data unit from the multiple candidate networks according to the routing information, wherein the first candidate network set includes the candidate networks routed to by the first target data unit; and divide the first candidate network into a reserved network or a network to be mapped according to the importance index of the multiple candidate networks in the first candidate network set.

[0206] Optionally, the current inference phase includes a decoding phase, the target task data includes multiple sets of target data units, and the target partitioning strategy corresponding to the decoding phase is used to indicate partitioning based on the target data unit sets respectively; the first partitioning module 1104 is further configured to partition multiple candidate networks into reserved networks and networks to be mapped based on the target partitioning strategy and routing information, including: For a first candidate network, a first target data unit set is determined from multiple target data unit sets, wherein the first candidate network is any one of the multiple candidate networks, and the first target data unit set includes multiple target data units routed to the first candidate network; Based on routing information, a second candidate network set corresponding to the first target data unit set is determined from multiple candidate networks, wherein the second candidate network set includes candidate networks that are routed to the first target data unit set; Based on the importance index of multiple candidate networks in the second candidate network set, the first candidate network is divided into reserved networks or networks to be mapped.

[0207] Optionally, the device also includes a second inference module, configured to, in the event of failure of candidate network partitioning, invoke multiple candidate networks based on routing information to continue inference on the target task data.

[0208] The task inference apparatus provided in the embodiments of this specification divides multiple candidate networks into reserved networks and networks to be mapped by using routing information based on the target task data. It determines the target reserved network and corresponding mapping parameters for the network to be mapped based on the mapping loss. Then, it calls the reserved network based on the mapping parameters to continue inference. This can reduce the number of candidate networks actually called, while using the target reserved network to approximately accept the output contribution of the network to be mapped, thereby reducing the inference overhead of the hybrid expert model and reducing the impact of directly omitting some candidate networks on the inference results.

[0209] The above is an illustrative scheme of a task reasoning device according to this embodiment. It should be noted that the technical solution of this task reasoning device and the technical solution of the task reasoning method described above belong to the same concept. For details not described in detail in the technical solution of the task reasoning device, please refer to the description of the technical solution of the task reasoning method described above.

[0210] Corresponding to the above method embodiments, this specification also provides embodiments of a content recommendation device. Figure 12 A schematic diagram of a content recommendation device according to one embodiment of this specification is shown. Figure 12 As shown, the device includes: The second acquisition module 1202 is configured to acquire routing information of the content to be recommended during the recommendation process, wherein the routing information is used to indicate multiple candidate networks participating in the recommendation process in the hybrid expert model. The second partitioning module 1204 is configured to partition multiple candidate networks into reserved networks and networks to be mapped based on routing information. The second determining module 1206 is configured to determine the target retention network corresponding to the network to be mapped from the retention network based on the mapping loss of the network to be mapped to the retention network, and to determine the mapping parameters of the network to be mapped to the target retention network. Recommendation module 1208 is configured to call the retention network based on the mapping parameters to continue recommending the content to be recommended.

[0211] The content recommendation apparatus provided in the embodiments of this specification obtains routing information during the recommendation process of the content to be recommended, divides multiple candidate networks into reserved networks and networks to be mapped, and determines the corresponding target reserved network and mapping parameters for the network to be mapped based on the mapping loss. Then, the reserved network is called based on the mapping parameters to continue the recommendation. This can reduce the number of candidate networks actually called, while using the target reserved network to approximately bear the output contribution of the network to be mapped, thereby reducing the inference overhead of the hybrid expert model in the content recommendation process and reducing the impact of directly omitting some candidate networks on the recommendation results.

[0212] The above is an illustrative scheme of a content recommendation device according to this embodiment. It should be noted that the technical solution of this content recommendation device and the technical solution of the content recommendation method described above belong to the same concept. For details not described in detail in the technical solution of the content recommendation device, please refer to the description of the technical solution of the content recommendation method described above.

[0213] Figure 13 A structural block diagram of a computing device according to one embodiment of this specification is shown. The components of the computing device 1300 include, but are not limited to, a memory 1310 and a processor 1320. The processor 1320 is connected to the memory 1310 via a bus 1330, and a database 1350 is used to store data.

[0214] The computing device 1300 also includes an access device 1340, which enables the computing device 1300 to communicate via one or more networks 1360. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1340 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Networks (WLAN) interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0215] In one embodiment of this specification, the above-described components of the computing device 1300 and Figure 13 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 13 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0216] The computing device 1300 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1300 can also be a mobile or stationary server.

[0217] The processor 1320 is used to execute computer programs / instructions, which, when executed by the processor, implement the steps of the above-mentioned task reasoning or content recommendation method.

[0218] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solution of the task reasoning or content recommendation method described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the task reasoning or content recommendation method described above.

[0219] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described task reasoning or content recommendation method.

[0220] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the task reasoning or content recommendation method described above. Details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the task reasoning or content recommendation method described above.

[0221] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described task reasoning or content recommendation method.

[0222] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solution of the task reasoning or content recommendation method described above. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the task reasoning or content recommendation method described above.

[0223] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0224] Computer instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in computer-readable media can be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0225] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0226] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0227] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A task reasoning method, characterized in that, include: During the reasoning process of the target task data, routing information of the target task data is obtained, wherein the routing information is used to indicate multiple candidate networks participating in the reasoning process in the hybrid expert model; Based on the routing information, the multiple candidate networks are divided into reserved networks and networks to be mapped; Based on the mapping loss of the network to be mapped to the retained network, the target retained network corresponding to the network to be mapped is determined from the retained network, and the mapping parameters of the network to be mapped to the target retained network are determined. Based on the mapping parameters, the retention network is invoked to continue inference on the target task data.

2. The method according to claim 1, characterized in that, The step of dividing multiple candidate networks into reserved networks and networks to be mapped based on the routing information includes: For the first candidate network, obtain the statistical index of the first candidate network, wherein the statistical index is obtained based on the output results of the first candidate network for multiple auxiliary data units, and the first candidate network is any one of the multiple candidate networks; Based on the routing weights of the first candidate network and the statistical indicators, the importance indicators of the first candidate network are determined. Based on the importance index of the first candidate network, the first candidate network is divided into the reserved network or the network to be mapped.

3. The method according to claim 2, characterized in that, The step of obtaining the statistical metrics of the first candidate network includes: The first candidate network is invoked to process the multiple auxiliary data units, thereby obtaining the output results corresponding to the multiple auxiliary data units respectively. The output results are statistically analyzed to obtain the statistical indicators of the first candidate network.

4. The method according to claim 1, characterized in that, The step of invoking the retention network based on the mapping parameters to continue inference on the target task data includes: For each of the target retention networks in the retention network, the target retention network is invoked, and the target task data is further inferred based on the routing weight of the target retention network and the corresponding mapping parameters; For each non-target reserved network in the reserved network, the non-target reserved network is invoked, and the target task data is further inferred based on the routing weight of the non-target reserved network.

5. The method according to claim 4, characterized in that, The step of invoking the target reserved network and, based on the routing weights of the target reserved network and the corresponding mapping parameters, continuing to infer the target task data includes: Based on the routing weight of the target network to be mapped and the mapping parameters from the target network to the target reserved network, the mapping routing weight from the target network to the target reserved network is determined, wherein the target network to be mapped is the network to be mapped corresponding to the target reserved network among the networks to be mapped; The target route weight of the target reserved network is determined based on the mapped route weight and the route weight of the target reserved network. The target retention network is invoked, and inference is continued on the target task data based on the target routing weight.

6. The method according to claim 1, characterized in that, The hybrid expert model comprises multiple original networks; Before determining the target retained network corresponding to the network to be mapped from the retained network based on the mapping loss of the network to be mapped to the retained network, and before determining the mapping parameters from the network to be mapped to the target retained network, the method further includes: Based on the auxiliary data units processed by the multiple original networks, a network set corresponding to the auxiliary data unit is determined, wherein the same network set includes the original networks that process the same auxiliary data unit. For a first original network and a second original network in a first network set, determine a first output result of the first original network and a second output result of the second original network, wherein the first network set is any one of the network sets, and the first original network and the second original network are any two original networks in the first network set; Based on the difference between the first output result and the second output result, determine the mapping loss of the first original network to the second original network and the mapping parameters of the first original network to the second original network.

7. The method according to any one of claims 1 to 6, characterized in that, Before dividing the multiple candidate networks into reserved networks and networks to be mapped based on the routing information, the method further includes: Based on the current inference stage of the target task data, a target partitioning strategy is determined, wherein different current inference stages correspond to different target partitioning strategies; The step of dividing multiple candidate networks into reserved networks and networks to be mapped based on the routing information includes: Based on the target partitioning strategy and the routing information, the multiple candidate networks are divided into the reserved network and the network to be mapped.

8. The method according to claim 7, wherein the current inference stage includes a pre-filling stage, the target task data includes multiple target data units, and the target partitioning strategy corresponding to the pre-filling stage is used to indicate partitioning based on the target data units respectively; The step of dividing multiple candidate networks into reserved networks and networks to be mapped based on the target partitioning strategy and the routing information includes: For a first candidate network, a first target data unit is determined from a plurality of target data units and routed to the first candidate network, wherein the first candidate network is any one of the plurality of candidate networks, and the first target data unit is any one of the target data units routed to the first candidate network; Based on the routing information, a first candidate network set corresponding to the first target data unit is determined from a plurality of candidate networks, wherein the first candidate network set includes the candidate networks to which the first target data unit is routed; Based on the importance index of multiple candidate networks in the first candidate network set, the first candidate network is divided into the reserved network or the network to be mapped.

9. The method according to claim 7, wherein the current inference stage includes a decoding stage, the target task data includes multiple target data unit sets, and the target partitioning strategy corresponding to the decoding stage is used to indicate partitioning based on the target data unit sets respectively; The step of dividing multiple candidate networks into reserved networks and networks to be mapped based on the target partitioning strategy and the routing information includes: For a first candidate network, a first target data unit set is determined from a plurality of target data unit sets, wherein the first candidate network is any one of the plurality of candidate networks, and the first target data unit set includes a plurality of target data units routed to the first candidate network; Based on the routing information, a second candidate network set corresponding to the first target data unit set is determined from a plurality of candidate networks, wherein the second candidate network set includes the candidate networks to which the first target data unit set is routed; Based on the importance indicators of multiple candidate networks in the second candidate network set, the first candidate network is divided into the reserved network or the network to be mapped.

10. The method according to any one of claims 1 to 6, characterized in that, The method further includes: If the candidate network partitioning fails, multiple candidate networks are invoked based on the routing information to continue reasoning on the target task data.

11. A content recommendation method, characterized in that, include: During the recommendation process of the content to be recommended, the routing information of the content to be recommended is obtained, wherein the routing information is used to indicate multiple candidate networks participating in the recommendation process in the hybrid expert model; Based on the routing information, the multiple candidate networks are divided into reserved networks and networks to be mapped; Based on the mapping loss of the network to be mapped to the retained network, the target retained network corresponding to the network to be mapped is determined from the retained network, and the mapping parameters of the network to be mapped to the target retained network are determined. Based on the mapping parameters, the retention network is invoked to continue recommending the content to be recommended.

12. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, It stores a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

14. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

Images