Model prediction method and model system suitable for multiple scenarios and multiple tasks

Abstract

The embodiments of this specification disclose a model prediction method and model system suitable for multiple scenarios and multiple tasks. The model system includes an input layer, a scenario layer, a task layer, and an output layer. After acquiring feature data, a scenario identifier, and a task identifier, the feature data and the scenario identifier are processed by the scenario layer to select one or more scenario-specific expert modules and one or more scenario-shared expert modules from multiple expert modules in the scenario layer to process the feature data respectively, and obtain a scene layer prediction result. The scene layer prediction result and the task identifier are processed by the task layer to select one or more task-specific expert modules and one or more task-shared expert modules from multiple expert modules in the task layer to process the scene layer prediction result respectively, and obtain a task layer prediction result. The task layer prediction result is processed by the output layer to obtain a model prediction result.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a model prediction method and a model system suitable for multiple scenarios and multiple tasks. Background Art

[0002] With the rapid development of artificial intelligence technology, machine learning models have been widely used in various daily production and life fields. At present, the more common practice is to train models for a single application scenario and a single prediction task, which makes the model training efficiency low. At the same time, due to the certain correlation between some scenarios and tasks, it is difficult to share data information under different scenarios / tasks for single-scenario single-task modeling. There are some studies on non-single-scenario single-task modeling, but these studies mainly focus on single-scenario multi-task learning (SSL&MTL) and multi-scenario single-task learning (MSL&STL). Single-scenario multi-task learning aims to train a model (framework) to achieve predictions under multiple optimization tasks (also called optimization objectives, referred to as tasks or objectives), but the applicable scenario is single and cannot be competent for multi-scenario prediction. Multi-scenario single-task learning aims to train a model (framework) to achieve predictions under multiple scenarios, but the optimization objective is single and cannot be competent for multi-objective (task) prediction.

[0003] In view of this, we hope to provide a multi-scenario and multi-task learning solution, that is, to train a model to achieve predictions in multiple scenarios and multiple tasks, further improve modeling efficiency, and at the same time share information to a greater extent to improve model prediction accuracy. Summary of the invention

[0004] One of the embodiments of this specification provides a model prediction method suitable for multiple scenes and multiple tasks. The method may include: obtaining feature data, a scene identifier and a task identifier, wherein the scene identifier is used to indicate a target scene selected from multiple scenes, and the task identifier is used to indicate a target task selected from multiple tasks; processing the feature data and the scene identifier through the scene layer to select one or more scene-specific expert modules and one or more scene-shared expert modules from multiple expert modules in the scene layer to process the feature data respectively, and obtain a scene layer prediction result; processing the scene layer prediction result and the task identifier through the task layer to select one or more task-specific expert modules and one or more task-shared expert modules from multiple expert modules in the task layer to process the scene layer prediction result respectively, and obtain a task layer prediction result; processing the task layer prediction result through the output layer to obtain a model prediction result.

[0005] One of the embodiments of this specification provides a model system suitable for multiple scenes and multiple tasks. The model system may include: an input layer, used to obtain feature data, a scene identifier and a task identifier, wherein the scene identifier is used to indicate a target scene selected from multiple scenes, and the task identifier is used to indicate a target task selected from multiple tasks; a scene layer, used to process the feature data and the scene identifier, to select one or more scene-specific expert modules and one or more scene-shared expert modules from multiple expert modules in the scene layer to process the feature data respectively, and obtain a scene layer prediction result; a task layer, used to process the scene layer prediction result and the task identifier, to select one or more task-specific expert modules and one or more task-shared expert modules from multiple expert modules in the task layer to process the scene layer prediction result respectively, and obtain a task layer prediction result; an output layer, used to process the task layer prediction result, and obtain a model prediction result.

[0006] One of the embodiments of this specification provides a model training method suitable for multiple scenes and multiple tasks. The method may include: obtaining feature data, scene identification, task identification and labels of training samples, wherein the scene identification is used to indicate a target scene selected from multiple scenes, and the task identification is used to indicate a target task selected from multiple tasks; inputting the feature data, scene identification and task identification of the training samples into a model system as described in any embodiment of this specification to obtain a model prediction result corresponding to the training sample; determining a first loss function value based on the model prediction result corresponding to the training sample and its label, wherein the first loss function value reflects the difference between the model prediction result and the label; determining a second loss function value based on the similarity between the scene weight vector and the scene index vector of each scene-specific expert module corresponding to the training sample, the similarity between the scene weight vector and the uniform distribution vector of each scene-shared expert module, the similarity between the task weight vector and the task index vector of each task-specific expert module, and the similarity between the task weight vector and the uniform distribution vector of each task-shared expert module, wherein the second loss function is negatively correlated with the sum of each similarity; adjusting the model parameters of the model system to simultaneously reduce the first loss function value and the second loss function value.

[0007] One of the embodiments of this specification provides a model prediction device suitable for multiple scenarios and multiple tasks, including a processor and a storage device, wherein the storage device is used to store instructions, wherein when the processor executes the instructions, the model prediction method described in any embodiment of this specification is implemented.

[0008] One of the embodiments of this specification provides a model training device suitable for multiple scenarios and multiple tasks, including a processor and a storage device, wherein the storage device is used to store instructions. When the processor executes the instructions, the model training method described in any embodiment of this specification is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0010] Figure 1 is an exemplary module diagram of a model system suitable for multiple scenarios and multiple tasks according to some embodiments of this specification;

[0011] Figure 2 is an exemplary flow chart of a model prediction method suitable for multiple scenarios and multiple tasks according to some embodiments of this specification;

[0012] Figure 3 is a data flow diagram of a single scene layer according to some embodiments of this specification;

[0013] Figure 4 It is a data flow diagram of a model system suitable for multiple scenarios and multiple tasks in a multi-level scenario according to some embodiments of this specification;

[0014] Figure 5 is a data flow diagram of a single task layer according to some embodiments of the specification;

[0015] Figure 6 is a data flow diagram of a single task layer according to other embodiments of the specification;

[0016] Figure 7 This is an exemplary flowchart of a model training method suitable for multiple scenarios and multiple tasks according to some embodiments of this specification. DETAILED DESCRIPTION

[0017] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0018] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0019] As shown in this specification, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0020] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0021] Machine learning can be divided into supervised learning and unsupervised learning, which are distinguished by whether the training data (samples) have labels. This specification relates to supervised learning, that is, samples have labels, or model inputs (such as input feature vectors) have corresponding labels. For example, for a classification model, the input can be an image of an animal such as a cat, dog, or sheep, and its corresponding label can be an animal species such as a cat, dog, or sheep.

[0022] Scenarios, also known as application scenarios, are used to describe the background of modeling or the specific field in which the model is used. In a relatively rough division, scenarios can be animal classification scenarios, user classification scenarios, item recommendation scenarios, etc. In a more fine-grained division, the animal classification scenario can be divided into cat classification, canine classification, or subdivided into image-based animal classification and call-based animal classification; the user search-based product recommendation scenario can be further subdivided based on the user's search entry or the user's intention. It should be understood that the definition of scenarios is relatively broad, flexible, and relative. In practical applications, scenarios can be divided according to the specific business needs or background of modeling. Tasks are often related to the ultimate purpose of model prediction. Similar to the definition of scenarios, tasks can also be divided according to corresponding business needs. Exemplarily, in a recommendation business based on user search, tasks can be set to include increasing user click-through rate, increasing user purchase rate, etc. In different scenarios (fixed tasks), the mapping relationship between input data and label space (label set of training samples) may be different. In different tasks (fixed scenarios), the mapping relationship between input data and label space may also be different. There are two feasible approaches to achieve prediction in multiple scenarios and multiple tasks: one is to train a model suitable for SSL&MTL for each scenario, that is, to train multiple models suitable for SSL&MTL; the other is to train a model suitable for MSL&STL for each task, that is, to train multiple models suitable for MSL&STL. However, training multiple models will bring huge maintenance costs to the iteration of the model. At the same time, it is difficult to obtain satisfactory training results using a single scenario data for training scenarios with small data volumes.

[0023] As an example only, in a search system of a certain platform, users can initiate a query request by entering a search term, and then obtain the goods (such as fund products) recommended by the system. The platform can be a network platform for selling goods, usually presented to users in the form of a website or application, and users can purchase goods or services in its user interface. Goods can be physical items, financial products, service products, etc. The platform can provide users with a diversified search portal. Taking financial products as an example, the search portal provided by the platform can include two channels: the main search page (a comprehensive homepage, denoted as HP) and the vertical search page (a highly targeted vertical page, denoted as VP). Each channel can be further divided into two domains: hot funds (denoted as BS) and fund products (denoted as RI). Therefore, there are a total of four subdivided scenarios, namely HP→BS, HP→RI, VP→BS, and VP→RI. Click-through rate (CTR) and conversion rate (Conversion Rate) are important optimization indicators of the search system. Therefore, for the search system, the two basic training tasks (goals) are to improve the click-through rate (referred to as click-through rate) and improve the conversion rate. Among them, the click-through rate can refer to the ratio of the number of times users click on recommended products to the number of times recommended products are displayed (exposed), and the conversion rate can refer to the ratio of the number of times users purchase recommended products (i.e. conversion) after clicking to the number of times users click on recommended products. It can be understood that the click / conversion behavior of users in different scenarios has different rules, and the basis for determining labels in different tasks is also different (for example, the label of the CTR task is determined based on whether the user clicks, and the label of the CVR task is determined based on whether the user conversion behavior occurs).

[0024] In view of this, the embodiments of this specification provide a model system and a model prediction and training method suitable for multiple scenarios and multiple tasks (MSL&MTL).

[0025] Figure 1 is an exemplary module diagram of a model system suitable for multiple scenarios and multiple tasks according to some embodiments of this specification. Figure 1 As shown, the system 100 may include an input layer 110 , a scene layer 120 , a task layer 130 , and an output layer 140 .

[0026] The input layer 110 may be used to obtain feature data, a scene identifier and a task identifier, wherein the scene identifier may be used to indicate a target scene selected from a plurality of scenes, and the task identifier may be used to indicate a target task selected from a plurality of tasks.

[0027] The scene layer 120 may include multiple expert modules (not shown). The scene layer 120 may be used to process the feature data and the scene identifier, to select one or more dedicated expert modules and one or more shared expert modules from the multiple expert modules of the scene layer 120 to process the feature data respectively, and obtain a scene layer prediction result.

[0028] In some embodiments, the scene layer 120 may further include a gating module 122 and a selection module 124 .

[0029] The gate control module 122 may be used to process the feature data and the scene identifier to obtain a scene weight vector for each of the multiple expert modules of the scene layer 120. The dimension of the scene weight vector is related to the number of the multiple scenes.

[0030] The selection module 124 is used to determine, for each of the multiple expert modules of the scene layer 120 , whether the expert module is a scene-specific expert module or a scene-shared expert module based on the scene weight vector of the expert module.

[0031] In some embodiments, the gating module 122 may also be used to process the feature data and the scene identifier to obtain an expert weight vector of the target scene. The dimension of the expert weight vector is related to the number of the multiple expert modules of the scene layer 120. The scene layer 120 may also include a weight adjustment module 126, which is used to retain the weights corresponding to the scene-specific expert modules and the scene-shared expert modules in the expert weight vector to obtain an updated expert weight vector.

[0032] In some embodiments, the scene layer 120 may further include a prediction module 128, which is used to: obtain multiple scene layer prediction components obtained by each expert module processing the feature data; and perform weighted summation of the multiple scene layer prediction components based on the updated expert weight vector to obtain the scene layer prediction result.

[0033] The task layer 130 may also include multiple expert modules (not shown). The task layer 130 may be used to process the scene layer prediction result and the task identifier, so as to select one or more task-specific expert modules and one or more task-shared expert modules from the multiple expert modules in the task layer 130 to process the scene layer prediction result respectively, and obtain the task layer prediction result.

[0034] In some embodiments, the task layer 130 may further include a gating module 132 and a selection module 134 .

[0035] The gate control module 132 may be used to process the scene layer prediction result and the task identifier to obtain a task weight vector for each of the multiple expert modules of the task layer 130. The dimension of the task weight vector is related to the number of the multiple tasks.

[0036] The selection module 134 may be configured to determine, for each of the plurality of expert modules of the task layer 130 , whether the expert module is a task-specific expert module or a task-shared expert module based on the task weight vector of the expert module.

[0037] In some embodiments, the gating module 132 may also be used to process the scene layer prediction results and the task identifier to obtain an expert weight vector for the target task. The dimension of the expert weight vector is related to the number of multiple expert modules in the task layer 130. The task layer 130 may also include a weight adjustment module 136 for retaining the weights corresponding to the task-specific expert modules and the task-shared expert modules in the expert weight vector to obtain an updated expert weight vector.

[0038] In some embodiments, the task layer 130 may further include a prediction module 138, which is used to: obtain multiple task prediction components obtained by each expert module processing the scene layer prediction result; and perform weighted summation of the multiple task prediction components based on the updated expert weight vector to obtain the task layer prediction result.

[0039] The output layer 140 can be used to process the task layer prediction results to obtain model prediction results.

[0040] Optionally, in some embodiments, the model system 100 may further include an acquisition module (not shown) for acquiring a search term entered by a user in a search portal of the platform; determining a product to be recommended based on the search term; determining feature data based on user characteristics of the user, features of the product to be recommended, and features of the user's historical query behavior; determining a scene identifier by taking the search portal as a target scene; determining a target task as one or more of increasing click-through rate and increasing conversion rate to determine a task identifier; and sending the feature data, the scene identifier, and the task identifier to the input layer 110.

[0041] For more details about the model system 100 and its modules, please refer to Figure 2 and its related description.

[0042] It should be understood that Figure 1The system and its modules shown can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented by hardware, software or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or a dedicated design hardware. Those skilled in the art will understand that the above methods and systems can be implemented using computer executable instructions and / or included in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the system and its modules of this specification. Not only can the hardware circuits such as ultra-large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc. be implemented, it can also be implemented with software such as executed by various types of processors, and it can also be implemented by a combination of the above hardware circuits and software (for example, firmware).

[0043] It should be noted that the above description of the system and its modules is only for convenience of description and does not limit this specification to the scope of the embodiments. It is understandable that for those skilled in the art, after understanding the principle of the system, it is possible to arbitrarily combine the modules or form a subsystem connected to other modules without deviating from this principle. For example, in some embodiments, the weight update module 126 and the prediction module 128 can be combined into one module. Such variations are within the scope of protection of this specification.

[0044] Figure 2 is an exemplary flow chart of a model prediction method suitable for multiple scenarios and multiple tasks according to some embodiments of this specification. In some embodiments, process 200 can be Figure 1 The system 100 shown in FIG. Figure 2 As shown, process 200 may include the following steps.

[0045] Step 210 , acquiring feature data, scene identifiers, and task identifiers. In some embodiments, step 210 may be performed by an acquisition module and the input layer 110 .

[0046] Feature data can reflect various features related to the prediction object, and can be set according to business needs in actual applications. Taking the search system as an example, feature data can reflect one or more of the following features: user features, features of the recommended products, and query behavior features. Among them, user features can include the user's gender, age, preferences, etc., features of the recommended products can include brand, price, origin, etc., and query behavior can include current query input, recent query frequency, and products that have been queried, clicked, purchased, or browsed in the historical time period, etc.

[0047] In some embodiments, the acquisition module can obtain the search terms entered by the user in a search portal of the platform, and determine the recommended products based on the search terms. The recommended products may be products or services corresponding to or related to the search terms. The number of recommended products may be one or more. When there are multiple recommended products, the multiple recommended products may be processed separately. In some embodiments, the acquisition module can further obtain the user's gender, age, occupation, preferences and other characteristics, the brand, price, origin and other characteristics of the recommended products, and the user's historical query behavior, and then obtain the feature data. The feature data may be in a coded form so that the input layer 110 can process it directly.

[0048] For category features (such as gender, brand), the category features of a given instance (prediction object, such as the user and / or the product to be recommended) can be converted into a one-hot vector. The dimension of the one-hot vector can be equal to the number of all categories of the category feature, and each dimension corresponds to each category one-to-one, where there is only one non-zero (usually 1) vector element and the rest of the vector elements are zero, to indicate that the given instance belongs to the target category corresponding to the non-zero element. For example, assuming there are n brands, an n-dimensional one-hot vector (denoted as v) can be used to represent the brand features. Let the t-th dimension element of the one-hot vector v (denoted as v t ) The corresponding brand is denoted as b t If the given instance belongs to brand b i , then the i-th dimension element of the unique hot vector v representing the brand feature of the instance can be set to 1, and the remaining elements can be set to 0, that is, v i =1 and v j =0(j≠i). For numerical features, feature binning can be performed, that is, converting them into categorical features, and then converting categorical features into one-hot vectors.

[0049] The scene identifier can be used to indicate the target scene selected from multiple scenes, and the task identifier can be used to indicate the target task selected from multiple tasks. Specifically, multiple scene identifiers corresponding to multiple scenes can be set one by one. For any instance, the target scene to which the instance belongs can be determined first, and then the scene identifier corresponding to the target scene can be determined. Similarly, multiple task identifiers corresponding to multiple tasks can be set one by one, and for each target task, the task identifier corresponding to the target task can be determined. It can be understood that the target scene to which a given instance belongs is unique (the scene is exclusive), that is, there is only one target scene for a given instance. The task is not exclusive, and according to actual needs, the target task can be one or more. Continuing with the previous example, the acquisition module can use the search entry selected by the user in the platform as the target scene, and then obtain the scene identifier. The acquisition module can also use one or more of improving conversion rate and improving click-through rate as the target task according to business needs, and then obtain the task identifier.

[0050] Since multiple scenes / tasks can be considered as multiple categories, in some embodiments, the one-hot vector can serve as a scene identifier. Referring to the above content, taking scene identifier as an example, assuming that there are m scenes, an m-dimensional one-hot vector (denoted as v) can be used to indicate the target scene to which a given instance belongs. It is better to denote the t-th dimension element of the one-hot vector v (denoted as v t ) The corresponding scene is recorded as s t If a given instance belongs to scene s i , then the i-th element of the scene identification vector v of the instance can be set to 1, and the remaining elements can be set to 0, that is, v i =1 and v j =0(j≠i). Similarly, the task identifier can be obtained.

[0051] In some embodiments, the input layer 110 may perform vector embedding processing on the feature data, scene identifier, and task identifier in the form of one-hot vectors to obtain the feature data, scene identifier, and task identifier in the form of embedded vectors and send them to the scene layer 120 for processing. For a given instance, multiple embedded vectors representing different features may be concatenated to obtain feature data in the form of embedded vectors for the instance and send them to the scene layer 120 for processing.

[0052] Step 220, processing the feature data and the scene identifier through the scene layer 120 to select one or more scene-specific expert modules and one or more scene-shared expert modules from multiple expert modules of the scene layer 120 to process the feature data respectively, to obtain a scene layer prediction result.

[0053] In some embodiments, the expert module can process feature data to obtain intermediate prediction results. In the model system used in this specification, the scene layer can have multiple expert modules. For different scenes, different experts can be selected from multiple expert modules to participate in the prediction of each scene, thereby adapting to multi-scene prediction. The expert module can be implemented based on a multi-layer perceptron (Multi-Layer Perceptron, MLP) or a deep neural network (Deep Neural Networks, DNN). In order to simplify the description, the expert module can be referred to as an expert (Expert).

[0054] At the scene level, if the relevance of an expert to a specific scene (referred to as scene j) is higher than that to other scenes, then this expert can be called a dedicated expert for scene j (i.e., the scene-specific expert module); if the difference between the relevance of an expert to all scenes is very small, then this expert is called a shared expert (for all scenes) (i.e., the scene-shared expert module). It can be understood that both dedicated experts and shared experts for scene j can be used for predictions under scene j. After expert selection, for instances belonging to scene j, the scene-level prediction results of the instance can be obtained based only on the output of dedicated experts for scene j and the output of scene-shared experts, that is, the output of unselected experts can be shielded. In addition, any expert can be both a scene-specific expert and a scene-shared expert, that is, scene-specific experts and scene-shared experts can have an intersection. In view of this, one or more scene-specific experts and one or more shared experts imply the union of the two.

[0055] It is worth noting that the scene layers are expandable (stackable). In the stacked scene layers: the first scene layer takes the feature data and the scene identifier as input, and the subsequent scene layers take the scene layer prediction results output by the previous scene layer and the scene identifier as input; the experts of the first scene layer take the feature data as input, and the subsequent scene layers take the scene layer prediction results output by the previous scene layer as input; after layer-by-layer expert selection, the scene layer prediction results output by the last scene layer can be sent to the task layer for processing. It can be understood that if the task layer is also expanded, the scene layer prediction results output by the last scene layer can be sent to the first task layer for processing. Stacking can be used to expand the parameter scale of the model system and increase the depth of the model system, which is beneficial to improve the prediction accuracy for prediction scenarios with more complex feature data. In some embodiments, stacking can also be used to achieve the correspondence between multiple scene layers and multiple levels within the scene, refine the processing granularity, and improve prediction accuracy. For related content, please refer to Figure 4 The following mainly takes a single scene layer (non-stacked scene layer) as an example to introduce the internal processing flow of the scene layer. For stacked scene layers, the internal processing flow of each scene layer is similar to that of a single scene layer.

[0056] In some embodiments, the gating module 122 of the scene layer 120 can process the expert input and the scene identification to obtain the scene weight vector of each of the multiple experts in the scene layer 120. The dimension of the scene weight vector can be consistent with the number of the multiple scenes. For example, when the multiple scenes are m scenes, the scene weight vector can be an m-dimensional vector, that is, the m elements of the scene weight vector correspond to the m scenes one by one (it may be considered that the j-th dimension element corresponds to scene j), and the j-th dimension element of the scene weight vector of each expert can be the correlation score between the expert and scene j. Then, for each of the multiple experts in the scene layer 120 (denoted as expert E k ), the selection module 120 of the scene layer 120 can select k The scene weight vector determines the expert E k Whether it is a scene-specific expert or a scene-shared expert. It is reasonable to assume that: if expert E k If the j-th dimension vector element of the scene weight vector is larger than other vector elements, it is considered that the expert E k Contains the proprietary information of scene j, and the corresponding expert E k can be classified as a dedicated expert in scenario j; if expert E k The values ​​of the vector elements in the scene weight vector are close, then it is considered that the expert E k Contains shared information of all scenarios, corresponding to the expert E k can be classified as a shared expert for all scenarios. In short, the sparsity of each expert's scenario weight vector reflects the probability of the expert being classified as a scenario-specific expert and a scenario-shared expert.

[0057] In some embodiments, the scene weight vectors of each expert can be arranged as row vectors to obtain a matrix G. Correspondingly, the column vectors of the G matrix can be regarded as the correlation scores of each expert and the scene for the same scene; that is, the scene weight vector of the expert can be recorded as G[k,:], and the column vector of the G matrix can be called the expert weight vector of a certain scene. In some embodiments, the gating module 122 of the scene layer 120 can use the parameter matrix corresponding to different scenes to process the feature data and the scene identification respectively to obtain the expert weight vector of each scene. It is not difficult to understand that this process can obtain the scene weight vector of each expert at the same time.

[0058] Taking scene j as an example, the processing logic of the gating module 122 of the scene layer 120 can be expressed as:

[0059] g j =S j [x,s]+η j (1)

[0060] Among them, g j represents the expert weight vector of scene j, and the dimension of the expert weight vector may be consistent with the number of experts in the scene layer 120. For example, when there are n experts in the scene layer 120, the expert weight vector may be an n-dimensional vector, that is, the n elements of the expert weight vector correspond to the n experts one by one (it may be considered that the k-th dimension element corresponds to the expert E k ), expert weight vector g j The k-th dimension element is the expert E k S is the relevance score to scene j. j is the parameter matrix corresponding to scene j, which can be obtained through learning, [,] represents vector concatenation, x represents expert input, s represents scene identification, S j [x,s] represents the matrix S j The product of and vector [x,s]. η j Represents a noise (disturbance) vector, such as a Gaussian noise vector. By adding disturbances, the selection space can be expanded to avoid some experts always being selected, which leads to a decrease in model reliability.

[0061] In some embodiments, the weight adjustment module 126 of the scene layer 120 can update the expert weight vector of the target scene according to the expert selection result. Specifically, assuming that the target scene is scene j, the weight adjustment module can retain the weights corresponding to the scene-specific experts and scene-shared experts in the expert weight vector of scene j to obtain an updated expert weight vector. It can be understood that the scene-specific experts here refer to the proprietary experts of the target scene. In addition, retaining can refer to maintaining the original weight (vector element) unchanged, and not retaining can refer to setting the weight (vector element) to 0.

[0062] In some embodiments, the scene layer prediction vector (of the current scene layer) can be obtained through the updated expert weight vector. Specifically, the prediction module 128 of the scene layer 120 can first obtain multiple scene layer prediction components obtained by processing expert input (feature data or scene layer prediction results of the previous scene layer) by each expert (of the current scene layer), and then weighted sum the multiple scene prediction components based on the updated expert weight vector to obtain the scene layer prediction result (of the current scene layer). It can be understood that for each specific scene (assuming it is scene j and the corresponding scene identifier s j ) can process the feature data x and scene identifier s of the instance through the scene layer (denoted as ScenarioLayer()) j , and obtain the scene layer prediction result of this instance (denoted as z j ), that is, z j =ScenarioLayer(x,s j ). When the scene layer 120 is expanded, the first scene layer is composed of feature data x and scene identifier sj As input, the subsequent scene layer uses the scene layer prediction result z output by the previous scene layer j (new expert input x) and the scene identifier s j For input.

[0063] In some embodiments, the selection module 124 of the scene layer 120 can determine the similarity between the scene weight vector of each expert and the scene index vector, and then select one or more scene-specific experts from multiple experts in the scene layer 120 based on the similarities. Wherein, the dimension of the scene index vector is equal to the scene weight vector, that is, when the multiple scenes are m scenes, the scene index vector is an m-dimensional vector, and the m dimensions (elements) correspond to the m scenes one by one. The scene index vector can be a one-hot vector, in which the element corresponding to the target scene is non-0 and the remaining elements are 0. In some embodiments, the non-0 elements in the scene index vector can be 1, and accordingly, the scene weight vector of each expert can be normalized (such as softmax processing), and the similarity between the normalized scene weight vector of each expert and the scene index vector is determined. Referring to the assumptions mentioned above, the higher the similarity between the scene weight vector of the expert and the scene index vector, the greater the probability that the expert is classified as a specific expert of the target scene (i.e., selected as a scene-specific expert). Therefore, experts associated with high similarity can be determined as scene-specific experts. For example, experts associated with similarity rankings in the top K1 positions can be determined as scene-specific experts, where K1 is a preset value.

[0064] In some embodiments, the selection module 124 of the scene layer 120 can determine the similarity between the scene weight vector of each expert and the uniform distribution vector, and then select one or more scene sharing experts from multiple experts in the scene layer 120 based on the similarities. Wherein, the dimension of the uniform distribution vector is equal to the scene weight vector, that is, when the multiple scenes are m scenes, the uniform distribution vector is an m-dimensional vector. The elements in the uniform distribution vector can be the same, for example, the elements in the m-dimensional uniform distribution vector are all equal to 1 / m (that is, the sum of all elements is equal to 1). In some embodiments, when the sum of all elements of the uniform distribution vector is equal to 1, the scene weight vector of each expert can be normalized (such as softmax processing), and the similarity between the normalized scene weight vector of each expert and the uniform distribution vector is determined. Referring to the assumptions mentioned above, the higher the similarity between the scene weight vector of the expert and the uniform distribution vector, the greater the probability that the expert is classified as a shared expert for all scenes (that is, selected as a scene sharing expert). Therefore, experts associated with high similarity may be determined as scene sharing experts. For example, experts associated with similarity rankings in the top K2 positions may be determined as scene sharing experts, where K2 is a preset value.

[0065] In this description, the two similarities mentioned can be obtained based on any feasible similarity algorithm, for example, cosine similarity, Euclidean distance, KL divergence, etc. Taking KL divergence as an example, the two similarities mentioned can be calculated as follows:

[0066]

[0067]

[0068] in: Indicates expert E k The similarity between the scene weight vector and the scene index vector, Indicates expert E k The similarity between the scene weight vector and the uniform distribution vector; KL() represents the KL divergence, p j represents the scene index vector whose element corresponding to scene j is 1, q represents the uniform distribution vector whose sum of all elements is equal to 1, Indicates expert E k Normalized scene weight vector.

[0069] Figure 3 The single scene layer is a data flow diagram of a single scene layer shown in some embodiments of the specification. The single scene layer of the example is a scene layer in a model system suitable for dual scenes and dual tasks.

[0070] Figure 3 Middle: There are n experts in the scene layer, namely E1~E n ; Scene IDs A Indicates that the target scene is scene A. Figure 3 The gate control module can process the expert input and scene identification s respectively through the parameter matrix corresponding to scene A and the parameter matrix corresponding to scene B A Get the expert weight vector g of scene A A and the expert weight vector g of scenario B B The expert weight vector g A and the expert weight vector g B (as column vectors) to form a G matrix, and then the row vector of the G matrix, i.e., the scenario weight vector of each expert (not shown), can be obtained as the basis for expert selection. After the expert selection by the selection module, the dedicated expert E1 of scenario A and the shared expert E3 of scenario A and scenario B are determined. The selected experts (scenario-specific experts / scenario-shared experts) are indicated by solid-line boxes, and the unselected experts such as E2 and E n The dotted box indicates that the target scene is scene A. The weight adjustment module can adjust the expert weight vector g of scene A according to the expert selection result. A Update, that is, keep g AThe weight g corresponding to the scene-specific expert E1 A [1] and the weight g corresponding to the scene sharing expert E3 A [3], get the updated expert weight vector The prediction module can obtain multiple scene-level prediction components (i.e., expert outputs) obtained by each expert processing the expert input, and based on the updated expert weight vector The prediction results of the scene layer are obtained by weighted summing of the prediction components of each scene layer.

[0071] In some embodiments, the scenario can be represented by a multi-level structure, where each level of the multi-level structure has one or more scenario categories. For example, the previous article introduces a two-level scenario: the first level has two scenario categories, namely the main search page (HP) and the vertical search page (VP) channels; the second level has two scenario categories, namely the hot funds (BS) and fund products (RI) domains. That is, there are a total of 4 possibilities for this two-level scenario, namely four sub-scenarios, namely HP→BS, HP→RI, VP→BS, and VP→RI.

[0072] For multi-level scenes, the scene identifier may include sub-identifiers corresponding to each level of the multi-level structure representing the scene, and the sub-identifier corresponding to each level indicates the target scene category at that level. Continuing with the previous example, the scene identifier of HP→BS may include two sub-identifiers, the first sub-identifier is used to indicate whether the first-level scene category of the current scene is HP or VP, and the second sub-identifier is used to indicate whether the second-level scene category of the current scene is BS or RI. In this case, multiple scene layers may be stacked, which include scene layers corresponding to each level, and each level may correspond to one or more scene layers. When there are multiple scene layers corresponding to a certain level, these scene layers may process the same sub-identifier.

[0073] Figure 4 The data flow of the model system suitable for multi-scenario and multi-task in multi-level scenarios is demonstrated.

[0074] For ease of understanding, Figure 4 In the model system shown, the multiple levels (L s Each level of the structure corresponds to only one scene layer, and the i-th scene layer (scene layer i) corresponds to the i-th level, where 1≤i≤L s The input layer is used to store feature data, scene identifiers (including the 1st to L s The sub-identifier corresponding to the level is recorded as ) and the task identifier are processed into an embedded vector form, the feature data and the sub-identifier s1 corresponding to the first level can be processed through the first scene layer to obtain the scene layer prediction result of the first scene layer. Next, for the i-th scene layer (2≤i≤L s), the scene prediction results of the previous scene layer (i-1 scene layer) and the sub-identifier corresponding to the i-th level can be processed through this scene layer (i-th scene layer) One or more scene-specific expert modules and one or more scene-shared expert modules are selected from multiple experts of the scene layer (i-th scene layer) to process the scene layer prediction results of the previous scene layer (i-1th scene layer) respectively, and the scene layer prediction results of the current scene layer (i-th scene layer) are obtained. s The scene layer prediction results of each scene layer are sent to the task layer for processing. Figure 4 As shown, the subsequent scene layer (the i-th scene layer) processed together with the scene layer prediction results of the previous scene layer (the i-1th scene layer) and the sub-identifier corresponding to the i-th level may also include the sub-identifiers corresponding to the 1st to the i-1th levels.

[0075] because In most cases (such as n i >2) is established, by multi-level (L s Each level of the scene sets the corresponding scene layer (the cost is determined by ) is less expensive than setting up a single scene layer (the cost is determined by The size of the model determines the size of the model), and the same number of ( Modeling of subdivided scenarios.

[0076] Step 230, processing the scene layer prediction result and the task identifier through the task layer 130, so as to select one or more task-specific expert modules and one or more task-shared expert modules from multiple expert modules in the task layer 130 to process the scene layer prediction result respectively, and obtain the task layer prediction result.

[0077] Similar to the scenario layer, the task layer can also include multiple experts. At the task layer, if the relevance of an expert to a specific task (denoted as task j) is higher than that to other tasks, then this expert can be called a dedicated expert for task j (i.e., the task-specific expert module); if the difference between the relevance of an expert to all tasks is very small, then this expert is called a shared expert (of all tasks) (i.e., the task-shared expert module). It can be understood that both dedicated experts and shared experts for task j can be used for predictions under task j. After expert selection, the task-layer prediction results under the task can be obtained based only on the output of dedicated experts for task j and the output of task-shared experts, that is, the output of unselected experts can be shielded. In addition, any expert can be both a task-specific expert and a task-shared expert, that is, there can be an intersection between task-specific experts and task-shared experts. In view of this, one or more task-specific experts and one or more shared experts imply the meaning of taking the union of the two.

[0078] Similar to the scene layer, the task layer is also expandable (stackable). Figure 4 In the stacked task layers: the first task layer (task layer 1) takes the scene layer prediction result and the task identifier as input, and the subsequent task layers take the task layer prediction result output by the previous task layer and the task identifier as input; the experts of the first task layer take the scene layer prediction result as input, and the subsequent task layers take the task layer prediction result output by the previous task layer as input; after layer-by-layer expert selection, the last task layer (task layer L t )The task layer prediction results output by the task layer can be sent to the output layer for processing.

[0079] In some embodiments, there may be multiple target tasks, and accordingly, there may be multiple task identifiers that correspond one to one with multiple target tasks. For example, each of the multiple tasks can be used as a target task. For each of the multiple target tasks, the task layer 130 can process the expert input (scene layer prediction result or the task layer prediction result of the previous task layer) and the task identifier corresponding to the target task to select one or more task-specific experts and one or more task-sharing experts from the multiple experts in the task layer 130 to process the scene layer prediction results respectively, and obtain the task layer prediction results corresponding to the target task. It can be understood that the task-specific expert here refers to the exclusive expert of the target task.

[0080] In some embodiments, when there are multiple target tasks and multiple task layers 130, for one of the target tasks, the following can be done: the first task layer 130 processes the scene layer prediction result and the task identifier corresponding to the target task, so as to select one or more task-specific experts and one or more task-sharing experts from multiple experts in the task layer (first task layer) to process the scene layer prediction result respectively, and obtain the task layer prediction result corresponding to the target task of the current task layer (first task layer); the subsequent task layer 130 (which may be recorded as the i-th task layer) processes the task layer prediction result corresponding to the target task of the previous task layer (i-1th task layer) and the task identifier corresponding to the target task, so as to select one or more task-specific experts and one or more task-sharing experts from multiple expert modules in the task layer (i-th task layer) to process the task layer prediction result corresponding to the target task of the previous task layer (i-1th task layer), and obtain the task layer prediction result corresponding to the target task of the current task layer (i-th task layer). And so on, until the last task layer outputs the task layer prediction result corresponding to the target task, and then outputs it to the output layer. Similarly, the task-specific experts here refer to the specific experts of the target task. For other target tasks, they are processed layer by layer according to the above process until the last task layer outputs the task-level prediction results corresponding to the other target tasks, and then outputs them to the output layer. In other words, each task layer can select different experts from the same group of experts for different target tasks to process the task-level prediction results of the previous task layer corresponding to different target tasks.

[0081] The following takes a single target task as an example to introduce the data processing flow of a single task layer. In some embodiments, the gating module 132 of the task layer 130 can process the expert input (the scene layer prediction result or the task layer prediction result output by the previous task layer) and the task identifier to obtain a task weight vector for each of the multiple experts in the task layer 130. Among them, the dimension of the task weight vector can be consistent with the number of the multiple tasks. For example, when the multiple tasks are m (only as an example, it does not mean that the number of multiple tasks must be the same as the number of multiple scenes) tasks, the task weight vector can be an m-dimensional vector, that is, the m elements of the task weight vector correspond one-to-one to the m tasks (it may be considered that the j-th dimension element corresponds to task j), and the j-th dimension element of each expert's task weight vector can be the correlation score between the expert and task j. Furthermore, for each of the multiple experts in the task layer 130 (denoted as expert E i ), the selection module 134 of the task layer 130 can select i The task weight vector determines the expert E i Whether it is a task-specific expert or a task-shared expert. It is reasonable to assume that: if E iIf the j-th dimension vector element of the task weight vector is larger than other vector elements, it is considered that expert E i Included tasks j Proprietary information of Expert E i can be classified as a dedicated expert on task j; if E i The values ​​of the elements in the task weight vector of are close, then it is considered that the expert E i Contains shared information for all tasks, corresponding to the expert E i can be classified as a shared expert for all tasks. In short, the sparsity of each expert's task weight vector reflects the probability of the expert being classified as a task-specific expert and a task-shared expert.

[0082] In some embodiments, the task weight vectors of each expert can be arranged as row vectors to obtain a matrix G. Correspondingly, the column vectors of the G matrix can be regarded as the correlation scores of each expert with the same task; that is, the task weight vector of the expert can be recorded as G[i,:], and the column vector of the G matrix can be called the expert weight vector of a certain task. In some embodiments, the gating module 132 of the task layer 130 can use the parameter matrices corresponding to different tasks to process the scene layer prediction results and task identifiers respectively to obtain the expert weight vectors of each task. It is not difficult to understand that this process can obtain the task weight vectors of each expert at the same time.

[0083] Taking task k as an example, the processing logic of the gate control module 132 of the task layer 130 can be expressed as:

[0084] g k =T k [x,s,t k ]+η k (4)

[0085] Among them, g k represents the expert weight vector of task k, and the dimension of the expert weight vector may be consistent with the number of experts in the task layer 130. For example, when there are n experts in the task layer 130, the expert weight vector may be an n-dimensional vector, that is, the n elements of the expert weight vector correspond to the n experts one by one (it may be considered that the i-th dimension element corresponds to the expert E i ), expert weight vector g k The i-th element of the expert E i The relevance score to task k. k is the parameter matrix corresponding to task k, which can be obtained through learning, [,] represents vector concatenation, x represents expert input, s represents scene identification, t k is the task identifier indicating task k, T k [x,s,t k ] represents the matrix T kand the vector [x,s,t k ] is the product of η. k Represents a noise (disturbance) vector, such as a Gaussian noise vector. By adding disturbances, it is possible to avoid some experts being always selected, which leads to a decrease in model reliability.

[0086] In some embodiments, the weight adjustment module 126 of the task layer 130 can update the expert weight vector of the target task according to the expert selection result. Specifically, assuming that the target task is task k, the weight adjustment module can retain the weights corresponding to the task-specific experts and task-shared experts in the expert weight vector of task k to obtain an updated expert weight vector. It can be understood that the task-specific experts here refer to the dedicated experts of the target task. In addition, retaining can mean maintaining the original weight unchanged, and not retaining can mean resetting the weight to 0.

[0087] In some embodiments, the task layer prediction vector (of the current task layer) can be obtained through the updated expert weight vector. Specifically, the prediction module 138 of the task layer 130 can first obtain the multiple task layer prediction components obtained by each expert (of the current task layer) processing the expert input (scene layer prediction result or the task layer prediction result of the previous task layer), and then weighted sum the multiple task prediction components based on the updated expert weight vector to obtain the task layer prediction result (of the current task layer). It can be understood that the task layer prediction result here refers to the task layer prediction result corresponding to the target task, and the output layer processes this task layer prediction result to obtain the model prediction result under the target task. It can be understood that for an instance from scene j, the scene layer prediction result (z j ) of the instance can be processed by the task layer (denoted as TaskLayer()) j ) and the task identifier t corresponding to task k k , and obtain the task layer prediction result of the instance under task k (denoted as z k ), that is, z k =TaskLayer(z j ,t k ). When the task layer 130 is expanded, the first task layer predicts the result z at the scene layer. j And the task identifier t k The subsequent task layer is the task layer prediction result z output by the previous task layer. k (new expert input) and task identifier t k For input.

[0088] In some embodiments, when the gating module 132 of the task layer 130 processes the expert input (the scene layer prediction result or the task layer prediction result of the previous task layer) and the task identifier to obtain the task weight vector of each of the multiple experts of the task layer 130 or the expert weight vector of the target task, the scene identifier (for example, Figure 4 Include sub-id For example, the expert weight vector (g) of the target task (task k) can be calculated according to formula (4): k ), or it can be calculated according to the following variation of formula (4):

[0089] g k =T k [x,t k ]+η k (5)

[0090] That is, the scene identifier s is not used to calculate the expert weight vector g k Required input.

[0091] In some embodiments, the selection module 134 of the task layer 130 can determine the similarity between the task weight vector and the task index vector of each expert, and then select one or more task-specific experts from multiple experts in the task layer 130 based on the similarities. Wherein, the dimension of the task index vector is equal to the task weight vector, that is, when the multiple tasks are m tasks, the task index vector is an m-dimensional vector, and the m dimensions (elements) correspond to the m tasks one by one. The task index vector can be a one-hot vector, in which the element corresponding to the target task is non-0, and the remaining elements are 0. In some embodiments, the non-0 elements in the task index vector can be 1, and accordingly, the task weight vector of each expert can be normalized (such as softmax processing), and the similarity between the normalized task weight vector and the task index vector of each expert is determined. Referring to the assumptions mentioned above, the higher the similarity between the expert's task weight vector and the task index vector, the greater the probability that the expert is classified as a specific expert of the target task (i.e., selected as a task-specific expert). Therefore, experts associated with high similarity can be determined as task-specific experts. For example, experts associated with similarity rankings in the top K1 positions can be determined as task-specific experts, where K1 is a preset value.

[0092] In some embodiments, the selection module 134 of the task layer 130 can determine the similarity between the task weight vector of each expert and the uniform distribution vector, and then select one or more task sharing experts from multiple experts in the task layer 130 based on the similarities. Wherein, the dimension of the uniform distribution vector is equal to the task weight vector, that is, when the multiple tasks are m tasks, the uniform distribution vector is an m-dimensional vector. The elements in the uniform distribution vector can be the same, for example, the elements in the m-dimensional uniform distribution vector are all equal to 1 / m (that is, the sum of all elements is equal to 1). In some embodiments, when the sum of all elements of the uniform distribution vector is equal to 1, the task weight vector of each expert can be normalized (such as softmax processing), and the similarity between the normalized task weight vector of each expert and the uniform distribution vector is determined. Referring to the assumptions mentioned above, the higher the similarity between the expert's task weight vector and the uniform distribution vector, the greater the probability that the expert is classified as a shared expert for all tasks (that is, selected as a task sharing expert). Therefore, experts with high similarity associations can be determined as task sharing experts, for example, experts with similarity associations ranked in the top K2 positions in similarity can be determined as task sharing experts, where K2 is a preset value. The calculation method of similarity can be found in the detailed description elsewhere in the article, and will not be repeated here.

[0093] Figure 5 It is a data flow diagram of a single task layer according to some embodiments of the specification. It can be understood that in the case of stacking task layers, Figure 5 The scenario layer shown is the first task layer.

[0094] Figure 5 Middle: There are n experts in the task layer, namely E1~E n ; Task ID t α Indicates that the target task is task α, and the task identifier is t β Indicates that the target task also includes task β. Figure 5 The gating module can firstly target the expert input and the task identifier t α Get the expert weight vector g for task α α and the expert weight vector g for task β β , where: the scene layer prediction results and task identifier t are processed by the parameter matrix corresponding to task α α , get the expert weight vector g for task α α ; Process the scene layer prediction results and task identifier t through the parameter matrix corresponding to task β α , get the expert weight vector g for task β β The expert weight vector g α and the expert weight vector g β(as column vectors) to form the G matrix, and then the row vector of the G matrix, i.e., the task weight vector of each expert (not shown), can be obtained as the basis for expert selection. After the expert selection by the selection module, the dedicated expert E1 for task α and the shared expert E2 for task α and task β are determined. At this time, for the target task α, the weight adjustment module can adjust the expert weight vector g for task α according to the expert selection result. α Update, that is: keep g α The weight g corresponding to the task-specific expert E1 in α [1] and the weight g corresponding to the task-sharing expert E2 α [2], get the updated expert weight vector Keep g β The weight g corresponding to the task sharing expert E2 β [2] and with the mission-specific expert E n The corresponding weight g β [n], get the updated expert weight vector For the target task α, the prediction module can obtain multiple task-level prediction components (i.e., expert outputs) obtained by each expert processing the scene-level prediction results, and based on the updated expert weight vector The weighted sum of the prediction components of each task layer is used to obtain the task layer prediction result corresponding to task α (denoted as task layer prediction result-α). Similarly, the gating module can be used for expert input and task identifier t β Get the expert weight vector g for task α α and the expert weight vector g for task β β , where: the scene layer prediction results and task identifier t are processed by the parameter matrix corresponding to task α β , get the expert weight vector g for task α α ; Process the scene layer prediction results and task identifier t through the parameter matrix corresponding to task β β , get the expert weight vector g for task β β The expert weight vector g α and the expert weight vector g β (as column vectors) to form the G matrix, and then the row vector of the G matrix, i.e., the task weight vector of each expert (not shown), can be obtained as the basis for expert selection. After the expert selection by the selection module, the dedicated expert E for task β is determined n And the shared expert E2 for task α and task β. Figure 5 The selected experts (task-specific experts / task-shared experts) are indicated by solid boxes, and the unselected experts such as E3 are indicated by dotted boxes. At this time, for the target task β, the weight adjustment module can adjust the expert weight vector g of task β according to the expert selection result. ββFor the target task β, the prediction module can obtain multiple task-level prediction components (i.e., expert outputs) obtained by each expert processing the scene-level prediction results, and based on the updated expert weight vector The weighted sum of the prediction components of each task layer is used to obtain the task layer prediction result corresponding to task β (denoted as task layer prediction result-β).

[0095] It can be seen that Figure 5 The task layer shown can share the same group of experts, but different gating modules, selection modules and weight adjustment modules can be set for different tasks so that different experts can be selected for prediction for different tasks. At this time, within the task layer, the task layer prediction process of different tasks (that is, the process of multiple experts and prediction modules calculating the task layer prediction results of different tasks) needs to be implemented serially.

[0096] Figure 6 It is a data flow diagram of a single task layer shown in other embodiments of the specification.

[0097] and Figure 5 compared to, Figure 6 The task layer shown includes multiple groups of experts. For example, a group of experts can be set for each of the multiple tasks. Figure 6 In the task layer shown in the figure, multiple groups of experts are required to process the expert input corresponding to different tasks respectively, such as processing task t by one group of experts. α The corresponding expert input ( Figure 6 The expert input on the left-α) gets the task t α The task-level prediction results and the processing task t through another set of expert input β The corresponding expert input ( Figure 6 The expert input on the right side -β) gets the task t β It can be understood that the number of groups of experts input can be consistent with the number of the multiple tasks. For example, when the multiple tasks are m tasks, m groups of experts can be set. Such a setting can make the task layer relative to Figure 5 The structure shown processes multiple tasks in parallel to improve processing efficiency.

[0098] Step 240, processing the task layer prediction result through the output layer 140 to obtain a model prediction result.

[0099] When there are multiple target tasks, the task layer prediction results corresponding to each target task can be processed separately through the output layer 140 to obtain the model prediction results under different target tasks. Figure 4When multiple target tasks include task α and task β, the task layer prediction result corresponding to task α can be processed by the output layer 140 to obtain the model prediction result under task α (model prediction result-α), and the task layer prediction result corresponding to task β can be processed by the output layer 140 to obtain the model prediction result under task β (model prediction result-β).

[0100] In some embodiments, the output layer 140 may include towers of MLPs corresponding to different tasks, for example, Figure 4 The Tower-α corresponding to task α and the Tower-β corresponding to task β in . For any task (such as task t α or task t β ), the task layer prediction result corresponding to the task can be input into the tower of the MLP corresponding to the task to obtain the model prediction result under the task, for example, Figure 4 The model prediction result in α or the model prediction result in β. That is, in, represents the model prediction result under task k, z k It represents the task layer prediction result under task k, MLP() represents the functional function of the multi-layer perceptron, and σ() represents the activation function (such as Sigmoid function).

[0101] In some embodiments, the output layer is used to perform dimensional conversion or numerical normalization processing on the prediction results of the task layer (such as converting the task prediction results in vector form into a numerical value between 0 and 1) to obtain the result form required by the prediction scenario. Combined with the feature data mentioned in step 210, the model prediction result can be the probability of recommending the recommended product to the user. When the target tasks include improving the click-through rate and improving the conversion rate, the output layer can output the probability of recommending the recommended product to the user under the target task of optimizing (improving) the click-through rate (which can be understood as the probability of being clicked by the user when recommending the recommended product to the user), and the output layer can also output the probability of recommending the recommended product to the user under the target task of optimizing (improving) the conversion rate (which can be understood as the probability of being purchased by the user when recommending the recommended product to the user).

[0102] Figure 7 is an exemplary flow chart of a model training method suitable for multiple scenarios and multiple tasks according to some embodiments of the specification. Figure 7 As shown, process 700 may include the following steps.

[0103] Step 710, obtaining feature data, scene identifiers, task identifiers and labels of training samples.

[0104] It can be understood that each training sample can be regarded as an instance.

[0105] Step 720, input the feature data, scene identifier and task identifier of the training sample into the model system 100 to obtain the model prediction result corresponding to the training sample.

[0106] For more details about the model system 100 and model predictions, please refer to Figure 1 , Figure 2 And the related description will not be repeated here.

[0107] Step 730: Determine a first loss function value based on the model prediction result and the label corresponding to the training sample, wherein the first loss function value reflects the difference between the model prediction result and the label.

[0108] Exemplarily, the first loss function can be constructed based on cross entropy. Specifically, the cross entropy of the jth training sample under task k is as follows:

[0109]

[0110] Among them, L k represents the cross entropy of the jth training sample under task k, represents the label value of the jth training sample under task k, The model prediction result of the jth training sample under task k, log() represents the logarithmic function.

[0111] Step 740, determines the value of the second loss function based on the similarity between the scene weight vector and the scene index vector of each scene-specific expert corresponding to the training sample, the similarity between the scene weight vector and the uniform distribution vector of each scene-shared expert, the similarity between the task weight vector and the task index vector of each task-specific expert, and the similarity between the task weight vector and the uniform distribution vector of each task-shared expert.

[0112] Based on the above assumptions, the higher the similarity, the more certain the classification of the proprietary experts and the shared experts, and accordingly, the more accurate the model prediction results. Therefore, the second loss function can be negatively correlated with the sum of the similarities, and by reducing the second loss function value, the accuracy of the model prediction result model can be improved. With reference to the above embodiment, when the similarity is negatively correlated with the KL divergence, the second loss function value can include the sum of the KL divergence corresponding to each similarity.

[0113] Step 750 , adjust the model parameters of the model system 100 to simultaneously reduce the first loss function value and the second loss function value.

[0114] According to the aforementioned embodiment, the model parameters of the model system 100 may include the parameter matrix S of the scene layer 120 j and the parameter matrix T of the task layer 130 k .

[0115] It should be noted that the above-mentioned description of the process is only for example and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to the process under the guidance of this specification. However, these modifications and changes are still within the scope of this specification. For example, in some embodiments, the total loss function may not include the second loss function.

[0116] The beneficial effects that may be brought about by the embodiments of this specification include but are not limited to: (1) providing a machine learning solution suitable for multiple scenarios and multiple tasks, which can save the maintenance cost of model iteration, and can also efficiently utilize the shared information between scenarios / tasks, which helps to improve learning efficiency; (2) selecting scenario / task-specific / shared experts through a general and flexible architecture can improve the accuracy of prediction results; (3) through the multi-layer expansion of the scenario layer, the computing cost can be greatly saved; (4) during model training, adding a second loss function term can enhance the expert selection decision-making ability of the model under the target scenario / target task; (5) experiments show that applying the model prediction method provided by the embodiments of this specification to the search recommendation business effectively improves the prediction accuracy of the product recommendation probability under different search entry scenarios, and significantly optimizes the click-through rate and conversion rate. It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects that may be produced may be any one or a combination of the above, or any other possible beneficial effects.

[0117] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation on the embodiments of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to the embodiments of this specification. Such modifications, improvements and corrections are suggested in the embodiments of this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0118] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment" and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different locations in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0119] In addition, it will be appreciated by those skilled in the art that the various aspects of the embodiments of this specification may be illustrated and described by a number of patentable categories or situations, including any new and useful process, machine, product or combination of substances, or any new and useful improvements thereto. Accordingly, the various aspects of the embodiments of this specification may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may all be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". In addition, the various aspects of the embodiments of this specification may be represented as a computer product located in one or more computer-readable media, the product including computer-readable program code.

[0120] A computer storage medium may include a propagated data signal containing computer program code, for example, in baseband or as part of a carrier wave. The propagated signal may be in a variety of forms, including electromagnetic, optical, etc., or a suitable combination. A computer storage medium may be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, device or apparatus to communicate, propagate or transmit the program for use. The program code on the computer storage medium may be transmitted via any suitable medium, including radio, cable, fiber optic cable, RF, or similar media, or any combination of the above media.

[0121] The computer program coding required for the operation of each part of the embodiment of this specification can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages ​​such as C language, VisualBasic, Fortran2003, Perl, COBOL2002, PHP, ABAP, dynamic programming languages ​​such as Python, Ruby and Groovy, or other programming languages, etc. The program coding can be run completely on the user's computer, or run on the user's computer as an independent software package, or partly run on the user's computer and partly run on the remote computer, or run completely on the remote computer or processing equipment. In the latter case, the remote computer can be connected to the user's computer by any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., by the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

[0122] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in the embodiments of this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of the embodiments of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing processing device or mobile device.

[0123] Similarly, it should be noted that in order to simplify the description of the embodiments disclosed in this specification, thereby helping to understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the objects of the embodiments of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0124] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification is hereby incorporated by reference in its entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this application (currently or later attached to this application) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.

[0125] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of the embodiments of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A model prediction method suitable for multiple scenarios and multiple tasks, comprising: Obtain feature data, scene identifiers, and task identifiers; The scene identifier is used to indicate a target scene selected from a plurality of scenes, and the task identifier is used to indicate a target task selected from a plurality of tasks; the feature data reflects one or more of the following features: user features, features of the product to be recommended, and query behavior features; Processing the feature data and the scene identifier through a scene layer, selecting one or more scene-specific expert modules and one or more scene-shared expert modules from a plurality of expert modules at the scene layer to process the feature data respectively, and obtaining a scene layer prediction result; Processing the scene layer prediction result and the task identifier through the task layer, so as to select one or more task-specific expert modules and one or more task-shared expert modules from a plurality of expert modules in the task layer to process the scene layer prediction result respectively, and obtain a task layer prediction result; Processing the task layer prediction result through the output layer to obtain a model prediction result; the model prediction result reflects the probability of recommending the to-be-recommended product to the user; The processing of the feature data and the scene identifier by the scene layer to select one or more scene-specific expert modules and one or more scene-shared expert modules from a plurality of expert modules at the scene layer to process the feature data respectively to obtain a scene layer prediction result includes: Processing the feature data and the scene identifier through a scene layer selection module to obtain a scene weight vector for each of the multiple expert modules of the scene layer; wherein the dimension of the scene weight vector is related to the number of the multiple scenes; For each of the plurality of expert modules of the scene layer, determining whether the expert module is a scene-specific expert module and whether the expert module is a scene-shared expert module based on the scene weight vector of the expert module; The step of processing the scene layer prediction result and the task identifier through the task layer to select one or more task-specific expert modules and one or more task-shared expert modules from a plurality of expert modules in the task layer to process the scene layer prediction result respectively to obtain the task layer prediction result includes: Processing the scene layer prediction result and the task identifier through the selection module of the task layer to obtain a task weight vector for each of the multiple expert modules of the task layer; wherein the dimension of the task weight vector is related to the number of the multiple tasks; For each of the plurality of expert modules of the task layer, it is determined whether the expert module is a task-specific expert module or a task-shared expert module based on the task weight vector of the expert module.

2. The method of claim 1, wherein: The target scenario is one, and the target tasks are one or more.

3. The method of claim 1, wherein: Also includes: Processing the feature data and the scene identifier through the selection module of the scene layer to obtain an expert weight vector of the target scene; wherein the dimension of the expert weight vector is related to the number of the multiple expert modules of the scene layer; The weights corresponding to the scene-specific expert modules and the scene-shared expert modules in the expert weight vector are retained to obtain an updated expert weight vector.

4. The method of claim 3, wherein: The feature data is processed by each scene-specific expert module and each scene-shared expert module to obtain scene-level prediction results, including: Acquire multiple scene prediction components obtained by each expert module processing the feature data; The scene layer prediction result is obtained by weighted summing of the multiple scene prediction components based on the updated expert weight vector.

5. The method of claim 1, wherein: For each of the plurality of expert modules of the scene layer, determining whether the expert module is a scene-specific expert module based on the scene weight vector of the expert module comprises: Determine the similarity between the scene weight vector and the scene index vector of each expert module; the dimension of the scene index vector is equal to the scene weight vector, and the element corresponding to the target scene in the scene index vector is non-0, and the other elements are 0; Based on the respective similarities, one or more scene-specific expert modules are selected from the plurality of expert modules at the scene level.

6. The method of claim 1, wherein: For each of the plurality of expert modules of the scene layer, determining whether the expert module is a scene sharing expert module based on the scene weight vector of the expert module comprises: Determine the similarity between the scene weight vector of each expert module and the uniform distribution vector; the dimension of the uniform distribution vector is equal to the scene weight vector, and the elements in the uniform distribution vector are the same; Based on the similarities, one or more scene-sharing expert modules are selected from the plurality of expert modules at the scene layer.

7. The method of claim 1, wherein: The scene is represented by a multi-level structure, each level of the multi-level structure has one or more scene categories; the scene identifier includes sub-identifiers corresponding to each level of the multi-level structure, and the sub-identifier corresponding to each level indicates the target scene category of the target scene at the corresponding level.

8. The method of claim 7, wherein the number of the scene layers is the same as the number of levels of the multi-level structure, and each scene layer corresponds to each level of the multi-level structure one by one; The processing of the feature data and the scene identifier by the scene layer to select one or more scene-specific expert modules and one or more scene-shared expert modules from a plurality of expert modules of the scene layer to respectively process the feature data to obtain a scene layer prediction result includes, for a scene layer corresponding to one of the first levels: The scene layer prediction result of the previous scene layer and the sub-identifier corresponding to the level are processed by the scene layer to select one or more scene-specific expert modules and one or more scene-shared expert modules from the multiple expert modules of the scene layer to respectively process the scene layer prediction result of the previous scene layer to obtain the scene layer prediction result of the current scene layer.

9. The method of claim 1, wherein: Also includes: Processing the scene layer prediction result and the task identifier through the selection module of the task layer to obtain an expert weight vector of the target task; wherein the dimension of the expert weight vector is related to the number of the multiple expert modules of the task layer; The weights corresponding to the task-specific expert modules and the task-shared expert modules in the expert weight vector are retained to obtain an updated expert weight vector.

10. The method according to claim 1 or 9, wherein: When the scene layer prediction result and the task identifier are processed by the task layer selection module to obtain the task weight vector of each of the multiple expert modules of the task layer and / or the expert weight vector of the target task, the scene identifier is processed together with the scene layer prediction result and the task identifier.

11. The method of claim 9, wherein: The scene-level prediction results are processed by each task-specific expert module and each task-shared expert module to obtain task-level prediction results, including: Acquire multiple task prediction components obtained by each expert module processing the scene layer prediction result; The multiple task prediction components are weighted and summed based on the updated expert weight vector to obtain the task layer prediction result.

12. The method of claim 1, wherein: For each of the plurality of expert modules of the task layer, determining whether the expert module is a task-specific expert module based on the task weight vector of the expert module comprises: Determine the similarity between the task weight vector and the task indicator vector of each expert module; the dimension of the task indicator vector is equal to the task weight vector, and the element corresponding to the target task in the task indicator vector is non-0, and the other elements are 0; Based on the respective similarities, one or more task-specific expert modules are selected from the plurality of expert modules at the task layer.

13. The method of claim 1, wherein: For each of the plurality of expert modules at the task layer, determining whether the expert module is a task sharing expert module based on the task weight vector of the expert module comprises: Determine the similarity between the task weight vector of each expert module and the uniform distribution vector; the dimension of the uniform distribution vector is equal to the task weight vector, and the elements in the uniform distribution vector are the same; Based on the respective similarities, one or more task-sharing expert modules are selected from the plurality of expert modules at the task layer.

14. The method of claim 2, wherein: There are multiple target tasks, and the corresponding task identifiers are also multiple and correspond one-to-one to the multiple target tasks; the scene layer prediction results and the task identifiers are processed by the task layer to select one or more task-specific expert modules and one or more task-shared expert modules from multiple expert modules in the task layer to process the scene layer prediction results respectively, and obtain the task layer prediction results, including for each of the multiple target tasks: The scene layer prediction result and the task identifier corresponding to the target task are processed through the task layer to select one or more task-specific expert modules and one or more task-shared expert modules from multiple expert modules in the task layer to process the scene layer prediction result respectively, so as to obtain the task layer prediction result corresponding to the target task.

15. The method of claim 14, wherein: There are multiple task layers; processing the scene layer prediction result and the task identifier corresponding to the target task through the task layer, selecting one or more task-specific expert modules and one or more task-sharing expert modules from multiple expert modules in the task layer to respectively process the scene layer prediction result, and obtaining the task layer prediction result corresponding to the target task, including for one of the task layers: When the task layer is the first task layer, the scene layer prediction result and the task identifier corresponding to the target task are processed by the task layer, so as to select one or more task-specific expert modules and one or more task-sharing expert modules from a plurality of expert modules in the task layer to process the scene layer prediction result respectively, and obtain the task layer prediction result corresponding to the target task of the current task layer; When the task layer is not the first task layer, the task layer prediction result corresponding to the target task of the previous task layer and the task identifier corresponding to the target task are processed through the task layer, so as to select one or more task-specific expert modules and one or more task-sharing expert modules from multiple expert modules in the task layer to respectively process the task layer prediction result corresponding to the target task of the previous task layer, and obtain the task layer prediction result corresponding to the target task of the current task layer.

16. The method of claim 14, wherein: The task layer prediction results corresponding to each target task are processed respectively through the output layer to obtain the model prediction results under different target tasks.

17. The method of claim 1, wherein: The acquiring of feature data, scene identifier and task identifier includes: Get the search term entered by the user in a search portal on the platform; Determine the products to be recommended based on the search terms; Acquire feature data based on user features, features of the recommended product, and historical query behavior features of the user; Taking the certain search entry as the target scene, and then determining the scene identifier; The target task is determined to be one or more of increasing the click rate and increasing the conversion rate, and then the task identifier is determined.

18. A model prediction device suitable for multiple scenarios and multiple tasks, comprising a processor and a storage device, wherein the storage device is used to store instructions, wherein: When the processor executes the instruction, the model prediction method as described in any one of claims 1 to 17 is implemented.

19. A model system suitable for multiple scenarios and multiple tasks, comprising: Input layer, used to obtain feature data, scene identification and task identification; The scene identifier is used to indicate a target scene selected from a plurality of scenes, and the task identifier is used to indicate a target task selected from a plurality of tasks; the feature data reflects one or more of the following features: user features, features of the product to be recommended, and query behavior features; A scene layer, used for processing the feature data and the scene identifier, so as to select one or more scene-specific expert modules and one or more scene-shared expert modules from a plurality of expert modules of the scene layer to process the feature data respectively, and obtain a scene layer prediction result; The task layer is used to process the scene layer prediction result and the task identifier, so as to select one or more task-specific expert modules and one or more task-shared expert modules from a plurality of expert modules in the task layer to process the scene layer prediction result respectively, so as to obtain the task layer prediction result; The output layer is used to process the prediction result of the task layer to obtain a model prediction result; the model prediction result reflects the probability of recommending the product to be recommended to the user; Wherein, the scene layer also includes a gating module and a selection module; The gating module is used to process the feature data and the scene identifier to obtain a scene weight vector of each of the multiple expert modules of the scene layer; wherein the dimension of the scene weight vector is related to the number of the multiple scenes; The selection module is used to determine, for each of the plurality of expert modules of the scene layer, whether the expert module is a scene-specific expert module and whether it is a scene-shared expert module based on the scene weight vector of the expert module; Wherein, the task layer also includes a gating module and a selection module; The gating module is used to process the scene layer prediction result and the task identifier to obtain a task weight vector for each of the multiple expert modules of the task layer; wherein the dimension of the task weight vector is related to the number of the multiple tasks; The selection module is used to determine, for each of the multiple expert modules of the task layer, whether the expert module is a task-specific expert module and whether it is a task-shared expert module based on the task weight vector of the expert module.

20. The model system of claim 19, wherein: The gating module is further used to process the feature data and the scene identifier to obtain an expert weight vector of the target scene; wherein the dimension of the expert weight vector is related to the number of the multiple expert modules of the scene layer; The scene layer also includes a weight adjustment module, which is used to retain the weights corresponding to the scene-specific expert modules and the scene-shared expert modules in the expert weight vector to obtain an updated expert weight vector.

21. The model system of claim 20, wherein: The scene layer also includes a prediction module, which is used to: Acquire multiple scene-level prediction components obtained by processing the feature data by each expert module; The scene layer prediction result is obtained by weighted summing the multiple scene layer prediction components based on the updated expert weight vector.

22. The model system of claim 19, wherein: The selection module is specifically used for: Determine the similarity between the scene weight vector and the scene index vector of each expert module; the dimension of the scene index vector is equal to the scene weight vector, and the element corresponding to the target scene in the scene index vector is non-0, and the other elements are 0; Based on the respective similarities, one or more scene-specific expert modules are selected from the plurality of expert modules of the scene layer.

23. The model system of claim 19, wherein: The selection module is specifically used for: Determine the similarity between the scene weight vector of each expert module and the uniform distribution vector; the dimension of the uniform distribution vector is equal to the scene weight vector, and the elements in the uniform distribution vector are the same; Based on the similarities, one or more scene-sharing expert modules are selected from the plurality of expert modules of the scene layer.

24. The model system of claim 19, wherein: The scene is represented by a multi-level structure, each level of the multi-level structure has one or more scene categories; the scene identifier includes sub-identifiers corresponding to each level of the multi-level structure, and the sub-identifier corresponding to each level indicates the target scene category of the target scene at the corresponding level.

25. The model system of claim 24, wherein the number of the scene layers is the same as the number of levels of the multi-level structure, and each scene layer corresponds to each level of the multi-level structure one by one; The scene layer corresponding to one of the levels is used for: The scene layer prediction result of the previous scene layer and the sub-identifier corresponding to the level are processed by the scene layer to select one or more scene-specific expert modules and one or more scene-shared expert modules from the multiple expert modules of the scene layer to respectively process the scene layer prediction result of the previous scene layer to obtain the scene layer prediction result of the current scene layer.

26. The model system of claim 19, wherein: The gating module is further used to process the scene layer prediction result and the task identifier to obtain an expert weight vector of the target task; wherein the dimension of the expert weight vector is related to the number of the multiple expert modules of the task layer; The task layer also includes a weight adjustment module, which is used to retain the weights corresponding to the task-specific expert modules and the task-shared expert modules in the expert weight vector to obtain an updated expert weight vector.

27. The model system of claim 19 or 26, wherein: When the scene layer prediction result and the task identifier are processed by the selection module of the task layer to obtain the weight vector corresponding to each of the multiple expert modules of the task layer and / or the expert weight vector of the target task, the scene identifier is also processed together with the scene layer prediction result and the task identifier.

28. The model system of claim 26, wherein: The task layer also includes a prediction module, which is used to: Acquire multiple task prediction components obtained by each expert module processing the scene layer prediction result; The task layer prediction result is obtained by weighted summing of the multiple task prediction components based on the updated expert weight vector.

29. The model system of claim 19, wherein: The selection module is specifically used for: Determine the similarity between the task weight vector and the task indicator vector of each expert module; the dimension of the task indicator vector is equal to the task weight vector, and the element corresponding to the target task in the task indicator vector is non-0, and the other elements are 0; Based on the respective similarities, one or more task-specific expert modules are selected from the plurality of expert modules of the task layer.

30. The model system of claim 19, wherein: The selection module is specifically used for: Determine the similarity between the task weight vector of each expert module and the uniform distribution vector; the dimension of the uniform distribution vector is equal to the task weight vector, and the elements in the uniform distribution vector are the same; Based on the respective similarities, one or more task-sharing expert modules are selected from the plurality of expert modules of the task layer.

31. The model system of claim 19, wherein: There are multiple target tasks, and the corresponding task identifiers are also multiple and correspond one-to-one to the multiple target tasks; for each of the multiple target tasks, the task layer is used to: Process the scene layer prediction result and the task identifier corresponding to the target task, select one or more task-specific expert modules and one or more task-shared expert modules from multiple expert modules in the task layer to process the scene layer prediction result respectively, and obtain the task layer prediction result corresponding to the target task.

32. The model system of claim 31, wherein: There are multiple task layers; in order to process the scene layer prediction result and the task identifier corresponding to the target task, one or more task-specific expert modules and one or more task-shared expert modules are selected from multiple expert modules in the task layer to process the scene layer prediction result respectively, and obtain the task layer prediction result corresponding to the target task, for one of the task layers: When the task layer is the first task layer, the task layer is used to process the scene layer prediction result and the task identifier corresponding to the target task, so as to select one or more task-specific expert modules and one or more task-sharing expert modules from multiple expert modules in the task layer to process the scene layer prediction result respectively, and obtain the task layer prediction result corresponding to the target task of the current task layer; When the task layer is not the first task layer, the task layer is used to process the task layer prediction result corresponding to the target task of the previous task layer and the task identifier corresponding to the target task, so as to select one or more task-specific expert modules and one or more task-sharing expert modules from multiple expert modules in the task layer to respectively process the task layer prediction result corresponding to the target task of the previous task layer, and obtain the task layer prediction result corresponding to the target task of the current task layer.

33. The model system of claim 31, wherein: The output layer is used to process the task layer prediction results corresponding to each target task respectively, and obtain the model prediction results under different target tasks.

34. The model system of claim 19, wherein: The system further comprises an acquisition module, configured to: Get the search term entered by the user in a search portal on the platform; Determine the products to be recommended based on the search terms; Determining feature data based on user features of the user, features of the product to be recommended, and features of the user's historical query behavior; Taking the certain search entry as the target scene, and then determining the scene identifier; Determine the target task as one or more of increasing click rate and increasing conversion rate, and then determine the task identifier; And, sending the feature data, scene identifier and task identifier to the input layer.

35. A model training method suitable for multiple scenarios and multiple tasks, comprising: Obtain feature data, scene identifiers, task identifiers, and labels of training samples; The scene identifier is used to indicate a target scene selected from a plurality of scenes, and the task identifier is used to indicate a target task selected from a plurality of tasks; Inputting feature data, scene identifiers, and task identifiers of the training samples into the model system described in any one of claims 19 to 34 to obtain model prediction results corresponding to the training samples; Determine a first loss function value based on the model prediction result and the label corresponding to the training sample, wherein the first loss function value reflects the difference between the model prediction result and the label; Based on the similarity between the scene weight vector and the scene index vector of each scene-specific expert module corresponding to the training sample, the similarity between the scene weight vector and the uniform distribution vector of each scene-shared expert module, the similarity between the task weight vector and the task index vector of each task-specific expert module, and the similarity between the task weight vector and the uniform distribution vector of each task-shared expert module, a second loss function value is determined, wherein the second loss function is negatively correlated with the sum of the similarities; The model parameters of the model system are adjusted to simultaneously reduce the first loss function value and the second loss function value.

36. A model training device suitable for multiple scenarios and multiple tasks, comprising a processor and a storage device, wherein the storage device is used to store instructions, wherein: When the processor executes the instructions, the model training method as described in claim 35 is implemented.

Images