Data processing method based on grouped hybrid expert model, electronic equipment and medium

Abstract

The embodiment of the invention provides a data processing method based on a grouped hybrid expert model, electronic equipment and a medium. According to the method, multiple pieces of input feature data are acquired, and for each piece of input feature data in the multiple pieces of input feature data, at least one target expert group matched with the input feature data is determined from multiple routing expert groups through a group routing unit; and determining at least one activation expert matched with the input feature data from each target expert group through an expert routing unit, and performing data processing on the input feature data through the sharing expert and the at least one activation expert to obtain a corresponding processing result. According to the method, the routing experts with the high adaptation degree with the input feature data are determined as the activated experts, distribution of the routing experts according to needs is achieved, the method only needs to carry out weight calculation on the routing experts in the target expert group, the calculation amount in the data processing process is reduced, the calculation cost is reduced, and the calculation efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, electronic device, and medium based on a group hybrid expert model. Background Art

[0002] With the application and development of artificial intelligence (AI) technology in deep learning, the complexity and number of parameters in AI models have increased exponentially. This has led to a surge in computing resources and power consumption during AI model training and inference. To balance model performance and computational cost, existing technologies have proposed a hybrid expert model.

[0003] The existing Mixed Expert Model (MoE) breaks down complex tasks into multiple subtasks and assigns a corresponding "expert" to each subtask, allowing the "expert" to handle the corresponding subtask. This process requires calculating the weights of all experts, assigning a corresponding expert to each subtask based on the weight, and activating the expert. Therefore, this method only needs to activate experts with high weights, reducing computational costs compared to traditional AI models. In addition, to ensure load balancing for each expert, the parameters of each expert in the existing MoE are the same size, resulting in an overabundance of experts when handling simple tasks and insufficient expert capacity when handling complex tasks, leading to low prediction accuracy.

[0004] Therefore, the existing MoE needs to calculate the weights of all experts and cannot allocate them on demand for tasks of different complexity, resulting in a waste of computing resources, high computing costs and low computing efficiency. Summary of the Invention

[0005] The embodiments of the present application provide a data processing method, electronic device, and medium based on a grouped hybrid expert model to solve the problem that the existing MoE needs to calculate all expert weights and cannot allocate them on demand for tasks of different complexities, resulting in waste of computing resources, high computing costs, and low computing efficiency.

[0006] In a first aspect, embodiments of the present application provide a data processing method based on a grouped hybrid expert model, wherein the grouped hybrid expert model includes: a shared expert group, multiple routing expert groups, a group routing unit, and an expert routing unit, wherein each routing expert group has a different parameter quantity, and multiple experts within each routing expert group have the same parameter quantity. The method includes:

[0007] Obtain multiple input feature data;

[0008] For each input feature data in the plurality of input feature data, determining, by a group routing unit, at least one target expert group that matches the input feature data from the plurality of routing expert groups;

[0009] Determining, by an expert routing unit, at least one activated expert that matches the input feature data from each target expert group;

[0010] The shared expert and the at least one activated expert perform data processing on the input feature data to obtain corresponding processing results.

[0011] In a possible implementation, the routing expert group corresponds to at least one group feature tag, and determining, by the group routing unit, at least one target expert group that matches the input feature data from the multiple routing expert groups includes:

[0012] For any routing expert group among the plurality of routing expert groups, matching at least one group feature tag of the routing expert group with the input feature data by the group routing unit to obtain a group inference weight corresponding to the routing expert group, wherein the group inference weight represents the inference accuracy of the routing expert group on the input feature data;

[0013] At least one target expert group that meets a group screening condition is determined based on the plurality of group reasoning weights. The group screening condition includes determining a routing expert group whose group reasoning weights are ranked within a first preset range as the target expert group.

[0014] In a possible implementation, the expert corresponds to at least one expert feature tag, and determining, by the expert routing unit, at least one activated expert matching the input feature data from each target expert group includes:

[0015] Acquire expert group information of the target expert group through the expert routing unit, wherein the expert group information includes: an expert group identifier and a target group reasoning weight;

[0016] For at least one undetermined expert in at least one target expert group, matching the expert feature label of each undetermined expert with the input feature data to obtain a corresponding initial inference weight;

[0017] Based on the target group reasoning weight, the initial reasoning weight of the corresponding undetermined expert is modified to obtain the expert reasoning weight corresponding to each of the undetermined experts;

[0018] At least one activated expert that meets an expert screening condition is determined based on at least one of the expert reasoning weights. The expert screening condition includes: determining a pending expert whose expert reasoning weight ranking is within a second preset range as an activated expert.

[0019] In a possible implementation, after determining at least one target expert group that meets a group screening condition based on the plurality of group reasoning weights, the method further includes:

[0020] determining, based on the group reasoning weight corresponding to each input feature data and the group screening condition, a first total number of times each routing expert group is determined as a target expert group, and a total group reasoning weight corresponding to each routing expert group;

[0021] A group routing loss function is determined based on the first total number of times, the total group inference weight, and a hidden layer dimension corresponding to each routing expert group, where the hidden layer dimension is in direct proportion to the group routing loss function.

[0022] In a possible implementation, after determining at least one activated expert that meets the expert screening condition based on at least one expert reasoning weight, the method further includes:

[0023] Determining a second total number of times each pending expert is determined as an activated expert and a total expert reasoning weight of each pending expert based on the expert reasoning weight corresponding to each input feature data and the expert screening condition;

[0024] An expert routing loss function corresponding to the undetermined expert is determined based on the second total number of times and the total expert reasoning weight.

[0025] In one possible implementation, the method further includes:

[0026] Determining an initial model loss function based on the processing results;

[0027] Determining a target model loss function based on the group routing loss function, the expert routing loss function, and the initial model loss function;

[0028] The grouped hybrid expert model is optimized based on the target model loss function.

[0029] In a possible implementation, the processing of the input feature data by the shared expert and the at least one activated expert to obtain a corresponding processing result includes:

[0030] Performing inference and prediction on the input feature data by the shared expert to obtain a shared inference result; and

[0031] Performing inference prediction on the input feature data by the at least one activated expert to obtain at least one inference result;

[0032] The shared inference result and the at least one inference result are summed to obtain the processing result.

[0033] In a second aspect, an embodiment of the present application provides a data processing device based on a grouped hybrid expert model, wherein the grouped hybrid expert model includes: a shared expert group, multiple routing expert groups, a group routing unit, and an expert routing unit, wherein each routing expert group has a different parameter quantity, and the parameter quantities of multiple experts in each routing expert group are the same. The device includes:

[0034] An acquisition module, used to obtain multiple input feature data;

[0035] a determination module, configured to determine, for each input feature data in the plurality of input feature data, at least one target expert group matching the input feature data from the plurality of routing expert groups through a group routing unit;

[0036] The determination module is further configured to determine, from each target expert group, at least one activated expert that matches the input feature data through an expert routing unit;

[0037] The processing module is used to perform data processing on the input feature data through the shared expert and the at least one activated expert to obtain corresponding processing results.

[0038] In a possible implementation, the routing expert group corresponds to at least one group feature tag, and the apparatus further includes: a matching module and a screening module;

[0039] The matching module is configured to match, for any routing expert group among the plurality of routing expert groups, at least one group feature label of the routing expert group with the input feature data via the group routing unit to obtain a group inference weight corresponding to the routing expert group, wherein the group inference weight represents the inference accuracy of the routing expert group on the input feature data;

[0040] The screening module is configured to determine at least one target expert group that meets a group screening condition based on the plurality of group reasoning weights, wherein the group screening condition includes determining as the target expert group a routing expert group whose group reasoning weights are ranked within a first preset range.

[0041] In a possible implementation, the expert corresponds to at least one expert feature tag, and the apparatus further includes: a correction module;

[0042] The acquisition module is further configured to acquire expert group information of the target expert group through the expert routing unit, wherein the expert group information includes: an expert group identifier and a target group reasoning weight;

[0043] The matching module is further configured to match the expert feature label of each undetermined expert with the input feature data for at least one undetermined expert in at least one target expert group to obtain a corresponding initial inference weight;

[0044] The correction module is used to correct the initial reasoning weight of the corresponding undetermined expert based on the target group reasoning weight to obtain the expert reasoning weight corresponding to each of the undetermined experts;

[0045] The determination module is further configured to determine, based on at least one of the expert reasoning weights, at least one activated expert that meets an expert screening condition, wherein the expert screening condition includes determining as an activated expert a pending expert whose expert reasoning weight ranking is within a second preset range.

[0046] In one possible implementation, the determining module is further configured to determine a first total number of times each routing expert group is determined as the target expert group, and a total group reasoning weight corresponding to each routing expert group, based on the group reasoning weight corresponding to each input feature data and the group screening condition;

[0047] The determination module is further configured to determine a group routing loss function based on the first total number of times, the total group inference weight, and a hidden layer dimension corresponding to each routing expert group, wherein the hidden layer dimension is proportional to the group routing loss function.

[0048] In a possible implementation, the determination module is further configured to determine, based on the expert reasoning weight corresponding to each input feature data and the expert screening condition, a second total number of times each pending expert is determined as an activated expert, and a total expert reasoning weight of each pending expert;

[0049] The determination module is further configured to determine an expert routing loss function corresponding to the undetermined expert based on the second total number of times and the total expert reasoning weight.

[0050] In a possible implementation, the determining module is further configured to determine an initial model loss function based on the processing result;

[0051] The determination module is further configured to determine a target model loss function based on the group routing loss function, the expert routing loss function, and the initial model loss function;

[0052] The processing module is further used to optimize the grouped hybrid expert model based on the target model loss function.

[0053] In a possible implementation, the apparatus further includes: a reasoning module;

[0054] The reasoning module is configured to perform reasoning and prediction on the input feature data through the shared expert to obtain a shared reasoning result; and

[0055] Performing inference prediction on the input feature data by the at least one activated expert to obtain at least one inference result;

[0056] The processing module is further configured to perform sum processing on the shared reasoning result and the at least one reasoning result to obtain the processing result.

[0057] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0058] The memory stores computer-executable instructions;

[0059] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0060] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementation methods of the first aspect.

[0061] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0062] The embodiment of the present application provides a data processing method based on a grouped hybrid expert model, in which the grouped hybrid expert model includes: a shared expert group, multiple routing expert groups, a group routing unit, and an expert routing unit. The parameter amounts of each routing expert group are different, and the parameter amounts of multiple experts in each routing expert group are the same. The method obtains multiple input feature data, and for each input feature data in the multiple input feature data, determines at least one target expert group that matches the input feature data from the multiple routing expert groups through the group routing unit, and determines at least one activated expert that matches the input feature data from each target expert group through the expert routing unit. Then, the input feature data is processed by the shared expert and the at least one activated expert to obtain the corresponding processing result, and the routing expert with a high degree of adaptability to the input feature data is determined as the activated expert, thereby realizing the on-demand allocation of routing experts. In addition, the method only needs to perform weight calculation on the routing experts in the target expert group, reducing the amount of calculation in the data processing process, thereby reducing the calculation cost and improving the calculation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0064] Figure 1 Schematic diagram of the data processing process based on the group hybrid expert model provided in this application Figure 1 ;

[0065] Figure 2 A schematic diagram of a grouped hybrid expert model provided in an embodiment of the present application;

[0066] Figure 3 A schematic diagram of the structure of a routing expert group provided in an embodiment of the present application;

[0067] Figure 4 Schematic diagram of the data processing method based on the grouped hybrid expert model provided in this application Figure 2 ;

[0068] Figure 5 Schematic diagram of the data processing method based on the grouped hybrid expert model provided in this application Figure 3 ;

[0069] Figure 6 A schematic diagram of the structure of a data processing device based on a grouped hybrid expert model provided by this application;

[0070] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application.

[0071] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0072] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0073] First, let’s explain the terms involved in this application:

[0074] Mixture of Experts: A machine learning approach that partitions an AI model into separate sub-networks (or “experts”), each specializing in a subset of the input data, to jointly perform a task.

[0075] MoE is a model based on the Transformer architecture. The core idea of ​​MoE is to decompose the target task into multiple subtasks, each of which is handled by a dedicated expert model. These expert models can be, for example, neural networks or other machine learning models.

[0076] MoEs include multiple expert models. During reasoning on the target task, the inference weights of all expert models for each subtask must be calculated to match the corresponding activated expert to each subtask. The activated experts then reason about the subtasks and output the inference results. Furthermore, in existing MoE architectures, each expert has the same number of parameters, resulting in an overabundance of expert capacity for simple tasks and insufficient expert capacity for complex ones, leading to low prediction accuracy.

[0077] Therefore, the existing MoE needs to calculate the weights of all experts and cannot allocate them on demand for tasks of different complexity, resulting in a waste of computing resources, high computing costs and low computing efficiency.

[0078] The data processing method based on the grouped hybrid expert model provided in the present application determines the target expert group corresponding to the input feature data from multiple routing expert groups through a group routing unit, determines at least one activated expert corresponding to the input feature data from each target expert group through an expert routing unit, and then processes the input feature data through a shared expert and at least one activated expert to obtain a processing result, thereby realizing on-demand allocation of routing experts. In addition, this method only requires weight calculation for the routing experts in the target expert group, reducing the computational complexity of the data processing process, thereby reducing computational costs and improving computational efficiency.

[0079] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0080] Figure 1 Schematic diagram of the data processing process based on the group hybrid expert model provided in this application Figure 1 ,The group hybrid expert model includes : a shared expert group, multiple routing expert groups, a group routing unit, and an expert routing unit. ,The parameter amount of each routing expert group is different, and the parameter amount of multiple experts in each routing expert group is the same, ,e.g. Figure 1 As shown, the method includes:

[0081] S101: Acquire multiple input feature data.

[0082] Specifically, the preprocessing layer divides the input data and performs feature representation on each sub-input data obtained from the division, obtaining multiple sub-input feature data. These sub-input feature data are then positionally encoded to obtain multiple input feature data corresponding to the input data. The grouped hybrid expert model processes each input feature data independently. Therefore, the input hidden layer of the grouped hybrid expert model obtains a single input feature data as input to the routing expert group, group routing unit, and expert routing unit. The input feature data includes data dimensions and position encodings.

[0083] It is understandable that the dimensions of the above-mentioned input feature data can be set according to the accuracy requirements of data processing, and this application does not impose any restrictions on this.

[0084] For example, the input data might be, for example, "What's the weather like tomorrow?" The preprocessing layer divides this input data into four tokens, positionally encodes these four tokens, and converts them into four high-dimensional vectors, resulting in four 1024-dimensional token vectors. These four 1024-dimensional tokens serve as the input feature data in this step. These four tokens include "tomorrow," "weather," "how," and "?". Each token has a corresponding positional encoding that characterizes the positional sequence of each token, for example, "tomorrow" is the first sequence, "weather" is the second sequence, "how" is the third sequence, and "?" is the fourth sequence. This positional sequence is used to reorganize the output feature data for each token to obtain the output data corresponding to the input data.

[0085] Figure 2 A schematic diagram of a group hybrid expert model provided in an embodiment of the present application is shown as follows: Figure 2 As shown, the input hidden layer of the grouped mixture of experts model obtains a token, Figure 2 Each circle in the input hidden layer represents a dimension of the above token. The vector of tokens obtained by the input hidden layer represents the input of the shared expert, group router, and expert router.

[0086] also, Figure 2In the example, 1 through Ns are shared experts, and Group1 through GroupNg are routing expert groups. Shared experts are general, reusable modules designed to improve data processing efficiency and model generalization by reusing expert models. All token predictions utilize shared experts. During training and use, shared experts learn common features that can be transferred across multiple token tasks, improving model generalization. Optionally, these shared experts employ a fully connected layer network structure.

[0087] For example, Figure 3 A schematic diagram of the structure of a routing expert group provided in an embodiment of the present application. Figure 3 As shown in the figure, there are Nr experts, divided into Ng groups, each containing N experts. Here, N = Nr / Ng. The expert parameters within each group are the same, but the expert model parameters vary between groups. The expert parameters set for group i are proportional to the expert's number of hidden layer channels, Wi. The minimum number of hidden layer channels among all groups is Wmin. The expert parameters for each group increase in size, and the N experts in each group have the same parameters. The parameters of each expert in Group 1 are proportional to Wmin.

[0088] like Figure 3 As shown in the figure, since the parameters of the N experts in the routing expert group are the same, the probability of the N experts in the group being selected is the same, which is conducive to achieving load balancing among the experts in the group. In addition, each group has an expert deployed on any GPU. Since the selection probability of the experts in each group is the same, the probability of each GPU being activated is the same. Therefore, the routing expert group and the GPU are decoupled. The selection of different routing expert groups will not cause the experts on a certain GPU to be over-activated, thereby achieving the technical effect of evenly utilizing the computing resources of each GPU. In addition, Figure 3 The layout shown avoids the problem of unbalanced GPU load, which causes the forward propagation to wait for the GPU with heavy computation burden, thereby reducing computation efficiency.

[0089] S102: For each input feature data in the plurality of input feature data, determine, by a group routing unit, at least one target expert group that matches the input feature data from a plurality of routing expert groups.

[0090] Specifically, the group routing unit obtains input feature data from the input hidden layer of the grouped hybrid expert model and, based on the built-in routing mechanism, matches the input feature data with the feature data that each routing expert group is adept at processing, obtaining a matching value. This matching value represents the inference performance of each routing expert group based on the input feature data. These multiple matching values ​​are sorted, and according to the preset routing mechanism, routing expert groups that meet the sorting range are selected as target expert groups. For example, the matching values ​​are sorted from largest to smallest, and the routing expert groups corresponding to the top five rankings are selected as target expert groups.

[0091] For example, Figure 2 As shown. First, the input feature data is obtained from the Input hidden layer. The input feature data is a multi-dimensional vector. Then, the Group Router calculates the matching degree between the feature data that the routing expert groups Group 1 to Group Ng are good at processing and the input feature data, and then selects the routing expert group with the matching degree in the top Kg (Top-Kg) as the target expert group. Figure 2 As shown, the target expert groups include: Group 1, Group Ng, and at least one other routing expert group.

[0092] This method uses a group routing unit to determine the target expert group for processing input feature data, so that the activated experts can be selected within the target expert group, reducing the computational complexity of determining the activated experts. In addition, this method uses a feature matching method to determine the target expert group, ensuring the inference effect of the input features.

[0093] S103: Determine, through the expert routing unit, at least one activated expert that matches the input feature data from each target expert group.

[0094] Specifically, after determining at least one target expert group corresponding to the input feature data, for each expert in the target expert group, the input feature data is matched against the feature data that each expert is adept at processing, resulting in a matching value. Based on this matching value, at least one activated expert is then determined. For example, the matching values ​​are sorted from largest to smallest, and the experts ranked in the top Kr (Top-Kr) are determined as activated experts. The specific value of the Top-Kr value can be set based on data processing requirements and is not limited in this application.

[0095] Exemplarily, the Expert Router obtains input feature data from the Input Hidden layer, matches the input feature data with feature data that experts in each target expert group are good at processing, obtains matching values, and then selects the top Kr experts with larger matching values ​​as activated experts.

[0096] Optionally, the complexity of the input feature data is determined based on lexical and semantic features of the input feature data, and the specific value of the Top-Kr is determined based on the complexity. For example, different levels of complexity correspond to different Top-Kr values, and the higher the complexity, the larger the Top-Kr value.

[0097] S104: Process the input feature data through the shared expert and at least one activated expert to obtain corresponding processing results.

[0098] Specifically, after determining at least one active expert, the at least one active expert is used to perform inference and prediction on the input feature data to obtain a routing expert inference result. Furthermore, a shared expert obtains the input feature data and performs inference and prediction on the input feature data to obtain a shared inference result. The processing result of the input feature data is then determined based on the routing expert inference result and the shared inference result. This processing result may be, for example, output feature data, whose dimensions are consistent with those of the input feature output.

[0099] For example, Figure 2 As shown in the figure, this grouped hybrid expert model has a total of Ns shared experts. The shared experts obtain input feature data from the Input Hidden layer and perform inference and prediction on this input feature data to produce shared output feature data. Furthermore, the activated experts selected by the Expert Router obtain input feature data and perform inference and prediction on this input feature data to produce routing expert output feature data. After obtaining the shared output feature data and the routing expert output feature data, the shared output feature data and the routing expert output feature data are concatenated to generate the output feature data in the Output Hidden layer.

[0100] In addition, each output feature data corresponding to the input feature data has a corresponding position code, and the multiple output feature data are combined according to the position code, and the output data corresponding to the input data is determined based on the combined output feature data.

[0101] The embodiment of the present application provides a data processing method based on a grouped hybrid expert model, in which the grouped hybrid expert model includes: a shared expert group, multiple routing expert groups, a group routing unit, and an expert routing unit. The parameter amounts of each routing expert group are different, and the parameter amounts of multiple experts in each routing expert group are the same. The method obtains multiple input feature data, and for each input feature data in the multiple input feature data, determines at least one target expert group that matches the input feature data from the multiple routing expert groups through the group routing unit, and determines at least one activated expert that matches the input feature data from each target expert group through the expert routing unit. Then, the input feature data is processed by the shared expert and the at least one activated expert to obtain the corresponding processing result, and the routing expert with a high degree of adaptability to the input feature data is determined as the activated expert, thereby realizing the on-demand allocation of routing experts. In addition, the method only needs to perform weight calculation on the routing experts in the target expert group, reducing the amount of calculation in the data processing process, thereby reducing the calculation cost and improving the calculation efficiency.

[0102] Figure 4 Schematic diagram of the data processing method based on the grouped hybrid expert model provided in this application Figure 2 ,like Figure 4 As shown, this embodiment Figures 1 to 3 Based on the embodiment, a data processing method based on a group hybrid expert model is described in detail. The method includes:

[0103] S401: Acquire multiple input feature data.

[0104] This step S401 is similar to the explanation of the above step S101 and will not be repeated here.

[0105] S402: For any routing expert group among the multiple routing expert groups, a group routing unit performs matching processing on at least one group feature label of the routing expert group and input feature data to obtain a group inference weight corresponding to the routing expert group.

[0106] Each routing expert group is assigned at least one group feature tag, which includes feature tags corresponding to the routing expert group's strengths in reasoning. For example, these feature tags may be feature vectors representing different reasoning domains, such as weather, emotions, and finance. Furthermore, the group reasoning weight represents the routing expert group's reasoning accuracy for the input feature data.

[0107] Specifically, the group routing unit calculates the matching degree between the input feature data and the group feature label of each routing expert group, and after obtaining the matching degrees corresponding to multiple routing expert groups, normalizes the matching degrees to obtain the group inference weight corresponding to each routing expert group.

[0108] Exemplarily, the group routing unit calculates the matching scores between at least one group feature label and the input feature data of each routing expert group in sequence through a trained neural network, and after obtaining the matching scores corresponding to multiple routing experts, uses softmax (normalized exponential function) to normalize the multiple matching scores to obtain the group inference weight corresponding to each routing expert group.

[0109] This method evaluates the group inference weight of each routing expert group on the input feature data through the matching degree between the group feature label and the input feature data, and quantifies the inference effect of each routing expert group on the input feature data. In the process of screening the target expert group, it helps to improve the screening efficiency of the target expert group and obtain the target expert group with better inference effect.

[0110] S403: Determine at least one target expert group that meets the group screening condition based on the multiple group reasoning weights.

[0111] Specifically, after obtaining the group reasoning weight corresponding to each routing expert group, a comparison and analysis of the multiple group reasoning weights is performed to screen out at least one target expert group that meets the group screening conditions. The group screening conditions include: determining the routing expert group whose group reasoning weight is ranked within a first preset range as the target expert group. For example, the multiple group reasoning weights are sorted according to the preset rules, and the routing expert group whose group reasoning weight is ranked within the first preset range is determined as the target expert group. Figure 2 As shown, the multiple group reasoning weights are sorted in descending order, and the routing expert groups corresponding to the multiple group reasoning weights ranked in Top-Kg are determined as the target expert groups.

[0112] This method determines the target expert group that meets the group screening criteria based on the group inference weights of each routing expert group on the input feature data, resulting in a target expert group with excellent inference performance. Furthermore, in the subsequent process of determining the activated experts from the target expert group, only the inference weights of the multiple experts in the target expert group on the input feature data need to be calculated, eliminating the need to calculate the inference weights of all experts in the grouped hybrid expert model on the input feature data. This reduces the amount of computational effort required for data processing during the inference process and improves computational efficiency.

[0113] S404: Obtain expert group information of the target expert group through the expert routing unit.

[0114] Specifically, each routing expert group carries an expert group identifier. After obtaining the group inference weights for the input feature data for each routing expert group in step S403, expert group information is generated based on the group inference weights and the expert group identifiers, and the expert group information is sent to the expert routing unit. The expert group information corresponding to the target expert group includes the expert group identifier and the target group inference weights. For example, the expert group information corresponding to the target expert group can be [001, 0.6].

[0115] S405 : For at least one undetermined expert in at least one target expert group, match the expert feature label of each undetermined expert with the input feature data to obtain a corresponding initial inference weight.

[0116] Specifically, each target expert group includes at least one pending expert, and the pending expert corresponds to at least one expert feature label. The at least one expert feature label of the pending expert is matched with the input feature data in sequence to determine whether the input feature data is a domain feature that the corresponding pending expert is good at inferring and predicting, and to obtain the initial inference weight of the above-mentioned pending expert for the input feature data.

[0117] For example Figure 2 Group 1 is the target expert group. There are three pending experts in Group 1. The expert feature labels of pending expert 1 are: [Calculus, Lagrange mean value theorem, derivative], the expert feature labels of pending expert 2 are: [Differential mean value theorem, Rolle's theorem, derivative], and the expert feature labels of pending expert 3 are: [Matrix operations, determinant, eigenvalue]. The input feature data is: [Lagrange]. The above three pending experts and the input feature data are matched to obtain the matching score of each pending expert and the input feature data. The three matching scores are then normalized to obtain the initial inference weight of pending expert 1 is 0.7, the inference weight of pending expert 2 is 0.2, and the inference weight of pending expert 3 is 0.1.

[0118] This method only calculates the initial inference weight of each pending expert in the target expert group for the input feature data, effectively reducing the computational complexity of the data processing process and thereby improving computational efficiency.

[0119] S406: Modify the initial reasoning weight of the corresponding pending expert based on the target group reasoning weight to obtain the expert reasoning weight corresponding to each pending expert.

[0120] Specifically, to further refine the inference performance of the candidate experts on the input feature data, the target group inference weight of the target expert group is used to modify the initial inference weight of each candidate expert in the group to obtain the expert inference weight corresponding to the candidate expert. For example, for the N candidate experts in Group 1, the target group inference weight of Group 1 is multiplied by the initial inference weights of the N candidate experts, and the multiple products are normalized to obtain the expert inference weight corresponding to each candidate expert in Group 1.

[0121] like Figure 2 As shown, the target group reasoning weights of Group1, Group Ng and other target expert groups and the initial reasoning weights of the pending experts are input into the corresponding product operation units, and the product operation units multiply the target group reasoning weights of each group and the initial reasoning weights of each pending expert in the group to obtain the expert reasoning weight corresponding to each pending expert.

[0122] S407: Determine at least one activated expert that meets the expert screening condition based on at least one expert reasoning weight.

[0123] For multiple pending experts corresponding to multiple target expert groups, based on the expert reasoning weight of each pending expert, the multiple expert reasoning weights are screened according to expert screening conditions to obtain at least one activated expert that meets the expert screening conditions. The expert screening conditions include: determining as an activated expert a pending expert whose expert reasoning weight is ranked within a second preset range.

[0124] For example, the target expert group includes Group 1, Group 5, and Group 14. The expert reasoning weights of all pending experts in Group 1, Group 5, and Group 14 are sorted from largest to smallest, and the experts with the expert reasoning weights in the top-Kr ranking range are determined as active experts. For example, if Top-Kr = 7, then the experts with the expert reasoning weights in the top 7 ranking range are determined as active experts.

[0125] Optionally, the expert reasoning weights can be modified based on the reasoning efficiency of each candidate expert. For example, the reasoning efficiency of each candidate expert is multiplied by the corresponding expert reasoning weight, and the multiple products are normalized to obtain the modified expert reasoning weights. This modified expert reasoning weight not only ensures reasoning effectiveness but also takes into account reasoning efficiency, which helps to select activated experts with high reasoning effectiveness and efficiency.

[0126] S408. Perform inference prediction on the input feature data through shared experts to obtain shared inference results; and perform inference prediction on the input feature data through at least one activated expert to obtain at least one inference result.

[0127] Specifically, after determining at least one activated expert corresponding to the input feature data, the input feature data is input into the at least one activated expert, which then performs inference and prediction on the input feature data to obtain at least one inference result. This inference result can, for example, be a feature vector with the same dimension as the input feature data. Furthermore, shared experts perform inference and prediction on the input feature data to obtain a shared inference result. This shared inference result has the same dimension as the input feature data.

[0128] S409: Add the shared reasoning result and at least one reasoning result to obtain a processing result.

[0129] Specifically, the shared inference result has the same dimension as the inference result. The shared inference result and the inference result are added together to obtain the inference result corresponding to the input feature data. The dimension of the inference result is the same as the input feature data. The inference result carries a position code. After obtaining the inference results of other input feature data, multiple inference results are integrated according to the position code to obtain the output data.

[0130] like Figure 2 As shown, the shared reasoning result and at least one reasoning result are input into the summing operation unit, and the summing operation unit sums the shared reasoning result and at least one reasoning result to obtain the reasoning result, and outputs the reasoning result to the Output Hidden layer.

[0131] The data processing method based on the group hybrid expert model provided in the embodiment of the present application obtains multiple input feature data, and for each routing expert group in multiple routing expert groups, matches at least one group feature label of each routing expert group with the input feature data through a group routing unit to obtain a group reasoning weight corresponding to the routing expert group. Then, based on the multiple group reasoning weights, at least one target expert group that meets the group screening conditions is determined, and at least one target expert group with the best reasoning ability for the input feature data is obtained, which helps to improve the processing efficiency of the input feature data and obtain more accurate processing results. In addition, each target expert group corresponds to expert group information, and the expert group information of the target expert group is obtained through the expert routing unit. For at least one pending expert in at least one target expert group, the expert feature label of each pending expert is matched with the input feature data to obtain the corresponding initial inference weight. The initial inference weight of the corresponding pending expert is corrected based on the target group inference weight of the target expert group to obtain the expert inference weight corresponding to each pending expert. Based on the at least one expert inference weight, at least one activated expert that meets the expert screening condition is determined, and then the input feature data is inferred and predicted by the shared expert to obtain a shared inference result. The input feature data is also inferred and predicted by the at least one activated expert to obtain at least one inference result. The shared inference result and the at least one inference result are summed to obtain a processing result. This method only calculates the initial inference weights of multiple experts in the target expert group, without having to calculate the inference weights of all experts in the model, thereby reducing the computational complexity of the data processing process and improving data processing efficiency. In addition, the target group reasoning weight reflects the reasoning ability of the routing expert group on the input feature data. This method corrects the initial reasoning weight of the pending experts in the group based on the target group reasoning weight, which is conducive to accurately determining the expert reasoning weight of each expert in the target expert group on the input feature data, thereby selecting more reasonable activated experts and improving the reasoning effect of the input feature data.

[0132] Figure 5 Schematic diagram of the data processing method based on the grouped hybrid expert model provided in this application Figure 3 ,like Figure 5 As shown, this embodiment Figures 1 to 4 Based on the embodiment, a method for optimizing a grouped hybrid expert model is described in detail. This method optimizes the grouped hybrid expert model in the process of determining the target expert group and activating the expert through the group routing unit and the expert routing unit. The method includes:

[0133] S501: Determine a first total number of times each routing expert group is determined as a target expert group and a total group reasoning weight corresponding to each routing expert group based on a group reasoning weight corresponding to each input feature data and a group screening condition.

[0134] Specifically, after the corresponding target expert groups are determined for multiple input feature data, each routing expert group corresponds to a group reasoning weight. Based on the group reasoning weight and the group screening conditions, the first total number of times each routing expert group is determined as the target expert group is calculated, and based on the group reasoning weight of each routing expert group for each input feature data, the total group reasoning weight of each routing expert group is calculated.

[0135] Optionally, the first total number of times for each target expert group is determined based on the following formula:

[0136]

[0137] in, is the normalization parameter, is the number of routing expert groups, is the number of target expert groups identified, is the total number of input feature data, To characterize the Input feature data, To characterize the Group Routing Experts Group, For the Group Routing Experts Group The group inference weight of the input feature data, To characterize the Group Routing Experts Group, For the Group Routing Experts Group The group inference weight of the input feature data, For group filtering conditions, The group reasoning weights of multiple routing expert groups corresponding to the input feature data are used to return the largest group reasoning weight among the multiple group reasoning weights. The target group inference weights.

[0138] Optionally, the total group reasoning weight of each routing expert group is determined based on the following formula:

[0139]

[0140] in, is the total number of input feature data, To characterize the Input feature data, To characterize the Group Routing Experts Group, For the Group Routing Experts Group The group inference weights of the input feature data.

[0141] S502: Determine a group routing loss function based on the first total number of times, the total group reasoning weight, and the hidden layer dimension corresponding to each routing expert group.

[0142] The hidden layer dimension is proportional to the group routing loss function. After obtaining the first total number of times and the total inference weight of each routing expert group, the group routing loss function is calculated based on the first total number of times, the total inference weight and the hidden layer dimension of each routing expert group.

[0143] Optionally, the group routing loss function is determined based on the following formula:

[0144]

[0145] in, is a balancing factor used to control the impact of the loss function on the group routing strategy. To characterize the Group Routing Experts Group, For the The hidden layer dimensions of the group routing expert group, is the smallest hidden layer dimension among all routing expert groups, For the The total number of first times the group routing expert group, For the The total reasoning weight of the group routing experts.

[0146] The group routing loss function penalizes expert groups that select large parameter counts. The hidden layer dimension of each expert group has a positive linear relationship with the number of parameters. As shown in the above formula, the higher the hidden layer dimension of the routing expert group, the greater the group routing loss. The model tends to select expert groups with smaller loss functions. Therefore, an increase in the group routing loss reduces the group inference weight of the corresponding routing expert group. This method helps reduce the inference process's reliance on experts with large parameter counts, thereby more rationally allocating and fully utilizing expert resources.

[0147] S503: Based on the expert reasoning weight corresponding to each input feature data and the expert screening condition, determine the second total number of times each pending expert is determined as an activated expert and the total expert reasoning weight of each pending expert.

[0148] Specifically, after determining multiple activated experts corresponding to multiple input feature data, each pending expert in the target expert group has an expert reasoning weight. Based on the expert reasoning weight and the expert screening conditions, the second total number of times each pending expert is determined as an activated expert is calculated, and the total expert reasoning weight corresponding to each pending expert is calculated based on the expert reasoning weight of each pending expert for each input feature data.

[0149] Optionally, the second total number of times each expert is activated is determined based on the following formula:

[0150]

[0151] in, To characterize the Group Target Expert Group, To characterize the Experts to be determined, is the number of all pending experts in all target expert groups, is the total number of input feature data, To characterize the Input feature data, is the normalization parameter, For the The first Undetermined experts on Expert reasoning weights for input feature data, To characterize the Experts to be determined, Expert screening conditions to filter out the largest front expert reasoning weights.

[0152] Optionally, the total expert reasoning weight corresponding to each pending expert is determined based on the following formula:

[0153]

[0154] in, To characterize the Group Target Expert Group, To characterize the Experts to be determined, is the total number of input feature data, To characterize the Input feature data.

[0155] S504: Determine an expert routing loss function corresponding to the undetermined expert based on the second total number of times and the total expert reasoning weight.

[0156] Among them, the expert routing loss function corresponding to each target expert group affects the expert reasoning weight of each expert in the routing expert group to balance the load of experts in the group and avoid excessive dependence of the model on some experts.

[0157] Optionally, the expert routing loss function within each target expert group is determined based on the following formula:

[0158]

[0159] in, To characterize the Group Target Expert Group, To characterize the Experts to be determined, is the number of all pending experts in all target expert groups, For the The first in the target expert group The total number of times a pending expert is determined as an activated expert. For the The first in the target expert group Experts to be determined.

[0160] S505: Determine the initial model loss function based on the processing result.

[0161] Specifically, the initial model loss function of the grouped hybrid expert model is calculated based on the actual output content corresponding to the processing results and the input feature data. This initial model loss function can minimize the inference error. Since experts with a large number of parameters can process complex input feature data more quickly and output more accurate processing results, if the initial model loss function is not constrained, the grouped hybrid expert model tends to select the expert group with a large number of parameters.

[0162] S506 , determining a target model loss function based on the group routing loss function, the expert routing loss function, and the initial model loss function; and optimizing the grouped hybrid expert model based on the target model loss function.

[0163] Specifically, the group routing loss function, the expert routing loss function and the initial model loss function are weightedly summed to obtain the target model loss function, and then the parameters of the group expert hybrid model are updated and optimized based on the target model loss function until the model converges.

[0164] It can be understood that when performing weighted summation on the group routing loss function, the expert routing loss function and the initial model loss function, the weight corresponding to each loss function is set based on the model training requirements, and this application does not impose any restrictions on this.

[0165] The data processing method based on the grouped hybrid expert model provided in the embodiment of the present application determines, based on the group reasoning weight and group screening conditions corresponding to each input feature data, a first total number of times each routing expert group is determined as the target expert group, and the total group reasoning weight corresponding to each routing expert group, and then determines the group routing loss function based on the first total number, the total group reasoning weight, and the hidden layer dimension corresponding to each routing expert group; determines, based on the expert reasoning weight and expert screening conditions corresponding to each input feature data, a second total number of times each expert is determined as an activated expert, and the total expert reasoning weight of each expert, and then determines the expert routing loss function of the corresponding expert based on the second total number and the total expert reasoning weight; determines the initial model loss function based on the processing result, and after obtaining the group routing loss function, the expert routing loss function, and the initial model loss function, determines the target model loss function based on the group routing loss function, the expert routing loss function, and the initial model loss function, and optimizes the grouped hybrid expert model based on the target model loss function. In this method, the initial model loss function favors expert groups with larger parameters, while the group routing loss function penalizes those that choose larger parameters. These two functions are combined to determine the target model loss function, which is then used for model optimization. This helps the grouped hybrid model select expert groups with appropriate parameters, thereby achieving on-demand allocation of expert resources. Furthermore, the expert routing loss function in this method influences the expert reasoning weights of each expert within the routing expert group, helping to balance the load of experts within the group and prevent the model from over-relying on a few experts.

[0166] Figure 6 This is a structural diagram of a data processing device based on a grouped hybrid expert model provided by the present application. The grouped hybrid expert model in the device includes: a shared expert group, multiple routing expert groups, a group routing unit, and an expert routing unit. The parameter amount of each routing expert group is different, and the parameter amount of multiple experts in each routing expert group is the same, such as Figure 6 As shown, the data processing device 60 based on the grouped hybrid expert model provided in this embodiment includes:

[0167] An acquisition module 601 is used to acquire a plurality of input feature data;

[0168] a determination module 602 configured to determine, for each input feature data in the plurality of input feature data, at least one target expert group matching the input feature data from the plurality of routing expert groups through a group routing unit;

[0169] The determination module 602 is further configured to determine, through an expert routing unit, at least one activated expert that matches the input feature data from each target expert group;

[0170] The processing module 603 is configured to process the input feature data using the shared expert and the at least one activated expert to obtain corresponding processing results.

[0171] In a possible implementation, the routing expert group corresponds to at least one group feature tag, and the apparatus further includes: a matching module 604 and a screening module 605;

[0172] The matching module 604 is configured to match, for any routing expert group among the plurality of routing expert groups, at least one group feature tag of the routing expert group with the input feature data via the group routing unit to obtain a group inference weight corresponding to the routing expert group, wherein the group inference weight represents the inference accuracy of the routing expert group on the input feature data;

[0173] The screening module 605 is configured to determine at least one target expert group that meets a group screening condition based on the plurality of group reasoning weights, wherein the group screening condition includes determining as the target expert group a routing expert group whose group reasoning weights are ranked within a first preset range.

[0174] In a possible implementation, the expert corresponds to at least one expert feature tag, and the apparatus further includes: a correction module 606;

[0175] The acquisition module 601 is further configured to acquire expert group information of the target expert group through the expert routing unit, wherein the expert group information includes: an expert group identifier and a target group reasoning weight;

[0176] The matching module is further configured to match the expert feature label of each undetermined expert with the input feature data for at least one undetermined expert in at least one target expert group to obtain a corresponding initial inference weight;

[0177] The correction module 606 is used to correct the initial reasoning weight of the corresponding undetermined expert based on the target group reasoning weight to obtain the expert reasoning weight corresponding to each of the undetermined experts;

[0178] The determination module 602 is further configured to determine, based on at least one of the expert reasoning weights, at least one activated expert that meets an expert screening condition, wherein the expert screening condition includes determining as an activated expert a pending expert whose expert reasoning weight is ranked within a second preset range.

[0179] In one possible implementation, the determining module 602 is further configured to determine a first total number of times each routing expert group is determined as a target expert group, and a total group reasoning weight corresponding to each routing expert group, based on the group reasoning weight corresponding to each input feature data and the group screening condition.

[0180] The determining module 602 is further configured to determine a group routing loss function based on the first total number of times, the total group inference weight, and a hidden layer dimension corresponding to each routing expert group, wherein the hidden layer dimension is proportional to the group routing loss function.

[0181] In one possible implementation, the determining module 602 is further configured to determine a second total number of times each expert is determined as an activated expert and a total expert reasoning weight of each expert based on the expert reasoning weight corresponding to each input feature data and the expert screening condition;

[0182] The determining module 602 is further configured to determine an expert routing loss function of a corresponding expert based on the second total number of times and the total expert reasoning weight.

[0183] In a possible implementation, the determining module 602 is further configured to determine an initial model loss function based on the processing result;

[0184] The determining module 602 is further configured to determine a target model loss function based on the group routing loss function, the expert routing loss function, and the initial model loss function;

[0185] The processing module 603 is further configured to optimize the grouped hybrid expert model based on the target model loss function.

[0186] In a possible implementation, the apparatus further includes: a reasoning module 607;

[0187] The reasoning module 607 is configured to perform reasoning and prediction on the input feature data through the shared expert to obtain a shared reasoning result; and

[0188] Performing inference prediction on the input feature data by the at least one activated expert to obtain at least one inference result;

[0189] The processing module 603 is further configured to add the shared reasoning result and the at least one reasoning result to obtain the processing result.

[0190] The data processing device based on the group hybrid expert model provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar, and are not described in detail in this embodiment.

[0191] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 7As shown, the electronic device 70 provided in this embodiment includes: at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, the memory 702 and the communication component 703 are connected via a bus 704.

[0192] During the specific implementation process, at least one processor 701 executes the computer-executable instructions stored in the memory 702, so that the at least one processor 701 performs the above method.

[0193] The specific implementation process of the processor 701 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0194] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0195] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0196] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0197] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0198] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0199] The readable storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0200] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0201] The division of units is merely a logical functional division; actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, any direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units, either through an interface, electrical, mechanical, or other means.

[0202] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0203] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0204] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present invention. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0205] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0206] Finally, it should be noted that those skilled in the art will readily identify other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The present invention is not limited to the precise structure described above and illustrated in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof. The scope of the present invention is limited solely by the appended claims.

Claims

1. A data processing method based on a grouped hybrid expert model, characterized in that: The grouped hybrid expert model includes: a shared expert group, multiple routing expert groups, a group routing unit, and an expert routing unit. The parameter amounts of each routing expert group are different, and the parameter amounts of multiple experts in each routing expert group are the same. The method includes: Obtain multiple input feature data; For each input feature data in the plurality of input feature data, determining, by a group routing unit, at least one target expert group that matches the input feature data from the plurality of routing expert groups; Determining, by an expert routing unit, at least one activated expert that matches the input feature data from each target expert group; The shared expert and the at least one activated expert perform data processing on the input feature data to obtain corresponding processing results.

2. The method according to claim 1, characterized in that The routing expert group corresponds to at least one group feature tag, and the determining, by the group routing unit, at least one target expert group matching the input feature data from the multiple routing expert groups includes: For any routing expert group among the plurality of routing expert groups, matching at least one group feature tag of the routing expert group with the input feature data by the group routing unit to obtain a group inference weight corresponding to the routing expert group, wherein the group inference weight represents the inference accuracy of the routing expert group on the input feature data; At least one target expert group that meets a group screening condition is determined based on the plurality of group reasoning weights. The group screening condition includes determining a routing expert group whose group reasoning weights are ranked within a first preset range as the target expert group.

3. The method according to claim 2, characterized in that The expert corresponds to at least one expert feature tag, and determining at least one activated expert matching the input feature data from each target expert group by the expert routing unit includes: Acquire expert group information of the target expert group through the expert routing unit, wherein the expert group information includes: an expert group identifier and a target group reasoning weight; For at least one undetermined expert in at least one target expert group, matching the expert feature label of each undetermined expert with the input feature data to obtain a corresponding initial inference weight; Based on the target group reasoning weight, the initial reasoning weight of the corresponding undetermined expert is modified to obtain the expert reasoning weight corresponding to each of the undetermined experts; At least one activated expert that meets an expert screening condition is determined based on at least one of the expert reasoning weights. The expert screening condition includes: determining a pending expert whose expert reasoning weight ranking is within a second preset range as an activated expert.

4. The method according to claim 3, characterized in that After determining at least one target expert group that meets a group screening condition based on the plurality of group reasoning weights, the method further includes: determining, based on the group reasoning weight corresponding to each input feature data and the group screening condition, a first total number of times each routing expert group is determined as a target expert group, and a total group reasoning weight corresponding to each routing expert group; A group routing loss function is determined based on the first total number of times, the total group inference weight, and a hidden layer dimension corresponding to each routing expert group, where the hidden layer dimension is in direct proportion to the group routing loss function.

5. The method according to claim 4, characterized in that After determining at least one activated expert that meets the expert screening condition based on at least one expert reasoning weight, the method further includes: Determining a second total number of times each pending expert is determined as an activated expert and a total expert reasoning weight of each pending expert based on the expert reasoning weight corresponding to each input feature data and the expert screening condition; An expert routing loss function corresponding to the undetermined expert is determined based on the second total number of times and the total expert reasoning weight.

6. The method according to claim 5, characterized in that The method further comprises: Determining an initial model loss function based on the processing results; Determining a target model loss function based on the group routing loss function, the expert routing loss function, and the initial model loss function; The grouped hybrid expert model is optimized based on the target model loss function.

7. The method according to claim 1, characterized in that The step of processing the input feature data by the shared expert and the at least one activated expert to obtain a corresponding processing result includes: Performing inference and prediction on the input feature data by the shared expert to obtain a shared inference result; and Performing inference prediction on the input feature data by the at least one activated expert to obtain at least one inference result; The shared inference result and the at least one inference result are summed to obtain the processing result.

8. A data processing device based on a grouped hybrid expert model, characterized in that: The grouped hybrid expert model includes: a shared expert group, multiple routing expert groups, a group routing unit, and an expert routing unit. The parameter amounts of each routing expert group are different, and the parameter amounts of multiple experts in each routing expert group are the same. The device includes: An acquisition module, used to obtain multiple input feature data; a determination module, configured to determine, for each input feature data in the plurality of input feature data, at least one target expert group matching the input feature data from the plurality of routing expert groups through a group routing unit; The determination module is further configured to determine, from each target expert group, at least one activated expert that matches the input feature data through an expert routing unit; The processing module is used to perform data processing on the input feature data through the shared expert and the at least one activated expert to obtain corresponding processing results.

9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images