Battery SOH and RUL joint prediction method based on HS-MH-MoE model and EIS data

Abstract

The invention discloses a battery SOH and RUL joint prediction method based on an HS-MH-MoE model and EIS data. The method comprises the following steps: S1, testing and collecting battery data: collecting battery EIS data by adopting a professional impedance spectroscopy test instrument; s2, fusing the different EIS data and the corresponding SOH and RUL data; s3, extracting EIS data features by using a multi-layer multi-head sparse MLP sub-expert model; s4, generating shared feature representation; and S5, simultaneously predicting SOH and RUL of the battery. According to the method, a simple multi-layer perceptron model is used as an expert sub-model to construct an HS-MH-MoE model, and EIS fusion data is used in one model to jointly predict SOH and RUL of the battery. The method has the comprehensive advantages of multi-source data fusion, multi-layer multi-head sparse model integration, multi-target prediction, feature sharing, precision, efficiency and the like, and is a high-performance and high-efficiency prediction model.

Description

Technical Field

[0001] The invention belongs to the field of electrochemical technology, and in particular to a battery SOH and RUL joint prediction method based on a HS-MH-MoE model and EIS data. Background Art

[0002] Battery State of Health (SOH) is an indicator that measures the performance level of a battery relative to its new state. It is used to monitor battery performance, predict life, evaluate safety, guide maintenance plans, and help users make replacement decisions. It plays an important role in the optimal use and safe management of batteries.

[0003] The remaining useful life (RUL) of a battery is an indicator that predicts how long a battery can be used in its current state. It helps with preventive maintenance, cost optimization, safety, resource planning, user experience, system reliability, product design improvement, regulatory compliance, and provides guidance for battery recycling and reuse. Common battery RULs include the number of remaining cycles.

[0004] Therefore, predicting battery SOH and RUL has high application value.

[0005] Electrochemical Impedance Spectroscopy (EIS) is a test method used to characterize the electrochemical process inside a battery. If the active material of the battery is reduced and the performance deteriorates, the EIS curve will change accordingly, so using EIS to predict SOH and RUL is a feasible path.

[0006] Generally speaking, when predicting SOH and RUL by testing EIS, two different sets of data are collected respectively. The quantity and dimension of the data are often different. Then, a separate prediction model is trained for each prediction target. The data utilization efficiency is not high, and the model training and maintenance costs are also high.

[0007] If multiple sets of EIS data are integrated and SOH and RUL are predicted simultaneously, it will improve data utilization efficiency, enhance model prediction accuracy and reduce model maintenance costs.

[0008] Mixture of Experts (MoE) is a machine learning model that distributes input data to different expert models through a gating network, with each expert model responsible for learning a specific part of the data. The outputs of these expert models are then combined to form the final prediction result. Sparse Mixture of Experts (SMoE) is designed for complex problems. It uses a gating network to weightedly aggregate the predictions of expert sub-models to achieve sparse activation and reduce computational costs. Hierarchical Sparse Mixture of Experts (HS-MoE) is an extension of SMoE, using multi-level experts and gating networks to form a tree structure to handle more complex tasks.

[0009] Multi-Head Mixture of Experts (MH-MoE) is a neural network architecture designed to improve the expressiveness and computational efficiency of the model. Each gating network of MH-MoE is a "head" that selects and activates the appropriate expert network based on the input data. Multiple gating networks (heads) can process input data in parallel. Multiple gating networks of MH-MoE work simultaneously, and each gating network generates its own weight distribution to select and activate different expert networks. The Multi-Head Mixture of Experts model achieves efficient computation and powerful expressiveness by combining multiple gating networks and expert networks.

[0010] Combining the Hierarchical Multi-Head Sparse Mixture of Experts (HS-MH-MoE) with the Multi-Head Sparse Mixture of Experts (MH-MoE) can build a more powerful model that can fully utilize the advantages of multi-level feature extraction and multi-head mechanism. This combined model is called the Hierarchical Multi-Head Sparse Mixture of Experts (HS-MH-MoE) in the present invention. Summary of the invention

[0011] In order to solve the above technical problems, the present invention uses a simple multi-layer perceptron (MLP) model as an expert sub-model to construct a HS-MH-MoE model, and uses EIS fusion data in one model to jointly predict battery SOH and RUL, thereby greatly reducing the model training and maintenance costs.

[0012] The technical solution adopted by the present invention to solve its technical problem is: A battery SOH and RUL joint prediction method based on HS-MH-MoE model and EIS data includes the following steps: Step S1. Test and collect battery data: Use professional impedance spectrum test instrument to collect battery EIS data, and the frequency range needs to be 10 -2 HZ to 10 5 HZ, select 5 to 60 frequency values ​​in the middle of the frequency range, preferably 60 as samples.

[0013] When collecting data, record the real and imaginary data of the impedance corresponding to each frequency, as well as the target value to be predicted, which is the corresponding battery SOH and RUL.

[0014] Step S2. Fuse different EIS data and corresponding SOH and RUL data: calculate the lowest common multiple of the numbers of different data sets, then upsample each data set so that the number of each data set is increased to their lowest common multiple, and vertically splice the two sets of data after the increase in number to obtain the fused data.

[0015] Step S3. Use a multi-layer multi-head sparse MLP sub-expert model to extract EIS data features: The HS-MH-MoE model includes a multi-layer multi-head sparse expert model; each sub-model at each level is an MLP model; the fused EIS data is used as input data for different MLP models to extract different features; the fused EIS data is used as input data for the gating network model to train the weights of different expert sub-models.

[0016] Furthermore, the MLP model is an expert sub-model, which adopts a multi-layer perceptron model and is a 3-layer network structure. The input layer receives input data, the middle layer is a 32-dimensional vector, the activation function is the relu function, and the output layer is a 32-dimensional vector. The activation function is relu, and the output layer is the feature expression of the input data.

[0017] Furthermore, the gated network model adopts a multi-layer perceptron model, which is a 3-layer network structure. The input layer receives input data, the middle layer is a 32-dimensional vector, the activation function is the relu function, and the output layer is a 3-dimensional vector. The activation function is softmax, which corresponds to the weights of the three expert sub-models respectively, and the sum of the three weights is 1.

[0018] Step S4. Generate shared feature representation: In each level, the EIS features extracted by the two MLP models with the highest probability are weighted and summed with the corresponding expert weights output by the gated network model to obtain the shared feature representation, which is used as the input of the subsequent prediction model.

[0019] Furthermore, the weighted summation process is as follows: The outputs of the three expert sub-models of the first level are respectively recorded as output_1_1, output_1_2 and output_1_3. These three outputs are all high-dimensional vectors. The present invention takes 32 dimensions as an example to represent the feature representation of each sub-expert model for the input data.

[0020] Each gating network model at the first level selects the outputs of the two experts with the highest probability, multiplies them with the corresponding weights and sums them up to obtain the weighted sum of the outputs of each gating network model and sub-expert model at the first level.

[0021] Then, the weighted sum values ​​output by each gated network model and sub-expert model of the first level are averaged to obtain the weighted sum average value of all sub-expert models of the first level.

[0022] Afterwards, the weighted sum average of all sub-expert models in the first layer is used as the input data of the second layer, and is input into each expert sub-model and each gating network model in the second layer respectively.

[0023] The outputs of the three expert sub-models of the second level are output_2_1, output_2_2 and output_2_3 respectively. These three outputs are all high-dimensional vectors. The present invention takes 32 dimensions as an example to represent the feature representation of each sub-expert model for the input data.

[0024] The three gating networks in the second layer independently calculate the weight of each expert sub-model in the second layer. The output of each gating network is a three-dimensional vector, which corresponds to the weights of the three expert sub-models, and the sum of the three weights is 1.

[0025] Then, each gating network model at the second level selects the outputs of the two experts with the highest probability, multiplies them with the corresponding weights and sums them up to obtain the feature representation of the weighted sum of the outputs of each gating network model at the second level and all sub-expert models.

[0026] Step S5. Simultaneously predict the battery SOH and RUL: After the shared feature representation at the second level is a fully connected layer, which further extracts the shared feature representation to obtain a high-level feature expression of EIS.

[0027] After the fully connected layer is the output layer of the model. The output layer has two linear regression models, and the corresponding predicted values ​​are the battery capacity, i.e., SOH, and the remaining number of training times, i.e., RUL.

[0028] The beneficial effects of the present invention are: The present invention takes into account the comprehensive advantages of multi-source data fusion, multi-layer multi-head sparse model integration, multi-target prediction, feature sharing, accuracy and efficiency. It is a high-performance and efficient battery SOH and RUL prediction model. Its technical advantages are as follows: 1. Multi-source data fusion: This method combines the EIS data tested under different conditions and the corresponding SOH and RUL data.

[0029] 2. Multi-layer multi-head sparse model integration: The multi-layer structure of HS-MH-MoE, with modules stacked layer by layer, extracts deep features of data and improves the accuracy of battery state prediction; the multi-head structure ensures multiple gated networks in each layer, focusing on different data patterns and making comprehensive and stable predictions; the sparse activation mechanism activates a small number of experts during prediction, reduces calculations, prevents overfitting, and improves prediction accuracy.

[0030] 3. Joint prediction model: By building a model that simultaneously predicts SOH and RUL, the interaction and dependency between the two can be captured more accurately.

[0031] 4. Data feature sharing: The fused EIS data is vertically spliced ​​and used as input data for each expert model and gated network model, and the data features are shared.

[0032] 5. MLP model feature extraction: The MLP model has a simple structure, is easy to learn, and has high training and prediction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a structural diagram of the model for joint prediction of battery SOH and RUL based on the HS-MH-MoE model and EIS fusion data of the present invention.

[0034] Figure 2 It is the overall flow chart of the joint prediction of battery SOH and RUL based on the HS-MH-MoE model and EIS fusion data of the present invention.

[0035] Figure 3 This is the structure diagram of the MLP sub-model at each level.

[0036] Figure 4 It is a structural diagram of the gated network model.

[0037] Figure 5 This is a sample diagram of battery SOH prediction results based on the HS-MH-MoE model and EIS fusion data.

[0038] Figure 6 This is a sample diagram of battery RUL prediction results based on the HS-MH-MoE model and EIS fusion data. DETAILED DESCRIPTION

[0039] like Figure 1 As shown, the input data is the battery EIS fusion data. The EIS comes from different test data sets, all of which are composed of real and imaginary parts. The figure takes two EIS data sets as an example, which can also be expanded to multiple data sets.

[0040] Next are the multi-layer sparse expert layer and the multi-head gated network layer. The present invention takes a two-layer structure, each layer has three MLP expert sub-models, and each layer has three gated network models based on a multi-layer perceptron (MLP) as an example.

[0041] The input data is input into the three expert sub-models of the first level respectively, and the corresponding features are extracted from different angles. The outputs of the three expert sub-models are output_1_1, output_1_2 and output_1_3 respectively. These three outputs are all high-dimensional vectors. The present invention takes 32 dimensions as an example to represent the feature representation of each sub-expert model for the input data.

[0042] The input data is input into the three gating networks of the first level respectively, and the weight of each expert sub-model in the first level is calculated independently. The output of each gating network is a three-dimensional vector, which corresponds to the weights of the three expert sub-models, and the sum of the three weights is 1.

[0043] Afterwards, each gating network model selects the outputs of the two experts with the highest probability, multiplies them with the corresponding weights and sums them up to obtain the weighted sum of the outputs of each gating network model and sub-expert model at the first level.

[0044] Then, the weighted sum values ​​output by each gated network model and sub-expert model of the first level are averaged to obtain the weighted sum average value of all sub-expert models of the first level.

[0045] Afterwards, the weighted sum average of all sub-expert models in the first layer is used as the input data of the second layer, and is input into each expert sub-model and each gating network model in the second layer respectively.

[0046] The outputs of the three expert sub-models of the second level are output_2_1, output_2_2 and output_2_3 respectively. These three outputs are all high-dimensional vectors. The present invention takes 32 dimensions as an example to represent the feature representation of each sub-expert model for the input data.

[0047] The three gating networks in the second layer independently calculate the weight of each expert sub-model in the second layer. The output of each gating network is a three-dimensional vector, which corresponds to the weights of the three expert sub-models, and the sum of the three weights is 1.

[0048] Then, each gating network model at the second level selects the outputs of the two experts with the highest probability, multiplies them with the corresponding weights and sums them up to obtain the feature representation of the weighted sum of the outputs of each gating network model at the second level and all sub-expert models.

[0049] Afterwards, the three feature representations output by the second level are connected to a fully connected layer.

[0050] Finally, through two linear regression layers, SOH and RUL are output simultaneously.

[0051] refer to Figure 2 The implementation process of the present invention includes the following steps: Step S1. Test and collect battery data: Testing and collecting battery EIS data requires professional impedance spectrum testing instruments, and needs to cover a certain frequency range. The frequency range can be set to 10 -2 HZ to 10 5 HZ, multiple frequency values ​​are selected at a certain interval in the middle of the frequency range, and the frequency values ​​are generally selected from 5 to 60. The present invention selects 60 as a sample.

[0052] When collecting data, it is necessary to record the real and imaginary data of the impedance corresponding to each frequency, as well as the target value that needs to be predicted, which is the corresponding battery SOH and RUL.

[0053] Step S2: Fusion of different EIS data and corresponding SOH and RUL data.

[0054] Since the EIS and the corresponding SOH and RUL are often tested in different test cycles, their quantity and dimensions are different. In the HS-MH-MoE prediction model, data from different sources need to be fused to ensure that the model input data has the same quantity.

[0055] When implementing EIS testing and the corresponding SOH and RUL assessments, these tests are usually performed independently in different cycles, resulting in differences in the amount and dimensions of the resulting data.

[0056] When building a multi-objective prediction model, the total amount of input data needs to be the same, and the number of training data in each batch must also be the same. In order to build a HS-MH-MoE prediction model, it is necessary to ensure that the total amount of input data is consistent, and the amount of data in each training batch must also be consistent. Therefore, the EIS data from different test environments and their related SOH and RUL data must be fused.

[0057] The fusion method is: calculate the least common multiple of the numbers of different data sets, then upsample each data set so that the number of each data set increases to their least common multiple, and vertically splice the two sets of data after the increase in number to obtain the fused data.

[0058] The process of data fusion involves the following steps: 1. Calculate the lowest common multiple of the number of different data sets; 2. Expand the number of each data set to this least common multiple through repetition; 3. Vertically splice the repeated multiple data sets to obtain fused data.

[0059] Step S3. Step S3. Use a multi-layer multi-head sparse MLP sub-expert model to extract EIS data features: The HS-MH-MoE model includes a multi-layer multi-head sparse expert model; each sub-model at each level is an MLP model; the fused EIS data is used as input data for different MLP models to extract different features; the fused EIS data is used as input data for the gating network model to train the weights of different expert sub-models.

[0060] The HS-MH-MoE model includes a multi-layer multi-head sparse expert model. Each sub-model at each level is a multi-layer perceptron model (MLP) model.

[0061] refer to Figure 3 ,The expert sub-model adopts a multi-layer perceptron model, which is a 3-layer network structure. The input layer receives the input data, the middle layer is a 32-dimensional vector, the activation function is the relu function, and the output layer is a 32-dimensional vector, the activation function is relu, and the output layer is the feature expression of the input data.

[0062] refer to Figure 4 The gated network model adopts a multi-layer perceptron model, which is a 3-layer network structure. The input layer receives input data, the middle layer is a 32-dimensional vector, the activation function is the relu function, and the output layer is a 3-dimensional vector. The activation function is softmax, which corresponds to the weights of the three expert sub-models, and the sum of the three weights is 1.

[0063] Step S4. Generate shared feature representation: At each level, the EIS features (Top-K) extracted by the two MLP models with the highest probability are weighted summed with the corresponding expert weights output by the gating network model to obtain the shared feature representation for the input of the subsequent prediction model.

[0064] The weighted summation process is as follows: The outputs of the three expert sub-models of the first level are respectively recorded as output_1_1, output_1_2 and output_1_3. These three outputs are all high-dimensional vectors. The present invention takes 32 dimensions as an example to represent the feature representation of each sub-expert model for the input data.

[0065] Each gating network model at the first level selects the outputs of the two experts with the highest probability, multiplies them with the corresponding weights and sums them up to obtain the weighted sum of the outputs of each gating network model and sub-expert model at the first level.

[0066] Then, the weighted sum values ​​output by each gated network model and sub-expert model of the first level are averaged to obtain the weighted sum average value of all sub-expert models of the first level.

[0067] Afterwards, the weighted sum average of all sub-expert models in the first layer is used as the input data of the second layer, and is input into each expert sub-model and each gating network model in the second layer respectively.

[0068] The outputs of the three expert sub-models of the second level are output_2_1, output_2_2 and output_2_3 respectively. These three outputs are all high-dimensional vectors. The present invention takes 32 dimensions as an example to represent the feature representation of each sub-expert model for the input data.

[0069] The three gating networks in the second layer independently calculate the weight of each expert sub-model in the second layer. The output of each gating network is a three-dimensional vector, which corresponds to the weights of the three expert sub-models, and the sum of the three weights is 1.

[0070] Then, each gating network model at the second level selects the outputs of the two experts with the highest probability, multiplies them with the corresponding weights and sums them up to obtain the feature representation of the weighted sum of the outputs of each gating network model at the second level and all sub-expert models.

[0071] Step S5. Simultaneously predict battery SOH and RUL: After the second-level shared feature representation, there is a fully connected layer, which further extracts the shared feature representation to obtain the high-level feature expression of EIS. The number of neurons in the fully connected layer is 16, and the activation function is the relu function.

[0072] After the fully connected layer is the output layer of the model. The output layer has two linear regression models, and the corresponding predicted values ​​are the battery capacity (SOH) and the remaining training times (RUL). The activation functions are all linear functions.

[0073] Since SOH and RUL are both continuous values, the loss function of each prediction target is the mean square error (MSE), and the overall loss function of the model is the sum of the two loss functions; the optimizer is rmsprop; the evaluation metric is the mean absolute error (MAE), which is the sum of the absolute values ​​of the differences between the target value and the predicted value. Example

[0074] 1. Prediction data display The present invention tests the effect on a public data set (https: / / github.com / YunweiZhang / ML-identify-battery-degradation), and uses the impedance data therein to predict the battery SOH and RUL. Among them, there are 1358 battery EIS and SOH data, and 525 battery EIS and RUL data. Referring to Tables 1 and 2, EIS has 120 features, and the feature numbers are 1 to 120, respectively. The first 60 features are the real part data of the impedance data corresponding to the frequency from high to low, and the last 60 features are the imaginary part data of the impedance data corresponding to the frequency from high to low.

[0075] In Table 1, f1 to f120 represent EIS frequency characteristics, and Capacity here represents the maximum capacity of the battery, namely SOH.

[0076] In Table 2, f1 to f120 represent EIS frequency characteristics, and RUL here represents the remaining cycle number of the battery.

[0077] 2. Data Fusion The above two data sets are divided into training set and test set in the ratio of 8:2, respectively, to obtain 1086 training data and 272 test data for predicting SOH, and 420 training data and 105 test data for predicting RUL.

[0078] In order to ensure that there is no intersection between the training data and the test data, data fusion is performed on the two training sets and the test set respectively.

[0079] The least common multiple of the 1086 training data for predicting SOH and the 420 training data for predicting RUL is 76020, so the training sets are upsampled respectively so that the number of each training set is increased to 76020.

[0080] The least common multiple of the 272 test data for predicting SOH and the 105 test data for predicting RUL is 28,560, so the test sets are upsampled respectively so that the number of each test set increases to 28,560.

[0081] 3. Training the model The present invention implements the model code on the Keras platform. The multi-target prediction model is trained on the training set, and the effect is tested on the test set. The sample batch size of the model training is 32, and the number of epochs of the model iteration is 50.

[0082] 4. Prediction results display (1) Display of SOH prediction results The mean of the test set of battery SOH is 30.5932, the standard deviation is 4.1284, and the mean of the prediction set is 30.5916, the standard deviation is 4.1157, which shows that the data distribution of the prediction results is basically consistent with the data distribution of the real data.

[0083] The root mean square error between the prediction set and the test set of battery SOH is only 0.2829, indicating that the prediction effect of the model is good.

[0084] The prediction results are shown in Table 3 below. Figure 5 As shown, Figure 5 The solid line in the upper middle represents the true value, the dotted line represents the predicted value, and the solid square represents the error value. The numbers represent the true values. It can be seen from the figure that the prediction effect is good.

[0085]

[0086] (2) Display of RUL prediction results The mean of the test set of battery cycle count is 149.2952, the standard deviation is 105.1953, and the mean of the prediction set is 149.9957, the standard deviation is 105.7829, which shows that the data distribution of the prediction results is basically consistent with the data distribution of the real data.

[0087] The root mean square error between the prediction set and the test set of battery RUL is only 4.504, indicating that the prediction effect of the model is good.

[0088] The prediction results are shown in Table 4. Figure 6 As shown, Figure 6 The solid line in the upper middle represents the true value, the dotted line represents the predicted value, and the solid square represents the error value. The numbers represent the true values. It can be seen from the figure that the prediction effect is good.

Claims

1. A battery SOH and RUL joint prediction method based on HS-MH-MoE model and EIS data, characterized in that: The steps include: Step S1. Test and collect battery data: Use professional impedance spectrum test instrument to collect battery EIS data, and the frequency range needs to be 10 -2 HZ to 10 5 HZ, select 5 to 60 frequency values ​​in the middle of the frequency range, preferably 60 as samples; When collecting data, record the real and imaginary data of the impedance corresponding to each frequency, as well as the target value to be predicted, which is the corresponding battery SOH and RUL; Step S2. Fusing different EIS data and corresponding SOH and RUL data: by calculating the least common multiple of the number of different data sets, and then upsampling each data set so that the number of each data set is increased to their least common multiple, and vertically splicing the two sets of data after the increase in number to obtain the fused data; Step S3. Use a multi-layer multi-head sparse MLP sub-expert model to extract EIS data features: The HS-MH-MoE model includes a multi-layer multi-head sparse expert model; each sub-model at each level is an MLP model; the fused EIS data is used as input data of different MLP models to extract different features; the fused EIS data is used as input data of the gating network model to train the weights of different expert sub-models; Step S4. Generate shared feature representation: In each level, the EIS features extracted by the two MLP models with the highest probability are weighted and summed with the corresponding expert weights output by the gated network model to obtain the shared feature representation, which is used as the input of the subsequent prediction model; Step S5. Simultaneously predicting battery SOH and RUL: After the second-level shared feature representation is a fully connected layer, which further extracts the shared feature representation to obtain a high-level feature expression of EIS; After the fully connected layer is the output layer of the model. The output layer has two linear regression models, and the corresponding predicted values ​​are the battery capacity, i.e., SOH, and the remaining number of training times, i.e., RUL.

2. According to claim 1, a battery SOH and RUL joint prediction method based on HS-MH-MoE model and EIS data is characterized in that: In step S3, the MLP model is an expert sub-model, which adopts a multi-layer perceptron model and is a 3-layer network structure. The input layer receives input data, the middle layer is a 32-dimensional vector, the activation function is the relu function, and the output layer is a 32-dimensional vector. The activation function is relu, and the output layer is the feature expression of the input data.

3. A battery SOH and RUL joint prediction method based on HS-MH-MoE model and EIS data according to claim 1, characterized in that: In step S3, the gated network model adopts a multilayer perceptron model, which is a 3-layer network structure. The input layer receives input data, the middle layer is a 32-dimensional vector, the activation function is the relu function, and the output layer is a 3-dimensional vector. The activation function is softmax, which corresponds to the weights of the three expert sub-models respectively, and the sum of the three weights is 1.

4. The method for jointly predicting battery SOH and RUL based on the HS-MH-MoE model and EIS data according to claim 1, characterized in that: In step S4, the weighted summation process is as follows: the outputs of the three expert sub-models of the first level are respectively recorded as output_1_1, output_1_2 and output_1_3, and these three outputs are all high-dimensional vectors. The present invention takes 32 dimensions as an example, which respectively represent the feature representation of each sub-expert model for the input data; Each gating network model at the first level selects the outputs of the two experts with the highest probability, multiplies them by the corresponding weights and sums them up to obtain the weighted sum of the outputs of each gating network model and sub-expert model at the first level; Then, the weighted sum values ​​output by each gating network model and sub-expert model of the first level are averaged to obtain the weighted sum average value of all sub-expert models of the first level; Afterwards, the weighted sum average of all sub-expert models in the first layer is used as the input data of the second layer, and is input into each expert sub-model and each gating network model in the second layer respectively; The outputs of the three expert sub-models of the second level are output_2_1, output_2_2 and output_2_3 respectively. These three outputs are all high-dimensional vectors. The present invention takes 32 dimensions as an example to represent the feature representation of each sub-expert model for the input data. The three gating networks in the second layer independently calculate the weight of each expert sub-model in the second layer. The output of each gating network is a 3D vector, which corresponds to the weight of the three expert sub-models, and the sum of the three weights is 1. Then, each gating network model at the second level selects the outputs of the two experts with the highest probability, multiplies them with the corresponding weights and sums them up to obtain the feature representation of the weighted sum of the outputs of each gating network model at the second level and all sub-expert models.

Images