A system for encoding a mixture of experts from a neural network, storing latent representations, a motor vehicle, a method and a program based on such a system

Abstract

The invention relates to a system for encoding a mixture of experts from a neural network, the experts being defined by parameters. The system comprises at least one storage means and an autoencoder which includes: - an encoder module (E) receiving the parameters and compressing them by encoding them into latent representations (R), the autoencoder cooperating with said storage means to store the latent representations (R) in said storage means; - a decoder module (D) reconstructing the mixture of experts from the latent representations (R) stored in said storage means. The invention also relates to a motor vehicle, a method, and a program based on such a system. Abstract figure: Fig. 1

Description

Title of the invention: ENCODING SYSTEM FOR A MIXTURE OF EXPERTS FROM A NEURAL NETWORK, STORING LATENTE REPRESENTATIONS, AUTOMOBILE, A METHOD AND A PROGRAM BASED ON SUCH A SYSTEM

[0001] The invention relates to the field of multiple expert model architectures. The invention is at the intersection between neural network systems, multiple expert model architectures, large language models (or "Large Language Models" in English, abbreviated "LLMs"), in the context of different fields, such as driving assistants, and also fields outside of vehicles.

[0002] Large language models have demonstrated innovative capabilities, leading a wave of innovations in generative artificial intelligence that are significantly impacting the technology. It is observed that large language models tend to improve performance as they increase in size. However, the computational costs associated with an increase in the number of model parameters can quickly become prohibitive. The Mixture of Experts (MoE) architecture is an efficient machine learning architecture that integrates multiple multilayer perceptron (MLP) models. These perceptron models are hereinafter referred to as "experts."Instead of relying on a single model to handle all aspects of a task, the MoE expert mix allows for the aggregation of multiple experts, with only a subset active for each input. This selective activation significantly increases computational efficiency, allowing models to scale in size without a corresponding increase in computational costs. Expert mix has been used in large language models such as Mistral's Mixtral 8*7B, Deepseek's Expert Mix, and others, to improve efficiency and performance by transferring input data through appropriate expert networks. These models can also be tailored to specific datasets to achieve performance boosts for target tasks.

[0003] In addition to expert mixtures, model compression techniques such as pruning and quantization are also crucial for reducing computational requirements. Large neural networks. Pruning involves removing significant weights or neurons from the model to simplify its structure and maintain performance. Quantization, on the other hand, reduces the precision of model parameters and activations, typically by converting high-precision values, such as 32-bit floating-point numbers, into lower-precision representations like 8-bit integers. This technique significantly decreases storage requirements and increases inference efficiency. Pruning and quantization can be implemented individually or in combination to compress models efficiently while preserving their accuracy.

[0004] Unfortunately, pruning can lead to a degradation in accuracy because significant weights may be removed. This is especially true in tasks that require detailed contextual understanding. Furthermore, determining which weights to prune is a complex task, and an incorrect selection can severely impact performance.

[0005] Furthermore, quantization, while reducing memory requirements, leads to a loss of accuracy due to rounding errors that can accumulate and degrade model performance. In addition, quantized models often require specialized hardware to achieve effective gains, which limits their practical deployment in standard hardware environments.

[0006] Pruning and quantization may require retraining to restore lost performance, which adds complexity. Furthermore, these methods are task-specific, meaning that models optimized for one task may not generalize well to others, thus limiting their adaptability.

[0007] The invention aims to solve the problem of large storage requirements and computational inefficiency in mixed expert (MoE) neural networks caused by the large number of parameters associated with individual experts. Traditional expert models store and process each expert's parameters separately, leading to redundancy and scalability issues. This invention aims to reduce the memory footprint and improve the efficiency of expert models by minimizing redundant parameter storage and enabling better analysis of relationships between experts.

[0008] To achieve this objective, the invention proposes an encoding system for a mixture of experts from a neural network, the experts being defined by parameters, the system comprising at least one storage means and an autoencoder which includes: - an encoder module receiving the parameters and compressing the parameters by encoding them into latent representations, the autoencoder cooperating with said storage means to store the latent representations in said storage means; - a decoder module reconstructing the mix of experts from the latent representations stored in said storage means.

[0009] In particular, the invention introduces an autoencoder to compress the parameters of each expert into compact latent representations. By replacing the parameters of individual experts with their latent codes and using a shared decoder, it significantly reduces the total number of parameters stored and processed. This modification eliminates redundancy by sharing the decoder among all experts and storing only essential latent information. The technical effect is a more efficient MoE model with reduced storage and computational costs, improved scalability, and the ability to analyze relationships between experts via their latent representations.

[0010] By using latent representations with an autoencoder and employing a shared decoder for all experts, the invention enables a substantial reduction in the total number of parameters. This method reduces redundancy and storage requirements more effectively than traditional approaches, leading to a greater proportion of parameter reduction compared to known solutions that store complete parameter sets for each expert individually. More specifically, for a single expert mixture (MoE) layer, this architecture facilitates a parameter reduction of over 90%, significantly improving model efficiency. If we use a so-called "DeepseekMoE" expert mixture and replace a single MoE layer, we could reduce the total model by 3.2%.

[0011] The delicate balance between model simplification and performance preservation is critically managed in this design. Reducing the number of parameters carries an inherent risk of diminishing model capabilities; however, the architecture has been optimized to ensure that efficiency gains do not come at the expense of accuracy. Compact latent representations are designed to efficiently encapsulate the essential features of each expert. Simultaneously, a shared decoder reconstructs the functionalities with high fidelity.

[0012] Preferably, the encoder module performs convolutions in several layers, in which for at least a first convolution layer, the encoder module implements the following operation: ho = ReLU ( ConvZch-^ ( x ) ) where h0 is the number of output channels of said layer; in which the decoder module performs an inverse operation for each of the convolution layers, and in which, for said first convolution layer, the decoder module implements the following operation: dn = ReLU(PixelShuffle(Conv2dWhn(z))).

[0013] This allows for a first encoding operation.

[0014] Preferably, for at least a second convolution layer, the encoder module implements the following operation: hj = ReLU ( Conv2dhk । ) ) for each second layer i, where U, and hj are input and output channels of the i-th layer, respectively, in which the decoder module performs an inverse operation for each second convolution layer, implementing the following operation: dj - LeakyReLU ( PixelShuffle ( Conv2d4h;+rh. ( di+1 ) ) ).

[0015] This allows for a second encoding operation.

[0016] Preferably, for a final convolution layer, the encoder module performs a latent spatial representation of dimension n, by implementing the following operation: z = LeakyReLU ( Con v2d]1r ( hn ) ) and in which for each layer I, the final reconstruction x restores the data to their initial input dimensions via the following operation: x = PixelShuffle(Conv2d4hi)^(d0)).

[0017] This allows for a third encoding operation.

[0018] Preferably, the encoding system further includes a means for training the autoencoder minimizing an error in reconstructing parameters from latent representations.

[0019] This allows for more precise encoding.

[0020] Preferably, said error is defined by the mean squared error (MSE): afv 0 A — 1 Vn IU £ I! 2where n is the number of samples in the set of x, a 7 — nj H *4 “ || data.

[0021] The invention further relates to a motor vehicle comprising an encoding system according to the invention. In particular, it is a vehicle driving assistance system.

[0022] Another object of the invention relates to a method for encoding a mixture of experts from a neural network, using an encoding system according to the invention, the encoding method comprising the following steps: - a self-encoding step in which the parameters are compressed by encoding them into latent representations, - a storage stage in which latent representations are stored; - a decoding stage in which the mix of experts is reconstructed from the latent representations.

[0023] Preferably, the encoding process further includes an error minimization step for reconstructing parameters from latent representations.

[0024] The invention also relates to a computer program comprising program code instructions for executing the steps of the encoding process according to the invention, when said program is running on a computer.

[0025] The invention will be further detailed by describing non-limiting embodiments, and based on the accompanying figures illustrating variants of the invention, including: - [Fig.l] schematically illustrates the structure of the autoencoder of the encoding system according to a preferred variant of the invention; - [Fig.2] schematically illustrates an example of the logic of a network mixing experts suitable for implementing the invention.

[0026] The invention presents a novel method for compressing and optimizing Mixed Expert (MoE) neural network architectures by using an autoencoder E to encode each expert's parameters into compact latent representations R. Instead of storing complete parameter sets for each expert, the method replaces them with their respective latent codes (representations R) and uses a shared decoder D to reconstruct the expert parameters when necessary. This approach significantly reduces the total number of parameters, decreases memory and storage requirements, while ensuring model performance. Furthermore, it allows for the analysis of relationships between experts through their latent representations R, thus providing information on the model's internal structure and facilitating further optimizations.

[0027] The mixture of experts (MoE) architecture is a powerful neural network model designed to handle complex tasks by dividing a problem space among several specialized subnetworks, called experts. Each expert focuses on a specific subset of the input data, and a control mechanism determines each expert's contribution to the final result.

[0028] In a mixture of experts (MoE) model, we define N experts, denoted E; where i = 1, 2, ..., N, illustrated in [Fig. 2]. A control function G(x), operating on the input x, determines the weight of each expert. This control network generates a probability distribution over the experts, guiding the concentration of the model, as shown in [Fig. 1]. The final result of the MoE model can be calculated as a weighted sum of the experts' results:

[0029] Q x ) gxj where Gi(x) represents the control weight for the expert, E; and E;(x) is the result of the expert E; given the input x. Regarding the architecture of the autoencoder E detailed in [Fig. 1], the encoder portion of the autoencoder E is designed to transform high-dimensional input data (hidden input IH) into a lower-dimensional latent representation R (hidden output OH). This process is achieved through several layers of convolution functions Cv and nonlinear activation, as illustrated in [Fig. 1]. The initial layer performs the first convolution operation in the encoder: ho = ReLU(ConvZdj^(X)) where h0 is the number of output channels of the first layer.

[0030] The following intermediate layers are defined by: hj — ReLU ( Conv2dh. ( hj-i ) ) for each layer i, where h, । and h; are respectively the input and output channels of the i-th layer.

[0031] The final stage of the encoder projects the output of the last encoder layer hn into the representation of the 1-dimensional latent space: z = LeakyReLU(Conv2dhn-^(hn))

[0032] Regarding the details of decoder D, the decoder part of the autoencoder is responsible for reconstructing the input data from its encoded form by reversing the encoder operations via oversampling and convolution layers, as illustrated in [Fig. 1]. The reference PS designates, for example, a function called "PixelShuffle". The initial oversampling in decoder D extends the encoded latent space z towards the output dimensions: dn = ReLU ( PixelShuffle ( Convld]-^ ( z ) ) )

[0033] The intermediate layers of the decoder progressively reconstruct the higher-dimensional data: d, = LeakyReLU ( PixelShuffle ( Conv2d4hj+i-^h; ( di+1 ) ) ) for each layer i.

[0034] The final reconstruction x restores the data to their original input dimensions: x = PixelShuffle ( Conv2d4ho^3 ( d0 ) )

[0035] Regarding the training of the autoencoder E, it allows minimizing the reconstruction error between the input and the output, defined by the mean squared error (MSE): a / „ * \ _ 1 V" II „ * l| 2where n is the number of samples in the set of a / — n2Lj=] || |j data.

[0036] Regarding the overall methodology, it integrates the autoencoder within a mix of MoE experts to efficiently compress expert parameters.

[0037] The key steps include the extraction of the expert parameters 0j and the design and training of the autoencoder architecture which uses convolutional layers to map these parameters to the latent codes (Representations R). A lower-dimensional latent space Z is defined and a shared decoder D reconstructs the expert parameters from these latent codes (R).

[0038] Model compression replaces the complete set of parameters 0! of each expert with its latent code zi(R), resulting in a significant reduction in parameter size. During inference or subsequent training, the reconstructed expert parameters are computed on the fly using the shared decoder D. The model computes the trigger weights Gj(x) and the output using the reconstructed experts: y = L^Gi(x) - E^êi)

[0039] Regarding the experimental results, the performance of the compressions obtained, and the impact on the accuracy of the model, were analyzed.

[0040] We started with a basic Deepseek MoE expert mixture model equipped with several layers, each containing 64 experts, where each expert had approximately 8.65 million parameters, totaling 553.65 million parameters per MoE layer.

[0041] We used our autoencoder model for this experiment, which consists of an encoder E with 11.24 million parameters and a decoder D with 26.04 million parameters. The parameters of each expert were encoded in a compact latent representation R of approximately 20,000 parameters (negligible compared to the total parameter size).

[0042] The results show that replacing a MoE layer with its latent representations R reduces the model parameters by about 527.61 million (553.65 million in total for a MoE layer, the number of decoder parameters to be subtracted).

[0043] Moreover, the results show that when three MoE layers are replaced, the compression effect is even more significant, with a total reduction of about 1.61 billion parameters (553.65 million multiplied by 4, the number of decoder parameters being subtracted).

[0044] To quantify the impact of our compression technique, we evaluated the model's performance using the perplexity of the Wikitext dataset as the primary measure. The table below shows the perplexity scores for the base and compressed models, as well as the number of compressed parameters:

[0045] Table 1: Comparison of the perplexity of different MoE model configurations with compression details. Model configuration Replaced layers Perplexity Reduced parameters (in millions) Base reference (original model) None 9.2937 0 Compressed model 10th layer 9.5541 527.61 (3.21% of the model) Compressed model 10th to 13th layers 10.4457 2188.56 (12.85% of the model)

[0046] Experimental results indicate that our compression method can significantly reduce the overall model size while having only a modest impact on model performance. The slight increase in perplexity with more replaced layers suggests a trade-off between model size and accuracy. However, the significant decrease in the number of parameters underscores the potential of our approach for deploying large MoE models in resource-constrained environments.

[0047] The proposed method offers an innovative solution to the challenges posed by large-scale MoE models. By integrating an autoencoder to compress expert parameters into latent R representations and using a shared D decoder, the method significantly reduces the model size while maintaining performance. This approach allows for a trade-off between memory consumption and LLM performance, which is useful for edge deployment.

[0048] The invention further relates to an encoding method and a corresponding program. The program can be loaded into the memory of a control unit of a motor vehicle serving as an on-board driving assistant or into an external development unit.

Claims

Demands

1. System for encoding a mixture of experts of a neural network, the experts being defined by parameters, the system comprising at least one storage means and an autoencoder which includes: - an encoder module (E) receiving the parameters and compressing the parameters by encoding them into latent representations (R), the autoencoder cooperating with said storage means to store the latent representations (R) in said storage means; - a decoder module (D) reconstructing the mixture of experts from the latent representations (R) stored in said storage means.

2. Encoding system according to claim 1, characterized in that the encoder module (E) performs convolutions in several layers (Cv), in that for at least one first convolution layer, the encoder module (E) implements the following operation: 1¾ = ReLU ( Conv2d3->ho ( x ) ) where h0 is the number of output channels of said layer; in that the decoder module (D) performs an inverse operation for each of the convolution layers, and in that for said first convolution layer, the decoder module (D) implements the following operation: dn - ReLU ( PixelShuffle( Conv2dH4hn ( z ) ) ).

3. An encoding system according to claim 2, characterized in that for at least one second convolution layer, the encoder module (E) implements the following operation: hi = ReLU(Conv2dh.(hj)) for each second layer i, where hi and hi are input and output channels of the i-th layer, respectively, in that the decoder module (D) performs an inverse operation for each second convolution layer, implementing the following operation: d| = LeakyReLU(PixelShuffle(Conv2d4h.irh.(di+1)))

4. Encoding system according to claim 3, characterized in that for a last convolution layer (hn), the encoder module (E) performs a latent spatial representation of dimension (1), by implementing the following operation: Z = LeakyReLU ( Con v2dhir*] ( hn ) ) and in that for each layer I, the final reconstruction x restores the data to their initial input dimensions via the following operation: x = PixelShuffle ( Conv2d4ho-»3 ( d0 ) ).

5. Encoding system according to any one of claims 1 to 4, further comprising a means for training the autoencoder minimizing an error in reconstructing parameters from the latent representations (R).

6. Encoding system according to claim 5, characterized in that said error is defined by the mean squared error (MSE): afv OA - IV11 II v 0 II 2where n is the number of samples in A t A, X ) - n ^.=1Xj - Xi || the dataset.

7. Motor vehicle comprising an encoding system according to any one of claims 1 to 6.

8. A method for encoding a mixture of experts from a neural network, using an encoding system according to any one of claims 1 to 6, the encoding method comprising the following steps: - a self-encoding step in which the parameters are compressed by encoding them into latent representations (R), - a storage step in which the latent representations (R) are stored; - a decoding step in which the mixture of experts is reconstructed from the latent representations (R).

9. Encoding method according to claim 8, further comprising an error minimization step for reconstructing parameters from latent representations (R).

10. Computer program comprising program code instructions for carrying out the steps of the encoding process according to any one of claims 8 to 9, when said program is running on a computer.

Images