Method and device for controlling the rendering of text generated from textual data

Abstract

The present invention relates to a method and device for controlling the rendering of text generated by a trained language model based on an expert model architecture including linear gate units configurable by weights and biases. During a learning phase, the method comprises: - obtaining (41) a permutation matrix of weights and biases of linear gate units to maximize similarity between the weights and biases of linear gate units of a first expert model and the weights and biases of linear gate units of a second expert model; - obtaining (42) permuted weights and biases by permuting the weights and biases of the linear gate units of the second expert model according to the permutation matrix obtained; and - replacing (43) the weights and biases of the linear gate units of the second expert model with the permuted weights and biases obtained. Figure for the abstract: Figure 4

Description

Title of the invention: Method and device for controlling the rendering of text generated from textual data technical field

[0001] The present invention relates to generative artificial intelligence, that is to say, to artificial intelligence systems capable of generating text, images or other media in response to prompts.

[0002] The present invention lies at the intersection of neural networks, architectures composed of multiple expert models, and large-scale language models, applicable to multiple fields such as driver assistance.

[0003] In particular, the present invention relates to the learning of these large language models. Technological background

[0004] Modern vehicles incorporate an increasing number of on-board systems that contribute to vehicle control by assisting the driver and / or to the comfort of the driver and passengers. Some of these systems are known as ADAS (Advanced Driver-Assistance System).

[0005] Some AD AS systems can use artificial intelligence to implement generators and other natural language processing according to particular road situations.

[0006] The emergence of large language models (LLMs) has recently revolutionized the field of natural language processing and generative artificial intelligence because these models are capable of generating texts close to those a human can generate, answering questions, translating texts into different languages, and even writing poetry. They have been used in various applications ranging from chatbots to writing assistants (Metz, C. (2021). AI language model GPT-3 reproduces human-like biases. MIT Technology Review).

[0007] It has been observed that large language models improve their performance as they grow larger, but the computational costs associated with increasing the parameters of these language models can quickly become prohibitive. Mix of Experts (MoE) (Fedus W, Dean J, Zoph B. A review of sparse expert models in deep learning[J]. arXiv preprint arXiv:2209.01667, 2022) is a machine learning architecture that aims to solve this computational cost problem.

[0008] A neural network organized according to a MoE architecture is a neural network in which a set of neural network parameters is distributed among so-called "expert" models. During the learning and inference phases, the MoE architecture routes input examples to specific expert model(s). Consequently, each example interacts with only a subset of the neural network parameters, unlike a more conventional approach where the entire neural network is used for each input during the learning and inference phases.

[0009] MoE architectures combine several expert models in the form of multilayer perceptrons (MLPs). A multilayer perceptron is a primary artificial neural network model consisting of at least three layers: an input layer, at least one hidden layer, and an output layer. It requires that the units of two adjacent layers be closely connected, which is why a large number of weighting parameters must be trained. Instead of relying on a single model to handle all aspects of a problem, MoE architectures allow for a significant increase in the size of large language models while maintaining reasonable computational costs, since not all expert models are active for every input.MoE architectures are widely used in current large-scale language models to improve the efficiency and performance of language understanding tasks by dynamically routing data through networks of specific expert models.

[0010] These large language models organized according to MoE architectures can be refined with relevant datasets in order to obtain better performance for specific tasks.

[0011] Before refining such models during a training phase, permutation symmetry alignment (Ainsworth, SK, Hayase, J., & Srinivasa, S. (2023). Git Re-Basin: Merging Models Modulo Permutation Symmetries. arXiv preprint: 2209.04836) can significantly reduce the number of iterations required for training. Indeed, the neural network can exploit inherent symmetries in its parameters, allowing it to align them with each other so that different sets of parameters lie within the same parameter subset. This greatly simplifies the problem of optimizing the neural network parameters without altering the output of the neural network. Recent research has provided theoretical guarantees for equi-variant permutation neural networks, demonstrating their efficiency in tasks that require processing the weights of multilayer perceptrons. and convolutional neural networks (Allan Zhou, Kaien Yang, Kaylee Burns, Adriano Cardace, Yiding Jiang, Samuel Sokota, J. Zico Kolter, Chelsea Finn. (2023). Permutation Equivariant Neural Functionals .arXiv preprint: 2302.14040).

[0012] Permutation symmetry refers to the property whereby certain neurons or groups of neurons in a neural network can be interchanged without altering the overall function of the neural network. This symmetry arises from the fact that the order of neurons within the layers of the neural network that perform element-wise operations does not affect the output of the neural network, provided that subsequent layers adapt accordingly. Permutation symmetry is also called neuronal alignment.

[0013] Figure 1 illustrates an example of loss in a partially trained neural network. The grayscale gradient indicates the values ​​of the loss function, with darker grays representing the highest losses and lighter grays the lowest losses. Lines 10 and 11 show optimization (minimization) paths of a loss function, where a gradient descent algorithm moves toward regions where the losses are lowest. The end of line 10 illustrates a point where permutation symmetry is implemented. Line 12 indicates that neurons or layers of the neural network have been rearranged (aligned) by permutation without affecting the overall function of the neural network.This rearrangement of neurons allows optimization to proceed efficiently from a new position in the parameter space, thus avoiding local minima and improving convergence during subsequent training of the neural network.

[0014] Permutation symmetries in neural networks can be expressed mathematically as follows.

[0015] Consider a classical neural network with L layers. Let xe pA be the input data of the neural network and zG p4 the activations of layer k. For a classical multilayer perceptron (MLP) model, a so-called forward pass f(x, 3) is defined as follows:

[0016]

[0017] [Math.l] / (r,0) =z / +i = <7(W z z z + ^) 1=1, ...,L in which <?(.) est une fonction d’activation par élément e CS( a weight matrix, g p'L; cst a bias vector and @ = |} L represents the set of parameters of the neural network.

[0018] Let us now consider p a permutation matrix representing a rearrangement of neurons in the layer.

[0019] A new set of parameters 0' can then be defined for which:

[0020] [Math.2] wy = pwt bi - P^ = Wl + ïPT

[0021] with pT the transpose of the permutation matrix P.

[0022] Since a permutation P in layer 1 is countered by an inverse permutation in layer l + 1 through w pT and ppT = J with / the identity matrix, the output of the neural network remains unchanged, i.e. that:

[0023] [Math.3] / (r, 0) = / (r,0')

[0024] It is known that a multilayer perceptron is formed of gate linear units. Gate linear units introduce a gate mechanism that allows for more expressive interactions within a neural network (Shazeer N. Glu variants improve transformer[J]. arXiv preprint arXiv:2002.05202, 2020).

[0025] Figure [Fig.2] schematically illustrates a linear unit with 20 gates.

[0026] The linear gate unit 20 includes a module 21 which implements a first linear transform of input data 23 of the linear gate unit 20. For example, the first linear transform is implemented by a first weight matrix and output data of the module 21 are obtained by multiplying a data vector formed by the input data 23 by the first weight matrix.

[0027] The linear gate unit 20 further includes a module 22 which implements a second linear transform of the input data 23 of the linear gate unit 20. For example, the second linear transform is implemented by a second weight matrix and output data of the module 22 are obtained by multiplying a data vector formed by the input data 23 by the second weight matrix.

[0028] The linear gate unit 20 also includes a module 24 which implements an activation function taking as input the output data of the module 22.

[0029] The linear gate unit 20 also includes a module 25 which implements a multiplication of the output data of the module 21 and the output data of the module 24.

[0030] Finally, the linear gate unit 20 includes a module 26 which implements a third linear transform of the output data from the module 25. For example, the

[0031]

[0032]

[0033]

[0034]

[0035]

[0036]

[0037]

[0038]

[0039]

[0040] The third linear transform is implemented by a third weight matrix and output data from module 26 (output data from the linear unit with gates 20) are obtained by multiplying a data vector formed by the output data from module 25 by the third weight matrix. The global function GLU(x) implemented by the linear unit with gates 20 can be defined by: [Math.4] GLU{x) = [(7^^ + ^)0^2 + ¼) M + è3 with [Math.5] XGP^'" an input data vector of the linear unit with gates 20; [Math.6] jy x is the first weight matrix allowing a projection of the vector formed by the input data 23 onto a hidden data vector. This projection is called a "gate projection". [Math.7] W- GP^' ” x is the second weight matrix allowing a projection of the vector formed by the input data 23 onto a hidden data vector. This projection is called "up projection" in English; [Math.8] bp b2 SP^' are vectors of bias; [Math.9] jy x ^out is the third weight matrix allowing a projection of the vector formed by the hidden data onto the output data vector 27 (output out data vector). This projection is called "down projection" in English; [Math. 10] b3 8 P4™1 is a vector of bias; [Math. 11] HJ is the activation function per element (for example a rectified linear unit in English ReLU for Rectified Linear Unit);

[0041] [Math. 12] is an element multiplication operator;

[0042] [Math. 13] ^^hidden dma are respectively dimensions of the input data vector 23, the hidden data vector and the output data vector 27.

[0043] In a dense multilayer perceptron model (not including linear gated units), the layers are stacked so that the output of one layer directly feeds into the next. Permutations are generally applied layer by layer, affecting only one layer at a time. In contrast, linear gated units process the input data vectors in parallel (modules 21 and 22-24) before combining them by element-wise multiplication (module 25), which introduces a control mechanism that modulates the information flow.

[0044] Due to the parallel structure and element-wise multiplication in linear gated units, permuting neurons in one channel without a corresponding permutation in the other parallel channel disrupts alignment and affects the output data of the linear gated unit and therefore of the neural network. Consequently, unlike dense multilayer perceptron models (which do not include linear gated units) where permutations can be applied independently to each layer, any permutation in a linear gated unit must be applied consistently to all three relevant linear layers.

[0045] Current permutation symmetry methods cannot be directly applied to the layers of a MoE architecture that combines several expert models in the form of multilayer perceptrons (MLPs) composed of linear gated units. This is because these permutation symmetry methods are primarily designed for dense multilayer perceptron architectures and assume a sequential flow of data through the layers, where the output of each layer is fed into the next. Permutations are then applied independently to each layer, and alignment focuses on matching layers between models. However, the layers of current large-scale model language MoE architectures combine linear gated units, which implies the simultaneous implementation of linear transformations that interact with each other through element-wise operations (Shazeer N. Glu variants improve transformer[J]. arXiv preprint arXiv:2002.05202, 2020).

[0046] The parallelism of linear gated units thus introduces interdependencies between layers that current permutation symmetry methods cannot handle. Permuting the neurons of one channel without simultaneously permuting the corresponding neurons of parallel channels disrupts the alignment and alters the output of the neural network. Consequently, the independent application of permutation to individual layers, as is the case in dense multilayer perceptron models, is inefficient for MoE architectures combining several expert models that take the form of multilayer perceptrons formed from linear gated units. Summary of the present invention

[0047] One object of the present invention is to solve at least one of the problems of the technological background described above.

[0048] According to a first aspect, the present invention relates to a method for controlling the rendering, by an embedded system of a vehicle, of text generated by a trained language model when textual data from at least one embedded system of a vehicle are presented as input to the language model, the language model being based on an architecture of so-called expert models, each expert model including linear units with gates that are parameterizable by weights and biases, said method comprising the following steps during a language model training phase: - obtaining a permutation matrix of weights and biases of linear gated units to maximize a similarity between weights and biases of linear gated units of a first expert model and weights and biases of linear gated units of a second expert model; - obtaining permuted weights and biases by permuting the weights and biases of the linear gate units of the second expert model according to the permutation matrix obtained; - replacing the weights and biases of the linear gate units of the second expert model with the permuted weights and biases obtained.

[0049] The method allows for the permutation alignment of the weights and biases of expert models used by a large language model and based on linear gated units, by introducing a unified permutation matrix that acts simultaneously on all parallel layers of these linear gated units. This method overcomes the challenges posed by the parallel structure of the linear gated units and ensures that the output of the neural network remains unchanged after the permutation. By simultaneously applying a single permutation to several layers of the expert models, the method differs from the usual methods used in multilayer perceptron networks. The permutation alignment implemented by the method facilitates large-scale language model post-training operations, such as merging, compression, and model tuning, which are important for deploying language models using a large-scale MoE architecture.

[0050] Weight and bias permutation alignment of expert models used by large language models allows for specialization in different aspects of vehicle-related tasks. For example, some expert models can focus on understanding technical language, while others handle user interaction or safety-critical vehicle commands. Process-based permutation alignment optimizes the performance of the large language model in specialized domains such as automotive engineering, customer support, or embedded systems.

[0051] The method allows the weights and biases of a reference expert model (first expert model) to be aligned with those of a target expert model (second expert model) by permutation, without affecting the output of the neural network. This facilitates the merging, compression, or adjustment of the language model for downstream tasks. By using a permutation matrix that preserves orthogonality properties, the method ensures that the rearrangement of neurons in the layers of the expert models does not modify the output of the neural network. The initial functionality of the language model is thus maintained despite internal reconfigurations (alignments).

[0052] Aligning the weights and biases of expert models by permuting facilitates a more uniform distribution of the parameters of the language model's neural network. This alignment simplifies the learning phase, accelerates the convergence of the learning algorithm, and improves learning efficiency by reducing the complexity of algorithms that often hinder gradient descent algorithms used for training deep neural networks.

[0053] From a resource perspective, aligning the weights and biases of expert layers by permuting them allows for more efficient use of computing resources. By directing only relevant queries to the specialized expert models of the language model, unnecessary data processing can be reduced, resulting in faster and more efficient operations. This is particularly advantageous in real-time automotive applications such as autonomous driving or real-time in-cockpit diagnostics. It can also lead to cost savings by reducing the energy and infrastructure required to run large language models.

[0054] Moreover, the method has a broad scope and can be used in any context where the development of large language models is a desirable key performance indicator. For example, functions such as Driver assistance and autonomous driving are possible use cases for which this process can be implemented.

[0055] According to a particular and non-limiting embodiment of the present invention, the weights of each linear gated unit are transformation matrix weights.

[0056] According to a particular and non-limiting embodiment of the present invention, each linear unit with gates implements: a first linear transformation of an input data of the linear unit with gates by using a first linear transformation matrix formed of weights and biases; a second linear transformation of said input data of the linear gated unit by using a second linear transformation matrix formed of weights and biases; an activation function providing an intermediate output data when an output data from the second linear transformation is present at the input of the activation function; a multiplication of an output value from the first linear transformation and an output value from the activation function; and a third linear transformation of a given result of said multiplication by using a third linear transformation matrix formed of weights and biases.

[0057] According to a particular and non-limiting embodiment of the present invention, obtaining the perturbation matrix includes a step of calculating a total similarity matrix by concatenating a first similarity matrix evaluating a similarity between the first transformation matrix of the first expert model and the first transformation matrix of the second expert model, a second similarity matrix evaluating a similarity between the second transformation matrix of the first expert model and the second transformation matrix of the second expert model, and a third similarity matrix evaluating a similarity between the third transformation matrix of the first expert model and the third transformation matrix of the second expert model.

[0058] According to a particular and non-limiting embodiment of the present invention, the perturbation matrix is ​​obtained so that the total similarity matrix is ​​close to the identity matrix.

[0059] According to a particular and non-limiting embodiment of the present invention, the activation function is an activation function by element.

[0060] According to a second aspect, the present invention relates to a device comprising a memory associated with a processor configured for the implementation of the steps of the process according to the first aspect of the present invention.

[0061] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

[0062] According to a fourth aspect, the present invention relates to a computer program which includes instructions adapted for carrying out the steps of the process according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0063] Such a computer program may use any programming language, and be in the form of source code, object code, or an intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0064] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the process according to the first aspect of the present invention.

[0065] On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium can include a storage means, such as a ROM, RAM, CD-ROM or a microelectronic circuit-type ROM, or a magnetic recording means or a hard disk drive.

[0066] On the other hand, this recording medium can also be a transmissible medium such as an electrical or optical signal, such a signal being able to be transmitted via an electrical or optical cable, by conventional or radio frequency, by self-directing laser beam, or by other means. The computer program according to the present invention can, in particular, be downloaded from an Internet-type network.

[0067] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to execute or to be used in the execution of the process in question. Brief description of the figures

[0068] Other features and advantages of the present invention will become apparent from the description of the particular and non-limiting embodiments of the present invention below, with reference to the attached Figures 1 to 5, in which:

[0069] [Fig-1] illustrates an example of loss of a neural network that has undergone partial learning.

[0070] [Fig.2] schematically illustrates a linear unit with gates.

[0071] [Fig.3] schematically illustrates a diagram of the steps of a rendering control process for text generated from input textual data from at least an embedded system of a vehicle according to a particular and non-limiting embodiment of the present invention.

[0072] [Fig.4] schematically illustrates a diagram of the steps of a method for permuting weights and biases of linear gate units during a learning phase of a language model, according to a particular and non-limiting embodiment of the present invention.

[0073] [Fig.5] schematically illustrates a device configured to control the rendering of text generated from input textual data from at least one embedded system of a vehicle according to a particular and non-limiting embodiment of the present invention. Description of examples of achievements

[0074] A method and a rendering control device for text generated from input textual data from at least one vehicle on-board system will now be described in the following with joint reference to Figures 1 to 5. The same elements are identified with the same reference signs throughout the following description.

[0075] The terms "first," "second" (or "firsts," "seconds"), etc., are used in this document by arbitrary convention to allow for the identification and distinction of different elements (such as operations, means, etc.) implemented in the embodiments described below. Such elements may be distinct or correspond to a single element, depending on the embodiment.

[0076] A vehicle, according to the present invention, corresponds, for example, to a vehicle with an internal combustion engine, with electric motor(s), or even a hybrid vehicle with an internal combustion engine and one or more electric motors. The vehicle thus corresponds, for example, to a land vehicle, such as a car, a truck, a bus, or a motorcycle.

[0077] The vehicle includes at least one embedded system which is configured to provide textual data, i.e. either sequences formed from one or more words of natural language.

[0078] According to a particular and non-limiting embodiment of the present invention, the textual data may be representative of a state of at least one component of the vehicle and / or of a road situation in which the vehicle is located.

[0079] According to a particular, non-limiting embodiment of the present invention, one of these embedded systems may include a touchscreen and memory configured for input and recording natural language words. Another example might be a voice recording system comprising a microphone and memory.

[0080] According to a particular and non-limiting embodiment of the present invention, an on-board vehicle system may be an ADAS system. For example, the system may be a road sign detection system that provides word sequences as soon as it detects a road sign. For example, a word sequence 'speed limit 80 km / h' may be provided by this ADAS system as soon as an 80 km / h speed limit sign is detected. Another ADAS system may provide word sequences as soon as weather conditions change.

[0081] These examples of AD AS systems are given only by way of example and the present invention extends to any type of AD AS system which would provide sequences of words according to a state of at least one organ of the vehicle and / or a road situation in which the vehicle is located.

[0082] According to a particular and non-limiting embodiment of the present invention, an embedded vehicle system may be a display system comprising means for interfacing with a vehicle user that enable the retrieval of textual data and the audio, visual, or audiovisual rendering of text. These interface means may, for example, include a touchscreen, a microphone and / or a speaker controlled by a computer, or a voice control system comprising a microphone and / or a speaker. The embedded system may include a memory configured to store words of natural language.

[0083] Fig. 3 schematically illustrates a diagram of the steps of a method for controlling the rendering of text generated from input textual data from at least one embedded system of a vehicle according to a particular and non-limiting embodiment of the present invention.

[0084] In a step 31, the input textual data is obtained from at least one embedded system.

[0085] For example, a user can use a speech recognition system to ask a question about a feature of an AD AS system. The speech recognition system then transcribes this question into a sequence of text data.

[0086] In a step 32, an embedded system of the vehicle implements a large-scale language model trained to generate text when the textual data obtained (step 31) are presented as input to the large-scale language model.

[0087] In a step 33, an on-board system of the vehicle is configured to provide a rendering of this generated text in either audio, visual or audio-visual form.

[0088] For example, this system may include means for converting text into audio, visual, or audio-visual signals. It may include, for example, loudspeakers or a screen.

[0089] The large-scale language model is organized according to an architecture MoE combines several expert models in the form of multilayer perceptrons. Each expert model is trained to be dedicated to a specific task. Each expert model is composed of linear gated units, meaning that each expert model consists of three layers: an input layer, a hidden layer, and an output layer. These layers implement three linear transformations in the form of weight matrices, as explained in relation to [Fig. 2].

[0090] To align the weights and biases of the linear gated units without modifying the output data of the neural network, a single permutation is used to simultaneously swap the weights and biases of the three layers of linear gated units of the language model's expert models. The use of a single permutation is justified by the interconnected nature of the parallel paths of the linear gated units.A simultaneous permutation means that the permutation is applied to the weights and biases of the first projection (gate projection), the second projection (up projection) and the third projection (down projection) of the linear gated units of an expert model.

[0091] Fig. 4 schematically illustrates a diagram of the steps of a method for permuting weights and biases of linear gate units during a language model learning phase, according to a particular and non-limiting embodiment of the present invention.

[0092] The language model is trained during a learning phase. During this learning phase, one or more permutations of neurons may be implemented. Each implementation of a permutation of neurons employs the process described below.

[0093] In a step 41, a permutation matrix P of gated linear unit weights and biases is obtained to maximize a similarity between gated linear unit weights and biases of a first expert model of the language model architecture and gated linear unit weights and biases of a second expert model.

[0094] In a step 42, permuted weights and biases are obtained by permuting the weights and biases of the linear gate units of the second expert model according to the permutation matrix P obtained.

[0095] According to a particular and non-limiting example of the present invention, the weights of each linear gated unit are transformation matrix weights.

[0096] For example, the weights of a linear gated unit of the input layer of the first (respectively second) expert model are grouped into a first weight matrix jy G xdhu <kn, les poids de la couche cachée d’une unité linéaire à portes de la couche cachée du premier (respectivement deuxième) modèle expert sont regroupés in a second weight matrix yy pdin x and the weights of a linear gated unit of the output layer of the first (respectively second) expert model are grouped in a third weight matrix yy^ ex dw. A linear gated unit of the first (respectively second) expert model includes bias vectors g as explained previously.

[0097] The first perturbed weight matrix yy' is then obtained by:

[0098] [Math. 14]

[0099] The second permuted weight matrix VU is then obtained by:

[0100] [Math. 15] •>

[0101] The third permuted weight matrix yy / is then obtained by:

[0102] [Math. 16] W3 = W2P t •>

[0103] and the permuted bias vectors are then obtained by:

[0104] [Math. 17] = P^ And [Math. 17] b2 - Pb2

[0105] In a step 43, the weights and biases of the linear gate units of the second expert model are replaced by the permuted weights and biases obtained.

[0106] For example, the weight matrices W2 and W3 respectively, are replaced by the permuted weight matrices W2' and W3' respectively and the bias vector &i, respectively b2 and ^3, is replaced by the vector ô / , respectively b2r and b3r.

[0107] According to a particular and non-limiting embodiment of step 41, a total similarity matrix is ​​calculated by concatenating a first similarity matrix evaluating a similarity between the first transformation matrix of the first expert model and the first transformation matrix of the second expert model, and a second similarity matrix evaluating a similarity between the second transformation matrix of the first expert model and the second matrix transformation of the second expert model and a third similarity matrix evaluating a similarity between the third transformation matrix of the first expert model and the third transformation matrix of the second expert model.

[0108] According to a particular and non-limiting embodiment of the present invention, the perturbation matrix P is obtained so that the total similarity matrix is ​​close to the identity matrix.

[0109] For example, the permutation matrix P of weights and biases of linear units with gates is obtained by an algorithm called the Hungarian algorithm or method, also known as the Kuhn-Munkres algorithm. This algorithm determines the permutation matrix P in polynomial time. It is an algorithm that allows finding a perfect matching of optimum (minimum or maximum) weights in a bipartite graph whose edges are associated with weights. The algorithm can be presented in several forms. The first is a fairly combinatorial presentation using matrices, the second is a presentation within the framework of linear optimization (https: / / fr.wikipedia.org / wiki / Algorithme_hongrois#Description_par_l'optimisation_lin%C3%A9aire). Basically, this algorithm considers two sets of weights as input and the permutation matrix P as output.In the context of the present invention, the first set of weights corresponds to the weights of the first expert model, and the second set of weights corresponds to the weights of the second expert model. The algorithm is iterative. At each iteration, a current weight permutation is selected from a set of possible permutations. For each pair of weight matrices consisting of a weight matrix of the first expert model and the corresponding weight matrix of the second expert model, a weight permutation axis is determined within said weight matrices based on the current weight permutation. This axis determines which weights in the weight matrix are permuted.A first intermediate weight matrix is ​​then formed from the weights of the first set of weights (first expert model), and a second intermediate weight matrix is ​​obtained by permuting the weights of the second set of weights (second expert model) along the permutation axis. The first and second intermediate weight matrices are resized to compute a dot product of these two intermediate weight matrices. A cumulative permutation cost (taking the form of a tensor) is then updated from this dot product. Another iteration is then performed with a new weight matrix from the first expert model and the corresponding weight matrix from the second expert model, and the cumulative permutation cost is updated at the end of each iteration.When all the weight matrices of the first and second expert model are considered (in this case 3), an element of the cumulative permutation cost is determined that minimizes the calculated scalar products. between the three weight matrices. If the value of this element is less than the current value of a previously calculated permutation cost element, then the algorithm terminates and the output permutation matrix P is determined based on the current permutation axis and that permutation cost element. If this element's value is greater than the current value of a previously calculated permutation cost element, another permutation is considered.

[0110] According to a particular and non-limiting embodiment of the present invention, the activation function is an activation function by element.

[0111] The method in [Fig. 4] can be applied to recent language models such as the open-source DeepSeek-MoE-16B® (DeepSeek-MoE). In particular, DeepSeek-MoE comprises 27 expert model layers, each layer hosting 64 expert models including linear gate units, for a total of 1728 expert models. The presence of numerous expert models including linear gate units in DeepSeek-MoE offers a substantial advantage for post-training manipulations, such as expert model merging. The use of perturbation according to the present invention significantly improves the ability to analyze the interactions between the different expert models, thus facilitating the refinement and customization of the language model. For example, the language model merging method known as MCSMoE (Li P, Zhang Z, Yadav P, et al. Merge, then compress: Demystify efficient SMoe with hints from its routing policy [J]).(arXiv preprint arXiv:2310.01334, 2023) requires neural alignments that have only been implemented on switching transformer models (and not based on linear gate units). The method according to the present invention makes it possible to apply the MCSMoe fusion method to large language models organized according to MoE architectures combining several expert models that take the form of multilayer perceptrons formed from linear gate units. This allows the use of this fusion method to be extended.

[0112] In terms of results, the inventor compared the perplexity of the Wikitext dataset (Merily S, Xiong C, Bradbury J, et al. Pointer sentinel mixture models[J]. arXiv preprint arXiv:1609.07843, 2016.) of the original Deepseek-MoE model and that of this trained model using permutations according to the process, using the data known as the "Language Model Evaluation Harness of EleutherAI" (Gao L, Tow J, Abbasi B, et al. A framework for few-shot language model evaluation [EB / OL]. Zenodo, 2024-07. Version vO.4.3. DOI: 10.5281 / zenodo.12608602. Available from: https: / / zenodo.org / records / 12608602). The results are identical, such as the word perplexity of 9.293 and the byte perplexity of 1.517. For example, if the same prompt is used for both models, the original model and the "swapped" model will produce the same result. answer. For example, to the question "an attention function can be described as the mapping of a query and a set of key-value pairs with an output, where the query, keys, values, and output are all vectors," the original model and the "permuted" model answer "a vector of attention scores, which are used to compute a weighted sum of the values. The attention function is a key element of the transformer architecture, which is used in many natural language processing tasks. The attention function is used to compute the context vector, which is used to compute the output of the transformer."

[0113] Figure 5 schematically illustrates a device configured to control the rendering of text generated from input textual data from at least one embedded vehicle system, according to a particular and non-limiting embodiment of the present invention.Device 5, for example, corresponds to a device embedded in a vehicle, such as a computer.

[0114] The device 5 is, for example, configured to carry out the steps of the processes described opposite Figures 3 and 4. Examples of such a device 5 include, but are not limited to, embedded electronic equipment such as a vehicle's on-board computer, an electronic control unit such as an ECU (Electronic Control Unit), a smartphone, a tablet, or a laptop computer. The elements of the device 5, individually or in combination, may be integrated into a single integrated circuit, into several integrated circuits, and / or into discrete components. The device 5 may be implemented in the form of electronic circuits or software (or computer) modules, or a combination of electronic circuits and software modules.

[0115] The device 5 comprises one (or more) processor(s) 50 configured to execute instructions for carrying out the steps of the process and / or for executing instructions from the software embedded in the device 5. The processor 50 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 5 further comprises at least one memory 51, for example, volatile and / or non-volatile memory, and / or includes a memory storage device that may include volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk, or optical disk.

[0116] The computer code of the embedded software(s) including the instructions to be loaded and executed by the processor is for example stored on memory 51.

[0117] According to various specific and non-limiting embodiments, the device 5 is coupled in communication with other similar devices or systems and / or with communication devices, for example a TCU (Telematic Control Unit), by for example via a communication bus or through dedicated input / output ports.

[0118] According to a particular and non-limiting embodiment, the device 5 includes a block 52 of interface elements for communicating with external devices, for example a remote server or the cloud, other nodes of the ad hoc network. The interface elements of block 52 include one or more of the following interfaces: - radio frequency RF interface, for example of the Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or of the Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or of the Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced; - USB interface (from the English "Universal Serial Bus" or "Universal Serial Bus" in French); - HDMI interface (from the English "High Definition Multimedia Interface", or "High Definition Multimedia Interface" in French); - LIN interface (from the English "Local Interconnect Network", or in French "Réseau interconnecté local").

[0119] According to another particular and non-limiting embodiment, the device 5 includes a communication interface 53 which enables communication with other devices (such as other computers in the embedded system) via a communication channel 54. The communication interface 53 corresponds, for example, to a transmitter configured to transmit and receive information and / or data via the communication channel 54. The communication interface 53 corresponds, for example, to a wired network of the CAN (Controller Area Network), CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by ISO 17458) or Ethernet (standardized by ISO / IEC 802-3) type.

[0120] According to a particular and non-limiting embodiment, the device 5 can provide output signals to one or more external devices, such as a display screen 55, touch or not, one or more speakers 56 and / or other peripherals 57 via output interfaces 58, 59, 60 respectively. According to a variant, one or more of the external devices is integrated into the device 5.

[0121] Of course, the present invention is not limited to the embodiments described above but extends to any process that would include secondary steps without thereby departing from the scope of the present invention. The same would apply to a device configured for the implementation of such a method.

[0122] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-powered vehicle, comprising the device 5 of [Fig.5].

Claims

Demands

1. A method for controlling the rendering of text by an embedded system in a vehicle, said method implemented by at least one processor comprising the following steps: - obtaining input textual data from an embedded speech recognition system in the vehicle; - generating the text by a language model when the obtained textual data is presented as input to said language model, said language model being based on an architecture of so-called expert models, each expert model including linear units with gates that are parameterizable by weights and biases; - rendering the generated text by the embedded system in the vehicle;said language model being trained during a learning phase comprising the following steps: - obtaining (41) a permutation matrix of weights and biases of linear gated units to maximize a similarity between the weights and biases of linear gated units of a first expert model and the weights and biases of linear gated units of a second expert model; - obtaining (42) permuted weights and biases by permuting the weights and biases of the linear gated units of the second expert model according to the permutation matrix obtained; - replacing (43) the weights and biases of the linear gated units of the second expert model with the permuted weights and biases obtained.

2. A method according to claim 1, wherein the weights of each linear gated unit are transformation matrix weights.

3. A method according to claim 2, wherein each linear gated unit implements: a first linear transformation of an input data item of the linear gated unit by using a first linear transformation matrix formed of weights and biases; a second linear transformation of said input data item of the linear gated unit by using a second linear transformation matrix formed of weights and biases; an activation function providing an intermediate output data item when an output data item of the second linear transformation is present at the input of the activation function; a multiplication of an output data from the first linear transformation and an output data from the activation function; and a third linear transformation of a result data from said multiplication by using a third linear transformation matrix formed of weights and biases.

4. A method according to claim 3, wherein obtaining the perturbation matrix comprises a step of calculating a total similarity matrix by concatenating a first similarity matrix evaluating a similarity between the first transformation matrix of the first expert model and the first transformation matrix of the second expert model, a second similarity matrix evaluating a similarity between the second transformation matrix of the first expert model and the second transformation matrix of the second expert model, and a third similarity matrix evaluating a similarity between the third transformation matrix of the first expert model and the third transformation matrix of the second expert model.

5. Method according to claim 4, wherein the perturbation matrix is ​​obtained such that the total similarity matrix is ​​close to the identity matrix.

6. A method according to any one of the preceding claims, wherein the activation function is an element-based activation function.

7. Device (5) comprising a memory (51) associated with at least one processor (50) configured for carrying out the steps of the process according to any one of claims 1 to 6.

8. Computer program comprising instructions for carrying out the method according to any one of claims 1 to 6, when such instructions are executed by a processor.

9. Computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the process according to any one of claims 1 to 6.

10. Vehicle comprising a device according to claim 7.

Images