Method and device for learning a vehicle in-car conversational agent

Abstract

The present invention relates to a method and device for learning a vehicle in-car conversational agent. The conversational agent implements a large language model (LLM), and is configured to implement natural language communication between a vehicle user and at least one vehicle-mounted system. To this end, deep learning of the LLM based on human preferences is implemented, with optimization of the LLM policy by projecting updated parameters from a current policy into a finite policy space to obtain a new LLM policy. Figure 1 (for the abstract)

Description

Title of the invention: Method and device for learning a vehicle cabin conversational agent technical field

[0001] The invention relates to methods and devices for learning a vehicle interior conversational agent, in particular but not exclusively a motor vehicle. Technological background

[0002] The development of artificial intelligence and large language models, known as LLM or agent LLM, (from the English "Large Language Model") makes it possible to imagine new human-machine interfaces implemented in a vehicle, for example to allow a driver and / or a passenger of the vehicle to control embedded systems of the vehicle or simply to communicate with an embedded system to obtain information on the state of the vehicle or on the environment in which the vehicle is traveling using natural language as a means of interaction.

[0003] To improve the efficiency and relevance of generative LLM agents, these models or agents are learned according to a method called reinforcement learning based on human preferences, called RLHF (from the English "Reinforcement Learning with Human Feedback") to align the text generated by the LLM model with user preferences on the responses to be provided to queries provided to the LLM model in the context of use of the LLM model (for example for communication in a vehicle cabin or more precisely to control in natural language a given embedded system).

[0004] One of the algorithms used in RLHF learning is known as PPO (from the English “Proximal Policy Optimization” or in French “Optimisation proximale de la politique”) and for example described in the document entitled “Proximal Policy Optimization Algorithms”, by John Schulman et al., published on August 28, 2017 under the reference arXiv: 1707.06347v2.

[0005] Despite the popularity of the PPO algorithm, certain problems are encountered, such as slow convergence without a solid theoretical guarantee of convergence to support its performance. In other words, when learning RLHF with PPO, not only is the process slow, but it is not possible to know whether the algorithm saturates or continues to improve. Consequently, this algorithm consumes a lot of computing resources without knowing whether or not it achieves high performance.

[0006] This problem is particularly acute for difficult tasks such as those required of a vehicle in-cabin assistant, such as generating natural language messages and responses on various topics ranging from vehicle status to navigation assistance (interface with the GPS system) or understanding the scene inside and outside the vehicle. The variety and difficulty of these tasks make the convergence problems described above particularly concerning in the context of a vehicle in-cabin assistant. Summary of the present invention

[0007] One object of the present invention is to solve at least one of the problems of the technological background described above.

[0008] An object of the present invention is, for example, to improve the learning of a conversational agent for a vehicle interior.

[0009] Another objective is to reduce the costs associated with the development of such a conversational agent.

[0010] According to a first aspect, the present invention relates to a method for learning a vehicle cabin conversational agent, the conversational agent implementing a large language model, called LLM, the conversational agent being configured to implement natural language communication between a vehicle user and at least one vehicle embedded system, the method being implemented by at least one processor and comprising the following steps: - supervised learning of a pre-trained LLM from a first training dataset representative of natural language interactions with at least one embedded system to obtain a basic LLM; - learning a reward model from a second training dataset representative of human-scored textual data obtained as output from the basic LLM in response to each query from a set of queries provided as input to the basic LLM; - Reinforcement learning of the basic LLM from a third training dataset and reward model to obtain the LLM implemented in the chatbot, the reinforcement learning including: • an update of parameters of a current policy of the basic LLM by maximizing rewards obtained from the reward model for output data of the basic LLM obtained for each prompt of a set of prompts from the third training dataset provided as input to the basic LLM, • an update of the policy by projecting the updated parameters into a finite policy space to obtain a new policy.

[0011] Using the updated policy parameters by reward maximization to determine the new policy allows convergence learning faster, which accelerates learning and reduces the resources needed to train a chatbot.

[0012] According to one variant, the policy parameter update and policy update steps are reiterated in such a way that the new policy obtained in one iteration becomes the current policy in the next iteration.

[0013] According to yet another variant, the updating of parameters of the current policy includes a determination of a gradient of the rewards obtained from the reward model, the maximization being obtained by a gradient climbing method.

[0014] According to a further variant, the method further includes a step of transmitting parameters representative of the LLM implemented in the conversational agent to a data storage device of the vehicle, the LLM being implemented in the form of a neural network in the vehicle.

[0015] According to a second aspect, the present invention relates to a learning device for a conversational agent in a vehicle interior, the device comprising a memory associated with a processor configured for the implementation of the steps of the process according to the first aspect of the present invention.

[0016] According to a third aspect, the present invention relates to a system comprising the device according to the second aspect of the present invention and a vehicle connected by wired or wireless communication to the device.

[0017] According to a fourth aspect, the present invention relates to a computer program which includes instructions adapted for carrying out the steps of the process according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0018] Such a computer program may use any programming language, and be in the form of source code, object code, or an intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0019] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the process according to the first aspect of the present invention.

[0020] On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium can include a storage means, such as a ROM, a CD-ROM or a microelectronic circuit-type ROM, or a magnetic recording means or a hard disk drive.

[0021] On the other hand, this recording medium can also be a transmissible medium such as an electrical or optical signal, such a signal being able to be transmitted via an electrical or optical cable, by conventional or Hertzian radio, by self-directed laser beam, or by other means. The computer program according to the present invention can, in particular, be downloaded onto an Internet-type network.

[0022] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to execute or to be used in the execution of the process in question. Brief description of the figures

[0023] Other features and advantages of the present invention will become apparent from the description of the particular and non-limiting embodiments of the present invention below, with reference to the attached Figures 1 to 4, in which:

[0024] [Fig-1] schematically illustrates a learning process of a converging agent sational of a vehicle passenger compartment, according to a particular embodiment of the present invention;

[0025] [Fig.2] schematically illustrates the optimization of a policy of the agent convert sational in the process of the [Fig.1], according to a particular embodiment of the present invention;

[0026] [Fig.3] schematically illustrates a device configured for learning a vehicle cabin conversational agent, according to a particular and non-limiting embodiment of the present invention.

[0027] [Fig.4] illustrates a flowchart of the different stages of a learning process of a vehicle cabin conversational agent, according to a particular and non-limiting embodiment of the present invention. Description of examples of achievements

[0028] A method and device for learning a vehicle cabin conversational agent will now be described in what follows with joint reference to Figures 1 to 4. The same elements are identified with the same reference signs throughout the description that follows.

[0029] The terms "first," "second" (or "firsts," "seconds"), etc., are used in this document by arbitrary convention to allow for the identification and distinction of different elements (such as operations, means, etc.) implemented in the embodiments described below. Such elements may be distinct or correspond to a single element, depending on the embodiment.

[0030] Fig. 1 schematically illustrates a learning process 1 of a conversational agent in a vehicle's interior, according to particular and non-limiting embodiments of the present invention.

[0031] The vehicle carrying the chatbot corresponds, for example, to a vehicle with an internal combustion engine, with electric motor(s) or even a hybrid vehicle with a combustion engine and one or more electric motors. The vehicle thus corresponds, for example, to a land vehicle, for example a car, a truck, a bus.

[0032] The conversational agent is based on a generative LLM, also called an agent LLM. Such a generative LLM is, for example, implemented in the form of a neural network, as known to a person skilled in the art.

[0033] The learning of the conversational agent, i.e. the LLM agent, is implemented by a computer or server-type data processing device. Examples of hardware implementations of such a device are described below with reference to [Fig. 3].

[0034] In what follows, the LLM is referred to as a language model or agent.

[0035] Reinforcement learning is a field of machine learning in which an agent, in this case the language model, learns a strategy or policy through interactions with its environment, in this case the passenger compartment of a vehicle and / or the vehicle's onboard systems. The agent takes measures or actions that affect the environment in which the agent is located, which in turn transitions to a new state and provides the agent with a reward. Rewards are feedback signals that allow the agent to adjust its action strategy. As the agent goes through training episodes or iterations, it adjusts its strategy to take sequences of actions that maximize its reward.

[0036] In a first operation 11 of the process, supervised learning of a pre-trained model is implemented, the supervised learning corresponding to supervised fine-tuning, known as SFT (Supervised Fine-Tune). Supervised fine-tuning is used to prepare the language model to generate its responses in a format expected by the users of the language model (i.e., the conversational agent), according to the environment in which the language model is used.

[0037] The language model is used for example to control one or more embedded systems of the vehicle and to communicate with the vehicle.

[0038] The embedded system(s) belong to a set of embedded systems comprising: - a navigation and geolocation system, also called a GNSS system (Geolocation and Navigation by a Satellite System), for example a GPS type system (from the English "Global Positioning System" or in French "Système de géo-positionnement par satellites") or Galileo; - an air conditioning system; - an indoor and / or outdoor environmental analysis or detection system vehicle ; - an infotainment system; - etc.

[0039] Supervised learning uses a first training dataset 10 representative of natural language interactions with the environment in which the language model is used, i.e. the vehicle's embedded system(s) such as one or more of the systems listed above.

[0040] The first training dataset includes labeled data in the format (prompt, response) relating to the environment of use of the language model and prepared or validated by humans to show the language model output from supervised learning how to respond to prompts (or requests made by a user in the vehicle).

[0041] The pre-trained language model used to obtain the updated language model (also called the base language model) corresponds, for example, to a Llama 2 type model, as described in the document entitled "Llama 2: Open Foundation and Fine-Tuned Chat Models", by Hugo Touvron et al., published on July 19, 2023 under the reference arXiv:2307.09288v2 or to a Mistral 7B type model, as described in the document entitled "Mistral 7B", by Albert Q. Jiang et al., published on October 10, 2023 under the reference arXiv:2310.06825vl.

[0042] In a second operation 13 of the process, a reward model is learned or trained from a second training dataset 12 representative of human-scored textual data obtained as output from the base language model (obtained during the first operation by updating the pre-trained language model) in response to each query from a set of queries provided as input to the base language model.

[0043] A reward corresponds to a scalar value measuring the alignment of the basic language model with the context in which the language model is used (in this case a vehicle interior) and the intention of the user using the language model.

[0044] Training the reward model corresponds to training another machine learning model, for example another language model or LLM modified to generate a scalar value instead of a sequence of words. The reward model takes as input the text generated by the base language model and provides as output a quality score (a scalar value).

[0045] Training (or learning) the reward model requires a training dataset corresponding to a set of text data generated by the base model (obtained in the first operation) according to queries or prompts provided as input to the base language model, the text data being labeled, This means that a score is assigned by a human to each text input (or response) obtained from the basic language model for a given prompt. The responses are ranked from best to worst. In one implementation, only two responses are output from the basic language model: one is classified or scored as correct (also called the chosen response), and the other is classified or scored as incorrect or rejected (called the rejected response). By being trained on the outputs and quality scores of the basic language model, the reward model creates a mathematical representation of human preferences within the context of the basic language model's application (in this case, the interior of a vehicle and one or more of the vehicle's embedded systems).

[0046] Learning a reward model is known to those skilled in the art and is described for example in the document entitled "Llama 2: Open Foundation and Fine-Tuned Chat Models", by Hugo Touvron et al., published on July 19, 2023 under the reference arXiv:2307.09288v2 or in the document entitled "Reward design with language models", by Minae Kwon et al., published on February 27, 2023 under the reference arXiv:2303.00001vl.

[0047] In a third operation 14 of the process, the basic language model obtained in the first operation 11 is updated during a training or reinforcement learning phase using a third training dataset and the reward model learned or trained in the second operation 13, such learning being supervised.

[0048] During this third operation, the policy (from the English "policy"), denoted ji, of the base language model is updated from the third dataset and the scores or rewards obtained as output from the reward model evaluating the responses of the base language model output in response to the prompts provided as input to the base language model.

[0049] Figure 2 illustrates a scheme for updating the policy ir of the base language model 21 as a function of the third training dataset 20 and the reward model 22. In response to a prompt denoted 'S;', from a set of prompts or requests (or "prompts" in English) of the third training dataset 20, the base language model 21 generates an action denoted a; via the current policy of the language model 21, which action a; is evaluated by the reward model which assigns it a score or reward denoted 'r / ', to which is associated a gradient denoted 'A0' which is calculated with respect to the parameters, denoted '0', of the current policy of the base language model.

[0050] The policy of a language model comprises a set of rules, for example parametric rules, which determine the behavior of an intelligence agent artificial, in this case the basic language model. In mathematical terms, a policy 'ji' is a function that takes a state 's' and returns an action 'a', i.e. ji(s) -> a.

[0051] The action 'a' belongs to a space of allowed actions which includes, for example, the entire vocabulary of tokens available for the basic language model. In other words, the action space is the set of possible linguistic outputs that the language model can generate for a given policy.

[0052] The set of states of the language model usage environment includes user prompts.

[0053] The reward measures the alignment of the language model response with the application context and user intent.

[0054] Reinforcement learning advantageously uses gradient climbing to maximize reward.

[0055] The third operation 13 corresponds to a reinforcement learning loop in which the base language model corresponds to the reinforcement learning agent, referred to as the RL agent. At each step or iteration of the learning loop, the base language model takes several prompts from the third training dataset and generates text. The output of the base language model is passed to the reward model, which provides a score that assesses its alignment with human preferences. The policy of the base language model is then updated to create outcomes that obtain a higher score on the reward model, the objective being to determine the policy parameters that maximize the score obtained from the reward model via a gradient-climbing function.

[0056] The third operation 13 thus comprises the following operations, for each iteration of the learning loop: - an update of parameters of a current policy of the base language model by maximizing rewards obtained from the reward model for base language output data in response to each prompt from a set of prompts in the third training dataset provided as input to the base language model; and - an update of the current policy, to obtain a new current policy, by projecting the updated parameters into a finite policy space.

[0057] The number of iterations of the learning loop corresponds to a parameter of the learning system. When a new policy is obtained, it becomes the current policy in the next iteration, and so on for each iteration. The number of iterations is, for example, equal to 10 or 100, or even 1000 or 10000 in other examples.

[0058] Updating the policy by projecting the parameters into a space of The finite lithics for selecting the new policy corresponding to the projection result can be seen as a form of regularization or constraint to maintain control over policy updates. This ensures that the base language model obtained after the various iterations does not deviate too much from the initial base language model obtained in the first operation.

[0059] The maximization of rewards is obtained by calculating the gradient associated with an expected reward according to the response of the basic language model evaluated by the reward model (the expected reward being included in the third training dataset) and by applying the so-called gradient climbing method.

[0060] The gradient of the value function V^p,) associated with the current policy ir is obtained by the following policy gradient theorem (PGT, from the English "Policy Gradient Theorem"):

[0061] [Math.l]

[0062] With s a state of a finite action space, p the target state distribution, y a discount factor and Qn(s,.) the state-action value function associated with the current policy jt.

[0063] Such a theorem is known to those skilled in the art and is described for example in the document entitled "A novel framework for policy mirror discent with general parame-terization and linear convergence", by Carlo Alfano et al, published on September 21, 2023 at the 37th conference on neural information processing systems and updated on January 15, 2024, an intermediate version of which was published on November 6, 2023 under the reference arXiv:2301.13139v3.

[0064] At each iteration, a surrogate loss is minimized and the projection of the updated parameters onto a probability simplex is a function of the finite space of actions associated with the basic language model.

[0065] It has been observed that the implementation of such reinforcement learning makes it possible to reduce the learning time from 1 / e2 to log(l / e) (or a multiple of log(l / e)), where e is the measure of performance up to the optimal solution.

[0066] The learning time with such a process is notably reduced compared to known methods such as the so-called PPG method (from the English "Proximal Policy Optimization" or in French "Optimisation proximale de la politique").

[0067] In a further operation, the language model obtained as a result of the learning described above is integrated into the vehicle. The parameters of the language model, i.e., the LLM parameters, are loaded into the memory of the The vehicle is used to implement the LLM-based conversational agent via one or more of the vehicle's onboard computers. LLM parameters are transmitted, for example, to a vehicle data storage device (e.g., memory) via a wired or wireless connection linking the device that learned the parameters to the vehicle. The LLM is implemented, for example, as a neural network by one or more of the vehicle's onboard computers.

[0068] Such a conversational agent then allows the driver of the vehicle or a passenger to converse in natural language with the vehicle, for example to obtain information on the current state of the vehicle or of one or more of its on-board systems, on the external environment of the vehicle analyzed by the vehicle's on-board sensors, on the internal environment of the vehicle's passenger compartment, or even to control one or more on-board systems in natural language with feedback from this or these on-board systems in natural language.

[0069] Fig. 3 schematically illustrates a device 3 configured for learning a vehicle cabin conversational agent implementing a large language model, called LLM, according to examples of embodiments of the invention.

[0070] Device 3 corresponds for example to a computer or server type data processing device, and more particularly a device configured to implement one or more artificial neural networks, for example one or more deep neural networks.

[0071] According to one variant, device 3 corresponds to a device configured for the implementation of the conversational agent, for example a device embedded in the vehicle, for example a computer.

[0072] Device 3 is, for example, configured to carry out the operations described opposite Figures 1 and 2 and / or the steps of the process described opposite [Fig. 4]. Examples of such a device 3 include, but are not limited to, a computer, a server, a laptop computer, a smartphone, a tablet, embedded electronic equipment such as a vehicle's on-board computer, or an electronic control unit such as an ECU (Electronic Control Unit). The elements of device 3, individually or in combination, can be integrated into a single integrated circuit, into several integrated circuits, and / or into discrete components. Device 3 can be implemented in the form of electronic circuits or software (or computer) modules, or a combination of electronic circuits and software modules.

[0073] The device 3 comprises one (or more) processor(s) 30 configured to execute instructions for carrying out the steps of the process and / or for executing instructions from the software embedded in the device 3. The processor 30 may include integrated memory, an input / output interface, and various circuits known to a person skilled in the art. The device 3 further comprises at least one memory 31 corresponding for example to volatile and / or non-volatile memory and / or comprises a memory storage device which may comprise volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.

[0074] The computer code of the embedded software(s) including the instructions to be loaded and executed by the processor is for example stored on memory 31.

[0075] According to various particular and non-limiting embodiments, the device 3 is coupled in communication with other similar devices or systems and / or with communication devices, for example a TCU (Telematic Control Unit), for example via a communication bus or through dedicated input / output ports.

[0076] The neural network can be trained by a first device 3 corresponding to a server or a series of servers, communicating with a second device 3 corresponding to a vehicle's computer, via a communication network. The internal parameters (or weights) of the neural network can then be transmitted by this remote device to the computer, which receives them. The computer is then able to implement the neural network and the classification process when an input image acquired by the vehicle's imaging system is presented as input to the trained neural network.

[0077] According to a particular and non-limiting embodiment, the device 3 comprises a block 32 of interface elements for communicating with external devices. The interface elements of the block 32 comprise one or more of the following interfaces: - Radio frequency (RF) interface, for example, Wi-Fi® (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (from the English "Long-Term Evolution" or in French "Evolution à long terme"), LTE-Advanced (or in French LTE-avancé); - USB interface (from the English "Universal Serial Bus" or "Universal Serial Bus" in French); - HDMI interface (from the English "High Definition Multimedia Interface", or "High Definition Multimedia Interface" in French); - LIN interface (from the English "Local Interconnect Network", or in French "Réseau interconnecté local").

[0078] Internal parameters (weights) of the trained and / or updated neural network can be received via the communication interface 33.

[0079] According to another particular and non-limiting embodiment, the device 3 includes a communication interface 33 which enables communication with other devices (such as other computers in the embedded system) via a communication channel 330. The communication interface 33 corresponds, for example, to a transmitter configured to transmit and receive information and / or data via the communication channel 330. The communication interface 33 corresponds, for example, to a wired network of the CAN (Controller Area Network), CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by ISO 17458) or Ethernet (standardized by ISO / IEC 802-3) type.

[0080] According to a particular and non-limiting embodiment, the device 3 can provide output signals to one or more external devices, such as a display screen 340, touch or not, one or more speakers 350 and / or other peripherals 360 (projection system) via output interfaces 34, 35 and 36 respectively. According to a variant, one or more of the external devices is integrated into the device 3.

[0081] Figure 4 illustrates a flowchart of the different stages of a method for learning a vehicle cabin conversational agent, the conversational agent implementing a large language model, known as an LLM, the conversational agent being configured to implement natural language communication between a vehicle user and at least one vehicle embedded system, according to a particular and non-limiting embodiment of the present invention. The method is implemented, for example, by device 3 in Figure 3.

[0082] In a first step 41, supervised learning of a pre-trained LLM is implemented from a first training dataset representative of natural language interactions with at least one embedded system to obtain a basic LLM.

[0083] In a second step 42, a reward model learning is implemented from a second training dataset representative of human-scored textual data obtained as output from the basic LLM in response to each query from a set of queries provided as input to the basic LLM.

[0084] In a third step 43, reinforcement learning of the basic LLM is implemented from a third training dataset and the reward model to obtain the LLM implemented in the conversational agent, reinforcement learning including: • an update of parameters of a current policy of the base LLM by maximizing rewards obtained from the reward model for output data of the base LLM obtained for each prompt of a set of prompts of the third training dataset provided as input to the base LLM, • an update of the policy by projecting the updated parameters into a finite policy space to obtain a new policy.

[0085] According to one variant, the variants and examples of the operations described in relation to [Fig.1] and / or 2 apply to the steps of the process in [Fig.4].

Claims

Demands

1. A method for learning a vehicle in-car conversational agent, said conversational agent implementing a large language model, said LLM, said conversational agent being configured to implement natural language communication between a user of said vehicle and at least one vehicle-embedded system, said method being implemented by at least one processor and comprising the following steps: - supervised learning (41) of a pre-trained LLM from a first training dataset representative of natural language interactions with said at least one vehicle-embedded system to obtain a basic LLM; - learning (42) of a reward model from a second training dataset representative of human-scored textual data obtained as output from said basic LLM in response to each query from a set of queries provided as input to said basic LLM;- reinforcement learning (43) of said base LLM from a third training dataset and said reward model to obtain said LLM implemented in the conversational agent, said reinforcement learning comprising: • an update of parameters of a current policy of said base LLM by maximizing rewards obtained from said reward model for output data of said base LLM obtained for each prompt from a set of prompts of said third training dataset provided as input to said base LLM, • an update of said policy by projecting the updated parameters into a finite policy space to obtain a new policy.;

2. A method according to claim 1, wherein the policy parameter update and policy update steps are repeated in such a way that the new policy obtained in one iteration becomes the current policy in the next iteration.

3. A method according to any one of claims 1 to 2, wherein said updating of current policy parameters comprises determining a gradient of the rewards obtained from said reward model, said maximization being achieved by a ramp-up method of gradient.

4. A method according to any one of claims 1 to 3, further comprising a step of transmitting representative parameters of said LLM implemented in the conversational agent to a data storage device of the vehicle, said LLM being implemented in the form of a neural network in said vehicle.

5. A computer program comprising instructions for carrying out the method according to any one of the preceding claims, when such instructions are executed by at least one processor.

6. Computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the process according to any one of claims 1 to 4.

7. Device (3) for learning a vehicle interior conversational agent, said device (3) comprising a memory (31) associated with at least one processor (30) configured to implement at least one step of the method according to any one of claims 1 to 4.

8. System comprising the device (3) according to claim 7 and a vehicle connected in communication to said device (3).

Images