Method and device for providing operating information for a vehicle

Abstract

The invention relates to a method (200) and a device for providing operating information for a vehicle, said method comprising the steps of: Receiving (210) a generated text; Determining (220) a list of labels from said received text and a first predetermined model; Receiving (230) data; Determining (240) a list of words from said list of labels, said received data, and a predetermined knowledge graph; Determining (250) a list of sentences from said list of words and a second predetermined model, said second model being a machine learning model in the field of natural language processing; Emitting (260) said list of sentences to said audio system to provide operating information for said vehicle by speech synthesis. Figure to be published for the abstract: Figure 2

Description

Title of the invention: Method and device for providing operating information for a vehicle Technical field of the invention

[0001] The invention is in the field of autonomous vehicle driving assistance systems. In particular, the invention relates to a method and a device for providing operating information for a vehicle having an audio voice recognition and synthesis system, to assist and guide an occupant of said vehicle. State of the art

[0002] The term "vehicle" means any type of vehicle such as a motor vehicle, a moped, a motorcycle, a storage robot in a warehouse, etc. The term "autonomous driving" of an "autonomous vehicle" means any method capable of assisting the driving of the vehicle. The method may thus consist of partially or totally steering the vehicle or providing any type of assistance to a natural person driving the vehicle. The method thus covers all autonomous driving, from level 0 to level 5 in the OICA scale, for International Organization of Motor Vehicle Manufacturers.

[0003] Vehicles marketed today include a great many features. The implementation of these features differs between different vehicle makes and / or models. Many of these implementations are complex and not very intuitive, especially for a new user. This new user needs to receive information on the vehicle's operation, through aids or guides, to understand these implementations. This new user must consult user instructions (also called instructions, user guide, user manual, user guide, etc.), instructions detailing, in writing, the steps to follow to activate, deactivate or use one of the features. When using the car, it is not possible to consult these instructions.In addition, some of these instructions are detailed in documents available on the internet or in documents only available from an after-sales service, because some instructions are not or are no longer up to date, in particular when these instructions are printed.

[0004] “Artificial intelligence”, or AI, means a machine, a computer device, designed from large quantities of data and capable of providing decisions or requests based on an expressed need. It is known that the design of these devices is based on learning a model from expressed needs, also called inputs, and expected decisions, also called outputs. The inputs and outputs of artificial intelligence are digital, interpretable and informative. matically. We are talking about machine learning models, the models are pre-trained (learning phase).

[0005] Natural Language Processing, NLP, is a field of artificial intelligence that focuses on the interaction between computers and human language. For example, NLP relies on machine learning techniques and models, such as neural networks, language models, classification and clustering algorithms, statistical methods, etc. These techniques allow NLP systems to process and understand structures, meanings, and nuances of natural language.

[0006] For example, version 3.5 of the artificial intelligence known as "ChatGPT®" is an NLP whose training data size contains 175 billion parameters to be trained. The training dataset contains 300 billion words, or about 570 gigabytes of text data.

[0007] Currently, NLP models are much too large (memory) and consume computing resources (processor CPU time, processor power, etc.). These models cannot be embedded in a computer (or ECU from the English "Electronic Control Unit") in a vehicle while delivering services in near real time, the computer being compatible (variation in temperature, humidity, etc.) with use in a "general public" vehicle (sold in several tens of thousands of copies).

[0008] It is known to use a "smartphone", a multifunctional cell phone or mobile phone, connected to the "cloud" (remote computer server, or server). This smartphone may include an application connected, by an electromagnetic link (4G, 5G, etc.) or similar, to this cloud to an AI of the ChatGPT® type. However, on the one hand, this requires a good connection quality and generally a paid subscription, and, on the other hand, these AIs are not suitable for providing vehicle operating information from a voice request, in particular providing contextualized information based on a vehicle make and / or model. In addition, this implies that the voice requests pass through the cloud and are recorded by 1TA. A user may wish not to have his conversations, made in the car, recorded by a third party, outside the car.

[0009] Audio speech recognition systems are also known that allow human speech to be converted into written text. These systems operate using signal processing algorithms that analyze the audio and identify the different components of language, such as phonemes, words, intonations, etc.

[0010] Some of these voice recognition systems are simpler, and can be implemented in a vehicle. There is no conversion of human speech into written text. From the identification of intonations or a predetermined sound sequence, Action requests such as "turn off" or "turn on" a car radio are recognized. These are simpler systems but they are limited to a very limited number of language component identifications. The expressed need is limited to a very limited number (around ten) of functions. These systems are very sensitive to ambient noise, intonation, accents, etc., and therefore do not allow for the provision of very varied operating information. Summary of the invention

[0011] An object of the present invention is to remedy the aforementioned problem, in particular provide accurate and relevant vehicle operating information, contextualized information based on the vehicle, with models that can be embedded and executed on ECUs compatible for use in a “general public” vehicle, without having to go through a cloud and have verbal requests recorded by a third party.

[0012] To this end, a first aspect of the invention relates to a method for providing operating information for a vehicle having an audio voice recognition and synthesis system, to assist and guide an occupant of said vehicle, said occupant issuing a verbal request, said method being implemented by a processor and comprising the steps of: • Receiving a text generated from said audio system processing said verbal request; • Determination of a list of labels from said received text and a first predetermined model, • said first model being a machine learning model in the field of natural language processing, • said first model being trained from • a predetermined set of texts representative of a verbal request for the provision of operating information for at least one vehicle, and • a predetermined set of said labels, • said labels being a group of characters, at least one word, used to identify and describe sets of words from said predetermined set of texts representative of a verbal request for the provision of operating information for at least one vehicle; • Reception of data, said data relating to said vehicle, one of said data being at least one identifier making it possible to determine a make and / or model of said vehicle; • Determination of a list of words from said list of labels, said received data and a predetermined knowledge graph, • said graph comprising • nodes, said nodes being representative of entities, the entities being predetermined from • said labels, and • a set of at least one word, said at least one word characterizing a predetermined set of received data; • arcs, said arcs being representative of a functional relationship between said entities, said arcs being described by at least one word; • Determination of a list of sentences from said list of words and a second predetermined model, said second model being a machine learning model from the natural language processing domain, • said second model being at least trained from a set of words from the knowledge graph; • Transmission of said list of phrases to said audio system to provide operating information of said vehicle by voice synthesis.

[0013] Thus, the first model and the second model are determined to the strict minimum, and therefore of minimum size, and are then able to be executed on an ECU, calculator, automobile (vehicle marketed to the "general public"). The pre-training of the models can be carried out on an identical make and / or model of vehicle, further limiting the size of the models. The size being limited, this allows a download in reasonable durations, not mobilizing the vehicle for several hours, these models can then be updated in a reasonable time. The voice request remains internal to the vehicle's computer system, it is not transmitted to a remote server. The confidentiality of the "conversations" is preserved. Also, the different functionalities of the vehicle are taken into account. By said data, the functionalities are contextualized in relation to the make and / or model of the vehicle, the occupant receives contextualized information..

[0014] Advantageously, said list of sentences is further determined from said received data, said second model is further trained from metadata relating to a predetermined set of received data.

[0015] Thus, the sentence list is contextualized according to the make and / or model of the vehicle. Said second model is even more optimized.

[0016] Advantageously, said received data also includes data relating to said occupant.

[0017] Thus, the list of sentences is also contextualized according to the occupant. The operating information is personalized giving an improved perceived quality of the vehicle. The information provided can also be more precise.

[0018] Advantageously, said first model is a neural network type model.

[0019] Advantageously, said second model is a neural network type model.

[0020] Advantageously, said first model and said second model are furthermore trained from a text extraction of user manuals for said vehicle.

[0021] Thus, the knowledge graph and the second model are simpler to predetermine and are simplified and optimized. The information provided is also more precise and is more consistent with existing notices.

[0022] Advantageously, said at least one vehicle is a vehicle of the same make and / or model as said vehicle.

[0023] Thus, the first model, the knowledge graph and the second model are even more optimized and reduced in size, taking into account only the make and / or model of the vehicle.

[0024] A second aspect of the invention relates to a device comprising an audio system, a memory associated with at least one processor configured to implement the method according to the first aspect of the invention.

[0025] The invention also relates to a vehicle comprising the device.

[0026] The invention also relates to a computer program comprising instructions which, when the program is executed by the device according to the second aspect of the invention, lead the latter to implement the method according to the first aspect of the invention. Brief description of the figures

[0027] Other characteristics and advantages of the invention will emerge from the description of the non-limiting embodiments of the invention below, with reference to the appended figures, in which:

[0028] [Fig.l] schematically illustrates a device, according to a particular example of embodiment of the present invention.

[0029] [Fig.2] schematically illustrates a method of providing operating information of a vehicle, according to a particular exemplary embodiment of the present invention. Detailed description of the invention

[0030] The invention is described below in its non-limiting application to the case of an autonomous motor vehicle traveling on a road or on a traffic lane. Other applications such as a robot in a storage warehouse or a motorcycle on a country road are also conceivable.

[0031] [Fig.l] represents an example of a device 101 included in the vehicle, in a network ("cloud") or in a server. This device 101 can be used as a centralized device in charge of at least certain steps of the method described below with reference to [Fig.2]. In one embodiment, it corresponds to an autonomous driving computer.

[0032] In the present invention, the device 101 is included in the vehicle.

[0033] This device 101 can take the form of a box comprising circuits printed, from any type of computer or even from a mobile phone (“smartphone”).

[0034] The device 101 comprises a random access memory 102 for storing instructions for the implementation by a processor 103 of at least one step of the method as described above. The device also comprises a mass memory 104 for storing data intended to be retained after the implementation of the method.

[0035] The device 101 may further comprise a digital signal processor (DSP) 105. This DSP 105 receives data to format, demodulate and amplify, in a manner known per se, this data.

[0036] The device 101 also comprises an input interface 106 for receiving the data implemented by the method according to the invention and an output interface 107 for transmitting the data implemented by the method according to the invention.

[0037] For example, the input interface 106 can receive the following data: position or geographical location of the vehicle, speed and / or acceleration of the vehicle, set or predetermined positions / speeds / accelerations, engine speed, position and / or travel of the clutch, brake and / or acceleration pedal, detection of other vehicles or objects, position or geographical location of the other vehicles or objects detected, speed and / or acceleration of the other vehicles or objects detected, operating states of sensors, confidence index of data originating from or processed by sensors and / or devices similar to the device 101. For example, the sensors capable of providing data are: GPS associated or not with mapping, tachometers, accelerometers, RADAR, LIDAR, lasers, ultrasound, camera, an audio system for recognition and voice synthesis, etc.

[0038] The interface 106 can also receive: a text or an alphanumeric sequence of characters, data relating to the vehicle such as a make, a model, an identifier making it possible to determine a make and / or a model of the vehicle, data relating to an occupant, etc.

[0039] For example, the output interface 107 can transmit data similar to the data received by the input interface 106. For example, the output interface 107 can also transmit: a list of labels, a list of words, a list of sentences, metadata relating to a predetermined set of received data, ...

[0040] [Fig.2] schematically illustrates a method 200 for providing information of operation of a vehicle, according to a particular embodiment of the present invention.

[0041] The method 200 is a method for providing operating information of a vehicle having an audio voice recognition and synthesis system, to assist and guide an occupant of said vehicle, said occupant issuing a verbal request, said method being implemented by the processor 103 and comprising several steps.

[0042] Audio recognition and speech synthesis systems are known. They are capable of converting a verbal request from an occupant into a text. By text is meant a succession of characters, for example alphanumeric, interpretable by computer. For example, these characters are coded, encoded, in ASCII, UTF-8, UTF-18 or other encodings known to those skilled in the art of character encoding.

[0043] The occupant may issue various verbal requests. For example, a verbal request may be the verbal expression issued by the occupant “How can I prevent my child from opening the door?”, when the occupant wishes to be assisted and guided on the activation procedure to prohibit the opening of a rear door by its interior handle, in other words to activate the mechanical child safety.

[0044] Step 210, RxTxt, is a step of receiving a text generated from said audio system processing said verbal request. Following the above verbal expression, said audio system is able to transmit to the device 101 a text such as "how can I prevent my child from opening the door". In one operating mode, the audio system is able to transmit metadata, such as metadata relating to a "Question" by an analysis of the intonation.

[0045] Step 220, LList, is a step of determining a list of labels from said received text and a first predetermined model.

[0046] Said first model is a machine learning model from the field of natural language processing, NLP, such as neural networks, language models, classification and clustering algorithms, statistical methods, etc. In one operating mode, said first model is a neural network type model whose design method (number of nodes, number of layers, choice of activation functions, etc.) and training method (parameterization of activation functions, etc.) are well known, the remaining difficulty is to succeed in choosing inputs and outputs of the models that are relevant to the problem that must be solved.

[0047] Thanks to the invention, this difficulty is overcome. Said first model is trained from a predetermined set of texts representative of a verbal request for the provision of operating information for at least one vehicle, and from a predetermined set of said labels, said labels being a group of characters, at least one word, used to identify and describe sets of words from said predetermined set of texts representative of a verbal request for the provision of operating information for at least one vehicle.

[0048] For example, a predetermined set of texts representative of a verbal request for the provision of operating information for at least one vehicle may consist of several subsets. To illustrate, a subset may be linked to an expression introducing a question such as the set {“How”, “help me”, “in what way”, ...}. In another example, a subset may be linked to a verb such as the set {“open”, “opens”, “close”, “closes”, “puts”, “put”, “secure”, ...}. In another example, a subset may be linked to a part or device of the vehicle such as the set {“door”, “door”, “opening”, “air conditioner”, “cold generator”, ...}.

[0049] For example, to illustrate, a label may be the expression "question" to identify an expression introducing a question. Another label may be the expression "activation" to identify an element "open", "opens", ... of a subset. Another label may be the expression "opening" to identify an element "door", "door", "opening", ... Thus, words or expressions having similar meanings or which appear in similar contexts will have the same label.

[0050] Advantageously, said first model is further trained from an extraction of text from user manuals of said vehicle. This makes it possible to determine quite simply the subsets linked to a verb, the subsets linked to a part or a device of the vehicle. In one operating mode, a label can be a reference to a paragraph or a part of a user manual, which then substantially limits the size of the first model.

[0051] The first model being predetermined, the automatic learning is carried out offline, before implementation in the RAM 102 or mass memory 104 of the device 101. All of the labels and lists of labels are available. All or part of the set can be used for the learning of the models described below or for the construction of the knowledge graph described below.

[0052] Step 230, RxData, is a step of receiving data, said data relating to said vehicle, one of said data being at least one identifier making it possible to determine a make and / or a model of said vehicle.

[0053] It is known that a vehicle is identified by its vehicle identification number or VIN, from the English “Vehicle Identification Number”, a standardized alphanumeric sequence of characters. The first characters make it possible to identify a brand or manufacturer such as a Peugeot® vehicle, a Cirtoën® vehicle, ..., a car model such as 208®, C3®, ..., a type of engine, a year of manufacture of the model, ... Also, the calculators, ECU or device similar to the device 101, of a vehicle have other data enabling the make and model of the vehicle to be identified. These computers are capable of transmitting data relating to said vehicle to the device 101.

[0054] Advantageously, said received data also includes data relating to said occupant. For example, using driver or passenger recognition devices, the first and last names of the occupants and / or the driver can be received.

[0055] For a given set of vehicles, for example vehicles of the same make and model, it is possible to construct, offline, a set of data that can be received. This set can be used for the learning of the models described below or for the construction of the knowledge graph described below.

[0056] Step 240, WList, is a step of determining a list of words from said list of labels, said received data and a predetermined knowledge graph. Said graph comprises nodes, said nodes being representative of entities, the entities being predetermined from said labels, and a set of at least one word, said at least one word characterizing a predetermined set of received data. Said graph also comprises arcs, said arcs being representative of a functional relationship between said entities, said arcs being described by at least one word.

[0057] Knowledge graphs, KG acronym for "Knowledge Graph", are known. A KG is a data structure that organizes knowledge in the form of graphs by linking entities and relationships. The difficulty is to have a list of entities. In the present invention, this list is determined from the set of possible labels of the first model and from the set of said at least one word, said at least one word characterizing a predetermined set of received data. Said at least one word can be considered similarly to other labels or additional labels that may have the value, for example, "Vehicle", "Peugeot®", "Citroën®", "208®", "C3®", ...

[0058] Once the list of entities has been obtained, a person skilled in the art knows how to construct the KG and determine the relationships between these entities. The KG is predetermined, it is therefore carried out “offline”. Only the model is implemented in the RAM 102 or mass memory 104. The model can be interpreted by computer in a known manner, making it possible to capture complex relationships between the entities and to provide semantics to the data. Thus, at the output of this KG model, from the list of labels determined in step 220, and from the data of step 230, we have a list of words. For example, this list of words can be a list of instructions such as “to activate child safety”, “open rear door”, “command in field”, “insert key”, “turn left if ARG”, “turn right if ARD”.

[0059] If said first model is further trained from an extraction of text from user manuals of said vehicle, the size of the KG model can be significantly reduced, an entity being able to be a reference to a paragraph or part of a user manual.

[0060] Step 250, SList, is a step of determining a list of sentences from said list of words and a second predetermined model, said second model being a machine learning model from the natural language processing domain, said second model being at least trained from a set of words from the knowledge graph.

[0061] Said second model is a machine learning model from the field of natural language processing, NLP, such as neural networks, language models, classification and clustering algorithms, statistical methods, etc. In one operating mode, said second model is a neural network type model whose design method (number of nodes, number of layers, choice of activation functions, etc.) and training method (parameterization of activation functions, etc.) are well known, the remaining difficulty is to succeed in choosing inputs and outputs of the models that are relevant to the problem that must be solved.

[0062] Said second model is trained from a set of words from the knowledge graph. It is known to extract all possible lists of words from the KG since we know all possible inputs (lists of labels, lists of at least one word). Said second model is predetermined, it is therefore carried out “offline”. Only the model is implemented in the RAM 102 or mass memory 104.

[0063] Advantageously, said second model is further trained from an extraction of text from user manuals of said vehicle, which simplifies the learning process, a list of sentences being able to be mainly a paragraph of the user manual. If the first model is also trained from an extraction of text from user manuals, the determination of the list of sentences is more robust to variations in verbal queries for the same operating request.

[0064] Advantageously, said list of phrases is further determined from said received data, said second model being further trained from metadata relating to a predetermined set of received data. Thus, the list of phrases is contextualized according to the make and / or model of the vehicle. Said second model is even more optimized. For example, a metadata item is “Citroën C3®”. The second model is only trained on operating aids for a Citroën C3® vehicle. It does not contain information on any other vehicle manufacturer or model. The second model implemented does not will include only information related to said vehicle.

[0065] In one operating mode, if said received data also includes data relating to said occupant, the list of sentences can be contextualized and personalized in relation to the occupant.

[0066] For example, a list of sentences might be "Dear Pierre, to activate the child safety lock on your Citroën C3® vehicle, first open the relevant rear door.", "The control is in the field of each rear door.", "It is indicated by a marking on the bodywork.", "Insert the key into the control as far as it will go.", "Turn the inserted key to the right for a left rear door. Turn the inserted key to the left for a right rear door.".

[0067] There are NLGs, from the English "Natural Language Generation", NLP sub-models specially designed to generate sentences from a list of instructions. Advantageously, the second model is obtained from an existing NLG after specialization, additional training and simplifications, from a set of words from the knowledge graph and, if necessary, from an extraction of text from user manuals of said vehicle and / or from metadata relating to a predetermined set of received data.

[0068] Step 260, TxSList, is a step of transmitting said list of sentences to said audio system to provide operating information of said vehicle by voice synthesis. The audio system is then able, by its voice synthesis system, to transmit the operating information to help and guide the occupant.

[0069] Advantageously, said at least one vehicle is a vehicle of the same make and / or model as said vehicle. Thus, the first model, the knowledge graph and the second model are even more optimized and of reduced size, taking into account only the make and / or model of the vehicle.

[0070] The present invention is not limited to the embodiments described above as examples: it extends to other variants.

[0071] For example, the method has been described according to a sequence of steps. Certain steps can be carried out in parallel or according to another sequence.

[0072] Thus, an exemplary embodiment has been described above in which words, expressions, groups of words, etc. are involved. An expression may also be a group of words. Advantageously, the words, expressions, and groups of words may be encoded to be easily interpretable by a computer system or by a device 101.

Claims

Claims

1. Method (200) for providing operating information of a vehicle having an audio voice recognition and synthesis system, to assist and guide an occupant of said vehicle, said occupant issuing a verbal request, said method being implemented by a processor (103) and comprising the steps of: - Receiving (210) a text generated from said audio system processing said verbal request; Determination (220) of a list of labels from said received text and a first predetermined model, • said first model being a machine learning model in the field of natural language processing, • said first model being trained from a predetermined set of texts representative of a verbal request for the provision of operating information for at least one vehicle, and from a predetermined set of said labels, • said labels being a group of characters, at least one word, used to identify and describe sets of words from said predetermined set of texts representative of a verbal request for the provision of operating information for at least one vehicle; Reception (230) of data, said data relating to said vehicle, one of said data being at least one identifier making it possible to determine a make and / or a model of said vehicle; Determining (240) a list of words from said list of labels, said received data and a predetermined knowledge graph, • said graph comprising nodes, said nodes being representative of entities, the entities being prede- terminated from said labels, and a set of at least one word, said at least one word characterizing a predetermined set of received data, and said graph comprising arcs, said arcs being representative of a functional relationship between said entities, said arcs being described by at least one word; - Determination (250) of a list of sentences from said list of words and a second predetermined model, said second model being a machine learning model from the natural language processing domain, • said second model being at least trained from a set of words from the knowledge graph; - Transmission (260) of said list of sentences to said audio system to provide operating information of said vehicle by voice synthesis.

2. The method of claim 1, wherein said list of sentences is further determined from said received data, said second model being further trained from metadata relating to a predetermined set of received data.

3. Method according to the preceding claim, said received data further comprises data relating to said occupant.

4. Method according to one of the preceding claims, wherein said first model is a neural network type model.

5. Method according to one of the preceding claims, in which said second model is a neural network type model.

6. Method according to one of the preceding claims, in said first model and said second model are further trained from an extraction of text from user manuals of said vehicle.

7. Method according to one of the preceding claims, wherein said at least one vehicle is a vehicle of the same make and / or model as said vehicle.

8. Device (101) comprising said audio system, a memory (102) associated with at least one processor (103) configured to implement the method according to one of the preceding claims.

9.

10. Vehicle comprising the device according to the preceding claim. Computer program comprising instructions which, when the program is executed by the device (101) according to claim 8, cause the latter to implement the method according to one of claims 1 to 7.

Images