Method and device for processing image data representing one or more images of a vehicle's environment

Abstract

Methods and devices exist for processing image data representative of an image of a vehicle's environment (10), the image data being obtained from an on-board camera (105, 106). To achieve this, the first voice data representing a voice request from this vehicle (10) are received from an on-board microphone (100) and processed to obtain the first text data (1010) representing a text query associated with said voice request. The text information is generated by feeding a query model trained with the first text data (1010), which queries the image data to produce visual functions correlated with the text information.The second text data (1020) representing a text response to the text query is generated by adding a language model trained with the text information; the second text data (1020) is then processed to obtain second expression data representing a voice response to the voice query. Figure 1.

Description

Title of the invention: Method and device for processing image data representative of one or more images of a vehicle's environment Technical field

[0001] The present application relates in general to image processing and, in particular, to natural language processing applied to image data. The present application also relates to the method and apparatus for processing image data representative of one or more images of the environment of a vehicle, for example a car. The present application also relates to a method and apparatus for assisting a driver in controlling a vehicle, for example an autonomous or semi-autonomous vehicle. The present application also relates to a method and device for processing audio and visual data by a generative assistant for vehicles operating with AI.

[0002] Context

[0003] This section is intended to introduce the reader to various aspects of the art, which may be related to various aspects of at least one exemplary embodiment of the present application described below. This discussion would be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present application.

[0004] Modern vehicles are generally equipped with a voice user interface configured to control and / or communicate with the vehicle's on-board systems using voice commands.

[0005] However, some of the advanced technology still requires text commands, which limits its use in a vehicle. Controlling some in-vehicle systems via touch interface requires the vehicle driver to use one of their hands to enter text commands, which can divert attention from the road, endangering the vehicle and its occupants.

[0006] Summary

[0007] The following section presents a simplified summary of at least one exemplary embodiment to provide a basic understanding of certain aspects of the present application. This summary is not a complete overview of an exemplary embodiment. It is not intended to identify key or critical elements of an embodiment. The following summary presents only certain aspects of at least one of the exemplary embodiments in a simplified form as a prelude to the more detailed description provided elsewhere in the document.

[0008] According to a first aspect of the present application, there is provided a method for processing image data representative of at least one image of an environment of a vehicle, the image data obtained from at least one on-board camera of the vehicle, the method consisting of: - to receive first expression data representative of a voice request from an on-board microphone of the vehicle; - processing first voice data representative of a text query associated with the voice query;- generating text information by feeding a trained query model with the first text data, the trained query model querying the image data to generate visual features based on the first text data, the text information being correlated with the visual features; - by generating second textual data representative of a textual response to the textual query by feeding a trained linguistic model with the textual information; - processing of text data of second text data to obtain voice data query.

[0009] Such a method makes it possible to obtain an answer to a question expressed vocally on the basis of the content of the images of the vehicle's environment, taking advantage of the advantageous capabilities of a linguistic model, for example, a large linguistic model. It allows the driver of a vehicle to obtain information about his environment while driving, the question being processed vocally and the answer to the question also being vocal.

[0010] In an exemplary embodiment, the trained linguistic model corresponds to a large linguistic model, called LLM.

[0011] In another exemplary embodiment, the trained query model corresponds to a query transformer comprising an image transformer interacting with an image encoder to obtain the visual features and a text transformer to obtain the textual information, the image transformer and the text transformer sharing the same self-attention layers.

[0012] In another exemplary embodiment, processing the first speech data involves predicting the first text data by feeding a self-supervised speech synthesis model.

[0013] In an additional exemplary embodiment, the speech-to-text model was trained on training data representative of noise specific to a vehicle environment.

[0014] In another exemplary embodiment, processing the second text data involves predicting the second voice data by feeding a text model to the self-monitored speech.

[0015] According to a second aspect of the present application, there is an apparatus for processing image data representing at least one image of an environment of a vehicle, in which the apparatus comprises a memory associated with at least one processor configured to implement the method in accordance with the first aspect of the present application.

[0016] According to a third aspect of the present application, a computer program product is provided, including instructions which, when the program is executed by one or more processors, cause the one or more processors to perform a method according to the first aspect of the present application.

[0017] According to a fourth aspect of the present application, there is a non-transitory storage means, a storage means carrying program code instructions for executing a method according to the first aspect of the present application.

[0018] The specific nature of at least one of the exemplary embodiments as well as other objects, advantages, features and uses of said at least one of these exemplary embodiments will be apparent from the following description of the examples taken in conjunction with the accompanying drawings. Brief description of the drawings

[0019] Reference will be made, for example, to the drawings which accompany the present application and in which:

[0020] [Fig.l] shows a schematic representation of a system and process for processing image data obtained from one or more cameras of a vehicle, according to at least one exemplary representation;

[0021] [Fig.2] shows a schematic representation of a portion of the system and process of [Fig.l] for obtaining a response to a query based on image data, in accordance with at least one exemplary embodiment;

[0022] [Fig. 3] illustrates a block diagram of an exemplary apparatus configured to process image data obtained from one or more cameras of the vehicle of [Fig. 1], in accordance with at least one exemplary representation; and

[0023] [Fig.4] shows a schematic block diagram of a method of processing image data obtained from one or more cameras of the vehicle of [Fig.l], according to at least one exemplary representation.

[0024] At least one of the exemplary embodiments is described in more detail below with respect to the accompanying figures, in which examples of at least one of the exemplary embodiments are illustrated. An exemplary embodiment, however, may be embodied in many alternative forms and should not be construed as limited to the examples presented herein. Accordingly, it should be understood that there is no intention to limit the exemplary embodiments to the particular forms disclosed. Rather, the disclosure is intended to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of this application.

[0025] [Fig.l] illustrates a schematic representation of an on-board system of a vehicle 10 and the process implemented by the system in accordance with exemplary embodiments.

[0026] [Fig.l] illustrates a vehicle 10. The vehicle 10 corresponds, for example, to a vehicle equipped with a combustion engine, an electric motor or even a hybrid vehicle equipped with a combustion engine and one or more electric motors, for example a plug-in hybrid vehicle. The vehicle 10 therefore corresponds, for example, to a land vehicle, for example a car, a truck, a bus or a motorcycle.

[0027] The vehicle 10 may be equipped with various ADAS (Advanced Driver-Assistance System) systems configured to assist the driver of the vehicle 10 in controlling the vehicle. The vehicle 10 may be a semi-autonomous or autonomous vehicle, i.e., a vehicle traveling with an autonomy level greater than or equal to 2 in an autonomy range between 1 and 5, a level 1 corresponding to a vehicle having a minimum level of autonomy and a level 5 corresponding to a vehicle having a maximum level of autonomy (i.e., a fully autonomous vehicle).

[0028] The vehicle 10 is equipped with one or more on-board cameras 105 and 106, for example one or two front cameras 105 and one rear camera 106. Each camera is configured to acquire images of parts of the external environment surrounding the vehicle 10.

[0029] The vehicle 10 is also equipped with one or more microphones 100 and one or more loudspeakers 104. The microphone(s) 100 and the loudspeaker(s) 104 are arranged inside the vehicle 10 in order to respectively acquire the voice commands, requests or queries of a passenger, for example the driver, and to emit sounds, for example a voice response to the voice commands, requests or queries addressed to the passenger.

[0030] The on-board systems, cameras, microphones and speakers are each controlled by one or more computers, for example ECUs (electronic control units). These computers form, for example, a multiplexed architecture for the realization of various services useful for the proper functioning of the vehicle and for helping the driver and / or passengers of the vehicle to control the vehicle 10 by controlling the on-board systems of the vehicle 10. The computers communicate and exchange data with each other via one or more computer buses, for example a communication bus such as CAN (Controller Area Network), CAN FD (Controller Area Network Flexible Data-Rate Bus Type), FlexRay (according to ISO 17458), LIN (Local Interconnect Network) or Ethernet (according to ISO / IEC 802-3).

[0031] The processing of the image data obtained or acquired by one or more cameras 105, 106 is advantageously implemented by one or more on-board computers of the vehicle 10. The operations of the process are illustrated by blocks 101, 102 and 103 of [Fig. 1].

[0032] Blocks 101 to 103 represent the indifferent operations / steps of a process / method and the processing units that implement these operations / steps of the process / method. Processing units 101 to 103 may be implemented as a single element or as separate elements, as a combination of hardware and software known to those skilled in the art.

[0033] In block 101, first voice data representative of a voice request is received from one or more onboard microphones 105.

[0034] The voice query can be made by the driver of the vehicle 10 while driving. For example, such a voice query corresponds for example to: "How many pedestrians are on the road?"

[0035] The first voice data is processed to obtain the first text data 1010 representative of a text query associated with the voice query. The text query thus generated corresponds to a text version of the voice query obtained by processing the first voice data. The first text data 1010 corresponds for example to a set of words resulting from the processing of the first voice data.

[0036] The conversion of the first voice data into first text data 1010 can be obtained by implementing any automatic voice recognition (ASR) method known to persons skilled in the art.

[0037] The first voice data are, for example, inputs into a speech synthesis model that generates the first text data. The speech synthesis model may correspond to a statistical model such as a hidden Markov model.

[0038] In one embodiment, the speech synthesis model corresponds to an encoder-decoder transformer model.

[0039] A transformer corresponds to a deep learning architecture that relies on the attention mechanism. A transformer is known to those skilled in the art and, for example, described in the article "Attention is All You Need", by Ashish Vaswani et al., published on June 12, 2017.

[0040] A transformer model uses an encoder / decoder architecture. The encoder includes encoder layers that process input tokens iteratively from one layer to the next. Input tokens are obtained by parsing the input data (e.g., speech data) through a byte-pair encoding tokenizer, and each token can be converted to a vector by looking up from an embedding table. The decoder includes decoder layers that iteratively process the encoder output as well as the decoder output tokens.

[0041] The speech-to-text model can be trained using several publicly available speech datasets, such as MLS (Multilingual LibriSpeech), Fleurs, MMS (Massively Multilingual Speech), as described in the article "Scaling Speech Technology to 1,000+ Languages", by Vincent Pratap al, published on May 22, 2023.

[0042] The speech-to-text transformer model may be initialized from a pre-trained encoder checkpoint and a pre-trained decoder checkpoint. For example, a pre-trained transformer-based language model, e.g., "Wav2Vec2", "Hubert" (as described in the paper by Vincent Pratap et al.) may serve as the encoder and pre-trained auto-coding models, e.g., "GPT2", as well as the pre-trained decoder portion of the sequence-to-sequence models, e.g., the decoder of "BART". may be used as the decoder. Depending on the architecture chosen for the decoder, the cross-attention layers may be initialized randomly. Initializing the speech-to-text model from a pre-trained encoder and decoder may require fine-tuning the model on a downstream task.

[0043] Fine-tuning of the model can also be achieved by introducing self-induced noise into the pre-training datasets to enable the model to better handle the specific environment of a moving vehicle.

[0044] In block 102, the second textual data 1020 representative of a textual response to the textual query is generated by feeding a trained language model, for example an LLM (Grand Language Model) with textual information. The textual information, for example the text tokens, can be obtained from a trained query model, for example a query transformer, fed with the first textual data generated in block 101.

[0045] The trained query model is configured to query the image data to generate visual features from one or more images acquired by the onboard cameras 105, 106 based on the first textual data, the textual information being correlated with the visual features.

[0046] The output of the query process may be processed (e.g., by projection operations) to match the dimension of the required embedding in the input of the language model generating a textual response to the textual query received in the query model input.

[0047] A detailed description of the exemplary embodiment of block 102 and related operations is provided below, with respect to [Fig.2].

[0048] In box 103, the second text data 1020 is processed to obtain second speech data representative of a voice response to the voice request received from one or more microphones 100.

[0049] The data of the second speech is then transmitted to one or more on-board loudspeakers 104 to give the response to the person, for example the driver, who pronounced the voice request.

[0050] By way of illustration, the voice response to the voice question "How many pedestrians are on the road?" corresponds for example to: "There are 5 pedestrians on the road." The response to the question is obtained from the content of one or more images acquired by the on-board cameras 105, 105 after the processing at block 102.

[0051] The processing at block 103 may correspond to any method known to those skilled in the art, for example, a method of converting language to speech, for example, a process called speech synthesis. Such a method is for example described in "From Text to Speech: The MITalk System", by Allen et al., published in 1987.

[0052] According to a non-limiting embodiment, the processing of the second textual data 1020 to generate the second voice data corresponds to a deep learning method based on a pre-trained text-to-speech model.

[0053] Like the speech-to-text model, the speech-to-text model is a model trained during a training process and is configured to classify the second text data 1020 to predict the second speech data.

[0054] The text-to-speech model may also correspond to an encoder-decoder transformer model. The training process of the speech synthesis model may be based on large-scale collections of unlabeled speech synthesis and text corpora. A joint pre-training method for aligning textual and acoustic information into a unified semantic space may be implemented, as described in "SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing," by Junyi Ao et al., published on May 24, 2022.

[0055] The text-to-speech model can correspond to the "SpeechT5" model presented in the aforementioned paper by Junyi Ao et al., whose framework consists of a shared encoder-decoder network and six specific modal (speech / text) pre / post-networks. After preprocessing the input speech / text through the pre-networks, the shared encoder / decoder network models the sequence-to-sequence transformation, and then the post-networks generate the output in the speech / text modality based on the output of the decoder.

[0056] The training process may further comprise, after the pre-training phase, a fine-tuning operation to refine the codec-decoder backbone by losing the downstream task.

[0057] [Fig.2] illustrates a schematic representation of the generation of a text response to the text query at block 102 according to the exemplary embodiments.

[0058] Blocks 21 to 24 represent indifferent operations / steps of the process / method implemented in block 102 and processing units, which may be hardware and / or software, implementing these operations / steps of the process / method in block 102.

[0059] A vision-based language model, for example a large-format vision-based language model, is implemented at block 102 to generate a text response represented by a second text data item 1020 to the text query / question represented by the first text data item 1010.

[0060] Block 22, corresponding to a query module, is configured to serve as a link or bridge between block 21, corresponding to an image encoder, and block 24, corresponding to an LLM. Block 23 is configured to transform the output embeddings of block 22 to correspond to the dimension of the text embedded in the LLM 24. Parts of such an architecture are described in "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models", by Junnan Li et al, published June 15, 2023.

[0061] A query transformer 22, called Q-Former, corresponds to a lightweight transformer that uses learnable query vectors to extract visual features from the frozen image encoder 21. It acts as an information bottleneck between the frozen image encoder 21 and the frozen LLM 24, where it feeds the most useful visual feature to the LLM 24 to output the response 1020 to the input text 1010. In a first pre-training phase, learning of the linguistic representation of vision is performed to enforce the Q-Ex 22 to learn the most relevant visual representation for the input text 1010. In a second pre-training phase, generative learning of vision to language is done by linking the output of the Q-Ex 22 to the frozen LLM 24 and training the Q-Ex 24 so that its visual output representation can be interpreted by the LLM 24.

[0062] Q-Former 24 is proposed as a training module for connecting a frozen image encoder 21 and a frozen LLM 24. It extracts a fixed number of output features from the image encoder, regardless of the input image resolution. It consists of two transformer sub-modules that share the same self-attention layers, the first being an image transformer that interacts with the frozen image encoder 21 for visual feature extraction and the second a text transformer that can function as a text encoder and a text decoder. A number of primeable query embeddings are created for the image transformer. Queries interact with each other through self-attention layers, and with frozen image features through cross-attention layers (inserted every other transformer block). Additionally, queries can interact with the input text 1010 through the same self-attention layers. Depending on the pre-training task, different self-attention masks can be used to control the interaction between queries and the input text 1010. The Q-Ex 22 is initialized with the pre-trained weights of BERT-base, while the cross-attention layers are randomly initialized.At the representation learning stage, Q-Former 22 is used in conjunction with the frozen image encoder 24 to perform pre-training using image-text pairs. The main goal of Q-Former is to learn how to extract the visual representation that is most informative from the input text 1010.

[0063] Three pre-training objectives are jointly optimized, with the input format and model parameters being shared across the three pre-training objectives. The main objective of using three pre-training strategies is to use a different masking strategy between queries and text to control their interaction. The three pre-training objectives are: - Matching images and texts; - Image-based text creation; and - Image-text contrastive learning.

[0064] Image-text contrastive learning is a typical contrastive learning where the model learns to align the image and text representations in a way that maximizes their mutual information. To do this, the similarity of a positive image-text pair is compared to that of negative pairs. The output query representation "Z" of the image transformer is aligned with the text representation "t" of the text transformer. A unimodal self-attention mask is used to prevent the queries and the input text 1010 from seeing each other (to prevent information leakage).

[0065] The image-based text generation loss is used to train the Q-Ex model 22 to generate text information based on given input images. However, the Q-Former architecture does not have a direct communication path between the frozen image encoder 21 and the text tokens corresponding to the text information. To generate text, the necessary information must first be extracted by the queries and then relayed to the text tokens through self-attention layers. Therefore, the queries are forced to extract features visual teristics that encompass all information in the input text 1010. To regulate query-text interaction, a multimodal causal self-attention mask is used. Queries can interact with each other, but they cannot attend to text tokens. On the other hand, each text token can respond to all queries and its previous text tokens.

[0066] The objective of image-text matching (ITM) is to establish a detailed correlation between image and text representations. The ITM task involves binary classification in which the model must predict whether an image-text pair is positive (matched) or negative (unmatched). This is achieved by using a bidirectional self-attention mask that allows all queries and texts to attend to each other, resulting in the output query embedding "Z" that captures multimodal information. These embeddings are then fed into a two-class linear classifier to obtain a logit for each class. The output matching score is obtained by averaging the logs of all queries.

[0067] The generative pre-training phase involves leveraging the generative linguistic capability of a frozen LLM by connecting it to Q-Ex 22 (with a frozen image encoder 21). A fully connected (FC) layer 23 is used to linearly project the output query embedding Z to match the dimension of the LLM's text embedding. These embedding projections serve as soft visual prompts that allow the LLM to incorporate visual information extracted by Q-Ex 22. Since Q-Ex 22 is already pre-trained to extract informative visual representations about the language, it functions as an information bottleneck that provides relevant information to the LLM while filtering out irrelevant visual data. This helps mitigate the forgetting problem by reducing the burden on the LLM to learn vision language alignment.

[0068] Two types of LLM can be used: decoder-based LLM and decoder-based LLM. For decoder-based LLM, it is pre-trained using language modeling loss, where frozen LLM is responsible for generating text conditioned on the visual representation of Q-Ex. On the other hand, for encoder-decoder-based LLM, it is pre-trained using prefix language modeling loss. In this approach, a text is split into two parts, where the prefix text is concatenated with the visual representation as input to the LLM encoder. The suffix text is then used as the generation target for the LLM decoder.

[0069] The vision-based language model may be refined following the pre-training process to improve understanding of road conditions and answering specific driver questions, for example in an autonomous driving scenario. To achieve this goal, a finishing dataset is used, as described in "Pano-AVQA: Audio-Visual Question-Based Question Answering on 360° Videos", by Heeseung Yun et al., published on October 11, 2021.

[0070] The image encoder 21 corresponds, for example, to VIT-g of EVA-CLIP®, such as a transformer described in "Scaling Vision Transformers", by Xiaohan Zhai et al., published on June 8, 2021. The LLM 24 may correspond to the response generation model OPT (Open Pre-training Transformer), as described in "OPT: Open Pre-training Transformer Language Models", by Susan Zhang et al., published on May 2, 2022. June 21, 2022 [3]. The image encoder 21 may, for example, be pre-trained for 250,000 steps in the first stage, and the LLM 24 may be pre-trained for 80,000 steps in the second stage. A batch size of 1680 is used for ViT-g, and a batch size of 1920 is used for OPT in the second stage. The training process uses an optimizer described in "ADAM: A Method for Stochastic Optimization," by Diederik P.Kingma and Jimmy Lei Ba, published on January 30, 2017, with parameters 1 = 0.9, 2 = 0.98 corresponding to hyper-parameters controlling the exponential decay rates of the moving averages, and a weight decay of 0.05. A cosinal learning rate decay rate is used with a maximum learning rate of le-4. In the second step, the learning rate is set to 5e-5.

[0071] [Fig. 3] presents a block diagram illustrating an exemplary system or apparatus 3 in which various exemplary aspects and embodiments are implemented.

[0072] The apparatus 3 may be incorporated as one or more devices, including the various components described below. In various embodiments, the apparatus 3 may be configured to implement one or more of the aspects described in the present application.

[0073] Examples of equipment that may constitute all or part of the apparatus 3 include the computer, the electronic control unit (ECU), the information processing device, or other communication devices. The elements of the apparatus 3, individually or in combination, may be incorporated into a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of the apparatus 3 may be distributed across multiple ICs and / or discrete components. In various embodiments, the apparatus 3 may be communicatively coupled to other similar systems, or other electronic devices, for example, via a communication bus 320 through input interfaces and / or dedicated output 32.

[0074] The apparatus 3 may include at least one processor 30 configured to execute instructions loaded therein to implement, for example, the various aspects described in the present application. The processor 30 may include integrated memory, an input output interface, and various other circuits known in the art. The apparatus 3 may include at least one memory 31 (e.g., a volatile memory device and / or a non-volatile memory device). The apparatus 3 may include a storage device, which may include non-volatile memory and / or volatile memory, including, but not limited to, an electrically erasable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (SRAM), flash memory, a magnetic disk drive, and / or an optical disk drive.The storage device may include an internal storage device, an attached storage device, and / or a network accessible storage device, as non-limiting examples.

[0075] The apparatus 3 may include an encoder / decoder module configured, for example, to process the image data to provide encoded / decoded text data, and the encoder / decoder module may include its own processor and memory. The encoder / decoder module may represent the module(s) that may be included in a device to perform the encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module may be implemented as a separate element or incorporated into the processor 30 as a combination of hardware and software known to those skilled in the art.

[0076] The program code to be loaded onto the processor 30 or the encoder / decoder to perform the various aspects described in the present application may be stored in a storage device and then loaded onto the memory 31 for execution by the processor 30. In accordance with various embodiments, one or more of the processors 30, the memory 31, the storage device, and the encoder / decoder module may store one or more different elements when performing the processes described in the present application. These stored elements may include, but are not limited to, data representative of the audio content, data representative of the video content, haptics-related data, a bitstream, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operational logic.

[0077] In several modes of execution, memory within the processor 30 and / or the encoder / decoder module may be used to store the instructions and provide working memory for processing that may be performed during processing, the encoding or decoding of data.

[0078] In other embodiments, however, memory external to the processing device (e.g., the processor may be the processor) may be used for one or more of these functions. The external memory may be memory 31 and / or the storage device, e.g., dynamic volatile memory and / or non-volatile flash memory. In at least one embodiment, fast external dynamic volatile memory such as random access memory may be used as working memory for processing data.

[0079] [Fig.4] shows a schematic block diagram of the steps of a method for processing image data representing at least one image of an environment of a vehicle 10, the image data being obtained from at least one on-board camera of the vehicle 10, in accordance with at least one exemplary embodiment.

[0080] The method can, for example, be implemented by an on-board device of the vehicle 10 or by the device 3 of [Fig.3].

[0081] During the first step 41, the first voice data representative of a voice request are received from an on-board microphone of the vehicle.

[0082] In the second step 42, the first voice data are processed to obtain the first textual data representative of a textual query associated with the voice query.

[0083] In the third step 43, the textual information is generated by feeding a trained query model with the first textual data, the trained query model querying the image data to generate visual features based on the first textual data, the textual information being correlated with the visual features.

[0084] In the fourth step 44, a second textual data representative of a textual response to the textual query is generated by feeding a linguistic model trained with the textual information.

[0085] In the fifth step 45, the second textual data is processed to obtain second speech data representative of a voice response to the voice request.

[0086] The terminology used herein is intended only to describe particular embodiments and is not intended to limit embodiments. It will be understood that the terms "includes / includes" and / or "includes / includes", when used in this specification, may specify the presence of stated features, elements, and / or components, for example, but do not preclude the presence or addition of one or more other features, elements, components, and / or groups of such elements. In addition, when an element is referred to as "reactive" or "connected" to another element, it may be directly reactive or connected to the other element, or intervening elements may be present. In contrast, when an element is called "directly sensitive" or "directly linked" to another element, there are no intervention elements present.

[0087] It should be noted that the use of any of the symbols / terms " / ", "and / or" and "at least one of", for example, in the cases of "A / B", "A and / or B" and "at least one of A and B", may be intended to encompass the selection of the first option (A) or the selection of the second option (B) only, or both options (A and B). For example, in the cases "A, B and / or C" and "at least one of options A, B and C", this wording is intended to encompass the selection of the first option (A) only or the second option (B) only, or the selection of the third option (C) only, or the selection of the first and second options (A and B) only, or the selection of the third option (C), the first and third options (A and C) only, or the second and third options (B and C) only, or the selection of all three options (A and B and C).This can be extended, as is clear, to any of the ordinary skills in this field and related arts, for as many items as are listed.

[0088] Various numerical values ​​may be used in the present application. The specific values ​​may be, for example, purposes and the described aspects are not limited to these specific values.

[0089] It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, such elements are not limited by these terms. These terms are used only to distinguish one element from another. For example, a first element could be referred to as a second element and, likewise, a second element could be referred to as a first element without departing from the teachings of this application. No order is implied between a first element and a second element.

[0090] Reference to "an exemplary embodiment" or "an exemplary embodiment" or "a single implementation" or "an implementation", as well as other variations thereof, is frequently used to indicate that a particular feature, structure, characteristic, and so forth (described in connection with the embodiment / implementation) is included in at least one embodiment / implementation. Thus, appearances of the phrase "in an exemplary embodiment" or "in an exemplary embodiment" or "in an implementation" or "in an implementation", as well as any other variations, appearing at various places in this application do not necessarily all refer to the same embodiment.

[0091] Similarly, the reference herein to "in accordance with an exemplary embodiment / example / implementation" or "in an exemplary embodiment / example / implementation", as well as other variations thereof, is frequently used to indicate that a particular feature, structure or characteristic (described in connection with the exemplary embodiment / example / implementation) may be included in at least one embodiment / example / implementation. least one exemplary embodiment / example / implementation. Thus, the appearances of the phrase "in accordance with an exemplary embodiment / example / implementation" or "in an exemplary embodiment / example / implementation" in various places in the specification do not necessarily all refer to the same exemplary embodiment / example / implementation, nor are they a separate or alternative exemplary embodiment / examples / implementation, necessarily mutually exclusive of other exemplary embodiments / examples / implementations.

[0092] Although not explicitly described, the embodiments / examples and variations present may be used in any combination or subcombination.

[0093] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or deleted to produce other implementations. In addition, one of ordinary skill in the art will understand that other structures and processes may be substituted for those disclosed and that the resulting implementations will have at least substantially the same function or functions, at least in substantially the same manner, to achieve at least substantially the same results as the disclosed implementations. Accordingly, these and other implementations are contemplated in this application.

Claims

Claims

1. Method for processing image data representative of at least one image of the environment of a vehicle (10), said image data being obtained from at least one on-board camera (105, 106) of said vehicle (10), the method comprising the steps of: - receiving (41) first voice data representative of a voice request from an on-board microphone (10) on said vehicle (10); - processing (42) the first voice data to obtain first text data (1010) representative of a text request associated with said voice request;- generating textual information (43) by feeding a trained query model (22) with said first textual data (1010), said trained query model (22) interrogating said image data to obtain visual characteristics based on said first textual data (1010), the textual information being correlated with these visual characteristics; - generating (44) second textual data (1020) representative of a textual response to said textual query by feeding a trained linguistic model (24) with said textual information; - processing (45) the second textual data (1020) to obtain second voice data representative of a voice response to the voice query.;

2. The method of claim 1, wherein said trained language model (24) corresponds to a large language model, called LLM.

3. A method according to claim 1 or 2, wherein said trained query model (22) corresponds to a query transformer comprising an image transformer interacting with an image encoder (21) to obtain said visual features and a text to obtain said textual information, said image transformer and said text transformer sharing the same self-attention layers.

4. Computer program comprising instructions for implementing the method according to one of the preceding claims, when these instructions are executed by a processor.

5. A non-transitory storage medium, which carries instructions for program code for carrying out the method according to one of claims 1 to 3.

6. Device (3) for processing image data representing at least one image of an environment of a vehicle, wherein said device comprises a memory (31) associated with at least one processor (30) configured to implement the method according to one of claims 1 to 3.

7. Vehicle (10) comprising the device (3) according to claim 6.

Images