Autonomous vehicles

Abstract

An autonomous vehicle comprises a sensor unit configured to generate an environmental signal corresponding to a surrounding environment. A vector generator unit generates a numeric vector from the environmental signal received from the sensor unit. A pre-processing unit receives the numeric vector and generates a vector embedding from that numeric vector. The vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle. A language model unit uses a machine learning language model to generate a textual data output based on the vector embedding. A controller receives the textual data output and generates one or more driving commands, and the controller operates the autonomous vehicle based on those commands.

Description

[0001] Autonomous Vehicles

[0002] Technical Field

[0003] This invention relates to autonomous vehicles, such as autonomous cars, trucks, buses, vans, and such like. Particularly, though not exclusively, the present invention is concerned with 'end-to-end' methods for autonomous driving.

[0004] Background Art

[0005] Modern scalable autonomous driving systems, whether they adopt an end-to-end approach using a single network, or a component-based configuration that combines learnable perception and motion planning modules, face common challenges. These systems often behave as 'black-boxes' in the decision-making process, making it especially difficult to endow them with out-of-distribution (OOD) reasoning and interpretability capabilities.

[0006] A fundamental challenge with modern autonomous driving systems is the lack of interpretability in the decision-making process. Current autonomous driving systems often struggle with the interpretation and prediction of complex, real-world driving scenarios, especially for rare and unusual ones. Understanding why a decision is made is crucial for understanding areas of uncertainty, building trust, enabling effective human-AI collaboration, and ensuring safety.

[0007] Some current solutions, known in the art perse, utilize sensor fusion methods and routelocating algorithms, emphasizing environmental perception and decision-making based on predetermined rules. However, these systems often struggle with complex situations, and their interpretation and response capabilities can be limited.

[0008] There has been large progress in end-to-end deep learning methods for autonomous systems in recent years. However, a fundamental challenge with modern autonomous driving systems is the lack of interpretability in the decision-making process. Understanding why a decision is made is crucial for understanding areas of uncertainty, building trust, enabling effective human-AI collaboration, and ensuring safety.

[0009] Where deep neural networks are used, a variety of explainability methods have been introduced to understand the underlying decision processes such networks. For example, there are model-agnostic interpretability methods that generate explanations for individual predictions. Other methods such as gradient-based methods, saliency maps, and attention maps target the inner operations of models to explain the decision-making process.

[0010] The Applicant has appreciated a need for an improved method of providing improved systems and methods for autonomous driving that, inter alia, provide for enhanced explainability. In particular, the Applicant has appreciated a need to provide an improved architecture for autonomous driving systems and controllers which is not a black box, but rather than can be inspected and interrogated by human operators. Such an arrangement would be technically advantageous for a number of reasons, including but not limited to improving the safety, reliability, and performance of autonomous vehicles. While regulatory and legal considerations may impose a need for explainability, critically there is a technical imperative to being able to understand the behaviour of the technical system - the control systems and the autonomous vehicle as a whole - explainability is of high technical importance.

[0011] Summary of the Invention

[0012] In accordance with a first aspect, embodiments of the present invention provide an autonomous vehicle comprising: a sensor unit configured to generate an environmental signal corresponding to a surrounding environment of the autonomous vehicle; a vector generator unit configured to generate a numeric vector from the environmental signal received from the sensor unit; a pre-processing unit which receives as an input the numeric vector generated by the vector generator unit, wherein the pre-processing unit generates a vector embedding from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; a language model unit configured to use a machine learning language model to generate a textual data output based on the vector embedding; and a controller configured to receive the textual data output and generate therefrom one or more driving commands, wherein the controller is further configured to operate the autonomous vehicle based on said one or more driving commands.

[0013] The first aspect of the invention extends to a control system for an autonomous vehicle, the control system comprising: a vector generator unit configured to generate a numeric vector from a received environmental signal; a pre-processing unit which receives as an input the numeric vector generated by the vector generator unit, wherein the pre-processing unit generates a vector embedding from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; a language model unit configured to use a machine learning language model to generate a textual data output based on the vector embedding; and a controller configured to receive the textual data output and generate therefrom one or more driving commands, wherein the controller is further configured to operate the autonomous vehicle based on said one or more driving commands.

[0014] The first aspect of the invention also extends to a computer-implemented method of operating an autonomous vehicle, the method comprising: receiving or generating an environmental signal corresponding to a surrounding environment of the autonomous vehicle; generating numeric vectors from the environmental signal; generating vector embeddings from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; using a machine learning language model to generate a textual data output based on the vector embedding; generating one or more driving commands from the textual data output; and operating the autonomous vehicle based on said one or more driving commands.

[0015] The first aspect of the invention further extends to a non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to carry out a method of operating an autonomous vehicle, the method comprising: receiving or generating an environmental signal corresponding to a surrounding environment of the autonomous vehicle; generating numeric vectors from the environmental signal; generating vector embeddings from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; using a machine learning language model to generate a textual data output based on the vector embedding; generating one or more driving commands from the textual data output; and operating the autonomous vehicle based on said one or more driving commands.

[0016] The first aspect of the invention further extends to a computer software product comprising instructions that, when executed by a processor, cause the processor to carry out a method of operating an autonomous vehicle, the method comprising: receiving or generating an environmental signal corresponding to a surrounding environment of the autonomous vehicle; generating numeric vectors from the environmental signal; generating vector embeddings from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; using a machine learning language model to generate a textual data output based on the vector embedding; generating one or more driving commands from the textual data output; and operating the autonomous vehicle based on said one or more driving commands.

[0017] Thus it will be appreciated that embodiments of the present invention provide an advantageous arrangement in which control of the autonomous vehicle is performed using a language model. The Applicant has appreciated that language models - particularly Large Language Models (LLMs) - show remarkable capabilities in terms of reasoning and generic situation understanding. The remarkable abilities of large language models (LLMs) demonstrate early signs of artificial general intelligence (AG I), exhibiting capabilities such as out-of-distribution (OOD) reasoning, common sense understanding, knowledge retrieval, and the ability to naturally communicate these aspects with humans. These capabilities align well with the focus areas of autonomous driving and robotics in general.

[0018] The Applicant has further appreciated how to harness such abilities into the autonomous driving domain, aiming to improve both generalization and interpretability. As a result, the Applicant presents a novel architecture that integrates an object-level vector modality into the language models, which may be pre-trained LLMs.

[0019] Embodiments of the present invention offer an innovative approach to process and react to real world driving scenarios, including rare and complex situations, by leveraging the general-purpose reasoning capability learned by language models such as LLMs, significantly enhancing the driving system's capability.

[0020] The technically advantageous application of machine learning language models in an autonomous vehicle system allows for a more comprehensive understanding of general physics, world understanding, and driving rules and scenarios. Integrating language models such as LLMs into the decision-making process enables the system to interpret scenes more accurately, predict actions more effectively, and ensure safer driving conditions. It allows reuse of the knowledge from (e.g. pre-trained) models that can help in certain edge cases that were not foreseen by the developers.

[0021] Textual or symbolic modality, with its inherent suitability for logical reasoning, knowledge retrieval, and human communication, serves as an excellent medium for harnessing the capabilities of LLMs. However, its linear sequential nature poses limitations for nuanced spatial understanding, a crucial aspect of autonomous navigation. Pioneering work in visual language models (VLMs) has begun to bridge this gap by merging visual and text modalities, enabling spatial reasoning with the power of pre-trained LLMs. However, to effectively incorporate the new modality into the language representation space requires extensive pretraining with a significant volume of labelled image data. Embodiments of the present invention advantageously integrate numeric vector modality, a type of data used in robotics for representing speed, actuator positions and distance measurements, into pre-trained LLMs. Such modality is considerably more compact than vision which alleviates some of the VLM scaling challenges. To make the object-level vector comprehensible to the language model (e.g. LLM), a language generator is introduced that translates this numerical data into textual descriptions.

[0022] The machine learning language model could be trained during use. However, in some embodiments, the machine learning language model is pre-trained to transform textual inputs to driving commands.

[0023] Certain embodiments of the present invention fuse vectorized object-level 2D scene representation into a pre-trained LLM using adapters. This fusion enables the model to directly interpret and reason about comprehensive driving situations. As a result, the LLM is empowered to serve as the 'brain' of the autonomous driving system. In a particular set of embodiments, the LLM can interact directly with the simulator to facilitate reasoning and action prediction.

[0024] As outlined later herein, the Applicant's results showcase a driving LLM agent capable of providing a human-readable output outlining the system's reasoning about actions based on the current scenario. In some embodiments, a user interface unit is configured to receive a user input prompt, and to input the user prompt into the language model unit, wherein the language model unit generates a response to the user prompt based on said user prompt and the vector embeddings; wherein the response is provided as an output. In such embodiments, a human may interact with the control system of the autonomous vehicle to determine not only how the control system will react to a given driving scenario, but also why. Being able to understand the reasoning of the control system can provide confidence that the system not only makes the correct decision, but also that it is doing so for the correct reason. Additionally, it is technically advantageous in terms of debugging and troubleshooting the control system - a human operator (such as an engineer or technician) can probe the technical workings of the control system with a query and obtain a response, rather than needing to analyse the system as a black box. In some embodiments, the response is provided as an output via the user interface unit.

[0025] The sensor unit provides an environmental signal which corresponds to the environment surrounding the autonomous vehicle. Those skilled in the art will appreciate that there are many different types of sensors, known in the art perse, that may be used on an autonomous vehicle to sense its surroundings. In some embodiments, the sensor unit comprises one or more sensors from a group comprising: a camera; a microphone; a sonar sensor; an ultrasonic sensor; a radar sensor; an ultra-wideband radio sensor; a lidar sensor; an optical flow sensor; a magnetometer; a thermometer; a barometer; a moisture sensor; a humidity sensor; an inertial measurement unit; a gyroscope; an accelerometer; a touch sensor; a laser rangefinder; an infrared sensor; a vehicle speed sensor; a yaw rate sensor; a wheel speed sensor; a current sensor; a fuel level sensor; a voltage sensor; an altitude sensor; a tilt sensor; a Hall effect sensor; an odometer; and a geopositioning sensor. It will be appreciated that the sensor unit may comprise any combination of sensor types, and may comprise more than one of the same type of sensor, as appropriate. The term "geopositioning sensor" should be understood to include sensors for use with GNSS systems such as GPS, Gallileo, BeiDou, etc.

[0026] The information about the surroundings of the autonomous vehicle is used to determine how the vehicle is controlled. There are many environmental variables that may be important for the control system to consider when determining how to control the autonomous vehicle. In some embodiments, the numeric vector representation of the surrounding environment of the autonomous vehicle comprises one or more numeric values corresponding to: a number of detected vehicles; respective distances between detected vehicles and the autonomous vehicle; respective locations of detected vehicles; respective orientations of detected vehicles; respective speeds of detected vehicles; and / or respective bounding boxes of detected vehicles.

[0027] In some potentially overlapping embodiments, the numeric vector representation of the surrounding environment of the autonomous vehicle comprises one or more numeric values corresponding to: a number of detected pedestrians; respective distances between detected pedestrians and the autonomous vehicle; respective locations of detected pedestrians; respective orientations of detected pedestrians; respective speeds of detected pedestrians; and / or respective bounding boxes of detected pedestrians.

[0028] Cyclists may be treated as a type of vehicle, a type of pedestrian, or another category. Thus, in some further potentially overlapping embodiments, the numeric vector representation of the surrounding environment of the autonomous vehicle comprises one or more numeric values corresponding to: a number of detected cyclists; respective distances between detected cyclists and the autonomous vehicle; respective locations of detected cyclists; respective orientations of detected cyclists; respective speeds of detected cyclists; and / or respective bounding boxes of detected cyclists.

[0029] Additionally, or alternatively, the environmental signal may include information corresponding to the state of traffic signals to which the autonomous vehicle is subject. Thus, in some further potentially overlapping embodiments, the numeric vector representation of the surrounding environment of the autonomous vehicle comprises one or more numeric values corresponding to: a detected traffic signal; respective distances between a detected traffic signal and the autonomous vehicle; a respective location of a detected traffic signal; and / or a respective state of a detected traffic signal. Such traffic signals may, for example, include traffic lights where the illumination state of different coloured lights indicates required or permitted behaviour (e.g. red meaning 'stop', green meaning 'go', and so on). Other 'dynamic' traffic signals that may be represented in the environmental signal include level crossing signs, smart motorway signs, and so on. The traffic signals may, additionally or alternatively, include 'static' traffic signals such as stop signs, give way markings, speed limit signs, road signage (e.g. to indicate road layout, lane allocations, etc.), warning signs, and so on.

[0030] The control of the autonomous vehicle may, in some embodiments, be based on more than the environmental signals. In particular, the numeric vectors which are provided to the pre-processing unit may correspond to additional information, beyond the environmental signal. In some embodiments, the vector generation unit is further configured to generate a numeric vector from one or more operational parameters corresponding to operation of the autonomous vehicle; wherein the numeric vector generated from the one or more operational parameters are input to the pre-processing unit such that the vector embeddings further comprises a structured semantic representation of the one or more operational parameters. Such operational parameters may relate to states and / or variables corresponding to the autonomous vehicle itself.

[0031] In some such embodiments, the numeric vector representation of the one or more operational parameters comprises one or more numeric values corresponding to: a location of the autonomous vehicle; a speed of the autonomous vehicle; an orientation of the autonomous vehicle; a trajectory of the autonomous vehicle; a bounding box of the autonomous vehicle; one or more actuator positions of the autonomous vehicle; an accelerator state of the vehicle; and / or a brake state of the vehicle.

[0032] Additionally, or alternatively, the control of the autonomous vehicle may be based on the destination of the vehicle. This may be an end-destination and / or it may be a route (i.e. a planned path from the current or initial location of the autonomous vehicle to the intended end-destination). In some embodiments, the vector generation unit is further configured to generate a numeric vector from destination information corresponding to the autonomous vehicle; wherein the numeric vector generated from the destination information are input to the pre-processing function executed by the pre-processing unit such that the vector embeddings further comprises a structured semantic representation of the destination information. In some such embodiments, the destination information comprises a route of the autonomous vehicle.

[0033] It will be appreciated that a numeric vector comprises an array of numeric values. Some such vectors may represent discrete sources of information that may be tokenised and then transformed into vectors. Other vectors may represent continuous sources of information (e.g. vehicle speed) that may not need to be tokenised. Those skilled in the art will appreciate that a token is a parsed segment or unit of data (e.g. text), where each token is selected from a 'library' or 'vocabulary' of possible tokens - i.e. there is a predetermined, finite list of tokens available. A vector is a numerical representation often used to encapsulate information about a token, sequence of tokens, or other types of data. Some (or potentially all) of the numeric vectors referred to herein may, at least in some particular embodiments, be tokenised. Thus, in some embodiments, one or more numeric vectors generated by the vector generator unit are tokenised such that the respective numeric vector comprises one or more tokens, each token being selected from a predetermined set of tokens. However, not all data needs to be 'split up' into different tokens. Those skilled in the art will appreciate that the tokenisation (or not) of data is dependent on the specific type of data and its nature - tokenisation may be technically advantageous for certain types of data. The tokenisation may, in some such embodiments, be performed by the vector generator unit.

[0034] There are several different pre-processing architectures that could be used in accordance with embodiments of the present invention. However, at least in some embodiments, the pre-processing unit comprises: a vector encoder configured to receive the numeric vector generated by the vector generator unit, the vector encoder being further configured to encode said numeric vector into a latent space, thereby generating latent vectors. Those skilled in the art of deep learning will appreciate that a latent space is, in effect, a multi-dimensional space in which meaningful representations of 'real world' events or entities are encoded. These encoded latent vectors are typically a compressed representation of the real world, where the compression provides for a lower dimension representation of the real world (which typically exists at higher dimensionality).

[0035] In a set of such embodiments, the vector encoder comprises: a multilayer perceptron layer configured to receive the numeric vector generated by the vector generator unit; and a cross-attention layer configured to receive an output of the multilayer perceptron layer and a plurality of weight values, said cross-attention layer being further configured to use a cross-attention mechanism to generate the latent vectors.

[0036] In a particular set of potentially overlapping embodiments, the pre-processing unit further comprises: a vector former configured to receive the latent vectors from the vector encoder and to generate the vector embeddings from the latent vectors.

[0037] In some such embodiments, the vector former comprises: a self-attention layer configured to receive the latent vectors; and a second cross-attention layer configured to receive an output of the self-attention layer and a second plurality of weight values, said second cross-attention layer being further configured to use a second cross-attention mechanism to generate the vector embeddings.

[0038] Similarly, there are several language model unit architectures that could be used in accordance with embodiments of the present invention. In some embodiments, the language model unit comprises: a language model transformer configured to receive the vector embeddings and to use a respective attention mechanism to generate the one or more driving commands.

[0039] Language models such as LLMs can learn knowledge on their own, however applying such knowledge to real world problems typically requires a fine-tuning process. Fine-tuning involves pre-training the language model using a sufficiently large training set and then adapting that language model to some real-world task. Language models may be finetuned by providing the model with a prompt and an answer (often with additional context) and the model is then trained to generate an answer like the answer originally provided for training.

[0040] In some embodiments, the language model transformer is frozen. Those skilled in the art will appreciate that the term 'frozen' should be understood to mean that the weights used by the language model are not updated during fine-tuning, i.e. the internal weights of the language model are fixed at particular values after the initial training of the language model, but are not changed during the later fine-tuning process.

[0041] In some such embodiments, the attention mechanism used by the transformer is dependent on a query matrix (Q.), a key matrix (K), and a value matrix (V). In some further such embodiments, the language model unit further comprises a Low-Rank Adaptation (LoRA) module configured to supply the query matrix (Q.), the key matrix (K), and the value matrix (V). In a set of such embodiments, one or more of said matrices are trainable. Such an arrangement may advantageously allow a frozen LLM to be 'fine-tuned' by the LoRA. The Applicant has appreciated that the use of language models to provide for a response to a user prompt is novel and inventive it its own right. Such an arrangement advantageously allows for an autonomous vehicle that can provide the reasoning used by the control system for its decisions, providing confidence that the system not only makes the correct decision, but also that it is doing so for the correct reason. As outlined previously, this is technically advantageous in terms of debugging and troubleshooting the control system - a human operator (such as an engineer or technician) can probe the technical workings of the control system with a query and obtain a response, rather than needing to analyse the system as a black box.

[0042] Thus, when viewed from a second aspect, embodiments of the present invention provide an autonomous vehicle comprising: a sensor unit configured to generate an environmental signal corresponding to a surrounding environment of the autonomous vehicle; a vector generator unit configured to generate a numeric vector from the environmental signal received from the sensor unit; a pre-processing unit which receives as an input the numeric vector generated by the vector generator unit, wherein the pre-processing unit generates a vector embedding from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; a language model unit configured to use a machine learning language model; and a user interface unit configured to receive a user input prompt, and to input the user prompt into the language model unit, wherein the language model unit generates a response to the user prompt based on said user prompt and the vector embeddings; wherein the response is provided as an output.

[0043] The second aspect of the invention extends to a user interaction system for an autonomous vehicle, the user interaction system comprising: a vector generator unit configured to generate a numeric vector from the environmental signal received from the sensor unit; a pre-processing unit which receives as an input the numeric vector generated by the vector generator unit, wherein the pre-processing unit generates a vector embedding from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; a language model unit configured to use a machine learning language model; and a user interface unit configured to receive a user input prompt, and to input the user prompt into the language model unit, wherein the language model unit generates a response to the user prompt based on said user prompt and the vector embeddings; wherein the response is provided as an output.

[0044] The second aspect of the invention also extends to a computer-implemented method of operating an autonomous vehicle, the method comprising: receiving or generating an environmental signal corresponding to a surrounding environment of the autonomous vehicle; generating numeric vectors from the environmental signal; generating vector embeddings from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; receiving a user input prompt; inputting the user prompt into a language model to generate a response to the user prompt based on said user prompt and the vector embeddings; providing the response as an output.

[0045] The second aspect of the invention further extends to a non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to carry out a method of operating an autonomous vehicle, the method comprising: receiving or generating an environmental signal corresponding to a surrounding environment of the autonomous vehicle; generating numeric vectors from the environmental signal; generating vector embeddings from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; receiving a user input prompt; inputting the user prompt into a language model to generate a response to the user prompt based on said user prompt and the vector embeddings; providing the response as an output.

[0046] The second aspect of the invention further extends to a computer software product comprising instructions that, when executed by a processor, cause the processor to carry out a method of operating an autonomous vehicle, the method comprising: receiving or generating an environmental signal corresponding to a surrounding environment of the autonomous vehicle; generating numeric vectors from the environmental signal; generating vector embeddings from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; receiving a user input prompt; inputting the user prompt into a language model to generate a response to the user prompt based on said user prompt and the vector embeddings; providing the response as an output.

[0047] The Applicant has further appreciated that the fusion of a numeric modality and language models in an autonomous vehicle is also novel and inventive in its own right. Thus, when viewed from a third aspect, embodiments of the present invention provide an autonomous vehicle comprising: a sensor unit configured to generate an environmental signal corresponding to a surrounding environment of the autonomous vehicle; a vector generator unit configured to generate a numeric vector from the environmental signal received from the sensor unit; a pre-processing unit which receives as an input the numeric vector generated by the vector generator unit, wherein the pre-processing unit generates a vector embedding from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; and a language model unit configured to use a machine learning language model, the language model being trained to generate an output based on the structure textual data.

[0048] The third aspect of the invention extends to a user interaction system for an autonomous vehicle, the user interaction system comprising: a vector generator unit configured to generate a numeric vector from the environmental signal received from the sensor unit; a pre-processing unit which receives as an input the numeric vector generated by the vector generator unit, wherein the pre-processing unit generates a vector embedding from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; and a language model unit configured to use a machine learning language model, the language model being trained to generate an output based on the structure textual data, data.

[0049] The third aspect of the invention also extends to a computer-implemented method of operating an autonomous vehicle, the method comprising: receiving or generating an environmental signal corresponding to a surrounding environment of the autonomous vehicle; generating numeric vectors from the environmental signal; generating vector embeddings from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; using a trained machine learning language model to generate an output based on the structure textual data.

[0050] The third aspect of the invention further extends to a non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to carry out a method of operating an autonomous vehicle, the method comprising: receiving or generating an environmental signal corresponding to a surrounding environment of the autonomous vehicle; generating numeric vectors from the environmental signal; generating vector embeddings from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; using a trained machine learning language model to generate an output based on the structure textual data. The third aspect of the invention further extends to a computer software product comprising instructions that, when executed by a processor, cause the processor to carry out a method of operating an autonomous vehicle, the method comprising: receiving or generating an environmental signal corresponding to a surrounding environment of the autonomous vehicle; generating numeric vectors from the environmental signal; generating vector embeddings from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; using a trained machine learning language model to generate an output based on the structure textual data.

[0051] The model may undergo a pretraining phase that employs a numeric upweighting loss mechanism, which enhances alignment between the numeric vector modality and the language latent representations. Following this, the unique architectural design may be trained to establish a robust baseline model.

[0052] To enhance the accuracy of number decoding, the Applicant has appreciated that it may be particularly advantageous to use weighted cross-entropy on number and symbol tokens In other words, number and symbol tokens are made more important than alphabetical tokens fA-Z', and 'a-z') or other tokens. This approach adds further penalization on the decoded numbers, hence increasing the performance of the learned representation embeddings.

[0053] Such an arrangement is novel and inventive in its own right and thus, when viewed from a fourth aspect, embodiments of the present invention provide a computer-implemented method of training a machine learning model for use in a control system for an autonomous vehicle, the control system comprising: a pre-processing unit which receives as an input a numeric vector comprising information relating to the autonomous vehicle and / or a surrounding environment of said autonomous vehicle, wherein the pre-processing unit generates a vector embedding from said numeric vector, wherein the vector embedding comprises a structured semantic representation of said information; and a language model unit configured to use a machine learning language model to generate a textual data output based on the vector embedding; wherein the generation of the vector embedding by the pre-processing unit is dependent on a first set of weights, and wherein the generation of the textual data output by the machine learning language model is dependent on a second set of weights; wherein the method comprises: determining a set of values for the first set of weights by minimising an error between a ground truth label set and a candidate label set generated by the control system; wherein each label set comprises a plurality of symbols, each symbol being a selected symbol from a predetermined set of symbols, said set of symbols comprising an alphabetical symbol sub-set and a numeric symbol subset; wherein the error between the ground truth label set and the candidate label set is determined such that differences between corresponding symbols in the ground truth label set and the candidate label set are allocated: a first error value when a given symbol in the candidate label set is different from the corresponding symbol in the ground truth label set when said corresponding symbol belongs to the alphabetic symbol sub-set; and a second error value when a given symbol in the candidate label set is different from the corresponding symbol in the ground truth label set when said corresponding symbol belongs to the numeric symbol sub-set, wherein the second error value is greater than the first error value.

[0054] The fourth aspect of the invention further extends to a non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to carry out a computer-implemented method of training a machine learning model for use in a control system for an autonomous vehicle, the control system comprising: a pre-processing unit which receives as an input a numeric vector comprising information relating to the autonomous vehicle and / or a surrounding environment of said autonomous vehicle, wherein the pre-processing unit generates a vector embedding from said numeric vector, wherein the vector embedding comprises a structured semantic representation of said information; and a language model unit configured to use a machine learning language model to generate a textual data output based on the vector embedding; wherein the generation of the vector embedding by the pre-processing unit is dependent on a first set of weights, and wherein the generation of the textual data output by the machine learning language model is dependent on a second set of weights; wherein the method comprises: determining a set of values for the first set of weights by minimising an error between a ground truth label set and a candidate label set generated by the control system; wherein each label set comprises a plurality of symbols, each symbol being a selected symbol from a predetermined set of symbols, said set of symbols comprising an alphabetical symbol sub-set and a numeric symbol subset; wherein the error between the ground truth label set and the candidate label set is determined such that differences between corresponding symbols in the ground truth label set and the candidate label set are allocated: a first error value when a given symbol in the candidate label set is different from the corresponding symbol in the ground truth label set when said corresponding symbol belongs to the alphabetic symbol sub-set; and a second error value when a given symbol in the candidate label set is different from the corresponding symbol in the ground truth label set when said corresponding symbol belongs to the numeric symbol sub-set, wherein the second error value is greater than the first error value.

[0055] The fourth aspect of the invention also extends to a computer software product comprising instructions that, when executed by a processor, cause the processor to carry out a computer-implemented method of training a machine learning model for use in a control system for an autonomous vehicle, the control system comprising: a pre-processing unit which receives as an input a numeric vector comprising information relating to the autonomous vehicle and / or a surrounding environment of said autonomous vehicle, wherein the pre-processing unit generates a vector embedding from said numeric vector, wherein the vector embedding comprises a structured semantic representation of said information; and a language model unit configured to use a machine learning language model to generate a textual data output based on the vector embedding; wherein the generation of the vector embedding by the pre-processing unit is dependent on a first set of weights, and wherein the generation of the textual data output by the machine learning language model is dependent on a second set of weights; wherein the method comprises: determining a set of values for the first set of weights by minimising an error between a ground truth label set and a candidate label set generated by the control system; wherein each label set comprises a plurality of symbols, each symbol being a selected symbol from a predetermined set of symbols, said set of symbols comprising an alphabetical symbol sub-set and a numeric symbol subset; wherein the error between the ground truth label set and the candidate label set is determined such that differences between corresponding symbols in the ground truth label set and the candidate label set are allocated: a first error value when a given symbol in the candidate label set is different from the corresponding symbol in the ground truth label set when said corresponding symbol belongs to the alphabetic symbol sub-set; and a second error value when a given symbol in the candidate label set is different from the corresponding symbol in the ground truth label set when said corresponding symbol belongs to the numeric symbol sub-set, wherein the second error value is greater than the first error value.

[0056] The fourth aspect of the present invention further extends to an autonomous vehicle comprising: a sensor unit configured to generate an environmental signal corresponding to a surrounding environment of the autonomous vehicle; a pre-processing unit which receives as an input a numeric vector comprising information relating to the autonomous vehicle and / or a surrounding environment of said autonomous vehicle, wherein the pre-processing unit generates a vector embedding from said numeric vector, wherein the vector embedding comprises a structured semantic representation of said information; and a language model unit configured to use a machine learning language model to generate a textual data output based on the vector embedding; wherein the generation of the vector embedding by the pre-processing unit is dependent on a first set of weights, and wherein the generation of the textual data output by the machine learning language model is dependent on a second set of weights; wherein the autonomous vehicle is configured such that: a set of values for the first set of weights is determined by minimising an error between a ground truth label set and a generated candidate label set; wherein each label set comprises a plurality of symbols, each symbol being a selected symbol from a predetermined set of symbols, said set of symbols comprising an alphabetical symbol sub-set and a numeric symbol subset; wherein the error between the ground truth label set and the candidate label set is determined such that differences between corresponding symbols in the ground truth label set and the candidate label set are allocated: a first error value when a given symbol in the candidate label set is different from the corresponding symbol in the ground truth label set when said corresponding symbol belongs to the alphabetic symbol sub-set; and a second error value when a given symbol in the candidate label set is different from the corresponding symbol in the ground truth label set when said corresponding symbol belongs to the numeric symbol sub-set, wherein the second error value is greater than the first error value.

[0057] The fourth aspect of the invention yet further extends to a control system for an autonomous vehicle, the control system comprising: a pre-processing unit which receives as an input a numeric vector comprising information relating to the autonomous vehicle and / or a surrounding environment of said autonomous vehicle, wherein the pre-processing unit generates a vector embedding from said numeric vector, wherein the vector embedding comprises a structured semantic representation of said information; and a language model unit configured to use a machine learning language model to generate a textual data output based on the vector embedding; wherein the generation of the vector embedding by the pre-processing unit is dependent on a first set of weights, and wherein the generation of the textual data output by the machine learning language model is dependent on a second set of weights; wherein the control system is configured such that: a set of values for the first set of weights is determined by minimising an error between a ground truth label set and a candidate label set generated by the control system; wherein each label set comprises a plurality of symbols, each symbol being a selected symbol from a predetermined set of symbols, said set of symbols comprising an alphabetical symbol sub-set and a numeric symbol subset; wherein the error between the ground truth label set and the candidate label set is determined such that differences between corresponding symbols in the ground truth label set and the candidate label set are allocated: a first error value when a given symbol in the candidate label set is different from the corresponding symbol in the ground truth label set when said corresponding symbol belongs to the alphabetic symbol sub-set; and a second error value when a given symbol in the candidate label set is different from the corresponding symbol in the ground truth label set when said corresponding symbol belongs to the numeric symbol sub-set, wherein the second error value is greater than the first error value.

[0058] The alphabetic symbol sub-set may, in some embodiments, comprise the symbols a-z and / or A-Z. The numeric symbol sub-set may, in some potentially overlapping embodiments, comprise the symbols 0-9, and may further comprise a negative sign symbol (-)•

[0059] In some embodiments of the fifth aspect of the invention, the control system further comprises a controller configured to receive the textual data output and generate therefrom one or more driving commands, wherein the controller is further configured to operate the autonomous vehicle based on said one or more driving commands. In some potentially overlapping embodiments of the fifth aspect of the invention, the control system further comprises vector generator unit configured to generate the numeric vector from a received environmental signal. In some such embodiments, the autonomous vehicle comprises a sensor unit configured to generate the environmental signal corresponding to a surrounding environment of the autonomous vehicle.

[0060] It will be appreciated that the optional features described hereinabove in respect of embodiments of the first aspect of the invention apply equally, where technically appropriate, to the other aspects of the invention outlined herein.

[0061] Where technically appropriate, embodiments of the invention may be combined. Embodiments are described herein as comprising certain features / elements. The disclosure also extends to separate embodiments consisting or consisting essentially of said features / elements.

[0062] Technical references such as patents and applications are incorporated herein by reference.

[0063] Any embodiments specifically and explicitly recited herein may form the basis of a disclaimer either alone or in combination with one or more further embodiments.

[0064] In the context of this specification "comprising" is to be interpreted as "including". Aspects of the invention comprising certain elements are also intended to extend to alternative embodiments "consisting" or "consisting essentially" of the relevant elements.

[0065] The term "vehicle" as used herein should be understood to mean any kind of vehicle intended to travel on roads where some mechanical and / or electrical propulsion is used to drive the vehicle, whether operated autonomously or not. This includes, but is not limited to: cars, motorbikes, trucks, buses, coaches, vans, lorries, campervans, motor caravans, minibuses, limousines, all-terrain vehicles (ATVs), tractors, and other such vehicles that are mechanically or electrically driven. Where context allows (e.g. in respect of other vehicles detected by the autonomous vehicle), the term "vehicle" further extends to non-driven vehicles, i.e. those without mechanical or electrical propulsion. This includes, but is not limited to: bicycles, unicycles, tricycles, quadracycles, rickshaws, carts, wagons, horse-drawn carts or carriages, and other such vehicles that are not mechanically or electrically driven.

[0066] The terms "data" is used in different contexts herein to refer to digital information, such as that represented by known bit structures within one or more programming languages. In use, data may refer to digital information that is stored as bit sequences within computer memory.

[0067] The term "structured semantic representation" refers to information which is organized, modelled, and encoded to capture the relationships, meanings, and attributes of the data in a way that is understandable and utilisable by computational systems like neural networks, and in particular language models such as LLMs. Thus those skilled in the art will readily appreciate that the term "structured semantic representation" as used herein means a vector embedding in a format usable as an input to the language model. The content of that vector embedding corresponds to the information contained in the environmental signal, operational parameters, and / or destination information, as appropriate.

[0068] Certain machine learning models may operate on structured arrays of data of a predefined bit format. Using terms of the art, these may be referred to as "vectors", as used herein. Those skilled in the art will appreciate that a vector is a type of tensor and is a term often used for a first rank or ID tensor. Conversely, a tensor may, in some cases, be understood to be a collection of vectors. For the avoidance of doubt, the terms "vector" and "tensor" can be used interchangeably where technical context allows.

[0069] It should be noted that for machine learning methods multidimensional arrays, e.g. with a defined extent in multiple dimensions, may be converted to different forms for representation (e.g., within memory) as a sequence or vector of values stored according to the predefined format (e.g., n-bit integer or floating point number, signed or unsigned). For example, images taken from a camera are typically a 3D tensor (typically containing RGB values), which can be converted into a vector, and potentially tokenised as outlined above. Large arrays such as images may also be split up into multiple 'patches', which may be operated on separately to ease processing and / or memory constraints (e.g. an image of size 1000 pixels by 1000 pixels could be split into patches of size 100 pixels by 100 pixels).

[0070] Such patches can then be merged back together after processing to combine the results, if necessary.

[0071] Brief Description of the

[0072] Certain embodiments of the present invention will now be described with reference to the accompanying drawings, in which:

[0073] Fig. 1 is a block diagram illustrating the architecture for driving with LLMs in accordance with an embodiment of the present invention;

[0074] Fig. 2 is a further block diagram illustrating the architecture for driving with LLMs;

[0075] Fig. 3 is a block diagram of the architecture of the driving LLM agent model;

[0076] Fig. 4 is a block diagram that provides an illustration of the Driving QA Dataset automatic labelling process; and

[0077] Fig. 5 is a schematic diagram illustrating a demonstration of the interaction with the pretrained driving LLM agent.

[0078] Detailed

[0079] Certain exemplary embodiments are described herein which relate to a controller for an autonomous vehicle, an autonomous vehicle, and methods of controlling autonomous vehicles. It will be appreciated that the autonomous vehicle and its various components - as well as systems and components it interacts with - are complex technical systems, and so the illustrations and descriptions provided herein are simplified for ease of reference. ms

[0080] A variety of explainability methods have been introduced to understand the underlying decision process of deep neural networks. For example, there are well-established modelagnostic interpretability methods that generate explanations for individual predictions. Other methods such as gradient based, saliency maps, and attention maps target the inner operations of models to explain the decision-making process.

[0081] In the field of autonomous vehicles, visual attention maps which highlight causally influential regions in driving images have been proposed which combine attention-based methods with natural language into an attention-based vehicle controller to provide natural language action descriptions and explanations based on a series of image frames. Such an architecture may be improved by integrating part of speech prediction and special token penalties.

[0082] Efforts have also been made to combine this with other explanatory methods. For example, attempts have been made to explain transformers by leveraging attentive class activation tokens, encoded features, their gradients, and their attention weights all together.

[0083] Embodiments of the present invention use text modality for explainability in autonomous driving.

[0084] Multi-modal LLMs in Driving Tasks

[0085] In recent times, there has been a notable trend towards integrating multiple modalities into unified large-scale models. Notable examples include VLMs which effectively combine language and images to accomplish tasks like image captioning, visual question answering, and image-text similarity. Another noteworthy advancement involves the fusion of information from six distinct modalities: text, image / video, audio, depth, thermal, and inertial measurements. This exciting development expands the possibilities for generating content using diverse data input and output types, while also enabling broader multimodal search capabilities. With camera sensors being one of the most common sensors used in autonomous driving, a natural step to incorporate language has been through VLMs. For example, some approaches use images and language directions to train a driving policy. The system learns to summarize its visual observations in natural language, predicts an appropriate action response (e.g. "I see a pedestrian crossing, so I stop"), and predicts the controls (i.e. driving commands), accordingly.

[0086] The Applicant has appreciated a novel approach in the domain of autonomous vehicles in which numeric vector modality is fused with language.

[0087] Architecture Overview

[0088] Fig. 1 is a block diagram illustrating the architecture for driving with LLMs in accordance with an embodiment of the present invention. Fig. 1 shows various components of an autonomous vehicle 100. These include: a sensor unit 102; an operational parameter unit 104; a destination information unit 106; a vector generator unit 108; a pre-processing unit 110; a language model unit 112; a controller 114; and a user interface unit 116. As outlined above, this is necessarily a simplification of a practical autonomous vehicle, which may have many other components not shown or described here for brevity.

[0089] The vector generator unit 108, pre-processing unit 110, language model unit 112, and controller 114 are referred to collectively as a control system 118. Some or all the components of the control system 118 may be embodied within a single device or processor. Alternatively, some or all the components of the control system 118 may be separate, discrete devices or processors that are physically distinct. However, for ease of reference, these are grouped logically as a 'control system' 118.

[0090] The sensor unit 102 is configured to generate an environmental signal 120 corresponding to a surrounding environment of the autonomous vehicle 100. There are many different types of sensor, known in the art perse, that may be used within the sensor unit 102, as appropriate. It will be appreciated that where multiple sensors are used, some of all of these may be physically distinct from one another, however for ease of reference they are referred to collectively as a 'unit' herein, even where multiple separate sensor devices are used. The sensor unit 102 may include one or more of: a camera; a microphone; a sonar sensor; an ultrasonic sensor; a radar sensor; an ultra-wideband radio sensor; a lidar sensor; an optical flow sensor; a magnetometer; a thermometer; a barometer; a moisture sensor; a humidity sensor; an inertial measurement unit; a gyroscope; an accelerometer; a touch sensor; a laser rangefinder; an infrared sensor; a vehicle speed sensor; a yaw rate sensor; a wheel speed sensor; a current sensor; a fuel level sensor; a voltage sensor; an altitude sensor; a tilt sensor; a Hall effect sensor; an odometer; and a geopositioning sensor.

[0091] The sensor unit 102 is used to determine the state of the world around the autonomous vehicle 100. This may include detecting entities and objects such as other vehicles, pedestrians, cyclists, traffic signals, road obstructions, and so forth, as well as the state of such detected entities and objects.

[0092] The environmental signals generated by the sensor unit 102 may include information relation to, by way of non- limiting example:

[0093] • a number of detected vehicles

[0094] • respective distances between detected vehicles and the autonomous vehicle

[0095] • respective locations of detected vehicles

[0096] • respective orientations of detected vehicles

[0097] • respective speeds of detected vehicles

[0098] • respective bounding boxes of detected vehicles

[0099] • a number of detected pedestrians

[0100] • respective distances between detected pedestrians and the autonomous vehicle

[0101] • respective locations of detected pedestrians

[0102] • respective orientations of detected pedestrians

[0103] • respective speeds of detected pedestrians

[0104] • respective bounding boxes of detected pedestrians

[0105] • a number of detected cyclists

[0106] • respective distances between detected cyclists and the autonomous vehicle

[0107] • respective locations of detected cyclists

[0108] • respective orientations of detected cyclists

[0109] • respective speeds of detected cyclists

[0110] • respective bounding boxes of detected cyclists

[0111] • a detected traffic signal

[0112] • respective distances between a detected traffic signal and the autonomous vehicle

[0113] • a respective location of a detected traffic signal a respective state of a detected traffic signal

[0114] Such traffic signals may, for example, include traffic lights where the illumination state of different coloured lights indicates required or permitted behaviour (e.g. red meaning 'stop', green meaning 'go', and so on). These may also include 'static' traffic signals such as stop signs, give way markings, speed limit signs, road signage (e.g. to indicate road layout, lane allocations, etc.), warning signs, level crossing signs, smart motorway signs, and so forth.

[0115] The operational parameter unit 104 provides information or data relating to the operational state of the autonomous vehicle (i.e. the 'ego' vehicle). The operational parameters provided by the operational parameter unit 104 may include information relation to, by way of non- limiting example:

[0116] • a location of the autonomous vehicle

[0117] • a speed of the autonomous vehicle

[0118] • an orientation of the autonomous vehicle

[0119] • a trajectory of the autonomous vehicle

[0120] • a bounding box of the autonomous vehicle

[0121] • one or more actuator positions of the autonomous vehicle

[0122] • an accelerator state of the vehicle

[0123] • a brake state of the vehicle

[0124] The destination information unit 106 provides information relating to the destination or route of the vehicle. This may be the coordinates of the destination, a list of upcoming directions (e.g. next right turn, second exit at roundabout, straight on for 5 miles, etc.), a next direction, or a route of the autonomous vehicle.

[0125] While in this embodiment, the control system 118 of the autonomous vehicle 100 makes use of the operational parameters and destination information, these may be omitted. The Applicant has appreciated that the autonomous vehicle may drive based on only the environmental signal obtained from the sensor unit 102, though the inclusion of the operational parameters and destination information is beneficial for enhanced functionality. The vector generator unit 108 is configured to generate a numeric vector 122 from: the environmental signal 120 received from the sensor unit 102, the operational parameters received from the operational parameter unit 104, and the destination information received from the destination information unit 106. Each numeric vector is an array of numerical values, which correspond to the information of the environmental signal, operational parameters, and destination information, respectively. There may be more than one vector created from any one of the environmental signal, operational parameters, and / or destination information. For example, separate vectors may be generated from the environmental signal corresponding to detected vehicles (as one vector) and detected pedestrians (as another vector).

[0126] The pre-processing unit 110 receives as an input the numeric vector 122 generated by the vector generator unit 108. The pre-processing unit 110 generates a vector embedding 124 from the numeric vector 122. The vector embeddings 124 comprises a structured semantic representation of the surrounding environment of the autonomous vehicle 100.

[0127] The language model unit configured to use a machine learning language model to generate textual outputs 126, which include one or more driving commands, based on the vector embedding. These textual outputs 126 include a semantic description of what the control system 118 is observing, what actions it has planned, and the reasoning behind its decisions.

[0128] The controller 114 is configured to receive the one or more driving commands within the output 126 of the LLM 126 and operate the autonomous vehicle based on said one or more driving commands, by supplying suitable signals 128 to the drive system 130 of the autonomous vehicle. This drive system 130 may be a suitable drive-by-wire system, the details of which are not outlined herein for sake of brevity. Those skilled in the art will appreciate that a drive-by-wire system is an arrangement in which driving functions such as acceleration, braking, and steering are controlled electronically.

[0129] Fig. 2 is a further block diagram illustrating the architecture for driving with LLMs. As can be seen in Fig. 2, the model is built on three key components: the vector encoder 202, vector former 204, and a frozen LLM 206 with a Low-Rank Adaptation (LoRA) module 208. The internal architecture of these components can be seen in more detail Fig. 3. This architecture is described below with reference to Figs. 1 through 3. For ease of illustration, some of the higher-level components described above with reference to Fig. 1 are omitted from Figs. 2 and 3.

[0130] The first component is the vector encoder 202. The four input vectors 122 initially pass through the Multilayer Perceptron (MLP) layers 302 (there are MLPs for each vector in this embodiment). A cross-attention layer 304 is employed to encode the vector feature into a latent space, producing a vector latent 306. At this stage, the ego feature is added to each learned input latent to accentuate the ego states.

[0131] The vector encoder 202 receives a number of latents 313 - in this particular embodiment there are thirty-two latents input to the cross-attention layer 304. Each of these latents 313 and the outputs of the MLPs 302 are arguments to the cross-attention layer 304. In other words, they are multiplied via a cross-attention operation, in a manner known in the art per se.

[0132] The second component is the vector former 204. This module comprises self-attention layers 308 and a cross-attention layer 310. There may, by way of example only, be multiple self-attention layers 308, for example seven such layers. Their efficient learning mechanisms enable the transformation of the vector latent 306 into an embedding 312 that the LLM 206 can decode.

[0133] The vector former 204 also receives a number of queries 314 - in this particular embodiment there are sixty-four queries input to the cross-attention layer 310 of the vector former 204. Thus the outputs of the self-attention layer 308 and the queries 314 are arguments to the cross-attention layer 310, and thus are multiplied via a cross-attention operation.

[0134] The third component is the frozen LLM 206, which receives the vector embedding 312 from the vector former 204, as well as a prompt embedding 316. The prompt embedding 316 is an embedding of the prompt given to the system, which might be a request for driving commands or a request for an explanation of what the system will do and why, for a given world state observed by the system from the environmental signals. The prompt and its embedding may be generated automatically and / or the prompt and its embedding may be derived from a user input given via the user interface unit 116 shown on Fig. 1.

[0135] The LLM 206 of this embodiment has an architecture where each block starts with a layer normalization followed by a multi-head self-attention mechanism. After this, there's a residual connection and another layer normalization. This is then followed by a feedforward layer and another residual connection.

[0136] The pretrained LLM typically includes query (Q.), key (K), and value (V) linear layers within the transformer model. As shown in Fig. 3, trainable rank decomposition matrices 318 - the trainable query (Q.), key (K), and value (V) matrices - are injected into each layer of the transformer of the LLM 206 (alongside the original Q, K, and V linear layers) for efficient finetuning of the pre-trained LLM. This equips the LLM 206 to generate driving-related answers based on the prompt and vector inputs. For example, embodiments of the present invention may utilize LLaMA-7b as the language model, though other models may readily be used, as appropriate. The Q, K, and V matrices 318 are generated by the LoRA module 208.

[0137] As can be seen in Fig. 2, the LLM 206 generates textual outputs 126, which include one or more driving commands, based on the vector embedding 312 and the prompt embedding 316. These textual outputs 126 include a semantic description of what the control system 118 is observing, what actions it has planned, and the reasoning behind its decisions.

[0138] By way of example only, Fig. 2 shows a particular scene 200 in which the autonomous vehicle 100 is driving behind another vehicle 201, which it is to overtake. The LLM 126 may produce a textual based output 216 that says:

[0139] "Here are my actions:

[0140] Accelerator pedal 0%

[0141] Brake pedal 81%

[0142] Steering 31% to the right Reason:

[0143] I need to slow down and overtake the vehicle in front of me"

[0144] The controller 114 receives this input 126 and extracts suitable signals 128 which are supplied in a drive-by-wire manner to the drive system 130. This will result in the accelerator pedal being set to 0%, the brake pedal being set to 81%, and the steering being set 31% to the right. The reasoning behind these is captured in the text and can be supplied as a separate output, either in response to a user query during driving operations, or as part of a debugging process, for example. This may be presented to a user via the user interface unit 116 or via some other means.

[0145] Structured Language Generation for Pseudo Vector Captioning

[0146] It can be seen, therefore, that a vector representation of the environment is mapped to an action for the vehicle dynamics with an attention-based neural network architecture. Vector representations are converted into language using a structured language generator. The object-level vectors contain semantically significant attributes such as the number of cars and pedestrians, their respective locations, orientations, speeds, bounding boxes and other attributes.

[0147] Fig. 4 is a block diagram that provides an illustration of the Driving QA Dataset automatic labelling process. The pre-processing provided by the vector encoders 202 and vector former 204 may be employed to craft language labels - automatically generated captions - which are derived from the vector space representation of a given scenario 400, as illustrated below.

[0148] In other words, the vector encoders 202, vector former 204, and LLM 206 collectively provide a language generator (lanGen) function 402. For example, this lanGen function 402 may provide an exemplary output 404 such as:

[0149] I'm observing 1 car and 1 pedestrian A moving car; Angle in degrees: 1.19; Distance: 9.98m; [My attention: 78%] A pedestrian; Angle in degrees: -41.90; Distance: 11.94m; [My attention: 22%]

[0150] My current speed is 11.96 mph.

[0151] There is a traffic light and it is red. It is 12.63m ahead.

[0152] The next turn is 58 degrees right in 14.51m.

[0153] [Here are my actions:]

[0154] [ - Accelerator pedal 0%]

[0155] [ - Brake pedal 80%]

[0156] [ - Steering straight]"

[0157] In this function 402, the variables vcar, vPed, veg0, and vrOute denote the vector information corresponding to cars, pedestrians, ego vehicle, and route, respectively - as outlined previously.

[0158] The term oricorresponds to the output from the RL agent, consisting of additional attention and action labels for guiding the action reasoning process. Attention labels are collected from RL policy attention layers.

[0159] This structured language generator function 402 enables the transformation of vector representations into understandable language captions 404. It crafts a comprehensive narrative of the current driving scenario, inclusive of the agent's observations, the agent's current state, and its planned actions.

[0160] This comprehensive contextual foundation enables the LLM 206 to conduct reasoning and construct appropriate responses 406 in a manner that humans can interpret and understand.

[0161] The inclusion of oriis optional, and so two different versions of pseudo labels may be generated to cater to different requirements:

[0162] Without Attention / Action: Employed during the representation pre-training stage where the inference of attentions and actions is not required. With Attention / Action: Utilized for VQA labeling with GPT during the fine-tuning stage. This equips GPT with the capacity to ask specific questions about attentions and actions, thereby empowering the driving LLM agent with the ability to reason about attentions and actions.

[0163] The QA dataset labelling and training processes are outlined in further detail below.

[0164] Driving QA Dataset Labelling

[0165] The Applicant has appreciated that having a large amount of data is typically the key to enabling the question answering ability of language models. This becomes particularly crucial when a new modality is introduced to the LLMs; in such circumstances, it may be highly beneficial to have a high-quality question-answering dataset relevant to the modality input.

[0166] Data labelled using ChatGPT has previously been shown to surpass crowd-sourced workers in terms of performance for text-annotation tasks. Embodiments of the present invention may utilize GPT-3.5 to generate a Driving QA dataset. This dataset is conditioned on the structured language input, serving as a representative for the vectors. To be specific, GPT- 3.5 may, in some embodiments, be tasked to emulate the role of a certified professional driving instructor. It will of course be appreciated that different language models, e.g. other LLMs, may be used.

[0167] Basic driving rules are supplied as pre-prompts, covering elements like traffic signal transitions and the driving side (e.g. if operating in a left-handed driving country). GPT-3.5 is tasked with creating sixteen sets of random questions and answers per scenario, considering the current driving scenario provided by language description generated by the lanGen 402. These question-answer pairs were generated based on two criteria: the answers should be derived from the given input observations and the model's reasoning, and the responses may be formatted in JSON line format for easier parsing. Those skilled in the art will appreciate that 'JSON' refers to JavaScript Object Notation (JSON) where a series of fields are defined and values for those fields specified using a machine-readable syntax. For example, the JSON data {"firstName": "John", "secondName": "Smith"} allows values for the firstName and secondName defined fields to be extracted.

[0168] To incentivize the self-generation of question-and-answer pairs that contain information from the input vector, some preset questions are provided. Simple guardrail rules are established too - GPT-3.5 is instructed to include a driving-related question that could not be inferred from the observation input. This way, the model's response would be something akin to "I'm unable to answer this question based on the observations I have," reducing the chance of hallucination.

[0169] Additionally, a random question unrelated to driving is included, instructing the model to respond with "As an Al Driver, the question you asked is out of my scope, but I can attempt to answer it." The model would then proceed to answer the question normally, thereby preserving the versatility of the language model as a general-purpose assistant.

[0170] Below are some examples of the question-answer pairs 406 generated by GPT-3.5:

[0171] Question: "What objects are you observing?"

[0172] Answer: "I'm observing a moving car, and two pedestrians. The car is coming from the opposite direction, and the pedestrians are crossing the street from both the right and left sides."

[0173] Question: "How are you going to drive in this situation and why?" Answer: "Here are my actions: - Accelerator pedal 0% - Brake pedal 7% - Steering straight. Reason: Because I'm maintaining a slow speed due to the pedestrians crossing, and I'm steering straight to keep our vehicle properly aligned with the road."

[0174] Question: "What is the protocol if the pedestrian crossing the road suddenly stops?"

[0175] Answer: "If a pedestrian suddenly stops, we should further decrease our speed, and if necessary, come to a complete stop to ensure their safety and avoid a collision. It's crucial to always be prepared for unpredictable pedestrian behaviour."

[0176] Question: "Can you check the tyre pressure?"

[0177] Answer: "I'm unable to answer this question based on the observation I have. My current input observations include objects, pedestrians, traffic lights, and driving actions."

[0178] Question: "What are the best tourist spots in London?"

[0179] Answer: "As an Al Driver, the question you asked is out of my scope, but I can try to answer it. Some famous tourist spots in London include the Tower of London, Buckingham Palace, The British Museum, The Shard, and the London Eye."

[0180] Training the Driving LLM Agent

[0181] Training the driving LLM agent involves formulating it as a Driving Question Answering (DQA) problem within the context of a language model. The key to this formulation is the integration of object-level vector modality with the pre-trained LLMs, creating a multimodal system capable of interpreting and interacting with both language and vector inputs.

[0182] A two-stage process is used to train the model for effectively fusing the object-level vector modality into the driving LLM agent.

[0183] In the first stage, the vector representation is grounded into an embedding that can be decoded by the LLM. This is done by freezing the language model and optimizing the weights of the vector encoders and the vector transformer.

[0184] In the second stage, the model is fine-tuned to the driving task, training it to answer driving-related questions and take appropriate actions based on its current understanding of the environment. Fig. 5 illustrates a demonstration of the interaction with the pretrained driving LLM agent. Notably, the driving LLM agent possesses the ability to anticipate future actions when prompted. Fig. 5 shows a scenario 500 in which the autonomous vehicle 502 (the ego vehicle) is approaching a crossroads, i.e. a four-way intersection. This intersection is governed via a series of traffic lights 504a-d.

[0185] There are several pedestrians 506 walking alongside the road, and several other vehicles 508, some of which are actively turning across the intersection in front of the autonomous vehicle 502. Note that not all of the pedestrians 506 and other vehicles 508 are labelled, for ease of illustration.

[0186] In this particular scenario 500, the traffic light 504a ahead of the autonomous vehicle 502 is red, i.e. the vehicle 502 should stop and wait rather than driving into the intersection.

[0187] As shown in the LLM output 510 of Fig. 5, a user prompt of "What's your action and why?" is met with the response:

[0188] "Here's my actions:

[0189] Accelerator pedal 0%

[0190] Brake pedal 35%

[0191] Steering straight.

[0192] Reason: I am applying the brake pedal to slow down the car as the traffic light ahead is red."

[0193] A follow up user prompt of "What will you do if the traffic light turns green?" is met with the response:

[0194] "I would release the brake pedal and accelerate the accelerator pedal to continue driving."

[0195] The LLM is able to produce not only an output suitable for conversion to driving commands and its reasoning, but it is also capable of providing an output outlining what it will do under a future change in condition (i.e. the light subsequently turning green). Training using a Simulator

[0196] To generate language-based grounded driving datasets, a realistic 2D simulator can be used with procedural generation of driving scenarios. In such arrangements, an RL agent is used to solved simulated scenarios using an object-level ground-truth representation of the driving scene.

[0197] Numeric Upweighting Loss for Vector Representation Pre-training

[0198] Integrating a new modality into pre-trained Large Language Models (LLMs) poses significant challenges due to the need for extensive data and computational resources. Embodiments of the present invention use a novel approach that uses structured language to bridge the vector space with the language embedding, with particular focus on numerical tokens.

[0199] In the pretraining phase, a strategy is employed in which the language model is frozen while the entire framework is trained end-to-end. This approach is designed to optimize the weights of the vector encoders and the vector former. Such an optimization process enables effective grounding of the vector representation into an embedding that can be directly decoded by the LLMs.

[0200] It is important to note that during this pretraining phase, attention and action elements are deliberately excluded from the structured language. This is because the focus at this stage is solely on representation training, and it is desirable to avoid prematurely integrating any reasoning components into the V-former.

[0201] To enhance the accuracy of number decoding, it may be particularly advantageous to use weighted cross-entropy on number and symbol tokens ('0-9', and In other words, number and symbol tokens are made more important than alphabetical tokens fA-Z', and 'a-z') or other tokens. This approach adds further penalization on the decoded numbers, hence increasing the performance of the learned representation embeddings. The pretraining was conducted on 100k simulator-collected data points with additional 200k uniform randomly generated data points over the course of 5 epochs. This totals to an equivalent of 1.5 million vector sample with vector captioning labels, providing a robust training set for the novel methodology. Through this approach, embodiments of the invention may effectively incorporate object-level vector modality into pre-trained LLMs, which provides a very good starting point for the fine-tuning stage.

[0202] Driving QA Fine-tuning

[0203] After the pre-training stage, the trainable LoRA module 208 is ingested to the LLM 206, and optimization of the weights of the vector encoder 202, vector former 204, and LoRA module 208 is performed in an end-to-end fashion on the Driving QA data collected earlier.

[0204] To train the driving LLM agent to output accurate action for driving, certain actiontriggering questions are added to the VQA dataset. These are questions that, when asked, require the agent to generate actions in specific format. For example, a question like "How are you going to drive in this case and why?" would require the agent to infer the action based on the vector input. A simple language-action grounding strategy with regular expression matching is used to extract the action required to control the car in a simulator.

[0205] To ensure the model pays sufficient attention to these important questions, the actiontriggering questions are upsampled with different expressions such as:

[0206] • "How are you going to drive in this situation?"

[0207] • "What actions are you taking?"

[0208] • "How are you driving in this situation?"

[0209] • "What are your planned actions for this situation?"

[0210] The pre-existing model is refined on 8,000 scenarios, each incorporating 16 driving QA entities, along with an additional vector captioning data and an action prediction QA pair. This results in an aggregate of 144,000 QA pairs. Training is conducted across 5 epochs until the model achieves convergence. The fine-tuning process optimizes the weights of the vector encoders and the Vector Former in an end-to-end manner. The outcome is a driving LLM agent that is capable of reasoning about its actions based on the current driving scenario. This demonstrates the power and potential of integrating object-level vector modality with pre-trained LLMs, particularly in the context of autonomous driving.

[0211] Evaluation and Results

[0212] This section aims to thoroughly assess the model's performance using a reserved evaluation set comprising 1,000 distinct driving scenarios. The vector captioning and vector QA data for these scenarios were labelled following the methodologies discussed herein.

[0213] Evaluation of Vector Captioning and Action Prediction

[0214] In this section, the results of the driving QA models on the vector captioning and action tasks are presented. The results on the model trained using the two-stage approach (with pretraining), and the model training only on driving QA dataset (without pretraining), are reported herein.

[0215] The Mean Absolute Error (MAE) is calculated for the predictions of the number of cars and pedestrians, denoted as respectively. Additionally, the accuracy of traffic light detection is measured, as well as the mean absolute distance error in meters for traffic light distance prediction, represented as ACCT L and DT L-

[0216] The MAE for ego states, including speed in mph and steering angle in percentage, are presented as Furthermore, the MAE is computed for the normalized acceleration and brake pressure, denoted as

[0217] Finally, the weighted cross entropy loss is reported for the token prediction on the evaluation set, indicated

[0218] Table 1: The evaluation result of vector captioning and action prediction

[0219] The results clearly demonstrate that pretraining with the novel numeric upsampling significantly enhances the agent's detection capabilities. There is a significant decrease in the MAE for both the car and pedestrian prediction tasks. This suggests that the pretrained model exhibits a higher level of accuracy in perceiving and quantifying the number of cars and pedestrians in its environment.

[0220] The pretrained model also shows a lower loss value, LtOken, which indicates an improvement in the overall effectiveness of the model's token predictions. Interestingly, the pretraining stage did not yield significant improvements in ego states decoding and action prediction. The pretrained model's performance in terms of traffic light detection accuracy, ACCT L, is similar to the model without pretraining. The distance error for traffic light detection, DT L, also remains relatively unchanged. This outcome could be attributed to the relatively small size of the driving QA dataset. The action prediction task is primarily learned during the finetuning stage, and with the current limited volume of data, the model might not be able to fully capture the complexity of driving actions. However, a key advantage of the disclosed method is its high scalability. As the driving QA dataset is expanded with more samples, it is anticipated that the performance of action prediction will improve correspondingly during the fine-tuning stage. Evaluation of Driving QA

[0221] To assess the quality of answers to open-ended questions about the driving environment, GPT-3.5 is used to grade the model's answers. This allows for quick and cheap evaluation of the model's capabilities on questions without a fixed answer such as summarising the surrounding, explaining the reason for taking a certain action, or describing the most important objects in the scene.

[0222] For evaluation, GPT-3.5 is prompted with the language-based observation description used during dataset generation, the question from the test set, and the answer of the model. GPT-3.5's task is to write a one-line assessment for each answer, followed by a score from 0 to 10 where 0 is worst and 10 is best. The final score of the model is the average over all question scores.

[0223] Grading the outputs of the LLM agent using GPT-3.5

[0224] Table 2: Grading the outputs of the LLM agent using GPT-3.5. Scores are in the range 0 to

[0225] 10, where 0 is worst and 10 is best.

[0226] The results show that pre-training improves the grading score of the model by 8.2% over no pre- training. For reference, also provided is the score when running the evaluation procedure with the answer labels provided by GPT-3.5 and a constant answer of "I don't know". The model, using only vectorized representations as input, can get a score of 7.88, much higher than the score of 2.4 when just answering with "I don't know". Conclusion

[0227] The Applicant believes that the proposed architecture, combined with the driving scenario QA task and the numeric upweighting pretraining methodology, alone and in combination, provide a technical foundation for the integration of numeric vector modality with LLMs in the context of autonomous driving.

[0228] The proposed approach holds potential for application for real-world driving scenarios. The present invention provides a significant step forward in the integration of numeric vector modality with LLMs in the context of autonomous driving.

[0229] While specific embodiments of the present invention have been described in detail, it will be appreciated by those skilled in the art that the embodiments described in detail are not limiting on the scope of the claimed invention.

Claims

Claims1. An autonomous vehicle comprising: a sensor unit configured to generate an environmental signal corresponding to a surrounding environment of the autonomous vehicle; a vector generator unit configured to generate a numeric vector from the environmental signal received from the sensor unit; a pre-processing unit which receives as an input the numeric vector generated by the vector generator unit, wherein the pre-processing unit generates a vector embedding from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; a language model unit configured to use a machine learning language model to generate a textual data output based on the vector embedding; and a controller configured to receive the textual data output and generate therefrom one or more driving commands, wherein the controller is further configured to operate the autonomous vehicle based on said one or more driving commands.

2. A control system for an autonomous vehicle, the control system comprising: a vector generator unit configured to generate a numeric vector from a received environmental signal; a pre-processing unit which receives as an input the numeric vector generated by the vector generator unit, wherein the pre-processing unit generates a vector embedding from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; a language model unit configured to use a machine learning language model to generate a textual data output based on the vector embedding; and a controller configured to receive the textual data output and generate therefrom one or more driving commands, wherein the controller is further configured to operate the autonomous vehicle based on said one or more driving commands.

3. The autonomous vehicle or control system as claimed in claim 1 or 2, wherein the machine learning language model is pre-trained to transform textual inputs to driving commands.

4. The autonomous vehicle or control system as claimed in any preceding claim, further comprising: a user interface unit configured to receive a user input prompt, and to input the user prompt into the language model unit, wherein the language model unit generates a response to the user prompt based on said user prompt and the vector embeddings; wherein the response is provided as an output; optionally wherein the response is provided as an output via the user interface unit.

5. The autonomous vehicle or control system as claimed in any preceding claim, wherein the sensor unit comprises one or more sensors from a group comprising: a camera; a microphone; a sonar sensor; an ultrasonic sensor; a radar sensor; an ultra- wideband radio sensor; a lidar sensor; an optical flow sensor; a magnetometer; a thermometer; a barometer; a moisture sensor; a humidity sensor; an inertial measurement unit; a gyroscope; an accelerometer; a touch sensor; a laser rangefinder; an infrared sensor; a vehicle speed sensor; a yaw rate sensor; a wheel speed sensor; a current sensor; a fuel level sensor; a voltage sensor; an altitude sensor; a tilt sensor; a Hall effect sensor; an odometer; and a geopositioning sensor.

6. The autonomous vehicle or control system as claimed in any preceding claim, wherein the numeric vector representation of the surrounding environment of the autonomous vehicle comprises one or more numeric values corresponding to: a number of detected vehicles; respective distances between detected vehicles and the autonomous vehicle; respective locations of detected vehicles; respective orientations of detected vehicles; respective speeds of detected vehicles; and / or respective bounding boxes of detected vehicles.

7. The autonomous vehicle or control system as claimed in any preceding claim, wherein the numeric vector representation of the surrounding environment of the autonomous vehicle comprises one or more numeric values corresponding to: a number of detected pedestrians; respective distances between detected pedestrians and the autonomous vehicle; respective locations of detected pedestrians; respective orientationsof detected pedestrians; respective speeds of detected pedestrians; and / or respective bounding boxes of detected pedestrians.

8. The autonomous vehicle or control system as claimed in any preceding claim, wherein the numeric vector representation of the surrounding environment of the autonomous vehicle comprises one or more numeric values corresponding to: a detected traffic signal; respective distances between a detected traffic signal and the autonomous vehicle; a respective location of a detected traffic signal; and / or a respective state of a detected traffic signal.

9. The autonomous vehicle or control system as claimed in any preceding claim, wherein the vector generation unit is further configured to generate a numeric vector from one or more operational parameters corresponding to operation of the autonomous vehicle; wherein the numeric vector generated from the one or more operational parameters are input to the pre-processing unit such that the vector embeddings further comprises a structured semantic representation of the one or more operational parameters.

10. The autonomous vehicle or control system as claimed in claim 9, wherein the numeric vector representation of the one or more operational parameters comprises one or more numeric values corresponding to: a location of the autonomous vehicle; a speed of the autonomous vehicle; an orientation of the autonomous vehicle; a trajectory of the autonomous vehicle; a bounding box of the autonomous vehicle; one or more actuator positions of the autonomous vehicle; an accelerator state of the vehicle; and / or a brake state of the vehicle.

11. The autonomous vehicle or control system as claimed in any preceding claim, wherein the vector generation unit is further configured to generate a numeric vector from destination information corresponding to the autonomous vehicle; wherein the numeric vector generated from the destination information are input to the pre-processing function executed by the pre-processing unit such that the vector embeddings further comprises a structured semantic representation of the destination information;optionally wherein the destination information comprises a route of the autonomous vehicle.

12. The autonomous vehicle or control system as claimed in any preceding claim, wherein one or more numeric vectors generated by the vector generator unit are tokenised such that the respective numeric vector comprises one or more tokens, each token being selected from a predetermined set of tokens; optionally wherein the tokenisation is performed by the vector generator unit.

13. The autonomous vehicle or control system as claimed in any preceding claim, wherein the pre-processing unit comprises: a vector encoder configured to receive the numeric vector generated by the vector generator unit, the vector encoder being further configured to encode said numeric vector into a latent space, thereby generating latent vectors.

14. The autonomous vehicle or control system as claimed in claim 13, wherein the vector encoder comprises: a multilayer perceptron layer configured to receive the numeric vector generated by the vector generator unit; and a cross-attention layer configured to receive an output of the multilayer perceptron layer and a plurality of weight values, said cross-attention layer being further configured to use a cross-attention mechanism to generate the latent vectors.

15. The autonomous vehicle or control system as claimed in claim 13 or 14, wherein the pre-processing unit further comprises: a vector former configured to receive the latent vectors from the vector encoder and to generate the vector embeddings from the latent vectors.

16. The autonomous vehicle or control system as claimed in claim 15, wherein the vector former comprises: a self-attention layer configured to receive the latent vectors; and a second cross-attention layer configured to receive an output of the self-attention layer and a second plurality of weight values, said second cross-attention layer beingfurther configured to use a second cross-attention mechanism to generate the vector embeddings.

17. The autonomous vehicle or control system as claimed in any preceding claim, wherein the language model unit comprises: a language model transformer configured to receive the vector embeddings and to use a respective attention mechanism to generate the one or more driving commands;18. The autonomous vehicle or control system as claimed in claim 16, wherein the attention mechanism used by the transformer is dependent on a query matrix (Q.), a key matrix (K), and a value matrix (V).

19. The autonomous vehicle or control system as claimed in claim 18, wherein the language model unit further comprises a Low-Rank Adaptation (LoRA) module configured to supply the query matrix (Q.), the key matrix (K), and the value matrix (V); optionally wherein one or more of said matrices are trainable.

20. A computer-implemented method of operating an autonomous vehicle, the method comprising: receiving or generating an environmental signal corresponding to a surrounding environment of the autonomous vehicle; generating numeric vectors from the environmental signal; generating vector embeddings from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; using a machine learning language model to generate a textual data output based on the vector embedding; generating one or more driving commands from the textual data output; and operating the autonomous vehicle based on said one or more driving commands.

21. A non-transitory computer-readable medium comprising instructions which, when executed by a processor, cause the processor to carry out a method of operating an autonomous vehicle, the method comprising:receiving or generating an environmental signal corresponding to a surrounding environment of the autonomous vehicle; generating numeric vectors from the environmental signal; generating vector embeddings from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; using a machine learning language model to generate a textual data output based on the vector embedding; generating one or more driving commands from the textual data output; and operating the autonomous vehicle based on said one or more driving commands.

22. A computer software product which, when executed by a processor, cause the processor to carry out a method of operating an autonomous vehicle, the method comprising: receiving or generating an environmental signal corresponding to a surrounding environment of the autonomous vehicle; generating numeric vectors from the environmental signal; generating vector embeddings from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; using a machine learning language model to generate a textual data output based on the vector embedding; generating one or more driving commands from the textual data output; and operating the autonomous vehicle based on said one or more driving commands.

23. A user interaction system for an autonomous vehicle, the user interaction system comprising: a vector generator unit configured to generate a numeric vector from the environmental signal received from the sensor unit; a pre-processing unit which receives as an input the numeric vector generated by the vector generator unit, wherein the pre-processing unit generates a vector embedding from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle;a language model unit configured to use a machine learning language model; and a user interface unit configured to receive a user input prompt, and to input the user prompt into the language model unit, wherein the language model unit generates a response to the user prompt based on said user prompt and the vector embeddings; wherein the response is provided as an output.

24. A user interaction system for an autonomous vehicle, the user interaction system comprising: a vector generator unit configured to generate a numeric vector from the environmental signal received from the sensor unit; a pre-processing unit which receives as an input the numeric vector generated by the vector generator unit, wherein the pre-processing unit generates a vector embedding from said numeric vector, wherein the vector embedding comprises a structured semantic representation of the surrounding environment of the autonomous vehicle; and a language model unit configured to use a machine learning language model, the language model being trained to generate an output based on the structure textual data, data.

25. A computer-implemented method of training a machine learning model for use in a control system for an autonomous vehicle, the control system comprising: a pre-processing unit which receives as an input a numeric vector comprising information relating to the autonomous vehicle and / or a surrounding environment of said autonomous vehicle, wherein the pre-processing unit generates a vector embedding from said numeric vector, wherein the vector embedding comprises a structured semantic representation of said information; and a language model unit configured to use a machine learning language model to generate a textual data output based on the vector embedding; wherein the generation of the vector embedding by the pre-processing unit is dependent on a first set of weights, and wherein the generation of the textual data output by the machine learning language model is dependent on a second set of weights; wherein the method comprises:determining a set of values for the first set of weights by minimising an error between a ground truth label set and a candidate label set generated by the control system; wherein each label set comprises a plurality of symbols, each symbol being a selected symbol from a predetermined set of symbols, said set of symbols comprising an alphabetical symbol sub-set and a numeric symbol subset; wherein the error between the ground truth label set and the candidate label set is determined such that differences between corresponding symbols in the ground truth label set and the candidate label set are allocated: a first error value when a given symbol in the candidate label set is different from the corresponding symbol in the ground truth label set when said corresponding symbol belongs to the alphabetic symbol sub-set; and a second error value when a given symbol in the candidate label set is different from the corresponding symbol in the ground truth label set when said corresponding symbol belongs to the numeric symbol sub-set, wherein the second error value is greater than the first error value.

Images