Method and apparatus for test scenario generation

Abstract

A computer implemented method comprises obtaining an input sequence 310, the input sequence comprising a natural language description of a test scenario for testing a component of an autonomous vehicle control system, processing 320 the input sequence using a natural language processing, NLP engine, such as a generative large language model, to generate information associated with at least one dynamic entity forming part of the test scenario. Based on the information, a computer readable data structure, such as an OpenScenario, comprising parameters defining the test scenario is generated 330, and simulated sensor data for a sensor(s) of the autonomous vehicle control system is generated 340 based on the computer readable data structure. The simulated sensor data is provided 350 as an input to a component of the autonomous vehicle control system. A method of training a generative natural language processing engine for generating a data structure defining a test scenario is also provided, including using training data and updating weights.

Description

TECHNICAL FIELD The present disclosure relates to a method and system for generating a test scenario for testing autonomous driving features of a vehicle. In particular, the present disclosure relates to a method and apparatus for generating a test scenario in response to a natural language description of a test scenario for testing a component of an autonomous vehicle control system. Aspects of the invention relate to a method, to a control system, and to a computer readable medium. BACKGROUND Autonomous driving systems and vehicles should be tested and validated to ensure consistent and reliable operation across a wide range of conditions and scenarios that may be encountered during operation of a vehicle. Testing can be performed on physical vehicles, for example pre-production or prototype vehicles, in supervised real-world driving. However, such physical vehicles may only be available relatively late on during the development process, and discovering an issue at a late stage in the development cycle may increase the cost and complexity of providing a fix. Software based testing, e.g. via simulation, may be performed at an earlier stage of the development cycle, for example using emulated or prototype sensors and processing modules. One approach to software testing of autonomous driving features uses OpenSCENARIO files that define parameters of a test environment, for example the presence and actions of other vehicles, hazards, etc. in a simulated scenario. However, to replicate the wide range of conditions that might be encountered during testing of a physical platform in the real-world, a large number of scenarios may need to be defined covering a wide range of test cases for each autonomous driving feature to be tested. Generating appropriate scenario definitions, such as OpenSCENARIO files, to cover a large number of different scenarios may require a significant amount of time and effort. It is an aim of the present invention to address one or more of the disadvantages associated with the prior art. SUMMARY OF THE INVENTION Aspects and embodiments of the invention provide a method, a control system, and a computer readable medium as claimed in the appended claims. According to an aspect of the present invention there is provided a computer implemented method comprising obtaining an input sequence comprising a natural language description of a test scenario for testing a component of an autonomous vehicle control system, processing the input sequence using a natural language processing, NLP, engine, and generating a data structure defining one or more aspects of the test scenario. Advantageously, a test scenario can be described in a user’s own words and used in a simulation to test an autonomous vehicle component. According to another aspect of the present invention there is provided a computer implemented method comprising obtaining an input sequence, the input sequence comprising a natural language description of a test scenario for testing a component of an autonomous vehicle control system, processing the input sequence using a natural language processing, NLP, engine to generate information associated with at least one dynamic entity forming part of the test scenario, based on the information associated with the at least one dynamic entity forming part of the test scenario, generating a computer readable data structure comprising a plurality of parameters defining the test scenario, generating simulated sensor data for at least one sensor of the autonomous vehicle control system based on the computer readable data structure, and providing the simulated sensor data as an input to the component of the autonomous vehicle control system. Advantageously, a test scenario can be described in a user’s own words and this natural language description of the test scenario can be interpreted by the NLP engine and converted to a file, or data structure, defining the test scenario, and a simulation can be performed to test parts of an autonomous vehicle control system based on the test scenario. This reduces the time and effort required to generate the test scenario definition over prior methods and allows accelerated testing, or testing of a greater number of scenarios, to be performed. Optionally, the information associated with the at least one dynamic entity comprises at least one of: a vehicle type associated with the dynamic entity; a position associated with the dynamic entity; a speed associated with the dynamic entity; and a trajectory associated with the dynamic entity. Advantageously, the NLP engine is able to determine multiple properties of entities described in the natural language description of the test scenario to allow complex test scenarios to be automatically generated and simulated testing based on the scenarios to be performed. Optionally, at least one of the position, speed and trajectory associated with the dynamic entity are defined relative to a vehicle in the test scenario that includes the autonomous vehicle control system comprising the component to be tested. Optionally, at least one of the position, speed and trajectory associated with the dynamic entity are defined as absolute values. Optionally, the NLP engine comprises a pre-trained NLP algorithm. Advantageously, the NLP engine is able to use available pre-trained NLP algorithms to reduce the cost and difficulty of creating the NLP engine. Optionally, the pre-trained NLP algorithm comprises one of: a Generative Pre-trained Transformer, GPT, model; Text-to-Text Transfer Transformer, T5, model; Pathways Language Model, PaLM; Large Language Model Meta Al, LLaMA; or another generative large language model. Advantageously, a wide range of available pre-trained NLP models can be used to implement the method. In embodiments, the NLP engine further comprises one or more output layers, each of the one or more output layers comprising a plurality of nodes of a neural network, and the method further comprises fine-tuning the NLP engine by training the one or more output layers using a set of training data, the training data comprising training example input / output pairs of natural language descriptions of a test scenario and an associated computer readable data structure comprising the plurality of parameters defining the test scenario. Advantageously, through further fine-tuning / training of the NLP engine including one or more further layers of nodes, the NLP engine may be trained to better understand the test scenario descriptions and convert these to a suitable data structure for performing the test simulation. Optionally, the computer readable data structure comprising the plurality of parameters defining the test scenario comprises an OpenSCENARIOfile. Advantageously, fine-tuning of the NLP engine may provide for direct generation of OpenSCENARIO files as output from the NLP engine based on the natural language description of the test scenario. In embodiments, processing the input sequence using the NLP engine comprises providing the input sequence as a prompt to the natural language processing engine, the prompt further comprising a template computer readable data structure, wherein the computer readable data structure comprising the plurality of parameters defining the test scenario is generated based on the template computer readable data structure. Advantageously, by use of a suitable prompt, the NLP engine can be influenced to generate the desired output format based on the natural language description of the test scenario without requiring further training, or fine-tuning, of the NLP engine (i.e. one-shot learning). Optionally, the template computer readable data structure comprises a Javascript Object Notation, JSON, template. Advantageously, JSON provides a common format that may be understood by the NLP engine and is easily converted into a suitable test scenario file for use in the simulation and testing. In embodiments, the method comprises parsing the computer readable data structure comprising the plurality of parameters defining the test scenario to generate an OpenScenario file defining the test scenario. Advantageously, the output of the NLP engine can be converted into an OpenScenario file which is a standard format used by available testing software for simulation and testing of autonomous driving features. Optionally, the component of the autonomous vehicle control system comprises a hardware module. Advantageously, the method can be used to simulate sensor inputs for hardware-in-the-loop testing, providing realistic inputs for hardware components, and any associated software provided on the hardware modules, allowing test scenarios to be quickly and easily developed and used to test portions of the autonomous control system prior to integration with other components. Optionally, the component of the autonomous vehicle control system comprises a software module. Advantageously, the data structure describing the test scenario can also be used in software-in-the-loop testing for validation of software components prior to integration with the corresponding hardware modules. In embodiments, generating simulated sensor data for at least one sensor of the autonomous vehicle control system further comprises obtaining a model of the at least one sensor and generating a simulated sensor output based on the model and the computer readable data structure. Advantageously, simulated sensor data can be generated based the output of the NLP engine and on models of physical sensors to be used with the autonomous control system to generate realistic sensor data for the simulation. In embodiments, the autonomous vehicle control system comprises at least one actuator controlled based on an output of the component of the autonomous vehicle control system, the computer implemented method further comprising monitoring an actuation of the actuator in response to providing the simulated sensor data as an input to the component of the autonomous vehicle control system. Advantageously, hardware-in-the-loop testing can include multiple hardware components including actuators to allow scenario-based testing of the control system. According to another aspect of the invention, there is provided a control system comprising a processor, and a memory comprising computer program instructions that when executed by the processor cause the apparatus to obtain an input sequence, the input sequence comprising a natural language description of a test scenario for testing a component of an autonomous vehicle control system, process the input sequence using an NLP engine to generate information associated with at least one dynamic entity forming part of the test scenario, based on the information associated with at least one dynamic entity forming part of the test scenario, generate a computer readable data structure comprising a plurality of parameters defining the test scenario, generate simulated sensor data for at least one sensor of the autonomous vehicle control system based on the computer readable data structure, and provide the simulated sensor data as in input to the component of the autonomous vehicle control system. Advantageously, a test scenario can be described in a user’s own words and this natural language description of the test scenario interpreted by the NLP engine and converted to a file, or data structure, defining the test scenario and a simulation performed to test parts of an autonomous vehicle control system based on the test scenario. This reduces the time and effort required to generate the test scenario definition over prior methods and allows accelerated testing to be performed. The control system may comprise one or more controllers collectively comprising at least one electronic processor having an electrical input for receiving an input signal; and at least one memory device electrically coupled to the at least one electronic processor and having instructions stored therein; and wherein the at least one electronic processor is configured to access the at least one memory device and execute the instructions thereon so as to: obtain an input sequence, the input sequence comprising a natural language description of a test scenario for testing a component of an autonomous vehicle control system; process the input sequence using an NLP engine to generate information associated with at least one dynamic entity forming part of the test scenario; based on the information associated with at least one dynamic entity forming part of the test scenario, generate a computer readable data structure comprising a plurality of parameters defining the test scenario; generate simulated sensor data for at least one sensor of the autonomous vehicle control system based on the computer readable data structure; and provide the simulated sensor data as in input to the component of the autonomous vehicle control system. Optionally, the component of the autonomous vehicle control system comprises a hardware module. Advantageously, the method can be used to simulate sensor inputs for hardware-in-the-loop testing, providing realistic inputs for hardware components, and any associated software provided on the hardware modules, allowing test scenarios to be quickly and easily developed and used to test portions of the autonomous control system prior to integration with other components. According to a further aspect of the invention, there is provided a method of training a generative natural language processing engine for generating a computer readable data structure comprising a plurality of parameters defining a test scenario based on an input sequence comprising a natural language description of a test scenario for testing a component of an autonomous vehicle control system, the natural language processing engine comprising a neural network including a pre-trained natural language processing, NLP, algorithm to receive a prompt comprising the input sequence, and one or more output layers, each layer of the one or more output layers comprising a plurality of nodes of a neural network, wherein the method comprises: receiving a set of training data to train the natural language processing engine, the training data comprising training example input / output pairs of natural language descriptions of a test scenario and an associated computer readable data structure comprising the plurality of parameters defining the test scenario, inputting the natural language description from one or more of the input / output pairs from the set of training data to obtain a predicted computer readable data structure, characterising an error between the predicted computer readable data structure and the computer readable data structure corresponding to the natural language description of the training set, and using an optimisation algorithm to update weights of the one or more output layers based on the characterised error. Advantageously, the described method allows fine-tuning training of a natural language processing engine using a pre-trained NLP engine to generate computer readable test scenario descriptions based on a natural language description of the test scenario. According to another aspect of the invention, there are provided computer readable instructions, which when executed by a computer, are arranged to perform a method of generating a computer readable data structure or a method of training a generative natural language processing engine as described above. Within the scope of this application it is expressly intended that the various aspects, embodiments, examples and alternatives set out in the preceding paragraphs, in the claims and / or in the following description and drawings, and in particular the individual features thereof, may be taken independently or in any combination. That is, all embodiments and / or features of any embodiment can be combined in any way and / or combination, unless such features are incompatible. The applicant reserves the right to change any originally filed claim or file any new claim accordingly, including the right to amend any originally filed claim to depend from and / or incorporate any feature of any other claim although not originally claimed in that manner. BRIEF DESCRIPTION OF THE DRAWINGS One or more embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which: Figure 1 shows a system for testing a component of an autonomous vehicle control system in a simulated test scenario according to embodiments of the invention; Figure 2 shows a method of generating a test scenario for testing a component of an autonomous vehicle control system according to embodiments of the invention; Figure 3 shows a method of providing simulated sensor data based on a natural language description of a test scenario according to embodiments of the invention; Figure 4 shows a computer system suitable for implementing a method of generating a test scenario according to embodiments of the invention; Figure 5 shows a natural language processing engine according to embodiments of the invention; Figure 6 shows a method of training a natural language processing engine according to embodiment of the invention; and Figure 7 shows operation of a method of training a natural language processing engine according to embodiments of the invention. DETAILED DESCRIPTION According to embodiments of the invention, a computer readable data structure comprising parameters defining a test scenario for testing a component of an autonomous vehicle control system, such as components providing Advanced Driver Assistance System (ADAS) features, is generated based on a natural language description of the test scenario. The computer readable data structure may then be used to generate simulated sensor data as part of a simulation of the test scenario, and the simulated sensor data may then be input to the component of the autonomous vehicle control system being tested. The natural language description of the test scenario is processed using a natural language processing algorithm to convert the natural language description into a computer readable data structure defining the parameters of the test scenario in an appropriate format to be read by simulation software used to generate inputs to the component under test. Embodiments may significantly reduce the time taken to generate test scenario files for use with simulation software for testing components of an autonomous vehicle control system, thereby facilitating generation and use of a greater number of test scenarios for more complete testing and validation of autonomous driving features. Figure 1 illustrates a system 100 for testing a component of an autonomous vehicle control system in a simulated test scenario according to embodiments of the invention. As illustrated in Figure 1, one or more computer readable data structures 110 are provided, each comprising parameters that define a test scenario are provided. For example, the computer readable data structures 110 may comprise OpenSCENARIO files according to the OpenSCENARIO standard that has been developed to provide a standardised description of traffic situations for the purpose of simulation. Alternate computer readable data structures 110 that may be used include High Level Scenario Description Language (HSDL) files. The data structures 110 are provided to a control system 120 that provides a simulator core operable to execute a simulation of traffic situations, along with the operation of an ego vehicle representing a vehicle including the autonomous vehicle control system under test, and provide one or more outputs based on the simulation. The outputs of the control system 120 are provided to a component 130 of the autonomous vehicle control system under test. The outputs may comprise simulated sensor data for at least one sensor of the autonomous vehicle control system. An output 140 of the component 130 may then be recorded and used to validate the operation of the component 130. For example, the output 140 may comprise one or more control signals to an actuator of the autonomous vehicle control system (e.g. to a control signal to activate brakes of the autonomous vehicle in response to a detected obstruction). In some embodiments, the output 140 may be fed back to the control system 120 to influence the behaviour of the ego vehicle, that is the vehicle controlled by the autonomous vehicle control system, in the simulation. In embodiments, the component 130 may comprise a hardware module of the vehicle, i.e. a controller with associated software to be provided on the vehicle as part of the autonomous vehicle control system. Furthermore, component 130 may represent a system or sub-system providing a portion of the autonomous vehicle control system comprising a number of discrete components that may be tested together. One or more simulated sensor signals may be provided to the hardware component to facilitate hardware-in-the-loop testing, providing realistic simulated sensor inputs for the hardware components as part of a testing process. In some embodiments, the output 140 of the component may be provided to an actuator of the autonomous vehicle control system and operation of the actuator in response to the simulated input signals may be monitored to validate correct control of the actuator by the component 130 as part of the simulated test scenario. In other embodiments, the component 130 may be a software module forming part of the autonomous vehicle control system, for example executed on emulated vehicle hardware to allow testing of the autonomous vehicle control system early in the development cycle before hardware components are available. The one or more outputs of the simulation may be each be based on a respective model of at least one sensor of the autonomous vehicle control system to generate realistic sensor signals for input to the component 130 based on the simulation scenario being executed on the control system 120. Simulation platforms for testing autonomous driving features with predefined test scenarios, such as using OpenSCENARIO files, have been developed and allow for testing of autonomous driving features in a controlled environment. The predefined test scenarios include parameters defining the presence and action of other vehicles, street furniture, hazards, etc. that may be encountered in a scenario. However, in order to provide thorough testing of components 130 across multiple test cases representative of a wide range of real world conditions, hundreds of test scenarios may need to be defined. Generating a computer readable data structure, such as an OpenSCENARIO file, for each of these hundreds of test scenarios may require significant resources as parameters defining the position and actions of each actor (e.g. each other vehicle) in the test scenario must be defined and incorporated into a predefined format for use by the simulation environment. Figure 2 illustrates a method 200 of automatically generating a computer readable data structure defining a test scenario based on a natural language description of the test scenario. According to the method of Figure 2, a selection may be made between generating a computer readable data structure 110 defining a test scenario of an appropriate format for input to the control system 120, such as an OpenSCENARIO file, or generating an intermediate data structure, such as a Javascript Object Notation (JSON) file, which may then be parsed into the appropriate format. According to the method of Figure 2, a natural language description of the test scenario is obtained 210 from an operator of the method, the natural language description describing the test scenario in the operators own words. As an example, a natural language description of a test scenario may be: • Input text description The ego vehicle is travelling with an initial speed of 60 kph and 5 meters from the starting point of the lane. A leading vehicle is positioned in front of the ego vehicle with a headway time of 2 seconds and travelling with an initial speed equal to the ego vehicle's speed. There are two blocking objects, a pedestrian and a bus, positioned in the same lane and 500 meters and 515 meters ahead of the ego vehicle, respectively. When the leading vehicle is 50 meters away from the pedestrian, it performs a lane change to cut out from blocking objects with a lateral velocity of 2 m / s. The scenario stops 10 seconds after the ego vehicle would reach the target without braking. A determination 220 may be made as to whether to directly generate a test scenario file 110, comprising a computer readable data structure defining the test scenario, or whether to generate an intermediate data structure. In some embodiments, determination 220 may not be present and the method may always proceed by directly generating test scenario file 110 defining the test scenario (e.g. an OpenSCENARIO file), or always proceed by generating intermediate data structure (e.g. a JSON file). In the case that an intermediate data structure is to be generated, the natural language description of the test scenario is processed at block 230 using a generative natural language processing, NLP, engine that is operable to convert the natural language description of the test scenario into a JSON file format. For example, the NLP engine may process the example description provided above and determine parameters such as speed and relative (or absolute) position of the ego vehicle, leading vehicle, pedestrian, and bus forming part of the scenario. A JSON file may then be generated including, for each identified entity, a data structure defining the determined parameters for that entity. The JSON file output by the NLP engine is then parsed at block 240 to generate the test scenario file 110 defining the test scenario (e.g. an OpenSCENARIO file) to be provided to the control system 120. Alternatively, in the case that the test scenario file 110 defining the test scenario (e.g. an OpenSCENARIO file) is to be generated by the NLP engine directly, the natural language description of the test scenario is processed by an NLP engine at block 250 to identify the entities present in the test scenario and to determine the associated parameters for each entity and then to generate the test scenario file 110. Whether the method proceeds via block 230, generating an intermediate data structure, or via block 250, generating the test scenario file 110 directly from the natural language description of the test scenario may depend on a characteristic, such as the complexity, of the natural language description of the test scenario. For example, in the case that the natural language description is relatively simple, it may be more reliable to generate the intermediate data structure which may then be parsed to generate the test scenario file 110. However, for more complex scenarios, direct generation of the test scenario file 110 by the NLP engine may be suitable for use. In some embodiments, the NLP engine may be trained to select whether to proceed via block 230 or block 250 based on the natural language description of the test scenario, for example by inclusion of a classifier layer on the output of the NLP engine, the classifier layer trained to classify the natural language description as more appropriate for direct conversion or conversion via the intermediate data structure. According to embodiments, the NLP engine processes the test scenario description to identify one or more dynamic entities forming part of the test scenario. For example, based on the example test scenario description above, an ego vehicle, a leading vehicle, a pedestrian and a bus may be identified. These entities may be represented in a computer readable data structure output by the NLP engine. An example data structure defining the entities is provided below: <Entities> < Scenarioobject name-"ego 1 ”> <CatalogReference catalogName= "VehicleCatalog" entryName="car_blue7> <ObjectController> <CatalogReference catalogName="Controllercatalog" entryName= "ALKSController” / > < / ObjectController> < / ScenarioObject> < ScenarioObject name="car2 "> <CatalogReference catalogName= "VehicleCatalog" entryName= "car_yellow7> < / ScenarioObject> <ScenarioObject name= "pedestrian3"> <CatalogReference catalogName= "PedestrianCatalog" entryName= ”pedestrian_adult" / > < / ScenarioObject> <ScenarioObject name= ”bus4"> <CatalogReference catalogName= ''VehicleCatalog'' entry Name= ”bus_blue'' / > < / ScenarioObject> < / Entities> Furthermore, the NLP engine interprets the natural language description of the test scenario to determine one or more initial conditions and actions, associated with the identified entities. In the above example, an initial speed and position relative to a traffic lane is defined for the ego vehicle as “60 kph” and “5m from the start of the lane”. The NLP engine identifies this information and includes corresponding parameters in the output data structure, for example: <Actions> <Private entityRef=''ego1"> <PrivateAction> <LongitudinalAction> <SpeedAction> <SpeedActionDynamics dynamics Shape="step" value="1.0"dynamicsDimension="time" / > <SpeedActionTarget> <AbsoluteTargetSpeed value= "16.67" / > < / SpeedActionTarget> < / SpeedAction> < / LongitudinalAction> < / PrivateAction> <PrivateAction> <TeleportAction> <Position> <LanePosition roadld=''0” laneid=”-3" s="0.0" offset="0.0"> <Orientation h^O.O” type="relative'' / > < / LanePosition> < / Position> < / TeleportAction> < / PmateAction> < / Private> Thus, the NLP engine processes the easily understandable, natural language, description of the test scenario and identifies the relevant features of the scenario, i.e. the dynamic entities and actions associated with the entities, and generates a computer readable data structure providing parameters defining the test scenario, e.g. parameters defining the entities forming part of the test scenario, the initial conditions such as speed and location of the entities, and any manoeuvres associated with the entities and forming part of the test scenario. This computer readable data structure may then be provided to the control system 120 to provide simulated sensor data to the component 130 of the autonomous vehicle control system to allow testing of the component 130. In embodiments, the NLP engine identifies information associated with at least one dynamic entity in the test scenario, the information including one or more of: a type of vehicle, e.g. a car, bus, etc.; a position associated with the dynamic entity, such as initial location of the entity in the test scenario; a speed associated with the dynamic entity; and a trajectory of associated with the dynamic entity, e.g. a change in direction and speed associated with a manoeuvre made by the entity as part of the test scenario. The position, speed, and / or trajectory may be defined relative to an ego vehicle of the test scenario, or may be defined in absolute terms according to a coordinate system of the test scenario environment. Simulations of test scenarios may also include simulations of an environment around the vehicle, for example street furniture, road layout, nearby buildings, etc. For example, some scenarios may include traversal of road junctions, traffic controls such as traffic lights, and the like. More realistic simulations may be achieved through the use of detailed pre-prepared map data defining the environment, for example map data may be captured from real-world locations through 3D scanning or similar. However, identifying an appropriate location or portion of the predefined map data for performing a particular test scenario may be difficult as it may be necessary to search a large proportion of the map data to identify a location having a number of specific road elements in a particular combination. According to the method 200 illustrated in Figure 2, an appropriate portion of the map data including a plurality of road segments required to perform a particular test scenario may be automatically identified based on the natural language description of the test scenario. In particular, the natural language description is processed by an NLP engine at block 260 to identify a plurality of road segments associated with the test scenario and based on these road segments a subset of the map matching, or at least substantially corresponding with, the requirements of the described test scenario is identified. An indication of the identified portion of the predefined map is then provided, as appropriate, to block 230 or 250 for use in generating the test scenario file 110. Thus, the method 200 is operable to obtain a natural language description of a test scenario, identify an appropriate portion of a map in which the test scenario may be simulated and generate the test scenario file 110, such as an OpenSCENARIO file, that defines parameters to define the actions and movements of one or more entities present in the test scenario on the road segments at the identified location in the map data. Figure 3 illustrates a computer implemented method 300 according to an embodiment of the present invention. The method 300 begins in block 310. In block 310, the method 300 comprises obtaining an input sequence comprising a natural language description, such as a sequence of words, describing a test scenario for testing a component of an autonomous vehicle control system. At block 320, the method 300 comprises processing the input sequence using a natural language processing engine, such as a generative artificial intelligence algorithm. The natural language processing engine processes the input sequence to generate information associated with at least one dynamic entity forming part of the test scenario, for example as discussed above. Based on the information associated with the at least one dynamic entity, a computer readable data structure is generated 330, the computer readable data structure comprising a plurality of parameters defining the test scenario. For example, the computer readable data structure may comprise an OpenSCENARIO or High Level Scenario Description Language (HSDL) file, or an intermediate data structure, such as a JSON, YAML (Yet Another Markup Language) or extensible Markup Language (XML) structure that may be parsed into an appropriate format for use by the control system 120. At block 340, the control system 120 generates simulated sensor data for at least one sensor of the autonomous vehicle control system based on the computer readable data structure. For example, the control system 120 may simulate a road traffic environment according to a test scenario and generate sensor data corresponding to the simulated environment to be provided to the component 130 under test. At block 350, the simulated sensor data is provided to the component 130 to allow the operation of the component 130 to be validated. Thus, the method 300 allows a test scenario to be automatically generated from a natural language description of the test scenario, for example in line with method 200, and then the test scenario may be simulated for testing the component 130 of the autonomous vehicle control system. Certain methods and systems as described herein may be implemented by a processor that processes program code that is retrieved from a non-transitory storage medium. Figure 4 illustrates an example of a computer system 400 operable to implement the described methods according to embodiments of the invention. Computer system 400 includes memory 410, one or more processors 420, network interface 430, and non-transitory computer-readable medium 440. The computer-readable medium 440 can be any medium that can contain, store, or maintain programs and / or data for use by or in connection with an instruction execution system or other system for giving effect to instructions. The computer-readable medium 440 can comprise any one of many physical media such as, for example, electronic, magnetic, optical, electromagnetic, or semiconductor media. More specific examples of suitable machine-readable media include, but are not limited to, a hard drive, a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory, or a portable storage such as, for example, a USB drive or disk. In Figure 4, the computer-readable storage medium 440 comprises program code to perform a method or implement a device as described herein. For example, the program code when executed may implement a method corresponding to the example shown in Figure 2 or Figure 3. Figure 5 illustrates an example of a natural language processing, NLP, engine 500 according to an embodiment of the present invention and suitable for implementing the method 200 of Figure 2. NLP engine 500 of Figure 5 comprises a pre-trained natural language processing engine 530 and at least one output layer 540 comprising a plurality of nodes of a neural network. According to some embodiments, a fine-tuning process may be performed in which the output layer 540 is trained to output data in a particular format, such as the desired computer readable data structure. In operation, an input sequence comprising natural language description of the test scenario 510 and a prompt 520 is input to the pre-trained NLP 530. As known in the art, the input sequence 510 and / or prompt 520 may be tokenised by dividing a sequence of words into a plurality of tokens, each token representing a word or sub-word piece of the input sequence. Each token may be assigned a numeric value to generate a vector of values that encodes the words of the input sequence 510 and / or prompt 520 to be input to the pre-trained NLP algorithm 530. The pre-trained NLP algorithm 530 receives the input sequence 510 and prompt 520 and generates an encoded vector that represents, or embeds, a meaning of the input sequence. For example, the NLP algorithm may receive a tokenised input sequence describing a test scenario described in the user’s own words and generate a contextualised embedding associated with the entities and actions present in the test scenario being described. In practice, the same test scenario may be described in different words by different users. However the NLP engine 530 is able to extract the underlying meaning of the descriptions and therefore the different descriptions of a particular test scenario would result in similar encoded vectors recognising the underlying meaning of the input sequences is the same. Pre-trained NLP algorithm 530 may comprise a generative neural network that has been trained on a large training set of natural language texts. Examples of training corpora used to train the pre-trained NLP algorithm 530 include Wikipedia pages and BooksCorpus. Examples of a pre-trained NLP algorithm 530 include a Generative Pre-Trained Transformer, GPT, architecture model; a Text-to-Text Transfer Transformer, T5, model; Pathways Language Model, PaLM; Large Language Model Meta Al, LLaMA; Megatron-LM model; or another generative large language model. The NLP engine 530 may output a vector embedding representing the entities and parameters of the test scenario, corresponding to the meaning of the natural language description of the test scenario. The vector embedding may be received by output layer 540 which generates a computer readable data structure comprising a plurality of parameters defining the test scenario based on the vector embedding. Figure 6 illustrates a computer implemented method 600 according to an embodiment of the present invention to train a natural language processing engine 500 for use in the methods of Figure 2 and Figure 3. Figure 7 illustrates an iterative process 700 corresponding to the method 600 of training the NLP engine 500 according to an embodiment of the invention. As illustrated in Figure 7, a corpora of training data 710 is provided comprising training example input / output pairs of natural language descriptions of test scenarios and associated computer readable data structures comprising a plurality of parameters defining the corresponding test scenario. In block 610 of the method 600, the set of training data 710 to be used to train the NLP engine 500, and in particular output layer 540, is received. In block 620, for one or more of the input / output pairs, the natural language description of the test scenario is input to the NLP engine 500. For each of the input values used to train the NLP engine 500, an example natural language description input sequence may be provided to NLP engine 500 and an output is generated by the NLP engine 500 in response, the output comprising a predicted computer readable data structure corresponding to the described test scenario. In block 630, the predicted computer readable data structure is compared with the associated computer readable data structure output 730 corresponding to the computer readable data structure of the input / output pair and an error between the predicted computer readable data structure and the computer readable data structure of the training data is characterised. For example, the predicted computer readable data structure may be compared in a comparator 740 with the actual computer readable data structure 730 provided in the training data 710 to characterise an error between the predicted result and the result provided in the training set. In block 640, an optimisation algorithm is used to update weights of the neural network based on the characterised error. Updated weights may be calculated by a tuning algorithm 720 that receives the output of comparator 740 and in response updates weights of the neural network of the NLP engine 500 using an appropriate optimisation algorithm. In embodiments, during training of the NLP engine 500, weights of the pre-trained NLP algorithm 530 may be frozen and weights of the output layer 540 updated to train the output layer 540 to output the data structure in a desired format. A number of example input sequences suitable for use as training for training the NLP engine 500 to convert the natural language description of a test scenario with to a desired computer readable data structure are provided below. Data structures, such as OpenSCENARIO files, corresponding with each input sequence may be generated by a user using traditional techniques to provide training output values corresponding with each input sequence. Example input sequences are: • The ego vehicle starts at a speed of 60kph and 5 meters behind a lead vehicle driving in the same lane. After 10 seconds, the leading vehicle starts to swerve left by 1.5 meters with a lateral acceleration of 0.3 m / sA2. After completing the left swerve, the leading vehicle returns to its original lane. Then, the leading vehicle swerves right by 1.5 meters before returning to its original lane again. The scenario ends after 50 seconds. • The ego vehicle is driving on a straight road with an initial speed of 60 kph. A blocking object is positioned partially on the ego vehicle lane, with a lateral offset of 1.5 meters and a longitudinal offset of 500 meters. The scenario stops 10 seconds after the ego vehicle would reach the target without braking. • The ego vehicle is driving at an initial speed of 60 kph and approaching a pedestrian that is standing still 500 meters ahead facing the road on the road curb. The pedestrian starts crossing the road at a speed of 5 kph when the ego vehicle is close enough that it would hit the pedestrian with the centre of its front bumper if it does not brake. The scenario ends 10 seconds after the ego vehicle would reach the pedestrian initial position without braking. According to some embodiments, prompt 520 may comprise a template computer readable data structure, or schema, defining a desired output format of the data structure to be provided by the NLP engine 500. The template may indicate particular parameters to be defined for entities present in the test scenario that should be determined by the NLP engine 500 from the input sequence 510. In some embodiments, the template may be provided as a JSON template. An example JSON template for defining the entities of the 15 test scenario is provided below: "type": "object", "properties": { "actors": { "type": "array", "items": { "type": "object", "properties": { "id": "integer, .start all id from 1", "actorType": { "type": "string", "enum": [ "ego", "car", "pedestrian", "cyclist", "truck", "bus", "van", "obstacle" }. "InitPos": { "type": "object", "properties": { "s": "number in meter, convert headway time to meters, relative value to absolute value, from the start of the lane”, "t": "number in meter, 0 if not specified", "heading": "number in degree, 90 for crossing" "initSpeed": "number in m / s, calculate absolute value from relative speed", "initRoadld": "integer", "initLaneld": "integer, 99 for road border / kerb / curb" "actions": { "type": "array", "items": { "type": "object", "properties": { "id": "integer", "eventType": { "type": "string", "enum": [ "crossing", "swerve", "turn_right", "turn_left", "changejanes", "U_tum", "accelerate", "decelerate" ] "actorld": "integer", "actionValue": { "type": "object,", "properties": { "valueType": { "type": "string", "enum": [ "accelerate_rate", "speed", "longitudinal_speed", "lateraLspeed”, "longitudinal_accelerate_rate", "lateral_accelerate_rate" "value": "number, in m / s2 for acceleration_rate" } 1 }, "trigger": { "type": "object", "properties": { "triggerType": "string", "enum": [ "time", "speed", "RelativeSpeed", "longitudinalDis", "relativeLongitudinalDis", "lateraloffset", "relativeLateralOffset", "collision", "actionCompleted", "doseToCollision" "rule": "string", "enum": [ "Greater", "Equal", "Less", "Completed" "value": "number, action id if type is actionCompleted", "refActorld": "integer, required only when relevant" "target": { "type": "object, required", "properties": { "valueType": "string", "enum": [ "time", "distance", "speed", "longitudinalDis", "lateraloffset", "collision", "laneld" "value": "number, traveling distance for distance / longitudinalDis / lateralDis", "refActorld": "integer, required if the value is relative to another actor" } "duration": { "type": "object", "properties": { "time": "number", "trigger": { "type": "object", "properties": { "valueType": "string, null if not specified", "enum": [ "distance", "actionCompleted" "value": "number, travelling distance or action id", "actorld": "integer, required for distance type" It will be appreciated that various changes and modifications can be made to the present invention without departing from the scope of the present application.

Claims

1. A computer implemented method comprising:obtaining an input sequence, the input sequence comprising a natural language description of a test scenario for testing a component of an autonomous vehicle control system;processing the input sequence using a natural language processing, NLP, engine to generate information associated with at least one dynamic entity forming part of the test scenario;based on the information associated with the at least one dynamic entity forming part of the test scenario, generating a computer readable data structure comprising a plurality of parameters defining the test scenario;generating simulated sensor data for at least one sensor of the autonomous vehicle control system based on the computer readable data structure; andproviding the simulated sensor data as an input to the component of the autonomous vehicle control system.

2. The computer implemented method of claim 1, wherein the information associated with the at least one dynamicentity comprises at least one of: a vehicle type associated with the dynamic entity; a position associated with the dynamic entity; a speed associated with the dynamic entity; and a trajectory associated with the dynamic entity.

3. The computer implemented method of claim 1 or claim 2, wherein the NLP engine comprises a pre-trained NLPalgorithm.

4. The computer implemented method of claim 3, wherein the pre-trained NLP algorithm comprises one of: aGenerative Pre-trained Transformer, GPT, model; Text-to-Text Transfer Transformer, T5, model; Pathways Language Model, PaLM; Large Language Model Meta Al, LLaMA; or another generative large language model.

5. The computer implemented method of claim 3 or claim 4, wherein the NLP engine further comprises one or moreoutput layers, each of the one or more output layers comprising a plurality of nodes of a neural network;The method further comprising fine-tuning the NLP engine by training the one or more output layers using a set of training data, the training data comprising training example input / output pairs of natural language descriptions of a test scenario and an associated computer readable data structure comprising the plurality of parameters defining the test scenario.

6. The computer implemented method of any of claims 1 to 5, wherein processing the input sequence using theNLP engine comprises providing the input sequence as a prompt to the natural language processing engine, the prompt further comprising a template computer readable data structure; andwherein the computer readable data structure comprising the plurality of parameters defining the test scenario is generated based on the template computer readable data structure.

7. The computer implemented method of any of claims 1 to 6, further comprising parsing the computer readabledata structure comprising the plurality of parameters defining the test scenario to generate an OpenScenario file defining the test scenario.8, The computer implemented method of any of claims 1 to 7, wherein the component of the autonomous vehiclecontrol system comprises a hardware module.

9. The computer implemented method of any of claims 1 to 8, wherein generating simulated sensor data for at leastone sensor of the autonomous vehicle control system further comprises obtaining a model of the at least one sensor and generating a simulated sensor output based on the model and the computer readable data structure.

10. The computer implemented method of any of claims 1 to 9, wherein the autonomous vehicle control systemcomprises at least one actuator controlled based on an output of the component of the autonomous vehicle control system, the computer implemented method further comprising monitoring an actuation of the actuator in response to providing the simulated sensor data as in input to the component of the autonomous vehicle control system.

11. The computer implemented method of any of claims 1 to 10, wherein the computer readable data structurecomprising the plurality of parameters defining the test scenario comprises an OpenScenario file.

12. An control system comprising:a processor; anda memory comprising computer program instructions that when executed by the processor cause the apparatus to:obtain an input sequence, the input sequence comprising a natural language description of a test scenario for testing a component of an autonomous vehicle control system;process the input sequence using an NLP engine to generate information associated with at least one dynamic entity forming part of the test scenario;based on the information associated with at least one dynamic entity forming part of the test scenario, generate a computer readable data structure comprising a plurality of parameters defining the test scenario;generate simulated sensor data for at least one sensor of the autonomous vehicle control system based on the computer readable data structure; andprovide the simulated sensor data as in input to the component of the autonomous vehicle control system.

13. The control system of claim 12, wherein the component of the autonomous vehicle control system comprises ahardware module.

14. A method of training a generative natural language processing engine for generating a computer readable datastructure comprising a plurality of parameters defining a test scenario based on an input sequence comprising a natural language description of a test scenario for testing a component of an autonomous vehicle control system, the natural language processing engine comprising a neural network including:a pre-trained natural language processing, NLP, algorithm to receive a prompt comprising the input sequence; andone or more output layers, each layer of the one or more output layers comprising a plurality of nodes of a neural network;wherein the method comprises:receiving a set of training data to train the natural language processing engine, the training data comprising training example input / output pairs of natural language descriptions of a test scenario and an associated computer readable data structure comprising the plurality of parameters defining the test scenario;5 inputting the natural language description from one or more of the input / output pairs from the set of training datato obtain a predicted computer readable data structure;characterising an error between the predicted computer readable data structure and the computer readable data structure corresponding to the natural language description of the training set; andusing an optimisation algorithm to update weights of the one or more output layers.10 15. Computer readable instructions which, when executed by a computer, are arranged to perform a methodaccording to any of claims 1 to 11, or 14.19

Images