Autonomous vehicle simulation

Abstract

A method of generating and augmenting image frames for use in an autonomous vehicle simulator. When augmenting an existing image frame, it is encoded to generate a sequence of scene tokens, and a sub-set of the scene tokens corresponding to an asset to be in-painted are determined. A trained world model is used to generate tokens for the asset, which when in-painting are used to replace the determine sub-set. The final sequence of scene tokens are then decoded to generate the image frame output suitable for use in the autonomous vehicle simulator.

Description

[0001] Autonomous Vehicle Simulation

[0002] Technical Field

[0003] This invention relates to autonomous vehicles, such as autonomous cars, trucks, buses, vans, and such like. Particularly, though not exclusively, the present invention is concerned with improving the digital testing and / or validation of autonomous vehicles using simulated environments.

[0004] Background Art

[0005] There has been a great deal of development in recent years in the field of autonomous vehicles, with ever more advanced self-driving vehicles becoming a reality. There are varying degrees of autonomy, often referred to as the 'levels' of automation, with these levels progressively outsourcing more of the driving functions from a human operator to a computer or 'automated driver system' (ADS). Much advancement has been made in recent years towards truly autonomous vehicles which can drive in a real-world road environment without any human interaction.

[0006] There are a number of different approaches to autonomous driving, with different paradigms being used. Some current autonomous driving systems, known in the art perse, make use of a modular paradigm in which a series of discrete processing stages make driving decisions based on predetermined rules. These modules may utilize sensor fusion methods and route-locating algorithms, emphasizing environmental perception. However, these systems often struggle with complex situations, and their interpretation and response capabilities can be limited. Furthermore, such modular systems can suffer from error propagation.

[0007] Other solutions make use of 'end-to-end' methods, and there has been substantial progress in end-to-end deep learning methods for autonomous systems in recent years. Those skilled in the art will appreciate that with an 'end-to-end' paradigm, inputs from the car's sensors are typically mapped directly to driving outputs. This approach may provide better performance with complex or unusual situations. End-to-end approaches may also be more reliable compared to modular approaches which can suffer from error propagation. The Al models used by autonomous vehicles, e.g. end-to-end models, typically require training. This training may be done offline (e.g. using simulators) and / or online (e.g. while driving in a real-world environment, for example with a human safety driver).

[0008] The Applicant has previously developed GAIA-1 (Generative Artificial Intelligence for Autonomy), which utilises a multi-modal approach that leverages video, text and action inputs to generate realistic driving videos. By training on a vast corpus of real-world UK urban driving data, the GAIA-1 model learns to predict the subsequent frames in a video sequence, resulting in an autoregressive (AR) prediction capability without needing any labels. This resembles the approach seen in large language models (LLMs).

[0009] GAIA-1 is a true world model that learns to understand and disentangle the important concepts of driving, including cars, trucks, buses, pedestrians, cyclists, road layouts, buildings, and traffic lights. What sets the GAIA-1 model apart from other models, known in the art perse, is its ability to provide fine-grained control over both ego-vehicle (i.e. the autonomous vehicle itself) behaviour and other essential scene features. Whether altering the ego-vehicle's behaviour or modifying the overall scene dynamics, this model offers unparalleled flexibility, making it an invaluable tool for accelerating the development of foundation models for autonomous driving.

[0010] Those skilled in the art will appreciate that a world model is a predictive model of the future that learns a general representation of the world in order to understand the consequences of its actions. The main use cases are pure representation learning, planning (look-ahead search), or learning a policy in the world model (neural simulator).

[0011] World modelling has been used as a pre-training task to learn a compact and general representation in a self-supervised way. Subsequently, using this representation as a state for traditional reinforcement learning (RL) algorithms significantly accelerated convergence speed. World models can also be utilized for look-ahead search, in order to plan by imagining the outcomes of future actions. They have proven to be highly effective in game environments or board games. Additionally, world models can be a solution to the sample efficiency issues of RL algorithms by acting as a simulator of the environment, although this assumes the world model is an accurate model of the environment.

[0012] World modelling may be cast as a single sequence model, treating states, actions, and rewards as simply a stream of data. The advantage of such a perspective is that world models can benefit from scaling properties of high-capacity sequence model architectures applied to large-scale unsupervised training. This is the approach that GAIA-1 takes, leveraging those scaling properties to model complex environments such as real-world driving scenes.

[0013] The Applicant has appreciated there is a critical need for realistic and versatile testing environments for autonomous driving models. These simulated testing environments are of critical importance for autonomous vehicles. By its very nature, an autonomous vehicle with unreliable or unpredictable behaviour could be extremely dangerous, posing a risk to humans and objects in the vicinity of the vehicle when operating under real world conditions. It is important, therefore, to be able to train and test the vehicle under simulation, where no harm will be caused.

[0014] Traditional procedural asset-based driving simulators such as CARLA do not provide for control over dynamic assets within a simulated scene.

[0015] There are neural Tenderers known in the art perse - such as UniSim developed by Waabi, and Mars developed by Z. Wu, et. al - that may jointly model static and dynamic scene elements with configurable positions. UniSim and Mars both use an object NeRF approach, in which they learn a differentiable Tenderer conditioned on an encoding for each asset to account for the appearance of that particular object. The biggest disadvantage with these approaches is that neither are easily extendable for pedestrians or deformable objects.

[0016] There are also known techniques for scene editing of neural scene representations using a text interface, such as lnstruct-NeRF2NeRF. Those skilled in the art will appreciate that such Neural Radiance Field (or 'NeRF') based approaches rely on overfitting on a 3D simulation to build a 3D model of an environment, from which novel views can be rendered. However, the Applicant has appreciated that NeRF-based techniques typically cannot deal with dynamic assets within a scene, at best being able to remove them.

[0017] Existing fully generative approaches - such as GAIA-1 developed by the Applicant, and commaVQ. developed by comma. ai - are not equipped with explicit 3D rendering techniques.

[0018] The Applicant has appreciated a need for improved techniques that enable the generation of image and video scenes suitable for the simulation of autonomous vehicles, for example to train the driving models across a diverse range of scenarios. In particular, having control over the inclusion of dynamic assets within the scene (such as other vehicles) - for example their position, orientation, size, and type - would be highly beneficial for creating realistic and versatile testing environments.

[0019] Summary of the Invention

[0020] In accordance with a first aspect, embodiments of the present invention provide a computer-implemented method of generating an augmented image frame for use in an autonomous vehicle simulator, the augmented image frame comprising a visual representation of a scene of a driving environment and at least one in-painted asset, the method comprising: obtaining an input image frame, the input image frame comprising an initial visual representation of a scene of the driving environment; encoding the image frame input to generate a sequence of scene tokens; determining a sub-set of the scene tokens in the sequence, said sub-set of tokens corresponding to an asset to be in-painted into the scene of the driving environment; inputting into a trained world model: i) at least a portion of the sequence of scene tokens; ii) data representative of the determined sub-set of tokens; and ill) data representative of the asset to be in-painted; using the world model to generate in-painted tokens to replace the sub-set of tokens in the sequence of scene tokens, thereby generating an augmented sequence of scene tokens; and decoding the augmented sequence of scene tokens to generate the augmented image frame.

[0021] The step of determining a sub-set of the scene tokens in the sequence, said sub-set of tokens corresponding to an asset to be in-painted into the scene of the driving environment may be understood as a step of determining a sub-set of the scene tokens in the sequence of scene tokens of the input image frame which are to be modified (i.e. the subset of tokens that are to be replaced by the in-painted tokens) by in-painting of the asset.

[0022] The step of encoding the image frame input to generate a sequence of scene tokens may comprise encoding the entire input image (i.e. all of the input image). Thus, this step may not involve the encoding of only a part or a portion of the input image frame.

[0023] The augmented sequence of scene tokens are augmented as compared to the sequence of scene tokens generated from encoding the input image frame (i.e. the 'original' sequence of scene tokens). The original sequence of scene tokens comprises both the determined sub-set of scene tokens and additional tokens that do not correspond to the asset to be inpainted (i.e. additional tokens that are not modified by the inpainting of the asset). Accordingly, the augmented sequence of scene tokens comprises the in-painted tokens and the additional tokens from the original sequence of tokens that do not correspond to the asset to be in-painted.

[0024] The step of decoding the augmented sequence of scene tokens may comprise decoding all of the augmented sequence of scene tokens together / simultaneously / in combination with one another. That is, both the in-painted tokens and the additional tokens from the original sequence of tokens that do not correspond to the asset to be in-painted are decoded together / simultaneously / in combination with one another.

[0025] This first aspect extends to a system configured to generate an augmented image frame for use in an autonomous vehicle simulator, the augmented image frame comprising a visual representation of a scene of a driving environment and at least one in-painted asset, the system comprising: an image encoder configured to receive an input image frame, the input image frame comprising an initial visual representation of a scene of the driving environment, said image encoder being further configured to encode the image frame input to generate a sequence of scene tokens; an asset mapper configured to determine a sub-set of the scene tokens in the sequence, said sub-set of tokens corresponding to an asset to be in-painted into the scene of the driving environment; a trained world model configured to: i) receive at least a portion of the sequence of scene tokens; ii) receive data representative of the determined sub-set of tokens; ill) receive data representative of the asset to be in-painted; and iv) generate in-painted tokens to replace the sub-set of tokens in the sequence of scene tokens, thereby generating an augmented sequence of scene tokens; and a decoder configured to decode the augmented sequence of scene tokens to generate the augmented image frame.

[0026] The system of the first aspect may be configured to implement the above discussed method of the first aspect, optionally in any optional form thereof.

[0027] The first aspect of the invention also extends to a non-transitory computer-readable medium comprising instructions which, when executed by a processor, cause the processor to carry out a method of generating an augmented image frame for use in an autonomous vehicle simulator, the augmented image frame comprising a visual representation of a scene of a driving environment and at least one in-painted asset, the method comprising: obtaining an input image frame, the input image frame comprising an initial visual representation of a scene of the driving environment; encoding the image frame input to generate a sequence of scene tokens; determining a sub-set of the scene tokens in the sequence, said sub-set of tokens corresponding to an asset to be in-painted into the scene of the driving environment; inputting into a trained world model: i) at least a portion of the sequence of scene tokens; ii) data representative of the determined sub-set of tokens; and iii) data representative of the asset to be in-painted; using the world model to generate in-painted tokens to replace the sub-set of tokens in the sequence of scene tokens, thereby generating an augmented sequence of scene tokens; and decoding the augmented sequence of scene tokens to generate the augmented image frame.

[0028] This first aspect of the invention further extends to a computer software product comprising instructions which, when executed by a processor, cause the processor to carry out a method of generating an augmented image frame for use in an autonomous vehicle simulator, the augmented image frame comprising a visual representation of a scene of a driving environment and at least one in-painted asset, the method comprising: obtaining an input image frame, the input image frame comprising an initial visual representation of a scene of the driving environment; encoding the image frame input to generate a sequence of scene tokens; determining a sub-set of the scene tokens in the sequence, said sub-set of tokens corresponding to an asset to be in-painted into the scene of the driving environment; inputting into a trained world model: i) at least a portion of the sequence of scene tokens; ii) data representative of the determined sub-set of tokens; and iii) data representative of the asset to be in-painted; using the world model to generate in-painted tokens to replace the sub-set of tokens in the sequence of scene tokens, thereby generating an augmented sequence of scene tokens; and decoding the augmented sequence of scene tokens to generate the augmented image frame.

[0029] The non-transitory computer-readable medium and / or the computer software product of the first aspect may comprise instructions which, when executed by a processor, cause the processor to carry out of the method of the first aspect, optionally in any optional form thereof. Thus it will be appreciated that embodiments of the present invention provide an advantageous arrangement which provides a mechanism for in-painting assets into a scene. This approach allows for control over assets incorporated into an existing scene, which may be dynamic assets that are difficult to handle with conventional techniques known in the art perse. This advantageously allows for the use of realistic images / videos captured in the real-world, while also allowing for a fine degree of control over the placement of dynamic assets into that scene so that an autonomous vehicle model can be trained and tested across a wide range of realistic scenarios.

[0030] Those skilled in the art will appreciate that a 'world model' is a machine learning model that provides a learned, structured representation and understanding of the environment. A world model reasons about a given scene's high-level components and dynamics. The use of the world model to place the asset based on the information provided advantageously leverages the learnings of the world model to ensure that the asset is placed in a way that is consistent with how the real world works. This allows the synthetic assets that are in-painted into the scene to be more realistic than had a simple in-painting technique (e.g. based on a bounding box alone) been used. As the autonomous vehicle models being trained and tested are intended for use in a real-world practical environment, it is of critical importance that the scenarios on which those models are trained and tested are realistic.

[0031] There are many world models which can be used with embodiments of the present invention. By way of example, the Applicant's own GAIA-1 is a world model suitable for use with embodiments the present invention. It will be appreciated, however, that other such world models could be used instead. The GAIA-1 model demonstrates the ability to manifest the generative rules of the real world. Emerging properties such as learning high- level structures, generalization, creativity, and contextual awareness indicate that the model can comprehend and reproduce the rules and behaviours of the world. Moreover, GAIA-1 exhibits understanding of 3D geometry, for example, by effectively capturing the intricate interplay of pitch and roll induced by road irregularities such as speed bumps. It showcases reactive behaviours of other agents demonstrating the ability to understand causality in decision making of road users. World models, such as GAIA-1, are the basis for the ability to predict what might happen next, which is fundamentally important for decision-making in autonomous driving.

[0032] As outlined above, the Applicant has appreciated that scene in-painting may play a crucial role in the digital testing and validation of autonomous vehicles by enabling the creation of diverse and complex test scenarios, using the knowledge of the world model. This is achieved by integrating realistic assets (e.g. pedestrians, cars, vans, traffic lights, etc) in a scene, which can originate from various sources.

[0033] The input image frame may be captured by an image capture device (for example a camera or image capture device as known in the art). The input image frame may be captured by an image capture device mounted on a vehicle.

[0034] The scenes (and / or input image frame(s)) may originate from pre-recorded videos. These are sourced from footage captured by vehicles, which inherently limits the perspective to the original vehicle's trajectory. In such scenarios, simulation and validation can be conducted through 'open-loop' testing. In open-loop testing, the decisions made by the autonomous vehicle being tested do not influence the vehicle's future state in the simulation. This setup allows for the evaluation of the AV's responses to predefined scenarios without altering the course of the simulation's background or trajectory.

[0035] The scenes (and / or input image frame(s)) may, additionally or alternatively, originate from rendering techniques. When employing rendering techniques such as Neural Radiance Fields (NeRF) or Gaussian Splatting, there is flexibility to manipulate the viewpoint within the scene. This capability facilitates 'closed-loop' testing, a method where the autonomous vehicle's actions can influence its future state within the simulation. Unlike open-loop testing, closed-loop testing allows for dynamic interaction with the simulated environment, enabling a more comprehensive assessment of the AV's decision-making and adaptability.

[0036] Regardless of the origin of the scenes, the process of in-painting assets into these environments remains consistent. The approach provided by embodiments of the present invention allows for the simulation of realistic and variable conditions, such as the introduction of pedestrians, other vehicles, and obstacles, providing a robust framework for thoroughly testing and validating autonomous driving systems under a wide range of conditions and scenarios.

[0037] In order to determine which tokens correspond to a particular asset, projection techniques may be used, at least in some embodiments. Projecting the asset onto the scene may generate an asset mask. The physical attributes of assets may be used to determine their impact on the image's tokenized representation. For example, by projecting the assets' position, orientation, size, and type onto the tokenized scene, the tokens which are 'occupied' by these assets may be determined. Those skilled in the art will appreciate that such projection techniques rely on geometry to determine which area(s) of the image will be affected by the placement of the asset(s).

[0038] The invention may be used to generate multiple images, which may in turn be used to generate a video (i.e. a sequence of images). Thus, in some embodiments, the method further comprises generating an augmented video for use in an autonomous vehicle simulator, the augmented video comprising a sequence including the augmented image frame and at least one further augmented image frame, the method further comprising, for each further augmented image frame: obtaining a current input image frame (i.e. an input image frame that is, with reference to the sequential position in which it appears in the video, sequentially correspondent to the augmented image frame to be generated), the current input image frame comprising an initial visual representation of a current scene of the driving environment; encoding the current image frame input to generate a current sequence of scene tokens; determining a current sub-set of the scene tokens in the current sequence, said sub-set of tokens corresponding to an asset to be in-painted into the current scene of the driving environment; inputting into the trained world model: i) at least a portion of the sequence of scene tokens; ii) a set of one or more previous sequences of scene tokens; ill) data representative of the determined current sub-set of tokens; and iv) data representative of the asset to be in-painted; using the world model to generate current in-painted tokens to replace the current sub-set of tokens in the current sequence of scene tokens, thereby generating a respective augmented sequence of scene tokens; and decoding the augmented sequences of scene tokens to generate the augmented video.

[0039] The step of determining a current sub-set of the scene tokens in the current sequence, said sub-set of tokens corresponding to an asset to be in-painted into the current scene of the driving environment may be understood as a step of determining a current sub-set of the scene tokens in the current sequence of scene tokens of the current input image frame which are to be modified (i.e. the current subset of tokens that are to be replaced by the in-painted tokens) by in-painting of the asset.

[0040] The step of encoding the current image frame input to generate a current sequence of scene tokens may comprise encoding the entire current input image (i.e. all of the current input image). Thus, this step may not involve the encoding of only a part or a portion of the current input image frame.

[0041] The respective augmented sequence of scene tokens are augmented as compared to the current sequence of scene tokens generated from encoding the current input image frame. The current sequence of scene tokens comprises both the determined current sub-set of scene tokens and additional tokens that do not correspond to the asset to be in-painted (i.e. additional tokens that are not modified by the inpainting of the asset). Accordingly, the respective augmented sequence of scene tokens comprises the in-painted tokens and the additional tokens from the current sequence of tokens that do not correspond to the asset to be in-painted.

[0042] The step of decoding the augmented sequences of scene tokens to generate the augmented video may comprise, in addition to decoding the augmented sequence of scene tokens corresponding to the augmented image frame, decoding the respective augmented sequence of scene tokens corresponding to the at least one further augmented image frame. The respective steps of decoding may occur together / in combination / simultaneously. Alternatively, the decoding of the augmented sequence of scene tokens corresponding to the augmented image frame and the respective augmented sequence of scene tokens corresponding to the at least one further augmented image frame may occur sequentially.

[0043] The step of decoding the respective augmented sequence of scene tokens may comprise decoding all of the respective augmented sequence of scene tokens together / simultaneously / in combination with one another. That is, both the in-painted tokens and the additional tokens from the current sequence of tokens that do not correspond to the asset to be in-painted are decoded together / simultaneously / in combination with one another.

[0044] The use of the principles of the present invention to generate augmented videos is novel and inventive in its own right and thus, when viewed from a second aspect, embodiments of the present invention provide a computer-implemented method of generating an augmented video for use in an autonomous vehicle simulator, the augmented video comprising a sequence of image frames each comprising a visual representation of a scene of a driving environment, wherein at least one of said image frames comprises an augmented image frame further comprising a visual representation of at least one inpainted asset, the method comprising: for each image frame in the augmented video:

[0045] I) obtaining a current input image frame, the current input image frame comprising an initial visual representation of a current scene of the driving environment;

[0046] II) encoding the current image frame input to generate a current sequence of scene tokens; and

[0047] III) responsive to determining that an asset is to be in-painted into the current scene of the driving environment to generate an augmented image frame: a) determining a current sub-set of the scene tokens in the current sequence corresponding to said asset; b) inputting into the trained world model: i) at least a portion of the sequence of scene tokens; ii) responsive to the current image frame not being a first image frame, a set of one or more previous sequences of scene tokens; iii) data representative of the determined current sub-set of tokens; and iv) data representative of the asset to be in-painted; c) using the world model to generate current in-painted tokens to replace the current sub-set of tokens in the current sequence of scene tokens, thereby generating a respective augmented sequence of scene tokens; and d) decoding the augmented sequences of scene tokens to generate the current augmented video.

[0048] Step a) of determining a current sub-set of the scene tokens in the current sequence corresponding to said asset may be understood as a step of determining a current sub-set of the scene tokens in the current sequence of scene tokens of the current input image frame which are to be modified (i.e. the current subset of tokens that are to be replaced by the in-painted tokens) by in-painting of the asset.

[0049] Step II of encoding the current image frame input to generate a current sequence of scene tokens may comprise encoding the entire current input image (i.e. all of the current input image). Thus, this step may not involve the encoding of only a part or a portion of the current input image frame.

[0050] The respective augmented sequence of scene tokens are augmented as compared to the current sequence of scene tokens generated from encoding the current input image frame. The current sequence of scene tokens comprises both the determined current sub-set of scene tokens and additional tokens that do not correspond to the asset to be in-painted (i.e. additional tokens that are not modified by the inpainting of the asset). Accordingly, the respective augmented sequence of scene tokens comprises the in-painted tokens and the additional tokens from the current sequence of tokens that do not correspond to the asset to be in-painted. The step of decoding the augmented sequences of scene tokens to generate the current augmented video may comprise decoding each of the respective augmented sequence of scene tokens (corresponding individually to each of the augmented image frames) together / in combination / simultaneously or sequentially.

[0051] The step of decoding (each of) the respective augmented sequence of scene tokens may comprise decoding all of the respective augmented sequence of scene tokens together / simultaneously / in combination with one another. That is, both the in-painted tokens and the additional tokens from the current sequence of tokens that do not correspond to the asset to be in-painted are decoded together / simultaneously / in combination with one another.

[0052] This second aspect extends to a system configured to generate an augmented video for use in an autonomous vehicle simulator, the augmented video comprising a sequence of image frames each comprising a visual representation of a scene of a driving environment, wherein at least one of said image frames comprises an augmented image frame further comprising a visual representation of at least one in-painted asset, the system comprising: an image encoder configured to receive a current input image frame, the current input image frame comprising an initial visual representation of a current scene of the driving environment, said image encoder being further configured to encode the current image frame input to generate a current sequence of scene tokens; an asset mapper configured to, for each image frame in the augmented video, responsive to determining that an asset is to be in-painted into the current scene of the driving environment to generate an augmented image frame, determine a current sub-set of the scene tokens in the sequence, said sub-set of tokens corresponding to said asset; a trained world model configured to, for each image frame in the augmented video: i) receive at least a portion of the sequence of scene tokens; ii) responsive to the current image frame not being a first image frame, receive a set of one or more previous sequences of scene tokens; ill) receive data representative of the determined current sub-set of tokens; iv) receive data representative of the asset to be in-painted; and v) generate in-painted tokens to replace the current sub-set of tokens in the current sequence of scene tokens, thereby generating a respective augmented sequence of scene tokens; and a decoder configured to decode the augmented sequences of scene tokens to generate the augmented video.

[0053] The system of the second aspect may be configured to implement the above discussed method of the second aspect, optionally in any optional form thereof.

[0054] The second aspect of the invention also extends to a non-transitory computer-readable medium comprising instructions which, when executed by a processor, cause the processor to carry out a method of generating an augmented video for use in an autonomous vehicle simulator, the augmented video comprising a sequence of image frames each comprising a visual representation of a scene of a driving environment, wherein at least one of said image frames comprises an augmented image frame further comprising a visual representation of at least one in-painted asset, the method comprising: for each image frame in the augmented video:

[0055] I) obtaining a current input image frame, the current input image frame comprising an initial visual representation of a current scene of the driving environment;

[0056] II) encoding the current image frame input to generate a current sequence of scene tokens; and

[0057] III) responsive to determining that an asset is to be in-painted into the current scene of the driving environment to generate an augmented image frame: a) determining a current sub-set of the scene tokens in the current sequence corresponding to said asset; b) inputting into the trained world model: i) at least a portion of the sequence of scene tokens; ii) responsive to the current image frame not being a first image frame, a set of one or more previous sequences of scene tokens; ill) data representative of the determined current sub-set of tokens; and iv) data representative of the asset to be in-painted; c) using the world model to generate current in-painted tokens to replace the current sub-set of tokens in the current sequence of scene tokens, thereby generating a respective augmented sequence of scene tokens; and d) decoding the augmented sequences of scene tokens to generate the current augmented video.

[0058] This second aspect of the invention further extends to a non-transitory computer-readable medium and / or a computer software product comprising instructions which, when executed by a processor, cause the processor to carry out a method of generating an augmented video for use in an autonomous vehicle simulator, the augmented video comprising a sequence of image frames each comprising a visual representation of a scene of a driving environment, wherein at least one of said image frames comprises an augmented image frame further comprising a visual representation of at least one inpainted asset, the method comprising: for each image frame in the augmented video:

[0059] I) obtaining a current input image frame, the current input image frame comprising an initial visual representation of a current scene of the driving environment;

[0060] II) encoding the current image frame input to generate a current sequence of scene tokens; and

[0061] III) responsive to determining that an asset is to be in-painted into the current scene of the driving environment to generate an augmented image frame: a) determining a current sub-set of the scene tokens in the current sequence corresponding to said asset; b) inputting into the trained world model: i) at least a portion of the sequence of scene tokens; ii) responsive to the current image frame not being a first image frame, a set of one or more previous sequences of scene tokens; ill) data representative of the determined current sub-set of tokens; and iv) data representative of the asset to be in-painted; c) using the world model to generate current in-painted tokens to replace the current sub-set of tokens in the current sequence of scene tokens, thereby generating a respective augmented sequence of scene tokens; and d) decoding the augmented sequences of scene tokens to generate the current augmented video.

[0062] The non-transitory computer-readable medium and / or the computer software product of the second aspect may comprise instructions which, when executed by a processor, cause the processor to carry out of the method of the second aspect, optionally in any optional form thereof.

[0063] It will be appreciated that, for the second (and onward) image frames, the tokens corresponding to at least some (and potentially all) previous image frames are provided to the world model to assist in its prediction of tokens for the current image frame. This may typically include at least the tokens corresponding to the immediately preceding image frame. This may include at least the tokens corresponding to the immediately preceding N image frames. For the first image frame, there will typically be no previous image frame and so that input to the world model may be omitted for that first image frame.

[0064] Not every image frame in the video is necessarily augmented, as there may be image frames in which no asset needs to be in-painted. In some embodiments, the augmented video further comprises one or more non-augmented video frames.

[0065] In some embodiments, the set of one or more previous sequences of scene tokens input to the world model comprise the respective sequence of scene tokens used to generate a previous image frame. In some such embodiments, the set of one or more previous sequences of scene tokens comprises one or more previous augmented sequences of scene tokens used to respectively generate a previous augmented image frame.

[0066] There are two different modes in which the world model can process the scene tokens. The first mode is an 'autoregressive' mode in which generates each token based on the sequence of previous tokens of the image, the specified attributes of the assets (such as their position, orientation, size, and type), and potentially other forms of conditioning that the model accepts (e.g. text). Thus, in some embodiments, the portion of the sequence of scene tokens comprises one or more scene tokens preceding the respective sub-set.

[0067] The other mode is a 'masking' mode in which the world model predicts only masked tokens (corresponding to the assets), based on the value of a sub-set of the other tokens, the specified attributes of the assets (such as their position, orientation, size, and type) and potentially other forms of conditioning that the model accepts (e.g. text). Thus, in some potentially overlapping embodiments, the portion of the sequence of scene tokens comprises one or more scene tokens following the respective sub-set. Thus in a particular set of embodiments, the portion of the sequence of scene tokens comprises all scene tokens in the sequence except for the respective sub-set.

[0068] It is possible to in-paint more than one asset into a given scene and thus, at least in some embodiments, a plurality of assets is in-painted into the scene of the driving environment.

[0069] In some such embodiments, the method comprises: responsive to determining that a plurality of assets is to be in-painted into the scene of the driving environment, determining a respective sub-set of the scene tokens corresponding to each asset to be in-painted into the scene of the driving environment; inputting into the trained world model data representative of each determined sub-set of tokens, and data representative of each asset to be in-painted; and using the world model to generate respective in-painted tokens to replace each sub-set of tokens in the sequence of scene tokens, thereby generating an augmented sequence of scene tokens.

[0070] The processing of multiple assets may be carried out either in series or in parallel. In some embodiments, the method further comprises using the world model to generate the respective in-painted tokens to replace each sub-set of tokens in series. It will be appreciated that 'in series' means sequentially, one after another (rather than simultaneously). The series processing of multiple assets to be in-painted is particularly suitable when using the 'autoregressive' mode. In the autoregressive mode, the tokens up until the sub-set for replacement (i.e. the tokens preceding the sub-set for replacement) are provided to the world model, which then predicts the next token(s) which in turn are used as the replacements.

[0071] In some alternative embodiments, comprising using the world model to generate the respective in-painted tokens to replace each sub-set of tokens in parallel. It will be appreciated that 'in parallel' means simultaneously, with multiple assets having the respective in-painted tokens generated at the same time. The parallel processing of multiple assets to be in-painted is particularly suitable when using the 'masking' mode. In the masking mode, all tokens except for the sub-set to be replaced are provided to the world model, so the sub-set need not be a continuous set of tokens (i.e. that are adjacent or connected to one another), with different regions of tokens being determined for replacement. As such, multiple regions within the image corresponding to different assets to be in-painted can be 'carved out' at the same time, with the remainder fed to the world model so that the world model can generate the replacement tokens for all of those regions simultaneously.

[0072] In some embodiments, the data representative of the asset to be in-painted comprises one or more of the group comprising: a position of the asset, an orientation of the asset, a size of the asset, and a type of the asset. All of these may be supplied and thus, in a particular set of embodiments, the data representative of the asset to be in-painted comprises: a position of the asset, an orientation of the asset, a size of the asset, and a type of the asset.

[0073] This data representative of the asset to be in-painted may be supplied via any suitable means. For example, a user interface may be provided through which the values may be provided. In-painting the dynamic assets into an existing image or video advantageously provides a way to use existing real-world footage and augment it with specific dynamic assets for the autonomous vehicle simulation. However, the Applicant has appreciated that in addition to in-painting an asset into an existing scene, similar principles could be applied to fully generate the scene itself together with the asset, thereby avoiding the need for existing images / videos. Thus, in accordance with a third aspect, embodiments of the present invention provide a computer-implemented method of generating an image frame for use in an autonomous vehicle simulator, the image frame comprising a generated visual representation of a scene of a driving environment and at least one generated asset, the method comprising: obtaining data representative of the asset to be generated; inputting the data representative of the asset to be generated into a trained world model; using the world model to generate a sequence of scene tokens, the sequence of scene tokens comprising: i) a sub-set of tokens corresponding to the asset; and ii) a further sub-set of tokens corresponding to the scene of the driving environment; and decoding the generated sequence of scene tokens to generate the image frame.

[0074] The step of decoding the generated sequence of scene tokens may comprise decoding all of the generated scene tokens together / simultaneously / in combination with one another. That is, both the subset and further sub-set of tokens are decoded together / simultaneously / in combination with one another.

[0075] This third aspect extends to a system configured to generate an image frame for use in an autonomous vehicle simulator, the image frame comprising a generated visual representation of a scene of a driving environment and at least one generated asset, the system comprising: a trained world model configured to: a) receive data representative of the asset to be generated; and b) generate a sequence of scene tokens, the sequence of scene tokens comprising: i) a sub-set of tokens corresponding to the asset; and ii) a further sub-set of tokens corresponding to the scene of the driving environment; and a decoder configured to decode the generated sequence of scene tokens to generate the image frame.

[0076] The system of the second aspect may be configured to implement the above discussed method of the second aspect, optionally in any optional form thereof.

[0077] The third aspect of the invention also extends to a non-transitory computer-readable medium comprising instructions which, when executed by a processor, cause the processor to carry out a method of generating an image frame for use in an autonomous vehicle simulator, the image frame comprising a generated visual representation of a scene of a driving environment and at least one generated asset, the method comprising: obtaining data representative of the asset to be generated; inputting the data representative of the asset to be generated into a trained world model; using the world model to generate a sequence of scene tokens, the sequence of scene tokens comprising: i) a sub-set of tokens corresponding to the asset; and ii) a further sub-set of tokens corresponding to the scene of the driving environment; and decoding the generated sequence of scene tokens to generate the image frame.

[0078] This third aspect of the invention further extends to a computer software product comprising instructions which, when executed by a processor, cause the processor to carry out a method of generating an image frame for use in an autonomous vehicle simulator, the image frame comprising a generated visual representation of a scene of a driving environment and at least one generated asset, the method comprising: obtaining data representative of the asset to be generated; inputting the data representative of the asset to be generated into a trained world model; using the world model to generate a sequence of scene tokens, the sequence of scene tokens comprising: i) a sub-set of tokens corresponding to the asset; and ii) a further sub-set of tokens corresponding to the scene of the driving environment; and decoding the generated sequence of scene tokens to generate the image frame.

[0079] The non-transitory computer-readable medium and / or the computer software product of the third aspect may comprise instructions which, when executed by a processor, cause the processor to carry out of the method of the third aspect, optionally in any optional form thereof.

[0080] The third aspect of invention may be used to generate multiple images, which may in turn be used to generate a video (i.e. a sequence of images). Thus, in some embodiments, the method further comprises generating a video for use in an autonomous vehicle simulator, the video comprising a sequence of the image frame and at least one further image frame, the method further comprising, for each further image frame: inputting into the trained world model: a) a set of one or more previous sequences of scene tokens; and b) data representative of the asset to be generated; using the world model to generate a current sequence of scene tokens, the current sequence of scene tokens comprising: i) a current sub-set of tokens corresponding to the asset; and ii) a further current sub-set of tokens corresponding to the scene of the driving environment; and decoding the generated sequences of scene tokens to generate the video.

[0081] The step of decoding the generated sequences of scene tokens to generate the video may comprise decoding each of the respective sequences of scene tokens (corresponding individually to each of the image frames) together / in combination / simultaneously or sequentially. The step of decoding (each of) the respective sequences of scene tokens may comprise decoding all of the respective sequence of scene tokens together / simultaneously / in combination with one another. That is, both the current and further sub-sets of tokens are decoded together / simultaneously / in combination with one another.

[0082] The use of the principles of the third aspect of the invention to generate augmented videos is novel and inventive in its own right and thus, when viewed from a fourth aspect, embodiments of the present invention provide a computer-implemented method of generating a video for use in an autonomous vehicle simulator, the video comprising a sequence of image frames each comprising a visual representation of a scene of a driving environment, wherein at least one of said image frames comprises a visual representation of at least one generated asset, the method comprising: for each image frame in the video: responsive to determining that an asset is to be generated within a current scene of the driving environment: a) inputting into the trained world model: i) responsive to the current image frame not being a first image frame, a set of one or more previous sequences of scene tokens; and ii) data representative of the asset to be generated; b) using the world model to generate a current sequence of scene tokens, the current sequence of scene tokens comprising: i) a current sub-set of tokens corresponding to the asset; and ii) a further current sub-set of tokens corresponding to the current scene of the driving environment; and c) decoding the generated sequences of scene tokens to generate the video.

[0083] The step of decoding the generated sequences of scene tokens to generate the video may comprise decoding each of the respective sequences of scene tokens (corresponding individually to each of the image frames) together / in combination / simultaneously or sequentially.

[0084] The step of decoding (each of) the respective sequences of scene tokens may comprise decoding all of the respective sequence of scene tokens together / simultaneously / in combination with one another. That is, both the current and further sub-sets of tokens are decoded together / simultaneously / in combination with one another.

[0085] This fourth aspect extends to a system configured to generate a video for use in an autonomous vehicle simulator, the video comprising a sequence of image frames each comprising a visual representation of a scene of a driving environment, wherein at least one of said image frames comprises a visual representation of at least one generated asset, the system comprising: a trained world model configured to: a) responsive to the current image frame not being a first image frame, receive a set of one or more previous sequences of scene tokens; b) receive data representative of the asset to be generated; and c) generate current sequence of scene tokens, the current sequence of scene tokens comprising: i) a current sub-set of tokens corresponding to the asset; and ii) a further current sub-set of tokens corresponding to the current scene of the driving environment; and a decoder configured to decode the sequences of scene tokens to generate the video.

[0086] The system of the fourth aspect may be configured to implement the above discussed method of the fourth aspect, optionally in any optional form thereof.

[0087] The fourth aspect of the invention also extends to a non-transitory computer-readable medium comprising instructions which, when executed by a processor, cause the processor to carry out a method of generating a video for use in an autonomous vehicle simulator, the video comprising a sequence of image frames each comprising a visual representation of a scene of a driving environment, wherein at least one of said image frames comprises a visual representation of at least one generated asset, the method comprising: for each image frame in the video: responsive to determining that an asset is to be generated within a current scene of the driving environment: a) inputting into the trained world model: i) responsive to the current image frame not being a first image frame, a set of one or more previous sequences of scene tokens; and ii) data representative of the asset to be generated; b) using the world model to generate a current sequence of scene tokens, the current sequence of scene tokens comprising: i) a current sub-set of tokens corresponding to the asset; and ii) a further current sub-set of tokens corresponding to the current scene of the driving environment; and c) decoding the generated sequences of scene tokens to generate the video.

[0088] This fourth aspect of the invention further extends to a computer software product comprising instructions which, when executed by a processor, cause the processor to carry out a method of generating a video for use in an autonomous vehicle simulator, the video comprising a sequence of image frames each comprising a visual representation of a scene of a driving environment, wherein at least one of said image frames comprises a visual representation of at least one generated asset, the method comprising: for each image frame in the video: responsive to determining that an asset is to be generated within a current scene of the driving environment: a) inputting into the trained world model: i) responsive to the current image frame not being a first image frame, a set of one or more previous sequences of scene tokens; and ii) data representative of the asset to be generated; b) using the world model to generate a current sequence of scene tokens, the current sequence of scene tokens comprising: i) a current sub-set of tokens corresponding to the asset; and ii) a further current sub-set of tokens corresponding to the current scene of the driving environment; and c) decoding the generated sequences of scene tokens to generate the video.

[0089] The non-transitory computer-readable medium and / or the computer software product of the fourth aspect may comprise instructions which, when executed by a processor, cause the processor to carry out of the method of the fourth aspect, optionally in any optional form thereof.

[0090] It is possible to generate more than one asset into a given scene and thus, at least in some embodiments, a plurality of assets is generated in the scene of the driving environment. In some such embodiments, the method comprises: responsive to determining that a plurality of assets is to be generated in the scene of the driving environment, inputting into the trained world model data representative of each asset to be generated; and using the world model to generate a respective sub-set of tokens corresponding to each asset.

[0091] It should be noted that there is no structural difference between 'autoregressive' or 'masked' modes in the fully generative schemas (i.e. the third and fourth aspects of the invention). This is because when all tokens are generated by the world model, the order of the generation (autoregressive one by one, or masked in parallel) does not require any changes to the model itself, though the choice of mode will change the quality and inference speed performance of the model, where the autoregressive mode may provide for higher quality outputs, while the masking mode may provide for faster performance. The term "trained" used herein in respect of the world model means that, at least at the time of generating the augmented images, the world model is already trained. It should be noted, however, that in some embodiments of certain aspects the invention, the training of that world model may form part of the invention. In some embodiments, the world model is trained by: i) receiving video training data, said video training data comprising a plurality of time-varying image frames; ii) encoding the image frames to generate training image encodings; ill) receiving asset training data, said asset training data comprising asset information; iv) encoding the asset information to generate asset encodings; v) temporally aligning the image encodings and asset encodings; vi) generating a sequence of input tokens from the temporally aligned image encodings and asset encodings; and vii) using an autoregressive transformer to model the sequence, thereby generating the pre-trained world model and the set of learned world model weights.

[0092] This training method is novel and inventive in its own right and thus, when viewed from a fifth aspect, embodiments of the present invention provide a method of training a world model for use in an autonomous vehicle simulator, said method comprising: i) receiving video training data, said video training data comprising a plurality of time-varying image frames; ii) encoding the image frames to generate training image encodings; ill) receiving asset training data, said asset training data comprising asset information; iv) encoding the asset information to generate asset encodings; v) temporally aligning the image encodings and asset encodings; vi) generating a sequence of input tokens from the temporally aligned image encodings and asset encodings; and vii) using an autoregressive transformer to model the sequence, thereby generating the pre-trained world model and the set of learned world model weights. In certain embodiments of any of the foregoing aspects, a trainable image encoder may be used for encoding the image frames to generate the training image encodings. In some embodiments, the method comprises training a set of image encoder weights of the image encoder. The image encoder may be pre-trained. This trainable image encoder is then used as the image encoder for encoding images used in the generation of images in accordance with certain embodiments of the present invention.

[0093] The asset information used in training the world model may comprise one or more of the group comprising: a position of the asset, an orientation of the asset, a size of the asset, and a type of the asset.

[0094] In some such embodiments, a trainable asset encoder is used for encoding the asset information to generate the asset encodings, wherein the method comprises training a set of asset encoder weights of the asset encoder. The asset encoder may be pre-trained.

[0095] Each image frame may be encoded and discretized into a plurality of tokens for input into the world model. The image frames may be downsampled by a rate D.

[0096] The Applicant has appreciated that different modalities may be used to train the world model, with additional modalities beyond the image / video data and associated asset information providing data on which the world model can be trained, enhancing the patterns it is able to learn from the sequences of data it is provided during training.

[0097] In some embodiments, the step of training the world model further comprises: i) receiving action training data, said action training data comprising a plurality of time-varying driving actions; ii) encoding the driving actions to generate action encodings; ill) temporally aligning the image encodings, asset encodings, and action encodings; and iv) generating the sequence of input tokens from the temporally aligned image encodings, asset encodings, and action encodings. These driving actions correspond to driving parameters, such as a speed, a curvature, amounts of acceleration or braking being applied, a steering angle, an acceleration rate, a braking rate, a curvature rate, or such like. The driving actions may be raw signals, or may be formatted in some suitable manner, e.g. provided in a vectorised form. The driving action data may comprise a scalar value representing each driving parameters (e.g. a scalar representing speed, a scalar representing curvature, etc.).

[0098] In some such embodiments, a trainable action encoder is used for encoding the driving actions to generate the action encodings, wherein the method comprises training a set of action encoder weights of the action encoder. The action encoder may be pre-trained.

[0099] In some embodiments, the step of training the world model further comprises: i) receiving textual training data, said textual training data comprising a plurality of time-varying text data; ii) encoding the text data to generate text encodings; ill) temporally aligning the image encodings, asset encodings, and text encodings; and iv) generating the sequence of input tokens from the temporally aligned image encodings, asset encodings, and text encodings.

[0100] The textual data provided may be a text-based description of a particular scenario. For example, textual training data may comprise text statements such as "I am approaching a crossing yielding to pedestrians" or "It is safe to move so I am now accelerating."

[0101] In some such embodiments, a trainable text encoder is used for encoding the text data to generate the text encodings, wherein the method comprises training a set of text encoder weights of the text encoder. The text encoder may be pre-trained.

[0102] In some embodiments, the step of training the world model further comprises: i) receiving auxiliary sensor training data, said auxiliary sensor training data comprising a plurality of time-varying data relating to measurements from one or more auxiliary sensors; ii) encoding the auxiliary sensor data to generate auxiliary sensor encodings; ill) temporally aligning the image encodings, asset encodings, and auxiliary sensor encodings; and iv) generating the sequence of input tokens from the temporally aligned image encodings, asset encodings, and auxiliary sensor encodings.

[0103] The auxiliary sensor data corresponds to measurements acquired from any auxiliary sensor(s) on the vehicle. Such auxiliary sensors may, in some embodiments, comprise one or more of: a radar sensor, a lidar sensor, an infrared sensor, a range sensor, a distance sensor, an ultrasonic sensor, a rain sensor, a temperature sensor, a pressure sensor, and a load sensor. The auxiliary sensor data may be raw signals, or may be formatted in some suitable manner, e.g. provided in a vectorised form. The auxiliary sensor data may comprise a scalar value representing each measurement (e.g. a scalar representing distance to another vehicle ahead of the autonomous vehicle, an array of infrared values, etc.).

[0104] In some such embodiments, a trainable auxiliary sensor encoder is used for encoding the auxiliary sensor data to generate the auxiliary sensor encodings, wherein the method comprises training a set of auxiliary sensor encoder weights of the auxiliary sensor encoder. The auxiliary sensor encoder may be pre-trained.

[0105] Each modality (images and assets; potentially together with actions, text, and / or auxiliary sensor data, as appropriate) is encoded separately and then the appropriate sets of encodings are aligned temporally such that the correct time-based sequence of contemporaneous events is preserved. The world model is then trained by using an autoregressive transformer that models the sequence of tokenised, time-ordered encodings.

[0106] The world model may be configured to generate output tokens in which some of the output tokens are the same as the input tokens and some of the output tokens (the determined sub-sets) are the replacement tokens generated by the world model. Those output tokens may be input to the decoder configured to generate a video output comprising a plurality of generated image frames from the output tokens. Thus, in accordance with such embodiments, the decoder can decode the tokens output by the world model back into images / video, thus providing for synthetically generated realistic driving videos to be output. The decoder may be a diffusion decoder.

[0107] The decoder may be a trainable component. Thus, in some embodiments, the method may further comprise training a set of decoder weights of the decoder.

[0108] It will be appreciated that the term 'decoder' as used generally herein means an 'image decoder', 'video decoder', and / or an 'image / video decoder' as appropriate.

[0109] When viewed from a sixth aspect, embodiments of the present invention provide an autonomous vehicle simulator configured to train a driving model for an autonomous vehicle using the image and / or video generated in accordance with any of the first, second, third, or fourth aspects of the invention. This autonomous vehicle simulator may utilise the augmented and / or generated images and / or video generated in accordance with certain embodiments described herein, as appropriate.

[0110] When viewed from a seventh aspect, embodiments of the present invention provide an autonomous vehicle configured to operate using a driving model, said driving model being trained using the autonomous vehicle simulator in accordance with the sixth aspect of the invention.

[0111] It will be appreciated that the optional features described hereinabove in respect of embodiments of any aspect of the invention apply equally, where technically appropriate, to the other aspects of the invention outlined herein.

[0112] Where technically appropriate, embodiments of the invention may be combined. Embodiments are described herein as comprising certain features / elements. The disclosure also extends to separate embodiments consisting or consisting essentially of said features / elements.

[0113] Technical references such as patents and applications are incorporated herein by reference. Any embodiments specifically and explicitly recited herein may form the basis of a disclaimer either alone or in combination with one or more further embodiments.

[0114] In the context of this specification "comprising" is to be interpreted as "including". Aspects of the invention comprising certain elements are also intended to extend to alternative embodiments "consisting" or "consisting essentially" of the relevant elements.

[0115] The term "vehicle" as used herein should be understood to mean any kind of vehicle intended to travel on roads where some mechanical and / or electrical propulsion is used to drive the vehicle, whether operated autonomously or not. This includes, but is not limited to: cars, motorbikes, trucks, buses, coaches, vans, lorries, campervans, motor caravans, minibuses, limousines, all-terrain vehicles (ATVs), tractors, and other such vehicles that are mechanically or electrically driven.

[0116] Where context allows (e.g. in respect of other vehicles detected by the autonomous vehicle), the term "vehicle" further extends to non-driven vehicles, i.e. those without mechanical or electrical propulsion. This includes, but is not limited to: bicycles, unicycles, tricycles, quadracycles, rickshaws, carts, wagons, horse-drawn carts or carriages, and other such vehicles that are not mechanically or electrically driven.

[0117] The term "data" is used in different contexts herein to refer to digital information, such as that represented by known bit structures within one or more programming languages. In use, data may refer to digital information that is stored as bit sequences within computer memory.

[0118] Certain machine learning models may operate on structured arrays of data of a predefined bit format. Using terms of the art, these may be referred to as "vectors", as used herein. However, the term "vector" is understood by those skilled in the art to cover multidimensional arrays or "tensors" as well. It should be noted that for machine learning methods multidimensional arrays, e.g. with a defined extent in multiple dimensions, may be "flattened" so as to be represented (e.g., within memory) as a sequence or vector of values stored according to the predefined format (e.g., n-bit integer or floating-point number, signed or unsigned). Hence, the term "tensor" as used herein covers multidimensional arrays with one or more dimensions (e.g., vectors, matrixes, volumetric arrays etc).

[0119] Brief Description of the

[0120] Certain embodiments of the present invention will now be described with reference to the accompanying drawings, in which:

[0121] Fig. 1 is a block diagram illustrating an in-painting technique for a single augmented image frame using an autoregressive approach in accordance with an embodiment of the invention;

[0122] Fig. 2 is a block diagram illustrating an in-painting technique for a multiple augmented image frames using an autoregressive approach in accordance with an embodiment of the invention;

[0123] Fig. 3 is a block diagram illustrating an in-painting technique for a single augmented image frame using a masking approach in accordance with an embodiment of the invention;

[0124] Fig. 4 is a block diagram illustrating an in-painting technique for a multiple augmented image frames using a masking approach in accordance with an embodiment of the invention;

[0125] Fig. 5 is a block diagram illustrating a generative technique for a single generated image frame in accordance with a further embodiment of the invention; and

[0126] Fig. 6 is a block diagram illustrating a generative technique for a multiple generated image frames in accordance with a further embodiment of the invention.

[0127] Detailed Description

[0128] Fig. 1 is a block diagram illustrating an in-painting technique for a single augmented image frame using an autoregressive approach in accordance with an embodiment of the invention. Fig. 1 shows a system 100 which includes an image encoder 102, an asset mapper 104, a world model 106, and a decoder 108. The image encoder 102 is configured to receive an input image frame 110. This input image frame 110 is an image (e.g. captured from a camera) that provides a visual representation of a scene of the driving environment. Typically this image frame 110 may contain static assets within the scene, for example the road, pavement / sidewalk, trees, buildings, road signage, and such like - objects that do not move dynamically.

[0129] The image encoder 102 encodes the image frame 110 as a sequence of scene tokens 112. In other words, the image encoder 102 takes the image and moves it into the 'token space'. Thus this initial step involves processing the scene image with the image encoder 102. The encoder 102 breaks down the image into a matrix of tokens, where each token represents a distinct segment or "tile" of the original image.

[0130] Information regarding a dynamic asset to be in-painted 114 are provided. This information includes the position (for example relative to the vehicle or image capture device), orientation (i.e. its angle, for example relative to the vehicle or image capture device), size (for example absolute size of the asset), and type of the asset. For example, this information may specify respectively that the desired asset should be:

[0131] • Position = 10 m ahead;

[0132] • Orientation = 23°;

[0133] • Size = 4.8 m in length, 1.9 m in width, and 1.4 m in height; and

[0134] • Type = Car

[0135] The asset mapper 104 projects the asset onto the scene to generate an asset mask 116. In this step, the physical attributes of assets— entities that we want to in-paint in the scene- are used to determine their impact on the image's tokenized representation. By projecting the assets' position, orientation, size, and type onto the tokenized scene, the system identifies which tokens are "occupied" by these assets.

[0136] This projection effectively overlays the assets onto the scene background, marking the areas of interaction or alteration. In other words, the resultant asset mask 116 provides a mapping for which of the scene tokens should contain the asset (shown as white squares in Fig. 1), and which are unaffected by it (shown as black squares in Fig. 1). The world model is pre-trained on image / video training data and asset information training data. The world model 106 receives data representative of the dynamic asset information 114, and the sequence of scene tokens 112 up to the point where it reaches the tokens set to be replaced according to the asset mask 116 - i.e. the tokens 117 preceding the sub-set of tokens set for replacement. The world model is instructed regarding which tokens it is to generate replacements for (as set by the asset mask 116).

[0137] The world model 106 plays a crucial role in simulating the evolution of the scene by generating tokens 118 that represent the assets' positions and interactions within the scene. It operates autoregressively, meaning it generates each token 118 based on the sequence of previous tokens of the image, the specified attributes of the assets (such as their position, orientation, size, and type), and potentially other forms of conditioning that the model accepts (e.g. text).

[0138] When the image includes multiple assets, the process repeats for each asset. After the world model 106 generates tokens for one asset, it seamlessly transitions to the next, integrating each into the scene in turn. Scene tokens not impacted by the assets are appended without alteration.

[0139] The result is an augmented sequence of scene tokens 120 in which the replacement tokens 118 have replaced the corresponding original tokens from the sequence generated by the image encoder 102.

[0140] The final step involves converting the set of tokens— now representing the in-painted scene— back into a visual format through the decoder 108. This process reverses the initial tokenization, translating the discrete tokenized representation back into a continuous visual scene 122.

[0141] Fig. 2 is a block diagram illustrating an in-painting technique for a multiple augmented image frames using an autoregressive approach in accordance with an embodiment of the invention. Elements having reference numerals starting '2' in Fig. 2 are alike in form and function with those elements having reference numerals starting with '1' in Fig. 1 unless technical context dictates otherwise. When multiple images are being processed, the image encoder 202 processes the scene's latest image 210. As before, the encoder 202 breaks down that image 210 into a matrix of tokens, where each token represents a distinct segment or "tile" of the original image.

[0142] The asset mapper 204 acts in the same manner described previously, projecting the asset to be in-painted onto the tokenised representation of the latest image 210, generating an asset mask 216.

[0143] The world model 206 again operates autoregressively. However, compared to the single image process described above with reference to Fig. 1, the world model 206 also receives the token sequences of previous images 224 (as augmented via the in-painting process, as appropriate), together with the tokens 217 corresponding to the current image frame up that precede the tokens set for replacement according to the asset mask 216.

[0144] The world model 206 then generates each token based on the sequence of all tokens of the previous images in the video, previous tokens of the current image, the specified attributes of the assets (such as their position, orientation, size, and type), and potentially other forms of conditioning that the model accepts (e.g. text).

[0145] As before, if there are multiple assets to be included within a given image, the process is repeated for each such asset in series.

[0146] The decoder 208 now converts the output to video 222. This involves converting the set of tokens that represent all frames in the video - now representing the in-painted scene - back into a visual format through the decoder 208. This process reverses the initial tokenization, translating the discrete tokenized representation back into a continuous visual scene.

[0147] Fig. 3 is a block diagram illustrating an in-painting technique for a single augmented image frame using a masking approach in accordance with an embodiment of the invention.

[0148] Elements having reference numerals starting '3' in Fig. 3 are alike in form and function with those elements having reference numerals starting with '1' in Fig. 1 unless technical context dictates otherwise.

[0149] The image encoder 302 and asset mapper 304 work in the same manner described previously with reference to Fig. 1. However, in this case, the world model 306 uses a 'masking' approach, rather than operating autoregressively. The world model 306 is provided with a set of tokens 319 containing all of the scene tokens except those masked off by the asset mask 316.

[0150] The world model 306 predicts only masked tokens (corresponding to the assets), based on the value of a sub-set of the other tokens, the specified attributes of the assets (such as their position, orientation, size, and type) and potentially other forms of conditioning that the model accepts (e.g. text).

[0151] This masking approach allows for the parallel processing of multiple assets to be in-painted simultaneously. Scene tokens not impacted by the assets are simply kept (rather than appended) without alteration.

[0152] As before, the decoder 308 converts the augmented sequence of tokens 320 (representing the in-painted scene and containing the generated replacement tokens 318) back to a visual image 322.

[0153] Fig. 4 is a block diagram illustrating an in-painting technique for a multiple augmented image frames using a masking approach in accordance with an embodiment of the invention. Elements having reference numerals starting '4' in Fig. 4 are alike in form and function with those elements having reference numerals starting with '3' in Fig. 3 unless technical context dictates otherwise.

[0154] When multiple images are being processed, the image encoder 402 processes the scene's latest image 410. As before, the encoder 402 breaks down that image 410 into a matrix of tokens, where each token represents a distinct segment or "tile" of the original image. The asset mapper 404 acts in the same manner described previously, projecting the asset to be in-painted onto the tokenised representation of the latest image 410, generating an asset mask 416.

[0155] The world model 406 is provided with a set of tokens 419 containing all of the scene tokens except those masked off by the asset mask 416. Compared to the single image process described above with reference to Fig. 3, the world model 406 also receives the token sequences of previous images 424 (as augmented via the in-painting process, as appropriate).

[0156] The world model 406 then generates each token based on the sequence of all tokens of the previous images in the video, previous tokens of the current image, the specified attributes of the assets (such as their position, orientation, size, and type), and potentially other forms of conditioning that the model accepts (e.g. text).

[0157] As described above with reference to Fig. 3, if there are multiple assets to be included within a given image, these can be processed simultaneously in parallel.

[0158] The decoder 408 then converts the output to video 422. This involves converting the set of tokens that represent all frames in the video - now representing the in-painted scene - back into a visual format through the decoder 408. This process reverses the initial tokenization, translating the discrete tokenized representation back into a continuous visual scene.

[0159] Fig. 5 is a block diagram illustrating a generative technique for a single generated image frame in accordance with a further embodiment of the invention. This system 500 is configured to generate an image frame for use in an autonomous vehicle simulator.

[0160] Unlike the embodiments described previously, in this embodiment the images frames are fully generated (i.e. both the assets and surrounding driving environment are generated), rather than using in-painting on an existing image. As such, it is not necessary to include an image encoder with this system. Similarly, no asset mapper is needed either since its role is to determine which tokens need to be generated by the world model, however in the fully generative case (as described with reference to Figs. 5 and 6), all tokens are to be generated so there is no explicit requirement for the asset mapper component in these cases.

[0161] The world model 506 receives information 514 regarding the physical attributes of assets to be included in the generated image. The world model 506 generates a sequence of scene tokens 520 comprising: a sub-set of tokens 518 corresponding to the asset; and a further sub-set of tokens 521 corresponding to the scene of the driving environment.

[0162] A decoder 508 is configured to decode the generated sequence of scene tokens 520 to generate the image frame 522.

[0163] Thus, in the fully generative case, only the asset information needs to be supplied to the world model 506, which then generates all the tokens required for subsequent decoding into a fully generated image frame.

[0164] Fig. 6 is a block diagram illustrating a generative technique for a multiple generated image frames in accordance with a further embodiment of the invention. Elements having reference numerals starting '6' in Fig. 6 are alike in form and function with those elements having reference numerals starting with '5' in Fig. 5 unless technical context dictates otherwise.

[0165] Compared to the single image process described above with reference to Fig. 5, the world model 606 also receives the generated token sequences of previous images 624. The world model uses those token sequences of previous images 624 in addition to the information 614 regarding the asset to be generated in order to generate the current sequence of tokens 620 for the current image frame. Each sequence of tokens 620 contains a sub-set of tokens 618 corresponding to the asset; and a further sub-set of tokens 621 corresponding to the scene of the driving environment.

[0166] The decoder 608 then converts the output to video 622. This involves converting the set of tokens that represent all frames in the video - the generated asset(s) and surrounding environment - into a visual format through the decoder 608. This process translates the newly generated discrete tokenized representation into a continuous visual scene.

[0167] Thus it will be appreciated that embodiments of the present invention provide improved techniques for generating realistic driving videos suitable for use in training an autonomous vehicle (by means of simulation) which permit 'artificial' dynamic assets to be fed into the system and 'overlaid' with either live or artificially generated video. This approach does not require complex developments to the neural state model to introduce dynamic agents. Other embodiments provide for full generation of the images / videos for use in the autonomous vehicle simulator, in which both the driving scene and dynamic assets within it are generated together using the world model.

[0168] In either case, the use of the world model allows for its knowledge of real-world rules and dynamics to be used to create realistic and versatile scenarios suitable for use in an autonomous vehicle simulator. Both the in-painting and generative cases provide technical advantages - the in-painting approach allows better re-simulation of real data, while the fully generative approach allows the creation of realistic data that is not identical to the real data collected in the real world.

[0169] It will be appreciated that this description provides examples for reference purposes, and the scope of the invention is defined by the claims.

Claims

Claims1. A computer-implemented method of generating an augmented image frame for use in an autonomous vehicle simulator, the augmented image frame comprising a visual representation of a scene of a driving environment and at least one in-painted asset, the method comprising: obtaining an input image frame, the input image frame comprising an initial visual representation of a scene of the driving environment; encoding the image frame input to generate a sequence of scene tokens; determining a sub-set of the scene tokens in the sequence, said sub-set of tokens corresponding to an asset to be in-painted into the scene of the driving environment; inputting into a trained world model: i) at least a portion of the sequence of scene tokens; ii) data representative of the determined sub-set of tokens; and iii) data representative of the asset to be in-painted; using the world model to generate in-painted tokens to replace the sub-set of tokens in the sequence of scene tokens, thereby generating an augmented sequence of scene tokens; and decoding the augmented sequence of scene tokens to generate the augmented image frame.

2. The method as claimed in claim 1, comprising generating an augmented video for use in an autonomous vehicle simulator, the augmented video comprising a sequence of the augmented image frame and at least one further augmented image frame, the method further comprising, for each further augmented image frame: obtaining a current input image frame, the current input image frame comprising an initial visual representation of a current scene of the driving environment; encoding the current image frame input to generate a current sequence of scene tokens; determining a current sub-set of the scene tokens in the current sequence, said sub-set of tokens corresponding to an asset to be in-painted into the current scene of the driving environment; inputting into the trained world model:i) at least a portion of the sequence of scene tokens; ii) a set of one or more previous sequences of scene tokens; iii) data representative of the determined current sub-set of tokens; and iv) data representative of the asset to be in-painted; using the world model to generate current in-painted tokens to replace the current sub-set of tokens in the current sequence of scene tokens, thereby generating a respective augmented sequence of scene tokens; and decoding the augmented sequences of scene tokens to generate the augmented video.

3. A computer-implemented method of generating an augmented video for use in an autonomous vehicle simulator, the augmented video comprising a sequence of image frames each comprising a visual representation of a scene of a driving environment, wherein at least one of said image frames comprises an augmented image frame further comprising a visual representation of at least one in-painted asset, the method comprising: for each image frame in the augmented video:I) obtaining a current input image frame, the current input image frame comprising an initial visual representation of a current scene of the driving environment;II) encoding the current image frame input to generate a current sequence of scene tokens; andIII) responsive to determining that an asset is to be in-painted into the current scene of the driving environment to generate an augmented image frame: a) determining a current sub-set of the scene tokens in the current sequence corresponding to said asset; b) inputting into the trained world model: i) at least a portion of the sequence of scene tokens; ii) responsive to the current image frame not being a first image frame, a set of one or more previous sequences of scene tokens; iii) data representative of the determined current sub-set of tokens; andiv) data representative of the asset to be in-painted; c) using the world model to generate current in-painted tokens to replace the current sub-set of tokens in the current sequence of scene tokens, thereby generating a respective augmented sequence of scene tokens; and d) decoding the augmented sequences of scene tokens to generate the current augmented video.

4. The method as claimed in claim 2 or 3, wherein the augmented video further comprises one or more non-augmented video frames.

5. The method as claimed in any of claims 2 to 4, wherein the set of one or more previous sequences of scene tokens input to the world model comprise the respective sequence of scene tokens used to generate a previous image frame.

6. The method as claimed in claim 5, wherein the set of one or more previous sequences of scene tokens comprises one or more previous augmented sequences of scene tokens used to respectively generate a previous augmented image frame.

7. The method as claimed in any preceding claim, wherein the portion of the sequence of scene tokens comprises one or more scene tokens preceding the respective sub-set.

8. The method as claimed in any preceding claim, wherein the portion of the sequence of scene tokens comprises one or more scene tokens following the respective sub-set.

9. The method as claimed in any preceding claim, wherein the portion of the sequence of scene tokens comprises all scene tokens in the sequence except for the respective sub-set.

10. The method as claimed in any preceding claim, wherein a plurality of assets is inpainted into the scene of the driving environment.

11. The method as claimed in claim 10, wherein the method comprises: responsive to determining that a plurality of assets is to be in-painted into the scene of the driving environment, determining a respective sub-set of the scene tokens corresponding to each asset to be in-painted into the scene of the driving environment; inputting into the trained world model data representative of each determined sub-set of tokens, and data representative of each asset to be in-painted; and using the world model to generate respective in-painted tokens to replace each sub-set of tokens in the sequence of scene tokens, thereby generating an augmented sequence of scene tokens.

12. The method as claimed in claim 11, comprising using the world model to generate the respective in-painted tokens to replace each sub-set of tokens in series.

13. The method as claimed in claim 11, comprising using the world model to generate the respective in-painted tokens to replace each sub-set of tokens in parallel.

14. The method as claimed in any preceding claim, wherein the data representative of the asset to be in-painted comprises one or more of the group comprising: a position of the asset, an orientation of the asset, a size of the asset, and a type of the asset.

15. A computer-implemented method of generating an image frame for use in an autonomous vehicle simulator, the image frame comprising a generated visual representation of a scene of a driving environment and at least one generated asset, the method comprising: obtaining data representative of the asset to be generated; inputting the data representative of the asset to be generated into a trained world model; using the world model to generate a sequence of scene tokens, the sequence of scene tokens comprising: i) a sub-set of tokens corresponding to the asset; and ii) a further sub-set of tokens corresponding to the scene of the driving environment; anddecoding the generated sequence of scene tokens to generate the image frame.

16. The method as claimed as claim 15, further comprising generating a video for use in an autonomous vehicle simulator, the video comprising a sequence of the image frame and at least one further image frame, the method further comprising, for each further image frame: inputting into the trained world model: a) a set of one or more previous sequences of scene tokens; and b) data representative of the asset to be generated; using the world model to generate a current sequence of scene tokens, the current sequence of scene tokens comprising: i) a current sub-set of tokens corresponding to the asset; and ii) a further current sub-set of tokens corresponding to the scene of the driving environment; and decoding the generated sequences of scene tokens to generate the video.

17. A computer-implemented method of generating a video for use in an autonomous vehicle simulator, the video comprising a sequence of image frames each comprising a visual representation of a scene of a driving environment, wherein at least one of said image frames comprises a visual representation of at least one generated asset, the method comprising: for each image frame in the video: responsive to determining that an asset is to be generated within a current scene of the driving environment: a) inputting into the trained world model: i) responsive to the current image frame not being a first image frame, a set of one or more previous sequences of scene tokens; and ii) data representative of the asset to be generated; b) using the world model to generate a current sequence of scene tokens, the current sequence of scene tokens comprising: i) a current sub-set of tokens corresponding to the asset; andii) a further current sub-set of tokens corresponding to the current scene of the driving environment; and c) decoding the generated sequences of scene tokens to generate the video.

18. The method as claimed in any preceding claim, wherein the world model is trained by: i) receiving video training data, said video training data comprising a plurality of time-varying image frames; ii) encoding the image frames to generate training image encodings; ill) receiving asset training data, said asset training data comprising asset information; iv) encoding the asset information to generate asset encodings; v) temporally aligning the image encodings and asset encodings; vi) generating a sequence of input tokens from the temporally aligned image encodings and asset encodings; and vii) using an autoregressive transformer to model the sequence, thereby generating the pre-trained world model and the set of learned world model weights.

19. A non-transitory computer-readable medium comprising instructions which, when executed by a processor, cause the processor to carry out the method of any of claims 1 to 18.

20. A computer software product comprising instructions which, when executed by a processor, cause the processor to carry out the method of any of claims 1 to 18.

21. A system configured to generate an augmented image frame for use in an autonomous vehicle simulator, the augmented image frame comprising a visual representation of a scene of a driving environment and at least one in-painted asset, the system comprising: an image encoder configured to receive an input image frame, the input image frame comprising an initial visual representation of a scene of the driving environment,said image encoder being further configured to encode the image frame input to generate a sequence of scene tokens; an asset mapper configured to determine a sub-set of the scene tokens in the sequence, said sub-set of tokens corresponding to an asset to be in-painted into the scene of the driving environment; a trained world model configured to: i) receive at least a portion of the sequence of scene tokens; ii) receive data representative of the determined sub-set of tokens; ill) receive data representative of the asset to be in-painted; and iv) generate in-painted tokens to replace the sub-set of tokens in the sequence of scene tokens, thereby generating an augmented sequence of scene tokens; and a decoder configured to decode the augmented sequence of scene tokens to generate the augmented image frame.

22. The system as claimed in claim 21, configured to generate an augmented video for use in an autonomous vehicle simulator, the augmented video comprising a sequence of the augmented image frame and at least one further augmented image frame, wherein: the image encoder is configured to receive a current input image frame, the current input image frame comprising an initial visual representation of a current scene of the driving environment, said image encoder being further configured to encode the current image frame input to generate a current sequence of scene tokens; the asset mapper is configured to determine a current sub-set of the scene tokens in the current sequence, said sub-set of tokens corresponding to an asset to be in-painted into the current scene of the driving environment; the trained world model is configured to: i) receive at least a portion of the sequence of scene tokens; ii) receive a set of one or more previous sequences of scene tokens; ill) receive data representative of the determined current sub-set of tokens; iv) receive data representative of the asset to be in-painted; andv) generate current in-painted tokens to replace the current sub-set of tokens in the current sequence of scene tokens, thereby generating a respective augmented sequence of scene tokens; and the decoder is configured to decode the augmented sequences of scene tokens to generate the augmented video.

23. A system configured to generate an augmented video for use in an autonomous vehicle simulator, the augmented video comprising a sequence of image frames each comprising a visual representation of a scene of a driving environment, wherein at least one of said image frames comprises an augmented image frame further comprising a visual representation of at least one in-painted asset, the system comprising: an image encoder configured to receive a current input image frame, the current input image frame comprising an initial visual representation of a current scene of the driving environment, said image encoder being further configured to encode the current image frame input to generate a current sequence of scene tokens; an asset mapper configured to, for each image frame in the augmented video, responsive to determining that an asset is to be in-painted into the current scene of the driving environment to generate an augmented image frame, determine a current sub-set of the scene tokens in the sequence, said sub-set of tokens corresponding to said asset; a trained world model configured to, for each image frame in the augmented video: i) receive at least a portion of the sequence of scene tokens; ii) responsive to the current image frame not being a first image frame, receive a set of one or more previous sequences of scene tokens; ill) receive data representative of the determined current sub-set of tokens; iv) receive data representative of the asset to be in-painted; and v) generate in-painted tokens to replace the current sub-set of tokens in the current sequence of scene tokens, thereby generating a respective augmented sequence of scene tokens; and a decoder configured to decode the augmented sequences of scene tokens to generate the augmented video.

24. The system as claimed in claim 22 or 23, wherein the augmented video further comprises one or more non-augmented video frames.

25. The system as claimed in any of claims 22 to 24, wherein the set of one or more previous sequences of scene tokens input to the world model comprise the respective sequence of scene tokens used to generate a previous image frame.

26. The system as claimed in claim 25, wherein the set of one or more previous sequences of scene tokens comprises one or more previous augmented sequences of scene tokens used to respectively generate a previous augmented image frame.

27. The system as claimed in any of claims 21 to 26, wherein the portion of the sequence of scene tokens comprises one or more scene tokens preceding the respective sub-set.

28. The system as claimed in any of claims 21 to 27, wherein the portion of the sequence of scene tokens comprises one or more scene tokens following the respective sub-set.

29. The system as claimed in any of claims 21 to 28, wherein the portion of the sequence of scene tokens comprises all scene tokens in the sequence except for the respective sub-set.

30. The system as claimed in any of claims 21 to 29, wherein a plurality of assets is inpainted into the scene of the driving environment.

31. The system as claimed in claim 30, wherein: the asset mapper is configured to, responsive to determining that a plurality of assets is to be in-painted into the scene of the driving environment, determine a respective sub-set of the scene tokens corresponding to each asset to be in-painted into the scene of the driving environment; and the trained world model is configured to:receive data representative of each determined sub-set of tokens, and data representative of each asset to be in-painted; and generate respective in-painted tokens to replace each sub-set of tokens in the sequence of scene tokens, thereby generating an augmented sequence of scene tokens.

32. The system as claimed in claim 31, wherein the world model is configured to generate the respective in-painted tokens to replace each sub-set of tokens in series.

33. The system as claimed in claim 31, wherein the world model is configured to generate the respective in-painted tokens to replace each sub-set of tokens in parallel.

34. The system as claimed in any of claims 21 to 33, wherein the data representative of the asset to be in-painted comprises one or more of the group comprising: a position of the asset, an orientation of the asset, a size of the asset, and a type of the asset.

35. A system configured to generate an image frame for use in an autonomous vehicle simulator, the image frame comprising a generated visual representation of a scene of a driving environment and at least one generated asset, the system comprising: a trained world model configured to: a) receive data representative of the asset to be generated; and b) generate a sequence of scene tokens, the sequence of scene tokens comprising: i) a sub-set of tokens corresponding to the asset; and ii) a further sub-set of tokens corresponding to the scene of the driving environment; and a decoder configured to decode the generated sequence of scene tokens to generate the image frame.

36. A system configured to generate a video for use in an autonomous vehicle simulator, the video comprising a sequence of image frames each comprising a visual representation of a scene of a driving environment, wherein at least one of said imageframes comprises a visual representation of at least one generated asset, the system comprising: a trained world model configured to: a) responsive to the current image frame not being a first image frame, receive a set of one or more previous sequences of scene tokens; b) receive data representative of the asset to be generated; and c) generate current sequence of scene tokens, the current sequence of scene tokens comprising: i) a current sub-set of tokens corresponding to the asset; and ii) a further current sub-set of tokens corresponding to the current scene of the driving environment; and a decoder configured to decode the sequences of scene tokens to generate the video.

37. A method of training a world model for use in an autonomous vehicle simulator, said method comprising: i) receiving video training data, said video training data comprising a plurality of time-varying image frames; ii) encoding the image frames to generate training image encodings; ill) receiving asset training data, said asset training data comprising asset information; iv) encoding the asset information to generate asset encodings; v) temporally aligning the image encodings and asset encodings; vi) generating a sequence of input tokens from the temporally aligned image encodings and asset encodings; and vii) using an autoregressive transformer to model the sequence, thereby generating the pre-trained world model and the set of learned world model weights.

38. An autonomous vehicle simulator configured to train a driving model for an autonomous vehicle using the image and / or video generated as claimed in any of claims 1 to 36.

39. An autonomous vehicle configured to operate using a driving model, said driving model being trained using the autonomous vehicle simulator as claimed in claim 38.

Images