System and method for base model-based reward design for autonomous driving

Abstract

The invention relates to a system and method for base model-based reward design for autonomous driving. Methods and systems for optimizing an action strategy of an autonomous vehicle machine learning model. An image corresponding to a vehicle surroundings is generated. These images are passed through an image encoder to generate an image-based embedding of the current state of the vehicle. A textual cue representing a target of an autonomous vehicle is passed through a textual encoder to generate a text-based embedding of the target. A similarity score representing a similarity between the image-based embedding of the current state and the text-based embedding of the target is determined. A reinforcement learning model for closed loop autonomous driving tasks is executed, where the similarity score is used as a reward function. An action strategy corresponding to vehicle control is optimized based on the reward function.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for foundation model-based reward design for autonomous driving. In an embodiment, a foundation model is used to output a similarity score that is used as a reward in a reinforcement learning model for autonomous driving training. Background Art

[0002] Autonomous vehicles, often referred to as self-driving or driverless vehicles, are vehicles capable of navigating and operating on roads and in various environments without direct human control. Autonomous vehicles use a combination of advanced technologies and sensors to sense their surroundings, make decisions, and perform driving tasks.

[0003] Autonomous vehicles are typically equipped with a variety of sensors, including lidar, radar, cameras, ultrasonic sensors, and sometimes additional technologies such as GPS and IMU (Inertial Measurement Unit). These sensors provide real-time data about the vehicle's surroundings, including the positions of other vehicles, pedestrians, road signs, and road conditions. The vehicle's on-board computer uses the data from the sensors to create a detailed map of the environment and sense objects and obstacles. This information is essential for navigation and collision avoidance.

[0004] Reinforcement learning (RL) can be used with autonomous driving, where the model learns optimal decision-making by interacting with the environment. After being trained on sensor data, the RL agent creates a state representation, defines an action space, and learns a policy that maps states to actions. Through simulation training and continuous learning, the RL model adapts to different driving scenarios. Integration with the perception system enhances the vehicle's safe navigation ability. Summary of the Invention

[0005] According to an embodiment, a method for optimizing an action policy of a machine learning model for an autonomous vehicle includes the following: generating an image of the environment of the autonomous vehicle based on vehicle sensor data representing a current state of the autonomous vehicle; passing the generated image through an image encoder to generate an image-based embedding of the current state of the autonomous vehicle; receiving a text prompt representing a target of the autonomous vehicle; passing the text prompt through a text encoder to generate a text-based embedding of the target; determining a similarity score representing a similarity between the image-based embedding of the current state and the text-based embedding of the target; executing a reinforcement learning model for a closed-loop autonomous driving task, where the similarity score is used as a reward in the reinforcement learning model; and optimizing the action policy of the reinforcement learning model based on the similarity score used as the reward, where the action policy is associated with a control command of the autonomous vehicle.

[0006] In another embodiment, a system is provided for optimizing an action policy of a machine learning model for an autonomous vehicle. The system includes one or more image sensors mounted to the autonomous vehicle and configured to generate images of the exterior of the autonomous vehicle representing the current state of the autonomous vehicle. The system includes one or more processors communicatively coupled to the one or more image sensors. The one or more sensors are programmed to perform the following: receive the generated images from the one or more image sensors; perform an image encoder on the generated images to generate an image-based embedding of the current state of the vehicle; receive a text prompt representing a goal of the autonomous vehicle; perform a text encoder on the text prompt to generate a text-based embedding of the goal; determine a similarity score representing the similarity between the image-based embedding of the current state and the text-based embedding of the goal; execute a reinforcement learning model for a closed-loop autonomous driving task, wherein the similarity score is used as a reward in the reinforcement learning model; and optimize the action policy of the reinforcement learning model based on the similarity score used as the reward, wherein the action policy is associated with control commands of the autonomous vehicle.

[0007] In another embodiment, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, cause the processors to perform any or all of the above steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 Illustrates a system for training a neural network according to an embodiment.

[0009] Figure 2 Illustrates a computer-implemented method for training and utilizing a neural network according to an embodiment.

[0010] Figure 3 Illustrates a schematic diagram of a control system configured to control a vehicle, which can be a partially autonomous vehicle, a fully autonomous vehicle, a partially autonomous robot, or a fully autonomous robot.

[0011] Figure 4 Illustrates a schematic diagram of a goal-directed reinforcement learning system enhanced by a base model according to an embodiment, wherein the output of the base model similarity score is used as a reward for reinforcement learning.

[0012] Figure 5 Illustrates a schematic diagram of using a base model (e.g., a vision-language model) within a reinforcement learning model scenario according to an embodiment.

[0013] Figure 6 Illustrates a schematic diagram of a contrastive learning model according to an embodiment.

[0014] Figure 7 Illustrated is a method for optimizing an action policy of a machine learning model for an autonomous vehicle according to an embodiment. Detailed implementation

[0015] Embodiments of the present disclosure are described herein. However, it is to be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The figures are not necessarily drawn to scale; some features may be enlarged or reduced to show details of particular components. Thus, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching one skilled in the art to employ the embodiments in various ways. As will be understood by one of ordinary skill in the art, the various features illustrated and described with reference to any one of the figures may be combined with features illustrated in one or more other figures to produce embodiments that are not explicitly illustrated or described. Combinations of the illustrated features provide representative embodiments for typical applications. However, for a particular application or implementation, various combinations and modifications of the features consistent with the teachings of the present disclosure may be desirable.

[0016] Unless the context clearly dictates otherwise, the articles "a," "an," and "the" as used herein refer to both singular and plural referents. For example, a "processor" programmed to perform various functions refers to one processor programmed to perform each function, or more than one processor programmed jointly to perform each of the various functions.

[0017] In the context of autonomous vehicles and machine learning (e.g., reinforcement learning), the terms "agent", "state", and "environment" are used. The term "agent" can refer to an entity that makes decisions and takes actions within an environment. This can include the autonomous vehicle itself, specifically its computational system that performs decision-making. The "environment" represents the external system or surrounding context with which the agent interacts. It includes things that the agent cannot control but can sense and affect through its actions. In autonomous driving, the environment can include roads, other vehicles, pedestrians, traffic signals, and any other relevant factors. The agent receives feedback and rewards from the environment based on its actions. The "state" is a representation of the current situation or configuration of the environment that the agent observes. It contains the relevant information that the agent needs to make decisions. In the context of autonomous driving, the state can include the position and speed of nearby vehicles, the current position of the autonomous vehicle, traffic conditions, and other relevant details. In the context of an autonomous vehicle navigating through traffic, the agent (autonomous vehicle) continuously senses the environmental state (traffic conditions, positions of other vehicles, etc.), decides on actions (such as steering, accelerating, and braking), and receives feedback from the environment in the form of rewards or punishments based on the consequences of its actions. In this context, reinforcement learning allows the agent to learn a strategy that maximizes the cumulative reward over time, leading to safe and efficient driving behavior.

[0018] Reinforcement learning seeks an optimal policy for sequential decision-making processes, making it an ideal tool for solving closed-loop tasks in the field of autonomous driving. The main challenge in effectively implementing reinforcement learning algorithms lies in the complex task of crafting a dense and well-shaped reward function. Sparse reward functions that encourage task completion or discourage failure are easy to specify, but they only work for simple tasks such as highway lane following. When faced with complex driving scenarios, the learning agent will get stuck at local optimal results because sparse reward functions do not incentivize exploration, and thus the learned action policy / strategy is suboptimal. To address this problem and define a good reward function, previous work has mainly focused on hand-engineering reward functions based on traffic rules. The hand-engineering approach carefully considers good reward functions that work well for individual tasks (e.g., parking, highway driving, etc.). However, it requires extensive expert knowledge to define the reward function, such that the learning agent cannot illegally obtain rewards without learning how to perform the task. Additionally, such reward functions lack universality across different tasks. Therefore, simple and general reward design techniques are necessary for non-expert users.

[0019] The latest developments in the foundation models have provided a brand-new direction for the design of the reward functions required in reinforcement learning. However, there has been no existing work on applying the foundation models to the reward design for autonomous driving so far. This is due to several reasons. For example, the current industrial research focus is mainly on open-loop tasks. Similarly, simulators like CARLA provide all the necessary ground-truth information, which usually leads to handcrafted reward designs. Since the foundation models can provide simple and general reward design techniques, it has the potential to become a promising and popular research area for solving closed-loop autonomous driving tasks.

[0020] This disclosure presents a novel method for using foundation models in the reward function definition process for autonomous driving. According to an embodiment, first, the foundation model is utilized to obtain visual state observations and target state embeddings. Then, the cosine distance between these two embeddings is determined as the reward value. Note that in the embodiment, the cosine distance rather than the cosine similarity is used here, which gives the ability to capture abstract language concepts.

[0021] Machine learning and neural networks are essential components of the inventions disclosed herein. Figure 1 A system 100 for training a neural network (e.g., a deep neural network) is shown. The system 100 may include an input interface for accessing training data 102 for the neural network. For example, as Figure 1 illustrated, the input interface may consist of a data storage interface 104, which can access the training data 102 from a data storage device 106. For example, the data storage interface 104 may be a memory interface or a persistent storage interface (such as a hard disk or SSD interface), but it can also be a personal area network, local area network, or wide area network interface, such as a Bluetooth, Zigbee, or Wi-Fi interface or an Ethernet or fiber optic interface. The data storage device 106 may be an internal data storage device of the system 100 (such as a hard disk drive or SSD), but it can also be an external data storage device, such as a network-accessible data storage device.

[0022] In some embodiments, the data storage device 106 may further include an untrained version of the data representation 108 of the neural network, which can be accessed by the system 100 from the data storage device 106. However, it will be appreciated that the training data 102 and the data representation 108 of the untrained neural network may also be accessed from different data storage devices, e.g., via different subsystems of the data storage interface 104. Each subsystem may be of the type of the data storage interface 104 as described above. In other embodiments, the data representation 108 of the untrained neural network may be internally generated by the system 100 based on the design parameters for the neural network, and thus the data representation 108 may not be explicitly stored on the data storage device 106.

[0023] The system 100 may further include a processor subsystem 110, which may be configured to provide, during operation of the system 100, an iterative function as a replacement for the stacked layers of the neural network to be trained. Here, the corresponding layers in the replaced stacked layers may have weights shared with each other and may receive the output of the previous layer as input, or for the first layer of the stacked layers, receive the initial activation and a portion of the input to the stacked layers. The processor subsystem 110 may also be configured to iteratively train the neural network using the training data 102. Here, the training iterations by the processor subsystem 110 may include a forward propagation portion and a backward propagation portion. The processor subsystem 110 may be configured to perform the forward propagation portion by determining an equilibrium point of the iterative function among other operations defining the executable forward propagation portion, at which the iterative function converges to a fixed point, where determining the equilibrium point includes using a numerical root-finding algorithm to find the root solution of the iterative function minus its input, and by providing the equilibrium point as a replacement for the output of the stacked layers in the neural network. The system 100 may further include an output interface for outputting the data representation 112 of the trained neural network; this data may also be referred to as the trained model data 112. For example, as also Figure 1 illustrated, the output interface may be constituted by the data storage interface 104, and in these embodiments, the interface is an input / output (“IO”) interface via which the trained model data 112 may be stored in the data storage device 106. For example, the data representation 108 defining the “untrained” neural network may be at least partially replaced by the data representation 112 of the trained neural network during or after training, since the parameters of the neural network (such as the weights, hyperparameters, and other types of parameters of the neural network) may be adapted to reflect the training on the training data 102. This is also Figure 1In FIG. 108, 112 are illustrated by reference numerals, and reference numerals 108, 112 refer to the same data record on the data storage device 106. In other embodiments, the data representation 112 may be stored separately from the data representation 108 that defines the "untrained" neural network. In some embodiments, the output interface may be separated from the data storage interface 104, but generally may be of the type described above for the data storage interface 104.

[0024] Figure 1 The system 100 shown in FIG. 1 is an example of a system that can be used to train the machine learning models described herein.

[0025] Figure 2 FIG. 2 depicts a system 200 that implements the machine learning models described herein (e.g., a base model and a reinforcement learning model having a reward based on the base model). The system 200 may include at least one computing system 202. The computing system 202 may include at least one processor 204, and the processor 204 is operably connected to the memory unit 208. The processor 204 may include one or more integrated circuits that implement the functions of a central processing unit (CPU) 206. The CPU 206 may be a commercially available processing unit that implements an instruction set, such as one of the x86, ARM, Power, or MIPS instruction set families. During operation, the CPU 206 may execute stored program instructions retrieved from the memory unit 208. The stored program instructions may include software that controls the operation of the CPU 206 to perform the operations described herein. In some examples, the processor 204 may be a system-on-chip (SoC) that integrates the functions of the CPU 206, the memory unit 208, a network interface, and an input / output interface into a single integrated device. The computing system 202 may implement an operating system for managing various aspects of the operation. Although one processor 204, one CPU 206, and one memory 208 are shown in FIG. 2, of course, more than one of each of the above may be utilized throughout the system. Figure 2 In FIG. 2, one processor 204, one CPU 206, and one memory 208 are shown, but of course, more than one of each of the above may be utilized throughout the system.

[0026] The memory unit 208 may include volatile and non-volatile memory for storing instructions and data. The non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 202 is deactivated or loses power. The volatile memory may include static and dynamic random access memory (RAM) for storing program instructions and data. For example, the memory unit 208 may store the machine learning model 210 or algorithm, the training data set 212 of the machine learning model 210, and the original source data set 216.

[0027] The computing system 202 may include a network interface device 222 configured to provide communication with external systems and devices. For example, the network interface device 222 may include a wired and / or wireless Ethernet interface defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards family. The network interface device 222 may include a cellular communication interface for communicating with a cellular network (e.g., 3G, 4G, 5G). The network interface device 222 may also be configured to provide a communication interface to an external network 224 or the cloud.

[0028] The external network 224 may be referred to as the World Wide Web or the Internet. The external network 224 may establish standard communication protocols between computing devices. The external network 224 may allow for the easy exchange of information and data between computing devices and the network. One or more servers 230 may communicate with the external network 224.

[0029] The computing system 202 may include an input / output (I / O) interface 220 configured to provide digital and / or analog input and output. The I / O interface 220 is used to transfer information between an internal storage device and external input and / or output devices (e.g., HMI devices). The I / O 220 interface may include associated circuitry or a bus network to transfer information between the (one or more) processors and the storage device. For example, the I / O interface 220 may include digital I / O logic lines that may be read or set by the (one or more) processors, handshake lines to monitor data transfer via the I / O lines, timing and counting facilities, and other structures known to provide such functionality. Examples of input devices include keyboards, mice, sensors, touchscreens, etc. Examples of output devices include monitors, touchscreens, speakers, head-up displays, vehicle control systems, etc. The I / O interface 220 may include additional serial interfaces for communicating with external devices (e.g., Universal Serial Bus (USB) interfaces). The I / O interface 220 may be referred to as an input interface (since it transfers data from external inputs such as sensors) or an output interface (since it transfers data to external outputs such as displays).

[0030] The computing system 202 may include a human-machine interface (HMI) device 218, and the human-machine interface (HMI) device 218 may include any device that enables the system 200 to receive control inputs. The computing system 202 may include a display device 232. The computing system 202 may include hardware and software for outputting graphical and text information to the display device 232. The display device 232 may include an electronic display screen, a projector, speakers, or other suitable devices for displaying information to a user or operator. The computing system 202 may also be configured to allow interaction with remote HMIs and remote display devices via the network interface device 222.

[0031] System 200 can be implemented using one or more computing systems. Although the example depicts a single computing system 202 implementing all the described features, it is intended that the various features and functions can be separated and implemented by multiple computing units that communicate with each other. The particular system architecture chosen may depend on a variety of factors.

[0032] System 200 can implement a machine learning algorithm 210 configured to analyze a raw source data set 216. The raw source data set 216 can include raw or unprocessed sensor data, which can represent an input data set for a machine learning system. The raw source data set 216 can include video, video clips, images, text-based information, audio or human speech, time series data (e.g., a pressure sensor signal that varies over time), and raw or partially processed sensor data (e.g., a radar map of an object). In some examples, the machine learning algorithm 210 can be a neural network algorithm (e.g., a deep neural network) designed to perform a predetermined function. For example, a neural network algorithm can be configured in an automotive application to identify street signs or pedestrians in an image. The (one or more) machine learning algorithms 210 can include algorithms configured to operate one or more of the machine learning models described herein (including the VLP base model).

[0033] The computing system 202 can store a training data set 212 for the machine learning algorithm 210. The training data set 212 can represent a previously constructed set of data for training the machine learning algorithm 210. The machine learning algorithm 210 can use the training data set 212 to learn the weighting factors associated with the neural network algorithm. The training data set 212 can include a source data set that has corresponding outcomes or results that the machine learning algorithm 210 attempts to replicate via the learning process. In this example, the training data set 212 can include input images that include objects (e.g., street signs). The input images can include various scenarios in which the objects are identified. The training data set 212 can also include text descriptions of the scenarios corresponding to the images detected by vehicle sensors (e.g., “A pedestrian is crossing the street”).

[0034] The machine learning algorithm 210 can operate in a learning mode using the training data set 212 as input. The machine learning algorithm 210 can use the data from the training data set 212 to perform multiple iterations. With each iteration, the machine learning algorithm 210 can update internal weighting factors based on the achieved results. For example, the machine learning algorithm 210 can compare the output results (e.g., a reconstructed or supplemented image in the case where image data is the input) with those included in the training data set 212. Since the training data set 212 includes the expected results, the machine learning algorithm 210 can determine when the performance is acceptable. After the machine learning algorithm 210 achieves a predetermined performance level (e.g., 100% consistent with the results associated with the training data set 212) or converges, the machine learning algorithm 210 can be executed using data not in the training data set 212. It should be understood that in the present disclosure, "convergence" can mean that a set (e.g., predetermined) number of iterations have occurred, or the residuals are small enough (e.g., the change in the approximate probability during the iteration process is less than a threshold), or other convergence conditions. The trained machine learning algorithm 210 can be applied to a new data set to generate annotated data. In the context of the reinforcement learning model described herein, a comparison is made between the commanded action A t of an autonomous vehicle and the reward based on the current state of the vehicle during or after commanding the action, and the model can be trained with an optimizer to reduce this loss (e.g., increase the reward), which can lead to convergence.

[0035] The machine learning algorithm 210 can be configured to identify specific features in the original source data 216. The original source data 216 can include multiple instances or input data sets for which supplementary results are desired. For example, the machine learning algorithm 210 can be configured to identify the presence of other objects (e.g., other cars, pedestrians, etc.) in a video image, annotate occurring events, and / or command the vehicle to take specific actions (planning) based on the location data (perception) of the detected objects and the predicted future movement / position (prediction) of the objects. The machine learning algorithm 210 can be programmed to process the original source data 216 to identify the presence of specific features. The machine learning algorithm 210 can be configured to identify the features in the original source data 216 as predetermined features (e.g., road signs, pedestrians, etc.). The original source data 216 can be derived from a variety of sources. For example, the original source data 216 can be actual input data collected by a machine learning system. The original source data 216 can be machine-generated for testing the system. As an example, the original source data 216 can include raw video images from a camera.

[0036] Figure 3FIG. depicts a schematic diagram of a control system 302 configured to control a vehicle 300, which may be a partially autonomous vehicle or a fully autonomous vehicle, a partially autonomous robot or a fully autonomous robot. The vehicle 300 and / or its control system 302 may incorporate one or more components of system 200 (such as computing system 202) to command an actuator 304 to perform a specific action based on processed readings from one or more sensors 306. For example, the control system 302 may be configured to utilize the planning model disclosed herein to control the movement of the vehicle via the actuator 304, where the planning model is trained via an optimizer.

[0037] One or more sensors 306 may include one or more image sensors (e.g., cameras, video sensors, radar sensors, ultrasonic sensors, lidar sensors), and / or position sensors (e.g., GPS). The sensors 306 may be configured to generate raw source data 216 indicative of the current state and / or environment associated with the vehicle. One or more of the one or more specific sensors may be integrated into the vehicle 300. In the context of agent identification and processing as described herein, the sensor 306 is a camera mounted to or integrated into the vehicle 300. Alternatively or additionally to the one or more specific sensors identified above, the sensor 306 may include a software module configured to determine the state of the actuator 304 when executed. Data generated from these sensors may be fused or otherwise combined to create a bird's-eye view (BEV) that provides spatio-temporal information associated with the vehicle and agents detected in the environment.

[0038] In embodiments where the vehicle 300 is a fully or partially autonomous vehicle, the actuator 304 may be embodied in the brakes, accelerator, propulsion system, engine, transmission system, or steering system (e.g., steering wheel) of the vehicle 300. For example, an actuator control command may be determined such that the actuator 304 is controlled so that the vehicle 300 avoids a collision with a detected agent. The detected agents may also be classified according to what the classifier deems these agents most likely to be (such as pedestrians or trees). The actuator control command may be determined based on the classification.

[0039] In other embodiments where the vehicle 300 is a fully or partially autonomous robot, the vehicle 300 may be a mobile robot configured to perform one or more functions via the actuator 304, such as flying, swimming, diving, and walking. The mobile robot may be an at least partially autonomous lawn mower or an at least partially autonomous cleaning robot. In such embodiments, an actuator control command may be determined such that the propulsion unit, steering unit, and / or braking unit of the mobile robot can be controlled so that the mobile robot can avoid a collision with the identified object.

[0040] As presented above, the present disclosure relates to the use of a base model in autonomous driving for the reward function definition process. In an embodiment, a base model enhanced goal-directed reinforcement learning (RL) architecture for closed-loop autonomous driving tasks is disclosed herein. Figure 4 FIG. illustrates a schematic overview of a base model enhanced goal-directed reinforcement learning system 400 according to an embodiment. The system 400 includes two main parts: a reward subsystem 402 that outputs a reward 404 based on the base model, and a reinforcement learning (RL) subsystem 406 that performs RL based on the reward 404 output by the base model 402 for training the autonomous control of a vehicle.

[0041] Generally, and as will be discussed in more detail herein, the current state of the vehicle is determined, for example, via one or more sensors 306 and associated processors and software. Associated images representing the images are generated by the one or more sensors 306. The determined state is used as an input to one or more vision-language models (VLMs) 408, where an image-based encoder 410 associated with the VLM is configured to generate an image-based embedding of the current state of the autonomous vehicle. At the same time, a text prompt related to a specific goal is provided to a large language model (LLM) 412, where a text-based encoder 414 associated with the LLM is configured to generate a text-based embedding associated with the goal. It is noted that, as will be explained below, in an embodiment, the goal is a negative goal (e.g., "collision") because it is easier to specify and more objective than a positive goal ("avoid collision"). These embeddings can be vectorized such that each encoder 410, 414 generates an associated vector representing the generated image in a learned embedding space, and the base model is configured to derive a similarity score 416, for example, on a scale between 0 and 1. Then, this similarity score is used as the reward 414 in the RL system 406, and the RL system 406 takes actions based on the state and the environmental reward by an agent (e.g., the autonomous vehicle 300).

[0042] Goal-directed reinforcement learning represents an RL algorithm that enables an agent (e.g., an autonomous vehicle) to learn an optimal action policy through rewards in order to achieve a specified goal. In the base model enhanced goal-directed reinforcement learning architecture disclosed herein, the reward function is defined as a function that quantifies the similarity between the current state of the agent and the goal. The enhanced base model encodes the current state and the goal of the agent such that the cosine similarity between them can be calculated and this cosine similarity is regarded as the reward value. In this way, the agent can obtain step-by-step rewards that guide it towards the goal state, which also promotes the reinforcement learning process compared to a sparse reward setting where the agent only receives a single feedback at the end of one trial.

[0043] In the autonomous driving scenario, although the current state of the agent can be easily represented by its sensory inputs (such as the images captured by its on-board cameras) and text descriptions, it is difficult to directly specify the target state. Specifically, the goal for autonomous driving is to drive safely, which is an abstract state. Establishing such an abstract concept based on detailed state representations, for example, image or language descriptions that describe the desired driving speed and following distance, etc., requires non-trivial manual formulation efforts, which makes it impractical and non-generalizable across different scenarios and tasks. On the contrary, it is much easier to simply describe the unwanted states using text, such as "self-collision accident". This linguistic opposite goal is specified to the agent, and a negative similarity reward is defined for the agent to move away from the opposite goal, rather than motivating the agent to reach the goal.

[0044] Given the agent state and the target description, the reward function r can be determined as follows:

[0045]

[0046] where FM 状态 and FM 目标 represent the base model encoders 410 and 414 for the current state and the target state, respectively.

[0047] To implement the reward function r in the above equation (1), three different methods are disclosed in this paper: text-to-image embedding (such as Figure 4 illustrated and described above), image-to-image embedding, and text-to-text embedding.

[0048] For the implementation of text-to-image embedding, text is used to describe the target state (e.g., "do not hit another vehicle"), and the images generated by the on-board vehicle cameras are used to represent the current state of the agent. Figure 5 An example of the pipeline of this implementation is shown in

[0049]

[0050] The reference to "CLIP" refers to the Contrastive Language-Image Pretraining (CLIP) model. CLIP was developed by OpenAI. It is designed to understand and connect images and natural language descriptions in a way that allows it to perform a wide range of visual and language tasks. In Figure 6The contrastive learning concept used in CLIP is shown. CLIP adopts a dual-encoder architecture, including a visual encoder, a text encoder, and a shared embedding space. The visual encoder processes images, while the text encoder processes natural language descriptions. The visual encoder is based on a visual model (such as a convolutional neural network (CNN)) and converts images into fixed-length vector representations. The text encoder processes these text descriptions by converting them into fixed-length vector representations. CLIP is a vision-language foundation model trained on open-world data using contrastive learning. Contrastive learning is a type of machine learning in which the model learns to distinguish between positive and negative data pairs. In the context of CLIP, "positive pairs" consist of semantically related images and text descriptions, while "negative pairs" consist of unrelated images and randomly selected text descriptions. During training, CLIP is designed to encourage the clustering of features from related text and image pairs into a common embedding space while pushing unrelated pairs apart. Although contrastive learning is not used to derive the similarity score r because dissimilar features are not pushed apart, the concept of using encoders to transform image data and natural language into a shared embedding space for similarity analysis is used in this paper.

[0051] Reference Figure 6 In the example embodied in, multiple images (in this example, one of the multiple images is an image of a tiger) are fed into the image encoder, and multiple text phrases (in this example, one of the multiple text phrases is something like "a photo of a tiger") are fed into the text encoder. In CLIP, several unrelated or dissimilar text phrases and images are also fed into the encoder for contrastive learning, although this is not required in the current disclosure. The image encoder produces image vectors with features I1, I2, …, I N , while the text encoder produces text vectors with features T1, T2, …, T N . The diagonal of the matrix produced according to this dot product shows the paired images and text according to the possible similarity of the paired images and text, while the off-diagonal represents unpaired image and text features.

[0052] Similarly, according to an embodiment of the present disclosure, an image representing the current state is fed into the text-based encoder 410, and a text-based prompt representing the target state (e.g., stated in a negative target format) is fed into the text-based encoder 414. The image encoder produces image vectors with features I1, I2, …, I N , while the text encoder produces vectors with language-based features L1, L2, …, L N . Then, the cosine distance between the two embeddings is determined according to Equation (2) above, where CLIP I and CLIPT respectively refer to the embeddings determined by the image-based encoder and the text-based encoder.

[0053] Then, this cosine distance is used as the reward R for the reinforcement learning model t . Referring back to Figure 5 , a high-level overview of the role of the vision-language model (VLM) in the reinforcement learning pipeline is illustrated. The goal expressed in natural language and the current state expressed with the aid of image data are input into the base model, and the similarity between the vector-based embeddings is derived and used as the reward in reinforcement learning. In particular, the agent utilizes the reward r t and the current state s t to perform a new action a t . As the action is taken, the environment is determined via the vehicle's sensors, and a new state s t+1 is determined after the action is executed, and this new state s t+1 is again used by the VLM to determine a new reward. The loop continues, and the vehicle increasingly learns which actions result in greater rewards (e.g., pushing r closer to 1 and farther from 0 when on a scale from 0 to 1).

[0054] In other embodiments, image-to-image embeddings are used instead of text-to-image embeddings. For the implementation of image-to-image embeddings, the target state is represented as an image rather than natural language text. The difference between this method and text-to-image embeddings is that here two image encoders (e.g., from the CLIP model) are utilized and these two image encoders are used to derive the embeddings for similarity calculation.

[0055] In some cases, it is difficult to capture the differences between the various states of the agent solely through images. For example, in two similar images, the speed of the vehicle may be very different. Therefore, in other embodiments, text-to-text embeddings are used instead of image-to-image embeddings or text-to-image embeddings. Here, the system uses the linguistic description of the generated image as the state of the agent. For some scenarios, the system can use a template-based linguistic description to describe detailed information about the surrounding vehicles, including position, speed, and collision time, etc. For more complex scenarios, the system can use a large vision-language model (LLM) to generate a free-form description of the image. The system can also use a large language base model (e.g., SentenceBERT) to encode the text-based state description and the target description. The cosine similarity of the two encodings is calculated, and equation (1) above is adopted to define the reward function.

[0056] Figure 7FIG. illustrates a method 700 for optimizing an action policy of a machine learning model for an autonomous vehicle. The method may be executed by one or more computing systems and processors disclosed herein (such as system 100, system 200, computing system 202, etc.).

[0057] At 702, one or more images of the vehicle's surrounding environment are generated. The images are generated based on vehicle sensors and / or accompanying data generated by the sensors (such as sensor 306, e.g., a camera). The images may include pedestrians, other vehicles, lane lines, traffic signals, etc. These images and / or accompanying data depict or represent the current state of the autonomous vehicle.

[0058] At 704, these images are passed through an image encoder (performed by the image encoder) to generate an image-based embedding of the current state of the autonomous vehicle. For example, the image encoder may be from a CLIP model, in which a vector with, for example, features I1, I2, ……, I N is generated.

[0059] At 706, a text prompt is generated and received, where the text prompt represents the goal (or inverse goal) of the autonomous vehicle. For example, the goal may be "collide with another vehicle" or "drive over the speed limit" or the like. At 708, these text prompts are passed through a text encoder (performed by the text encoder) to generate a text-based embedding of the goal. For example, the text encoder may be from a CLIP model, in which a text vector with features T1, T2, ……, T N is generated. As explained above, the "goal" used may actually be an inverse goal because some goals are easier to objectively process in the negative form than in the positive form. For example, "do not collide with a vehicle" may be ambiguous because in infinite space, a vehicle can exist without colliding with another vehicle. Thus, the goal may be "collide with a vehicle" which can be easily determined, and the inverse result of this goal may be used during the determination of the reward (e.g., see equation (1) above).

[0060] At 710, a similarity score is determined. The similarity score represents the similarity between the image-based embedding of the current state and the text-based embedding of the goal. For example, the similarity score can be obtained using equation (1) above, and this similarity score can be used as the reward r in a reinforcement learning model. Cosine similarity can be used to determine the similarity score. In some embodiments, the dot product function can measure the similarity between two vectors.

[0061] At 712, a reinforcement learning model for a closed-loop autonomous driving task (e.g., highway driving, parking, urban driving, etc.) is executed. Here, the determined similarity score from 710 is used as a reward in the reinforcement learning model loop, such as those shown and described above Figures 4 to 5 as shown. At 714, the action policy of the reinforcement learning model is optimized based on the iteration of the reinforcement learning model that uses the similarity score as a reward during the iteration. The action policy is associated with control commands of the autonomous vehicle, such as steering, acceleration, braking, lane change, and so on. When the reward is getting larger (e.g., when on a scale from 0 to 1, closer to 1 rather than 0), the reinforcement learning policy is optimized through iteration.

[0062] While the exemplary embodiments have been described above, it is not intended that these embodiments describe all possible forms covered by the claims. The words used in the specification are descriptive words rather than restrictive, and it is to be understood that various changes can be made without departing from the spirit and scope of the present disclosure. As previously mentioned, the features of the various embodiments can be combined to form additional embodiments of the invention that may not be explicitly described or illustrated. While the various embodiments may have been described as providing advantages or being preferred over other embodiments or prior art implementations with respect to one or more desired characteristics, those of ordinary skill in the art will recognize that one or more features or characteristics may be compromised to achieve the desired overall system attributes, depending on the particular application and implementation. These attributes can include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, suitability, weight, manufacturability, ease of assembly, etc. Thus, to the extent that any embodiment is described as less desirable with respect to one or more features than other embodiments or prior art implementations, these embodiments are not outside the scope of the present disclosure and may be desirable for a particular application.

Claims

1. A method for optimizing an action policy of a machine learning model for an autonomous vehicle, the method comprising: Generating an image of the environment of the autonomous vehicle based on vehicle sensor data representing the current state of the autonomous vehicle; Passing the generated image through an image encoder to generate an image-based embedding of the current state of the autonomous vehicle; Receiving a text prompt representing the goal of the autonomous vehicle; Passing the text prompt through a text encoder to generate a text-based embedding of the goal; Determining a similarity score representing the similarity between the image-based embedding of the current state and the text-based embedding of the goal; Executing a reinforcement learning model for a closed-loop autonomous driving task, wherein the similarity score is used as a reward in the reinforcement learning model; And Optimizing the action policy of the reinforcement learning model based on the similarity score used as the reward, wherein the action policy is associated with control commands of the autonomous vehicle.

2. The method according to claim 1, further comprising: Executing a base model to perform the determination of the similarity score.

3. The method according to claim 2, wherein the similarity score is determined as follows: where r represents the reward utilized in the reinforcement learning model, FM 状态 represents the image-based embedding of the current state of the autonomous vehicle, and FM 目标 represents the text-based embedding of the target.

4. The method according to claim 1, wherein the text prompt is a manually crafted text prompt and is not generated by a machine learning model.

5. The method according to claim 1, wherein the determination of the similarity score includes obtaining an inverse result of the similarity between the image-based embedding of the current state and the text-based embedding of the goal.

6. The method according to claim 1, wherein the image encoder is part of a vision-language model (VLM) configured to generate a vector representing the generated image in a learned embedding space.

7. The method according to claim 6, wherein the text encoder is part of a large language model (LLM) configured to generate a vector representing the goal in a learned embedding space.

8. A system for optimizing an action policy of a machine learning model for an autonomous vehicle, the system comprising: One or more image sensors mounted to the autonomous vehicle and configured to generate images outside the autonomous vehicle representing the current state of the autonomous vehicle; And One or more processors communicatively coupled to the one or more image sensors, the one or more processors being programmed to: Receive the generated images from the one or more image sensors, Execute an image encoder on the generated images to generate an image-based embedding of the current state of the vehicle, Receive a text prompt representing the goal of the autonomous vehicle, Execute a text encoder on the text prompt to generate a text-based embedding of the goal, Determine a similarity score representing the similarity between the image-based embedding of the current state and the text-based embedding of the goal, Execute a reinforcement learning model for closed-loop autonomous driving tasks, where the similarity score is used as a reward in the reinforcement learning model, and Optimize the action policy of the reinforcement learning model based on the similarity score used as the reward, where the action policy is associated with the control commands of the autonomous vehicle.

9. The system according to claim 8, wherein the one or more processors are further programmed to: Execute a base model to perform the determination of the similarity score.

10. The system according to claim 9, wherein the similarity score is determined as follows: where r represents the reward utilized in the reinforcement learning model, FM 状态 represents the image-based embedding of the current state of the autonomous vehicle, and FM 目标 represents the text-based embedding of the target.

11. The system according to claim 8, wherein the text prompt is a manually crafted text prompt and is not generated by a machine learning model.

12. The system according to claim 8, wherein the determination of the similarity score includes obtaining an inverse result of the similarity between the image-based embedding of the current state and the text-based embedding of the target.

13. The system according to claim 8, wherein the image encoder is part of a vision-language model (VLM), and the image encoder is configured to generate a vector representing the generated image in a learned embedding space.

14. The system according to claim 13, wherein the text encoder is part of a large language model (LLM), and the text encoder is configured to generate a vector representing the target in a learned embedding space.

15. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the following actions, including: Generate an image of the environment of the autonomous vehicle based on vehicle sensor data representing the current state of the autonomous vehicle; Pass the generated image through an image encoder to generate an image-based embedding of the current state of the autonomous vehicle; Receive a text prompt representing the target of the autonomous vehicle; Pass the text prompt through a text encoder to generate a text-based embedding of the target; Determine a similarity score representing the similarity between the image-based embedding of the current state and the text-based embedding of the target; Execute a reinforcement learning model for closed-loop autonomous driving tasks, where the similarity score is used as a reward in the reinforcement learning model; And Optimize the action policy of the reinforcement learning model based on the similarity score used as the reward, where the action policy is associated with the control commands of the autonomous vehicle.

16. The non-transitory computer-readable storage medium according to claim 15, wherein when the instructions are executed by the one or more processors, the instructions cause the one or more processors to perform additional actions, including: Execute a base model to perform the determination of the similarity score.

17. The non-transitory computer-readable storage medium according to claim 16, wherein the similarity score is determined as follows: where r represents the reward utilized in the reinforcement learning model, FM 状态 represents the image-based embedding of the current state of the autonomous vehicle, and FM 目标 represents the text-based embedding of the target.

18. The non-volatile computer-readable storage medium according to claim 15, wherein the text prompt is a manually crafted text prompt and is not generated by a machine learning model.

19. The non-volatile computer-readable storage medium according to claim 15, wherein the determination of the similarity score includes obtaining a reverse result of the similarity between the image-based embedding of the current state and the text-based embedding of the target.

20. The non-volatile computer-readable storage medium according to claim 15, wherein the image encoder is part of a vision-language model (VLM), and the vision-language model is configured to generate a vector representing the generated image in the learned embedding space.

Images