Reward function for a vehicle

Abstract

Examples described herein provide a computer-implemented method that includes receiving, by a processing device, a current state of a vehicle. The method also includes predicting, by the processing device, a future state of the vehicle based at least in part on the current state of the vehicle using an output of an artificial intelligence model. The method also includes calculating, by the processing device, a reward associated with the future state of the vehicle using an adjustable reward function, the adjustable reward function including a plurality of adjustable coefficients. The method also includes training, by the processing device, the artificial intelligence model based at least in part on the reward.

Description

Technical Field

[0001] This disclosure relates to vehicles and more specifically to reward functions for vehicles. Background Technology

[0002] Modern vehicles (e.g., cars, motorcycles, boats, or any other type of automobile) can be equipped with vehicle communication systems that facilitate different types of communication between vehicles and other entities. For example, vehicle communication systems can provide vehicle-to-infrastructure (V2I), vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), and / or vehicle-to-grid (V2G) communication. In general, these can be referred to as vehicle-to-everything (V2X) communication, which enables information to be communicated from the vehicle to any other suitable entity. Various applications (e.g., V2X applications) can use V2X communication to send and / or receive safety messages, maintenance messages, vehicle status messages, and so on.

[0003] Modern vehicles may also include one or more cameras that provide reversing assistance, capture images of the driver to determine driver drowsiness or attention, provide road images while the vehicle is in motion for collision avoidance purposes, and provide structural recognition, such as road signs. For example, a vehicle may be equipped with multiple cameras and can use images from multiple cameras (called “surround-view cameras”) to create a “surround” or “bird’s-eye view” of the vehicle. Some cameras (called “remote cameras”) may be used to capture remote images (e.g., for object detection, structural recognition, etc., for collision avoidance).

[0004] Such vehicles can also be equipped with sensors, such as radar and LiDAR devices, for target tracking. Target tracking involves identifying a target object and tracking it over time as it moves relative to the vehicle observing it. Images from one or more cameras on the vehicle can also be used to perform target tracking.

[0005] These communication protocols, cameras, and / or sensors can be used for autonomous vehicle operation. Summary of the Invention

[0006] In one exemplary embodiment, a computer-implemented method is provided. The method includes receiving a current state of a vehicle by a processing device. The method further includes predicting a future state of the vehicle by the processing device using the output of an artificial intelligence model, at least in part based on the current state of the vehicle. The method also includes calculating a reward associated with the future state of the vehicle by the processing device using an adjustable reward function, the adjustable reward function including a plurality of adjustable coefficients. Finally, the method includes training the artificial intelligence model by the processing device at least in part based on the reward.

[0007] In another example, one of the adjustable factors is the adjustable factor for the magnitude of the vehicle's speed from the desired driving speed.

[0008] In another example, one of the adjustable factors is the adjustable factor that causes the vehicle's speed to decrease from the desired driving speed.

[0009] In another example, one of the adjustable factors is the adjustable factor for the magnitude of the jerk caused by the change in vehicle speed.

[0010] In another example, one of the adjustable coefficients is the adjustable discount factor.

[0011] In another example, one of the adjustable factors is the adjustable factor for the magnitude of a successful handover.

[0012] In another example, the adjustable reward function rewards both full success and partial success.

[0013] In another example, the adjustable reward function takes into account both current and future rewards.

[0014] In another example, the adjustable reward function rewards efficiency.

[0015] In another example, an adjustable reward function rewards comfort.

[0016] In another example, the adjustable reward function takes into account the tradeoff between efficiency and comfort.

[0017] In another example, the adjustable reward function is a function of the vehicle's current state s, action a, and the vehicle's future state s'.

[0018] In another example, the adjustable reward function is represented as:

[0019]

[0020] Where C speed It is the speed adjustment coefficient; v h It is the actual speed of the vehicle; v desired C is the vehicle's desired speed. temp It is the adjustable coefficient that controls the decrease in vehicle speed from the desired driving speed; C Jerk It is the adjustable coefficient of the jerk amplitude; J 2 It is the square of the jerk; γ is the adjustable discount factor; I is the indicator function; and C handoff It is an adjustable coefficient representing the extent of a successful handover.

[0021] In another example, the method also includes controlling the vehicle, at least in part, based on an artificial intelligence model.

[0022] In another example, the adjustable reward function supports driver-in-the-loop functionality.

[0023] In another exemplary embodiment, a system includes a memory having computer-readable instructions. The system also includes a processing means for executing the computer-readable instructions, which control the processing means to perform operations. The operations include receiving a current state of a vehicle. The operations further include using the output of an artificial intelligence model to predict a future state of the vehicle, at least in part based on the current state of the vehicle. The operations also include using an adjustable reward function to compute a reward associated with the future state of the vehicle, the adjustable reward function including a plurality of adjustable coefficients. The operations further include training the artificial intelligence model at least in part based on the reward.

[0024] In another example, the adjustable reward function is a function of the vehicle's current state s, action a, and the vehicle's future state s'.

[0025] In another example, the adjustable reward function is represented as:

[0026]

[0027] Where C speed It is the speed adjustment coefficient; v h It is the actual speed of the vehicle; v desired C is the vehicle's desired speed. temp It is an adjustable coefficient that controls the rate at which the vehicle speed decreases from the desired speed; C Jerk It is the adjustable coefficient of the jerk amplitude; J 2 It is the square of the jerk; γ is the adjustable discount factor; I is the index function; and C handoff It is an adjustable coefficient representing the extent of a successful handover.

[0028] In another example, the operation further includes controlling the vehicle based at least in part on an artificial intelligence model.

[0029] In yet another exemplary embodiment, a computer-readable storage medium is provided, comprising program instructions executable by a processor to cause the processor to perform operations. The operations include receiving a current state of a vehicle. The operations further include using the output of an artificial intelligence model to predict a future state of the vehicle, at least in part based on the current state of the vehicle. The operations also include using an adjustable reward function to compute a reward associated with the future state of the vehicle, the adjustable reward function including a plurality of adjustable coefficients. The operations further include training the artificial intelligence model at least in part based on the reward.

[0030] The above-described features and advantages, as well as other features and advantages of this disclosure, will become apparent when the following detailed description is taken in conjunction with the accompanying drawings. Attached Figure Description

[0031] Other features, advantages, and details appear only as examples in the following detailed description, which is described in detail with reference to the accompanying drawings, in which:

[0032] Figure 1 A vehicle including sensors and a processing system is depicted according to one or more embodiments described herein;

[0033] Figure 2 An architecture for supporting autonomous vehicle operation using an artificial intelligence model and an adjustable reward function, according to one or more embodiments described herein, is depicted.

[0034] Figure 3 A flowchart depicts a method for implementing a reward function for autonomous vehicle operation according to one or more embodiments described herein; and

[0035] Figure 4 A block diagram is depicted of a processing system for implementing the techniques described herein, according to exemplary embodiments. Detailed Implementation

[0036] The following description is exemplary in nature and is not intended to limit the invention or its application or use. It should be understood that in all the drawings, corresponding reference numerals denote the same or corresponding parts and features. As used herein, the term module refers to processing circuitry, which may include application-specific integrated circuits (ASICs), electronic circuitry, processors (shared, dedicated, or grouped) and memories executing one or more software or firmware programs, combinational logic circuitry, and / or other suitable components that provide the said functionality.

[0037] The technical solutions described herein provide an adjustable reward function for autonomous vehicle (AV) operation. Specifically, one or more embodiments described herein provide a robust and comprehensive design of the reward (or “cost”) function for driver-in-the-loop autonomous driving applications. “Driver-in-the-loop” refers to the ability of a driver (e.g., the vehicle operator) to gain control of the autonomous vehicle. Driver gaining control of the AV can occur either by the driver intentionally taking over control or by the AV relinquishing control to the driver (e.g., the AV knowing it can no longer control the vehicle well).

[0038] Autonomous driving agents controlling autonomous vehicles need to consistently evaluate different maneuvers and plan for safe, comfortable, and efficient driving. To this end, various planning methods optimize motion plans by evaluating potential maneuvers based on criteria of interest, so that a plan that maximizes reward (or minimizes cost) can be found or approximated. One or more embodiments described herein provide a comprehensive and general reward function design that considers driver-in-the-loop criteria and can be used by various optimization algorithms to generate suitable AV driving strategies for autonomous driving. The AV driving strategy specifies how the autonomous vehicle should operate. The AV driving strategy encodes and extracts information about the autonomous vehicle's environment relevant to the decision. This information is used to construct the behavior / motion plan to be performed by the autonomous vehicle. Information can be collected from the autonomous vehicle (referred to as the "target vehicle") and / or other surrounding vehicles (which may or may not be autonomous) (referred to as the "other vehicle" or "other vehicle").

[0039] One or more embodiments described herein address these and other shortcomings of the prior art by providing a reward function that reflects the expected behavior of the AV agent, said reward function incorporating criteria of efficiency, comfort, and task completion. The trade-offs between these three criteria can be adjusted / personalized. Furthermore, one or more embodiments described herein support rewarding the driver-in-the-loop system by increasing the reward function with partial task completion rewards. One or more embodiments described herein provide a method for designing and tuning a reward function for optimizing autonomous driving plans, suitable for driver-in-the-loop applications, and can be used by any utility-based method, such as online-search or learning methods (e.g., reinforcement learning), to generate optimized driving plans.

[0040] Figure 1 A vehicle 100 including a sensor and processing system 110 according to one or more embodiments described herein is depicted. Figure 1 In the example, vehicle 100 includes a processing system 110, cameras 120, 121, 122, 123, cameras 130, 131, 132, 133, radar sensor 140, and lidar sensor 141. Vehicle 100 may be a car, truck, van, bus, motorcycle, boat, aircraft, or other suitable vehicle 100.

[0041] Cameras 120-123 are panoramic cameras that capture images of the exterior and vicinity of vehicle 100. The images captured by cameras 120-123 together form a surround view (sometimes referred to as a "top view" or "bird's-eye view") of vehicle 100. These images are useful for maneuvering the vehicle (e.g., parking, reversing, etc.). Cameras 130-133 are remote cameras that capture images of the exterior of the vehicle and are located further away from vehicle 100 than cameras 120-123. These images can be used, for example, for object detection and avoidance. It should be understood that although eight cameras 120-123 and 130-133 are shown, more or fewer cameras may be implemented in various embodiments.

[0042] The captured images can be displayed on a monitor (not shown) to provide the driver / operator of vehicle 100 with an external view of vehicle 100. The captured images can be displayed as live images, still images, or some combination thereof. In some examples, the images can be combined to form a composite view, such as a surround view.

[0043] Radar sensor 140 measures the distance to a target object by emitting electromagnetic waves and measuring the reflected waves. This information is useful for determining the distance / position of the target object relative to vehicle 100.

[0044] The LiDAR (Light Detection and Ranging) sensor 141 measures the distance to a target object (e.g., another vehicle 154) by illuminating the target with a pulsed laser and measuring the reflected pulse with the sensor. This information is useful for determining the distance / position of the target object relative to vehicle 100.

[0045] Data generated from cameras 120-123, 130-133, radar sensor 140, and / or LiDAR sensor 141 can be used to detect and / or track target objects relative to vehicle 100. Examples of target objects include other vehicles (e.g., other vehicle 154), vulnerable road users (VRUs) such as pedestrians, bicycles, animals, potholes, oil on the road surface, debris on the road surface, fog, floods, etc.

[0046] Processing system 110 can be communicatively coupled to remote processing system 150, which may be an edge processing node as part of an edge processing environment, a cloud processing node as part of a cloud processing environment, etc. Processing system 110 can also be communicatively coupled to one or more other vehicles (e.g., other vehicle 154). In some examples, processing system 110 is directly communicatively coupled to processing system 150 and / or other vehicle 154 (e.g., using V2V communication), while in other examples, processing system 110 is indirectly communicatively coupled to processing system 150 and / or other vehicle 154, for example, via a network. For example, processing system 110 may include a network adapter (not shown) (see example...). Figure 4 (Network adapter 426). The network adapter enables the processing system 110 to transmit data to and / or receive data from other sources, such as other processing systems, data repositories, etc., including the remote processing system 150 and other vehicles 154. As an example, the processing system 110 may send data to and / or receive data from the remote processing system 150 directly and / or via network 152.

[0047] Network 152 represents any combination of suitable communication networks of one or different types, such as wired networks, public networks (e.g., the Internet), private networks, wireless networks, cellular networks, or any other suitable private and / or public networks. Furthermore, network 152 may have any suitable communication range associated with it and may include, for example, global networks (e.g., the Internet), metropolitan area networks (MANs), wide area networks (WANs), local area networks (LANs), or personal area networks (PANs). Additionally, network 152 may include any type of medium capable of carrying network traffic, including but not limited to coaxial cable, twisted pair, optical fiber, hybrid fiber-coaxial (HFC) media, microwave terrestrial transceivers, radio frequency communication media, satellite communication media, or any combination thereof. According to one or more embodiments described herein, remote processing system 150, another vehicle 154, and processing system 110 communicate via vehicle-to-infrastructure (V2I), vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), and / or vehicle-to-grid (V2G) communications.

[0048] This document further describes the characteristics and functionality of the components of processing system 110. Processing system 110 of vehicle 100 assists in decision-making within the context of autonomous vehicle operation, which may include the use of reward functions as described herein. Specifically, processing system 110 includes a data / communication engine 112, an autonomous agent engine 114, and a control engine 116. Data / communication engine 112 receives / collects data, such as data from sensors associated with vehicle 100, and / or receives data from other sources such as remote processing system 150 and / or other vehicles 154. Autonomous agent engine 114 acts as an intelligent agent performing operations with at least some degree of autonomy. For example, autonomous agent engine 114 autonomously operates vehicle 100 to achieve a goal. The goal may be navigation from location A to location B. Autonomous agent engine 114 may utilize machine learning capabilities to achieve the goal, as further described herein. Control engine 116 controls vehicle 100, such as performing driving maneuvers (e.g., changing lanes, changing speed, etc.). Although not shown, the processing system 110 may include other components, engines, modules, etc., such as processors (e.g., central processing units, graphics processing units, microprocessors, etc.), memory (e.g., random access memory, read-only memory, etc.), data storage (e.g., solid-state drives, hard disk drives, etc.), input and / or output devices (e.g., displays, touchpads, microphones, speakers, etc.).

[0049] Various aspects of this disclosure can utilize machine learning capabilities to accomplish the various operations described herein. More specifically, one or more embodiments described herein can combine and utilize rule-based decision-making and artificial intelligence (AI) reasoning to accomplish the various operations described herein. The term “machine learning” broadly describes the functionality of electronic systems that learn from data. Machine learning systems, modules, or engines (e.g., autonomous agent engine 114) can include trainable machine learning algorithms, which can be trained, for example, in an external cloud environment, to learn functional relationships between currently unknown inputs and outputs, and the resulting model can be used to determine the use of agent computer programs. In one or more embodiments, the machine learning functionality can be implemented using artificial neural networks (ANNs) that have the ability to be trained to perform currently unknown functions. In machine learning and cognitive science, ANNs are a family of statistical learning models inspired by the biological neural networks of animals, particularly the brain. ANNs can be used to estimate or approximate systems and functions that depend on a large number of inputs.

[0050] An ANN can be embodied as a so-called "neuromorphic" system of interconnected processor elements that act as analog "neurons" and exchange "messages" with each other in the form of electronic signals. Similar to the so-called "plasticity" of synaptic neurotransmitter connections that transmit information between biological neurons, the connections in an ANN that transmit electronic messages between analog neurons are given numerical weights corresponding to the strength of a given connection. These weights can be adjusted and tweaked empirically, allowing the ANN to adapt to inputs and learn. For example, an ANN for handwriting recognition is defined by a set of input neurons that can be activated by pixels of an input image. After being weighted and transformed by a function determined by the network designer, the activation of these input neurons is then passed on to other downstream neurons, often referred to as "hidden" neurons. This process is repeated until an output neuron is activated. The activated output neuron determines which character to read.

[0051] One type of machine learning is reinforcement learning, which deals with how an agent (e.g., autonomous agent engine 114) takes actions to achieve a desired goal while maximizing rewards. Reinforcement learning uses data in the form of state-action pairs, with the goal of maximizing future rewards over multiple time steps. In reinforcement learning, the agent (e.g., autonomous agent engine 114) is said to be in state s, which is the state the agent perceives. The agent takes action a. t Its agent can make movements in the environment at time t. When taking action a... t Afterwards, the environment undergoes a state change s at time t+1. t+1 The observations will be sent back to the agent in the form of [a specific format]. The agent will also receive a reward [amount]. t It measures the agent's action a at time t. t Feedback on success or failure. Reinforcement learning is useful for decision-making in autonomous vehicle operating environments.

[0052] One or more embodiments described herein provide a method for designing and tuning a reward function to optimize an autonomous driving plan generated using reinforcement learning. The proposed adjustable reward function is applicable to driver-in-the-loop applications and can be used by any utility-based method, such as online search or learning methods like reinforcement learning, to generate optimized driving plans.

[0053] A driving strategy can be considered acceptable when three success criteria are met: task completion, efficiency, and comfort / consistency. The task completion criterion is typically reflected in reaching a desired destination / condition. The efficiency criterion is completing the task in the fastest possible way within certain environmental constraints. The comfort / consistency criterion is based on decisiveness and the execution of a smooth and comfortable sequence of maneuvers in achieving its goals. To achieve these three criteria, the following adjustable reward function is proposed:

[0054]

[0055] Where C speed It is the speed adjustment coefficient; v h It is the actual speed of the vehicle; v desired C is the vehicle's desired speed. temp It is an adjustable coefficient that controls the rate at which the vehicle speed decreases from the desired speed; C Jerk It is an adjustable coefficient for the acceleration amplitude, which is the derivative of the acceleration (or instantaneous change), and it is the derivative of the vehicle's actual speed; J 2 γ is the square of the jerk, i.e., the instantaneous change in acceleration; γ is an adjustable discount factor; I is an index function (e.g., returning 0 when its predicate is false and 1 when it is true), whose predicate (condition) is user-defined and can be scenario-specific. success Corresponding to the success condition, I handoff Corresponding to the handover event; and C handoff It is the adjustable amplitude coefficient of the handover event.

[0056] The adjustable reward function is a function of the environment state *s*, the action *a*, and the future environment state *s'*. The first term is an "alive" dense reward (at every step), primarily used for standardization. The second term is a velocity deviation penalty term parameterized by multipliers and a temperature coefficient. The third term is an integral over the squared acceleration of the motion (and / or any derivative or artifact thereof). The fourth and fifth terms are task completion terms, rewarding full and partial (handover) success. It should be understood that the following components of the reward function are adjustable: C speed C temp C Jerk ,γ; and C handoff .

[0057] The adjustable reward function rewards both full and partial success. Suppose that under certain scenarios, the autonomous agent (e.g., Autonomous Agent Engine 114) may fail to achieve its objectives. Such failures typically result in high reward costs. To avoid this, the autonomous agent has the option to "hand over" driving control to a human operator / driver (e.g., in a driver-in-the-loop system such as L2 / 3 Advanced Driver Assistance Systems (ADAS)), or alternatively, the responsible human operator / driver may choose to actively overtake the autonomous agent to avoid undesirable outcomes. The adjustable reward function includes indicator components for rewarding "full success" or "partial success," and one or more partial success components to indicate the handover result (triggered by the driver or agent). For example, C handoff The coefficient balances availability with failure risk. In other words, higher availability will result in lower C. handoff As a result, a lower failure risk will be higher C.handoff The result. End users can define the logic behind the metric function to achieve flexibility, which may depend on the scenario.

[0058] Adjustable reward functions also consider when to reward: now or in the future. For example, in the planning domain, it is widely accepted to design discount reward functions using a discount factor γ. Adjustable reward functions take the discount factor γ and amplify its effect by using it for normalization of the adjustable reward function. The implicit assumption is that the future reward after success will be maximized, so negative numbers (the 2nd and 3rd reward terms) will be zero, and non-zero dense rewards will include a +1 "real-time" term. Following this assumption, and under the same γ discount regime, the "task completion" terms (the 4th and 5th) are amplified by a 1 / 1 - γ multiplier, which reflects the sum of an infinite future +1 geometric progression. With this, the same γ parameter can be reused to control the future of the discount, as well as greed and long-term planning, which are highly relevant planning concepts.

[0059] The adjustable reward function also rewards efficiency. The efficiency term (the second term) measures the deviation from certain desired driving speeds (e.g., maximum speed limits, driver feedback, etc.). For simplicity (and without loss of generality), it can be assumed that the driving speed is equal to or less than the desired speed, resulting in a term in the range [0,1] that penalizes deviations from the desired speed. The magnitude and decay of this term are determined by the coefficient C. speed and C temp Parameterization. Initial adjustments to these two coefficients are useful for reflecting the effects of low-speed driving.

[0060] The adjustable reward function also rewards comfort. It can be assumed that the autonomous agent tends to converge and maintain a certain arbitrary speed for most of its journey. Under this assumption, any change in speed will be applied with acceleration or deceleration for a short period and then converge back to zero. Therefore, to minimize the overuse of acceleration and deceleration, and the changes to these (which are known to cause passenger discomfort), the adjustable reward function uses C... Jerk The reward function of the coefficient penalizes the sum of the squared jerk components of the exercise plan under evaluation. Furthermore, any other artifacts that minimize this process can be inserted into this term (e.g., handling the non-differential part of the jerk profile). Here, again, preliminary adjustments can be made to the interrelationships between different comfort-related terms.

[0061] Adjustable reward functions also consider the trade-off between efficiency and comfort. For example, the second and third terms of the reward function represent efficiency and comfort, respectively. This trade-off can be achieved through a weighted C... speed With C Jerk To control.

[0062] Figure 2 An architecture 200 is depicted according to one or more embodiments described herein, supporting the use of an artificial intelligence model 210 and an adjustable reward function to train an autonomous agent (e.g., an autonomous agent engine 114) for autonomous vehicle operation. In this example, logic 204 generates input 211 to the artificial intelligence model 210. Logic 204 and the artificial intelligence model 210 together form the autonomous agent engine 114. Architecture 200 uses the output 215 of the artificial intelligence model 210 (e.g., a reinforcement learning model, an online search model, etc.) to simulate the actions of vehicle 100 using a simulator 220.

[0063] For example, vehicles 100 and 201a and 201b are traveling along road 202. Vehicles 201a and 201b are... Figure 1 Examples of other vehicles 154 are collectively referred to as "vehicle 201". It should be understood that one or more of vehicles 201 can be configured similarly to... Figure 1 Each of the vehicles shown and described herein, or in some examples of vehicles 201, may be configured to be different from vehicle 100.

[0064] Each of vehicles 100 and 201 collects data about itself and transmits it to a processing system for processing (e.g., using data / communication engine 112). According to one example, vehicle 201 collects data about itself and transmits it to processing system 110 of vehicle 100 for processing by autonomous agent 114. According to another example, vehicles 100 and 201 collect data about themselves and transmit it to remote processing system 150.

[0065] The data (i.e., information) provided by vehicle 201 can indicate the vehicle's perception in the environment and can generate estimated motion state of vehicle 201, lane assignment, vehicle intention (e.g., direction of travel, intention to merge / change lanes, etc.), and / or any other information useful for decision-making.

[0066] Vehicle 100 has a current state s (also referred to as the "perception state"). The autonomous agent engine 114 of vehicle 100 evaluates possible actions, such as following behind vehicle 201a, merging in front of vehicle 201b, and merging behind vehicle 201b. To evaluate these possible actions, the autonomous agent engine 114 uses the illustrated artificial intelligence model 210. The artificial intelligence model 210 takes data about vehicle 100, vehicles 201a, 201b, and any other relevant data (e.g., data about road 202, data about the surrounding environment, etc.) as input 211. The artificial intelligence model 210 evaluates the current state s, multiple actions a1…a2…a3…a4…a5…a6…a7…a8…a9 ... nThe AI ​​model 210 learns a policy π by considering future states s'. The AI ​​model 210 learns to rank / rate multiple actions a1…an based on its estimates of their expected future rewards with respect to an adjustable reward function. Actual samples from the adjustable reward function are used to supervise the above estimates and to determine which actions are likely to be more or less desirable than others by considering the future consequences of the actions. In this way, the AI ​​model 210 is trained. The AI ​​model 210 may rely on an input layer 212, hidden layers 213 (which may be one or more hidden layers), and an output layer 214 that produces an output 215 (e.g., a policy).

[0067] Simulator 220 receives output from autonomous agent engine 114 (represented by logic 204 and AI model 210) regarding which action *a* to apply next. Simulator 220 then advances its environment state to the next state *s'* and returns it to autonomous agent engine 114 along with an intermediate reward *r* calculated using an adjustable reward function. In the next step, autonomous agent engine 114 uses the previous state *s'*, which is now the current state *s* (e.g., the current state of the updated environment), and processes it through AI model 210 to again determine the next action *a* to apply. After storing most of these interactions as tuples of (s, a, r, s'), AI model 210 is updated under the supervision of *r* for each state-action selection throughout the agent simulator interactions.

[0068] More specifically, simulator 220 (via AI model 210) makes a decision (action a) from autonomous agent engine 114 and propels the actor (e.g., vehicles 201a, 201b) in the environment surrounding vehicle 100 (without propelling vehicle 100). Simulator 220 then simulates the environment's progression over time based on action a and uses action a to control (e.g., using control engine 116) vehicle 100. Control inputs (e.g., acceleration, trajectory, etc.) are fed back to simulator 220, which generates a new state s' and sends it back to logic 204, as indicated by arrow 221. When one iteration of simulation is complete, s' becomes the current state s of the environment, and logic 204 takes the new state s of the environment as input, converts it into a numerical input vector (e.g., input 211), which is then fed to AI model 210. AI model 210 outputs a numerical output vector 215, representing the index of the maximum reward score. Simulator 220 then maps the action to a trajectory. Now, simulator 220 knows how to advance vehicle 100 at the next time step. A dataset is generated by iterating this process between logic 204, AI model 210, and simulator 220. For each iteration, the following is generated: current state s, action a, future state s', and intermediate reward r. Once the dataset is constructed (which can be represented as a tuple of (s, a, r, s'), AI model 210 can improve based on the collected data to maximize the total reward, thereby enabling AI model 210 to make better decisions (e.g., maximizing r over time).

[0069] Figure 3 A flowchart is depicted illustrating a method 300 for implementing a reward function for autonomous vehicle operation according to one or more embodiments described herein. Method 300 can be performed by any suitable system or device, such as… Figure 1 Processing system 110, remote processing system 150, Figure 4 The processing system 400, or any other suitable processing system and / or processing device (e.g., a processor). Now refer to Figure 1 The element description method 300 is used for 2 and / or 2, but is not limited thereto.

[0070] At box 302, the autonomous agent engine 114 of processing system 110 receives the current state of the vehicle. At box 304, the autonomous agent engine 114 of processing system 110 uses the output of an artificial intelligence model (e.g., artificial intelligence model 210) to predict the future state of vehicle 100, at least in part, based on the current state of vehicle 100. At box 306, the autonomous agent engine 114 of processing system 110 uses an adjustable reward function to calculate a reward associated with the future state of vehicle 100. The adjustable reward function includes multiple adjustable coefficients as described herein. For example, as described herein, the adjustable coefficients can be adjusted to refine the reward based on efficiency, comfort, and task completion.

[0071] At box 308, the artificial intelligence model is trained and / or updated at least in part based on rewards. That is, the association between the inputs (initial state and actions) and the outputs (future states and rewards based on the initial state and actions) is learned and / or updated.

[0072] In some examples, the control engine 116 of the processing system 110 controls the vehicle 100 based at least in part on an artificial intelligence model. For example, the autonomous agent engine 114 makes driving decisions based at least in part on the artificial intelligence model 210 and some supplementary pre-programmed logic (referred to as a “driving strategy”) that wraps it. The vehicle 100 is then controlled by implementing the driving strategy. For example, the driving strategy may provide commands to the vehicle 100 to accelerate, decelerate, steer, etc., to perform actions to achieve a goal. This action can be selected from one of several actions. This improves vehicle operation by providing a safe, comfortable, and efficient ride for the occupants of the vehicle 100. This makes the vehicle 100 operate more efficiently, provides a better riding experience for its occupants, and in a safer manner.

[0073] The reward function described herein can be used in different types of artificial intelligence, such as reinforcement learning, online search, etc. In some embodiments, such as during reinforcement learning, the reward function is used during the offline training process. In other embodiments, such as during online search, the reward function is used directly online during real-time (or near-real-time) inference.

[0074] Additional processes may also be included, and it should be understood that... Figure 3 The processes described herein are illustrative and other processes may be added or existing processes may be removed, modified or rearranged without departing from the scope and spirit of this disclosure.

[0075] It should be understood that this disclosure can be implemented in conjunction with any type of computing environment now known or developed in the future. For example, Figure 4A block diagram of a processing system 400 for implementing the techniques described herein is depicted. In the example, the processing system 400 has one or more central processing units (processors) 421a, 421b, 421c, etc. (collectively referred to as processors 421 and / or processing devices). In aspects of this disclosure, each processor 421 may include a Reduced Instruction Set Computer (RISC) microprocessor. The processor 421 is connected to system memory (e.g., random access memory (RAM) 424) and various other components via a system bus 433. Read-only memory (ROM) 422 is connected to the system bus 433 and may include a Basic Input / Output System (BIOS) that controls certain basic functions of the processing system 400.

[0076] Further described are input / output (I / O) adapter 427 and network adapter 426 connected to system bus 433. I / O adapter 427 may be a Small Computer System Interface (SCSI) adapter that communicates with hard disk 423 and / or storage device 425 or any other similar component. I / O adapter 427, hard disk 423, and storage device 425 are collectively referred to herein as mass storage 434. Operating system 440 for execution on processing system 400 may be stored in mass storage 434. Network adapter 426 interconnects system bus 433 with external network 436, enabling processing system 400 to communicate with other such systems.

[0077] A display (e.g., a display monitor) 435 is connected to the system bus 433 via a display adapter 432, which may include a graphics adapter and a video controller for improving the performance of graphics-intensive applications. In one aspect of this disclosure, adapters 426, 427, and / or 432 may be connected to one or more I / O buses that are connected to the system bus 433 via an intermediate bus bridge (not shown). Suitable I / O buses for connecting peripheral devices (e.g., hard disk controllers, network adapters, and graphics adapters) typically include common protocols such as Peripheral Component Interconnect (PCI). Additional input / output devices are shown connected to the system bus 433 via a user interface adapter 428 and the display adapter 432. A keyboard 429, a mouse 430, and a speaker 431 (or other suitable input and / or output, such as a touchscreen in an infotainment system) may be interconnected to the system bus 433 via a user interface adapter 428, which may include, for example, a super I / O chip integrating multiple device adapters into a single integrated circuit. One or more cameras 120-123, 130-133 are also connected to system bus 433.

[0078] In some aspects of this disclosure, the processing system 400 includes a graphics processing unit 437. The graphics processing unit 437 is a specialized electronic circuit designed to manipulate and modify memory to accelerate the creation of images in a frame buffer for output to a display. Generally, the graphics processing unit 437 is highly efficient in manipulating computer graphics and image processing, and has a highly parallel architecture, making it more efficient than a general-purpose CPU in algorithms that process large blocks of data in parallel.

[0079] Therefore, as configured herein, the processing system 400 includes processing power in the form of a processor 421, storage capacity including system memory (e.g., RAM 424) and mass storage 434, input devices such as a keyboard 429 and a mouse 430, and output capacity including a speaker 431 and a display 435. In some aspects of this disclosure, a portion of the system memory (e.g., RAM 424) and mass storage 434 jointly stores an operating system 440 to coordinate the functions of the various components shown in the processing system 400.

[0080] Various examples of this disclosure have been described for illustrative purposes, but are not intended to be exhaustive or limiting to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described techniques. The terminology used herein has been chosen to best explain the principles of the technology, its practical application, or technical improvements to technologies found in the market, or to enable others skilled in the art to understand the technology disclosed herein.

[0081] While the foregoing disclosure has been described with reference to exemplary embodiments, those skilled in the art will understand that various changes can be made and equivalents can be substituted for elements therein without departing from its scope. Furthermore, many modifications can be made to adapt particular situations or materials to the teachings of this disclosure without departing from its essential scope. Therefore, it is intended that the technology be limited to the specific embodiments disclosed, but rather to include all embodiments falling within the scope of this application.

Claims

1. A computer-implemented method, comprising: The current status of the vehicle is received by the processing equipment; The processing device uses the output of an artificial intelligence model to predict the future state of the vehicle, at least in part, based on the vehicle's current state. The processing device uses an adjustable reward function to calculate the reward associated with the future state of the vehicle; the adjustable reward function includes multiple adjustable coefficients. The artificial intelligence model is trained by the processing device based at least in part on rewards; and At least partially based on artificial intelligence models to autonomously control the vehicle, The adjustable reward function rewards both complete success and partial success. Partial success includes situations where, based on the determination that autonomous vehicle control may not achieve complete success, vehicle control is transferred back to the user. The adjustable reward function is expressed as: in It is the speed adjustment coefficient; This is the vehicle's actual speed; It is the vehicle's expected speed; It is an adjustable coefficient that reduces the vehicle's speed from the desired driving speed; It is the adjustable coefficient of the jerk amplitude; It is the square of the jerk; It is an adjustable discount factor; It is an index function; and It is an adjustable coefficient representing the extent of a successful handover. It is a successful indicator function and It is the indicator function for the transfer.

2. The computer-implemented method according to claim 1, wherein, One of the adjustable coefficients is the adjustable coefficient for the magnitude of the acceleration caused by changes in vehicle speed.

3. The computer-implemented method according to claim 1, wherein, The adjustable reward function takes into account both current and future rewards.

4. The computer-implemented method according to claim 1, wherein, Adjustable reward functions take into account the trade-off between efficiency and comfort.

5. The computer-implemented method according to claim 1, wherein, The adjustable reward function is a function of the vehicle's current state s, action a, and future state s'.

6. The computer-implemented method according to claim 1, wherein, Adjustable reward functions support driver-in-the-loop functionality.

Images