Autonomous vehicles and methods for training autonomous vehicles in edge-case scenarios

Abstract

This disclosure relates to autonomous vehicles and methods for training autonomous vehicles in edge-case scenarios. In one embodiment, a method for training an autonomous vehicle model includes: autonomously controlling an autonomous vehicle within an environment using the autonomous vehicle model; receiving sensor data corresponding to one or more of the operation of the autonomous vehicle and the environment; generating an autonomous interruption request based at least in part on the sensor data; transmitting the autonomous interruption request to a remote operator; receiving control signals from the remote operator; controlling the autonomous vehicle according to the control signals from the remote operator; collecting additional sensor data while controlling the autonomous vehicle using the control signals from the remote operator; and training the autonomous vehicle model by providing the additional sensor data and control signals to the autonomous vehicle model.

Description

Technical Field

[0001] This disclosure relates to autonomous vehicles and methods for training autonomous vehicles in edge-case scenarios. Background Technology

[0002] Autonomous vehicles, such as Level 5 autonomous vehicles, are capable of navigating roads in an environment without human control. Such autonomous vehicles may include an autonomous driving system with a trained model that can operate to acquire sensor data about the vehicle and its environment, and generate a vehicle control system to drive the vehicle's trajectory within the environment.

[0003] In many situations, autonomous vehicles may encounter novel scenarios, such as rare edge cases. Examples could be uniquely configured intersections, ramp crests providing short-range visibility, and atypical driving behaviors of other vehicles in the environment. When an autonomous vehicle encounters such a scenario, the autonomous driving system can generate a low-confidence value, prompting it to take remedial actions, such as pulling over or other types of maneuvers.

[0004] Sensor data surrounding new scenarios can be collected and evaluated offline at a later time, and then used to further train or otherwise update the training model of the autonomous vehicle system, enabling the autonomous vehicle to subsequently learn how to operate in the new scenario. However, such offline processes are time-consuming and may delay the updating of the training model, and may also increase the costs associated with updating the training model with new scenarios. Furthermore, sensor data reflecting how a human driver would navigate the scenario is unavailable because the autonomous vehicle takes remedial actions that are not how a human would handle the scenario.

[0005] Therefore, alternative systems and methods may be needed for training autonomous vehicle models. Summary of the Invention

[0006] In one embodiment, a method for training an autonomous vehicle model includes: autonomously controlling an autonomous vehicle in an environment using the autonomous vehicle model; receiving sensor data corresponding to one or more of the operation of the autonomous vehicle and the environment; generating an autonomous interruption request based at least in part on the sensor data; transmitting the autonomous interruption request to a remote operator; receiving a control signal from the remote operator; controlling the autonomous vehicle according to the control signal from the remote operator; collecting additional sensor data while controlling the autonomous vehicle using the control signal from the remote operator; and training the autonomous vehicle model by providing the additional sensor data and the control signal to the autonomous vehicle model.

[0007] In another embodiment, a vehicle includes a plurality of sensors operable to generate sensor data on at least one of the autonomous vehicle's operation and the environment. The vehicle also includes a vehicle control system operable to move the autonomous vehicle within the environment. The vehicle further includes: an autonomous vehicle model receiving training on autonomously controlling the autonomous vehicle within the environment; one or more processors; and a non-transitory computer-readable medium. The non-transitory computer-readable medium stores instructions that, when executed by the one or more processors, cause the one or more processors to: autonomously control the autonomous vehicle within the environment using the autonomous vehicle model and the vehicle control system; generate an autonomous interruption request based at least in part on the sensor data; transmit the autonomous interruption request to a remote operator; receive control signals from the remote operator; control the autonomous vehicle using the vehicle control system according to the control signals from the remote operator; collect additional sensor data while controlling the autonomous vehicle using the control signals from the remote operator; and train the autonomous vehicle model by providing the additional sensor data and the control signals to the autonomous vehicle model. Attached Figure Description

[0008] Figure 1 An example autonomous vehicle is shown according to one or more embodiments described and illustrated herein.

[0009] Figure 2 An example autonomous vehicle communicating with a remote operator is shown, according to one or more embodiments described and illustrated herein.

[0010] Figure 3 Example components of an example autonomous vehicle according to one or more embodiments described and illustrated herein are shown.

[0011] Figure 4A An example human operator is shown who is a remote operator according to one or more embodiments described and illustrated herein.

[0012] Figure 4B An example remote operator of an expert system is shown, based on one or more embodiments described and illustrated herein.

[0013] Figure 5 An example method for autonomously controlling and training an autonomous vehicle, according to one or more embodiments described and illustrated herein, is shown.

[0014] Figure 6 A flowchart of an example method for autonomously controlling an autonomous vehicle is shown. Detailed Implementation

[0015] Embodiments of this disclosure relate to autonomous vehicles and systems and methods for training autonomous vehicle models. Currently, sensor data from autonomous vehicles is retrieved, analyzed, and used for retraining in an offline process following an autonomous driving session performed by the autonomous vehicle. For example, sensor data of the autonomous vehicle is generated and stored while it is driving in an environment. This sensor data is later analyzed, for example, classified by driving scenario or maneuver, and can be used to further train the autonomous vehicle model, thereby improving the performance of the autonomous vehicle.

[0016] When an autonomous vehicle encounters an unfamiliar scenario, its model may lack the confidence to generate a satisfactory trajectory for navigation. Consequently, the model may produce low confidence values ​​below a threshold. In such cases, the autonomous vehicle can be programmed to perform remedial maneuvers, such as pulling over to the side of the road and stopping, or taking some other action. These remedial maneuvers may not correspond to how a human driver would manually operate the vehicle in such a scenario. Therefore, the autonomous vehicle model should be trained to handle many different scenarios, some of which may be rare edge cases. However, since there is no human driver in the autonomous vehicle, sensor data on how a human driver would operate the vehicle in that scenario cannot be generated, as the autonomous vehicle takes remedial actions rather than navigating the environment like a human driver.

[0017] In embodiments of this disclosure, a remote operator controls the vehicle during scenarios where the autonomous vehicle cannot perform actions while operating in autonomous driving mode. For any and all possible reasons, the autonomous vehicle generates an autonomous interruption request prompted by the remote operator. As a non-limiting example, an autonomous interruption request may be generated when the autonomous vehicle model generates a confidence value below a threshold (or the autonomous vehicle's confidence does not meet some other metric), or when an assistance system (e.g., a collision avoidance system) generates an assistance signal that briefly takes over control of the autonomous vehicle.

[0018] A remote operator provides wireless control signals from a remote location, which the autonomous vehicle uses to navigate within a specific scenario environment. The remote operator can be a human operator at a remote operating facility, who remotely controls the autonomous vehicle until the vehicle completes the driving scenario and autonomous control is satisfactory. As another example, due to the power and processing constraints of the autonomous vehicle, the remote operator can be an expert system with a trained model that is more capable than the autonomous vehicle model running on it. As described in more detail below, the expert system's training model can include one or more large language models trained on the rules of roads in various jurisdictions, enabling the remote operator to skillfully control the autonomous vehicle in many different scenarios. For example, the remote operator as an expert system can be executed at a dedicated facility or hosted in a cloud environment.

[0019] During remote control of the autonomous vehicle, additional sensor data is generated by the vehicle's various sensors. This additional sensor data indicates how the autonomous vehicle should approach maneuvering in a specific scenario where an autonomous interruption request exists. In an embodiment, the additional sensor data is used to further train the autonomous vehicle model, enabling the autonomous vehicle to learn how to drive in an environment where a remote operator takes over. Thus, in similar scenarios in the future, a remote operator may not be required.

[0020] The following describes in detail various embodiments of autonomous vehicles and systems and methods for training autonomous vehicle models.

[0021] Now for reference Figure 1 An example autonomous vehicle 102 is schematically illustrated. The autonomous vehicle 102 can be any type of autonomous vehicle. For example, the autonomous vehicle 102 can be a Level 4 or Level 5 autonomous vehicle capable of driving without human intervention. The autonomous vehicle 102 shown has any number of sensors 104, such as cameras, lidar sensors, radar sensors, proximity sensors, speedometers, inertial measurement units (IMUs), steering angle sensors, braking sensors, occupancy sensors, and any other sensors capable of detecting the attributes of the autonomous vehicle 102 and the environment in which the autonomous vehicle 102 is navigating. The example autonomous vehicle 102 also includes a Global Positioning System (GPS) device 108, which is configured to receive position data from one or more satellites orbiting the Earth. As described in more detail below, the autonomous vehicle 102 has an autonomous driving system 106 capable of receiving sensor data from the multiple sensors 104, the GPS device 108, and any other data sources, and generating a trajectory for driving within the environment used by the autonomous vehicle 102. Autonomous driving system 106 includes autonomous vehicle model 124 trained to develop effective trajectories for autonomous vehicle 102 (see [link]). Figure 3In some embodiments, the autonomous driving system 106 may include other modules, such as a heuristic rule-based module that receives trajectories from the autonomous vehicle model 124 for verification before generating control signals. In other embodiments, heuristic rule-based modules are not utilized.

[0022] Figure 2 An autonomous vehicle 102 is shown driving on road 168 within an environment. The autonomous vehicle 102 is driven autonomously by an autonomous driving system 106 without a human driver. As the autonomous vehicle 102 drives on road 168, its sensors 104 generate data about the vehicle's operation (e.g., speed, steering angle, acceleration, and any other data indicative of the autonomous vehicle 102's operation) and data about the environment (e.g., data about other vehicles, pedestrians, number of lanes, road curvature, and any other environmental properties on road 168).

[0023] In most cases, the autonomous vehicle 102 can successfully navigate road 168 without any additional assistance. However, in some scenarios, the autonomous driving system 106 lacks confidence that it can generate a successful trajectory based on the sensor data it receives. Uncertainty may arise from driving scenarios where the autonomous driving system 106 has not been trained, such as edge-case scenarios that rarely occur. The embodiments are not limited to any type of scenario that could be problematic for the autonomous driving system 106. Non-limiting examples of problematic scenarios include a group of cyclists sharing road 168 with the autonomous vehicle 102, intersections with sharp right or left turns, slopes that limit the field of view of sensor 104 beyond the crest of a hill, dense fog, and animals in road 168. Any number of scenarios may cause the autonomous driving system 106 to lack confidence in developing a successful trajectory. However, a human driver is capable of performing maneuvers for manual navigation in these scenarios.

[0024] During problematic scenarios, the autonomous driving system 106 of autonomous vehicle 102 may generate an autonomous disengagement request or otherwise report that it cannot generate a satisfactory trajectory for that particular scenario. An autonomous disengagement request may be generated when the autonomous driving system 106 has a confidence value below a threshold, or when some other confidence metric is not met. As another example, an autonomous disengagement request may be generated when driver assistance signals of autonomous vehicle 102 are activated and generate one or more signals to control autonomous vehicle 102. For example, collision avoidance system 120 (see...) Figure 3The autonomous driving system 106 can generate a braking control signal indicating a scenario where the trajectory of the autonomous driving system 106 is unsatisfactory. In this case, an autonomous driving discontinuation request can be generated. As another example, if the lane keeping assist system 116 frequently generates control signals to keep the autonomous vehicle 102 within the lane lines, it can indicate that the trajectory generated by the autonomous driving system 106 is unsatisfactory, and an autonomous discontinuation request can be generated.

[0025] An autonomous interruption request may be provided to the remote operator 112 via one or more wireless signals 110, such as through a cellular communication network, satellite communication network, or any other communication network. Upon receiving the autonomous interruption request (or any other signal or command instructing authorized remote control of the autonomous vehicle 102), the remote operator 112 generates a control signal, which is then transmitted to the autonomous vehicle 102 via the communication network as a wireless signal 110. The control signal may include, but is not limited to, acceleration signals, braking signals, and steering signals. The remote operator 112 may successfully navigate the vehicle through the specific scenario until it is appropriate to transfer control of the autonomous vehicle 102 back to the autonomous driving system 106.

[0026] While the remote operator 112 is controlling the autonomous vehicle 102, the sensor 104 continues to generate additional sensor data. This additional sensor data can be useful information for learning how to navigate the vehicle correctly and successfully during a specific scenario. The additional sensor data provides information such as speed, acceleration, deceleration, steering angle, path, and other information about the attributes of the trajectory taken by the autonomous vehicle 102 under remote control.

[0027] The additional sensor data is then provided as training data to the autonomous driving system 106 to further train it on how to formulate a trajectory for a specific scenario that triggered the autonomous driving interruption request. More specifically, sensor data within the time window of the autonomous driving interruption request can be provided as training data to the autonomous driving system 106. The time window may begin a specific amount of time before the autonomous driving interruption request and end a specific amount of time after the autonomous driving interruption request or after control of the vehicle is once again transferred to the autonomous driving system 106. In this way, the sensor data reflects what driving conditions caused the scenario to occur and what trajectory the remote operator 112 provides to resolve the driving scenario. In some embodiments, a classifier is used to classify the driving scenario. Driving classifications can also be provided as training data to the autonomous driving system 106. Driving classifications are not limited to this disclosure and may include intersection traversal, merging, lane changing, road agent problems, pedestrian and sensor occlusion.

[0028] Now for reference Figure 3 The diagram schematically illustrates additional components of an example autonomous vehicle 102. The autonomous vehicle 102 includes various driver assistance systems, such as a lane-keeping assist system 116, an adaptive cruise control system 118, and a collision avoidance system 120. These systems can be used by the autonomous vehicle 102 to provide control signals when needed and, as described above, generate an autonomous interruption request if necessary. The autonomous vehicle 102 also includes vehicle control systems, such as, but not limited to, a propulsion system 126 (e.g., a motor or engine and accelerator) for providing mechanical propulsion to the autonomous vehicle 102, a steering system 128 for lateral control of the movement of the autonomous vehicle 102, and a braking system 130 for decelerating the autonomous vehicle 102. The vehicle control systems receive control signals from the autonomous driving system 106, the driver assistance systems, or a remote operator as needed. The control signals control the various vehicle control systems, enabling the autonomous vehicle 102 to complete a successful trajectory within its environment. As described above, the autonomous vehicle 102 also includes multiple sensors, such as speedometers, lidar sensors, radar, cameras, and other sensors capable of generating sensor data indicative of the properties of the autonomous vehicle 102 and its environment. The autonomous vehicle 102 also includes networking hardware 170 to generate and receive wireless signals 110 for communication between the autonomous vehicle 102 and the remote operator 112.

[0029] The autonomous driving system 106 includes, but is not limited to, an autonomous driving stack stored in a non-transitory computer-readable medium 122, which includes software components and sub-components for performing tasks of autonomously controlling the autonomous vehicle 102 within the environment. The autonomous driving system 106 also includes an autonomous vehicle model 124, which is trained using training data and sensor data as described above regarding autonomous interruption requests and control by the remote operator 112. Each of the driver assistance system, the autonomous driving system 106, the vehicle control system, and the sensors and communication devices can communicate with each other to provide data for successful navigation within the environment.

[0030] Now for reference Figure 4AAn example remote operator 112 configured as a human operator 134 is shown. As a non-limiting example, the human operator 134 could be an employee in an office or other location providing remote vehicle control services. The human operator 134 interfaces with the control system 136 to generate control signals, which are then transmitted to the autonomous vehicle 102 via wireless signal 110. The control system may have vehicle-like input devices, such as an accelerator pedal, brake pedal, and steering wheel. For example, the human operator 134 could sit in a simulated vehicle cockpit with various vehicle input devices and an electronic display showing the environment of the remote autonomous vehicle 102, allowing the human operator 134 to use the control system 136 to generate control signals to control the autonomous vehicle 102. In other embodiments, the human operator 134 may use one or more joysticks as input devices. Other input devices capable of generating control signals may also be utilized. The outputs of these input devices of the control system 135 can then be transmitted to the autonomous vehicle 102 via one or more communication networks, enabling the autonomous vehicle 102 to be remotely controlled.

[0031] When remote operator 112 receives an autonomous interruption request, a communication channel is established between human operator 134 and control system 136, allowing sensor data to be provided to remote operator 112 and control signals to be provided to autonomous vehicle 102. For example, a video feed using camera data from sensor 104 of autonomous vehicle 102 is provided to and displayed on control system 136, allowing human operator 134 to view the environment and generate vehicle control signals accordingly. These control signals are then provided to the autonomous vehicle.

[0032] Figure 4B Another remote operator 138 is shown, configured as an expert system not controlled by a human. Due to limitations in the battery power and computing capabilities of the autonomous vehicle 102, Figure 4B The remote operator 138 can be a more advanced and capable autonomous driving system than the autonomous driving system 106 running on the autonomous vehicle 102. Typically, when the remote operator 138 receives an autonomous interruption request, control of the autonomous vehicle 102 is transferred to the remote operator 138's expert system, which generates control signals for the autonomous vehicle 102, enabling the autonomous vehicle 102 to successfully navigate the scenarios encountered by the autonomous vehicle 102. For example, the remote operator as an expert system can operate from a dedicated facility or be hosted in a cloud environment.

[0033] An exemplary remote operator 138 includes one or more non-transitory computer-readable storage components 142 and one or more processors 144, the processors 144 being operable to execute instructions stored on the one or more storage components 142 to perform the remotely controlled vehicle operations described herein. The one or more storage components 142 may store non-transitory computer-readable media, one or more trained autonomous vehicle models, and / or other software components for remotely and autonomously controlling the autonomous vehicle 102.

[0034] The remote operator 138 also includes networked hardware 146 capable of operating to communicate with the autonomous vehicle 102 via one or more communication networks. For example, the networked hardware 146 enables control signals generated by the remote operator 138 to be transmitted to the autonomous vehicle 102 and enables sensor data to be transmitted from the autonomous vehicle 102 to the remote operator 138.

[0035] The example remote operator 138 also includes one or more large language models 140 that provide knowledge to the remote operator 138. For example, the one or more large language models 140 may be trained on road rules for various jurisdictions. Different jurisdictions have different road rules. The road rules of one country or state may differ from those of another country or state. The remote operator 138 determines the location and jurisdiction of the autonomous vehicle 102 based on sensor data provided to it, and then selects an appropriate large language model 140 for that jurisdiction. The remote operator 138 then uses the road rules provided by the large language model 140 (or other sources) for that jurisdiction.

[0036] In some embodiments, the remote operator 138 may include a number of trained models for many individual scenarios. Thus, each model may be an expert in navigation within a specific scenario. As a non-limiting example, these expert models may be configured as a large language model 140. One model may be trained to navigate the vehicle in the presence of animals on the road, while another model may be trained to navigate the vehicle in the presence of dense fog.

[0037] Remote operator 138 receives sensor data from autonomous vehicle 102 and makes a decision about the type of driving scenario. For example, remote operator 138 may perform classification or classification algorithms on the sensor data to determine the scenario experienced by autonomous vehicle 102. Any known or undeveloped classification or classification algorithms may be utilized. In a foggy environment, remote operator 138 may use sensor data (e.g., camera data) and classify the driving scenario as a foggy driving scenario. Remote operator 138 may then utilize a trained model trained in the foggy driving scenario. It should be understood that in other embodiments, a single trained model is used instead of multiple dedicated trained models.

[0038] Figure 5 An example process flow according to one or more embodiments of this disclosure is illustrated. At block 176, while sensor data generated by one or more sensors 104 of the vehicle is stored, the autonomous vehicle 102 autonomously navigates the environment using one or more autonomous vehicle models 124. As described below, the sensor data may relate to any vehicle operation or attribute of the environment. When an autonomous interruption request is present, the process moves to block 178, where, as described above, the autonomous vehicle 102 is either remotely controlled by a human operator or by an expert system. In addition to the sensor data, remote operator control signals for remotely controlling the autonomous vehicle 102 are also stored.

[0039] Then, at box 180, the autonomous vehicle model 124 is further trained using sensor data and remote operator control signals. This offline training process enables the autonomous vehicle model 124 to learn from control signals provided by the remote operator regarding autonomous interruption requests, which may be the result of edge situations unfamiliar to the autonomous vehicle model 124. In this way, the autonomous vehicle model 124 learns how to navigate in edge situations.

[0040] At box 182, the performance of the updated autonomous vehicle model 124 is then evaluated during testing. In one example, the autonomous vehicle model 124 is tested in a simulated environment. The simulated environment allows a virtual vehicle operating using the updated autonomous vehicle model 124 to determine how well the autonomous vehicle model 124 performs by passing through simulated edge cases that lead to an autonomous interruption request. In another example, the autonomous vehicle model 124 is set up on a physical vehicle performing the autonomous vehicle model 124 on a closed route. After determining that the autonomous vehicle model 124 performs according to quality criteria, it is then deployed to the autonomous vehicle, for example, via a software update (e.g., an over-the-air software update or a wired software update). The process continues to box 176, where the autonomous vehicle operates and generates sensor data. In this way, the autonomous vehicle model 124 can be continuously trained and improved to handle new edge cases.

[0041] Now for reference Figure 6 A flowchart of an example method 150 for autonomously controlling an autonomous vehicle 102 is shown. In block 152, method 150 includes autonomously controlling the autonomous vehicle 102 within an environment using an autonomous vehicle model. In block 154, sensor data is received and stored while the autonomous vehicle is driving autonomously. The sensor data corresponds to one or more of the autonomous vehicle's operation and the environment, such as the speeds of vehicles and road agents (e.g., other vehicles and pedestrians) in the environment.

[0042] In block 156, the method includes generating an autonomous interruption request based at least in part on sensor data. For example, the autonomous driving system 106 may generate an autonomous interruption request to take over control of the autonomous vehicle 102 upon the occurrence of a confidence value below a threshold or other event or cause.

[0043] In block 158, the method continues by transmitting an autonomous interruption request to a remote operator who evaluates sensor data and generates control signals corresponding to one or more trajectories. In block 160, the autonomous vehicle 102 receives control signals from the remote operator. In block 162, the method continues by controlling the autonomous vehicle according to control signals provided by the remote operator. In block 164, the method collects additional sensor data while controlling the autonomous vehicle using control signals from the remote operator. In block 166, the method continues by training the autonomous vehicle model by providing additional sensor data and control signals to the autonomous vehicle model.

[0044] It should now be understood that embodiments of this disclosure relate to autonomous vehicles and systems and methods for training autonomous vehicle models. In embodiments, a remote operator controls the vehicle during scenarios where the autonomous vehicle cannot operate in autonomous driving mode. Additional sensor data is collected while the autonomous vehicle is remotely controlled. This additional sensor data is used to further train the autonomous vehicle model, such that over time, the autonomous vehicle model improves its performance and can successfully navigate through new and / or challenging driving scenarios.

[0045] It should be noted that the terms “substantially” and “about” are used herein to indicate the degree of uncertainty that may be attributable to any quantitative comparison, value, measurement, or other representation. These terms are also used herein to indicate the degree to which a quantitative representation may differ from the stated reference value without causing a change in the fundamental function of the subject matter under discussion.

[0046] While specific embodiments have been shown and described herein, it should be understood that various other changes and modifications may be made without departing from the spirit and scope of the claimed subject matter. Furthermore, although various aspects of the claimed subject matter have been described herein, these aspects need not be used in combination. Therefore, the appended claims are intended to cover all such changes and modifications within the scope of the claimed subject matter.

Claims

1. A method for training an autonomous vehicle model, the method comprising: The autonomous vehicle model is used to autonomously control autonomous vehicles within the environment; Receive sensor data, which corresponds to one or more of the operation of the autonomous vehicle and the environment; The autonomous interruption request is generated at least in part based on the sensor data; Transmit the autonomous interruption request to the remote operator; Receive control signals from the remote operator; The autonomous vehicle is controlled according to the control signals from the remote operator; While controlling the autonomous vehicle using the control signals from the remote operator, additional sensor data is collected; as well as The autonomous vehicle model is trained by providing the additional sensor data and the control signals to the autonomous vehicle model.

2. The method of claim 1, further comprising classifying the driving scenario within the time window surrounding the generation of the autonomous interruption request based on the sensor data and the additional sensor data.

3. The method according to claim 2, wherein, The autonomous vehicle model is trained by providing the additional sensor data to the additional sensor data, which is obtained by classifying driving scenarios.

4. The method according to claim 1, wherein, The remote operator is human-controlled.

5. The method according to claim 1, wherein, The remote operator is an autonomous remote control system.

6. The method according to claim 5, wherein, The autonomous remote control system includes a large language model.

7. The method according to claim 5, wherein, The autonomous remote control system is trained on rules regarding the jurisdiction.

8. The method according to claim 1, wherein, When the confidence value of the autonomous vehicle model is lower than the threshold based on the sensor data, the autonomous interruption request is generated.

9. The method according to claim 1, wherein, The autonomous interruption request is generated after receiving an assisted driving signal from the assisted driving system.

10. The method according to claim 9, wherein, The driver assistance system includes lane keeping assist, collision avoidance system, or adaptive cruise control system.

11. An autonomous vehicle, comprising: Multiple sensors, which are operable to generate sensor data on at least one of the operation of the autonomous vehicle and the environment; A vehicle control system capable of operating to move the autonomous vehicle within the environment; An autonomous vehicle model that is trained on autonomously controlling the autonomous vehicle within the environment; One or more processors; as well as A non-transitory computer-readable medium storing instructions, which, when executed by the one or more processors, are used for: The autonomous vehicle model and the vehicle control system are used to autonomously control the autonomous vehicle within the environment; The autonomous interruption request is generated at least in part based on the sensor data; Transmit the autonomous interruption request to the remote operator; Receive control signals from the remote operator; Using the vehicle control system, the autonomous vehicle is controlled according to the control signals from the remote operator; While controlling the autonomous vehicle using the control signals from the remote operator, additional sensor data is collected; as well as The autonomous vehicle model is trained by providing the additional sensor data and the control signals to the autonomous vehicle model.

12. The autonomous vehicle according to claim 11, wherein, The computer-readable instructions further enable the vehicle to classify driving scenarios within a time window surrounding the generation of the autonomous interruption request based on the sensor data and the additional sensor data.

13. The autonomous vehicle according to claim 12, wherein, The autonomous vehicle model is further trained by classifying driving scenarios.

14. The autonomous vehicle according to claim 11, wherein, The remote operator is human-controlled.

15. The autonomous vehicle according to claim 11, wherein, The remote operator is an autonomous remote control system.

16. The autonomous vehicle according to claim 15, wherein, The autonomous remote control system includes a large language model.

17. The autonomous vehicle according to claim 15, wherein, The autonomous remote control system is trained on rules regarding the jurisdiction.

18. The autonomous vehicle according to claim 11, wherein, When the confidence value of the autonomous vehicle model is lower than the threshold based on the sensor data, the autonomous interruption request is generated.

19. The autonomous vehicle according to claim 11, wherein, The autonomous interruption request is generated after receiving an assisted driving signal from the assisted driving system.

20. The autonomous vehicle according to claim 19, wherein, The driver assistance system includes lane keeping assist, collision avoidance system, or adaptive cruise control system.

Images