Safe and secure remote control for autonomous driving

Abstract

The system may receive video streams from cameras on vehicles within the traffic network. The system may also receive inputs indicating the acceleration and steering angle of a simulated preceding vehicle. Based on the acceleration and steering inputs, the system may determine the attitude of the simulated vehicle relative to a reference attitude and display an overlay of the simulated vehicle's representation in the video stream in pixel coordinates based on the attitude. From the pixel coordinates, the system may determine the spatial coordinates of the simulated vehicle within the traffic network, where the spatial coordinates are relative to at least one of the traffic network or the cameras. The system may transmit the spatial coordinates to the vehicle to cause it to follow a path based on the spatial coordinates.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to autonomous vehicles, and more specifically to transmitting information for causing a vehicle to follow a predicted path.

BACKGROUND ART

[0002] Autonomous vehicles (AVs) provide human drivers with the convenience of efficient travel from one location to another without the need to pay attention to road conditions. An AV can be described as an autonomous (e.g., computer-controlled) vehicle that is capable of traveling on roads while complying with traffic rules and norms. However, even the best autonomous vehicle programming cannot account for or appropriately respond to all conditions and situations that may arise during operation of an autonomous vehicle. Furthermore, an autonomous vehicle may encounter conditions and situations where it can benefit from the assistance of a human operator (e.g., a remote operator).

SUMMARY OF THE INVENTION

[0003] This specification discloses aspects, features, elements, implementations, and embodiments of transmitting information for causing a vehicle to follow a predicted path.

[0004] One aspect of the disclosed embodiments is a method comprising: receiving a video stream from a camera of a vehicle in a transportation network; receiving an acceleration input and a steering input; determining a simulated vehicle pose relative to a reference pose based on the acceleration input and the steering input; displaying an overlay of a simulated vehicle representation in the video stream at pixel coordinates based on the pose; determining spatial coordinates of the simulated vehicle in the transportation network from the pixel coordinates, wherein the spatial coordinates are relative to at least one of the transportation network or the camera; and transmitting the spatial coordinates to the vehicle to cause the vehicle to follow a path based on the spatial coordinates.

[0005] Another aspect of the disclosed embodiment is an apparatus including a memory and a processor configured to execute instructions stored in the memory, the instructions being for receiving a video stream from a camera of a vehicle in a traffic network, receiving acceleration input and steering input, determining a simulated vehicle attitude relative to a reference attitude based on the acceleration input and steering input, displaying an overlay of a representation of the simulated vehicle in the video stream in pixel coordinates based on the attitude, determining the spatial coordinates of the simulated vehicle in the traffic network from the pixel coordinates, the spatial coordinates being relative to at least one of the traffic network or the camera, and transmitting the spatial coordinates to the vehicle in order to cause the vehicle to follow a path based on the spatial coordinates.

[0006] Another aspect of the disclosed embodiments is a non-temporary computer-readable medium storing instructions operable to cause one or more processors to perform an operation, the operation including receiving a video stream from a camera of a vehicle in a traffic network; receiving acceleration inputs and steering inputs; determining a simulated vehicle attitude relative to a reference attitude based on the acceleration inputs and steering inputs; displaying an overlay of a representation of the simulated vehicle in the video stream in pixel coordinates based on the attitude; determining the spatial coordinates of the simulated vehicle in the traffic network from the pixel coordinates, the spatial coordinates being relative to at least one of the traffic network or the camera; and transmitting the spatial coordinates to the vehicle in order to cause the vehicle to follow a path based on the spatial coordinates.

[0007] Modifications of these and other aspects, features, elements, implementations, and embodiments of the methods, apparatus, procedures, and algorithms disclosed herein will be described in further detail below. [Brief explanation of the drawing]

[0008] Various aspects of the methods and apparatus disclosed herein will become clearer by referring to the examples provided in the following description and drawings. In the drawings, similar reference numerals refer to similar elements unless otherwise indicated.

[0009] [Figure 1] This figure shows some examples of vehicles that can implement the embodiments, features, and elements disclosed herein.

[0010] [Figure 2] This figure shows some examples of vehicle transport and communication systems that can implement the embodiments, features, and elements disclosed herein.

[0011] [Figure 3] A block diagram showing a remote vehicle support center according to the implementation of this disclosure.

[0012] [Figure 4] This is a block diagram of an example system for displaying an overlay that includes a representation of a simulated preceding vehicle and its predicted path.

[0013] [Figure 5] This is an example of converting spatial coordinates to pixel coordinates.

[0014] [Figure 6] This is an example of a graphical user interface (GUI) that displays an overlay including a simulated preceding vehicle and its predicted path.

[0015] [Figure 7] This is an example diagram illustrating the determination of the predicted path and breadcrumb trail of a simulated preceding vehicle for use in the route planning process of an autonomous vehicle.

[0016] [Figure 8] This is a diagram illustrating an example of spatial coordinates in a transportation network.

[0017] [Figure 9] This is a diagram of an example using the Ackermann model.

[0018] [Figure 10] This is a flowchart of an example process for displaying an overlay including a simulated preceding vehicle and its predicted path. [Modes for Carrying Out the Invention]

[0019] Remote operation enables an autonomous vehicle (AV) to ensure driving safety, for example, in situations where the artificial intelligence (AI) system of the AV cannot determine an optimal action policy. For example, the AV may not be able to sufficiently evaluate, classify, and / or understand the intentions of other road users. Other road users may include pedestrians, construction workers, police officers, construction machinery, vehicles, cyclists, or other stationary objects (e.g., buildings, road signs, etc.). For example, evaluating road conditions may be outside the sensing capability of the AV, such as when identifying whether other road users exist around a corner obscured by a building, or when identifying whether other road users exist on the opposite side of a hill, and the like. Furthermore, resolving some situations (e.g., obstacle situations) may require the AV to deviate from normal (e.g., legally permitted and socially acceptable) driving rules in a manner that is unacceptable without human supervision.

[0020] Remote control may be enabled by allowing a remote operator to remotely navigate a vehicle by utilizing video streams of the vehicle's front and surrounding environment. However, in some cases, it may take time for the remote operator to generate a route for the vehicle. For example, the remote operator may need to determine multiple waypoints on a map and then transmit the route to the vehicle in order to establish a route. Depending on the complexity of the situation, the time required to create the route may result in an unreasonable delay before the vehicle can proceed. As a result, remote control may not be effective unless the video stream has low latency. Furthermore, because the remote operator is not physically located inside the vehicle, it may be difficult for the remote operator to visualize the driving situation.

[0021] Implementations of the present disclosure address issues such as these by displaying an overlay within a video stream at a graphical user interface (GUI) to improve remote operation of vehicles. In some implementations, the system can receive a video stream from one or more cameras of a vehicle within a transportation network. The system can also receive acceleration inputs and steering inputs for a simulated preceding vehicle, i.e., a simulated vehicle that (virtually) travels ahead of the vehicle within the transportation network. The system can determine the pose of the simulated vehicle relative to a reference pose based on the acceleration inputs and the steering inputs. The system can display an overlay of a representation of the simulated vehicle within the video stream at pixel coordinates based on the pose. The system can determine spatial coordinates of the simulated vehicle within the transportation network from the pixel coordinates, where the spatial coordinates are relative to at least one of the transportation network or the camera. The system can transmit the spatial coordinates to the vehicle to cause the vehicle to follow a path based on the spatial coordinates. In some implementations, the system can generate the overlay to assist a remote operator. In some implementations, the reference pose corresponds to the pose of the vehicle relative to at least one of the transportation network or the camera. In some implementations, the system may receive a reset input for resetting the pose of the simulated vehicle to the reference pose.

[0022] The overlay of the simulated vehicle may be drawn within a video stream received from a front camera of the automobile. In some implementations, the representation of the simulated vehicle includes a representation of a set of wheels, a chassis, a front bumper, and a lead bumper that mimics a vehicle. In some implementations, the representation of the simulated vehicle is displayed within the video stream after the acceleration input exceeds a predetermined threshold.

[0023] In some implementations, the system may determine a predicted path for the simulated vehicle based on the simulated vehicle's attitude and at least one of the acceleration input and steering input. The system may display an overlay of the predicted path representation within the video stream. The overlay of the simulated vehicle's predicted path may be drawn within the video stream received from the vehicle's front camera. In some implementations, the overlay allows the remote operator to visualize the path the simulated vehicle would travel given previous and / or current acceleration and steering inputs, thereby enabling them to take measures, for example, to avoid obstacles along the path.

[0024] In some implementations, the system may determine the speed of the simulated vehicle, and the system may transmit information to the vehicle representing at least one of the following: steering input, acceleration input, speed, and the yaw component of the simulated vehicle's attitude.

[0025] Refer to the following diagrams to describe some implementation configurations in more detail.

[0026] Figure 1 is a diagram of an example of a vehicle 1050 that can implement the embodiments, features, and elements disclosed herein. The vehicle 1050 may include a chassis 1100, a powertrain 1200, a controller 1300, wheels 1400 / 1410 / 1420 / 1430, or any other elements or combinations of elements of the vehicle. For simplicity, the vehicle 1050 is shown to include four wheels 1400 / 1410 / 1420 / 1430, but any other propulsion device such as a propeller or tracks may be used. In Figure 1, the lines interconnecting elements such as the powertrain 1200, the controller 1300, and the wheels 1400 / 1410 / 1420 / 1430 indicate that information such as data or control signals, power such as electricity or torque, or both information and power may be communicated between the elements. For example, the controller 1300 may receive power from the powertrain 1200 and communicate with the powertrain 1200, the wheels 1400 / 1410 / 1420 / 1430, or both, to control the vehicle 1050, which may include accelerating, decelerating, steering, or otherwise controlling the vehicle 1050.

[0027] The powertrain 1200 includes the power source 1210, the transmission 1220, the steering unit 1230, the vehicle actuator 1240, or any other elements or combinations of elements of the powertrain such as the suspension, drive shafts, axles, or exhaust system. Although shown separately, the wheels 1400 / 1410 / 1420 / 1430 may also be included in the powertrain 1200.

[0028] The power source 1210 may be any device or combination of devices capable of operating to provide energy such as electrical energy, chemical energy, or thermal energy. For example, the power source 1210 may include an engine such as an internal combustion engine, an electric motor, or a combination of an internal combustion engine and an electric motor, and may be operated to provide energy as driving force to one or more of the wheels 1400 / 1410 / 1420 / 1430. In some embodiments, the power source 1210 may include a potential energy unit such as one or more dry cell batteries, solar cells, fuel cells, or any other device capable of providing energy, such as nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), or lithium-ion (Li-ion).

[0029] The transmission 1220 receives energy from the power source 1210 and transmits that energy to the wheels 1400 / 1410 / 1420 / 1430 to provide driving force. The transmission 1220 may be controlled by the controller 1300, the vehicle actuator 1240, or both. The steering unit 1230 may be controlled by the controller 1300, the vehicle actuator 1240, or both, and controls the wheels 1400 / 1410 / 1420 / 1430 to steer the vehicle. The vehicle actuator 1240 may receive signals from the controller 1300 and actuate or control the power source 1210, the transmission 1220, the steering unit 1230, or any combination thereof, to operate the vehicle 1050.

[0030] In some embodiments, the controller 1300 includes a location unit 1310, an electronic communication unit 1320, a processor 1330, a memory 1340, a user interface 1350, a sensor 1360, an electronic communication interface 1370, or any combination thereof. Although shown as a single unit, any one or more elements of the controller 1300 may be integrated into any number of separate physical units. For example, the user interface 1350 and the processor 1330 may be integrated into a first physical unit, and the memory 1340 may be integrated into a second physical unit. Although not shown in Figure 1, the controller 1300 may include a power source such as a battery. Although shown as separate elements, the location unit 1310, the electronic communication unit 1320, the processor 1330, the memory 1340, the user interface 1350, the sensor 1360, the electronic communication interface 1370, or any combination thereof may be integrated into one or more electronic units, circuits, or chips.

[0031] In some embodiments, the processor 1330 includes any device or combination of devices that can manipulate or process signals or other information, whether currently existing or to be developed, including optical processors, quantum processors, molecular processors, or combinations thereof. For example, the processor 1330 may include one or more dedicated processors, one or more digital signal processors, one or more microprocessors, one or more controllers, one or more microcontrollers, one or more integrated circuits, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more programmable logic arrays (PLAs), one or more programmable logic controllers (PLCs), one or more state machines, or any combination thereof. The processor 1330 may be operably coupled to a location unit 1310, a memory 1340, an electronic communication interface 1370, an electronic communication unit 1320, a user interface 1350, a sensor 1360, a powertrain 1200, or any combination thereof. For example, the processor may be operably coupled to the memory 1340 via a communication bus 1380.

[0032] In some embodiments, the processor 1330 may be configured to execute instructions, including instructions for remote operation, which may be used to operate the vehicle 1050 from a remote location, including an operation center. Instructions for remote operation may be stored within the vehicle 1050 or received from an external source, such as a server computing device, which may include a traffic management center or a cloud-based server computing device. The processor 1330 may be configured to execute instructions for following a predicted path, as described herein.

[0033] Memory 1340 may include any tangible non-temporary computer-usable medium or computer-readable medium that can contain, store, communicate, or transport machine-readable instructions or any information associated therewith for use by or in connection with the processor 1330. Memory 1340 is, for example, one or more solid-state drives, one or more memory cards, one or more removable media, one or more read-only memories, one or more random-access memories, one or more disks including one or more solid-state drives, hard disks, floppy disks, optical disks, magnetic cards or optical cards, or any type of non-temporary medium suitable for storing electronic information, or any combination thereof.

[0034] The electronic communication interface 1370 may be a wireless antenna, a wired communication port, an optical communication port, or any other wired or wireless unit that can interface with a wired or wireless electronic communication medium 1500, as shown in the figure.

[0035] The electronic communication unit 1320 may be configured to transmit or receive signals via a wired or wireless electronic communication medium 1500, for example, via an electronic communication interface 1370. Although not explicitly shown in Figure 1, the electronic communication unit 1320 may be configured to transmit, receive, or both, via any wired or wireless communication medium such as radio frequency (RF), ultraviolet (UV), visible light, optical fiber, wire, or a combination thereof. Figure 1 shows a single electronic communication unit 1320 and a single electronic communication interface 1370, but any number of communication units and any number of communication interfaces may be used. In some embodiments, the electronic communication unit 1320 may include a dedicated short-range communication (DSRC) unit, a wireless safety unit (WSU), a cellular communication unit such as an IEEE 802.11p (Wi-Fi-P), Long-Term Evolution (LTE), or 5G transceiver, or a combination thereof.

[0036] The positioning unit 1310 may determine geographical location information of the vehicle 1050, including but not limited to longitude, latitude, altitude, direction of travel, or speed. For example, the positioning unit includes a Global Positioning System (GPS) unit such as a Wide Area Augmentation System (WAAS) Compatible National Marine Electronics Association (NMEA) unit, a radio triangulation unit, or a combination thereof. The positioning unit 1310 may be used, for example, to obtain information representing the current bearing of the vehicle 1050, the current position of the vehicle 1050 in two or three dimensions, the current angular orientation of the vehicle 1050, or a combination thereof.

[0037] The user interface 1350 may include any unit that can be used as an interface by a person, including a virtual keypad, a physical keypad, a touchpad, a display, a touchscreen, a speaker, a microphone, a video camera, a sensor, and a printer. The user interface 1350 may be operably coupled to the processor 1330 or to any other element of the controller 1300, as shown in the figure. Although shown as a single unit, the user interface 1350 may include one or more physical units. For example, the user interface 1350 may include a voice interface for performing voice communication with a person and a touch display for performing visual and touch-based communication with a person.

[0038] Sensor 1360 may include one or more sensors, such as a sensor array, which may be operable to provide information that can be used to control the vehicle. Sensor 1360 can provide information about the current operating characteristics of the vehicle or its surroundings. Sensor 1360 may include, for example, a speed sensor, an acceleration sensor, a steering angle sensor, a traction-related sensor, a braking-related sensor, or any sensor or combination of sensors operable to report information about any aspect of the current dynamic condition of the vehicle 1050.

[0039] In some embodiments, the sensor 1360 may include sensors that are capable of acquiring information about the physical environment surrounding the vehicle 1050. For example, one or more sensors may detect road shape and obstacles such as fixed obstacles, vehicles, cyclists, and pedestrians. In some embodiments, the sensor 1360 may be one or more video cameras, laser detection systems, infrared detection systems, acoustic detection systems, or any other suitable type of in-vehicle environment detection device or combination of devices that are currently known or to be developed, or may include these. In some embodiments, the sensor 1360 and the positioning unit 1310 are combined.

[0040] Although not shown separately, the vehicle 1050 may include a track controller. For example, controller 1300 may include a track controller. The track controller may be operable to acquire information describing the current state of the vehicle 1050 and the route planned for the vehicle 1050, and to determine and optimize a track for the vehicle 1050 based on this information. In some embodiments, the track controller outputs a signal that can be operated to control the vehicle 1050 so that it follows the track determined by the track controller. For example, the output of the track controller may be an optimized track that may be supplied to the powertrain 1200, wheels 1400 / 1410 / 1420 / 1430, or both. In some embodiments, the optimized track may be a set of control inputs, such as steering angles, where each steering angle corresponds to a point in time or position. In some embodiments, the optimized track may be one or more paths, lines, curves, or combinations thereof.

[0041] One or more of the wheels 1400 / 1410 / 1420 / 1430 may be a steering wheel that is turned to a steering angle under the control of the steering unit 1230, a drive wheel to which torque is applied to propel the vehicle 1050 under the control of the transmission 1220, or a steering drive wheel that steers and propels the vehicle 1050.

[0042] The vehicle may include units or elements not shown in Figure 1, such as a housing, a Bluetooth® module, a frequency modulation (FM) radio unit, a near-field communication (NFC) module, a liquid crystal display (LCD) display unit, an organic light-emitting diode (OLED) display unit, a speaker, or any combination thereof.

[0043] Figure 2 illustrates some examples of a vehicle transport and communication system 2000 that can implement the embodiments, features, and elements disclosed herein. The vehicle transport and communication system 2000 includes a vehicle 2100, such as the vehicle 1050 shown in Figure 1, and one or more external objects, such as an external object 2110, which may include any form of transport, such as the vehicle 1050 shown in Figure 1, pedestrians, cyclists, and any form of structure, such as a building. The vehicle 2100 may travel through one or more parts of the traffic network 2200 and may communicate with the external object 2110 via one or more electronic communication networks 2300. Although not explicitly shown in Figure 2, the vehicle may traverse areas that are not explicitly or completely included in the traffic network, such as off-road areas. In some embodiments, the traffic network 2200 may include one or more vehicle detection sensors 2202, such as guidance loop sensors, which may be used to detect the movement of vehicles on the traffic network 2200.

[0044] The electronic communication network 2300 may be a multiple access system that provides communication such as voice communication, data communication, video communication, messaging communication, or a combination thereof between the vehicle 2100, the external object 2110, and the operation center 2400. For example, the vehicle 2100 or the external object 2110 may receive information such as information representing the traffic network 2200 from the operation center 2400 via the electronic communication network 2300.

[0045] The operation center 2400 includes a controller device 2410 which includes some or all of the features of the controller 1300 shown in Figure 1. The controller device 2410 can monitor and adjust the movement of vehicles, including autonomous vehicles. The controller device 2410 may monitor the state or condition of vehicles such as vehicle 2100 and external objects such as external objects 2110. The controller device 2410 can receive vehicle data and infrastructure data, including any of the following: vehicle speed, vehicle position, vehicle operating status, vehicle destination, vehicle route, vehicle sensor data, external object speed, external object position, external object operating status, external object destination, external object route, and external object sensor data.

[0046] Furthermore, the controller device 2410 can establish remote control of one or more vehicles, such as the vehicle 2100, or external objects, such as the external object 2110. In this way, the controller device 2410 may remotely operate the vehicles or external objects from a remote location. The controller device 2410 may exchange (transmit or receive) status data with the vehicle 2100, the external object 2110, or computing devices, such as the vehicle 2100, the external object 2110, or the server computing device 2500, via wireless communication links such as the wireless communication link 2380 or wired communication links such as the wired communication link 2390.

[0047] The server computing device 2500 may include one or more server computing devices that exchange (transmit or receive) status signal data with one or more vehicles or computing devices, including a vehicle 2100, an external object 2110, or an operation center 2400, via an electronic communication network 2300.

[0048] In some embodiments, the vehicle 2100 or external object 2110 communicates via a wired communication link 2390, wireless communication links 2310 / 2320 / 2370, or any combination of any number or types of wired or wireless communication links. For example, as shown, the vehicle 2100 or external object 2110 communicates via a terrestrial wireless communication link 2310, via a non-terrestrial wireless communication link 2320, or a combination thereof. In some implementations, the terrestrial wireless communication link 2310 includes an Ethernet link, a serial link, a Bluetooth link, an infrared (IR) link, an ultraviolet (UV) link, or any link capable of providing electronic communication.

[0049] Vehicle 2100 or other vehicles, or external objects such as external object 2110, may communicate with other vehicles, external objects, or the operation center 2400. For example, the host or target vehicle 2100 may receive one or more vehicle-to-vehicle messages, such as basic safety messages (BSMs), from the operation center 2400 via the direct communication link 2370 or via the electronic communication network 2300. For example, the operation center 2400 may broadcast messages to host vehicles within a defined broadcast range, such as 300 meters, or to a defined geographical area. In some embodiments, vehicle 2100 receives messages via a third party, such as a signal repeater (not shown) or another remote vehicle (not shown). In some embodiments, vehicle 2100 or external object 2110 periodically transmits one or more vehicle-to-vehicle messages based on a defined interval, such as 100 milliseconds.

[0050] Vehicle-to-vehicle messages may include vehicle identification information, geospatial state information such as longitude, latitude, or altitude information, geospatial position accuracy information, vehicle acceleration information, yaw rate information, speed information, vehicle orientation information, braking system status data, throttle information, steering wheel angle information, or vehicle routing information, or vehicle operation status information such as vehicle size information, headlight status information, turn signal information, wiper status data, transmission information, or any other information or combination of information related to the transmitting vehicle status. For example, transmission status information indicates whether the transmitting vehicle's transmission is in neutral, parked, forward, or reverse.

[0051] In some embodiments, the vehicle 2100 communicates with the electronic communications network 2300 via an access point 2330. The access point 2330 may include a computing device and may be configured to communicate with the vehicle 2100, the electronic communications network 2300, the operations center 2400, or a combination thereof, via wired or wireless links 2310 / 2340. For example, the access point 2330 is a base station, base transceiver station (BTS), Node-B, extended Node-B (eNode-B), home Node-B (HNode-B), wireless router, wired router, hub, relay, switch, or any similar wired or wireless device. Although shown as a single unit, the access point may include any number of interconnected elements.

[0052] Vehicle 2100 may communicate with the electronic communications network 2300 via satellite 2350 or other non-terrestrial communications equipment. Satellite 2350 may include computing devices and may be configured to communicate with vehicle 2100, the electronic communications network 2300, the operations center 2400, or a combination thereof via one or more communication links 2320 / 2360. Although shown as a single unit, the satellite may include any number of interconnected elements.

[0053] The electronic communication network 2300 may be any type of network configured to provide voice, data, or any other type of electronic communication. For example, the electronic communication network 2300 includes a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a mobile or cellular telephone network, the Internet, or any other electronic communication system. The electronic communication network 2300 may use communication protocols such as the Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Internet Protocol (IP), Real-time Transport Protocol (RTP), Hypertext Transport Protocol (HTTP), or a combination thereof. Although shown as a single unit, the electronic communication network may include any number of interconnected elements.

[0054] In some embodiments, the vehicle 2100 communicates with the operation center 2400 via an electronic communication network 2300, an access point 2330, or a satellite 2350. The operation center 2400 may include one or more computing devices that can exchange (transmit or receive) data with vehicles such as the vehicle 2100, external objects including external objects 2110, or computing devices such as a server computing device 2500.

[0055] In some embodiments, the vehicle 2100 identifies a portion or state of the traffic network 2200. For example, the vehicle 2100 may include one or more on-board sensors 2102, such as the sensor 1360 shown in Figure 1, which includes a speed sensor, a wheel speed sensor, a camera, a gyroscope, a light sensor, a laser sensor, a radar sensor, an acoustic sensor (e.g., a microphone or acoustic sensor), a compass, or any other sensor or device or combination thereof that can determine or identify a portion or state of the traffic network 2200.

[0056] The vehicle 2100 may traverse one or more parts of the transportation network 2200 using information communicated via the electronic communication network 2300, such as information representing the transportation network 2200, information identified by one or more onboard sensors 2102, or a combination thereof. An external object 2110 may perform all or part of the above-described communications and operations with respect to the vehicle 2100.

[0057] For simplicity, Figure 2 shows vehicle 2100 as the host vehicle, and external objects 2110, a transportation network 2200, an electronic communications network 2300, and an operation center 2400. However, any number of vehicles, networks, or computing devices may be used. In some embodiments, the vehicle transport and communication system 2000 includes devices, units, or elements not shown in Figure 2. Although vehicle 2100 or external objects 2110 are shown as a single unit, a vehicle may include any number of interconnected elements.

[0058] Although the vehicle 2100 is shown as communicating with the operation center 2400 via the electronic communication network 2300, the vehicle 2100 (and external objects 2110) may communicate with the operation center 2400 via any number of direct or indirect communication links. For example, the vehicle 2100 or external object 2110 may communicate with the operation center 2400 via a direct communication link such as a Bluetooth communication link. For simplicity, Figure 2 shows one of the transportation networks 2200 and one of the electronic communication networks 2300, but any number of networks or communication devices may be used.

[0059] Figure 3 is a block diagram showing a remote vehicle support center 3000 according to an implementation of the present disclosure. The remote vehicle support center 3000 may also be called a remote system or remote operation center. The remote vehicle support center 3000 includes a fleet manager 3010, a plurality of vehicle managers (i.e., remote operators), including but not limited to remote operators 3020 and 3030, and a plurality of vehicles, including but not limited to vehicles 3040, 3050, 3060, and 3070.

[0060] The fleet manager 3010 may include a device that includes some or all of the features of the controller 1300 shown in Figure 1 or the controller device 2410 shown in Figure 2. The fleet manager 3010 can monitor and coordinate the movement of remote operators, including remote operators 3020 / 3030, as well as vehicles, including autonomous vehicles, and vehicles 3040, 3050, 3060, and 3070. Monitoring and coordinating remote operators may include assigning, allocating, or deassigning vehicles to remote operators, reviewing and monitoring remote operator performance data, and assigning remote operators to geographical areas. In one implementation, there may be multiple fleet managers, and these fleet managers may be managed by or under the authority of other fleet managers.

[0061] Remote operator 3020 can monitor the status or condition of vehicles, including vehicles 3040 and 3050. As shown in Figure 3, remote operator 3020 is assigned vehicles 3040 and 3050. The assignment of vehicles to remote operators may be performed by a fleet manager, such as fleet manager 3010.

[0062] The remote operator 3030 can monitor the status or condition of vehicles, including vehicles 3060 and 3070. As shown in Figure 3, the remote operator 3030 is assigned vehicles 3060 and 3070. The assignment of vehicles to the remote operator may be performed by a fleet manager, such as fleet manager 3010. The assignment of vehicles to the remote operator may be automated using machine learning techniques.

[0063] In one implementation, a remote operator can cluster or group vehicles, establish communication with the occupants inside the vehicles, remotely operate the vehicles, and coordinate the movement of vehicles around various obstacles such as traffic congestion or through the transportation network. The remote operator can also interact with other remote operators to assist in monitoring and managing the vehicles.

[0064] The fleet, including vehicles 3040 / 3050 / 3060 / 3070, and including vehicles such as vehicle 2100 shown in Figure 2, is monitored or coordinated by the fleet manager 3010. The vehicles may be operated autonomously or by a human driver and may exchange (transmit and receive) vehicle data relating to the state or conditions of the vehicle and its surroundings, including any of the following: vehicle speed, vehicle position, vehicle operating status, vehicle destination, vehicle route, vehicle sensor data, external object speed, and external object position.

[0065] In this specification, any statement describing an action performed by an autonomous vehicle (e.g., executing, deciding, starting, receiving, calculating, judging, etc.) should be understood as referring to some appropriate module of the AV performing that action. Such a module may be stored in the AV's memory, such as memory 1340 in Figure 1, or executed by a processor, such as processor 1330 in Figure 1. Such a module may be partially or completely included in a controller device, such as controller device 2410 in Figure 2, or may be partially or completely executed by the AV's processor, the operation center's processor, or a combination thereof. For example, the statement "the AV determines the trajectory" can be understood to mean "a module in the AV determines the trajectory" or "the AV's trajectory planning module determines the trajectory."

[0066] Figure 4 is a block diagram of an example of system 4000 for displaying an overlay 4080 including a vehicle, a simulated preceding vehicle, and a predicted path of the simulated preceding vehicle. For brevity, the “simulated preceding vehicle” may be referred to herein as the “simulated vehicle” or “preceding vehicle.” For example, system 4000 may be implemented by the controller device 2410 in Figure 2. In another example, system 4000 may be implemented by a computing device or server in the operation center 2400. System 4000 may be used by remote operators to perform remote control of vehicles, such as remote operators 3020 / 3030 in Figure 3 performing remote control of vehicles 3040 / 3050 / 3060 / 3070.

[0067] System 4000 may include a kinematics component 4005, a pathfinding component 4010, a transformation component 4020, and a drawing component 4030. System 4000 can receive video streams from cameras of vehicles in a traffic network (e.g., vehicle 2100 in traffic network 2200 in Figure 2) (e.g., sensor 1360 on vehicle 1050 in Figure 1). For example, the camera may be configured as a forward-facing camera positioned on top of the vehicle. System 4000 can also receive simulated driving inputs 4040 based on input from a remote operator. The simulated driving inputs 4040 may represent the steering angle and / or acceleration of a simulated vehicle, which may be received via a remote control device operated by the remote operator, such as a game controller, joystick, keyboard, mouse, steering wheel, accelerator pedal, brake pedal, or any combination thereof. The term “acceleration” encompasses both positive and negative acceleration, i.e., deceleration, where a positive acceleration input may include pressing the accelerator pedal in the driving simulator cockpit, and a negative acceleration input may include releasing the accelerator pedal or pressing the brake pedal in the driving simulator cockpit. System 4000 can also receive vehicle position and orientation inputs, i.e., attitude inputs 4035, which represent (i) the vehicle's position as spatial coordinates determined, for example, by a Global Positioning System (GPS) system, and (ii) the vehicle's orientation as yaw angle, pitch angle, and roll angle determined, for example, by a GPS system and / or an Inertial Measurement Unit (IMU) system. System 4000 can then determine the position and orientation, i.e., attitude, of a vehicle in a traffic network, a simulated vehicle, and a simulated vehicle's predicted path, based on the attitude inputs 4035 and the simulated driving inputs 4040. For example, system 4000 can utilize the kinematic component 4005 to determine the position and orientation, i.e., attitude, of simulated vehicles within a traffic network and the predicted paths of the simulated vehicles.The kinematics component 4005 receives simulated driving inputs 4040, such as steering angle and / or acceleration, and can calculate the simulated vehicle motion relative to the vehicle. The kinematics component 4005 can then use trigonometry and / or geometric calculations to determine the three-dimensional spatial coordinates and attitude angles of the simulated vehicle and generate a kinematics output 4045, which includes the attitude of the simulated vehicle. In some embodiments, the position of the simulated vehicle may be represented by a tuple of three-dimensional spatial coordinates, such as (x, y, z), which corresponds to the position of a predetermined point on the simulated vehicle, for example, the midpoint on the rear axle of the simulated vehicle. In some embodiments, the orientation of the simulated vehicle may be represented by a tuple of rotation angles around each orthogonal axis, such as (φ, θ, ζ), with the origin at a predetermined point on the simulated vehicle, for example, the midpoint on the rear axle of the simulated vehicle. The pathfinding component 4010 receives simulated driving inputs 4040, such as the steering angle and / or acceleration of the simulated vehicle, and a kinematic output 4045, and can use the Ackermann model to determine the radius of the curve of the predicted path of the simulated vehicle. The pathfinding component 4010 can then use trigonometry and / or geometric calculations to determine the three-dimensional spatial coordinates of the predicted path of the simulated vehicle and generate a path output 4050.

[0068] Referring further to Figure 5, in some cases the spatial coordinates determined by the kinematics component 4005 and / or the pathfinding component 4010 may be world coordinates 5010. For example, world coordinates 5010 may be coordinates relative to a transportation network (e.g., transportation network 2200). In some cases the spatial coordinates determined by the kinematics component 4005 and / or the pathfinding component 4010 may be camera coordinates 5020. For example, camera coordinates 5020 may be coordinates relative to a camera on a vehicle. When the spatial coordinates are world coordinates 5010, the transformation component 4020 can transform world coordinates 5010 to camera coordinates 5020.

[0069] Referring again to Figure 4, the system 4000 can determine from spatial coordinates two-dimensional pixel coordinates in the video stream and / or map stream corresponding to the positions of the vehicle, the simulated vehicle, and the predicted path of the simulated vehicle. For example, the system 4000 can utilize a transformation component 4020 to process spatial coordinates from the kinematic output 4045 and / or path output 4050. The transformation component 4020 can refer to camera parameters 4060 specific to the vehicle's camera to determine the pixel coordinates. For example, the camera parameters 4060 may indicate the camera's orientation, such as the camera's position on the vehicle and its height from the ground, as well as the camera's resolution. Based on the camera parameters 4060, the transformation component 4020 can transform the spatial coordinates into pixel coordinates and generate a transformation output 4070.

[0070] Referring further to Figure 5, the transformation component 4020 can transform spatial coordinates (e.g., world coordinates 5010 or camera coordinates 5020) into film coordinates 5030. The film coordinates 5030 may correspond to coordinates in the camera's focal plane (e.g., the video stream). The transformation component 4020 can further transform the film coordinates 5030 into pixel coordinates 5040. The pixel coordinates 5040 may correspond to the coordinates of pixels in a GUI or hypertext markup language (HTML) canvas used by a remote operator.

[0071] Referring again to Figure 4, the system 4000 can utilize the drawing component 4030 to generate a first overlay 4080 which is output to a GUI or HTML canvas used by a remote operator. The first overlay 4080 may include a representation of the simulated vehicle at a position in the video stream based on pixel coordinates (e.g., pixel coordinates 5040) and the orientation of the simulated vehicle. The drawing component 4030 can draw the simulated vehicle portion of the first overlay 4080 into the video stream. In some embodiments, the simulated vehicle may be represented by limited components of the vehicle (e.g., only the wheels and chassis), and in other embodiments, the simulated vehicle may be represented by the complete vehicle (e.g., the car body). In some embodiments, the simulated vehicle may be represented by a wireframe, a bounding box, one or more shapes, a rendering, or any combination thereof. In some embodiments, the simulated vehicle portion of the first overlay 4080, or a part thereof, may be drawn in the video stream as opaque, semi-transparent, or transparent. The first overlay 4080 may further include a representation of the predicted path of the simulated vehicle, based on pixel coordinates (e.g., pixel coordinates 5040) and the orientation of the simulated vehicle. The drawing component 4030 can apply spline interpolation based on the pixel coordinates, and the visualization of the predicted path portion of the first overlay 4080 can be drawn as a smooth curve representing the predicted path in the video stream. In some embodiments, the predicted path portion, or part thereof, of the first overlay 4080 may be drawn as opaque, semi-transparent, or transparent in the video stream.

[0072] Referring again to Figure 4, the system 4000 can utilize the drawing component 4030 to generate a second overlay 4080 which is output to a GUI or HTML canvas used by a remote operator. In some embodiments, the first and second overlays may be considered the same overlay. The second overlay 4080 may include a representation of the vehicle at a position on the map stream based on pixel coordinates (e.g., pixel coordinates 5040) and vehicle orientation. The drawing component 4030 can draw the vehicle portion of the second overlay 4080 onto the map stream. In some embodiments, the vehicle may be represented by limited components of the vehicle (e.g., wheels and chassis only), and in other embodiments, the vehicle may be represented by the complete vehicle (e.g., body). In some embodiments, the vehicle may be represented by a wireframe, a bounding box, one or more shapes, a rendering, or any combination thereof. In some embodiments, the vehicle portion of the second overlay 4080, or a part thereof, may be drawn as opaque, semi-transparent, or transparent within the video stream.

[0073] The second overlay 4080 may further include a representation of the simulated vehicle at its position on the map stream, based on its pixel coordinates (e.g., pixel coordinates 5040) and the orientation of the simulated vehicle. The drawing component 4030 can draw the simulated vehicle portion of the second overlay 4080 onto the map stream. In some embodiments, the simulated vehicle may be represented by limited components of the vehicle (e.g., only the wheels and chassis), while in other embodiments, the simulated vehicle may be represented by the complete vehicle (e.g., the car body). In some embodiments, the simulated vehicle may be represented by a wireframe, a bounding box, one or more shapes, a rendering, or any combination thereof. In some embodiments, the simulated vehicle portion of the second overlay 4080, or a part thereof, may be drawn on the map stream as opaque, semi-transparent, or transparent. The second overlay 4080 may further include a representation of the predicted path of the simulated vehicle, based on its pixel coordinates (e.g., pixel coordinates 5040) and the orientation of the simulated vehicle. The drawing component 4030 can apply spline interpolation based on pixel coordinates, and can visualize the predicted path portion of the second overlay 4080 as a smooth curve representing the predicted path on the map stream. The drawing component 4030 can further visualize the predicted path portion of the second overlay 4080 as a smooth curve representing the predicted path on the map stream. In some embodiments, the predicted path portion of the second overlay 4080, or a part thereof, may be drawn as opaque, semi-transparent, or transparent within the video stream.

[0074] Figure 6 shows an example of GUI 6000 displaying overlays 6010 and 6020. In some embodiments, overlays 6010 and 6020 may be considered the same overlay, such as overlay 4080 in Figure 4. Overlay 6010 may include a simulated vehicle overlay 6012 and a simulated vehicle predicted path overlay 6014, each of which may be drawn independently on one or more video streams. Overlay 6020 may include a simulated vehicle overlay 6022, a simulated vehicle predicted path overlay 6024, and a vehicle overlay 6026, each of which may be drawn independently on one or more map streams. GUI 6000 can be used to assist a remote operator in performing remote control of a vehicle, such as when a remote operator 3020 / 3030 in Figure 3 performs remote control of a vehicle 3040 / 3050 / 3060 / 3070. For example, system 4000 in Figure 4 can generate overlays 6010 and 6020, which together correspond to overlay 4080 in Figure 4. GUI 6000 may include an HTML canvas or web page displayed through a web interface, GUI may include one or more display monitors, and / or GUI 6000 may include one or more augmented reality displays, for example, via an augmented reality headset.

[0075] GUI6000 may include video streams 6030 from cameras on vehicles within the traffic network (e.g., vehicle 2100 in traffic network 2200 in Figure 2). For example, video stream 6030 may correspond to a camera configured as a forward-facing camera located on top of the vehicle. In some implementations, GUI6000 may include additional video streams from additional cameras on the vehicle. For example, GUI6000 may include a second video stream 6040 from a second camera configured as a left-facing camera and a third video stream 6050 from a third camera configured as a right-facing camera. The additional cameras may provide the remote operator with an additional viewpoint to further improve situational awareness. The overlay 6010, or a part thereof, may be drawn independently and / or simultaneously within the video streams of multiple cameras at the pixel coordinates of each video stream (e.g., pixel coordinate 5040). For example, Figure 6 shows the overlay 6010 drawn on video stream 6030 and the second video stream 6040.

[0076] GUI6000 may include one or more map streams from a mapping service that show an aerial (plan) view of the environment around a vehicle (e.g., vehicle 2100 in the traffic network 2200 in Figure 2). For example, GUI6000 may include a satellite map stream 6060 containing aerial satellite imagery of the environment and a road map stream 6070 containing lines, shapes, symbols, colors, and an abstract representation of the environment through such elements. Map streams 6060 and 6070 may provide the remote operator with additional viewpoints to further improve situational awareness. Overlay 6020, or a portion thereof, may be drawn independently and / or simultaneously on multiple map streams at the pixel coordinates of each map stream (e.g., pixel coordinate 5040). For example, Figure 6 shows the overlay 6020 (including the vehicle overlay 6026, the simulated vehicle overlay 6022, and the predicted route 6024) drawn on the satellite map stream 6060, and the simulated vehicle overlay 6026 drawn on the road map stream 6070.

[0077] The GUI 6000 may include a remote control support area 6080 for displaying information that can further improve the situational awareness of remote control. For example, the remote control support area 6080 may display information related to the simulated driving input 4040 in Figure 4, such as the steering angle and / or acceleration of the simulated vehicle. The remote control support area 6080 may further display the speed of the simulated vehicle or the distance between simulated vehicles. Such speed and / or distance may be determined by the kinematic component 4005 of the system 4000. The remote control support area 6080 may further display information related to the current procedure of remote control, such as whether the vehicle is requesting remote control (e.g., "requesting"), whether remote control is in progress ("operating"), or whether the vehicle has returned from remote control and is back in autonomous driving mode ("returning"). The remote control support area 6080 may further display information relating to one or more factors that the vehicle requested remote control, such as the detection and / or identification of an obstacle that the vehicle cannot autonomously bypass appropriately, adequately, or safely (e.g., “obstacle”).

[0078] In some cases, the video streams 6030, 6040, and 6050 that can be viewed by the remote operator may be delayed relative to real-time video captured by one or more cameras on a vehicle in the traffic network (e.g., vehicle 2100 in the traffic network 2200 in Figure 2). This delay may be caused by data transmission delays (e.g., network latency) between the vehicle in the traffic network, e.g., on the road, and the system 4000. As a result, if the remote operator assumes direct remote control of the vehicle, the remote operator will experience response delays caused by a first delay in transmitting remote control commands (e.g., steering angle or acceleration) to the vehicle and a second delay in receiving a subsequent video stream from the vehicle that may reflect the updated position and / or orientation of the vehicle as a result of executing the received remote control commands. Such response delays may make direct remote control of the vehicle difficult and / or dangerous. However, in the embodiments described herein, the remote operator assumes direct operation of a simulated preceding vehicle rather than the vehicle, and the vehicle follows a path determined by the position of the simulated vehicle. In this way, the remote operator is operating the vehicle indirectly. Since system 4000 can receive the simulated driving input 4040 without suffering the aforementioned transmission delay between the vehicle on the road and system 4000, the drawing component 4030 can draw the overlay 6010 (e.g., overlay 4080 in Figure 4) within the video streams 6030, 6040, and / or 6050 in near real-time with respect to the simulated driving input 4040. Thus, the remote operator does not experience response delays that may be caused by data transmission delays (e.g., network latency).

[0079] The route search component 4010 can determine a predicted route 6014 for the simulated vehicle based on motion data associated with the simulated vehicle. For example, given a simulated driving input 4040 (e.g., previous and / or current steering angles and acceleration), the route search component 4010 can determine possible routes for the simulated vehicle. Referring further to Figure 7, 7020 represents a vehicle with a camera 7010 (e.g., vehicle 2100 in the traffic network 2200 in Figure 2), and 7030 represents a simulated vehicle. The angle θ is the steering angle indicated by the simulated driving input 4040, and point A is the midpoint of the front axle of the simulated vehicle. The distance between point A and point B may be "d" meters, where d is a value representing the predicted length of the predicted path 6014, determined by either (i) a pre-configured or user-configurable distance, or (ii) a distance calculated by the kinematics component 4005 based on a pre-configured or user-configurable time period, the simulated vehicle's speed, and the acceleration of the preceding vehicle (for example, by the kinematic equation d = s(t) + (1 / 2)(a)(t^2), where d is the distance, s is the speed, t is the time period, and a is the acceleration). Given the angle θ and the distance d, the pathfinding component 4010 can determine the thick curved path 7040 (for example, the predicted path 6014 in Figure 6) that the simulated vehicle will take to reach point B. Note that the pathfinding component 4010 may not have the position of point B or the angle α. However, the pathfinding component 4010 can use the angle θ to determine the radius of the curve. Once the curve is determined, the pathfinding component 4010 can determine point B on the curve such that the length of AB is distance d. Once the line segment AB is determined, the pathfinding component 4010 can use trigonometric functions to find the angle α.

[0080] Figure 8 shows representations of vehicle 8020 and simulated vehicle 8030, which may represent vehicle 7020 and simulated vehicle 7030 in Figure 7, respectively. The coordinate system used by the pathfinding component 4010 may have an origin 8022 for vehicle 8020, where the origin 8022 corresponds to the position of camera 7010 on vehicle 7020, which may be mounted at its midpoint on the top of vehicle 7020. In some cases, camera 7010 may be at other locations on vehicle 7020, and the transformation component 4020 can determine an offset between the origin 8022 and the other location, for example, to the midpoint of the rear axle 8024 of vehicle 8020. In this example, the positive X-axis may point to the right of the vehicle, the positive Z-axis may point upward perpendicular to the ground, and the positive Y-axis may point forward of the vehicle.

[0081] Since the predicted path of the simulated vehicle (e.g., the predicted path overlay 6014 in Figure 6 or the thick curved path 7040 in Figure 7) corresponds to a path that the simulated vehicle could take, rather than a path that the vehicle could actually take, it may be more convenient for the coordinate system used by the pathfinding component 4010 to have the origin 8032 of the simulated vehicle 8030, which may correspond to any convenient point on the simulated vehicle 8030, for example, its midpoint on the top of the simulated vehicle 8030, or the midpoint of the rear axle 8034 of the simulated vehicle 8030. In such cases, the kinematic component 4005 can determine an offset between any point on vehicle 8020 and any point on simulated vehicle 8030, for example, between the origin 8022 (or origin 8024) of vehicle 8020 and the origin 8032 (or origin 8034) of simulated vehicle 8030, where the offset is a function of the relative distance and orientation of the simulated vehicle 8030 to vehicle 8020. In this example, the positive X' axis may point to the right of the simulated vehicle, the positive Z' axis may point upward perpendicular to the ground, and the positive Y' axis may point forward of the simulated vehicle.

[0082] Referring again to Figure 7, the simulated vehicle 7030 may be modeled based on vehicle 7020, so that the dimensions of the simulated vehicle 7030 may be identical to those of vehicle 7020. As a result, any point on vehicle 7020 can be projected onto an equivalent point on the simulated vehicle 7030, for example, the coordinates of a camera 7010 mounted on vehicle 7020 can be projected onto an equivalent point on the simulated vehicle 7030. For example, the coordinates of the wheels of the simulated vehicle can be determined using the vehicle dimensions, the position of the camera 7010 mounted on vehicle 7020, and the projection of coordinates from vehicle 7020 to the simulated vehicle 7030. In one implementation, the coordinates of the camera 7010 of vehicle 7020 may first be projected onto the simulated vehicle 7030, for example via the kinematics component 4005, and then the dimensions of vehicle 7020 may be applied to the simulated vehicle 7030 to determine the coordinates of the wheels of the simulated vehicle 7030, for example via the pathfinding component 4010. In another implementation, the dimensions of vehicle 7020 may first be applied to vehicle 7020 to determine the coordinates of the wheels of vehicle 7020, for example via the pathfinding component 4010, and then those coordinates may be projected onto the simulated vehicle 7030, for example via the kinematics component 4005.

[0083] As an example of the latter implementation, the camera 7010 may be mounted on a line connecting the midpoints of the front and rear axles of the vehicle, resulting in the following exemplary dimensions. Distance from the front axle of camera 7010: 1666mm Distance from the rear axle of camera 7010: 1030mm Vehicle tread width: 1256mm In this case, the (X,Y) coordinates of the front left wheel may be (-1256 / 2, 1666) = (-628, 1666), the (X,Y) coordinates of the front right wheel may be (1256 / 2, 1666) = (628, 1666), the (X,Y) coordinates of the rear left wheel may be (-1256 / 2, -1030) = (-628, -1030), and the (X,Y) coordinates of the rear right wheel may be (1256 / 2, -1030) = (628, -1030). The ground may be considered flat. As a result, the Z-coordinate of any point on the ground may be (-1 * camera height). That is, if camera 7010 is mounted at a height of 1600 mm, a point on the ground may have a coordinate of -1600. These coordinates can be projected onto the simulated vehicle 7030.

[0084] Referring further to Figure 9, the pathfinding component 4010 can use the Ackermann model to determine the radius of the curved path relative to the rear axle of the simulated vehicle 9030 (for example, the simulated vehicle 7030 in Figure 7 or the simulated vehicle 8030 in Figure 8). For example, from the Ackermann model, we can see the following: α i =tan -1 (L / (RT / 2))...(Formula 1) α0 = tan -1 (L / (R+T / 2))...(Formula 2)

[0085] The pathfinding component 4010 can use equation (1) or equation (2) to determine the radius "R". For example, if R is determined using equation (1), it will be as follows: R=(L / tan(α i ))+T / 2...(Formula 3)

[0086] Radius R is the radius of the curve traced by the midpoint of the rear axle. Each point on the vehicle may trace a separate curve along the path. The pathfinding component 4010 can determine the curves traced by the midpoints of the front wheels and front axle. For example, the pathfinding component 4010 can determine the aforementioned radius of these curves using dimensions (T, L) and the Pythagorean theorem. These curves may be arcs of different concentric circles. The centers of these circles may be located at a distance R from the midpoint of the rear axle, as shown in Figure 9. The centers may be located on either the left or right side of the rear axle, depending on the turning direction. By determining the turning direction and R, the pathfinding component 4010 can determine the coordinates of the centers using the coordinate system described with respect to Figure 8.

[0087] Referring again to Figure 7, the path 7060 (shown as a dashed line) that vehicle 7020 may follow can be based on the path 7050 (shown as a dotted line) that simulated vehicle 7030 traveled over a certain period of time. In some cases, paths 7050 and 7060 may be identical, and in other cases, path 7060 may deviate from path 7050, as will be described later herein (the deviations shown in Figure 7 are exaggerated for illustrative purposes). Path 7050 may include a time series of position coordinates, or “waypoints,” describing the position of a given reference point on simulated vehicle 7030 at consecutive points in time. For example, Figure 7 shows six waypoints, each waypoint shown as a filled circle (i.e., a dot), one of which waypoint 7052 is specifically labeled “t3:(x3, y3),” as will be described below. Each such waypoint corresponds to the location where a predetermined reference point 7054 of the simulated vehicle 7030 was located at a previous time. For example, the reference point 7054 may be the midpoint of the rear axle, as shown by the white circle in Figure 7. In some embodiments, the position coordinates of each waypoint may include a tuple of two-dimensional coordinates, such as (x,y), on a predetermined plane, such as the ground plane, for example, the XY (or X'-Y') plane shown in Figure 8. In Figure 7, the specifically labeled waypoint 7052 corresponds to the position coordinates (x3,y3) 7056 describing the location where the reference point 7054 was located at time t3.

[0088] A series of waypoints may also be called a "breadcrumb trail." Figure 7 shows a breadcrumb trail 7058 (shown as a thick solid line), where each waypoint, e.g., waypoint 7052, is connected to the previous and / or next waypoint by a straight line. When the time interval between consecutive waypoints is short, the breadcrumb trail 7058 more closely mimics the travel path 7050. If the time interval between consecutive waypoints is excessively long, interpolation between waypoints can be used to construct a breadcrumb trail that more closely mimics the travel path 7050 of the simulated vehicle 7030. Any suitable interpolation method between waypoints, such as spline interpolation, may be used.

[0089] When the remote operator begins remote control of the simulated vehicle 7030, the breadcrumb trail 7058 starts with the simulated vehicle 7030 in a reference attitude. In some implementations, the reference attitude corresponds to the simulated vehicle 7030's attitude (i.e., position and orientation) being identical to that of vehicle 7020. Essentially, the simulated vehicle 7030 is (virtually) superimposed on vehicle 7020. Referring again to Figure 6, when the remote operator begins driving the simulated vehicle, for example, the simulated preceding vehicle 7030, the simulated vehicle overlay 6012 may not be visible in the video streams 6030 / 6040 / 6050 because the simulated vehicle and the vehicle, for example, vehicle 7020, have the same attitude. However, when the remote operator starts driving the simulated vehicle, for example by pressing the accelerator pedal in the driving simulator cockpit and / or turning the steering wheel in the driving simulator cockpit, generating a driving input 4040, the simulated vehicle may move forward, and therefore the simulated vehicle overlay 6012 may become visible in the video streams 6030 / 6040 / 6050. In some cases, the remote operator driving the simulated vehicle may want to start over, i.e., reset the simulated vehicle to its reference posture. For example, the remote operator may have driven the simulated vehicle into an obstacle or hazard. In such a case, the remote operator may input a reset signal to the system 4000, e.g., the kinematics component 4005. The reset signal may be included in the simulated driving input 4040. The reset signal may be performed by any suitable means, e.g., by the remote operator pressing a reset button. Upon reset, the current breadcrumb trail 7058 is substantially removed.

[0090] The breadcrumb trail 7058 and / or breadcrumb reset signal may be transmitted to the vehicle 7020 (e.g., vehicle 2100 in Figure 2) via a communication network (e.g., electronic communication network 2300 in Figure 2). For example, the transmission of the breadcrumb trail 7058 may be initiated and / or performed by a system 4000 which may be implemented by the controller device 2410 in Figure 2. For example, the breadcrumb trail 7058 may be received by a communication interface 1370 coupled to a communication unit 1320, as shown in vehicle 1050 in Figure 1. Waypoints of the breadcrumb trail 7058 (e.g., waypoint 7052) may be transmitted one at a time at periodic or nonperiodic intervals, for example, coordinate (x1, y1) may be transmitted at or around time t1, coordinate (x2, y2) may be transmitted at or around time t2, and so on. Alternatively, multiple waypoints may be transmitted at once; for example, coordinates (x1, y1) and (x2, y2) may be transmitted at or around time t2. In either case, time points t1, t2, etc., may be explicitly encoded as timestamp values ​​within the transmitted data, or they may be implicitly encoded as a function of the arrival time of the transmitted data in the vehicle, which is determined or approximated by the time the data is received by the communication unit 1320 in Figure 1, for example. For example, the communication unit 1320 may pass multiple received waypoints along with their shared arrival time to the processor 1330. The processor 1330 may record consecutive shared arrival times and calculate an approximate timestamp for each waypoint based on the arrival interval and the number of waypoints received at once. For example, if a previous shared arrival time is recorded as tA, and waypoints w1, w2, and w3 arrive at the vehicle at time tB, the processor 1330 may use simple linear interpolation to determine the timestamp of w1 as tA, the timestamp of w2 as tA+(1 / 3)(tB-tA) (i.e., one-third of the period from tA to tB), and the timestamp of w3 as tA+(2 / 3)(tB-tA) (i.e., two-thirds of the period from tA to tB).In some embodiments, nonlinear interpolation may be used. In some embodiments, transmission delay (e.g., network latency) may be taken into consideration when determining the timestamps of multiple waypoints received at once.

[0091] An autonomous vehicle that requests remote control and subsequently receives breadcrumb trail waypoints as a result of remote control activities such as those described above may process the waypoints to determine the path to follow. The vehicle may be vehicle 1050 in Figure 1 and includes a communication unit 1320 configured to receive waypoints and a processor 1330 configured to process waypoints. The processor 1330 may implement a route planning algorithm that determines and selects the optimal path to follow based on the waypoints. The route planning algorithm may determine the path to follow that closely or roughly matches the breadcrumb trail, e.g., breadcrumb trail 7058. The amount of deviation between the breadcrumb trail and the determined path to follow may be based on a preset or user-configurable tolerance parameter. Alternatively or additionally, the trajectory controller of vehicle 1050 may optimize the path the vehicle follows. The vehicle may decide to deviate from the optimal path based on its own determination of whether the optimal path is safe to follow (e.g., whether there are any obstacles), using its own sensors and safety modules. Many embodiments of the safety-protected remote operation disclosed herein are independent of specific implementations of route planning algorithms and / or route optimization algorithms.

[0092] In some embodiments, waypoints transmitted to an autonomous vehicle may include information in addition to spatial coordinates and explicit or implicit timestamp values. Such additional information may be used by the autonomous vehicle to further optimize the path it takes and how it takes that path. For example, waypoints may include additional information (e.g., parameters) about the simulated vehicle, such as the steering angle, speed, acceleration, yaw, and yaw rate (i.e., rate of change of yaw over time). Some or all of these parameters may be used by the autonomous vehicle to smooth the received breadcrumb trail, to compensate for oversteer or understeer of the simulated vehicle, or to account for other artifacts caused by the remote operator or remote control system (in some embodiments, these actions may be considered path optimization). Furthermore, some or all of the above parameters may be used by the autonomous vehicle to determine one or more speeds to apply when taking the path determined from the breadcrumb trail. These one or more speeds may be called “speed profiles”. If one or more velocities include a directional component, for example, one or more velocity vectors, the profile may be called a “velocity vector profile.” Similarly, some or all of the parameters described above may be used to determine one or more accelerations to be applied when an autonomous vehicle follows a path determined from a breadcrumb trail. These one or more accelerations may be called an “acceleration profile.” In general, speed, velocity vectors, and / or acceleration profiles may be called a “motion profile,” individually or collectively.

[0093] For example, if a remote operator needs to drive a simulated vehicle over a speed bump or curb to bypass an obstacle, the remote operator may drive the simulated vehicle at a higher speed when approaching the speed bump or curb and at a lower speed when crossing the speed bump or curb. If the autonomous vehicle does not receive speed data for the simulated vehicle included in the waypoint, the autonomous vehicle may cross the speed bump or curb at an uncomfortably fast or dangerous speed. However, in some cases, the autonomous driving function of the autonomous vehicle may identify speed bumps or curbs in its path and adjust its speed appropriately even without receiving speed data for the simulated vehicle included in the waypoint.

[0094] Referring again to Figure 7, vehicle 7020 determines and selects the optimal route 7060 to follow based on the received breadcrumb trail 7058. In some implementations, vehicle 7020 must receive a waypoint 7052 of a threshold quantity to constitute the breadcrumb trail 7058 necessary for determining and selecting the optimal route to follow and / or for causing vehicle 7020 to begin following route 7060. In some implementations, the threshold quantity required to determine and select the optimal route to follow may be different from the threshold quantity required for causing vehicle 7060 to begin following route 7060. In some embodiments, instead of the threshold quantity of waypoint 7052, a threshold distance between the simulated vehicle 7030 and vehicle 7020 may be required to determine and select the optimal route to follow and / or for causing vehicle 7020 to begin following route 7060.

[0095] For the sake of brevity, process 10000 is illustrated and described herein as a series of steps or actions. However, the steps or actions according to this disclosure may be performed in various orders and / or simultaneously. Furthermore, other steps or actions not presented and described herein may be used. Moreover, not all illustrated steps or actions are necessarily required to implement the technology according to the subject matter disclosed.

[0096] In 10010, the system can receive video streams from cameras on vehicles within the traffic network. For example, system 4000 can receive video stream 6030 from camera 7010 of vehicle 2100 within the traffic network 2200. The system can receive video streams to enable the remote control of vehicles. For example, the system may be used by remote operators 3020 / 3030 to perform remote control of vehicles 3040 / 3050 / 3060 / 3070.

[0097] In 10020, the system can receive acceleration input and steering input via a remote control device operated by a remote operator, such as a game controller, joystick, keyboard, mouse, steering wheel, accelerator pedal, brake pedal, or any combination thereof. The remote control device may present a physical interface to the remote operator, or it may present a virtual interface to the remote operator, for example, as a GUI having a virtual accelerator pedal, virtual brake pedal, and virtual steering wheel.

[0098] In 10030, the system can determine the simulated vehicle attitude relative to a reference attitude based on acceleration input and steering input. The attitude includes the position and orientation of the simulated vehicle. In some embodiments, the reference attitude corresponds to the vehicle's attitude relative to at least one of a traffic network or a camera.

[0099] In 10040, the system can display an overlay of a simulated vehicle representation within the video stream in a pixel coordinate system based on its orientation. When the simulated vehicle is in a reference orientation, the overlay may not be visible within the video stream. In some implementations, the acceleration component of the simulated driving input must exceed a predetermined threshold before the simulated vehicle representation is displayed within the video stream.

[0100] In 10050, the system can determine the spatial coordinates of a simulated vehicle within a traffic network from its pixel coordinates, where the spatial coordinates are relative to at least one of the traffic network or a camera. For example, the kinematic component 4005 of system 4000 can use trigonometry and / or geometric calculations to determine the three-dimensional spatial coordinates and attitude angles of the simulated vehicle and generate a kinematic output 4045 that includes the position and orientation of the simulated vehicle.

[0101] In 10060, the system can transmit spatial coordinates to the vehicle in order to cause the vehicle to follow a path based on spatial coordinates. For example, the vehicle may be vehicle 1050 in Figure 1 and include a communication unit 1320 configured to receive spatial coordinates and a processor 1330 configured to process spatial coordinates. The processor 1330 may implement a route planning algorithm that determines and selects the optimal path to follow based on the spatial coordinates. The route planning algorithm may determine a path to follow that closely or roughly matches the spatial coordinates. The amount of deviation between the spatial coordinates and the determined path to follow may be based on a preset or user-configurable tolerance parameter. Alternatively or additionally, the trajectory controller of vehicle 1050 may optimize the path that the vehicle follows. The vehicle may decide to deviate from the optimal path based on the vehicle's determination of whether the optimal path is safe to follow (e.g., whether there are obstacles) using its own sensors and safety modules.

[0102] Process 1000 may include additional steps and embodiments not shown in Figure 10. For example, the system may determine a predicted path for a simulated vehicle based on the attitude of the simulated vehicle and at least one of the acceleration input and steering input, and display an overlay of a representation of the predicted path within the video stream. The predicted path may be determined using spatial coordinates with respect to at least one of the traffic network or the camera. For example, the pathfinding component 4010 of system 4000 may receive simulated driving inputs 4040, such as the steering angle and / or acceleration of the simulated vehicle, and a kinematic output 4045, and use the Ackermann model to determine the radius of the curve of the predicted path for the simulated vehicle. The pathfinding component 4010 may then use trigonometry and / or geometric calculations to determine the three-dimensional spatial coordinates of the predicted path for the simulated vehicle and generate a path output 4050.

[0103] The system can transmit information to the vehicle, in addition to spatial coordinates, such as parameters, which may include information representing at least one of the following components of the simulated vehicle's attitude: steering input, acceleration input, simulated vehicle speed, and some or all of the simulated vehicle's attitude, such as the yaw component. For example, the kinematic component 4005 of system 4000 can determine the simulated vehicle's speed, acceleration, and attitude via attitude input 4035. Some or all of these parameters may be used by the autonomous vehicle to smooth the received set of spatial coordinates, to compensate for oversteer or understeer of the simulated vehicle, or to account for other artifacts caused by the remote operator or remote control system (in some embodiments, these actions may be considered as path optimization). Furthermore, some or all of the above parameters may be used by the autonomous vehicle to determine a velocity profile, velocity vector profile, and / or acceleration profile (e.g., collectively, a motion profile) to be applied when following the path determined from the spatial coordinates.

[0104] Where used herein, the terms “example,” “embodiment,” “implementation,” “aspect,” “feature,” or “element” indicate that they are examples, illustrations, or illustrative. Unless expressly indicated otherwise, any example, embodiment, implementation, aspect, feature, or element is independent of any other example, embodiment, implementation, aspect, feature, or element and may be used in combination with any other example, embodiment, implementation, aspect, feature, or element.

[0105] As used herein, the terms “determine” and “identify,” or any variation thereof, include, in any manner, selecting, verifying, calculating, searching, receiving, determining, establishing, obtaining, or otherwise identifying or determining, using one or more of the devices shown and described herein.

[0106] As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or it is evident from the context, “X contains A or B” is intended to indicate any of the natural inclusive permutations. That is, if X contains A, if X contains B, or if X contains both A and B, “X contains A or B” is satisfied in any of the aforementioned cases. In addition, the articles ("a" and "an") used in this application and the attached claims should generally be interpreted as meaning “one or more” unless otherwise specified or it is evident from the context that they refer to a singular form.

[0107] Furthermore, for the sake of brevity, the drawings and descriptions herein may include sequences or series of steps or stages, but the elements of the methods disclosed herein may be performed in various orders or simultaneously. In addition, the elements of the methods disclosed herein may be performed in conjunction with other elements not expressly presented and described herein. Furthermore, not all elements of the methods described herein are necessary to implement the methods according to this disclosure. While embodiments, features, and elements are described herein in specific combinations, each embodiment, feature, or element may be used independently or in various combinations, with or without other embodiments, features, and elements.

[0108] The embodiments, examples, and implementations described above are provided for the purpose of facilitating the understanding of this disclosure and are not intended to limit it. Conversely, this disclosure encompasses various modifications and equivalent structures that fall within the scope of the appended claims, and its scope should be given the broadest possible interpretation to include all such modifications and equivalent structures that are legally permissible.

Claims

1. Receiving video streams from cameras on vehicles within the transportation network, Receiving acceleration input and steering input, Based on the acceleration input and steering input, the simulated vehicle attitude relative to the reference attitude is determined. In the pixel coordinates based on the aforementioned posture, an overlay of the simulated vehicle representation is displayed within the video stream. Determining the spatial coordinates of the simulated vehicle within the transportation network from the aforementioned pixel coordinates, wherein the spatial coordinates are relative to at least one of the transportation network or the camera, To cause the vehicle to follow the path based on the aforementioned spatial coordinates, the spatial coordinates are transmitted to the vehicle. Methods that include...

2. The method according to claim 1, wherein the reference posture corresponds to the posture of the vehicle with respect to at least one of the traffic network or the camera.

3. The method according to claim 1, further comprising resetting the attitude of the simulated vehicle to the reference attitude in response to receiving a reset input.

4. Based on the simulated vehicle's attitude and at least one of the acceleration input and the steering input, the predicted path of the simulated vehicle is determined. Display an overlay of the predicted path representation within the video stream. The method according to claim 1, further comprising:

5. Determining the speed of the simulated vehicle, Transmitting information to the vehicle representing at least one of the steering input, acceleration input, velocity, and yaw component of the attitude. The method according to claim 1, further comprising:

6. The method according to claim 1, wherein the representation of the simulated vehicle is displayed in the video stream after the acceleration input exceeds a predetermined threshold.

7. The method according to claim 1, wherein the simulated vehicle representation includes representations of a set of wheels, chassis, front bumper, and lead bumper, each of which is modeled based on the vehicle.

8. Memory and A processor configured to execute instructions stored in the aforementioned memory A device equipped with, The aforementioned instruction is, Receiving video streams from cameras on vehicles within the transportation network, Receiving acceleration input and steering input, Based on the acceleration input and steering input, the simulated vehicle attitude relative to the reference attitude is determined. In the pixel coordinates based on the aforementioned posture, an overlay of the simulated vehicle representation is displayed within the video stream. Determining the spatial coordinates of the simulated vehicle within the transportation network from the aforementioned pixel coordinates, wherein the spatial coordinates are relative to at least one of the transportation network or the camera, To cause the vehicle to follow the path based on the aforementioned spatial coordinates, the spatial coordinates are transmitted to the vehicle. A device for performing a certain task.

9. The apparatus according to claim 8, wherein the reference posture corresponds to the posture of the vehicle with respect to at least one of the traffic network or the camera.

10. The apparatus according to claim 8, wherein the command includes a command for resetting the attitude of the simulated vehicle to the reference attitude in response to receiving a reset input.

11. The aforementioned instruction is, Based on the simulated vehicle's attitude and at least one of the acceleration input and the steering input, the predicted path of the simulated vehicle is determined. The apparatus according to claim 8, comprising an instruction for displaying an overlay of the predicted path representation within the video stream.

12. The aforementioned instruction is, Determine the speed of the simulated vehicle, The apparatus according to claim 8, comprising a command for transmitting information to the vehicle representing at least one of the steering input, the acceleration input, the velocity, and the yaw component of the attitude.

13. The apparatus according to claim 8, wherein the representation of the simulated vehicle is displayed in the video stream after the acceleration input exceeds a predetermined threshold.

14. The apparatus according to claim 8, wherein the representation of the simulated vehicle includes a representation of a set of wheels, chassis, front bumper, and lead bumper that mimic the vehicle.

15. A non-temporary computer-readable medium for storing instructions that can cause one or more processors to perform an action, The aforementioned operation is, Receiving video streams from cameras on vehicles within the transportation network, Receiving acceleration input and steering input, Based on the acceleration input and steering input, the simulated vehicle attitude relative to the reference attitude is determined. In the pixel coordinates based on the aforementioned posture, an overlay of the simulated vehicle representation is displayed within the video stream. Determining the spatial coordinates of the simulated vehicle within the transportation network from the aforementioned pixel coordinates, wherein the spatial coordinates are relative to at least one of the transportation network or the camera, To cause the vehicle to follow the path based on the aforementioned spatial coordinates, the spatial coordinates are transmitted to the vehicle. Non-temporary computer-readable media, including [specific examples of such media].

16. The non-temporary computer-readable medium according to claim 15, wherein the reference posture corresponds to the posture of the vehicle with respect to at least one of the traffic network or the camera.

17. The non-temporary computer-readable medium according to claim 15, further comprising the operation of resetting the attitude of the simulated vehicle to the reference attitude in response to receiving a reset input.

18. The aforementioned operation is, Based on the simulated vehicle's attitude and at least one of the acceleration input and the steering input, the predicted path of the simulated vehicle is determined. Display an overlay of the predicted path representation within the video stream. A non-temporary computer-readable medium according to claim 15, further comprising:

19. The aforementioned operation is, Determining the speed of the simulated vehicle, Transmitting information to the vehicle representing at least one of the steering input, acceleration input, velocity, and yaw component of the attitude. A non-temporary computer-readable medium according to claim 15, further comprising:

20. The non-temporary computer-readable medium according to claim 15, wherein the representation of the simulated vehicle is displayed in the video stream after the acceleration input exceeds a predetermined threshold.

21. The non-temporary computer-readable medium according to claim 15, wherein the representation of the simulated vehicle includes representations of a set of wheels, chassis, front bumper, and lead bumper, each of which is modeled based on the vehicle.

Images