Remote control platform, user device, and method for operating an autonomous vehicle

Abstract

To provide methods, systems, devices, and apparatus for remote control platforms. [Solution] The remote control platform includes sensors. The sensors are configured to detect one or more objects in the environment surrounding the vehicle. The remote control platform includes an electronic control unit. The electronic control unit is coupled to the sensors. The electronic control unit is configured to determine the orientation of the vehicle and a route that includes the direction of travel. The electronic control unit is configured to drive the vehicle along a route based on one or more objects, the direction of travel of the vehicle, and its orientation.

Description

Technical Field

[0001] The present disclosure relates to a system, method, apparatus, and / or device for remotely controlling a vehicle via a smartphone and / or gesture input.

Background Art

[0002] Vehicle manufacturers have begun to provide remote control driving systems that enable users to control their vehicles. For example, some manufacturers enable control to move the vehicle from where the user is parking to a desired location. These autonomous driving functions enable the vehicle to leave the parking lot and drive around obstacles to the owner's location. Essentially, the owner's vehicle can be "called" from a location a threshold distance away to the owner. However, these systems may require line of sight and / or may not be able to adapt to changes in the environment, resulting in a near-collision situation. Therefore, these systems are limited in that they only need to tell the user to reach or go to a certain location, and in determining the optimal route for the vehicle to the desired location. This results in the user giving up control of the route by which the vehicle travels to the desired location.

[0003] Therefore, there is a need for a system, apparatus, and / or method to increase the amount of user remote control of a vehicle in order to enhance the capabilities of the remote control function.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

[0005] Generally, one aspect of the subject matter described herein can be embodied in a remote control platform for a vehicle. The remote control platform includes sensors configured to detect one or more objects in the environment surrounding the vehicle. The remote control platform includes an electronic control unit, which is coupled to the sensors. The electronic control unit is configured to determine the orientation of the vehicle and a route, including the direction of travel. The electronic control unit is configured to drive the vehicle along the route based on one or more objects, the direction of travel, and the orientation of the vehicle.

[0006] These and other embodiments may optionally include one or more of the following features: The electronic control unit can be configured to determine the current position of the vehicle. The electronic control unit can be configured to classify one or more objects in the surrounding environment into corresponding categories of a plurality of categories. The electronic control unit can be configured to determine the position of each of one or more objects relative to the vehicle. The electronic control unit can be configured to provide the user device with the current position of the vehicle, the corresponding categories, and the position of each of one or more objects in order to draw a graphical representation on the user device.

[0007] The electronic control unit can be configured to update the position of one or more objects relative to the vehicle. The electronic control unit can be configured to provide the updated position of one or more objects to track the movement of one or more objects. The graphical representation to be depicted may include the vehicle's current position, a corresponding category, and one or more icons representing the position of one or more objects. The electronic control unit can be configured to receive driving commands from a user device. Driving commands may include the vehicle's orientation and route. The electronic control unit can be configured to drive the vehicle around one or more objects while driving in the orientation the vehicle has acquired.

[0008] The electronic control unit can be configured to receive driving commands from the user device, including the vehicle's speed. The electronic control unit can be configured to drive the vehicle based on the vehicle's speed. The electronic control unit can be configured to determine when one or more objects in the surrounding environment intersect the vehicle's route. The electronic control unit can be configured to cancel the vehicle's navigation along the vehicle's route. The electronic control unit can be configured to provide a notification indicating the cancellation of the vehicle's navigation along the route. The electronic control unit can be configured to provide sensor data of the environment around the vehicle. The electronic control unit can be configured to provide a proposed alternative route for the vehicle to travel around one or more objects that intersect the vehicle's route.

[0009] In other embodiments, the subject can be embodied in a user device. The user device includes memory, which is configured to store computer-readable instructions, including a software application. The user device includes a processor coupled to the memory, which is configured to execute the software application and perform an operation to drive a vehicle. The operation includes drawing on a display the first position of the vehicle and the first positions of one or more objects adjacent to the vehicle's current position. The operation includes obtaining a driving command, which includes the orientation of the vehicle and the route the vehicle will travel. The operation includes providing a driving command to a remote control platform to drive the vehicle along the route around one or more objects.

[0010] In other embodiments, the subject can be embodied in a method for driving an autonomous vehicle. The method includes a processor determining the current position of the vehicle. The method includes a processor detecting one or more objects in the environment surrounding the current position of the vehicle. The method includes a processor driving or controlling the vehicle based on the vehicle's route and orientation. [Brief explanation of the drawing]

[0011] Other systems, methods, features, and advantages of the present invention will become apparent to those skilled in this art by examining the following figures and detailed description. Components shown in the drawings are not necessarily scaled down / enlarged to a constant ratio and may be exaggerated to better illustrate important features of the present invention.

[0012] [Figure 1] This is a block diagram of an example remote control system relating to an aspect of the invention. [Figure 2] This is a block diagram of an example user device of the remote control system shown in Figure 1, relating to an aspect of the invention. [Figure 3] This is an example process flowchart illustrating how to control or operate a vehicle using the remote control platform of the remote control system shown in Figure 1, relating to an aspect of the invention. [Figure 4] This is an example process flowchart for providing driving commands to operate a vehicle using the user device of the remote control system shown in Figure 1, relating to an aspect of the invention. [Figure 5] This is an example process flowchart for generating driving commands to operate a vehicle using the user device of the remote control system shown in Figure 1, relating to an aspect of the invention. [Figure 6] This figure shows an example graphic representation of a vehicle and its surrounding environment on the user device of the remote control system shown in Figure 1, relating to an aspect of the invention. [Modes for carrying out the invention]

[0013] Disclosed herein are systems, apparatus, and methods for remotely controlling a vehicle via a smartphone and / or gesture input. The remote control system includes a remote control platform and a user device. The remote control platform displays the locations of the vehicle and one or more objects surrounding the vehicle on a map, so that the user can create or generate a route for the vehicle to travel around one or more objects to a destination. The remote control platform provides the locations of the vehicle and one or more objects so that the user device can generate a drawing of the environment surrounding the vehicle, so that the user can have more detailed control of the vehicle within that environment. The remote control platform provides the user of the user device with more contextual information about the environment around the vehicle, so that the user can provide more detailed driving commands and more precise control of the vehicle via the user device. For example, user input could specify driving commands to remotely operate the vehicle to drive 5 feet forward, turn left, and then drive another 10 feet to reach a desired location, rather than simply specifying to arrive at a particular destination.

[0014] More detailed driving commands allow the user to specify the route the vehicle will take around one or more objects in its surrounding environment. This allows the user to control the precise path the vehicle will take from its starting point to its destination. Furthermore, the user can control other functions of the vehicle, such as its speed, direction, and orientation.

[0015] Other benefits and advantages include the ability to monitor objects in the surrounding environment in real time. For example, some of these objects may be other vehicles, and the remote control platform can identify when these objects are moving. If these objects are in the vehicle's path, or are about to enter the vehicle's path, the remote control platform can disable or cancel the vehicle's navigation to prevent an accident. The remote control platform can also notify the user, suggest an alternative route, or allow manual control.

[0016] Additionally, the remote control platform can identify or recognize the user's visual gestures for remotely controlling the vehicle. The remote control platform can capture user gestures and, based on those gestures, generate driving commands to direct the vehicle along a route. This allows the user to control the vehicle via gesture input.

[0017] Figure 1 is a block diagram of the remote control system 100. The remote control system 100 can be retrofitted to the vehicle 102, coupled to it, integrated with it, contained within it, or completely separate from the vehicle 102. The remote control system 100 may include, or be coupled to, a remote control platform 104, a user device 106, and / or an external database 108. The user device 106 may be a personal device, mobile device such as a smartphone, tablet, or other electronic device that can display notifications, activate applications, or interact with the vehicle 102 via a wireless or wired connection. The remote control system 100 uses the user device 106 to draw a graphical representation of the vehicle 102 and the objects surrounding the vehicle 102, and to obtain and display driving commands for controlling the vehicle 102 on a map. Figure 2 is a block diagram of the user device 106.

[0018] The remote control system 100 can have or use a network 110 to communicate between different components, such as between a vehicle 102, a user device 106, a remote control platform 104, and / or an external database 108. The network 110 can be a dedicated short range communication (DSRC) network, a local area network (LAN), a wide area network (WAN), a cellular network, the Internet, or a combination thereof that connects, couples, and / or communicates between different components of the remote control system 100.

[0019] The remote control system 100 can include or be coupled to an external database 108. A database is any collection of information organized for searching and retrieval, such as by a computer, and the database can be organized in tables, schemas, queries, reports, or any other data structure. The database can use any number of database management systems. The external database 108 can include a third-party server or website that stores or provides information. The information can include real-time information, information updated periodically, or information entered by a user. A server can be a computer on a network used to provide services, such as access to files or sharing of peripherals, to other computers on the network.

[0020] The external database 108 can be a map database that includes the locations of various structures or objects that obstruct the route of the vehicle 102, such as trees, buildings, or other objects. The external database 108 can be a traffic database. The traffic database can include the locations of other vehicles and / or can track the locations of other vehicles, such as the movement of other vehicles.

[0021] The remote control system 100 includes a remote control platform 104. The remote control platform 104 detects or determines the position of the vehicle 102 and the position of one or more objects in the surrounding environment that are close to the vehicle 102, such as within a threshold distance. The remote control platform 104 can detect the movement of the vehicle 102 and the movement of one or more objects and can provide the user device 106 with the position, movement, and type of object so that the user device 106 can create or generate a drawing of the environment surrounding the vehicle 102. Furthermore, the remote control platform 104 can receive driving commands from the user device 106 and control or operate the vehicle 102 to avoid obstacles or hazards in order to reach a specified destination. This allows the user to have greater control over the navigation, control, and / or operation of the vehicle 102 via the user device 106 or through gesture input.

[0022] The remote control platform 104 may include an electronic control unit 112, a memory 114, a network access device 116, and / or one or more sensors 118. The remote control platform 104 may include a navigation unit 120 and / or a user interface 122. The remote control platform 104 may include one or more other components of the vehicle 102, such as a brake 124, an accelerator 126, a steering 128, a motor and / or a generator 130, an engine 132, a battery 134, and / or a battery management and control unit (BMCU) 136.

[0023] Vehicle 102 is a means of transport capable of transporting people, objects, or devices that are permanently or temporarily attached. Vehicle 102 may be a means of transport equipped with self-propelled wheels, such as an automobile, a sports car, a truck, a bus, a van, or other vehicles powered by a motor, battery, or fuel cell. For example, Vehicle 102 may be an electric vehicle, a hybrid vehicle, a hydrogen fuel cell vehicle, a plug-in hybrid vehicle, or any other type of vehicle having a fuel cell stack, motor and / or generator. Other examples of vehicles include bicycles, trains, aircraft, boats, and any other form of transportable means of transport. Vehicle 102 may be semi-autonomous or autonomous; that is, Vehicle 102 may be self-driving and capable of driving without human input. An autonomous vehicle may have and be able to use one or more sensors and / or navigation units for autonomous driving.

[0024] The remote control system 100 includes, or is coupled to, one or more processors, such as the electronic control unit (ECU) 112 of the remote control platform 104, or the processor 202 of the user device 106. One or more processors, such as the ECU 112 or processor 202, can be implemented as a single processor or multiple processors. For example, the ECU 112 or processor 202 may be a microprocessor, data processor, microcontroller, or other controller, and can be electrically coupled to some or all other components in the vehicle 102 and / or the remote control system 100. One or more processors can draw a graphical representation of the positions of objects in the vehicle 102 and / or the surrounding environment. One or more processors can also receive driving commands entered by the user and / or drive in that manner. The ECU 112 can be coupled to memory 114, and the processor 202 can be coupled to memory 204.

[0025] The remote control system 100 has a memory 114 in the remote control platform 104 and a memory 204 in the user device 106. Memory 114 can be coupled to the ECU 112 and can store instructions executed by the ECU 112, and memory 204 can be coupled to the processor 202 and can store instructions executed by the processor 202. Memory 114 and / or memory 204 may include one or more random access memories (RAM), read-only memories (ROM), or other volatile or non-volatile memories. Memories 114 and 204 may be non-temporary memories or data storage devices such as hard disk drives, solid-state disk drives, hybrid disk drives, or other suitable data storage devices, and may further store machine-readable instructions that can be loaded and executed by the ECU 112 and / or processor 202. For example, memory 204 may store a software application that draws a graphical representation of the environment surrounding the vehicle 102 and can be used to access driving instructions that control or operate the vehicle 102.

[0026] The user device 106 may include a user interface 206 and / or a network access device 208, and the remote control platform 104 may include a user interface 122 and / or a network access device 116. One or more user interfaces 122, 206 may receive user input indicating driving commands used to operate, control, or drive the vehicle 102. One or more user interfaces 122, 206 may also receive configuration tolerances indicating when objects present a hazard and intersect the path or route of the vehicle 102. One or more user interfaces 122, 206 may also provide notifications and / or provide a graphical representation of the environment around the vehicle 102 so that the user can input a route to drive the vehicle 102 from a starting position to a destination position.

[0027] One or more user interfaces 122, 206 may include input / output devices that receive user input from user interface elements, buttons, dials, microphones, keyboards, or touchscreens. One or more user interfaces 122, 206 may provide output to output devices such as displays, speakers, audio or visual indicators, or refreshable braille displays.

[0028] One or more network access devices 116, 208 may include communication ports or channels such as one or more dedicated narrow-range communication (DSRC) units, Wi-Fi units, Bluetooth® units, radio frequency identification (RFID) tags or readers, or cellular network units for accessing cellular networks (such as 3G, 4G, or 5G). One or more network access devices 116, 208 may transmit and receive data from different components of different entities of the remote control system 100, such as the user device 106, the external database 108, and / or the vehicle 102.

[0029] The remote control platform 104 includes a navigation unit 120. The navigation unit 120 can be integrated with the vehicle 102 or it can be a separate unit coupled to the vehicle 103. Instead of the navigation unit 120, the vehicle 102 may include a Global Positioning System (GPS) unit (not shown) for detecting location data, including the vehicle's current location and date / time information. In this regard, the ECU 112 can perform the functions of the navigation unit 120 based on data received from the GPS unit. The navigation unit 120 or the ECU 112 can perform navigation functions. Navigation functions may include, for example, route and route set prediction, providing navigation commands, and receiving user input such as validation of predicted routes and route sets or destinations.

[0030] The remote control platform 104 includes one or more sensors 118. One or more sensors 118 may include a camera 118a. The camera 118a can capture image data of the surrounding environment, and the image data can be processed or analyzed to recognize structures or objects such as trees, buildings, or other vehicles in the environment surrounding the vehicle 102, and their relative distance from the vehicle 102. The image data can also be used to determine the orientation of the vehicle 102, such as whether the vehicle 102 is facing forward or backward. One or more sensors 118 may include proximity sensors 118b. The proximity sensors 118b can use lidar, radar, or other ranging techniques to detect or determine objects that are close to the vehicle 102, such as within a threshold distance. The proximity sensors 118b can also be used to determine the relative distance to those objects. One or more sensors 118 may include other sensors 118c. Other sensors 118c may include a vehicle speed sensor that can be used to measure the speed of the vehicle 102, and / or an accelerometer or gyroscope that can be used to determine the orientation of the vehicle 102. One or more sensors 118 can also be used to determine the orientation of the vehicle 102 and to make the vehicle 102 drive autonomously. For example, one or more sensors 118 can be used to control the vehicle 102 to drive along a route included in a driving command. One or more sensors 118 can be used to drive around one or more objects, maintain the relative distance between the vehicle 102 and one or more objects as indicated in the driving command, and / or control the vehicle 102 to follow a path or route indicated in the driving command.

[0031] The remote control system 100 may include, or be coupled to, one or more vehicle components. The remote control system 100 may control one or more vehicle components to operate or control the vehicle 102 to travel along a route from a starting position to a destination position and around one or more objects in the surrounding environment. One or more vehicle components may include a motor and / or generator 130. The motor and / or generator 130 may convert electrical energy into mechanical power, such as torque (rotational force), and mechanical power into electrical energy. The motor and / or generator 130 may be coupled to a battery 134. The motor and / or generator 130 may convert energy from the battery 134 into mechanical power and return energy to the battery 134, for example, through regenerative braking. The vehicle 102 may include one or more additional power generation devices, such as an engine 132 or a fuel cell stack (not shown). The engine 132 burns fuel to provide power in place of, and / or in addition to, the power supplied by the motor and / or generator 130.

[0032] The battery 134 can be coupled to the motor and / or generator 130, supply electrical energy to the motor and / or generator 130, and receive electrical energy from the motor and / or generator 130. The battery 134 may include one or more rechargeable batteries and can power the remote control system 100 even when the vehicle 102 is not operating.

[0033] The BMCU136 can be coupled to the battery 134 and can control and manage the charging and discharging of the battery 134. The BMCU136 can, for example, measure parameters used to determine the state of charge (SOC) of the battery 134 using a battery sensor. The BMCU136 can control the battery 134.

[0034] One or more vehicle components may include a brake 124, an accelerator 126, and / or a steering wheel 128. The remote control platform 104 can control or operate the brake 124, accelerator 126, and / or steering wheel 128 to move the vehicle 102 along a route around the surrounding environment. For example, the remote control platform 104 can apply or release the brake 124, or press or release the accelerator 126, and / or adjust the angle of the steering wheel 128 to increase the speed of the vehicle 102, decrease the speed, and / or change the direction of travel.

[0035] Figure 3 is a flowchart of the process 300 for controlling or operating the vehicle 102 using driving commands. One or more computers or one or more data processing devices, for example, a properly programmed ECU 112 of the remote control platform 104 of the remote control system 100 in Figure 1, can implement the process 300.

[0036] The remote control platform 104 acquires navigation map information (302). The navigation map information may include the current position of the vehicle 102 and / or the positions of one or more objects in the environment surrounding the vehicle 102. The remote control platform 104 can acquire navigation map information using the navigation unit 120 and / or the external database 108. For example, the remote control platform 104 can acquire the current position of the vehicle 102 using a Global Positioning System device and provide the current position of the vehicle 102 to the external database 108. The external database can then provide the information, and the remote control platform 104 can receive the positions of one or more objects in the environment surrounding the vehicle 102, such as the positions of other vehicles listed on the map, the positions of other structures such as buildings, and / or the positions of other objects such as trees.

[0037] The remote control platform 104 detects or identifies one or more objects in the environment surrounding the vehicle 102 (304). The remote control platform 104 can draw the outline or create the shape of each of the one or more objects in the environment surrounding the vehicle 102. The remote control platform 104 can compare the drawn outline or created shape of each of the one or more objects in the environment surrounding the vehicle 102 with a library of objects and determine whether the object matches one of the library objects. If there is a match, the remote control platform 104 can identify the object in the surrounding environment as a library object such as a tree, sign, or other vehicle, and recognize the object in the surrounding environment as such.

[0038] Additionally, the remote control platform 104 can use one or more sensors 118 to determine the position of one or more objects surrounding the vehicle 102. The position may be the relative position of one or more objects to the vehicle 102. For example, the remote control platform 104 can use proximity sensors 118b or analyze image data from cameras 118a to determine the distance and orientation of one or more objects to the vehicle 102. Object identification displays can be associated with each of the one or more objects along with their positions.

[0039] When one or more objects are identified or detected, the remote control platform 104 can classify one or more objects in the environment surrounding the vehicle 102 (306). When the remote control platform 104 matches one or more objects with library objects, the library objects can be associated with categories such as "buildings," "trees," or "vehicles," or with other categories such as "stationary objects" or "moving objects." The remote control platform 104 can associate the categories of the matched library objects with the categories of the objects, and can label or tag one or more objects to identify the categories of one or more objects.

[0040] The remote control platform 104 provides the user device 106 with a classification of one or more objects, the location of the vehicle 102, and / or the location of one or more objects in the surrounding environment (308). The remote control platform 104 can send or transmit the classification of one or more objects, the location of the vehicle 102, and / or the location of one or more objects in the surrounding environment to the user device 106 via the network 110 using the network access device 116. The classification and / or location can be used to generate a drawing of the environment surrounding the vehicle 102 so that the user can use the user device 106 to drive or control the vehicle 102.

[0041] The remote control platform 104 can acquire or detect gesture input (310). The remote control platform 104 can use one or more sensors 118, such as a camera 118a, to capture gesture input. For example, the remote control platform 104 can capture image data of the surrounding environment and identify individuals, such as the driver or owner, within a threshold distance of the vehicle 102. The remote control platform 104 can then identify in the image data that the user is moving or that a part of the user, such as the user's hand, is moving. The gesture input can be used to determine driving commands.

[0042] The remote control platform 104 can acquire user input (312). The remote control platform 104 can receive user input from the user device 106. User input can indicate driving commands to drive or control the vehicle 102. User input can be used to determine the vehicle 102's speed, direction, orientation, starting position, destination, and route from the starting position to the destination, around one or more objects in the environment surrounding the vehicle 102. User input can also indicate the vehicle 102's path from the starting position to the destination, relative to the locations of one or more objects.

[0043] The remote control platform 104 can determine driving commands based on user input and / or gesture input provided to the user interface (314). The remote control platform 104 can interpret driving commands from user input and / or gesture input. The remote control platform 104 can determine whether to use user input and / or gesture input based on configuration settings pre-set by the user and / or based on the identity of the vehicle 102 driver, which can be recognized from data captured by one or more cameras 118a.

[0044] A driving command for vehicle 102 may include the vehicle's speed, direction of travel, orientation, starting position, destination, and route from the starting position to the destination, around one or more objects in the environment surrounding vehicle 102. The route may include the path of vehicle 102 to the locations of one or more objects.

[0045] The remote control platform 104 can use mappings to associate user input received from the user device 106 with driving commands such as the vehicle's speed or direction, orientation, and / or route. Similarly, the remote control platform 104 can use mappings to associate received gesture inputs with driving commands. Driving commands can be translated into, or associated with, specific actions or controls of the vehicle 102, such as the operation of the brakes 124, accelerator 126, and / or steering 128.

[0046] For example, the remote control platform 104 can associate a curve in a path indicated by user input with an angle setting of the steering 128 that causes the vehicle 102 to change direction. The angle of the curve can correspond to the amount of the angle setting of the steering 128. In another example, the remote control platform 104 can associate a straight line extending by a first length, indicated by user input, with the speed of the user's finger drag on the user device 106, and by pressing the accelerator 126 of the vehicle 102, a first distance in the straight line, with the vehicle 102 traveling at a specific speed. If the user drags their finger faster on the user device 106, the vehicle 102 can be accelerated. And if the user moves their finger further in the straight line, the vehicle 102 will travel a longer distance along the straight line. Similarly, if the user stops their finger on the screen of the user device 106, this can indicate to the remote control platform 104 that the brake 124 is being applied. The distance between the user's finger path on the display of the user device 106 and the object icon can directly correspond to the distance between the vehicle's route and the object's position in the surrounding environment. Therefore, the user's finger path directly mimics, or faithfully reflects, the actual route taken by the vehicle 102 in the surrounding environment. Furthermore, the longer the path drawn by the user's finger on the display, the longer the straight-line path that the vehicle 102 travels in the surrounding environment.

[0047] In another example, the remote control platform 104 can recognize that the user is waving their hand in a specific direction. The waving can indicate the direction in which the vehicle 102 is traveling, and the speed of the waving can indicate the speed at which the vehicle 102 is traveling. In yet another example, the remote control platform 104 can recognize that the user is raising their hand above their head, which can indicate that the vehicle 102 should stop moving.

[0048] In some implementations, the user device 106 can receive user input or gesture input from the user, associate the user input with a driving command, and / or interpret it. The user device 106 can associate the user input with a driving command, and / or interpret it, and provide the driving command to the remote control platform 104. The association, interpretation, and / or generation of driving commands from user input and / or gesture input can be learned or improved using machine learning algorithms.

[0049] Once a driving command is determined, the remote control platform 104 determines whether one or more objects pose a hazard to the vehicle 102 (316). One or more objects may pose a hazard to the vehicle 102 when they intersect or cross the route of the vehicle 102. The remote control platform 104 can compare the route of the vehicle 102 included in the driving command with the locations of one or more objects, and if the route of the vehicle 102 crosses within a threshold distance of the locations of one or more objects, the remote control platform 104 can determine that the objects pose a hazard to the vehicle 102.

[0050] Similarly, the remote control platform 104 can identify one or more objects as "moving objects" based on the category of one or more objects, and can intermittently and periodically update the positions of the moving objects. Based on the updated positions, the remote control platform 104 can plot the paths of the one or more moving objects on a map and determine the trajectories or paths of the one or more moving objects. If the trajectories or paths of the one or more moving objects intersect with the route of the vehicle 102, the remote control platform 104 can determine that the objects pose a hazard.

[0051] If one or more objects do not pose a danger to the vehicle 102, the remote control platform 104 operates or controls the vehicle 102 (318). If one or more objects pose a danger to the vehicle 102, the remote control platform 104 notifies the user (320) and / or cancels the navigation or control of the vehicle 102 (322).

[0052] If one or more objects do not pose a danger to the vehicle 102, the remote control platform 104 will operate or control the vehicle 102 (318). The remote control platform 104 can operate or control the vehicle 102 based on driving commands. Driving commands, such as increasing speed, turning, braking, or other driving commands, are translated into or associated with the actions of the vehicle 102. The remote control system 100 can press or stop pressing the accelerator 126 and / or apply or stop applying the brake 124 to control the amount of speed, acceleration, and / or deceleration of the vehicle 102. Furthermore, the remote control platform 104 can use the steering 128 to steer the vehicle 102, such as turning. The remote control platform 104 can change gears to operate the vehicle 102 in driving mode, reverse mode, or neutral mode. The remote control platform 104 can also change the orientation of the vehicle 102, such as by rotating it 180 degrees to drive the vehicle in the opposite direction while avoiding one or more objects. Other controllable operations of the vehicle 102 include discharging electrical energy from the battery 134 to the motor and / or generator 130, and / or controlling the state of the engine 132 or throttle to move the wheels of the vehicle 102.

[0053] In some implementations, the remote control platform 104 can operate other functions of the vehicle 102, such as the windshield wipers, headlights, windows, heating, ventilation, and air conditioning (HVAC) units, or other functions of the vehicle 102. For example, when the vehicle 102 reaches the user's desired location, the remote control platform 104 can activate seat heaters or HVAC units on cold days to make the vehicle 102 more comfortable.

[0054] If one or more objects pose a hazard to the vehicle 102, the remote control platform 104 provides a notification to the user device 106 (320). The notification may indicate to the user that there is a hazard on the vehicle 102's route and may request additional driving commands. The notification may indicate the cancellation of the vehicle 102's navigation and / or may provide image data of the surrounding environment to enable the user to drive around the one or more objects posing a hazard and / or to identify the hazard. The notification may include a suggestion of an alternative route for the vehicle 102 to drive around the one or more objects posing a hazard to the vehicle 102. The alternative route may include sensor data of the relative position or location of one or more objects to assist in the vehicle 102's navigation around the one or more objects.

[0055] In some implementations, the remote control platform 104 can provide image data to the user device 106 along with notifications. The image data may include live or real-time images of the surrounding environment from one or more cameras 118a. The remote control platform 104 can provide the user with an option to manually control the vehicle 102, which, when activated, allows the user to manually drive the vehicle 102 or manually control the vehicle's operation using live or real-time images captured by one or more cameras 118a.

[0056] The remote control platform 104 can cancel the navigation of vehicle 102 when a hazard is detected. The remote control platform 104 can override any commands to one or more vehicle components to operate, move, or control vehicle 102 in order to stop vehicle 102 and / or switch off vehicle 102 to prevent vehicle 102 from moving. For example, to prevent vehicle 102 from moving, the remote control platform 104 can cut off the discharge of electrical energy from battery 134, stop engine 132, apply brakes 124, or render vehicle 102 inoperable. The cancellation can be performed automatically when a hazard is detected in the vehicle 102's route.

[0057] In some implementations, the remote control platform 104 can receive a user command to automatically cancel the navigation or control of the vehicle 102, regardless of whether or not there is a hazard detected by one or more sensors 118. For example, the user can simply click a button on the display of the user device 106 (e.g., an emergency or safe stop button) to cause the remote control platform 104 to switch off the vehicle 102, immediately stop the vehicle 102, or prevent the vehicle 102 from moving. This is advantageous because it allows the user to stop the vehicle 102 even when no hazard is present, and acts as an additional safety feature.

[0058] Figure 4 is a flowchart of process 400 for providing driving commands to drive vehicle 102. One or more computers or one or more data processing devices, for example, a properly programmed processor 202 of the user device 106 of the remote control system 100 in Figure 1, can implement process 400.

[0059] The user device 106 obtains the location of one or more objects in the vehicle 102 and / or the surrounding environment, and a classification of one or more objects (402). The user device 106 can obtain the location of one or more objects in the vehicle 102 and / or the surrounding environment, along with a classification of one or more objects, from the remote control platform 104, and can use this information to generate a rendering of the environment on the user interface 206 of the user device 106, such as a display.

[0060] The user device 106 generates icons (404) for the vehicle 102 and one or more objects. Icon generation can be based on the corresponding category of one or more objects. For example, a tree icon 602 can be generated to represent a tree, another vehicle icon 604 can be generated to represent another vehicle, and a vehicle icon 606 can be generated to represent the user's own vehicle. The user device 106 can use a variety of other icons to represent different structures or other objects that may be encountered in the environment surrounding the vehicle 102.

[0061] Once the icons are generated, the user device 106 can draw the icons on its display (406). The distance, orientation, and position of the icons of one or more objects relative to the vehicle 102 can faithfully reflect or mimic the actual distance, orientation, and position of one or more objects relative to the vehicle 102, such that even the distance between the icons corresponds to or correlates with the distance between one or more objects and the vehicle 102. This reproduces an accurate depiction of the environment surrounding the vehicle 102.

[0062] The user device can receive user input such as taps, selections, swipes, or user contact with the user interface 206 (408). User input can correspond to driving commands for controlling the vehicle 102. The user device 106 can receive user input via the user interface 206, such as via a touchscreen display. Upon receiving user input, the user device 106 can generate driving commands based on the user input (410). In some implementations, the remote control platform 104 can convert user input into driving commands. Figure 5 further describes the process 500 for associating user input with driving commands.

[0063] The user device 106 can provide driving commands to the remote control platform 104 (412). The user device 106 can use the network access device 208 to send driving commands to the remote control platform 104 that enable the remote control platform to operate or control one or more vehicle components that operate the vehicle 102. The network access device 116 can send driving commands via the network 110.

[0064] Figure 5 is a flowchart of the process 500 for generating driving commands to operate the vehicle 102. One or more computers or one or more data processing devices, for example, a properly programmed processor 202 of the user device 106 of the remote control system 100 in Figure 1, can implement the process 500.

[0065] The user device 106 can receive a selection of an icon, such as a vehicle icon 606, at a first position 610 (502). The selection may be, for example, a selection of the vehicle icon 606, as shown in Figure 6, which may indicate that the user intends to generate a driving command for the vehicle 102. The selection may occur, for example, when the user device 106 detects that pressure has been applied to the display at the first position 610.

[0066] The vehicle icon 606 may be a first position 610 on the display of the user device 106, surrounded by other icons such as other vehicle icons 604 and / or a tree icon 602. The other icons may be positioned on the display at a location or position relative to the first position 610, which represents the starting position of the vehicle 102. The first position 610 may be the starting position or current position of the vehicle 102. The starting position may be the beginning of the route of the vehicle 102.

[0067] The selection of an icon can also indicate the orientation of the vehicle 102. For example, if the user selects the vehicle icon 606, the vehicle icon 606 can switch between different orientations in which the vehicle 102 is traveling, such as a forward direction where the vehicle 102 is traveling forward, and a backward direction where the vehicle 102 is traveling backward.

[0068] The user device 106 can obtain the movement of the icon to a second position (504). The user device 106 can receive user input indicating that the user is dragging, sliding, or moving the vehicle icon 606 from a first position 610 to a second position 612. Dragging, sliding, or moving the vehicle icon 606 can indicate a route that the vehicle 102 will travel around one or more objects, such as other vehicles represented by other vehicle icons 604. Dragging, sliding, or moving the vehicle icon 606 around one or more other icons, such as other vehicle icons 604 and / or tree icons 602, can be represented by a path 608 on the display of the user device 106. The distance between the path 608 and the other vehicle icons 604 or tree icons 602 can correspond to the distance that the vehicle 102 maintains from the other vehicles represented by other vehicle icons 604 or the trees represented by tree icons 602.

[0069] The user device 106 can detect the release of the icon (moving the mouse or other object away from the icon) at the second position 612 (506). The second position 612 may be a destination position, and the icon release can represent the end of a route at the destination position. The user device 106 can detect the icon release, for example, when pressure is removed from the display at the second position 612. The icon release can trigger the generation of an operation command based on a sequence of user inputs on the user interface 206, including selection, dragging, and subsequent pressure removal.

[0070] When the user device 106 detects the release of the icon, the user device 106 can calculate or measure the speed of movement along the path 608 from the first position 610 to the second position 612 (508). The calculated or measured speed of the user input movement can correspond to and indicate the speed at which the vehicle 102 travels along the route. The measured speed can be directly proportional to the speed of the vehicle 102 traveling along the route around one or more objects. If the speed of movement of the pressure applied on the user device 106 increases, the speed at which the vehicle 102 travels can be increased, and if the speed of movement of the pressure applied on the user device 106 decreases, the speed at which the vehicle 102 travels can also be decreased.

[0071] The user device 106 can determine the relative distance between the vehicle icon 606's movement path 608 and other icons (510). The user device 106 can calculate the relative distance between the path 608 and the positions of other icons. The relative distance can correspond to the relative distance that the vehicle 102 maintains with respect to one or more objects.

[0072] The user device 106 generates driving commands (512). The user device 106 generates driving commands based on user input indicating the relative distance between the path 608 and other icons, the measured speed of movement along the path 608, the selection of an icon at the first position 610, and the release of the icon at the second position 612. The user device 106 can associate the first position 610 with the departure position of the vehicle 102, the second position 612 with the destination position of the vehicle 102, the path 608 with the route, the measured speed of movement along the path 608 with the vehicle's speed, and the relative distance between the path 608 and other icons with the relative distance between the vehicle 102's route and the position of one or more objects. These driving commands are then provided to the remote control platform 104 to operate or control the vehicle 102.

[0073] Embodiments of the invention have been disclosed in an illustrative form. Therefore, the terminology used throughout should be read non-restrictively. While minor modifications to the teachings herein may be conceivable to those familiar with the art, it should be understood that the scope of the patent relating to what is described herein is intended to be limited to all such embodiments that reasonably fall within the scope of the technological advance to which it is contributed, and that this scope should not be limited except with regard to the accompanying claims and their equivalents. The inventions disclosed herein include the following embodiments: [Aspect 1] A remote control platform for vehicles, A sensor configured to detect one or more objects in the environment surrounding the vehicle, The system comprises an electronic control unit coupled to the sensor, The aforementioned electronic control unit is Determine the direction of the vehicle and the route including the direction of travel. Based on the one or more objects, the direction of travel, and the orientation of the vehicle, the system is configured to make the vehicle travel along the route. Remote control platform. [Aspect 2] The aforementioned electronic control unit is Determine the current position of the aforementioned vehicle, The one or more objects in the surrounding environment are classified into corresponding categories of multiple categories, Determine the position of each of the one or more objects relative to the vehicle, The system is configured to provide the user device with the current position of the vehicle, the corresponding category, and the position of each of the one or more objects in order to draw a graphical representation on the user device. The remote control platform described in Embodiment 1. [Aspect 3] The aforementioned electronic control unit is The position of each of the one or more objects relative to the vehicle is updated, The updated position of each of the one or more objects relative to the vehicle is configured to track the movement of the one or more objects. The remote control platform described in Embodiment 2. [Aspect 4] The remote control platform according to embodiment 2, wherein the depicted graphic representation includes one or more icons representing the current position of the vehicle, the corresponding category, and the respective positions of one or more objects. [Aspect 5] The aforementioned electronic control unit is The system is configured to obtain driving commands from the user device, which include the direction of the vehicle and the route of the vehicle. In order to drive the vehicle based on the one or more objects, the direction of travel, and the orientation of the vehicle, the electronic control unit drives the vehicle around the one or more objects while the vehicle is traveling in the acquired orientation. The remote control platform described in Embodiment 1. [Aspect 6] The aforementioned electronic control unit is The system is configured to acquire driving commands from the user device, including the vehicle's speed. The aforementioned electronic control unit is configured to drive the vehicle based on the vehicle's driving speed. The remote control platform described in Embodiment 1. [Aspect 7] The aforementioned electronic control unit is It is determined that one or more of the surrounding environment intersects with the route of the vehicle, The system is configured to cancel the vehicle's navigation along the aforementioned route. The remote control platform described in Embodiment 1. [Aspect 8] The aforementioned electronic control unit is Provide a notification indicating the cancellation of the vehicle's navigation along the aforementioned route. The image data of the environment surrounding the vehicle is provided. The system is configured to provide a proposed alternative route for the vehicle to travel around one or more objects that intersect the route of the vehicle. The remote control platform described in Embodiment 7. [Aspect 9] The system further includes a second sensor configured to detect gesture input, The electronic control unit is further configured to drive the vehicle along the route based on the gesture input. The remote control platform described in Embodiment 1. [Aspect 10] The remote control platform according to embodiment 1, wherein the gesture input includes a hand movement indicating an additional driving command, which includes the driving direction and driving speed. [Aspect 11] User device, Memory configured to store computer-readable instructions, including software applications, The system comprises a processor coupled to the memory and configured to run the software application in order to perform operations for driving the vehicle, The aforementioned operation is, To draw the first position of the vehicle and the first positions of one or more objects adjacent to the current position of the vehicle on the display. To obtain a driving order that includes the direction of the vehicle and the route on which the vehicle will travel, The system includes providing driving commands to a remote control platform in order to drive the vehicle along the route around one or more of the aforementioned objects. User device. [Aspect 12] The aforementioned operation further, To obtain the current position of the vehicle, Obtaining the position of one or more objects adjacent to the current position of the vehicle, Obtaining a category for each of the one or more objects mentioned above, A user device according to embodiment 11, having the following features. [Aspect 13] The aforementioned operation further, The vehicle and the generation of icons for each of the one or more objects based on the category for each of the one or more objects, On the display, the current position of the vehicle and the positions of one or more objects are drawn using the generated icons. A user device according to embodiment 12, having the following features. [Aspect 14] To obtain the driving order which includes the direction of the vehicle and the route, Receiving a first user input that includes the starting position of the said route, the ending position of the said route, and the path between the said starting position and the said ending position, Receiving a second user input indicating the orientation of the vehicle when the vehicle is traveling along the route, A user device according to embodiment 11, which includes the following: [Aspect 15] The aforementioned operation further, Receiving a second position of the vehicle and a second position of each of the one or more objects, which is different from the first position of the vehicle. On the display, instead of the first position of the vehicle and the first position of each of the one or more objects, the second position of the vehicle and the second position of each of the one or more objects are drawn. A user device according to embodiment 11, having the following features. [Aspect 16] The change between the second position of the vehicle and the first position of the vehicle corresponds to the movement of the vehicle. Any change between the second position of each of the one or more objects and the first position of each of the one or more objects corresponds to the movement of the corresponding object. User device as described in aspect 15. [Aspect 17] A method for operating an autonomous vehicle, The processor determines the current position of the vehicle, The processor detects one or more objects in the environment surrounding the vehicle's current location, The processor drives or controls the vehicle based on the vehicle's route and orientation. A method that has [Aspect 18] Determining the position of each of the one or more objects, To draw the position of each of the one or more objects and the current position of the vehicle on the display, On the display, when the vehicle travels a route around one or more objects, the position of each of the one or more objects and the current position of the vehicle are updated. The method according to embodiment 17, further comprising the above. [Aspect 19] It further has the ability to detect gesture input, Driving or controlling the aforementioned vehicle is further based on the gesture input, The method described in aspect 17. [Aspect 20] Receiving user input indicating the route, which includes a starting position, an ending position, and the path between the starting position and the ending position, Determining the route based on the user input, The method according to embodiment 17, further comprising the above.

Claims

1. A remote control platform for vehicles, A sensor configured to detect one or more objects in the environment surrounding the vehicle, The system comprises an electronic control unit coupled to the sensor, The aforementioned electronic control unit is Determine the direction of the vehicle and the route including the direction of travel. Based on the one or more objects, the direction of travel, and the orientation of the vehicle, the system is configured to make the vehicle travel along the route. Remote control platform.

2. The aforementioned electronic control unit is Determine the current position of the aforementioned vehicle, One or more objects in the surrounding environment are classified into corresponding categories of a plurality of categories. Determine the position of each of the one or more objects relative to the vehicle, The system is configured to provide the user device with the current position of the vehicle, the corresponding category, and the position of each of the one or more objects in order to draw a graphical representation on the user device. The remote control platform according to claim 1.

3. The aforementioned electronic control unit is The position of each of the one or more objects relative to the vehicle is updated, The updated position of each of the one or more objects relative to the vehicle is configured to provide for tracking the movement of the one or more objects. The remote control platform according to claim 2.

4. The remote control platform according to claim 2, wherein the depicted graphic representation includes one or more icons representing the current position of the vehicle, the corresponding category, and the respective positions of one or more objects.

5. The aforementioned electronic control unit is The system is configured to obtain driving commands from the user device, which include the direction of the vehicle and the route of the vehicle. In order to drive the vehicle based on the one or more objects, the direction of travel, and the orientation of the vehicle, the electronic control unit drives the vehicle around the one or more objects while the vehicle is traveling in the acquired orientation. The remote control platform according to claim 1.

6. The aforementioned electronic control unit is The system is configured to acquire driving commands from the user device, including the vehicle's speed. The aforementioned electronic control unit is configured to drive the vehicle based on the vehicle's driving speed. The remote control platform according to claim 1.

7. The aforementioned electronic control unit is It is determined that one or more objects in the surrounding environment intersect with the route of the vehicle, The system is configured to cancel the vehicle's navigation along the aforementioned route. The remote control platform according to claim 1.

8. The aforementioned electronic control unit is Provide a notification indicating the cancellation of the vehicle's navigation along the aforementioned route. The image data of the environment surrounding the vehicle is provided. The system is configured to provide a proposed alternative route for the vehicle to travel around one or more objects that intersect the vehicle's route. The remote control platform according to claim 7.

9. The system further includes a second sensor configured to detect gesture input, The electronic control unit is further configured to drive the vehicle along the route based on the gesture input. The remote control platform according to claim 1.

10. The remote control platform according to claim 1, wherein the gesture input includes a hand movement indicating an additional driving command, which includes the driving direction and driving speed.

11. User device, Memory configured to store computer-readable instructions, including software applications, The system comprises a processor coupled to the memory and configured to run the software application in order to perform operations for driving the vehicle, The aforementioned operation is, To draw the first position of the vehicle and the first positions of one or more objects adjacent to the current position of the vehicle on the display. To obtain a driving order that includes the direction of the vehicle and the route on which the vehicle will travel, The system includes providing the driving commands to a remote control platform in order to drive the vehicle along the route around one or more of the aforementioned objects. User device.

12. The aforementioned operation further, To obtain the current position of the vehicle, Obtaining the position of one or more objects that are close to the current position of the vehicle, Obtaining the category for each of the one or more objects mentioned above, The user device according to claim 11, having the following features.

13. The aforementioned operation further, The vehicle and the generation of icons for each of the one or more objects based on the category for each of the one or more objects, On the display, the current position of the vehicle and the positions of one or more objects are drawn using the generated icons. The user device according to claim 12, having the following features.

14. To obtain the driving order which includes the direction of the vehicle and the route, Receiving a first user input that includes the starting position of the said route, the ending position of the said route, and the path between the said starting position and the said ending position, Receiving a second user input indicating the orientation of the vehicle when the vehicle is traveling along the route, The user device according to claim 11, which includes the following:

15. The aforementioned operation further, Receiving a second position of the vehicle and a second position of each of the one or more objects, which is different from the first position of the vehicle. On the display, instead of the first position of the vehicle and the first position of each of the one or more objects, the second position of the vehicle and the second position of each of the one or more objects are drawn. The user device according to claim 11, having the following features.

16. The change between the second position and the first position of the vehicle corresponds to the movement of the vehicle. Any change between the second position of each of the one or more objects and the first position of each of the one or more objects corresponds to the movement of the corresponding object. The user device according to claim 15.

17. A method for operating an autonomous vehicle, The processor determines the current position of the vehicle, The processor detects one or more objects in the environment surrounding the vehicle's current location, The processor drives or controls the vehicle based on the vehicle's route and orientation. A method that has

18. Determining the position of each of the one or more objects, To draw the position of each of the one or more objects and the current position of the vehicle on the display, On the display, when the vehicle travels a route around one or more objects, the position of each of the one or more objects and the current position of the vehicle are updated. The method according to claim 17, further comprising:

19. It further has the ability to detect gesture input, Driving or controlling the aforementioned vehicle is further based on the gesture input, The method according to claim 17.

20. Receiving user input indicating the route, which includes a starting position, an ending position, and the path between the starting position and the ending position, Determining the route based on the user input, The method according to claim 17, further comprising:

Images