System, method, and computer program product for autonomous and user-controlled vehicle call to a target

Abstract

A processor coupled to the memory is configured to receive an identification of a geographic location associated with a target specified by a user remote from the vehicle. Using a machine learning model, a representation of at least a portion of the vehicle's surroundings is generated using sensor data from one or more sensors of the vehicle. At least a portion of a path to a target location corresponding to the received geographic location is calculated using the generated representation of at least that portion of the vehicle's surroundings. At least one command is provided for automatically navigating the vehicle based on the determined path and updated sensor data from at least a portion of the one or more sensors of the vehicle.

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application is a continuation of U.S. Patent Application No. 16 / 272,273, filed on February 11, 2019, entitled "AUTONOMOUS AND USER CONTROLLED VEHICLE SUMMON TO A TARGET", which claims priority to the specification thereof, and the disclosure of the above application is hereby incorporated by reference in its entirety.

Background Art

[0002] A human driver is typically required to operate a vehicle, but the driving tasks performed by the driver are often complex and thus may be highly fatiguing. For example, retrieving or summoning a vehicle parked in a congested parking lot or garage can be inconvenient and time - consuming. The procedure often includes walking to one's vehicle, performing a three - point turn, inching forward from a tight space without touching adjacent vehicles or walls, and then driving back to the previous position. Some vehicles can be remotely operated, but the path driven by the vehicle is typically limited to a single straight - line path in either the forward or reverse direction, the steering range is limited, and no intelligence is required to navigate the vehicle along its predefined path.

Summary of the Invention

[0003] One embodiment includes a system. The system receives an identification of a geographical location associated with a target specified by a user remote from the vehicle, uses a machine learning model to generate a display of at least a portion of the vehicle's surrounding environment using sensor data from one or more sensors of the vehicle, calculates at least a portion of a route to a target location corresponding to the received geographical location using the generated display of at least a portion of the vehicle's surrounding environment, and provides at least one command for automatically navigating the vehicle based on the determined route and updated sensor data from at least a portion of one or more sensors of the vehicle, and a memory coupled to the processor and configured to provide instructions to the processor.

[0004] Another embodiment includes a method. The method includes receiving an identification of a geographical location associated with a target specified by a user remote from the vehicle, using a neural network to generate a display of at least a portion of the vehicle's surrounding environment using sensor data from one or more sensors of the vehicle, calculating at least a portion of a route to a target location corresponding to the received geographical location using the generated display of at least a portion of the vehicle's surrounding environment, and providing at least one command for automatically navigating the vehicle based on the determined route and updated sensor data from at least a portion of one or more sensors of the vehicle.

[0005] Yet another embodiment includes a computer program product embodied in a non-transitory computer-readable storage medium and including computer instructions. The purpose of the computing instructions is to receive an identification of a geographical location associated with a target specified by a user located remotely from the vehicle, to use a neural network to generate a display of at least a portion of the surrounding environment of the vehicle using sensor data from one or more sensors of the vehicle, to use the generated display of at least a portion of the surrounding environment of the vehicle to calculate at least a portion of a route to a target location corresponding to the received geographical location, and to provide at least one command for automatically navigating the vehicle based on the determined route and updated sensor data from at least a portion of one or more sensors of the vehicle.

Brief Description of the Drawings

[0006] In the following detailed description and the accompanying drawings, various embodiments of the present invention are disclosed.

[0007]

Figure 1

[0008]

Figure 2

[0009]

Figure 3

[0010]

Figure 4

[0011]

Figure 5

[0012]

Figure 6

[0013]

Figure 7

[0014]

Figure 8

[0015]

Figure 9

MODE FOR CARRYING OUT THE INVENTION

[0016] The present invention can be implemented in various ways, including as a process, an apparatus, a system, a composition of matter, a computer program product embodied on a computer-readable storage medium, and / or a processor such as a processor configured to execute instructions stored in a memory coupled to the processor and / or instructions provided by this memory. In this specification, these implementations, or any other form that the present invention may take, may sometimes be referred to as techniques. Generally, the order of the disclosed process steps may be changed within the scope of the present invention. Unless otherwise specified, components such as processors or memories described as being configured to perform a task are implemented as general components temporarily configured to perform that task at a given time or as special components manufactured to perform that task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data such as computer program instructions.

[0017] Hereinafter, one or more embodiments of the present invention will be described in detail together with the accompanying drawings showing the principles of the present invention. The description of the present invention is made in relation to such embodiments, but the present invention is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications, and equivalents. In the following description, many specific details are set forth in order to provide a thorough understanding of the present invention. These details are presented for purposes of illustration, and thus the present invention may be practiced without including some or all of these specific details, in accordance with the claims. Details of well-known techniques in the technical field related to the present invention are omitted so as not to unnecessarily obscure the present invention.

[0018] Disclosed is a technology for autonomously calling a vehicle to a destination. A target geographical location is provided, and the vehicle automatically navigates to the target location. For example, a user provides a location by dropping a pin on a graphical map user interface at the destination location. As another example, the user calls the vehicle to the user's location by designating the user's location as the destination location. Also, the user may select a destination location based on an executable route detected for the vehicle. The destination location may be updated (e.g., when the user is moving around), and thereby the vehicle updates its route to the destination location. Using the designated destination location, the vehicle navigates by generating a display of the vehicle's surrounding environment using sensor data such as video data captured by a camera. In some embodiments, the display is an occupancy grid that details drivable and non-drivable spaces. In some embodiments, the occupancy grid is generated from the camera's sensor data using a neural network. The display of the environment may be further enhanced by auxiliary data such as additional sensor data (including radar data), map data, or other inputs. Using the generated display, a route from the vehicle's current location to the destination location is planned. In some embodiments, the route is generated based on vehicle motion parameters such as the turning radius of the model, vehicle width, vehicle body length, etc. The optimal route is selected based on selection parameters such as distance, speed, number of gear changes, etc. As the vehicle automatically navigates using the selected route, the environment display is continuously updated. For example, as the vehicle moves towards the destination, new sensor data is captured and the occupancy grid is updated. Safety checks that can disable or modify the automatic navigation are continuously performed. For example, ultrasonic sensors can be utilized to detect the potential for collisions. The user can monitor the navigation and cancel the vehicle call at any time. In some embodiments, the vehicle must continuously receive a virtual heartbeat signal from the user in order to continue navigating by the vehicle. The virtual heartbeat can be used to indicate that the user is actively monitoring the vehicle's progress.In some embodiments, the user selects a route and remotely controls the vehicle. For example, the user can remotely control the vehicle by controlling the steering angle, direction, and / or speed of the vehicle. As the user remotely controls the vehicle, safety checks that can disable and / or modify the user control are continuously performed. For example, if an object is detected, the vehicle can be stopped.

[0019] In some embodiments, the system includes a processor configured to receive an identification of a geographical location associated with a target specified by a user located remotely from the vehicle. For example, a user waiting at a pick-up location may specify their geographical location as the destination target. The destination target is received by a vehicle parked at a remote parking spot. In some embodiments, a machine learning model is utilized and sensor data from one or more sensors of the vehicle is used to generate a representation of at least a portion of the vehicle's surrounding environment. For example, sensors such as cameras, radar, ultrasonic, or other sensors capture data of the vehicle's surrounding environment. The data is supplied as input to a neural network using a trained machine learning model, and a representation of the vehicle's surrounding environment is generated. As an example, the representation may be an occupancy grid that describes the drivable space available for the vehicle to pass through. In some embodiments, at least a portion of the route to the target location corresponding to the received geographical location is calculated using the representation generated for at least that portion of the vehicle's surrounding environment. For example, using an occupancy grid representing the vehicle's surrounding environment, a route for navigating the vehicle to the target destination is selected. The value at an entry of the occupancy grid may be the probability and / or cost of passing through the location associated with the grid entry by vehicle. In some embodiments, based on the determined route and updated sensor data from at least a portion of one or more sensors of the vehicle, at least one command for automatically navigating the vehicle is provided. For example, a vehicle controller provides actuator commands for controlling the speed and steering of the vehicle using the selected plan. The commands are used to navigate the vehicle along the selected route. As the vehicle navigates, updated sensor data is captured and the representation of the vehicle's surrounding environment is updated. In various embodiments, the system includes a memory coupled to the processor and configured to provide instructions to the processor.

[0020] FIG. 1 is a flowchart showing one embodiment of a process for automatically navigating a vehicle to a destination target. In some embodiments, the process of FIG. 1 is used to summon a vehicle to a geographical location specified by a user. The user may specify the target location using a mobile app, key fob, vehicle GUI, or the like. Using the target location, the vehicle automatically navigates from its starting position to the target position. In some embodiments, the target location is the user's location, and thus the target location is dynamic. For example, as the user moves, the vehicle's target destination moves to the new location where the user is. In some embodiments, the target location is specified indirectly via a calendar or planning software, etc. For example, the user's calendar is parsed and used to determine the target destination and time from a calendar event. The calendar event may include the location of the event and a time such as an end time. The destination is selected based on the location of the event, and the time is selected based on the end time of the event. The vehicle automatically navigates to arrive at its location at the end time of a dinner, wedding, restaurant reservation, or other end. In some embodiments, the vehicle uses a specified time such as an arrival time or departure time to navigate to the target destination. For example, the user can specify the time at which the vehicle should start an automatic navigation or departure to a specified destination. As another example, the user can specify the time at which the vehicle should arrive at a specified destination. In this case, the vehicle departs before the specified time in order to arrive at the destination at the specified time. In some embodiments, the arrival time is configured considering a threshold time, anticipating the difference between the estimated travel time and the actual travel time. In various embodiments, the process of FIG. 1 is executed in an autonomous vehicle. In some embodiments, a remote server communicating with the autonomous vehicle executes a part of the summoning function. In some embodiments, the process of FIG. 1 is implemented using at least in part the autonomous vehicle system of FIG. 7.

[0021] At 101, a destination is received. For example, a user selects a "Find Me" function from a mobile app on a smartphone device. The "Find Me" function determines the user's location and transmits the user's location to a vehicle calling module. In various embodiments, the vehicle calling module is implemented on one or more modules that may be present on and / or remote from the vehicle. Depending on the embodiment, the target destination may not be the user's location. For example, the user may select a location on a map as the target destination. As another example, the location may be based on a location associated with a calendar event. Depending on the embodiment, the location is selected by dropping a pin icon on the map at the target destination. The location may be longitude and latitude. Depending on the embodiment, the location may include elevation, altitude, or similar measures. For example, in a multi-level parking garage, the location includes an elevation component for distinguishing different levels in the multi-level parking garage. In various embodiments, the destination is a geographical location.

[0022] Depending on the embodiment, the destination includes a position as well as an orientation. For example, to specify both a position and an orientation, a car icon, a triangle icon, or another icon with an orientation is used. By specifying the orientation, the vehicle navigates to the selected target destination and turns in the intended orientation selected by the user. Depending on the embodiment, the orientation is determined by orientation proposals such as proposals based on the environment, the user, other vehicles, and / or other appropriate inferences. For example, an appropriate orientation for the vehicle, such as the direction in which the vehicle should travel on a one-way road, may be determined using a map. As another example, the calling function determines an appropriate orientation of the road based on lanes, signals, other vehicles, etc. For example, the direction in which other vehicles are heading can be used as an orientation proposal. Depending on the embodiment, the orientation is based on the orientation proposed by other autonomous vehicles. For example, depending on the embodiment, the vehicle calling function can communicate with and / or query other vehicles to determine their orientations and use the provided results as orientation proposals. In various embodiments, one or more proposals for the orientation are weighted and used to determine the orientation of the target destination.

[0023] Depending on the embodiment, the destination is determined by responding to a query. For example, the user may request their vehicle to arrive at a designated parking lot at a predetermined time using, for example, a GUI, voice, or another query type. The destination is determined by querying a search engine such as a map search database of the destination parking lot. The search results may be narrowed down using the user's calendar and / or address book. Once the parking lot is identified, instructions and / or traffic flow for entering and / or exiting the parking lot are determined and used in the selection of the destination. Next, the destination is provided to the vehicle calling module.

[0024] Depending on the embodiment, the received destination is a multi-part destination. For example, the destination requires reaching one or more waypoints before reaching the final destination. The waypoints may be used to perform additional control on the route the vehicle takes when navigating to the final destination. For example, the waypoints may be used to follow a preferred traffic pattern for an airport, a parking lot, or another destination. The waypoints may be used to pick up one or more additional passengers or others along the way. For example, the destination can incorporate delays and / or stops for picking up and / or dropping off passengers or goods along the way.

[0025] Depending on the embodiment, the received destination first undergoes one or more verifications and / or safety inspections. For example, the destination may be restricted based on distance so that the user can only select destinations within a predetermined distance (or radius) from the vehicle, such as 100 meters, 10 meters, or another appropriate distance. Depending on the embodiment, this distance is based on local rules and / or regulations. Depending on the embodiment, the destination location must be a valid stopping position. For example, sidewalks, crosswalks, intersections, lakes, etc. are typically not valid stopping positions, and the user may be prompted to select a valid location. Depending on the embodiment, the received destination is changed from the destination initially selected by the user, taking into account safety concerns such as enforcing a valid stopping position.

[0026] In 103, a route to the destination is determined. For example, one or more routes are determined to navigate the vehicle from its current position to the destination received at 101. In some embodiments, the route is determined using a route planner module such as the route planner module 705 of FIG. 7. In various embodiments, the route planner module may be implemented using a cost function with appropriate weights applied to different routes to the destination. In some embodiments, the selected route is based on potential route arcs starting from the vehicle position. The potential route arcs are limited by vehicle dynamics such as the vehicle steering range. In various embodiments, each potential route has a cost. For example, a potential route with a sharp turn has a higher cost than a smoother route. As another example, a potential route with a large speed change has a higher cost than a route with a small speed change. As yet another example, a potential route with many gear changes (e.g., a change from reverse to forward) has a higher cost than a route with few gear changes. In some embodiments, a route with a higher likelihood of encountering a given object is weighted differently from a route with a lower likelihood of encountering the object. For example, the route is weighted based on pedestrians, vehicles, animals, traffic volume, poor lighting, bad weather, tolls, and other likelihoods of encounter. In various embodiments, when determining a route to follow to the destination, a high-cost route is less preferred than a low-cost route.

[0027] At 105, the vehicle navigates to the destination. Using the route to the destination determined at 103, the vehicle automatically navigates to the destination received at 101. Depending on the embodiment, the route may be composed of a plurality of smaller sub - routes. The sub - routes may be implemented in various ways, such as by driving in different gears like forward or reverse. Depending on the embodiment, the route may include actions such as opening a garage door, closing a garage door, passing through a parking gate, waiting for a car lift, verifying payment, charging, making a phone call, sending a message, among others. Additional actions may require stopping the vehicle and / or performing actions that manipulate the surrounding environment. Depending on the scenario, the final destination may be a destination close to the destination received at 101. For example, depending on the scenario, the destination received at 101 may be unreachable, and thus, the final destination is brought as close as possible to that destination. For example, if the user selects a sidewalk, the final destination is a position on the road adjacent to the sidewalk. As another example, if the user selects a crosswalk, the final destination is not within the crosswalk but a position on the road close to the crosswalk.

[0028] In some embodiments, one or more safety inspection items are continuously checked while navigating. For example, using auxiliary data such as sensor data from ultrasonic sensors or other sensors, obstacles such as pedestrians, vehicles, speed bumps, traffic control signals, etc. are identified. The potential for a collision with an object may stop the current navigation and / or modify the route to the destination. In some embodiments, the route to the destination includes the speed at which to move along the route. As another example, in some embodiments, the heartbeat from the user's mobile app must be received in order to continue navigation. The heartbeat may be implemented by asking the user to maintain a connection with the called vehicle, for example, by continuously pressing a button GUI element in the mobile calling app. As another example, the user must keep the button on the key fob depressed (or maintain contact with the sensor) for the vehicle to navigate automatically. When contact is lost, the automatic navigation ends. In some embodiments, when navigation ends, the vehicle safely decelerates and the vehicle is placed in a safe stationary mode, such as pulling over to the side of the road, into a nearby parking space, etc. In some embodiments, depending on the scenario, the vehicle may stop immediately when contact is lost. For example, the user may release the call button on the mobile app, similar to a dead man's device, to indicate that the vehicle should stop immediately. In various embodiments, the deceleration applied to stop the vehicle depends on the vehicle's environment (e.g., whether other vehicles are driving nearby, such as behind the vehicle), whether there are obstacles or pedestrians that may collide in front of the vehicle's route, the speed at which the vehicle is traveling, or other appropriate parameters. In some embodiments, the vehicle's driving speed is limited to a low maximum speed, for example.

[0029] In 107, the vehicle calling function is completed. Upon completion, one or more completion actions may be performed. For example, a notification that the vehicle has arrived at the selected target destination is sent. In some embodiments, the vehicle arrives at a destination approximating its target destination and a notification of the vehicle's position is presented to the user. In some embodiments, the notification is sent to a mobile app, a key fob (e.g., as indicated by a state change associated with the key fob), via a text message, and / or via another appropriate notification channel. Additional instructions may be provided to guide the user to the vehicle. In various embodiments, the vehicle may be placed in a parking lot and one or more vehicle settings may be triggered. For example, the interior lighting of the vehicle may be turned on. The floor lighting may be activated to enhance visibility for entering the vehicle. One or more exterior lights, such as turn indicators, parking lamps, and / or hazard lamps, may be turned on. Exterior lights, such as the front headlights, may be activated to enhance visibility for an approaching occupant. The directional light that is activated may be directed in the direction where the occupant is predicted to appear as they attempt to reach the vehicle. In some embodiments, an audio alert or an audio notification such as music is played. Based on the user's preference, welcome music or a similar audio may be played inside the vehicle. Similarly, the temperature control of the vehicle may be activated to condition the interior environment of the vehicle, such as warming or cooling the interior to a desired temperature and / or humidity for the occupant. The seat may be warmed (or ventilated). The warming of the steering wheel may be activated. The vents can be directed according to preference. The doors may be unlocked and opened for the occupant. If the destination is a charging station, the vehicle can be oriented so that its charger port aligns with the charger. In some embodiments, the user can set vehicle preferences, including interior environment, interior lighting, exterior lighting, audio system, and other vehicle preferences, anticipating the occupant.

[0030] In various embodiments, the position of the vehicle is updated during and upon completion of a call. For example, the updated position may be reflected in the companion mobile app. In some embodiments, the completion act includes updating an annotated map of the traversed route and encountered environment with the latest sensor data that has been captured and analyzed. The annotated map may be updated to reflect potential obstacles, traffic signs, parking preferences, traffic patterns, pedestrian walking patterns, and / or other suitable route planning metadata that may be useful for future navigation and / or route planning. For example, in the annotated map, encountered speed bumps, crosswalks, potholes, empty parking lots, charging locations, gas stations, etc. are updated. In some embodiments, data corresponding to the encounter is used as potential training data to improve call functions such as perception, route planning, safety verification, and other functions for the current vehicle / user as well as other vehicles and users. For example, an empty parking lot may be used for route planning of other vehicles. As another example, a charging station or gas station is used in route planning. The vehicle can be routed to the charging station and its charger port can be oriented to align with the charger.

[0031] Figure 2 is a flowchart showing one embodiment of a process for receiving a target destination. In some embodiments, the target destination is selected and / or provided by the user. The destination target is associated with a geographical location and is used as a goal for automatically navigating the vehicle. In the example shown, the process of receiving a target destination can start from two or more starting points. The two starting points in Figure 2 are two examples of starting the reception of the destination. Additional methods are also possible. In some embodiments, the process of Figure 2 is executed at 101 of Figure 1.

[0032] At 201, the location of the destination is received. In some embodiments, the destination is provided as a geographical location. For example, longitude and latitude values are provided. In some embodiments, altitude is also provided. For example, the altitude associated with a particular floor of a multi-story parking garage is provided. In various embodiments, the destination location is provided by the user, via a smartphone device, via the vehicle's media console, or by other means. In some embodiments, this location is received along with the time associated with departure or arrival at the destination. In some embodiments, one or more destinations are received. For example, in some scenarios, a multi-stage destination including multiple stop locations is received. In some embodiments, the destination location includes an orientation or direction of travel. For example, the direction of travel indicates the direction the vehicle should face upon arrival at the destination.

[0033] At 203, the user location is determined. For example, the user's location is determined using a global positioning system or other location recognition technology. In some embodiments, the user's location is approximated by the location of a key fob, the user's smartphone device, or another device controlled by the user. In some embodiments, the user's location is received as a geographical location. Similar to 201, this location may be a pair of longitude and latitude, and in some embodiments, may include altitude. In some embodiments, the user's location is dynamic and is continuously updated, or updated at predetermined intervals. For example, the user can move to a new location and the received location is updated. In some embodiments, the destination location includes an orientation or direction of travel. For example, the direction of travel indicates the direction the vehicle should face upon arrival at the destination.

[0034] At 205, the destination is verified. For example, the verification of the destination is performed to confirm that the destination is reachable. In some embodiments, the destination must be within a predetermined distance from the starting position of the vehicle. For example, some regulations require restricting the automatic navigation of the vehicle to 50 meters or less. In various embodiments, if the destination is invalid, the user may be requested to provide a new location. In some embodiments, an alternative destination may be proposed to the user. For example, if the user selects the wrong direction, direction candidates are provided. As another example, if the user selects a no-parking area, a valid parking area such as the nearest valid parking area is proposed. Once verified, the selected destination is provided to step 207.

[0035] At 207, the destination is provided to the route planner module. In some embodiments, the route of the vehicle is determined using the route planner module. The destination provided to the route planner module at 207 is the verified destination. In some embodiments, the destination is provided as a destination including one or more locations, for example, a plurality of stop locations. The destination may include a two-dimensional location such as a latitude and longitude location. Alternatively, in some embodiments, the destination includes altitude. For example, in a multi-level parking garage, a bridge, or other predetermined drivable areas, a plurality of drivable planes have the same two-dimensional location. The destination may also include a travel direction for specifying the direction in which the vehicle should face upon arrival at the destination. In some embodiments, the route planner module is the route planner module 705 of FIG. 7.

[0036] Figure 3 is a flowchart showing an embodiment of a process for automatically navigating a vehicle to a destination target. For example, using the process of Figure 3, a display of the vehicle's surrounding environment is generated, which is used to determine one or more routes to the destination. The vehicle automatically navigates using the determined route. As additional sensor data is updated, the display of the surrounding environment is updated. In some embodiments, the process of Figure 3 is implemented using the autonomous vehicle system of Figure 7. In some embodiments, step 301 is executed at 101 of Figure 1, steps 303, 305, 307 and / or 309 are executed at 103 of Figure 1, steps 311 and / or 313 are executed at 105 of Figure 1, and / or step 315 is executed at 107 of Figure 1. In some embodiments, a neural network is used to guide the route that the vehicle follows. For example, using a machine learning network, steering values and / or acceleration values are predicted to navigate the vehicle to follow a route to the destination target.

[0037] At 301, a destination is received. For example, a geographical location is received via a mobile app, key fob, the vehicle's control center, or another suitable device. In some embodiments, the destination is a location and orientation. In some embodiments, the destination includes altitude and / or time. In various embodiments, the destination is received using the process of Figure 2. In some embodiments, the destination is dynamic and a new destination may be received as appropriate. For example, if the user selects the "Find Me" function, the destination is updated by following the user's location. Essentially, the vehicle can follow the user like a pet.

[0038] At 303, video data is received. For example, camera image data is received using one or more camera sensors attached to a vehicle. In some embodiments, the image data is received from sensors that cover the surrounding environment of the vehicle. The video data may be preprocessed to enhance the usefulness of the data for analysis. For example, one or more filters may be applied to reduce noise in the video data. In various embodiments, the video data is continuously captured and the surrounding environment of the vehicle is updated.

[0039] At 305, a drivable space is determined. In some embodiments, the drivable space is determined by applying inferences to the video data received at 303 using a neural network. For example, a convolutional neural network (CNN) is applied and the video data is used to determine the drivable and non-drivable spaces of the surrounding environment of the vehicle. The drivable space includes areas where the vehicle can travel. In various embodiments, the drivable space is free of obstacles so that the vehicle can travel using a path through the determined drivable space. In various embodiments, a machine learning model is trained to determine the drivable space and is deployed in the vehicle to automatically analyze and determine the drivable space from the image data.

[0040] In some embodiments, the video data is supplemented with additional data such as additional sensor data. The additional sensor data may include ultrasonic, radar, lidar, audio, or other suitable sensor data. Also, annotated data such as map data may be included as additional data. For example, an annotated map may annotate vehicle lanes, speed lines, intersections, and / or other driving metadata. The additional data may be used as an input to a machine learning model or may be consumed downstream when creating occupancy grids to improve the results in determining the drivable space.

[0041] At 307, an occupancy grid is generated. The occupancy grid that displays the environment of the vehicle is generated using the drivable space determined at 305. In some embodiments, the occupancy grid is a two-dimensional occupancy grid that displays the entire plane in which the vehicle is located (e.g., 360 degrees along the longitude and latitude axes). In some embodiments, the occupancy grid includes a third dimension that also describes altitude. For example, areas with multiple drivable paths at different altitudes, such as multi-story parking structures, elevated roadways, etc., can be displayed by a three-dimensional occupancy grid.

[0042] In various embodiments, the occupancy grid includes a drivability value at each grid position corresponding to one position in the surrounding environment. The drivability value at each position may be the probability that the position is drivable. For example, a sidewalk can be designed to have a drivability value of zero, while a gravel road can have a drivability value of 0.5. The drivability value may be a normalized probability having a range of 0 to 1 and is based on the drivable space determined at 305. In some embodiments, each position of the grid includes a cost metric associated with the cost (or disadvantage / advantage) of passing through that position. The cost value of each grid position within the occupancy grid is based on the drivable value. The cost value may further depend on additional data such as preference data. For example, route preferences can be configured to avoid toll roads, carpool lanes, school zones, etc. In various embodiments, the route preference data can be learned via a machine learning model and is determined at 305 as part of the drivable space. In some embodiments, the route preference data is configured by the user and / or operator to optimize the route adopted for navigation to the destination received at 301. For example, the route preference can be optimized to enhance safety, convenience, travel time, and / or comfort, among other objectives. In various embodiments, the preference is an additional weight used to determine the cost value of each position grid.

[0043] In some embodiments, the occupancy grid is updated using auxiliary data such as additional sensor data. For example, ultrasonic sensor data that captures neighboring objects is used to update the occupancy grid. Additionally, sensor data from lidar, radar, audio, etc. may also be used. In various embodiments, an annotated map may be partially used to generate the occupancy grid. For example, roads and their characteristics (speed limits, lanes, etc.) can be used to reinforce video data for generating the occupancy grid. As another example, occupancy data from other vehicles can be used to update the occupancy grid. For example, neighboring vehicles equipped with similar functions can share sensor data and / or occupancy grid results.

[0044] In some embodiments, the occupancy data is initialized using the most recently generated occupancy grid. For example, when the vehicle no longer captures new data and / or parks or stops, the final occupancy grid is saved. For example, when the vehicle is called and an occupancy grid is needed, the last generated occupancy grid is loaded and used as the initial occupancy grid. This optimization significantly improves the accuracy of the initial grid. For example, some objects may be difficult to detect from a stationary state, but assume they were detected in the last saved occupancy grid during vehicle movement approaching the current parking position.

[0045] At 309, the route goal is determined. A search is performed to determine a route for navigating the vehicle from its current position to the destination received at 301 using the occupancy grid. The potential routes are based in part on the vehicle's motion characteristics such as turning radius, vehicle width, vehicle body length, etc. Each vehicle model may be configured to have specific vehicle motion characteristics. In some embodiments, the route search is set to implement configurable constraints and / or goals. Examples of constraints include that the vehicle cannot drive on the sidewalk, that the vehicle should limit the number of sharp turns, that the vehicle should limit the number of gear changes (e.g., from reverse to forward, or vice versa), etc. The constraints / goals may be implemented as weighted costs in the cost function. In various embodiments, the initial position of the vehicle includes x, y, and heading values. The x and y values may correspond to longitude and latitude values. One or more potential routes are determined from the initial position towards reaching the goal. In some embodiments, the route is composed of one or more route primitives such as arc primitives. A route primitive describes a route (and route goal) along which the vehicle can navigate to reach the destination.

[0046] In various embodiments, the selected path goal is selected based on a cost function. The cost function is executed for each of the potential paths. Each potential path passes through a series of grids of the occupancy grid, and each grid has a cost value for rewarding or penalizing passing through its grid position. For the path goal, the path with the optimal cost value is selected. The path goal may include one or more path primitives, such as arcs, for modeling the movement of the navigating vehicle. For example, a pair of two path primitives can represent the reverse movement of the vehicle and the subsequent forward movement. The reverse path is displayed as one arc, and the forward path is displayed as another arc. As another example, the vehicle path may include a three-point turn. Each primitive of the turn can be represented as an arc path. At the end of each path primitive, the vehicle has new x, y, and heading values. In some embodiments, the vehicle includes an altitude value, for example, to support navigation between different floors of a multi-story parking lot. Although arc path primitives are used to define the goal path, other suitable geometric primitives may also be used. In some embodiments, each path includes a speed parameter. For example, the speed parameter can be used to propose a driving speed along the path. Initial speed, maximum speed, acceleration, maximum acceleration, and other speed parameters can be used to control how the vehicle navigates along the path. In some embodiments, the maximum speed is set low to prevent the vehicle from exceeding its speed. For example, a low maximum speed may be implemented to allow for quick intervention by the user.

[0047] In various embodiments, the determined route goal is communicated to the vehicle controller for navigating along the route. In some embodiments, the route goal is first converted into a series of waypoints along the route for use by the vehicle controller. In some embodiments, route planning is continuously executed and a new route may be determined while traversing the current route. In some embodiments, the intended route is no longer reachable, for example, the route is blocked, and a new route is determined. Route planning at 309 may be executed less frequently than other functions of the process of FIG. 3. For example, route planning may be executed less frequently than the determination of the drivable space at 305 and / or the vehicle control for navigation at 311. By making the determination of the route goal less frequent than the update of the occupancy grid, the process of route planning is more efficient and utilizes a more accurate external representation.

[0048] In some embodiments, two or more routes to the destination are viable and multiple route goals are provided to the user. For example, the user is presented with two or more route goals as options. The user can select the route goal to use to navigate the vehicle to the destination, for example, using a GUI, voice command, or the like. As an example, the user is presented with two routes from the vehicle to the destination. The first route has a shorter estimated travel time but more direction changes and requires frequent gear changes. The second route is smoother but takes more time. The user can select the route goal from these two options. In some embodiments, the user can select the route goal and modify the route. For example, the user can adjust the selected route to navigate around a particular obstacle such as a congested intersection.

[0049] At 311, the vehicle automatically navigates to the route goal. Using one or more route goals determined at 309, the vehicle automatically navigates along the goal route from the current position to reach the arrival destination. For example, the vehicle controller receives the goal route and then implements the vehicle control necessary to navigate the vehicle along the route. The route goal may be received as a series of waypoints along the route for navigation. In some embodiments, the vehicle controller converts route primitives, such as arcs, into waypoints. In various embodiments, the vehicle is controlled by the vehicle controller transmitting actuator parameters to the vehicle actuators. In some embodiments, the vehicle controller is the vehicle controller 707 of FIG. 7, and the vehicle actuators are the vehicle actuators 713 of FIG. 7. Steering, braking, acceleration, and / or other operating functions are actuated using the vehicle actuators.

[0050] In some embodiments, as the vehicle navigates to the route goal, the user can adjust the navigation / operation of the vehicle. For example, the user can adjust the navigation by providing an input such as "Steer further to the left". As an additional example, the user can increase or decrease the speed of the vehicle and / or adjust the steering angle during navigation.

[0051] At 313, it is determined whether the vehicle has arrived at the destination received at 301. If the vehicle has arrived at the destination, the process continues to 315. If the vehicle has not arrived at the destination, the process loops back to 301 to potentially receive an updated destination and automatically navigate to the selected (potentially updated) destination. In various embodiments, the vehicle may have arrived at the destination but not be at the exact destination location. For example, the selected destination may not be a drivable location or may no longer be a drivable location. As an example, another vehicle may be parked at the selected destination. As yet another example, the user may move to a location such as a passenger waiting area that is not drivable. In certain situations, the vehicle is determined to have arrived at the destination if it has arrived at a location that is determined to be the closest location to which the vehicle can reach the destination. The closest reachable destination may be based on the route determined at 309. In some embodiments, the closest reachable destination is based on a cost function used to calculate a potential route between the current location and the destination. In some embodiments, the difference between the reached location and the received destination is based on the accuracy of the technology available for location determination. For example, the vehicle may be parked within the accuracy of the available global positioning system.

[0052] At 315, the call is completed. In various embodiments, when the call function is completed, one or more completion actions are performed. For example, the completion actions described in connection with 107 of FIG. 1 are performed. In some embodiments, as a completion action, the occupancy grid generated at 307 is saved and / or exported. The occupancy grid may be saved locally to the vehicle and / or saved to a remote server. Once saved, the grid can be used by the vehicle and / or shared with other vehicles having potentially overlapping routes.

[0053] FIG. 4 is a flowchart showing an embodiment of a process for training and applying a machine learning model to generate a display of the surrounding environment of a vehicle. In some embodiments, the process of FIG. 4 is used to determine a drivable space for generating an occupancy grid using at least partially sensor data. The sensor data used to train and / or apply a trained machine learning model may correspond to image data captured from a vehicle using a camera sensor. In some embodiments, this process is used to create and deploy a machine learning model for the autonomous vehicle system of FIG. 7. In some embodiments, the process of FIG. 4 is used to determine a drivable space at 305 in FIG. 3 and to generate an occupancy grid at 307 in FIG. 3.

[0054] At 401, training data is prepared. In some embodiments, a training data set is created using sensor data including image data. The sensor data may include still images and / or videos from one or more cameras. Additionally, relevant sensor data may be provided using sensors such as radar, lidar, ultrasonic, etc. In various embodiments, the sensor data is paired with corresponding vehicle data to assist in identifying features of the sensor data. For example, position data and position change data can be used to identify the positions of relevant features such as lanes, traffic control signals, objects, etc. within the sensor data. In some embodiments, the training data is prepared to train a machine learning model to identify a drivable space. The prepared training data may include data for training, validation, and testing. In some embodiments, the data format is compatible with the machine learning model used in the deployed deep learning application.

[0055] In 403, a machine learning model is trained. For example, the machine learning model is trained using the data prepared in 401. In some embodiments, the model is a neural network such as a convolutional neural network (CNN). In various embodiments, the model includes a plurality of intermediate layers. In some embodiments, the neural network may include multiple layers including a plurality of convolutional layers and pooling layers. In some embodiments, the training model is verified using a validation data set created from the received sensor data. In some embodiments, the machine learning model is trained to predict an operable space from image data. For example, the operable space of the surrounding environment of a vehicle can be inferred from an image captured by a camera. In some embodiments, the image data is enhanced with other sensor data such as radar data or ultrasonic sensor data to improve accuracy.

[0056] In 405, the trained machine learning model is deployed. For example, the trained machine learning model is installed in a vehicle as an update to a deep learning network. In some embodiments, the deep learning network is part of a perception module such as the perception module 703 of FIG. 7. The trained machine learning model may be installed as an over-the-air update. In some embodiments, this update is a firmware update transmitted using a wireless network such as WiFi or a cellular network. In some scenarios, the newly trained machine learning model is installed during a vehicle inspection.

[0057] At 407, sensor data is received. For example, the sensor data is captured from one or more sensors of a vehicle. In some embodiments, the sensor is an image sensor such as the image sensor 701 of FIG. 7 and / or the additional sensor 709 of FIG. 7 that is used to capture video data. The image sensors may include cameras mounted behind the windshield, forward and / or side cameras mounted on the pillars, rear cameras, and other image sensors. In various embodiments, the sensor data is in a format that is used as an input by a machine learning model trained at 403 or is converted to the format. For example, the sensor data may be raw image data or processed image data. In some embodiments, the sensor data is data captured from ultrasonic sensors, radar, LiDAR sensors, microphones, or other suitable technologies. In some embodiments, the sensor data is preprocessed using an image preprocessor such as an image preprocessor during a preprocessing step. For example, the image may be normalized by removing distortion, noise, etc.

[0058] At 409, the trained machine learning model is applied. For example, the machine learning model trained at 403 is applied to the sensor data received at 407. In some embodiments, the application of the model is performed using a deep learning network by a perception module such as the perception module 703 of FIG. 7. In various embodiments, by applying the trained machine learning model, the drivable space is identified and / or predicted. For example, the drivable space in the surrounding environment of the vehicle is inferred. In various embodiments, by applying the machine learning model, the vehicle, obstacles, lanes, traffic control signals, map functions, distances to objects, speed limits, etc. are identified. The detected features can be used to determine the drivable space. In some embodiments, using traffic control and other driving functions, navigation parameters such as speed limits, stop positions, parking areas, oncoming directions, etc. are determined. For example, stop signs, parking spaces, lanes, and other traffic control functions are detected and utilized to determine the drivable space and driving parameters.

[0059] At 411, an occupancy grid is generated. For example, using the output of the trained machine learning model applied at 409, an occupancy grid for path planning to determine a goal path for navigating the vehicle is generated. The occupancy grid may be generated as described in relation to 307 of FIG. 3 and / or may be generated using the process of FIG. 5.

[0060] FIG. 5 is a flowchart showing an embodiment of a process for generating an occupancy grid. For example, the occupancy grid can be generated for path planning using the drivable space determined from sensor data. The occupancy grid can be augmented by additional sensor data, metadata from additional sources such as an annotated map, and / or previously generated occupancy grids. In some embodiments, the process of FIG. 5 is executed at 103 of FIG. 1, 307 of FIG. 3, and / or 411 of FIG. 4. In some embodiments, the process of FIG. 5 is implemented using the autonomous vehicle system of FIG. 7.

[0061] In some embodiments, the occupancy grid is generated prior to route planning. For example, the generated occupancy grid is then presented to the user for review via a GUI on a smartphone device, or a display in the vehicle, etc. The user can view the occupancy grid and select a target destination. The user can specify where on the curb they want the vehicle to be positioned. Once selected, the vehicle can navigate to the target destination.

[0062] At 501, the stored occupancy grid is loaded. In some embodiments, a pre-generated occupancy grid corresponding to the current position of the vehicle is loaded. The accuracy of the vehicle's surrounding environment can be improved by initializing the occupancy grid with a pre-stored occupancy grid. This optimization significantly improves the accuracy of the initial grid. For example, some objects may be difficult to detect from a stationary state, but should be detected in the last saved occupancy grid when the vehicle was moving closer to the current parking position.

[0063] At 503, the drivable space is received. For example, the drivable space is received as an output from a neural network such as a convolutional neural network. In some embodiments, the drivable space is continuously updated as new sensor data is captured and analyzed. The received drivable space may be segmented into grid positions. In some embodiments, in addition to the drivable space, other video-based measurements are received. For example, objects such as curbs, vehicles, pedestrians, cones, others, etc. are detected and received.

[0064] At 505, auxiliary data is received. For example, the auxiliary data is used to update the occupancy grid and further enhance its accuracy. The auxiliary data may include data from sensors such as ultrasonic sensors or radars. Also, the auxiliary data may include occupancy data generated from other vehicles. For example, a vehicle's mesh network may share occupancy data based on the time and location of the occupancy data. As another example, the occupancy grid may be reinforced using data from an annotated map. Map data such as speed limits, lanes, drivable space, traffic patterns, etc. may be loaded via the data of the annotated map.

[0065] In some embodiments, safety data used to disable or modify navigation is received as auxiliary data. For example, a collision warning system inputs data at 505 that is used to invalidate grid values to reveal potential or imminent collisions. In some embodiments, the auxiliary data includes data provided by the user. For example, the user may include images or videos that are useful for identifying destinations such as parking positions and / or orientations. The received data may be used to modify the occupancy grid.

[0066] At 507, the occupancy grid is updated. The occupancy grid is updated using the data received at 503 and / or 505. The updated grid may include, for each grid position, a value corresponding to the probability value of that grid position. The value may be a cost value associated with navigating through that grid position. In some embodiments, the value also includes an operable value corresponding to the probability that the grid position is an operable area. In various embodiments, the updated occupancy grid may be stored and / or exported. For example, the grid data may be uploaded to a remote server or stored locally. As an example, the grid data may be stored when the vehicle is parked and may later be used to initialize the grid. As another example, the grid data may be shared with associated vehicles such as multiple vehicles where a route or potential route intersects.

[0067] When the occupancy grid is updated, the process loops back to 503 and the occupancy grid is continuously updated with newly received data. For example, as the vehicle navigates along a route, new operable data and / or auxiliary data is received and the occupancy grid is updated. The newly updated grid may be used to refine and / or update the goal route used for automatic navigation. For example, a space that was previously free may be blocked at that time. Similarly, a space that was previously blocked may be open.

[0068] FIG. 6 is a flowchart showing an embodiment of a process for automatically navigating to a destination target. The process of FIG. 6 can be used to navigate a vehicle from its current position to a destination target position using the determined planned goal. In various embodiments, navigation is automatically performed by a vehicle controller using vehicle actuators to modify the steering and speed of the vehicle. The process of FIG. 6 implements a plurality of safety checks to enhance the safety of navigating the vehicle. The safety checks enable navigation to be terminated and / or modified. For example, it is possible to implement a virtual heartbeat that requires the user to constantly maintain contact with the vehicle to confirm that the user is monitoring the progress of the vehicle. In some embodiments, navigation utilizes a route goal to navigate the vehicle along an optimal route to the destination target. The route goal may be received as a route primitive such as an arc, a series of points along a selected route, and / or another form of route primitive. In some embodiments, the process of FIG. 6 is executed at 105 of FIG. 1 and / or 311 of FIG. 3. In some embodiments, this process is implemented using the autonomous vehicle system of FIG. 7.

[0069] In some embodiments, the process of FIG. 6 can be used by a user to remotely control the vehicle. For example, the user can remotely control the vehicle by controlling the steering angle, direction, and / or speed of the vehicle through vehicle adjustment. As the user remotely controls the vehicle, safety checks that can disable and / or modify user control are constantly performed. For example, if an object is detected, or if communication with the remote user is interrupted, the vehicle can be stopped.

[0070] At 601, vehicle adjustments are determined. For example, vehicle speed and steering adjustments are determined so as to hold the vehicle on the route goal. In some embodiments, the vehicle adjustments are determined by a vehicle controller such as vehicle controller 707 of FIG. 7. In some embodiments, the vehicle controller determines distances, speeds, headings, and / or other driving parameters for controlling the vehicle. In some embodiments, a maximum speed of the vehicle is determined and used to limit the vehicle speed. The maximum speed may be implemented to increase navigation safety and / or to allow sufficient reaction time for the user to terminate the calling function.

[0071] At 603, the vehicle is adjusted to keep its route along the route goal. For example, the vehicle adjustments determined at 601 are implemented. In some embodiments, a vehicle actuator such as vehicle actuator 713 of FIG. 7 implements the vehicle adjustments. The vehicle actuator adjusts the steering and / or speed of the vehicle. In various embodiments, all adjustments can be logged and uploaded to a remote server for later scrutiny. For example, if there are safety concerns, the vehicle's operation, route goal, destination location, current location, travel speed, and / or other driving parameters may be scrutinized to identify potential areas for improvement.

[0072] At 605, the vehicle is operated according to the vehicle adjustments and the vehicle operation is monitored. For example, the vehicle operates according to the instructions by the vehicle adjustments applied at 603. In various embodiments, the operation of the vehicle is monitored to implement safety, comfort, performance, efficiency, and other operation parameters.

[0073] At 607, it is determined whether a fault has been detected. If a fault has been detected, the process continues to 611. If no fault has been detected, the process proceeds to 605 and vehicle operation monitoring continues. In some embodiments, the fault is detected by a collision or object sensor such as an ultrasonic sensor. In some embodiments, the fault may be transmitted via a network interface. For example, a fault detected by another vehicle may be shared and received. In various embodiments, the detected fault can be used to notify other components of the autonomous vehicle system, such as those related to the generation of the occupancy grid, but is also received by the navigation component so that the vehicle can respond immediately to the detected fault.

[0074] At 609, it is determined whether communication with the user has been interrupted. If communication with the user has been interrupted, the process continues to 611. If communication with the user has not been interrupted, the process proceeds to 605 and vehicle operation monitoring continues. In some embodiments, continuous communication with the user is required to activate the automatic navigation. For example, a virtual heartbeat is transmitted from the user. The heartbeat may be transmitted from the user's smartphone device or from another suitable device such as a key fob. In some embodiments, as long as the user activates the virtual heartbeat, the virtual heartbeat is received and communication is not interrupted. When the user stops transmitting the virtual heartbeat, in response, communication with the user is considered to have been interrupted and the vehicle navigation responds accordingly at 611.

[0075] In some embodiments, the virtual heartbeat is implemented (and continuously transmitted to maintain communication) as long as the user continuously touches a heartbeat switch, button or other user interface device. When the user breaks contact with the appropriate user interface element, the virtual heartbeat is no longer transmitted and communication is interrupted.

[0076] At 611, navigation is disabled. In response to detecting a fault and / or losing contact with the user, the automatic navigation is disabled. For example, if the vehicle is traveling at a low speed, the vehicle can stop immediately. If the vehicle is traveling faster, the vehicle stops safely. A safe stop may require gentle braking and determining a safe stop position such as the side of the road or a parking lot. In various embodiments, disabling the navigation may require the user to look ahead and continue the automatic navigation. In some embodiments, the disabled navigation resumes when the detected fault no longer exists.

[0077] In some embodiments, if the navigation is disabled due to a detected fault, the vehicle is re-routed to the destination using a new route. For example, a new route goal is determined to avoid the detected fault. In various embodiments, the occupancy grid is updated to include the detected fault, and a new route goal is determined using the updated occupancy grid. When a viable new route is determined, the navigation can resume in a timely manner, for example, when contact is re-established. In some embodiments, the new route is determined at the time the navigation is disabled. For example, when contact is lost, the route to the destination is re-checked. If appropriate, an existing route can be used; otherwise, a new route can be selected. In various embodiments, if no viable route is found, the vehicle remains stopped. For example, the vehicle is brought to a complete stop.

[0078] FIG. 7 is a block diagram showing an embodiment of an autonomous vehicle system for automatically navigating a vehicle to a destination target. This autonomous vehicle system includes various components that can be used in combination to automatically navigate a vehicle to a target geographical location. In the illustrated example, the autonomous vehicle system includes an in-vehicle component 700, a remote interface component 751, and a navigation server 761. The in-vehicle component 700 is a component mounted on the vehicle. The remote interface component 751 is one or more remote components that can be used remotely from the vehicle to automatically navigate the vehicle. For example, the remote interface component 751 includes a smartphone app executed on a smartphone device, a key fob, a GUI for controlling the vehicle such as a website, and / or another remote interface component. The navigation server 761 is an optional server used to facilitate the navigation function. The navigation server 761 is a remote server and can function as a remote cloud server and / or storage. Depending on the embodiment, the autonomous vehicle system of FIG. 7 is used to implement functions associated with the processes of FIGS. 1-6 and the user interfaces of FIGS. 8-9.

[0079] In the illustrated example, the in-vehicle component 700 is an autonomous vehicle system including a video sensor 701, a perception module 703, a path planner module 705, a vehicle controller 707, an additional sensor 709, a safety controller 711, a vehicle actuator 713, and a network interface 715. In various embodiments, the different components are communicatively connected. For example, sensor data from the video sensor 701 and the additional sensor 709 is supplied to the perception module 703. The output of the perception module 703 is supplied to the path planner module 705. The output of the path planner module 705 and the sensor data from the additional sensor 709 are supplied to the vehicle controller 707. In some embodiments, the output of the vehicle controller 707 is a vehicle control command that is supplied to the vehicle actuator 713 to control vehicle operations such as the speed, braking, and / or steering of the vehicle, among others. In some embodiments, the sensor data from the additional sensor 709 is supplied to the vehicle actuator 713 to perform additional safety checks. In various embodiments, the safety controller 711 is connected to one or more components such as the perception module 703, the vehicle controller 707, and / or the vehicle actuator 713 to implement safety checks in each module. For example, the safety controller 711 may receive additional sensor data from the additional sensor 709 to disable the automatic navigation.

[0080] In various embodiments, sensor data, machine learning results, perception module results, path planning results, safety controller results, among others, can be transmitted to the navigation server 761 via the network interface 715. For example, the sensor data can be transmitted to the navigation server 761 via the network interface 715 to collect training data for improving the performance, comfort and / or safety of the vehicle. In various embodiments, the network interface 715 is used, among other things, to communicate with the navigation server 761, make phone calls, send and / or receive text messages, and transmit sensor data based on vehicle operation. Depending on the embodiment, the in-vehicle component 700 may optionally include additional or fewer components. For example, in some embodiments, the in-vehicle component 700 includes an image pre-processor (not shown) for enhancing sensor data. As another example, the image pre-processor may be used to normalize or transform an image. In some embodiments, noise, distortion and / or blur are removed or reduced during the pre-processing step. In various embodiments, the image is adjusted or normalized to improve the analysis results of machine learning. For example, the white balance of an image is adjusted, among other things, taking into account different lighting operating conditions such as daylight, sunny, cloudy, dusk, sunrise, sunset, and nighttime conditions. As another example, an image captured with a fish-eye lens may be distorted, and the image pre-processor may be used to transform the image to remove or correct the warping. In various embodiments, one or more of the in-vehicle components may be allocated to a remote server such as the navigation server 761.

[0081] Depending on the embodiment, the video sensor 701 includes one or more video sensors. In various embodiments, the video sensor 701 may be attached to the vehicle at different positions of the vehicle and / or directed in one or more different directions. For example, the video sensor 701 may be attached to the front, side, rear, and / or roof of the vehicle, and / or in other directions, such as forward, backward, or sideways. Depending on the embodiment, the video sensor 701 is an image sensor such as a high dynamic range camera. For example, a forward-facing high dynamic range camera captures image data in front of the vehicle. Depending on the embodiment, a plurality of sensors for capturing data are attached to the vehicle. For example, depending on the embodiment, eight surround cameras are attached to the vehicle, providing a 360-degree field of view around the vehicle up to 250 meters ahead. Depending on the embodiment, the camera sensors include a wide-angle front camera, a narrow-angle front camera, a rearview camera, a front-view side camera, and / or a rear-view side camera. Various camera sensors are used to capture the surrounding environment of the vehicle, and the captured images are provided for deep learning analysis.

[0082] Depending on the embodiment, the video sensor 701 is not attached to the vehicle as an in-vehicle component 700. For example, the video sensor 701 may be attached to a neighboring vehicle and / or to a road or the environment, and be included as part of a deep learning system for capturing sensor data. Depending on the embodiment, the video sensor 701 includes one or more cameras that capture the road surface on which the vehicle is traveling. For example, one or more forward-facing cameras and / or pillar cameras capture road markings within the lane in which the vehicle is traveling. The video sensor 701 may include both image sensors capable of capturing still images and / or videos. The data may be captured over a period of time, such as a series of data captured over a period of time.

[0083] In some embodiments, the perception module 703 is used to analyze sensor data to generate a display of the vehicle's surrounding environment. In some embodiments, the perception module 703 utilizes a machine learning network trained to generate an occupancy grid. The perception module 703 may take as input sensor data including data from the vision sensor 701 and / or additional sensors 709 using a deep learning network. The deep learning network of the perception module 703 may be an artificial neural network such as a convolutional neural network (CNN) trained with respect to inputs such as sensor data, and its output is provided to the path planner module 705. As an example, the output may include a drivable space of the vehicle's surrounding environment. In some embodiments, the perception module 703 receives at least sensor data as input. Additional inputs may include scene data describing the vehicle's surrounding environment and / or vehicle specifications such as the vehicle's operating characteristics. The scene data may include scene tags describing the vehicle's surrounding environment such as rainfall, wet roads, snowfall, puddles, heavy traffic, highways, cities, school districts, others. In some embodiments, the perception module 703 is utilized in the processes of 103 in FIG. 1, 305 and / or 309 in FIG. 3, 411 in FIG. 4, and / or FIG. 5.

[0084] In some embodiments, the path planner module 705 is a path planning component for selecting an optimal path to navigate the vehicle from one location to another. The path planning component may utilize an occupancy grid and a cost function to select the optimal route. In some embodiments, a potential path is composed of one or more path primitives such as arc primitives that model the vehicle's operating characteristics. In some embodiments, the path planner module 705 is utilized in step 103 of FIG. 1 and / or step 309 of FIG. 3. In various embodiments, the path planner module 705 is executed less frequently or with a lower update frequency than other components such as the perception module 703 and the vehicle controller 707.

[0085] In some embodiments, the vehicle controller 707 processes the output of the path planner module 705 and is utilized to convert the selected path into vehicle control actions or commands. In some embodiments, the vehicle controller 707 is utilized to control the vehicle to automatically navigate to the selected destination target. In various embodiments, the vehicle controller 707 can adjust the speed, acceleration, steering, braking, etc. of the vehicle by sending commands to the vehicle actuator 713. For example, in some embodiments, the vehicle controller 707 is used to control the vehicle to maintain the vehicle's position along the path from its current location to the selected destination.

[0086] In some embodiments, the vehicle controller 707 is used to control vehicle lighting such as brake lights, turn indicators, headlights, etc. In some embodiments, the vehicle controller 707 is used to control the audio state of the vehicle such as the vehicle's sound system, playback of voice alerts, activation of microphones, activation of horns, etc. In some embodiments, the vehicle controller 707 is used to control a notification system including a warning system to inform the driver and / or passengers of driving events such as potential collisions or approaching the intended destination. In some embodiments, the vehicle controller 707 is used to adjust sensors such as the vehicle's sensor 701. For example, the vehicle controller 707 may be used to change parameters of one or more sensors such as azimuth correction, changing the output resolution and / or format type, increasing or decreasing the capture speed, adjusting the captured dynamic range, adjusting the camera focus, activating and / or deactivating the sensor, etc.

[0087] Depending on the embodiment, in addition to the video sensor 701, one or more sensors are included as additional sensors 709. In various embodiments, the additional sensors 709 may be attached to the vehicle at different positions of the vehicle and / or directed in one or more different directions. For example, the additional sensors 709 may be attached to the front, side, rear and / or roof of the vehicle, among others, and may be directed forward, backward, sideways, or in other directions. Depending on the embodiment, the additional sensors 709 include, inter alia, each of radar, audio, LiDAR, inertial, odometry, position, and / or ultrasonic sensors. Ultrasonic sensors and / or radar sensors may be used to capture details of the surroundings. For example, twelve ultrasonic sensors may be attached to the vehicle to detect both hard and soft objects. Depending on the embodiment, forward radar is utilized to capture data of the surrounding environment. In various embodiments, the radar sensor can capture details of the surroundings regardless of heavy rain, fog, dust, and other vehicles. Various sensors are used to capture the surrounding environment of the vehicle, and the captured images are provided for deep learning analysis.

[0088] Depending on the embodiment, the additional sensors 709 are not attached to the vehicle of the vehicle component 700. For example, the additional sensors 709 may be attached to a nearby vehicle and / or attached to a road or environment and are included as part of an autonomous vehicle system for capturing sensor data. Depending on the embodiment, the additional sensors 709 include one or more non-video sensors that capture the road surface on which the vehicle is traveling. Depending on the embodiment, the additional sensors 709 include position sensors such as a global positioning system (GPS) sensor for determining the position and / or change in position of the vehicle.

[0089] According to some embodiments, the safety controller 711 is a safety component used to implement safety inspections of in-vehicle components 700. According to some embodiments, the safety controller 711 receives sensor inputs from the video sensor 701 and / or the additional sensor 709. When an object is detected and / or when there is a high likelihood of a collision, the safety controller 711 can notify various components of an imminent safety issue. According to some embodiments, the safety controller 711 can interrupt and / or enhance the results of the perception module 703, the vehicle controller 707, and / or the vehicle actuator 713. According to some embodiments, the safety controller 711 is used to determine how to respond to detected safety concerns. For example, at low speeds, the vehicle can be stopped immediately, but at high speeds, the vehicle must be safely decelerated and parked in a safe location. In various embodiments, the safety controller 711 communicates with the remote interface component 751 to detect whether a live connection is established between the in-vehicle component 700 and the user of the remote interface component 751. For example, the safety controller 711 can monitor the virtual heartbeat from the remote interface component 751, and when the virtual heartbeat is no longer detected, it can trigger a safety alert to end or modify the automatic navigation. According to some embodiments, the process of FIG. 6 is at least partially implemented by the safety controller 711.

[0090] In some embodiments, the vehicle actuator 713 is used to implement specific operation control of the vehicle. For example, the vehicle actuator 713 initiates a change in the speed and / or steering of the vehicle. In some embodiments, the vehicle actuator 713 sends operation commands to the drive inverter and / or the steering rack. In various embodiments, when a potential collision is detected, the vehicle actuator 713 implements a safety check based on inputs from the additional sensor 709 and / or the safety controller 711. For example, the vehicle actuator 713 immediately stops the vehicle.

[0091] In some embodiments, the network interface 715 is a communication interface for transmitting and / or receiving data including voice data. In various embodiments, the network interface 715 connects to and conducts a voice call, transmits and / or receives text messages, transmits sensor data, receives updates to the perception module including an updated machine learning model, and interfaces with a remote server to retrieve environmental conditions including weather conditions and forecasts, traffic conditions, traffic rules and regulations, etc. For example, the network interface 715 may be used to receive commands and / or updates to operation parameters for the sensor 701, the perception module 703, the route planner module 705, the vehicle controller 707, the additional sensor 709, the safety controller 711, and / or the vehicle actuator 713. The machine learning model of the perception module 703 may be updated using the network interface 715. As another example, the network interface 715 may be used to update the firmware of the video sensor 701 and / or the operation goals of the route planner module 705 such as the weighted cost. As yet another example, the network interface 715 may be used to transmit occupancy grid data to the navigation server 761 for sharing with other vehicles.

[0092] According to some embodiments, the remote interface component 751 is one or more remote components that can be used remotely from the vehicle to automatically navigate the vehicle. For example, the remote interface component 751 includes a smartphone app executed on a smartphone device, a key fob, a GUI for controlling the vehicle such as a website, and / or another remote interface component. The user can initiate a call function to automatically navigate the vehicle to a selected destination target specified by a geographical location. For example, the user can cause the vehicle to discover the user and then track the user. As another example, the user can specify a parking position using the remote interface component 751, and the vehicle automatically navigates to the specified position or the safest reachable position closest to the specified position.

[0093] According to some embodiments, the navigation server 761 is an optional remote server that includes remote storage. The navigation server 761 can store occupancy grids and / or occupancy data that can be used later to initialize newly generated occupancy grids. According to some embodiments, the navigation server 761 is used to synchronize occupancy data between different vehicles. For example, vehicles in an area with fresh occupancy data can be started or updated with occupancy data generated from other vehicles. In various embodiments, the navigation server 761 can communicate with the in-vehicle component 700 via the network interface 715. According to some embodiments, the navigation server 761 can communicate with the remote interface component 751. According to some embodiments, one or more components or sub-components of the in-vehicle component 700 are implemented on the navigation server 761. For example, the navigation server 761 can execute processes such as perception processing and / or path planning and provide the necessary results to the in-vehicle component 700.

[0094] FIG. 8 is a diagram showing an embodiment of a user interface for automatically navigating a vehicle to a destination target. In some embodiments, the user interface of FIG. 8 is used to initiate and / or monitor the processes of FIGS. 1-6. In some embodiments, the user interface of FIG. 8 is a user interface of a smartphone application and / or is the remote interface component 751 of FIG. 7. In some embodiments, the functions associated with FIG. 8 are initiated at 203 of FIG. 2 to navigate the vehicle to the user's location. For example, a user interacting with a user interface on a smartphone device initiates a "find me" action to navigate the vehicle to the location of the user's smartphone device that accurately approximates the user's location. In the illustrated example, the user interface 800 includes a user interface component map 801, a dialog window 803, a vehicle locator element 805, a user locator element 809, and an active call area element 807.

[0095] In some embodiments, the user interface 800 displays the map 801 using the position of the vehicle indicated by the vehicle locator element 805 and the position of the user indicated by the user locator element 809. The area through which the vehicle passes when automatically navigating to the user is indicated by the active call area element 807. In various embodiments, the active call area element 807 is a circle that displays the maximum distance within which the vehicle is allowed to travel. In some embodiments, the active call area element 807 takes into account the line of sight from the user to the vehicle, and only the area including this line of sight is allowed for automatic navigation.

[0096] In the illustrated example, map 801 is a satellite map, although alternative maps may be used. In some embodiments, map 801 may be manipulated to view different locations, for example, by panning or zooming map 801. In various embodiments, map 801 includes a three-dimensional view (not shown) that allows a user to select different altitudes, such as different levels in a multi-story parking garage. For example, areas having a drivable area at different altitudes are highlighted and can be viewed in an exploded view.

[0097] In some embodiments, vehicle locator element 805 displays both the position and orientation (or direction of travel) of the vehicle. For example, the direction the vehicle is facing is indicated by the direction in which the arrow of vehicle locator element 805 points. User locator element 809 indicates the position of the user. In some embodiments, the positions of other potential passengers are also displayed, for example, in a different color (not shown). In some embodiments, the center of map 801 is at vehicle locator element 805. In various embodiments, other data, such as the original (or starting) position of the vehicle, the current position of the user, the closest position reachable from the user's current position, etc., may be used as the center of map 801.

[0098] In the illustrated example, the dialog window 803 includes a text description such as "Press and hold to start, or tap the map to select a destination" to inform the user of how to activate the calling function. In some embodiments, the default action is to navigate the vehicle to the user. The default action is activated by selecting the "Find Me" button, which is part of the dialog window 803. In some embodiments, when the "Find Me" action is enabled, the selected route is displayed on the user interface (not shown in FIG. 8). As the vehicle is navigated to the user's location, the vehicle locator element 805 is updated to reflect the new location of the vehicle. Similarly, as the user moves, the user locator element 809 is updated to reflect the new location of the user. In some embodiments, the trajectory indicates the change in the user's position. In various embodiments, the user must continue to touch the virtual heartbeat button so that automatic navigation can continue. When the user releases the virtual heartbeat button, automatic navigation stops. In some embodiments, the heartbeat button is the "Find Me" button of the dialog window 803. In some embodiments, separate forward and reverse buttons each function as a virtual heartbeat button to approve automatic navigation in the forward and reverse directions, respectively (not shown).

[0099] In some embodiments, for example, an additional "Find" function (not shown) can automatically navigate the vehicle to the location of a selected person in order to pick up a passenger other than the user. In various embodiments, pre-identified locations that can be selected as destination targets are available for display (not shown). For example, the location may be pre-identified as a valid parking location or a waiting location to pick up a passenger.

[0100] FIG. 9 is a diagram showing an embodiment of a user interface for automatically navigating a vehicle to a destination target. Depending on the embodiment, the user interface of FIG. 9 is used to initiate and / or monitor the processes of FIGS. 1-6. Depending on the embodiment, the user interface of FIG. 9 is the user interface of a smartphone application and / or the remote interface component 751 of FIG. 7. Depending on the embodiment, the functions associated with FIG. 9 are initiated at 201 of FIG. 2 and navigate the vehicle to the position specified by the user. For example, a user who interacts with the user interface on a smartphone device drops a pin to specify a destination target that is the navigation destination of the vehicle. In the illustrated example, the user interface 900 includes a user interface component map 901, a dialog window 903, a vehicle locator element 905, a user locator element 909, an active call area element 907, and a destination target element 911.

[0101] Depending on the embodiment, the user interface 900 displays the map 901 using the position of the vehicle indicated by the vehicle locator element 905 and the position of the user indicated by the user locator element 909. The area through which the vehicle passes when automatically navigating to the user is indicated by the active call area element 907. In various embodiments, the active call area element 907 is a circle that displays the maximum distance within which the vehicle is permitted to travel, and the user can only select a destination target within the active call area element 907. Depending on the embodiment, the active call area element 907 takes into account the line of sight from the user to the vehicle, and only the area including this line of sight is permitted for automatic navigation.

[0102] In the illustrated example, map 901 is a satellite map, although alternative maps may be used. In some embodiments, map 901 may be manipulated to view different locations, for example, by panning or zooming map 901. In various embodiments, map 901 includes a three-dimensional view (not shown) that allows a user to select different altitudes, such as different levels in a multi-level parking garage. For example, areas having drivable areas at different altitudes are highlighted and can be viewed in an exploded view to select a destination target.

[0103] In some embodiments, vehicle locator element 905 displays both the position and orientation (or direction of travel) of the vehicle. For example, the direction the vehicle is facing is indicated by the direction in which the arrow of vehicle locator element 905 points. User locator element 909 indicates the position of the user. In some embodiments, the positions of other potential passengers are also displayed, for example, in a different color (not shown). In some embodiments, the center of map 901 is at vehicle locator element 905. In various embodiments, other data may be used as the center of map 901, such as the original (or starting) position of the vehicle, the current position of the user, the closest position reachable from the user's current position, the selected destination target displayed by destination target element 911, among others.

[0104] In the illustrated example, the dialog window 903 includes a text description such as "Press and hold to start, or tap on the map to select a destination" to inform the user of how to activate the calling function. The user can select a target destination by selecting a position within the valid calling area element 907 on the map 901. The destination target element 911 is displayed on the selected valid position. In the illustrated example, a pin icon is used for the destination target element 911. In some embodiments, an icon having a direction, such as an arrow or a vehicle icon (not shown), is used as the destination target element 911. The icon of the destination target element 911 can be operated to select the direction of the final destination. In various embodiments, the selected direction is verified to confirm that this direction is valid. For example, on a one-way road, a destination direction against traffic may not be allowed. To obtain approval for the selected position associated with the destination target element 911, the user can select the "Clear Pin" dialog of the dialog window 903. To start automatic navigation, the user selects the "Start" button of the dialog window 903. In some embodiments, when automatic navigation is enabled, the selected route is displayed on the user interface (not shown in FIG. 9). As the vehicle is navigated to the destination target position, the vehicle locator element 905 is updated to reflect the new position of the vehicle. Similarly, as the user moves, the user locator element 909 is updated to reflect the new position of the user. In some embodiments, the trajectory indicates the change in the user's position. In various embodiments, the user must continue to touch the virtual heartbeat button so that automatic navigation can continue. When the user releases the virtual heartbeat button, automatic navigation stops. In some embodiments, the heartbeat button is the "Start" button of the dialog window 903. In some embodiments, separate forward and reverse buttons each function as a virtual heartbeat button to confirm automatic navigation in the forward and reverse directions, respectively (not shown).

[0105] The embodiments described so far have been detailed for the purpose of clarity of understanding, but the present invention is not limited to such details. There are many alternative ways to implement the present invention. The disclosed embodiments are illustrative and not restrictive.

Claims

1. A processor, receiving an identification of a geographical location associated with a target specified by a user located remotely from a vehicle, using a machine learning model to generate an occupancy grid of at least a portion of the surrounding environment of the vehicle using sensor data from one or more sensors of the vehicle, the occupancy grid of the at least a portion of the surrounding environment being formed by a plurality of grid positions associated with one or more planes corresponding to a drivable space or a non-drivable space, using the occupancy grid to calculate at least a portion of a route to a target position corresponding to the received geographical location, and providing at least one command for automatically navigating the vehicle based on the at least a portion of the route and updated sensor data from at least a portion of one or more sensors of the vehicle, the automatically navigating of the vehicle being aborted in response to a determination that a heart rate signal has not been received from the user's device, a processor configured as such; a memory coupled to the processor and configured to provide instructions to the processor, a system comprising.

2. The system according to claim 1, wherein the target specified by the user is a dynamically updated current position of the user's mobile device.

3. The system according to claim 1 or claim 2, wherein the geographical location is based on a position of a global positioning system detected by the user's mobile device.

4. The system according to claim 1, wherein the target specified by the user includes an orientation that the vehicle should face upon arrival at the target position.

5. The system according to claim 1, wherein the target is specified by the user on a map.

6. The system according to claim 1, wherein the user selects the target within a threshold distance from a detected position of a device used by the user to specify the target within a limited geographical area.

7. The system according to claim 1, wherein the user selects the target within a limited geographical area associated with the current position of the vehicle.

8. The system according to claim 1, wherein the ability of the user to specify the target is enabled based on a determination that the position of the user's device is within a threshold distance from the current position of the vehicle.

9. The system according to any one of claims 1 to 8, wherein the speed of the vehicle while automatically navigating is limited to be equal to or lower than a specified speed limit.

10. The system according to any one of claims 1 to 9, wherein the target position is dynamically updated based on the periodically updated geographical position of the user's device.

11. The system according to any one of claims 1 to 10, wherein the one or more sensors of the vehicle include a plurality of cameras.

12. The system according to any one of claims 1 to 11, wherein the occupancy grid of at least a portion of the surrounding environment is generated using auxiliary sensor data obtained using one or both of an ultrasonic sensor or a radar sensor.

13. The system according to any one of claims 1 to 12, wherein the processor is further configured to receive a specified target time associated with the target specified by the user.

14. The system according to any one of claims 1 to 13, wherein the occupancy grid of at least a portion of the surrounding environment of the vehicle includes a three-dimensional occupancy grid of a portion of the surrounding environment.

15. The system according to any one of claims 1 to 14, wherein the target position is determined including a determination as to whether the geographical position of the specified target is a valid position to be occupied by the vehicle.

16. The system according to any one of claims 1 to 15, wherein the target position is determined based on the sensor data from the one or more sensors.

17. Receiving an identification of a geographical position associated with a target specified by a user located remotely from the vehicle; Generating an occupancy grid of at least a portion of the surrounding environment of the vehicle using sensor data from one or more sensors of the vehicle by means of a neural network, wherein the occupancy grid of the at least a portion of the surrounding environment is formed by a plurality of grid positions associated with one or more planes corresponding to drivable space or non-drivable space; Calculating at least a portion of a route to a target position corresponding to the received geographical position using the occupancy grid; Providing at least one command for automatically navigating the vehicle based on the at least a portion of the route and updated sensor data from at least a portion of one or more sensors of the vehicle, wherein automatically navigating the vehicle is aborted in response to a determination that a heartbeat signal has not been received from the user's device. A method comprising the steps.

18. A computer program product, wherein the computer program product is embodied in a non-transitory computer-readable storage medium and Receiving an identification of a geographical position associated with a target specified by a user located remotely from the vehicle; Generating an occupancy grid of at least a portion of the surrounding environment of the vehicle using sensor data from one or more sensors of the vehicle by means of a neural network, wherein the occupancy grid of the at least a portion of the surrounding environment is formed by a plurality of grid positions associated with one or more planes corresponding to drivable space or non-drivable space; Calculating at least a portion of a route to a target position corresponding to the received geographical position using the occupancy grid; Providing at least one command for automatically navigating the vehicle based on at least a portion of the path and updated sensor data from at least a portion of one or more sensors of the vehicle, wherein automatically navigating the vehicle is aborted in response to a determination that no heartbeat signal has been received from the user's device, steps; A computer program product including computer instructions for.

Images