Autonomous and user-controlled vehicle call to target

Abstract

The invention relates to autonomous and user-controlled vehicle call to a target. A processor coupled to the memory is configured to receive an identification of a geographic location associated with a target specified by a user remote from the vehicle. A representation of at least a portion of an environment surrounding a vehicle is generated using a machine learning model using sensor data from one or more sensors of the vehicle. At least a portion of a path to a target location corresponding to the received geographic location is calculated using the generated representation of at least a portion of the environment around the vehicle. At least one command for automatically navigating the vehicle is provided based on the determined path and updated sensor data from at least a portion of one or more sensors of the vehicle.

Description

[0001] This application is a divisional application of the patent application for invention with the application number 202080027623.1 and the invention title "Autonomous and User-Controlled Vehicle Summon to a Target", which entered the Chinese national phase on October 9, 2021, from the international application with the international application number PCT / US2020 / 017316 and the international filing date February 7, 2020.

[0002] Cross - Reference to Related Applications

[0003] This application is a continuation application of and claims priority to U.S. Patent Application No. 16 / 272,273, entitled "AUTONOMOUS AND USER CONTROLLED VEHICLE SUMMON TO A TARGET", filed on February 11, 2019, the disclosure of which is incorporated herein by reference in its entirety. BACKGROUND OF THE INVENTION

[0004] Human drivers are typically required to operate vehicles, and the driving tasks performed by them are often complex and can be exhausting. For example, retrieving or summoning a parked car from a crowded parking lot or garage can be both inconvenient and tedious. The operation typically involves walking to one's vehicle, three-point turning, pulling out from the edge of a narrow space without touching adjacent vehicles or walls, and driving to the location from which one previously came. Although some vehicles are capable of remote operation, the routes traveled by the vehicles are typically limited to a single straight path in the forward or backward direction, with limited steering range, and cannot intelligently navigate the vehicle along its own path. SUMMARY OF THE INVENTION

[0005] One embodiment includes a system. The system includes a processor and a memory, the processor being configured to: receive an identification of a geographical location associated with a target specified by a user remote from the vehicle; use a machine learning model to generate a representation of at least a portion of the environment around the vehicle using sensor data from one or more sensors of the vehicle; use the generated representation of at least a portion of the environment around the vehicle to calculate at least a portion of a path to a target location corresponding to the received geographical location; and provide at least one command for automatically navigating the vehicle based on the determined path and at least a portion of updated sensor data from one or more sensors of the vehicle; the memory being coupled to the processor and configured to provide instructions to the processor.

[0006] Another embodiment includes a method. The method includes receiving an identification of a geographical location associated with a target specified by a user remote from the vehicle; using a neural network to generate a representation of at least a portion of the environment around the vehicle using sensor data from one or more sensors of the vehicle; using the generated representation of at least a portion of the environment around the vehicle to calculate at least a portion of a path to a target location corresponding to the received geographical location; and providing at least one command for automatically navigating the vehicle based on the determined path and at least a portion of updated sensor data from one or more sensors of the vehicle.

[0007] Yet another embodiment includes a computer program product, which is implemented on a non-transitory computer-readable storage medium and includes computer instructions. The computer instructions are for: receiving an identification of a geographical location associated with a target specified by a user remote from the vehicle; using a neural network to generate a representation of at least a portion of the environment around the vehicle using sensor data from one or more sensors of the vehicle; using the generated representation of at least a portion of the environment around the vehicle to calculate at least a portion of a path to a target location corresponding to the received geographical location; and providing at least one command for automatically navigating the vehicle based on the determined path and at least a portion of updated sensor data from one or more sensors of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Various embodiments of the present invention are disclosed in the following detailed description and the drawings.

[0009] Figure 1 is a flowchart showing an embodiment of a process for automatically navigating a vehicle to a destination target.

[0010] Figure 2 is a flowchart showing an embodiment of a process for receiving a target destination.

[0011] Figure 3 is a flowchart showing an embodiment of a process for automatically navigating a vehicle to a destination target.

[0012] Figure 4 is a flowchart showing an embodiment of a process for training and applying a machine learning model to generate a representation of the environment around the vehicle.

[0013] Figure 5 is a flowchart showing an embodiment of a process for generating an occupancy grid.

[0014] Figure 6 is a flowchart showing an embodiment of a process for automatically navigating to a destination target.

[0015] Figure 7It is a block diagram showing an embodiment of an autonomous vehicle system for automatically navigating a vehicle to a destination target.

[0016] Figure 8 It is a diagram showing an embodiment of a user interface for automatically navigating a vehicle to a destination target.

[0017] Figure 9 It is a diagram showing an embodiment of a user interface for automatically navigating a vehicle to a destination target. Detailed Description

[0018] The present invention can be implemented in many ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored on and / or provided by a memory coupled to the processor. In this specification, any other form that these implementations or the present invention may take can be referred to as a technique. Generally, within the scope of the present invention, the order of the steps of the disclosed processes can be changed. Unless otherwise stated, components such as processors or memories described as being configured to perform a task can be implemented as general components temporarily configured to perform the task at a given time or as specific components manufactured to perform the task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0019] A detailed description of one or more embodiments of the present invention and the accompanying drawings illustrating the principles of the present invention are provided below. The present invention is described in connection with these embodiments, but the present invention is not limited to any embodiment. The scope of the present invention is defined only by the claims, and the present invention includes many alternatives, modifications, and equivalents. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. These details are provided for purposes of illustration, and the present invention may be practiced without some or all of these specific details in accordance with the claims. For clarity, technical materials known in the technical field related to the present invention are not described in detail so as not to unnecessarily obscure the present invention.

[0020] A technology for autonomously summoning a vehicle to a destination is disclosed. A target geographical location is provided and the vehicle automatically navigates to the target location. For example, a user provides the location by placing a pin at the destination location on a graphical map user interface. As another example, the user summons the vehicle to the user's location by designating the user's location as the destination location. The user can also select the destination location based on a feasible path detected for the vehicle. The destination location can be updated (e.g., if the user moves around) to direct the vehicle to update its path to the destination location. Using the specified destination location, the vehicle navigates by leveraging sensor data (such as visual data captured by a camera) to generate a representation of the environment around the vehicle. In some embodiments, the representation is an occupancy grid that details drivable and non-drivable spaces. In some embodiments, the occupancy grid is generated from camera sensor data using a neural network. The representation of the environment can be further enhanced with auxiliary data, such as additional sensor data (including radar data), map data, or other inputs. Using the generated representation, a path is planned from the vehicle's current location to the destination location. In some embodiments, the path is generated based on the vehicle's operating parameters (such as the turning radius of the model, vehicle width, vehicle length, etc.). An optimal path is selected based on selection parameters (such as distance, speed, number of gear shifts, etc.). When the vehicle autonomously navigates using the selected path, the representation of the environment is continuously updated. For example, when the vehicle travels to the destination, new sensor data is captured and the occupancy grid is updated. Safety checks are continuously performed, and the safety checks can override or modify the autonomous navigation. For example, ultrasonic sensors can be used to detect potential collisions. The user can monitor the navigation at any time and cancel the summoned vehicle. In some embodiments, the vehicle must continuously receive a virtual heartbeat signal from the user for the vehicle to continue navigating. The virtual heartbeat can be used to indicate that the user is actively monitoring the vehicle's progress. In some embodiments, the user selects the path and remotely controls the vehicle. For example, the user can control the steering angle, direction, and / or speed of the vehicle to remotely control the vehicle. When the user remotely controls the vehicle, safety checks are continuously performed, and the safety checks can override and / or modify the user control. For example, if an object is detected, the vehicle can be stopped.

[0021] In some embodiments, a system includes a processor configured to receive an identification of a geographical location associated with a destination specified by a user remote from the vehicle. For example, a user waiting at a pick-up location specifies her or his geographical location as a destination. The destination is received by a vehicle parked in a remote parking space. In some embodiments, a machine learning model is used to generate a representation of at least a portion of the environment around the vehicle using sensor data from one or more sensors of the vehicle. For example, sensors such as cameras, radar, ultrasonic, or other sensors capture data of the environment around the vehicle. The data is fed as input to a neural network using a trained machine learning model to generate a representation of the environment around the vehicle. As an example, the representation can be an occupancy grid that describes the drivable space through which the vehicle can navigate. In some embodiments, at least a portion of a path to a target location corresponding to the received geographical location is calculated using the generated representation of at least a portion of the environment around the vehicle. For example, an occupancy grid representing the environment around the vehicle is used to select a path for navigating the vehicle to the target destination. The value at an entry in the occupancy grid can be a probability and / or cost associated with driving through the location associated with the grid entry. In some embodiments, at least one command for automatically navigating the vehicle is provided based on the determined path and updated sensor data from at least a portion of one or more sensors of the vehicle. For example, a vehicle controller uses the selected plan to provide actuator commands for controlling the speed and steering of the vehicle. These commands are used to navigate the vehicle along the selected path. As the vehicle navigates, updated sensor data is captured to update the representation of the environment around the vehicle. In various embodiments, the system includes a memory coupled to the processor and configured to provide instructions to the processor.

[0022] Figure 1 is a flowchart showing an embodiment of a process for automatically navigating a vehicle to a destination. In some embodiments, Figure 1The process is used to summon the vehicle to a geographical location specified by the user. The user can use a mobile application, a remote key fob, the vehicle's GUI, etc. to specify the target location. Using the target location, the vehicle automatically navigates from its starting position to the target location. In some embodiments, the target location is the user's location and the target location is dynamic. For example, as the user moves, the vehicle's target destination moves to the user's new location. In some embodiments, the target location is specified indirectly, such as via a calendar or planning software. For example, the user's calendar is parsed and used to determine the target destination and time from calendar events. Calendar events can include the location and time of the event, such as the end time. The destination is selected based on the location of the event, and the time is selected based on the end time of the event. The vehicle automatically navigates to arrive at that location at the end time (such as at the end of a dinner, wedding, restaurant reservation, etc.). In some embodiments, the vehicle uses a specified time (such as arrival or departure time) to navigate to the target destination. For example, the user can specify the time when the vehicle should start automatically navigating or leave a specified destination. As another example, the user can specify the time when the vehicle should arrive at a specified destination. The vehicle will depart before the specified time in order to arrive at the destination at the specified time. In some embodiments, the arrival time is configured with a threshold time to allow for an estimate of the difference between the travel time and the actual travel time. In various embodiments, Figure 1 the process runs on an autonomous vehicle. In some embodiments, a remote server communicating with the autonomous vehicle performs part of the summon function. In some embodiments, Figure 1 the process is implemented at least in part using Figure 7 the autonomous vehicle system.

[0023] At 101, a destination is received. For example, the user selects the "Find Me" feature from a mobile application on a smart phone device. The "Find Me" feature determines the user's location and transmits the user's location to the vehicle summon module. In various embodiments, the vehicle summon module is implemented across one or more modules, and one or more modules can be present on the vehicle and / or away from the vehicle. In some embodiments, the target destination may not be the user's location. For example, the user can select a location on a map as the target destination. As another example, the location can be based on the location associated with a calendar event. In some embodiments, the location is selected by placing a fixed icon at the target destination on the map. The location can be longitude and latitude. In some embodiments, the location can include altitude, elevation, or a similar measure. For example, in a parking lot, the location includes a height component for differentiating different floors of a multi-level parking lot. In various embodiments, the destination is a geographical location.

[0024] In some embodiments, the destination includes a location and an orientation. For example, a car icon, a triangle icon, or other icons with an orientation are used to specify the location and orientation. By specifying the orientation, the vehicle navigates to the selected target destination and faces the expected orientation selected by the user. In some embodiments, the orientation is determined by orientation suggestions, such as suggestions based on the environment, the user, other vehicles, and / or other suitable inferences. For example, a map can be used to determine the correct orientation of the vehicle, such as the direction in which the vehicle is traveling on a one-way road. As another example, based on lane lines, traffic signals, other vehicles, etc., the summon function determines the appropriate orientation of the road. For example, the direction in which other vehicles are facing can be used as an orientation suggestion. In some embodiments, the orientation is based on the orientation suggested by other autonomous vehicles. For example, in some embodiments, the vehicle summon function can communicate with and / or query other vehicles to determine their orientations and use the provided results as orientation suggestions. In various embodiments, one or more orientation suggestions are weighted and used to determine the target destination orientation.

[0025] In some embodiments, the destination is determined by completing a query. For example, a user can request (e.g., using a GUI, voice, or other query type) for her or his vehicle to arrive at a specified parking lot at a certain time. The destination is determined by querying a search engine (such as a map search database) for the destination parking lot. The user's calendar and / or address book can be used to refine the search results. Once the parking lot is identified, the instructions and / or traffic flow for entering and / or leaving the parking lot are determined and used to select the destination. Then, the destination is provided to the vehicle summon module.

[0026] In some embodiments, the received destination is a multi-part destination. For example, the destination requires arriving at one or more waypoints before reaching the final destination. Waypoints can be used to exert additional control over the path taken by the vehicle when navigating to the final destination. For example, waypoints can be used to follow the preferred traffic patterns for an airport, a parking lot, or other destinations. Waypoints can be used to pick up one or more additional passengers, etc. For example, delays and / or pauses can be incorporated into the destination to pick up and / or drop off passengers or objects.

[0027] In some embodiments, the received destination is first subject to one or more verification and / or security checks. For example, the destination can be restricted based on distance such that a user can only select a destination that is within a certain distance (or radius), such as 100 meters, 10 meters, or another suitable distance, from the vehicle. In some embodiments, the distance is based on local regulations and / or statutes. In some embodiments, the destination location must be a valid parking location. For example, sidewalks, crosswalks, intersections, lakes, etc. are generally not valid parking locations, and the user can be prompted to select a valid location. In some embodiments, the received destination is modified from the initially selected destination of the user to account for safety issues, such as enforcing a valid parking location.

[0028] At 103, a path to the destination is determined. For example, one or more paths are determined for navigating the vehicle from its current location to the destination received at 101. In some embodiments, the path is determined using a path planning module (such as Figure 7 the path planning module 705). In various embodiments, the path planning module can be implemented using a cost function, where appropriate weights are applied to different paths to the destination. In some embodiments, the selected path is based on potential path arcs originating from the vehicle's location. The potential path arcs are limited by the vehicle's operating dynamics, such as the vehicle's turning range. In various embodiments, each potential path has a cost. For example, a potential path with a sharp turn has a higher cost than a smoother path. As another example, a potential path with a large speed change has a higher cost than a path with a smaller speed change. As another example, a potential path with more gear changes (e.g., a change from reverse to forward) has a higher cost than a path with fewer gear changes. In some embodiments, paths with a higher likelihood of encountering certain objects are weighted differently from paths with a lower likelihood of encountering objects. For example, paths are weighted based on the likelihood of encountering pedestrians, vehicles, animals, traffic, insufficient lighting, adverse weather, tolls, etc. In various embodiments, when determining the path to take to reach the destination, a high-cost path is not preferred over a low-cost path.

[0029] At 105, the vehicle navigates to a destination. Using the path to the destination determined at 103, the vehicle automatically navigates to the destination received at 101. In some embodiments, the path consists of multiple smaller sub - paths. Different sub - paths can be achieved by driving in different gears (such as forward or reverse). In some embodiments, the path includes actions such as opening a garage door, closing a garage door, passing through a parking gate, waiting for an elevator, confirming toll payment, charging, making a call, sending a message, etc. Additional actions may require stopping the vehicle and / or performing operations to manipulate the surrounding environment. In some cases, the final destination is a destination close to the destination received at 101. For example, in some cases, the destination received at 101 is inaccessible, so the final destination is very close to reaching that destination. For example, in the case where the user selects a sidewalk, the final destination is a location on the road adjacent to the sidewalk. As another example, in the case where the user selects a crosswalk, the final destination is a location on the road near but not in the crosswalk.

[0030] In some embodiments, one or more safety checks are continuously confirmed during navigation. For example, auxiliary data (such as sensor data from ultrasonic or other sensors) is used to identify obstacles such as pedestrians, vehicles, speed bumps, traffic control signals, etc. The likelihood of a potential collision with an object terminates the current navigation and / or modifies the path to the destination. In some embodiments, the path to the destination includes the speed of travel along the path. As another example, in some embodiments, a heartbeat must be received from the user's mobile application to continue navigation. The heartbeat can be achieved by requiring the user to maintain a connection with the summoned vehicle, for example, by continuously pressing a button GUI element in the mobile summon application. As another example, the user must continue to hold down the button on the remote key (or maintain contact with the sensor) for the vehicle to automatically navigate. If the connection is lost, the automatic navigation will terminate. In some embodiments, the termination of navigation safely decelerates the vehicle and places the vehicle in a safe stationary mode, such as parked at the roadside, a nearby parking space, etc. In some embodiments, the loss of connection can immediately stop the vehicle depending on the situation. For example, the user can release the summon button on the mobile application to indicate that the vehicle should stop immediately, similar to an emergency brake switch. In various embodiments, the deceleration used to stop the vehicle depends on the vehicle's environment (e.g., whether there are other vehicles driving nearby, such as behind the vehicle), the presence of a potential collision (such as an obstacle or pedestrian in front of the vehicle's path), the speed at which the vehicle is traveling, or other appropriate parameters. In some embodiments, the speed at which the vehicle is traveling is limited to, for example, a lower maximum speed.

[0031] At 107, the vehicle summon function is completed. Upon completion, one or more completion actions can be performed. For example, sending a notification that the vehicle has arrived at the selected target destination. In some embodiments, the vehicle arrives at a destination near the target destination and the user is presented with a notification of the vehicle's location. In some embodiments, the notification is sent to a mobile application, a key card (e.g., indicated by a status change associated with the key card), via a text message and / or via another suitable notification channel. Additional directions can be provided to guide the user to the vehicle. In various embodiments, the vehicle can be placed in a parking lot and one or more vehicle settings can be triggered. For example, the interior lighting of the vehicle can be turned on. Floor lighting can be activated to increase visibility when entering the vehicle. One or more exterior lights can be turned on, such as flashers, parking lights, and / or hazard lights. Exterior lights such as headlights can be activated to increase visibility for passengers approaching the vehicle. Actuated directional lights can be pointed in the expected direction from which the passengers will arrive to reach the vehicle. In some embodiments, an audio notification such as an audio alert or music is played. Welcome music or a similar audio can be played in the cabin based on the user's preference. Similarly, the climate control of the vehicle can be enabled to prepare the climate of the cabin for the passengers, such as heating or cooling the cabin to a desired temperature and / or humidity. The seats can be heated (or ventilated). The heated steering wheel can be activated. The vents can be reoriented based on preference. The doors can be unlocked and opened for the passengers. If the destination is a charging station, the vehicle can be oriented so that its charger port is aligned with the charger. In some embodiments, the user can configure the preferences of the vehicle according to the expectations of the passengers, including cabin climate, interior lighting, exterior lighting, audio system, and other vehicle preferences.

[0032] In various embodiments, during and after the summon is completed, the location of the vehicle is updated. For example, the updated location can be reflected in the accompanying mobile application. In some embodiments, the completion action includes updating an annotated map of the traveled path and encountered environment with recently captured and analyzed sensor data. The annotated map can be updated to reflect potential obstacles, traffic signals, parking preferences, traffic patterns, pedestrian walking patterns, and / or other suitable path planning metadata that may be useful for future navigation and / or path planning. For example, speed bumps, crosswalks, potholes, empty parking spaces, charging locations, gas stations, etc. encountered are updated to the annotated map. In some embodiments, the data corresponding to the encountered objects is used as potential training data to improve the summon function of the current vehicle / user as well as other vehicles and users, such as perception, path planning, safety verification, and other functions. For example, an empty parking space can be used for path planning of other vehicles. As another example, a charging station or a gas station is used for path planning. The vehicle can be routed to the charging station and oriented so that its charger port is aligned with the charger.

[0033] Figure 2 is a flowchart showing an embodiment of a process for receiving a target destination. In some embodiments, the target destination is selected and / or provided by a user. The destination target is associated with a geographical location and serves as a target for an autonomous navigation vehicle. In the illustrated example, the process of receiving the target destination can be initiated from more than one starting point. Figure 2 Two starting points are two examples of initiating destination reception. Other methods are possible. In some embodiments, Figure 2 the process of Figure 1 is executed at 101 of

[0034] At 201, the destination location is received. In some embodiments, the destination is provided as a geographical location. For example, longitude and latitude values are provided. In some embodiments, altitude is also provided. For example, an altitude associated with a particular floor of a parking lot is provided. In various embodiments, the destination location is provided by the user via a smart phone device, via the vehicle's media console, or by other means. In some embodiments, the location is received along with the time associated with leaving or arriving at the destination. In some embodiments, one or more destinations are received. For example, in some scenarios, a multi-step destination including more than one stop is received. In some embodiments, the destination location includes an orientation or heading. For example, the heading indicates the direction the vehicle should face when arriving at the destination.

[0035] At 203, the user location is determined. For example, the user's location is determined using the Global Positioning System or another location-aware technology. In some embodiments, the user's location is approximated by a key card, the user's smart phone device, or the location of another device in the user's control. In some embodiments, the user's location is received as a geographical location. Similar to 201, the location can be a latitude and longitude pair and can include altitude in some embodiments. In some embodiments, the user's location is dynamic and is continuously updated or updated at specific intervals. For example, the user can move to a new location and the received location is updated. In some embodiments, the destination location includes an orientation or heading. For example, the heading indicates the direction the vehicle should face when arriving at the destination.

[0036] At 205, confirm the destination. For example, verify the destination to confirm that the destination is reachable. In some embodiments, the destination must be within a certain distance of the vehicle's origin location. For example, certain regulations may require the vehicle to automatically navigate no more than 50 meters. In various embodiments, an invalid destination may require the user to provide a new location. In some embodiments, suggest alternative destinations to the user. For example, if the user selects a wrong orientation, provide a suggested orientation. As another example, if the user selects a no-parking area, suggest a valid parking area, such as the nearest valid parking area. Once verified, the selected destination is provided to step 207.

[0037] At 207, provide the destination to the path planning module. In some embodiments, the path planning module is used to determine the vehicle's route. At 207, provide the verified destination to the path planning module. In some embodiments, the destination is provided as one or more locations, for example, a destination with multiple stops. The destination can include two-dimensional locations, such as latitude and longitude locations. Alternatively, in some embodiments, the destination includes altitude. For example, certain drivable areas (such as multi-level parking lots, bridges, etc.) have multiple drivable planes for the same two-dimensional location. The destination can also include a heading for specifying the orientation that the vehicle should face when it reaches the destination. In some embodiments, the path planning module is Figure 7 the path planning module 705.

[0038] Figure 3 is a flowchart showing an embodiment of a process for automatically navigating a vehicle to a destination target. For example, using Figure 3 the process, generate a representation of the environment around the vehicle and use it to determine one or more paths to the destination. The vehicle automatically navigates using the determined path(s). As additional sensor data is updated, the representation of the environment around is also updated. In some embodiments, Figure 3 the process uses Figure 7 the autonomous vehicle system to execute. In some embodiments, the step of 301 is executed at Figure 1 101 of Figure 1 the step of 303, 305, 307, and / or 309 is executed at Figure 1 103 of Figure 1 the step of 311 and / or 313 is executed at

[0039] At 301, receive a destination. For example, receive a geographical location via a mobile application, a remote key, through a vehicle's control center, or other suitable device. In some embodiments, the destination is a location and orientation. In some embodiments, the destination includes altitude and / or time. In various embodiments, the destination is received using Figure 2 a process. In some embodiments, the destination is dynamic and new destinations can be received appropriately. For example, in the case where a user selects a "find me" feature, the destination is updated to follow the user's location. In essence, the vehicle can follow the user like a pet.

[0040] At 303, receive visual data. For example, receive camera image data using one or more camera sensors fixed to the vehicle. In some embodiments, the image data is received from sensors that cover the environment around the vehicle. The visual data can be pre-processed to improve the usefulness of the data for analysis. For example, one or more filters can be applied to reduce the noise of the visual data. In various embodiments, visual data is continuously captured to update the environment around the vehicle.

[0041] At 305, determine a drivable space. In some embodiments, a neural network is used to determine the drivable space by applying inference to the visual data received at 303. For example, a convolutional neural network (CNN) is applied using the visual data to determine the drivable and non-drivable spaces for the environment around the vehicle. The drivable space includes the area where the vehicle can travel. In various embodiments, the drivable space is free of obstacles such that the vehicle can use a path through the determined drivable space to travel. In various embodiments, a machine learning model is trained to determine the drivable space, and the machine learning model is deployed on the vehicle to automatically analyze and determine the drivable space from the image data.

[0042] In some embodiments, the visual data is supplemented with additional data, such as additional sensor data. The additional sensor data can include ultrasonic, radar, lidar, audio, or other suitable sensor data. The additional data can also include annotation data, such as map data. For example, an annotated map can annotate lanes, speed lines, intersections, and / or other driving metadata. The additional data can be used as an input to the machine learning model or consumed downstream when creating an occupancy grid to improve the results of determining the drivable space.

[0043] At 307, an occupancy grid is generated. Using the drivable space determined at 305, an occupancy grid representing the environment of the vehicle is generated. In some embodiments, the occupancy grid is a two-dimensional occupancy grid representing the entire plane in which the vehicle is located (e.g., 360 degrees along the longitude and latitude axes). In some embodiments, the occupancy grid includes a third dimension to account for height. For example, an area with multiple drivable paths at different heights (such as a multi-level parking structure, overpass, etc.) can be represented with a three-dimensional occupancy grid.

[0044] In various embodiments, the occupancy grid contains a drivability value at each grid position corresponding to a location in the surrounding environment. The drivability value for each location can be the probability that the location is drivable. For example, a sidewalk can be designed to have a zero drivability value, while gravel has a drivability value of 0.5. The drivability value can be a normalized probability ranging from 0 to 1 and is based on the drivable space determined at 305. In some embodiments, each position in the grid includes a cost metric associated with the cost (or penalty / reward) of traversing that position. The cost value for each grid position in the occupancy grid is based on the drivability value. The cost value can also depend on additional data such as preference data. For example, path preferences can be configured to avoid toll roads, carpool lanes, school zones, etc. In various embodiments, path preference data can be learned via a machine learning model and determined as part of the drivable space at 305. In some embodiments, the path preference data is configured by the user and / or operator to optimize the path taken to navigate to the destination received at 301. For example, path preferences can be optimized to improve objectives such as safety, convenience, travel time, and / or comfort. In various embodiments, the preference is an additional weight used to determine the cost value of each location grid.

[0045] In some embodiments, the occupancy grid is updated using auxiliary data such as additional sensor data. For example, ultrasonic sensor data capturing nearby objects is used to update the occupancy grid. Other sensor data such as lidar, radar, audio, etc. can also be used. In various embodiments, an annotated map can be used in part to generate the occupancy grid. For example, roads and their attributes (speed limits, lanes, etc.) can be used to enhance visual data to generate the occupancy grid. As another example, occupancy data from other vehicles can be used to update the occupancy grid. For example, adjacent vehicles with similar equipped functionality can share sensor data and / or occupancy grid results.

[0046] In some embodiments, the last generated occupancy grid is used to initialize the occupancy data. For example, when the vehicle no longer captures new data and / or is parked or powered off, the last occupancy grid is saved. When an occupancy grid is needed, e.g., when the vehicle is summoned, the last generated occupancy grid is loaded and used as the initial occupancy grid. This optimization significantly improves the accuracy of the initial grid. For example, some objects may be difficult to detect when stationary, but as the vehicle approaches its current parked position and moves, these objects will still be detected in the last saved occupancy grid.

[0047] At 309, a path target is determined. Using the occupancy grid, a search is performed to determine a path for navigating the vehicle from its current position to the destination received at 301. The potential path is based in part on the vehicle's operating characteristics, such as turning radius, vehicle width, vehicle length, etc. Each vehicle model can be configured with specific vehicle operating characteristics. In some embodiments, the path finding is configured to implement configurable constraints and / or goals. Example constraints include that the vehicle cannot drive sideways, the vehicle should limit the number of sharp turns, the vehicle should limit the number of gear shifts (e.g., from reverse to forward or vice versa), etc. The constraints / goals can be implemented as weighted costs in a cost function. In various embodiments, the vehicle's initial position includes x, y, and heading values. The x and y values can correspond to longitude and latitude values. One or more potential paths starting from the initial position towards reaching the target are determined. In some embodiments, the path consists of one or more path primitives (such as arc primitives). The path primitives describe the paths (and path targets) along which the vehicle can navigate to reach the destination.

[0048] In various embodiments, the selected path target is selected based on a cost function. The cost function is executed on each of the potential paths in the potential path set. Each potential path traverses a set of cells of the occupancy grid, where each cell has a cost value to reward or penalize travel through the cell location. The path with the optimal cost value is selected as the path target. The path target can include one or more path primitives, such as arcs, to model the movement of the navigating vehicle. For example, a pair of two path primitives can represent moving the vehicle backward and then forward. The reverse path is represented as one arc, and the forward path is represented as another arc. As another example, the vehicle path can include a three-point turn. Each primitive of the turn can be represented as an arc path. At the end of each path primitive, the vehicle will have new x, y, and heading values. In some embodiments, the vehicle includes, for example, a height value to support navigation between different floors of a parking lot. Although arc path primitives are used to define the target path, other suitable geometric primitives can also be used. In some embodiments, each path includes a speed parameter. For example, the speed parameter can be used to suggest the travel speed along the path. Initial speed, maximum speed, acceleration, maximum acceleration, and other speed parameters that can be used to control how the vehicle navigates along the path. In some embodiments, the maximum speed is set to a low speed and is used to prevent the vehicle from traveling too fast. For example, a lower maximum speed can be enforced to allow for quick user intervention.

[0049] In various embodiments, the determined path target is passed to the vehicle controller to navigate along the path. In some embodiments, the path target is first converted into a set of path points along the path for use by the vehicle controller. In some embodiments, path planning runs continuously and can determine a new path while traversing the current path. In some embodiments, the expected path is no longer reachable, e.g., the path is blocked and a new path is determined. The path planning at 309 can run at a lower frequency than Figure 3 other functions of the process. For example, the path planning can run at a lower frequency than the frequency at which the drivable space is determined at 305 and / or the vehicle is controlled to navigate at 311. By determining the path target at a lower frequency than the frequency of determining the occupancy grid, the path planning process is more efficient and utilizes a more accurate representation of the world.

[0050] In some embodiments, more than one route to a destination is viable, and multiple path goals are provided to the user. For example, the user is presented with two or more path goals as options. For example, the user can select a path goal for navigating the vehicle to the destination using a GUI, voice commands, etc. For example, the user is presented with two paths from the vehicle to the destination. The first path is estimated to take less time, but has more turns and requires frequent gear shifts. The second path is smoother, but takes more time. The user can select a path goal from these two options. In some embodiments, the user can select a path goal and modify the route. For example, the user can adjust the selected route to bypass a specific obstacle, such as a busy intersection.

[0051] At 311, the vehicle automatically navigates to the path goal. Using one or more path goals determined at 309, the vehicle automatically navigates from its current position along the (one or more) target path to reach the destination. For example, the vehicle controller receives the (one or more) target path and then implements the vehicle control required to navigate the vehicle along the path. The (one or more) path goals can be received as a set of path points along the path to be navigated. In some embodiments, the vehicle controller converts path primitives, such as arcs, into path points. In various embodiments, the vehicle is controlled by sending actuator parameters from the vehicle controller to the vehicle actuators. In some embodiments, the vehicle controller is Figure 7 the vehicle controller 707 and the vehicle actuators are Figure 7 the vehicle actuators 713. Using the vehicle actuators, steering, braking, accelerating, and / or other operating functions are actuated.

[0052] In some embodiments, when the vehicle is navigating to a path goal, the user can adjust the navigation / operation of the vehicle. For example, the user can adjust the navigation by providing an input such as "turn more to the left". As an additional example, the user can increase or decrease the speed of the vehicle while navigating and / or adjusting the steering angle.

[0053] At 313, it is determined whether the vehicle has reached the destination received at 301. If the vehicle has reached the destination, the process proceeds to 315. If the vehicle has not reached the destination, the process loops back to 301 to potentially receive an updated destination and automatically navigate to the (potentially updated) selected destination. In various embodiments, the vehicle has reached the destination, but may not be at the exact location of the destination. For example, the selected destination may not be or may no longer be a drivable location. As one example, another vehicle may be parked at the selected destination. As another example, the user may be moved to a non-drivable location such as a passenger waiting area. In some cases, if the vehicle has reached the location determined to be the closest reachable destination, the vehicle is determined to have reached the destination. The closest reachable destination may be based on the path determined at 309. In some embodiments, the closest reachable destination is based on a cost function used to calculate the potential path between the current location and the destination. In some embodiments, the difference between the arrival location and the received destination is based on the accuracy of the technology available for determining the location. For example, the vehicle may park within the accuracy range of the available global positioning system.

[0054] At 315, the summon is complete. In various embodiments, the summon function is completed and one or more completion actions are performed. For example, the completion actions described in 107 with respect to Figure 1 are performed. In some embodiments, the occupancy grid generated at 307 is saved and / or exported as a completion action. The occupancy grid may be saved locally on the vehicle and / or saved to a remote server. After being saved, the grid may be used by the vehicle and / or shared with other vehicles having potentially overlapping paths.

[0055] Figure 4 is a flowchart showing an embodiment of a process for training and applying a machine learning model to generate a representation of the environment around a vehicle. In some embodiments, Figure 4 the process is used to determine the drivable space for generating an occupancy grid using at least in part sensor data. The sensor data used for training and / or applying the trained machine learning model may correspond to image data captured from the vehicle using a camera sensor. In some embodiments, the process is used to create and deploy a machine learning model for Figure 7 the autonomous vehicle system. In some embodiments, Figure 4 the process is used to determine the drivable space at 305 of Figure 3 and generate an occupancy grid at 307 of Figure 3 .

[0056] At 401, prepare training data. In some embodiments, a training data set is created using sensor data that includes image data. The sensor data can include still images and / or videos from one or more cameras. Additional sensors such as radar, lidar, ultrasonic, etc. can be used to provide relevant sensor data. In various embodiments, the sensor data is paired with corresponding vehicle data to help identify features of the sensor data. For example, the position and changes in position data can be used to identify the positions of relevant features (such as lane lines, traffic control signals, objects, etc.) in the sensor data. In some embodiments, the training data is prepared to train a machine learning model to identify drivable space. The prepared training data can include data for training, validation, and testing. In some embodiments, the format of the data is compatible with the machine learning model used in the deployed deep learning application.

[0057] At 403, train the machine learning model. For example, the machine learning model is trained using the data prepared at 401. In some embodiments, the model is a neural network, such as a convolutional neural network (CNN). In various embodiments, the model includes multiple intermediate layers. In some embodiments, the neural network can include multiple layers, and the multiple layers include multiple convolutional layers and pooling layers. In some embodiments, a validation data set created from the received sensor data is used to validate the trained model. In some embodiments, the machine learning model is trained to predict drivable space based on image data. For example, the drivable space of the environment around the vehicle can be inferred from the images captured by the camera. In some embodiments, the image data is enhanced with other sensor data such as radar or ultrasonic sensor data to improve accuracy.

[0058] At 405, deploy the trained machine learning model. For example, the trained machine learning model is installed on the vehicle as an update to the deep learning network. In some embodiments, the deep learning network is part of a perception module such as Figure 7 the perception module 703. The trained machine learning model can be installed as an over-the-air update. In some embodiments, the update is a firmware update transmitted using a wireless network such as WiFi or cellular network. In some cases, the newly trained machine learning model is installed during vehicle maintenance.

[0059] At 407, receive sensor data. For example, the sensor data is captured from one or more sensors of the vehicle. In some embodiments, the sensor is a vision sensor, such as Figure 7 the vision sensor 701 for capturing visual data, and / or Figure 7Additional sensor 709. The vision sensor may include an image sensor, such as a camera mounted behind the windshield, a front and / or side camera mounted on a pillar, a rear camera, etc. In various embodiments, the sensor data is in a format that serves as an input to the machine learning model trained at 403 or is converted to that format. For example, the sensor data can be raw or processed image data. In some embodiments, the sensor data is data captured from ultrasonic sensors, radar, LiDAR sensors, microphones, or other suitable technologies. In some embodiments, the sensor data is preprocessed using an image preprocessor such as an image preprocessor during a preprocessing step. For example, the image can be normalized to remove distortion, noise, etc.

[0060] At 409, the trained machine learning model is applied. For example, the machine learning model trained at 403 is applied to the sensor data received at 407. In some embodiments, the application of the model is performed by a perception module such as Figure 7 perception module 703 using a deep learning network. In various embodiments, by applying the trained machine learning model, the drivable space can be identified and / or predicted. For example, the drivable space in the environment around the vehicle can be inferred. In various embodiments, the vehicle, obstacles, lanes, traffic control signals, map features, object distances, speed limits, etc. are identified by applying the machine learning model. The detected features can be used to determine the drivable space. In some embodiments, traffic control and other driving features are used to determine navigation parameters, such as speed limits, parking positions, parking areas, facing orientations, etc. For example, stop signs, parking spaces, lane lines, and other traffic control features are detected and used to determine the drivable space and driving parameters.

[0061] At 411, an occupancy grid is generated. For example, using the output of the trained machine learning model applied at 409, an occupancy grid for path planning is generated to determine a target path for navigating the vehicle. The occupancy grid can be generated as described for Figure 3 307 and / or using the Figure 5 process.

[0062] Figure 5 is a flowchart showing an embodiment of the process for generating an occupancy grid. For example, using the drivable space determined from the sensor data, an occupancy grid for path planning can be generated. The occupancy grid can be augmented using additional sensor data, metadata from additional sources (such as an annotated map), and / or a previously generated occupancy grid. In some embodiments, Figure 5 the process is at Figure 1 103 of Figure 3 307 of Figure 4 411 ofFigure 5 The process uses Figure 7 an autonomous vehicle system to execute.

[0063] In some embodiments, an occupancy grid is generated before route planning. For example, an occupancy grid is generated and then presented to the user for inspection, such as via a GUI on a smart phone device or via a display in the vehicle. The user can view the occupancy grid and select a target destination. The user can specify which part of the roadside the user wishes the vehicle to park at. Once selected, the vehicle can navigate to the target destination.

[0064] At 501, the saved occupancy grid is loaded. In some embodiments, a previously generated occupancy grid corresponding to the current position of the vehicle is loaded. By initializing the occupancy grid using the previously saved occupancy grid, the accuracy of the environment around the vehicle can be improved. This optimization significantly improves the accuracy of the initial grid. For example, some objects may be difficult to detect when stationary, but as the vehicle approaches the current parking position and moves, these objects will still be detected in the last saved occupancy grid.

[0065] At 503, the drivable space is received. For example, the drivable space is received as the output of a neural network (such as a convolutional neural network). In some embodiments, the drivable space is continuously updated as new sensor data is captured and analyzed. The received drivable space can be segmented into grid positions. In some embodiments, in addition to the drivable space, other vision-based measurements are received. For example, objects such as curbs, vehicles, pedestrians, cones, etc. are detected and received.

[0066] At 505, auxiliary data is received. For example, the auxiliary data is used to update and further refine the accuracy of the occupancy grid. The auxiliary data can include data from sensors such as ultrasonic sensors or radars. The auxiliary data can also include occupancy data generated from other vehicles. For example, the mesh network of the vehicle can share occupancy data based on the time and location of the occupancy data. As another example, annotated map data can be used to augment the occupancy grid. Map data such as speed limits, lanes, drivable space, traffic patterns, etc. can be loaded via the annotated map data.

[0067] In some embodiments, safety data for overriding or modifying navigation is received as auxiliary data. For example, a collision warning system inputs data at 505 to override grid values to account for potential or pending collisions. In some embodiments, the auxiliary data includes data provided from the user. For example, the user can include an image or video to assist in identifying the destination, such as a parking position and / or orientation. The received data can be used to modify the occupancy grid.

[0068] At 507, the occupancy grid is updated. The data received at 503 and / or 505 is used to update the occupancy grid. The updated grid may include values at each grid location corresponding to probability values of the grid locations. These values may be cost values associated with navigating through the grid location. In some embodiments, the values also include drivable values corresponding to the probability that the grid location is a drivable area. In various embodiments, the updated occupancy grid may be saved and / or exported. For example, the grid data may be uploaded to a remote server or saved locally. For example, when the vehicle is parked, the grid data may be saved and later used to initialize the grid. As another example, the grid data may be shared with related vehicles (such as vehicles having paths or potential paths that intersect).

[0069] Once the occupancy grid is updated, the process loops back to 503 to continuously update the occupancy grid with newly received data. For example, when the vehicle is navigating along a path, new drivable and / or assist data is received to update the occupancy grid. The newly updated grid may be used to refine and / or update the target path for autonomous navigation. For example, previously empty spaces may now be blocked. Similarly, previously blocked spaces may be open.

[0070] Figure 6 is a flowchart showing an embodiment of a process for autonomous navigation to a destination target. Figure 6 The process can be used to navigate a vehicle from a current location of the vehicle to a destination target location using the determined planning goal. In various embodiments, the navigation is automatically performed by a vehicle controller using vehicle actuators to modify the steering and speed of the vehicle. Figure 6 The process implements multiple safety checks to enhance the safety of vehicle navigation. The safety checks allow termination and / or modification of the navigation. For example, a virtual heartbeat may be implemented, which requires the user to continuously maintain contact with the vehicle to confirm that the user is monitoring the progress of the vehicle. In some embodiments, the navigation utilizes path goals to navigate the vehicle along an optimal path to the destination target. The path goals may be received as path primitives (such as arcs), a set of points along the selected path, and / or another form of path primitive. In some embodiments, Figure 6 The process is at Figure 1 105 of and / or Figure 3 at 311 of. In some embodiments, the process uses Figure 7 the autonomous vehicle system of.

[0071] In some embodiments, Figure 6The process can be used by a user to remotely control a vehicle. For example, the user can control the steering angle, direction, and / or speed of the vehicle via vehicle adjustments to remotely control the vehicle. When the user remotely controls the vehicle, a safety check is continuously performed, and the safety check can override and / or modify the user control. For example, if an object is detected or the connection to the remote user is lost, the vehicle can be stopped.

[0072] At 601, vehicle adjustments are determined. For example, vehicle speed and steering adjustments are determined to keep the vehicle on a path target. In some embodiments, the vehicle adjustments are determined by a vehicle controller such as Figure 7 vehicle controller 707. In some embodiments, the vehicle controller determines distances, speeds, orientations, and / or other driving parameters for controlling the vehicle. In some embodiments, a maximum vehicle speed is determined and used to limit the vehicle speed. The maximum speed can be enforced to increase the safety of navigation and / or to allow the user to have sufficient reaction time to terminate the summon function.

[0073] At 603, the vehicle is adjusted to maintain its route along the path target. For example, the vehicle adjustments determined at 601 are implemented. In some embodiments, vehicle actuators such as Figure 7 vehicle actuator 713 implement the vehicle adjustments. The vehicle actuator adjusts the steering and / or speed of the vehicle. In various embodiments, all adjustments are recorded and can be uploaded to a remote server for later viewing. For example, in the case of a safety issue, vehicle drive, path target, destination location, current location, driving speed, and / or other driving parameters can be inspected to identify potential areas for improvement.

[0074] At 605, the vehicle is operated according to the vehicle adjustments and the operation of the vehicle is monitored. For example, the vehicle operates as indicated by the vehicle adjustments applied at 603. In various embodiments, the operation of the vehicle is monitored to enforce safety, comfort, performance, efficiency, and other operating parameters.

[0075] At 607, it is determined whether an obstacle is detected. In the case where an obstacle is detected, the process proceeds to 611. In the case where no obstacle is detected, the process proceeds to 605, where the operation of the vehicle continues to be monitored. In some embodiments, the obstacle is detected by a collision or object sensor, such as an ultrasonic sensor. In some embodiments, the obstacle can be communicated via a network interface. For example, an obstacle detected by another vehicle can be shared and received. In various embodiments, the detected obstacle can be used to notify other components of the autonomous vehicle system, such as components related to occupancy grid generation, but is also received at the navigation component to allow the vehicle to immediately adjust for the detected obstacle.

[0076] At 609, it is determined whether the connection with the user is lost. In the case where the connection with the user is lost, the process proceeds to 611. In the case where the connection with the user is not lost, the process proceeds to 605, where the operation of the vehicle continues to be monitored. In some embodiments, continuous connection with the user is required to activate the automatic navigation. For example, a virtual heartbeat is sent from the user. The heartbeat can be sent from the user's smartphone device or other suitable device (such as a key fob). In some embodiments, as long as the user activates the virtual heartbeat, the virtual heartbeat is received and the connection is not lost. In response to the user no longer sending the virtual heartbeat, the connection with the user is considered lost and the navigation of the vehicle makes an appropriate response at 611.

[0077] In some embodiments, as long as the user continuously connects with a heartbeat switch, button, or other user interface device, the virtual heartbeat can be implemented (and continuously sent to maintain the connection). Once the user disconnects from the appropriate user interface element, the virtual heartbeat is no longer transmitted and the connection is lost.

[0078] At 611, the navigation is overridden. In response to detecting an obstacle and / or the loss of connection with the user, the automatic navigation is overridden. For example, if the vehicle is traveling at a low speed, the vehicle can stop immediately. If the vehicle is traveling at a higher speed, the vehicle will stop safely. A safe stop may require gradual braking and determining a safe stop location, such as the side of the road or a parking space. In various embodiments, overriding the navigation may require the user to actively resume the automatic navigation. In some embodiments, once the detected obstacle no longer exists, the overridden navigation is resumed.

[0079] In some embodiments, in the case where the navigation is overridden due to the detection of an obstacle, the vehicle is rerouted to the destination using a new path. For example, a new path target that can avoid the detected obstacle is determined. In various embodiments, the occupancy grid is updated using the detected obstacle, and the updated occupancy grid is used to determine the new path target. Once a viable new path is determined, the navigation can continue at an appropriate time, such as once the connection is re-established. In some embodiments, a new path is determined when the navigation is overridden. For example, when the connection is lost, the path to the destination is reconfirmed. If appropriate, the existing path can be used, or a new path can be selected. In various embodiments, if no viable path can be found, the vehicle remains stopped. For example, the vehicle is completely blocked.

[0080] Figure 7is a block diagram showing an embodiment of an autonomous vehicle system for automatically navigating a vehicle to a destination target. The autonomous vehicle system includes different components that can be used together to automatically navigate the vehicle to a target geographical location. In the illustrated example, the autonomous vehicle system includes an on-vehicle component 700, a remote interface component 751, and a navigation server 761. The on-vehicle component 700 is a component installed on the vehicle. The remote interface component 751 is one or more remote components that can be used away from the vehicle to automatically navigate the vehicle. For example, the remote interface component 751 includes a smartphone application running on a smartphone device, a key fob, a GUI for controlling the vehicle such as a website, and / or another remote interface component. The navigation server 761 is an optional server for facilitating navigation features. The navigation server 761 is a remote server and can be used as a remote cloud server and / or storage device. In some embodiments, Figure 7 the autonomous vehicle system is used to implement Figures 1 to 6 the process and functions associated with Figures 8 to 9 the user interface.

[0081] In the illustrated example, the on-vehicle component 700 is an autonomous vehicle system that includes a vision sensor 701, a perception module 703, a path planning module 705, a vehicle controller 707, additional sensors 709, a safety controller 711, vehicle actuators 713, and a network interface 715. In various embodiments, the different components are communicatively connected. For example, sensor data from the vision sensor 701 and the additional sensors 709 is fed to the perception module 703. The output of the perception module 703 is fed to the path planning module 705. The output of the path planning module 705 and sensor data from the additional sensors 709 are fed to the vehicle controller 707. In some embodiments, the output of the vehicle controller 707 is a vehicle control command that is fed to the vehicle actuators 713 to control the operation of the vehicle, such as the speed, braking, and / or steering of the vehicle. In some embodiments, sensor data from the additional sensors 709 is fed to the vehicle actuators 713 to perform additional safety checks. In various embodiments, the safety controller 711 is connected to one or more components, such as the perception module 703, the vehicle controller 707, and / or the vehicle actuators 713, to implement safety checks at each module. For example, the safety controller 711 can receive additional sensor data from the additional sensors 709 to override the automatic navigation.

[0082] In various embodiments, sensor data, machine learning results, perception module results, path planning results, safety controller results, etc. can be sent to the navigation server 761 via the network interface 715. For example, sensor data can be transmitted via the network interface 715 to the navigation server 761 to collect training data to improve the performance, comfort, and / or safety of the vehicle. In various embodiments, the network interface 715 is used to communicate with the navigation server 761, make phone calls, send and / or receive text messages, and transmit sensor data based on the operation of the vehicle, and for other reasons. In some embodiments, the vehicle component 700 may include more or fewer components as the case may be. For example, in some embodiments, the vehicle component 700 includes an image preprocessor (not shown) to enhance sensor data. As another example, the image preprocessor can be used to normalize an image or transform an image. In some embodiments, noise, distortion, and / or blur are removed or reduced during the preprocessing step. In various embodiments, the image is adjusted or normalized to improve the results of machine learning analysis. For example, the white balance of the image is adjusted to account for different lighting operating conditions, such as daylight, sunny, cloudy, dusk, sunrise, sunset, and night conditions, etc. As another example, an image captured with a fish-eye lens can be distorted and the image preprocessor can be used to transform the image to remove or modify the distortion. In various embodiments, one or more components of the vehicle component can be distributed to a remote server such as the navigation server 761.

[0083] In some embodiments, the vision sensor 701 includes one or more vision sensors. In various embodiments, the vision sensor 701 can be fixed to the vehicle at different positions of the vehicle, and / or oriented in one or more different directions. For example, the vision sensor 701 can be fixed to the front, side, rear, and / or roof of the vehicle in the forward, backward, lateral, etc. directions. In some embodiments, the vision sensor 701 is an image sensor such as a high-dynamic range camera. For example, a high-dynamic range front camera captures image data in front of the vehicle. In some embodiments, the vehicle is attached with multiple sensors for capturing data. For example, in some embodiments, eight surround cameras are fixed to the vehicle and provide 360-degree visibility around the vehicle, with a range up to 250 meters. In some embodiments, the camera sensors include a wide front camera, a narrow front camera, a rear camera, a front side camera, and / or a rear side camera. Various camera sensors are used to capture the environment around the vehicle, and the captured images are provided for deep learning analysis.

[0084] In some embodiments, the vision sensor 701 is not mounted to the vehicle of the vehicle component 700. For example, the vision sensor 701 can be mounted on an adjacent vehicle and / or fixed to the road or environment and be included as a learning system for capturing sensor data. In some embodiments, the vision sensor 701 includes one or more cameras that capture the road surface on which the vehicle travels. For example, one or more front and / or pillar cameras capture lane markings of the lane in which the vehicle travels. The vision sensor 701 can include an image sensor capable of capturing still images and / or video. Data can be captured over a period of time, such as a sequence of data captured over a period of time.

[0085] In some embodiments, the perception module 703 is used to analyze the sensor data to generate a representation of the environment around the vehicle. In some embodiments, the perception module 703 utilizes a trained machine learning network to generate an occupancy grid. The perception module 703 can utilize a deep learning network to take as input sensor data that includes data from the vision sensor 701 and / or additional sensors 709. The deep learning network of the perception module 703 can be an artificial neural network, such as a convolutional neural network (CNN), which is trained on inputs such as sensor data and whose output is provided to the path planning module 705. As an example, the output can include the drivable space of the environment around the vehicle. In some embodiments, the perception module 703 receives at least the sensor data as input. Additional inputs can include scene data that describes the environment around the vehicle and / or vehicle specifications, such as the operating characteristics of the vehicle. The scene data can include scene labels that describe the environment around the vehicle, such as rain, wet road, snow, mud, high density traffic, highway, city, school zone, etc. In some embodiments, the perception module 7031 is utilized at Figure 1 103 of Figure 3 305 and / or 309 of Figure 4 411 and / or Figure 5 the process of

[0086] In some embodiments, the path planning module 705 is a path planning component for selecting an optimal path to navigate the vehicle from one location to another. The path planning component can utilize the occupancy grid and a cost function to select the optimal route. In some embodiments, the potential paths consist of one or more path primitives, such as arc primitives, that model the operating characteristics of the vehicle. In some embodiments, the path planning module 705 is utilized at Figure 1 step 103 and / or Figure 3 step 309 of

[0087] In some embodiments, the vehicle controller 707 is configured to process the output of the path planning module 705 and convert the selected path into vehicle control operations or commands. In some embodiments, the vehicle controller 707 is configured to control the vehicle to automatically navigate to the selected destination target. In various embodiments, the vehicle controller 707 may adjust the speed, acceleration, steering, braking, etc. of the vehicle by transmitting instructions to the vehicle actuator 713. For example, in some embodiments, the vehicle controller 707 is configured to control the vehicle to maintain the position of the vehicle on the path from the current position to the selected destination.

[0088] In some embodiments, the vehicle controller 707 is configured to control vehicle lighting, such as brake lights, turn signals, headlights, etc. In some embodiments, the vehicle controller 707 is configured to control the vehicle audio conditions, such as the vehicle's audio system, play audio alerts, enable microphones, enable speakers, etc. In some embodiments, the vehicle controller 707 is configured to control the notification system including the warning system to notify the driver and / or passengers of driving events, such as potential collisions or approaching a predetermined destination. In some embodiments, the vehicle controller 707 is configured to adjust sensors such as the vehicle's sensor 701. For example, the vehicle controller 707 may be used to change the parameters of one or more sensors, such as modifying the orientation, changing the output resolution and / or format type, increasing or decreasing the capture rate, adjusting the captured dynamic range, adjusting the focus of the camera, enabling and / or disabling the sensor, etc.

[0089] In some embodiments, in addition to the vision sensor 701, the additional sensor 709 further includes one or more other sensors. In various embodiments, the additional sensor 709 may be fixed to the vehicle, at different positions of the vehicle, and / or oriented in one or more different directions. For example, the additional sensor 709 may be fixed to the front, side, rear, and / or roof of the vehicle in the forward, rearward, lateral, etc. directions. In some embodiments, the additional sensor 709 includes radar, audio, LiDAR, inertial, odometer, position, and / or ultrasonic sensors, etc. Ultrasonic and / or radar sensors may be used to capture surrounding details. For example, twelve ultrasonic sensors may be fixed to the vehicle to detect hard and soft objects. In some embodiments, the forward radar is used to capture data of the surrounding environment. In various embodiments, the radar sensor is capable of capturing surrounding details despite heavy rain, fog, dust, and other vehicles. Various sensors are used to capture the environment around the vehicle, and the captured images are provided for deep learning analysis.

[0090] In some embodiments, the additional sensor 709 is not mounted to the vehicle of the vehicle component 700. For example, the additional sensor 709 may be mounted on an adjacent vehicle and / or fixed to the road or environment and is included as part of an autonomous vehicle system for capturing sensor data. In some embodiments, the additional sensor 709 includes one or more non-visual sensors that capture the road surface on which the vehicle travels. In some embodiments, the additional sensor 709 includes a position sensor, such as a Global Positioning System (GPS) sensor for determining the vehicle's position and / or change in position.

[0091] In some embodiments, the safety controller 711 is a safety component for performing safety checks on the vehicle component 700. In some embodiments, the safety controller 711 receives sensor inputs from the vision sensor 701 and / or the additional sensor 709. If an object and / or a collision is likely to occur, the safety controller 711 may notify different components of an impending safety issue. In some embodiments, the safety controller 711 is capable of interrupting and / or enhancing the results of the perception module 703, the vehicle controller 707, and / or the vehicle actuator 713. In some embodiments, the safety controller 711 is used to determine how to respond to a detected safety issue. For example, at low speeds, the vehicle may stop immediately, but at high speeds, the vehicle must safely decelerate and stop in a safe location. In various embodiments, the safety controller 711 communicates with the remote interface component 751 to detect whether a real-time connection is established between the user of the vehicle component 700 and the remote interface component 751. For example, the safety controller 711 may monitor the virtual heartbeat from the remote interface component 751 and, in the case where the virtual heartbeat is no longer detected, may trigger a safety alert to terminate or modify the automatic navigation. In some embodiments, Figure 6 the process of... is implemented at least in part by the safety controller 711.

[0092] In some embodiments, the vehicle actuator 713 is used to implement specific operation controls of the vehicle. For example, the vehicle actuator 713 initiates a change in vehicle speed and / or steering. In some embodiments, the vehicle actuator 713 sends operation commands to the drive inverter and / or the steering rack. In various embodiments, in the case of a detected potential collision, the vehicle actuator 713 performs a safety check based on inputs from the additional sensor 709 and / or the safety controller 711. For example, the vehicle actuator 713 causes the vehicle to stop immediately.

[0093] In some embodiments, network interface 715 is a communication interface for sending and / or receiving data including voice data. In various embodiments, network interface 715 includes a cellular or wireless interface for interfacing with a remote server, making and receiving voice calls, sending and / or receiving text messages, transmitting sensor data, receiving updates to the perception module (including updated machine learning models), and retrieving environmental conditions (including weather conditions and forecasts, traffic conditions, traffic rules and regulations, etc.). For example, network interface 715 can be used to receive instructions and / or updated operating parameters from sensor 701, perception module 703, path planning module 705, vehicle controller 707, additional sensor 709, safety controller 711, and / or vehicle actuator 713. The machine learning model of perception module 703 can be updated using network interface 715. As another example, network interface 715 can be used to update the firmware of vision sensor 701 and / or the operating target of path planning module 705, such as a weighted cost. As yet another example, network interface 715 can be used to transmit occupancy grid data to navigation server 761 for sharing with other vehicles.

[0094] In some embodiments, remote interface component 751 is one or more remote components that can be used away from the vehicle to automatically navigate the vehicle. For example, remote interface component 751 includes a smartphone application running on a smartphone device, a key fob, a GUI for controlling the vehicle such as a website, and / or another remote interface component. A user can initiate a summon function to automatically navigate the vehicle to a selected destination target specified by a geographical location. For example, the user can have the vehicle find and follow the user. As another example, the user can use remote interface component 751 to specify a parking location, and the vehicle will automatically navigate to the specified location or the nearest location where it can safely reach the specified location.

[0095] In some embodiments, navigation server 761 is an optional remote server and includes remote storage. Navigation server 761 can store occupancy grids and / or occupancy data, which can be used later to initialize newly generated occupancy grids. In some embodiments, navigation server 761 is used to synchronize occupancy data across different vehicles. For example, occupancy data generated by other vehicles can be used to initiate or update a vehicle in an area with new occupancy data. In various embodiments, navigation server 761 can communicate with in-vehicle component 700 via network interface 715. In some embodiments, navigation server 761 can communicate with remote interface component 751. In some embodiments, one or more components or partial components of in-vehicle component 700 are implemented on navigation server 761. For example, navigation server 761 can perform processes such as perception processing and / or path planning, and provide the required results to in-vehicle component 700.

[0096] Figure 8 is a diagram showing an embodiment of a user interface for automatically navigating a vehicle to a destination target. In some embodiments, Figure 8 the user interface is used to initiate and / or monitor Figures 1 to 6 the process of Figure 8 the user interface is the user interface of a smartphone application and / or Figure 7 the remote interface component 751 of Figure 8 associated functions are initiated at 203 of Figure 2 to navigate the vehicle to the user's location. For example, a user interacting with the user interface on a smartphone device activates the "Find Me" action to navigate the vehicle to the location of the user's smartphone device, which is accurately close to the user's location. In the example shown, the user interface 800 includes the user interface component map 801, the dialogue window 803, the vehicle locator element 805, the user locator element 809, and the effective summons area element 807.

[0097] In some embodiments, the user interface 800 displays the map 801, where the vehicle position is indicated at the vehicle locator element 805 and the user position is indicated at the user locator element 809. The area that the vehicle can cross when automatically navigating to the user is indicated by the effective summons area element 807. In various embodiments, the effective summons area element 807 is a circle showing the maximum distance allowed for the vehicle to travel. In some embodiments, the effective summons area element 807 takes into account the line of sight from the user to the vehicle, and only the area with line of sight is allowed for automatic navigation.

[0098] In the example shown, the map 801 is a satellite map, although alternative maps can be used. In some embodiments, the map 801 can be manipulated to view different locations, for example, by panning or zooming the map 801. In various embodiments, the map 801 includes a three-dimensional view (not shown) to allow the user to select different heights, such as different floors of a parking lot. For example, areas with drivable regions at different heights are highlighted and can be viewed in a breakdown view.

[0099] In some embodiments, the vehicle locator element 805 displays both the position and orientation (or heading) of the vehicle. For example, the direction the vehicle is facing is indicated by the direction the arrow of the vehicle locator element 805 points. The user locator element 809 indicates the position of the user. In some embodiments, the positions of other potential passengers are also displayed, for example, in a different color (not shown). In some embodiments, the map 801 is centered on the vehicle locator element 805. In various embodiments, other data can be used as the center of the map 801, such as the original (or starting) position of the vehicle, the current position of the user, the closest reachable position of the user's current position, etc.

[0100] In the example shown, the dialogue window 803 includes a text description such as "Press and hold to start, or tap on the map to select a destination" to inform the user how to activate the summon feature. In some embodiments, the default action is to navigate the vehicle to the user. The default action is activated by selecting the "Find Me" button that is part of the dialogue window 803. In some embodiments, once the "Find Me" action is enabled, the selected path is displayed on the user interface ( Figure 8 not shown). When the vehicle is navigating to the user's location, the vehicle locator element 805 is updated to reflect the new position of the vehicle. Similarly, as the user moves, the user locator element 809 is updated to reflect the new position of the user. In some embodiments, a trajectory shows the change in the user's position. In various embodiments, the user must keep contact with the virtual heartbeat button to allow the automatic navigation to continue. Once the user releases the virtual heartbeat button, the automatic navigation stops. In some embodiments, the heartbeat button is the "Find Me" button of the dialogue window 803. In some embodiments, separate forward and backward buttons are used as virtual heartbeat buttons to confirm automatic navigation in the forward and backward directions respectively (not shown).

[0101] In some embodiments, an additional "Find" function (not shown) can automatically navigate the vehicle to the position of a selected person, for example, to pick up a passenger different from the user. In various embodiments, pre-identified positions that can be selected as destination targets can be displayed (not shown). For example, a position can be pre-identified as a valid parking or waiting position for picking up a passenger.

[0102] Figure 9 is a diagram showing an embodiment of a user interface for automatically navigating a vehicle to a destination target. In some embodiments, Figure 9 the user interface is for initiating and / or monitoring Figures 1 - 6 the process. In some embodiments, Figure 9 the user interface is the user interface of a smartphone application and / or Figure 7 the remote interface component 751. In some embodiments, related to Figure 9The associated function is initiated at Figure 2 201 of Figure 2 to navigate the vehicle to a location specified by the user. For example, a user interacting with the user interface on a smart phone device would drop a pin to specify a destination target for the vehicle to navigate to. In the illustrated example, the user interface 900 includes a user interface component map 901, a dialogue window 903, a vehicle locator element 905, a user locator element 909, an active summon area element 907, and a destination target element 911.

[0103] In some embodiments, the user interface 900 displays a map 901 where the vehicle location is indicated at the vehicle locator element 905 and the user location is indicated at the user locator element 909. The area that the vehicle can traverse when automatically navigating to the user is indicated by the active summon area element 907. In various embodiments, the active summon area element 907 is a circle that shows the maximum distance that the vehicle is allowed to travel, and the user can only select a destination target within the active summon area element 907. In some embodiments, the active summon area element 907 takes into account the line of sight from the user to the vehicle, and only the area with line of sight is allowed for automatic navigation.

[0104] In the illustrated example, the map 901 is a satellite map, although alternative maps can be used. In some embodiments, the map 901 can be manipulated to view different locations, e.g., by panning or zooming the map 901. In various embodiments, the map 901 includes a three-dimensional view (not shown) to allow the user to select different heights, such as different floors of a parking lot. For example, areas with drivable regions of different heights would be highlighted and can be viewed in an exploded view to select a destination target.

[0105] In some embodiments, the vehicle locator element 905 shows both the location and orientation (or heading) of the vehicle. For example, the direction the vehicle is facing is indicated by the direction the arrow of the vehicle locator element 905 points. The user locator element 909 indicates the user's location. In some embodiments, the locations of other potential passengers are also shown, e.g., in a different color (not shown). In some embodiments, the map 901 is centered on the vehicle locator element 905. In various embodiments, other data can be used as the center of the map 901, such as the vehicle's original (or starting) location, the user's current location, the closest reachable location to the user's current location, the selected destination target represented by the destination target element 911, etc.

[0106] In the illustrated example, the dialogue window 903 includes text descriptions such as "Press and hold to start, or tap on the map to select a destination" to inform the user on how to activate the summon function. The user can select a target destination by choosing a location within the valid summon area element 907 on the map 901. A destination target element 911 is displayed at the selected valid location. In the illustrated example, a fixture icon is used for the destination target element 911. In some embodiments, an icon with an orientation (not shown) (such as an arrow or a vehicle icon) is used as the destination target element 911. The icon of the destination target element 911 can be manipulated to select the final destination orientation. In various embodiments, the selected orientation is verified to confirm that the orientation is valid. For example, a destination orientation facing traffic may not be allowed on a one-way street. To clear the selected location associated with the destination target element 911, the user can select the "Clear Fixture" dialog box of the dialogue window 903. To initiate the auto-navigation, the user selects the "Start" button of the dialogue window 903. In some embodiments, once the auto-navigation is enabled, the selected path will be displayed on the user interface ( Figure 9 not shown). When the vehicle is navigating to the destination target location, the vehicle locator element 905 is updated to reflect the new position of the vehicle. Similarly, as the user moves, the user locator element 909 is updated to reflect the new position of the user. In some embodiments, a trajectory shows the change in the user's position. In various embodiments, the user must keep in contact with a virtual heartbeat button to allow the auto-navigation to continue. Once the user releases the virtual heartbeat button, the auto-navigation stops. In some embodiments, the heartbeat button is the "Start" button of the dialogue window 903. In some embodiments, separate forward and backward buttons are used as the virtual heartbeat button to confirm auto-navigation in the forward and backward directions respectively (not shown).

[0107] Although some of the foregoing embodiments have been described in some detail for purposes of clear understanding, the present invention is not limited to the details provided. There are many alternative ways to implement the present invention. The disclosed embodiments are illustrative rather than restrictive.

Claims

1. A system, comprising: one or more sensors configured to generate sensor data by capturing the real - world environment around the autonomous system, and one or more processors configured to: obtain information indicating a location to which the autonomous system is to navigate; use a machine - learning model and the sensor data to generate a representation of the real - world environment around the autonomous system, wherein the machine - learning model is configured to determine a plurality of drivable spaces and a plurality of non - drivable spaces around the autonomous system based on the sensor data; generate an occupancy grid including a plurality of grid positions, each grid position corresponding to a different one of the plurality of drivable spaces or the plurality of non - drivable spaces; wherein the occupancy grid includes one or more drivability values at each of the plurality of grid positions, and wherein a drivability value at a grid position represents a numerical probability that the corresponding position in the real - world environment is drivable and is selected from a plurality of drivability values within a range of numerical drivability values; and cause the autonomous system to navigate based on a path associated with navigating to the location, wherein the path is calculated based on the representation of the real - world environment.

2. The system according to claim 1, wherein the location is selected via a user interface of an application configured to execute on a mobile device.

3. The system according to claim 1, wherein the location is based on a global positioning system location associated with a mobile device.

4. The system according to claim 3, wherein the location is updated during navigation based on the location associated with the mobile device.

5. The system according to claim 1, wherein navigation is aborted in response to information indicating a lack of user input to an application configured to execute on a mobile device.

6. The system according to claim 1, wherein the one or more sensors include at least one visual sensor.

7. The system according to claim 1, wherein the one or more sensors include one or more of a camera, a radar, or an ultrasonic sensor.

8. The system according to claim 7, wherein the one or more sensors include lidar.

9. The system according to claim 1, wherein the path is calculated based on one or more cost metrics associated with the one or more drivability values.

10. The system according to claim 9, wherein multiple paths are calculated, and wherein the path is selected according to the one or more cost metrics and a cost function that assigns costs to the multiple paths.

11. A method implemented by a system of one or more processors, the system communicating with one or more sensors configured to generate sensor data by capturing the real - world environment around an autonomous system, and the method comprising: obtain information indicating a location to which the autonomous system is to navigate; Generate a representation of the real-world environment around the autonomous system using the machine learning model and the sensor data, wherein the machine learning model is configured to determine a plurality of drivable spaces and a plurality of non-drivable spaces around the autonomous system based on the sensor data; Generate an occupancy grid, the occupancy grid including a plurality of grid positions, each grid position corresponding to a different one of the plurality of drivable spaces or the plurality of non-drivable spaces, wherein the occupancy grid includes one or more drivability values at each of the plurality of grid positions, and wherein the drivability value at a grid position represents the numerical probability that the corresponding position in the real-world environment is drivable and is selected from a plurality of drivability values within a range of numerical drivability values; and Cause the autonomous system to navigate based on a path associated with navigating to the position, wherein the path is calculated based on the representation of the real-world environment.

12. The method according to claim 11, wherein the position is selected via a user interface of an application configured to execute on a mobile device.

13. The method according to claim 11, wherein the position is based on a global positioning system position associated with the mobile device.

14. The method according to claim 13, wherein during navigation the position is updated based on the position associated with the mobile device.

15. The method according to claim 11, wherein navigation is aborted in response to information indicating a lack of user input to an application configured to execute on a mobile device.

16. The method according to claim 11, wherein the one or more sensors include at least one visual sensor.

17. The method according to claim 11, wherein the one or more sensors include one or more of a camera, a radar, or an ultrasonic sensor.

18. The method according to claim 17, wherein the one or more sensors include lidar.

19. The method according to claim 11, wherein the path is calculated based on one or more cost metrics associated with the one or more drivability values, and wherein a plurality of paths are calculated, and wherein the path is selected according to the one or more cost metrics and a cost function that assigns costs to the plurality of paths.

20. A non-transitory computer storage medium storing instructions that, when executed by a system including one or more processors, cause the one or more processors to perform operations, wherein the system communicates with one or more sensors configured to generate sensor data by capturing a real-world environment around an autonomous system, and wherein the operations include: Obtain information indicating a position to which the autonomous system is to navigate; Generate a representation of the real-world environment around the autonomous system using the machine learning model and the sensor data, wherein the machine learning model is configured to determine a plurality of drivable spaces and a plurality of non-drivable spaces around the autonomous system based on the sensor data; Generate an occupancy grid, the occupancy grid including a plurality of grid positions, each grid position corresponding to a different one of the plurality of drivable spaces or the plurality of non-drivable spaces, wherein the occupancy grid includes one or more drivability values at each of the plurality of grid positions, and wherein the drivability value at a grid position represents a numerical probability that the corresponding position in the real-world environment is drivable and is selected from a plurality of drivability values within a range of numerical drivability values; and cause the autonomous system to navigate based on a path associated with navigating to the location, wherein the path is calculated based on the representation of the real-world environment.

Images