Autonomous and user-controlled vehicle summoning to a destination

Abstract

A processor coupled to the memory is configured to receive an identification of a geographic location associated with a target specified by a user remote from the vehicle. Using a machine learning model, a representation of at least a portion of an environment around the vehicle is generated using sensor data from one or more sensors of the vehicle. At least a portion of a path to a target location corresponding to the received geographic location is calculated using the generated representation of at least a portion of the environment around the vehicle. At least one command for automatically navigating the vehicle is provided based on the determined path and updated sensor data from at least a portion of the one or more sensors of the vehicle.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation of and claims priority to U.S. patent application No. 16 / 272,273, filed on February 11, 2019, entitled “AUTONOMOUS AND USER CONTROLLED VEHICLE SUMMON TO A TARGET,” the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0003] Human drivers are often required to operate the vehicle, and the driving tasks performed by them are often complex and can be exhausting. For example, retrieving or summoning a parked car from a crowded parking lot or garage can be inconvenient and tedious. Maneuvers often involve walking to one's vehicle, making three-point turns, driving out of tight spots without contacting adjacent vehicles or walls, and driving to a location one had previously come from. Although some vehicles are capable of remote operation, the routes that the vehicle travels are often limited to a single straight path in either the forward or reverse direction, with limited steering range, and an inability to intelligently navigate the vehicle along its own path. Summary of the invention

[0004] One embodiment includes a system. The system includes a processor and a memory, the processor being configured to: receive an identification of a geographic location associated with a target specified by a user remote from a vehicle; generate a representation of at least a portion of an environment around the vehicle using sensor data from one or more sensors of the vehicle using a machine learning model; calculate at least a portion of a path to a target location corresponding to the received geographic location using the generated representation of at least a portion of the environment around the vehicle; and provide at least one command for automatically navigating the vehicle based on the determined path and updated sensor data from at least a portion of the one or more sensors of the vehicle; the memory being coupled to the processor and configured to provide instructions to the processor.

[0005] Another embodiment includes a method. The method includes receiving an identification of a geographic location associated with a target specified by a user remote from a vehicle; generating, with a neural network, a representation of at least a portion of an environment around the vehicle using sensor data from one or more sensors of the vehicle; calculating at least a portion of a path to a target location corresponding to the received geographic location using the generated representation of at least a portion of the environment around the vehicle; and providing at least one command for automatically navigating the vehicle based on the determined path and updated sensor data from at least a portion of the one or more sensors of the vehicle.

[0006] Yet another embodiment includes a computer program product embodied in a non-transitory computer-readable storage medium and comprising computer instructions for: receiving an identification of a geographic location associated with a target specified by a user remote from a vehicle; generating a representation of at least a portion of an environment surrounding the vehicle using sensor data from one or more sensors of the vehicle using a neural network; calculating at least a portion of a path to a target location corresponding to the received geographic location using the generated representation of at least a portion of the environment surrounding the vehicle; and providing at least one command for automatically navigating the vehicle based on the determined path and updated sensor data from at least a portion of the one or more sensors of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0008] Figure 1 is a flow chart illustrating an embodiment of a process for automatically navigating a vehicle to a destination object.

[0009] Figure 2 is a flow chart illustrating an embodiment of a process for receiving a target destination.

[0010] Figure 3 is a flow chart illustrating an embodiment of a process for automatically navigating a vehicle to a destination object.

[0011] Figure 4 is a flow chart illustrating an embodiment of a process for training and applying a machine learning model to generate a representation of the environment surrounding a vehicle.

[0012] Figure 5 is a flow chart illustrating an embodiment of a process for generating an occupancy grid.

[0013] Figure 6 is a flow chart illustrating an embodiment of a process for automatically navigating to a destination object.

[0014] Figure 7 is a block diagram illustrating an embodiment of an autonomous vehicle system for automatically navigating a vehicle to a destination object.

[0015] Figure 8 is a diagram illustrating an embodiment of a user interface for automatically navigating a vehicle to a destination object.

[0016] Fig. 9 is a diagram illustrating an embodiment of a user interface for automatically navigating a vehicle to a destination object. DETAILED DESCRIPTION

[0017] The present invention may be implemented in a variety of ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored on and / or provided by a memory coupled to the processor. In this specification, these implementations or any other form that the present invention may take may be referred to as techniques. In general, the order of the steps of the disclosed processes may be changed within the scope of the present invention. Unless otherwise stated, components such as processors or memories described as being configured to perform a task may be implemented as general components that are temporarily configured to perform the task at a given time or as specific components manufactured to perform the task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0018] Provided below is a detailed description of one or more embodiments of the present invention and drawings illustrating the principles of the present invention. The present invention is described in conjunction with these embodiments, but the present invention is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications and equivalents. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. These details are provided for the purpose of example, and the present invention can be practiced according to the claims without some or all of these specific details. For the sake of clarity, technical materials known in the technical field related to the present invention are not described in detail to avoid unnecessary confusion of the present invention.

[0019] A technology for autonomously summoning a vehicle to a destination is disclosed. A target geographic location is provided and the vehicle automatically navigates to the target location. For example, a user provides a location by placing a pin at the destination location on a graphical map user interface. As another example, a user summons a vehicle to the user's location by specifying the user's location as the destination location. The user can also select a destination location based on a feasible path detected for the vehicle. The destination location can be updated (for example, if the user moves around) to guide the car to update its path to the destination location. Using the specified destination location, the vehicle navigates by utilizing sensor data (such as visual data captured by a camera) to generate a representation of the environment around the vehicle. In some embodiments, the representation is an occupancy grid detailing drivable and non-drivable spaces. In some embodiments, the occupancy grid is generated from camera sensor data using a neural network. The representation of the environment can be further enhanced with auxiliary data, such as additional sensor data (including radar data), map data, or other inputs. Using the generated representation, a path is planned from the current location of the vehicle to the destination location. In some embodiments, the path is generated based on operating parameters of the vehicle (such as a turning radius of the model, vehicle width, vehicle length, etc.). The optimal path is selected based on selected parameters such as distance, speed, number of gear shifts, etc. As the vehicle navigates automatically using the selected path, the representation of the environment is continuously updated. For example, as the vehicle drives to the destination, new sensor data is captured and the occupancy grid is updated. Safety checks are continuously performed, and safety checks can override or modify the automatic navigation. For example, ultrasonic sensors can be used to detect potential collisions. The user can monitor the navigation at any time and cancel the summoned vehicle. In some embodiments, the vehicle must continuously receive a virtual heartbeat signal from the user in order for the vehicle to continue navigating. The virtual heartbeat can be used to indicate that the user is actively monitoring the progress of the vehicle. In some embodiments, the user selects the path and the user remotely controls the vehicle. For example, the user can control the steering angle, direction, and / or speed of the vehicle to remotely control the vehicle. When the user remotely controls the vehicle, safety checks are continuously performed, and safety checks can override and / or modify user controls. For example, if an object is detected, the vehicle can be stopped.

[0020] In some embodiments, a system includes a processor configured to receive an identification of a geographic location associated with a target specified by a user remote from a vehicle. For example, a user waiting at a pickup location specifies her or his geographic location as a destination target. The destination target is received by a vehicle parked in a remote parking space. In some embodiments, a machine learning model is used to generate a representation of at least a portion of the environment around the vehicle using sensor data from one or more sensors of the vehicle. For example, sensors such as cameras, radars, ultrasonic waves, or other sensors capture data of the environment around the vehicle. The data is fed as input to a neural network using a trained machine learning model to generate a representation of the environment around the vehicle. As an example, the representation may be an occupancy grid describing a drivable space that the vehicle can use to navigate through. In some embodiments, at least a portion of a path to a target location corresponding to the received geographic location is calculated using the generated representation of at least a portion of the environment around the vehicle. For example, an occupancy grid representing the environment around the vehicle is used to select a path for navigating the vehicle to a target destination. The value at the entry in the occupancy grid may be a probability and / or cost associated with driving through a location associated with a grid entry. In some embodiments, at least one command for automatically navigating the vehicle is provided based on the determined path and updated sensor data from at least a portion of one or more sensors of the vehicle. For example, the vehicle controller uses the selected plan to provide actuator commands for controlling the speed and steering of the vehicle. These commands are used to navigate the vehicle along the selected path. As the vehicle navigates, updated sensor data is captured to update the representation of the environment around the vehicle. In various embodiments, the system includes a memory coupled to the processor and configured to provide instructions to the processor.

[0021] Figure 1 is a flow chart illustrating an embodiment of a process for automatically navigating a vehicle to a destination object. In some embodiments, Figure 1The process is used to summon a vehicle to a geographic location specified by a user. The user can specify the target location using a mobile application, a remote key, a GUI of the vehicle, etc. Using the target location, the vehicle automatically navigates from its starting location to the target location. In some embodiments, the target location is the user's location and the target location is dynamic. For example, as the user moves, the target destination of the vehicle moves to the user's new location. In some embodiments, the target location is indirectly specified, such as via a calendar or planning software. For example, the user's calendar is parsed and used to determine the target destination and time from a calendar event. The calendar event may include the location and time of the event, such as the end time. The destination is selected based on the location of the event, and the time is selected based on the end time of the event. The vehicle automatically navigates to arrive at the location at the end time (such as at the end of a dinner, wedding, restaurant reservation, etc.). In some embodiments, the vehicle navigates to the target destination using a specified time (such as an arrival or departure time). For example, a user can specify a time when the vehicle should start automatic navigation or leave a specified destination. As another example, a user can specify a time when the vehicle should arrive at a specified destination. The vehicle will depart before the specified time so as to arrive at the destination at the specified time. In some embodiments, the arrival time is configured with a threshold time to allow estimation of the difference between the travel time and the actual travel time. In various embodiments, Figure 1 The process is run on the autonomous vehicle. In some embodiments, a remote server in communication with the autonomous vehicle performs a portion of the summoning function. In some embodiments, Figure 1 The process uses at least part of Figure 7 autonomous vehicle systems.

[0022] At 101, a destination is received. For example, a user selects a "find me" feature from a mobile application on a smart phone device. The "find me" feature determines the user's location and transmits the user's location to a vehicle summoning module. In various embodiments, the vehicle summoning module is implemented across one or more modules, one or more of which may be present on the vehicle and / or remote from the vehicle. In some embodiments, the target destination may not be the user's location. For example, a user may select a location on a map as the target destination. As another example, the location may be based on a location associated with a calendar event. In some embodiments, the location is selected by placing a fixture icon on a map at the target destination. The location may be longitude and latitude. In some embodiments, the location may include a height, altitude, or similar measurement. For example, in a parking lot, the location includes a height component for distinguishing different floors of a multi-story parking lot. In various embodiments, the destination is a geographic location.

[0023] In some embodiments, the destination includes a location and an orientation. For example, a car icon, a triangle icon, or other icons with an orientation are used to specify a location and an orientation. By specifying an orientation, the vehicle navigates to the selected target destination and faces the expected orientation selected by the user. In some embodiments, the orientation is determined by an orientation suggestion, such as a suggestion based on the environment, the user, other vehicles, and / or other appropriate inferences. For example, a map can be used to determine the correct orientation of a vehicle, such as the direction in which a vehicle is traveling on a one-way road. As another example, based on lane lines, traffic signals, other vehicles, etc., the summoning function determines the appropriate orientation of the road. For example, the direction that other vehicles are facing can be used as an orientation suggestion. In some embodiments, the orientation is based on an orientation suggested by other autonomous vehicles. For example, in some embodiments, the vehicle summoning function can communicate and / or query other vehicles to determine their orientation and use the provided results as orientation suggestions. In various embodiments, one or more orientation suggestions are weighted and used to determine the target destination orientation.

[0024] In some embodiments, the destination is determined by completing a query. For example, a user may request (e.g., using a GUI, voice, or other query type) that her or his vehicle arrive at a specified parking lot at a certain time. The destination is determined by querying a search engine (such as a map search database) for the destination parking lot. The user's calendar and / or address book may be used to refine the search results. Once the parking lot is identified, instructions and / or traffic flow for entering and / or leaving the parking lot are determined and used to select a destination. The destination is then provided to the vehicle summoning module.

[0025] In some embodiments, the received destination is a multi-part destination. For example, the destination requires reaching one or more waypoints before reaching the final destination. Waypoints can be used to provide additional control over the path the vehicle takes when navigating to the final destination. For example, waypoints can be used to follow a preferred mode of transportation for an airport, parking lot, or other destination. Waypoints can be used to pick up one or more additional passengers, etc. For example, delays and / or pauses can be incorporated into the destination to pick up and / or drop off passengers or objects.

[0026] In some embodiments, the received destination first passes one or more verifications and / or safety checks. For example, the destination can be limited based on distance so that the user can only select destinations within a certain distance (or radius) (such as 100 meters, 10 meters, or another appropriate distance) from the vehicle. In some embodiments, the distance is based on local regulations and / or laws. In some embodiments, the destination location must be a valid parking location. For example, sidewalks, crosswalks, intersections, lakes, etc. are generally not valid parking locations, and the user can be prompted to select a valid location. In some embodiments, the received destination is modified from the user's initial selected destination to take into account safety issues, such as enforcing a valid parking location.

[0027] At 103, a path to the destination is determined. For example, one or more paths are determined for navigating the vehicle from its current location to the destination received at 101. In some embodiments, the paths are mapped using a path planning module (such as Figure 7 The path planning module 705 of the present invention is determined. In various embodiments, the path planning module can be implemented using a cost function, in which appropriate weights are applied to different paths to the destination. In some embodiments, the selected path is based on a potential path arc originating from the vehicle position. The potential path arc is limited to the vehicle's operating dynamics, such as the vehicle's steering range. In various embodiments, each potential path has a cost. For example, a potential path with a sharp turn has a higher cost than a smoother path. As another example, a potential path with a larger speed change has a higher cost than a path with a smaller speed change. As another example, a potential path with more gear changes (e.g., a change from reverse to forward) has a higher cost than a path with fewer gear changes. In some embodiments, the weight of a path with a higher probability of encountering certain objects is different from that of a path with a lower probability of encountering objects. For example, a path is weighted based on the probability of encountering pedestrians, vehicles, animals, traffic, insufficient light, bad weather, tolls, etc. In various embodiments, when determining the path to be taken to reach the destination, a high-cost path is not superior to a low-cost path.

[0028] At 105, the vehicle navigates to the destination. Using the path to the destination determined at 103, the vehicle automatically navigates to the destination received at 101. In some embodiments, the path consists of multiple smaller sub-paths. Different sub-paths can be achieved by driving in different gears (such as forward or reverse). In some embodiments, the path includes actions such as opening a garage door, closing a garage door, passing through a parking door, waiting for an elevator, confirming toll payment, charging, making a call, sending a message, etc. Additional actions may require stopping the vehicle and / or performing operations to manipulate the surrounding environment. In some cases, the final destination is a destination close to the destination received at 101. For example, in some cases, the destination received at 101 is unreachable, so the final destination is very close to reaching the destination. For example, in the case where the user selects a sidewalk, the final destination is a location on the road adjacent to the sidewalk. As another example, in the case where the user selects a crosswalk, the final destination is a location on the road near the crosswalk but not in the crosswalk.

[0029] In some embodiments, one or more safety checks are continuously confirmed while navigating. For example, auxiliary data (such as sensor data from ultrasonic or other sensors) is used to identify obstacles, such as pedestrians, vehicles, speed bumps, traffic control signals, etc. The possibility of a potential collision with an object terminates the current navigation and / or modifies the path to the destination. In some embodiments, the path to the destination includes the speed of travel along the path. As another example, in some embodiments, a heartbeat must be received from the user's mobile application to continue navigation. The heartbeat can be implemented by requiring the user to maintain connection with the summoned vehicle, such as by continuously pressing a button GUI element in the mobile summoning application. As another example, the user must continue to press and hold the button on the remote control key (or maintain contact with the sensor) for the vehicle to automatically navigate. If the contact is lost, the automatic navigation will terminate. In some embodiments, the termination of navigation safely decelerates the vehicle and puts the vehicle in a safe stationary mode, such as parked on the side of the road, a nearby parking space, etc. In some embodiments, the loss of contact can stop the vehicle immediately according to the situation. For example, the user can release the summon button on the mobile application to indicate that the vehicle should stop immediately, similar to an emergency brake switch. In various embodiments, the deceleration used to stop the vehicle depends on the vehicle's environment (e.g., whether there are other vehicles traveling nearby, such as behind the vehicle), whether there is a potential collision (such as an obstacle or pedestrian ahead in the vehicle's path), the speed the vehicle is traveling, or other appropriate parameters. In some embodiments, the speed at which the vehicle is traveling is limited, for example, to a lower maximum speed.

[0030] At 107, the vehicle summoning function is completed. Upon completion, one or more completion actions may be performed. For example, a notification is sent that the vehicle has arrived at the selected target destination. In some embodiments, the vehicle arrives at a destination close to the target destination and the user is presented with a notification of the vehicle's location. In some embodiments, the notification is sent to a mobile application, a key card (e.g., indicated by a state change associated with the key card) via a text message and / or via another appropriate notification channel. Additional directions may be provided to guide the user to the vehicle. In various embodiments, the vehicle may be placed in a parking lot and one or more vehicle settings may be triggered. For example, the interior lighting of the vehicle may be turned on. Floor lighting may be enabled to increase visibility of entering the vehicle. One or more exterior lights may be turned on, such as flashers, parking lights, and / or hazard lights. Exterior lights such as headlights may be activated to increase visibility of passengers approaching the vehicle. The actuated directional lights may point to the intended direction from which the passenger will arrive to reach the vehicle. In some embodiments, audio notifications such as audio alarms or music are played. Welcome music or similar audio may be played in the cabin based on the user's preferences. Similarly, the vehicle's climate control can be enabled to prepare the cabin's climate for the passengers, such as heating or cooling the cabin to a desired temperature and / or humidity. The seats can be heated (or ventilated). The heated steering wheel can be activated. The vents can be redirected based on preferences. The doors can be unlocked and opened for the passengers. If the destination is a charging station, the vehicle can be oriented so that its charger port is aligned with the charger. In some embodiments, the user can configure the vehicle's preferences according to the passengers' expectations, including cabin climate, interior lighting, exterior lighting, audio system, and other vehicle preferences.

[0031] In various embodiments, during and after the call is completed, the location of the vehicle is updated. For example, the updated location can be reflected in the accompanying mobile application. In some embodiments, the completion action includes updating an annotated map of the path traversed and the environment encountered with the recently captured and analyzed sensor data. The annotated map can be updated to reflect potential obstacles, traffic signals, parking preferences, traffic patterns, pedestrian walking patterns, and / or other appropriate path planning metadata that may be useful for future navigation and / or path planning. For example, speed bumps, crosswalks, potholes, empty parking spaces, charging locations, gas stations, etc. encountered are updated to the annotated map. In some embodiments, data corresponding to the encountered objects are used as potential training data to improve the calling functions of the current vehicle / user and other vehicles and users, such as perception, path planning, safety verification, and other functions. For example, an empty parking space can be used for path planning of other vehicles. As another example, a charging station or a gas station is used for path planning. The vehicle can be routed to the charging station and oriented so that its charger port is aligned with the charger.

[0032] Figure 2 is a flow chart illustrating an embodiment of a process for receiving a target destination. In some embodiments, the target destination is selected and / or provided by a user. The destination target is associated with a geographic location and is used as a target for an automated navigation vehicle. In the illustrated example, the process of receiving a target destination can be initiated from more than one starting point. Figure 2 The two starting points are two examples of initiating destination reception. Other methods are also possible. In some embodiments, Figure 2 The process in Figure 1 101 executions.

[0033] At 201, a destination location is received. In some embodiments, the destination is provided as a geographic location. For example, longitude and latitude values ​​are provided. In some embodiments, an altitude is also provided. For example, an altitude associated with a particular floor of a parking lot is provided. In various embodiments, the destination location is provided by a user via a smart phone device, via a media console of a vehicle, or by other means. In some embodiments, the location is received along with a time associated with leaving or arriving at the destination. In some embodiments, one or more destinations are received. For example, in some scenarios, a multi-step destination including more than one stop is received. In some embodiments, the destination location includes an orientation or heading. For example, a heading indicates the direction the vehicle should be facing when arriving at the destination.

[0034] At 203, the user's location is determined. For example, the user's location is determined using a global positioning system or another location-aware technology. In some embodiments, the user's location is approximated by the location of a key fob, the user's smart phone device, or another device in the user's control. In some embodiments, the user's location is received as a geographic location. Similar to 201, the location can be a longitude and latitude pair, and in some embodiments can include an altitude. In some embodiments, the user's location is dynamic and is continuously updated or updated at specific intervals. For example, the user can move to a new location and the received location is updated. In some embodiments, the destination location includes an orientation or heading. For example, the heading indicates the direction the vehicle should face when arriving at the destination.

[0035] At 205, the destination is confirmed. For example, the destination is verified to confirm that the destination is reachable. In some embodiments, the destination must be within a certain distance from the vehicle's origin location. For example, certain regulations may require that the vehicle automatically navigate no more than 50 meters. In various embodiments, an invalid destination may require the user to provide a new location. In some embodiments, an alternative destination is suggested to the user. For example, if the user selects an incorrect orientation, a suggested orientation is provided. For another example, if the user selects a no-parking area, a valid parking area is suggested, such as the nearest valid parking area. Once verified, the selected destination is provided to step 207.

[0036] At 207, the destination is provided to the path planning module. In some embodiments, the path planning module is used to determine the route of the vehicle. At 207, the verified destination is provided to the path planning module. In some embodiments, the destination is provided as one or more locations, for example, a destination with multiple stops. The destination may include a two-dimensional location, such as a latitude and longitude location. Alternatively, in some embodiments, the destination includes an altitude. For example, certain drivable areas (such as multi-story parking lots, bridges, etc.) have multiple drivable planes for the same two-dimensional location. The destination may also include a heading for specifying the orientation that the vehicle should face when arriving at the destination. In some embodiments, the path planning module is Figure 7 The path planning module 705.

[0037] Figure 3 is a flow chart illustrating an embodiment of a process for automatically navigating a vehicle to a destination object. For example, using Figure 3 The process of generating a representation of the environment around the vehicle and using it to determine one or more paths to the destination. The vehicle automatically navigates using the determined (one or more) paths. As additional sensor data is updated, the representation of the environment around is also updated. In some embodiments, Figure 3 The process uses Figure 7 In some embodiments, step 301 is performed by an autonomous vehicle system. Figure 1 101 is performed, steps 303, 305, 307 and / or 309 are performed at Figure 1 103 is performed, steps 311 and / or 313 are performed at Figure 1 The step 105 is performed, and / or the step 315 is performed Figure 1 107. In some embodiments, a neural network is used to guide a path to be followed by a vehicle. For example, using a machine learning network, steering and / or acceleration values ​​are predicted for navigating a vehicle to follow a path to a destination target.

[0038] At 301, a destination is received. For example, a geographic location is received via a mobile application, a key fob, a vehicle control center, or other appropriate device. In some embodiments, the destination is a location and an orientation. In some embodiments, the destination includes an altitude and / or a time. In various embodiments, the destination is received using Figure 2 In some embodiments, the destination is dynamic and new destinations can be received appropriately. For example, in the case where the user selects the "find me" feature, the destination is updated to follow the user's location. Essentially, the vehicle can follow the user like a pet.

[0039] At 303, visual data is received. For example, camera image data is received using one or more camera sensors mounted on the vehicle. In some embodiments, the image data is received from a sensor covering the environment surrounding the vehicle. The visual data may be pre-processed to improve the usefulness of the data for analysis. For example, one or more filters may be applied to reduce noise in the visual data. In various embodiments, visual data is continuously captured to update the environment surrounding the vehicle.

[0040] At 305, determine a drivable space. In some embodiments, a drivable space is determined using a neural network by applying reasoning to the visual data received at 303. For example, a convolutional neural network (CNN) is applied using the visual data to determine drivable space and non-drivable space for the environment surrounding the vehicle. The drivable space includes an area in which the vehicle can travel. In various embodiments, the drivable space is free of obstacles so that the vehicle can travel using a path through the determined drivable space. In various embodiments, a machine learning model is trained to determine the drivable space, and the machine learning model is deployed on the vehicle to automatically analyze and determine the drivable space from the image data.

[0041] In some embodiments, the visual data is supplemented with additional data, such as additional sensor data. The additional sensor data may include ultrasonic, radar, lidar, audio, or other appropriate sensor data. The additional data may also include annotation data, such as map data. For example, an annotated map may annotate lanes, speed lines, intersections, and / or other driving metadata. The additional data may be used as input to a machine learning model or consumed downstream when creating an occupancy grid to improve the results of determining drivable space.

[0042] At 307, an occupancy grid is generated. Using the drivable space determined at 305, an occupancy grid representing the environment of the vehicle is generated. In some embodiments, the occupancy grid is a two-dimensional occupancy grid representing the entire plane (e.g., 360 degrees along the longitude and latitude axes) in which the vehicle is located. In some embodiments, the occupancy grid includes a third dimension to account for altitude. For example, an area with multiple drivable paths at different altitudes (such as a multi-story parking structure, an overpass, etc.) can be represented by a three-dimensional occupancy grid.

[0043] In various embodiments, the occupancy grid contains a drivability value at each grid position corresponding to a position in the surrounding environment. The drivability value of each position can be the probability that the position is drivable. For example, a sidewalk can be designed to have a drivability value of zero, while gravel has a drivability value of 0.5. The drivability value can be a normalized probability ranging between 0 and 1, and is based on the drivable space determined at 305. In some embodiments, each position of the grid includes a cost metric associated with the cost (or penalty / reward) of passing through this position. The cost value of each grid position in the occupancy grid is based on the drivable value. The cost value can also depend on additional data such as preference data. For example, the path preference can be configured to avoid toll roads, carpool lanes, school zones, etc. In various embodiments, the path preference data can be learned via a machine learning model and determined as part of the drivable space at 305. In some embodiments, the path preference data is configured by a user and / or operator to optimize the path taken by navigating to the destination received at 301. For example, the path preference can be optimized to improve goals such as safety, convenience, travel time, and / or comfort. In various embodiments, the preference is an additional weight used to determine the cost value of each location grid.

[0044] In some embodiments, the occupancy grid is updated using auxiliary data such as additional sensor data. For example, ultrasonic sensor data that captures nearby objects is used to update the occupancy grid. Other sensor data such as lidar, radar, audio, etc. may also be used. In various embodiments, the occupancy grid may be generated in part using annotated maps. For example, roads and their attributes (speed limits, lanes, etc.) may be used to augment visual data to generate an occupancy grid. As another example, occupancy data from other vehicles may be used to update the occupancy grid. For example, adjacent vehicles with similarly equipped functionality may share sensor data and / or occupancy grid results.

[0045] In some embodiments, the occupancy data is initialized using the last generated occupancy grid. For example, when the vehicle is no longer capturing new data and / or is parked or powered off, the last occupancy grid is saved. When an occupancy grid is needed, for example, when the vehicle is summoned, the last generated occupancy grid is loaded and used as the initial occupancy grid. This optimization significantly improves the accuracy of the initial grid. For example, some objects may be difficult to detect when stationary, but as the vehicle moves closer to the current parking location, these objects will still be detected in the last saved occupancy grid.

[0046] At 309, a path target is determined. Using the occupancy grid, a search is performed to determine a path for navigating the vehicle from its current position to the destination received at 301. Potential paths are based in part on the operating characteristics of the vehicle, such as turning radius, vehicle width, vehicle length, etc. Each vehicle model can be configured with specific vehicle operating characteristics. In some embodiments, path finding is configured to implement configurable restrictions and / or goals. Example restrictions include that the vehicle cannot travel sideways, the vehicle should limit the number of sharp turns, the vehicle should limit the number of gear shifts (e.g., from reverse to forward or vice versa), etc. Restrictions / goals can be implemented as weighted costs in a cost function. In various embodiments, the initial position of the vehicle includes x, y, and heading values. The x and y values ​​can correspond to longitude and latitude values. Determine one or more potential paths starting from the initial position toward the destination. In some embodiments, the path consists of one or more path primitives (such as arc primitives). Path primitives describe the path (and path target) along which the vehicle can navigate to reach the destination.

[0047] In various embodiments, the selected path target is selected based on a cost function. A cost function is performed on each potential path in the potential path. Each potential path passes through a set of grids of the occupied grid, where each grid has a cost value to reward or penalize driving through the grid position. The path with the best cost value is selected as the path target. The path target may include one or more path primitives, such as arcs, to model the movement of the navigating vehicle. For example, a pair of two path primitives may represent moving the vehicle backward and then moving forward. The reverse path is represented as an arc, and the forward path is represented as another arc. As another example, the vehicle path may include a three-point turn. Each primitive of the turn may be represented as an arc path. At the end of each path primitive, the vehicle will have new x, y, and heading values. In some embodiments, the vehicle includes, for example, an altitude value to support navigation between different floors of a parking lot. Although the arc path primitive is used to define the target path, other appropriate geometric primitives may also be used. In some embodiments, each path includes a speed parameter. For example, the speed parameter may be used to suggest a travel speed along the path. Initial speed, maximum speed, acceleration, maximum acceleration, and other speed parameters that may be used to control how the vehicle navigates along the path. In some embodiments, the maximum speed is set to a low speed and is used to prevent the vehicle from traveling too fast. For example, a lower maximum speed can be enforced to allow a user to intervene quickly.

[0048] In various embodiments, the determined path objectives are communicated to the vehicle controller for navigation along the path. In some embodiments, the path objectives are first converted to a set of waypoints along the path for use by the vehicle controller. In some embodiments, path planning is run continuously and new paths can be determined while traversing the current path. In some embodiments, the expected path is no longer reachable, for example, the path is blocked and a new path is determined. The path planning at 309 may be performed in a more Figure 3 For example, path planning may be run less frequently than determining the drivable space at 305 and / or the vehicle being controlled to navigate at 311. By determining the path target less frequently than determining the occupancy grid, the path planning process is more efficient and utilizes a more accurate representation of the world.

[0049] In some embodiments, more than one route to a destination is possible, and multiple path targets are provided to the user. For example, a user is shown two or more path targets as options. For example, a user can select a path target for navigating a vehicle to a destination using a GUI, voice commands, etc. For example, a user is shown two paths from a vehicle to a destination. The first path is estimated to take less time, but has more turns and requires frequent gear changes. The second path is smoother, but takes more time. The user can select a path target from these two options. In some embodiments, the user can select a path target and modify the route. For example, the user can adjust the selected route to bypass a specific obstacle, such as a busy intersection.

[0050] At 311, the vehicle automatically navigates to the path target. Using the one or more path targets determined at 309, the vehicle automatically navigates from its current location along the target path(s) to reach the destination. For example, a vehicle controller receives the target path(s) and then implements the vehicle control required to navigate the vehicle along the path. The path target(s) may be received as a set of path points along the path to be navigated. In some embodiments, the vehicle controller converts path primitives such as arcs into path points. In various embodiments, the vehicle is controlled by sending actuator parameters from the vehicle controller to the vehicle actuators. In some embodiments, the vehicle controller is Figure 7 The vehicle controller 707 and the vehicle actuator are Figure 7 Vehicle actuators 713. Using the vehicle actuators, steering, braking, acceleration and / or other operating functions are actuated.

[0051] In some embodiments, the user can adjust the navigation / operation of the vehicle while the vehicle is navigating to a path destination. For example, the user can adjust the navigation by providing input such as "turn left more." As an additional example, the user can increase or decrease the speed of the vehicle while navigating and / or adjusting the steering angle.

[0052] At 313, it is determined whether the vehicle has arrived at the destination received at 301. If the vehicle has arrived at the destination, the process continues to 315. If the vehicle has not yet arrived at the destination, the process loops back to 301 to possibly receive an updated destination and automatically navigate to the (possibly updated) selected destination. In various embodiments, the vehicle has arrived at the destination, but may not be at the exact location of the destination. For example, the selected destination may not or may no longer be a drivable location. As an example, another vehicle may be parked at the selected destination. As another example, the user may be moved to a non-drivable location such as a passenger waiting area. In some cases, if the vehicle has arrived at a location determined to be the closest drivable destination, the vehicle is determined to have arrived at the destination. The closest drivable destination may be based on the path determined at 309. In some embodiments, the closest drivable destination is based on a cost function for calculating a potential path between the current location and the destination. In some embodiments, the difference between the arrival location and the received destination is based on the accuracy of the technology available for determining the location. For example, the vehicle may be parked within the accuracy range of the available global positioning system.

[0053] At 315, the call is completed. In various embodiments, the call function is completed and one or more completion actions are performed. For example, Figure 1 The completion action described in 107 of FIG. In some embodiments, the occupancy grid generated at 307 is saved and / or exported as a completion action. The occupancy grid can be saved locally in the vehicle and / or saved to a remote server. After saving, the grid can be used by the vehicle and / or shared with other vehicles with potential overlapping paths.

[0054] Figure 4 is a flow chart illustrating an embodiment of a process for training and applying a machine learning model to generate a representation of the environment surrounding a vehicle. In some embodiments, Figure 4 The process is used to determine the drivable space for generating an occupancy grid using, at least in part, sensor data. The sensor data used to train and / or apply the trained machine learning model may correspond to image data captured from the vehicle using a camera sensor. In some embodiments, the process is used to Figure 7 In some embodiments, Figure 4 The process is used to Figure 3 The drivable space is determined at 305 and Figure 3 An occupancy grid is generated at 307 locations.

[0055] At 401, training data is prepared. In some embodiments, a training data set is created using sensor data including image data. The sensor data may include still images and / or videos from one or more cameras. Additional sensors such as radar, lidar, ultrasound, etc. may be used to provide relevant sensor data. In various embodiments, the sensor data is paired with corresponding vehicle data to help identify features of the sensor data. For example, changes in position and position data can be used to identify the location of relevant features (such as lane lines, traffic control signals, objects, etc.) in the sensor data. In some embodiments, training data is prepared to train a machine learning model to identify drivable space. The prepared training data may include data for training, validation, and testing. In some embodiments, the format of the data is compatible with the machine learning model used on the deployed deep learning application.

[0056] At 403, a machine learning model is trained. For example, the machine learning model is trained using the data prepared at 401. In some embodiments, the model is a neural network, such as a convolutional neural network (CNN). In various embodiments, the model includes multiple intermediate layers. In some embodiments, the neural network may include multiple layers, including multiple convolutional layers and pooling layers. In some embodiments, the training model is validated using a validation data set created from the received sensor data. In some embodiments, the machine learning model is trained to predict the drivable space based on the image data. For example, the drivable space of the environment around the vehicle can be inferred from the image captured from the camera. In some embodiments, the image data is enhanced with other sensor data such as radar or ultrasonic sensor data to improve accuracy.

[0057] At 405, the trained machine learning model is deployed. For example, the trained machine learning model is installed on the vehicle as an update to the deep learning network. In some embodiments, the deep learning network is such as Figure 7 The trained machine learning model can be installed as an over-the-air update. In some embodiments, the update is a firmware update transmitted using a wireless network such as WiFi or a cellular network. In some cases, the newly trained machine learning model is installed when the vehicle is serviced.

[0058] At 407, sensor data is received. For example, the sensor data is captured from one or more sensors of the vehicle. In some embodiments, the sensor is a visual sensor, such as a sensor for capturing visual data. Figure 7 Visual sensor 701, and / or Figure 7Additional sensors 709. The visual sensors may include image sensors, such as cameras mounted behind the windshield, front and / or side cameras mounted on pillars, rear cameras, and the like. In various embodiments, the sensor data is in a format or converted into a format that is used as input by the machine learning model trained at 403. For example, the sensor data may be raw or processed image data. In some embodiments, the sensor data is data captured from an ultrasonic sensor, a radar, a LiDAR sensor, a microphone, or other appropriate technology. In some embodiments, the sensor data is preprocessed during a preprocessing step using an image preprocessor such as an image preprocessor. For example, the image may be normalized to remove distortion, noise, and the like.

[0059] At 409, the trained machine learning model is applied. For example, the machine learning model trained at 403 is applied to the sensor data received at 407. In some embodiments, the application of the model is performed by, for example, Figure 7 The perception modules such as the perception module 703 are executed using a deep learning network. In various embodiments, by applying a trained machine learning model, a drivable space can be identified and / or predicted. For example, the drivable space in the environment around the vehicle can be inferred. In various embodiments, vehicles, obstacles, lanes, traffic control signals, map features, object distances, speed limits, etc. are identified by applying a machine learning model. The detected features can be used to determine the drivable space. In some embodiments, traffic control and other driving features are used to determine navigation parameters such as speed limits, parking locations, parking areas, facing orientations, etc. For example, stop signs, parking spaces, lane lines, and other traffic control features are detected and used to determine drivable space and driving parameters.

[0060] At 411, an occupancy grid is generated. For example, using the output of the trained machine learning model applied at 409, an occupancy grid for path planning is generated to determine a target path for navigating the vehicle. The occupancy grid may be as described with respect to Figure 3 307 described and / or used Figure 5 process to generate.

[0061] Figure 5 is a flow chart illustrating an embodiment of a process for generating an occupancy grid. For example, using the drivable space determined based on sensor data, an occupancy grid for path planning can be generated. The occupancy grid can be augmented using additional sensor data, metadata from additional sources (such as annotated maps), and / or previously generated occupancy grids. In some embodiments, Figure 5 The process in Figure 1 103, Figure 3 307 and / or Figure 4 411 is executed. In some embodiments, Figure 5 The process of using Figure 7 to be performed by autonomous vehicle systems.

[0062] In some embodiments, an occupancy grid is generated prior to route planning. For example, an occupancy grid is generated and then presented to the user for review, such as via a GUI on a smart phone device or via a display in the vehicle. The user can review the occupancy grid and select a target destination. The user can specify which part of the roadside the user wants the vehicle to stop at. Once selected, the vehicle can navigate to the target destination.

[0063] At 501, a saved occupancy grid is loaded. In some embodiments, a previously generated occupancy grid corresponding to the current position of the vehicle is loaded. By initializing the occupancy grid using a previously saved occupancy grid, the accuracy of the environment around the vehicle can be improved. This optimization significantly improves the accuracy of the initial grid. For example, some objects may be difficult to detect when stationary, but as the vehicle moves closer to the current parking position, these objects will still be detected in the last saved occupancy grid.

[0064] At 503, a drivable space is received. For example, the drivable space is received as an output of a neural network (such as a convolutional neural network). In some embodiments, the drivable space is continuously updated as new sensor data is captured and analyzed. The received drivable space can be divided into grid locations. In some embodiments, in addition to the drivable space, other vision-based measurements are received. For example, objects such as curbs, vehicles, pedestrians, cones, etc. are detected and received.

[0065] At 505, auxiliary data is received. For example, the auxiliary data is used to update and further refine the accuracy of the occupancy grid. The auxiliary data may include data from sensors such as ultrasonic sensors or radars. The auxiliary data may also include occupancy data generated from other vehicles. For example, a mesh network of vehicles may share occupancy data based on the time and location of the occupancy data. As another example, annotated map data may be used to augment the occupancy grid. Map data such as speed limits, lanes, drivable space, traffic patterns, etc. may be loaded via the annotated map data.

[0066] In some embodiments, safety data for overriding or modifying navigation is received as auxiliary data. For example, a collision warning system inputs data at 505 to overlay grid values ​​to account for potential or pending collisions. In some embodiments, the auxiliary data includes data provided from a user. For example, a user may include an image or video to help identify a destination, such as a parking location and / or orientation. The received data may be used to modify the occupancy grid.

[0067] At 507, the occupancy grid is updated. The occupancy grid is updated using the data received at 503 and / or 505. The updated grid may include a value at each grid location corresponding to the probability value of the grid location. These values ​​may be cost values ​​associated with navigating through the grid location. In some embodiments, the values ​​also include a drivable value corresponding to the probability that the grid location is a drivable area. In various embodiments, the updated occupancy grid may be saved and / or exported. For example, the grid data may be uploaded to a remote server or saved locally. For example, when a vehicle is parked, the grid data may be saved and used later to initialize the grid. As another example, the grid data may be shared with related vehicles, such as vehicles having a path or potential path intersection.

[0068] Once the occupancy grid is updated, the process loops back to 503 to continuously update the occupancy grid with newly received data. For example, as the vehicle navigates along a path, new drivable and / or auxiliary data is received to update the occupancy grid. The newly updated grid can be used to refine and / or update the target path for autonomous navigation. For example, a previously empty space may now be blocked. Similarly, a previously blocked space may be open.

[0069] Figure 6 is a flow chart illustrating an embodiment of a process for automatically navigating to a destination object. Figure 6 The process can be used to navigate the vehicle from the current position of the vehicle to the destination target position using the determined planning goal. In various embodiments, navigation is automatically performed by the vehicle controller using the vehicle actuators to modify the steering and speed of the vehicle. Figure 6 The process implements multiple safety checks to increase the safety of vehicle navigation. The safety checks allow navigation to be terminated and / or modified. For example, a virtual heartbeat can be implemented that requires the user to continuously maintain contact with the vehicle to confirm that the user is monitoring the vehicle's progress. In some embodiments, navigation utilizes a path target to navigate the vehicle along an optimal path to a destination target. The path target can be received as a path primitive (such as an arc), a set of points along a selected path, and / or another form of path primitive. In some embodiments, Figure 6 The process in Figure 1 105 and / or Figure 3 In some embodiments, the process uses Figure 7 to be performed by autonomous vehicle systems.

[0070] In some embodiments, Figure 6The process may be used by a user to remotely control a vehicle. For example, a user may control the steering angle, direction, and / or speed of the vehicle via vehicle adjustments to remotely control the vehicle. When a user remotely controls a vehicle, safety checks are continuously performed and safety checks may override and / or modify user controls. For example, the vehicle may be stopped if an object is detected or contact with the remote user is lost.

[0071] At 601, vehicle adjustments are determined. For example, vehicle speed and steering adjustments are determined to keep the vehicle on a path target. In some embodiments, vehicle adjustments are determined by, for example, Figure 7 The vehicle controller 707 of the vehicle controller 708 may determine the vehicle speed. In some embodiments, the vehicle controller determines the distance, speed, orientation, and / or other driving parameters used to control the vehicle. In some embodiments, a maximum vehicle speed is determined and used to limit the vehicle speed. The maximum speed may be enforced to increase the safety of navigation and / or to allow the user sufficient reaction time to terminate the summon function.

[0072] At 603, the vehicle is adjusted to maintain its route along the path target. For example, the vehicle adjustments determined at 601 are implemented. In some embodiments, such as Figure 7 Vehicle actuators such as vehicle actuators 713 of the vehicle actuators 714 implement vehicle adjustments. The vehicle actuators adjust the steering and / or speed of the vehicle. In various embodiments, all adjustments are recorded and can be uploaded to a remote server for later review. For example, in the case of a safety issue, the vehicle drive, path target, destination location, current location, travel speed, and / or other driving parameters can be checked to identify potential areas for improvement.

[0073] The vehicle is operated according to the vehicle adjustments and the operation of the vehicle is monitored at 605. For example, the vehicle operates as indicated by the vehicle adjustments applied at 603. In various embodiments, the operation of the vehicle is monitored to enforce safety, comfort, performance, efficiency, and other operating parameters.

[0074] At 607, a determination is made as to whether an obstacle is detected. In the event that an obstacle is detected, the process continues to 611. In the event that an obstacle is not detected, the process continues to 605 where the operation of the vehicle continues to be monitored. In some embodiments, the obstacle is detected by a collision or object sensor, such as an ultrasonic sensor. In some embodiments, the obstacle may be communicated via a network interface. For example, an obstacle detected by another vehicle may be shared and received. In various embodiments, the detected obstacle may be used to notify other components of the autonomous vehicle system, such as components related to occupancy grid generation, but is also received at the navigation component to allow the vehicle to immediately adjust for the detected obstacle.

[0075] At 609, it is determined whether contact with the user is lost. In the case where contact with the user is lost, the process continues to 611. In the case where contact with the user is not lost, the process continues to 605, and at 605, the operation of the vehicle continues to be monitored. In some embodiments, continuous contact with the user is required to activate automatic navigation. For example, a virtual heartbeat is sent from the user. The heartbeat can be sent from the user's smart phone device or other appropriate device (such as a key card). In some embodiments, as long as the user activates the virtual heartbeat, the virtual heartbeat is received and the contact is not lost. In response to the user no longer sending the virtual heartbeat, the contact with the user is deemed to be lost and the navigation of the vehicle responds appropriately at 611.

[0076] In some embodiments, as long as the user continues to contact the heartbeat switch, button or other user interface device, the virtual heartbeat can be implemented (and continuously sent to keep in touch). Once the user is disconnected from the appropriate user interface element, the virtual heartbeat is no longer transmitted and the contact is lost.

[0077] At 611, navigation is overridden. In response to detecting an obstacle and / or loss of contact with the user, the automatic navigation is overridden. For example, if the vehicle is traveling at a low speed, the vehicle can stop immediately. If the vehicle is traveling at a higher speed, the vehicle will stop safely. Safe parking may require gradual braking and determining a safe parking location, such as a curb or parking space. In various embodiments, overriding navigation may require the user to actively continue automatic navigation. In some embodiments, overriding navigation is resumed once the detected obstacle no longer exists.

[0078] In some embodiments, in the event that an obstacle is detected that overrides navigation, a new path is used to reroute the vehicle to the destination. For example, a new path target is determined that avoids the detected obstacle. In various embodiments, the occupancy grid is updated using the detected obstacle, and the new path target is determined using the updated occupancy grid. Once a feasible new path is determined, navigation can continue when appropriate, for example, once contact is reestablished. In some embodiments, a new path is determined when navigation is overridden. For example, when contact is lost, the path to the destination is reconfirmed. If appropriate, the existing path can be used or a new path can be selected. In various embodiments, if a feasible path cannot be found, the vehicle remains stopped. For example, the vehicle is completely blocked.

[0079] Figure 7is a block diagram illustrating an embodiment of an autonomous vehicle system for automatically navigating a vehicle to a destination target. The autonomous vehicle system includes different components that can be used together to automatically navigate the vehicle to a target geographic location. In the example shown, the autonomous vehicle system includes an on-board component 700, a remote interface component 751, and a navigation server 761. The on-board component 700 is a component that is installed on the vehicle. The remote interface component 751 is one or more remote components that can be used away from the vehicle to automatically navigate the vehicle. For example, the remote interface component 751 includes a smartphone application running on a smartphone device, a remote key fob, a GUI such as a website for controlling the vehicle, and / or another remote interface component. The navigation server 761 is an optional server for facilitating navigation features. The navigation server 761 is a remote server and can be used as a remote cloud server and / or storage device. In some embodiments, Figure 7 Autonomous vehicle systems are used to achieve Figures 1 to 6 process and Figures 8 to 9 Functionality associated with the user interface.

[0080] In the illustrated example, the onboard component 700 is an autonomous vehicle system that includes a visual sensor 701, a perception module 703, a path planning module 705, a vehicle controller 707, additional sensors 709, a safety controller 711, a vehicle actuator 713, and a network interface 715. In various embodiments, different components are communicatively connected. For example, sensor data from the visual sensor 701 and the additional sensor 709 are fed to the perception module 703. The output of the perception module 703 is fed to the path planning module 705. The output of the path planning module 705 and the sensor data from the additional sensor 709 are fed to the vehicle controller 707. In some embodiments, the output of the vehicle controller 707 is a vehicle control command that is fed to the vehicle actuator 713 to control the operation of the vehicle, such as the speed, braking, and / or steering of the vehicle. In some embodiments, the sensor data from the additional sensor 709 is fed to the vehicle actuator 713 to perform additional safety checks. In various embodiments, the safety controller 711 is connected to one or more components, such as the perception module 703, the vehicle controller 707, and / or the vehicle actuator 713, to implement safety checks at each module. For example, the safety controller 711 can receive additional sensor data from the additional sensor 709 to cover the automatic navigation.

[0081] In various embodiments, sensor data, machine learning results, perception module results, path planning results, safety controller results, etc. can be sent to the navigation server 761 via the network interface 715. For example, the sensor data can be transmitted to the navigation server 761 via the network interface 715 to collect training data to improve the performance, comfort and / or safety of the vehicle. In various embodiments, the network interface 715 is used to communicate with the navigation server 761, make calls, send and / or receive text messages, and transmit sensor data based on the operation of the vehicle, as well as other reasons. In some embodiments, the vehicle-mounted component 700 may include more or fewer components as appropriate. For example, in some embodiments, the vehicle-mounted component 700 includes an image preprocessor (not shown) to enhance sensor data. As another example, the image preprocessor can be used to normalize the image or transform the image. In some embodiments, noise, distortion and / or blur are removed or reduced during the preprocessing step. In various embodiments, the image is adjusted or normalized to improve the results of the machine learning analysis. For example, the white balance of the image is adjusted to take into account different lighting operating conditions, such as daylight, sunny, cloudy, dusk, sunrise, sunset and night conditions. As another example, an image captured with a fisheye lens may be distorted and an image preprocessor may be used to transform the image to remove or modify the distortion.In various embodiments, one or more of the in-vehicle components may be distributed to a remote server such as navigation server 761 .

[0082] In some embodiments, the visual sensor 701 includes one or more visual sensors. In various embodiments, the visual sensor 701 can be fixed to the vehicle at different locations of the vehicle, and / or oriented in one or more different directions. For example, the visual sensor 701 can be fixed to the front, side, rear and / or roof of the vehicle in the forward, rearward, lateral and other directions. In some embodiments, the visual sensor 701 is an image sensor such as a high dynamic range camera. For example, a front camera with a high dynamic range captures image data in front of the vehicle. In some embodiments, the vehicle is attached with multiple sensors for capturing data. For example, in some embodiments, eight surround cameras are fixed to the vehicle and provide 360-degree visibility around the vehicle, with a range of up to 250 meters. In some embodiments, the camera sensor includes a wide front camera, a narrow front camera, a rear view camera, a front view side camera and / or a rear view side camera. Various camera sensors are used to capture the environment around the vehicle, and the captured images are provided for deep learning analysis.

[0083] In some embodiments, the visual sensor 701 is not mounted to the vehicle of the vehicle-mounted component 700. For example, the visual sensor 701 can be mounted on an adjacent vehicle and / or fixed to a road or environment and included as a learning system for capturing sensor data. In some embodiments, the visual sensor 701 includes one or more cameras that capture the road surface on which the vehicle is traveling. For example, one or more front and / or pillar cameras capture lane markings of the lane in which the vehicle is traveling. The visual sensor 701 may include an image sensor capable of capturing still images and / or video. The data can be captured over a period of time, such as a sequence of data captured over a period of time.

[0084] In some embodiments, the perception module 703 is used to analyze sensor data to generate a representation of the environment around the vehicle. In some embodiments, the perception module 703 utilizes a trained machine learning network to generate an occupancy grid. The perception module 703 may utilize a deep learning network to take as input sensor data including data from the visual sensor 701 and / or additional sensors 709. The deep learning network of the perception module 703 may be an artificial neural network, such as a convolutional neural network (CNN), which is trained on inputs such as sensor data and its output is provided to the path planning module 705. As an example, the output may include a drivable space of the environment around the vehicle. In some embodiments, the perception module 703 receives at least sensor data as input. Additional inputs may include scene data and / or vehicle specifications describing the environment around the vehicle, such as operating characteristics of the vehicle. The scene data may include scene labels describing the environment around the vehicle, such as rain, wet roads, snow, mud, high-density traffic, highways, cities, school areas, etc. In some embodiments, the perception module 7031 Figure 1 103, Figure 3 305 and / or 309, Figure 4 411 and / or Figure 5 is utilized in the process.

[0085] In some embodiments, the path planning module 705 is a path planning component for selecting an optimal path to navigate the vehicle from one location to another. The path planning component can use an occupancy grid and a cost function to select the optimal route. In some embodiments, the potential path is composed of one or more path primitives (such as arc primitives) that model the operating characteristics of the vehicle. In some embodiments, the path planning module 705 Figure 1 Step 103 and / or Figure 3 In various embodiments, the path planning module 705 runs at a slower frequency or is updated less frequently than other components such as the perception module 703 and the vehicle controller 707.

[0086] In some embodiments, the vehicle controller 707 is used to process the output of the path planning module 705 and convert the selected path into vehicle control operations or commands. In some embodiments, the vehicle controller 707 is used to control the vehicle to automatically navigate to the selected destination target. In various embodiments, the vehicle controller 707 can adjust the speed, acceleration, steering, braking, etc. of the vehicle by transmitting instructions to the vehicle actuator 713. For example, in some embodiments, the vehicle controller 707 is used to control the vehicle to maintain the position of the vehicle on the path from the current location to the selected destination.

[0087] In some embodiments, the vehicle controller 707 is used to control vehicle lighting, such as brake lights, turn signals, headlights, etc. In some embodiments, the vehicle controller 707 is used to control vehicle audio conditions, such as the vehicle's sound system, play audio alerts, enable microphones, enable horns, etc. In some embodiments, the vehicle controller 707 is used to control notification systems including warning systems to notify the driver and / or passengers of driving events, such as potential collisions or approaching a predetermined destination. In some embodiments, the vehicle controller 707 is used to adjust sensors such as the vehicle's sensors 701. For example, the vehicle controller 707 can be used to change the parameters of one or more sensors, such as modifying the orientation, changing the output resolution and / or format type, increasing or decreasing the capture rate, adjusting the captured dynamic range, adjusting the focus of the camera, enabling and / or disabling sensors, etc.

[0088] In some embodiments, in addition to the visual sensor 701, the additional sensor 709 also includes one or more other sensors. In various embodiments, the additional sensor 709 can be fixed to the vehicle, at different locations of the vehicle, and / or oriented in one or more different directions. For example, the additional sensor 709 can be fixed to the front, side, rear and / or roof of the vehicle in the forward, backward, lateral and other directions. In some embodiments, the additional sensor 709 includes radar, audio, LiDAR, inertial, odometer, position and / or ultrasonic sensors, etc. Ultrasonic and / or radar sensors can be used to capture surrounding details. For example, twelve ultrasonic sensors can be fixed to the vehicle to detect hard objects and soft objects. In some embodiments, the forward radar is used to capture data of the surrounding environment. In various embodiments, despite heavy rain, fog, dust and other vehicles, the radar sensor is still able to capture surrounding details. Various sensors are used to capture the environment around the vehicle, and the captured images are provided for deep learning analysis.

[0089] In some embodiments, the additional sensors 709 are not mounted to the vehicle of the onboard component 700. For example, the additional sensors 709 can be mounted on adjacent vehicles and / or fixed to the road or environment and included as an autonomous vehicle system for capturing sensor data. In some embodiments, the additional sensors 709 include one or more non-visual sensors that capture the road surface on which the vehicle is traveling. In some embodiments, the additional sensors 709 include position sensors, such as global positioning system (GPS) sensors for determining vehicle location and / or changes in location.

[0090] In some embodiments, the safety controller 711 is a safety component for implementing safety checks on the on-board component 700. In some embodiments, the safety controller 711 receives sensor input from the visual sensor 701 and / or the additional sensor 709. If an object is detected and / or a collision is likely to occur, the safety controller 711 can notify different components of an immediate pending safety issue. In some embodiments, the safety controller 711 is capable of interrupting and / or enhancing the results of the perception module 703, the vehicle controller 707 and / or the vehicle actuator 713. In some embodiments, the safety controller 711 is used to determine how to respond to a detected safety issue. For example, at low speeds, the vehicle can stop immediately, but at high speeds, the vehicle must safely slow down and stop in a safe position. In various embodiments, the safety controller 711 communicates with the remote interface component 751 to detect whether a real-time connection is established between the on-board component 700 and the user of the remote interface component 751. For example, the safety controller 711 can monitor a virtual heartbeat from the remote interface component 751 and can trigger a safety alarm to terminate or modify automatic navigation if the virtual heartbeat is no longer detected. In some embodiments, Figure 6 The process is at least partially implemented by the security controller 711.

[0091] In some embodiments, the vehicle actuator 713 is used to implement specific operational controls of the vehicle. For example, the vehicle actuator 713 initiates changes in vehicle speed and / or steering. In some embodiments, the vehicle actuator 713 sends operating commands to the drive inverter and / or steering rack. In various embodiments, in the event of a potential collision being detected, the vehicle actuator 713 implements a safety check based on inputs from the additional sensors 709 and / or the safety controller 711. For example, the vehicle actuator 713 stops the vehicle immediately.

[0092] In some embodiments, the network interface 715 is a communication interface for sending and / or receiving data including voice data. In various embodiments, the network interface 715 includes a cellular or wireless interface for interfacing with a remote server, connecting and making voice calls, sending and / or receiving text messages, transmitting sensor data, receiving updates to the perception module (including updated machine learning models), and retrieving environmental conditions (including weather conditions and forecasts, traffic conditions, traffic rules and regulations, etc.). For example, the network interface 715 can be used to receive instructions and / or operating parameters of the sensor 701, the perception module 703, the path planning module 705, the vehicle controller 707, the additional sensor 709, the safety controller 711, and / or the vehicle actuator 713. The machine learning model of the perception module 703 can be updated using the network interface 715. As another example, the network interface 715 can be used to update the firmware of the visual sensor 701 and / or the operating objectives of the path planning module 705, such as weighted costs. As yet another example, the network interface 715 can be used to transmit the occupancy grid data to the navigation server 761 for sharing with other vehicles.

[0093] In some embodiments, remote interface component 751 is one or more remote components that can be used away from the vehicle to automatically navigate the vehicle. For example, remote interface component 751 includes a smart phone application running on a smart phone device, a remote key, a GUI such as a website for controlling the vehicle, and / or another remote interface component. The user can initiate a summon function to automatically navigate the vehicle to a selected destination target specified by a geographic location. For example, the user can let the vehicle find and follow the user. As another example, the user can specify a parking location using remote interface component 751, and the vehicle will automatically navigate to the specified location or the nearest location to safely reach the specified location.

[0094] In some embodiments, the navigation server 761 is an optional remote server and includes a remote storage device. The navigation server 761 can store occupancy grids and / or occupancy data, which can be used to initialize a newly generated occupancy grid later. In some embodiments, the navigation server 761 is used to synchronize occupancy data across different vehicles. For example, occupancy data generated by other vehicles can be used to initiate or update vehicles in an area with new occupancy data. In various embodiments, the navigation server 761 can communicate with the vehicle-mounted component 700 via the network interface 715. In some embodiments, the navigation server 761 can communicate with the remote interface component 751. In some embodiments, one or more components or parts of the vehicle-mounted component 700 are implemented on the navigation server 761. For example, the navigation server 761 can perform processing such as perception processing and / or path planning, and provide the required results to the vehicle-mounted component 700.

[0095] Figure 8 is a diagram illustrating an embodiment of a user interface for automatically navigating a vehicle to a destination object. In some embodiments, Figure 8 User interface for initiating and / or monitoring Figures 1 to 6 In some embodiments, Figure 8 The user interface is the user interface of a smartphone application and / or Figure 7 Remote interface component 751. In some embodiments, Figure 8 The associated functions are Figure 2 203 is initiated to navigate the vehicle to the user's location. For example, a user interacting with a user interface on a smart phone device activates a "find me" action to navigate the vehicle to the location of the user's smart phone device, which is accurately close to the user's location. In the example shown, the user interface 800 includes a user interface component map 801, a dialog window 803, a vehicle locator element 805, a user locator element 809, and an effective summoning area element 807.

[0096] In some embodiments, user interface 800 displays map 801 with the vehicle location indicated at vehicle locator element 805 and the user location indicated at user locator element 809. The area that the vehicle can traverse when automatically navigating to the user is indicated by valid summon area element 807. In various embodiments, valid summon area element 807 is a circle that displays the maximum distance the vehicle is allowed to travel. In some embodiments, valid summon area element 807 takes into account the line of sight from the user to the vehicle, and only areas with line of sight are allowed for automatic navigation.

[0097] In the example shown, map 801 is a satellite map, although alternative maps may be used. In some embodiments, map 801 may be manipulated to view different locations, for example, by panning or zooming map 801. In various embodiments, map 801 includes a three-dimensional view (not shown) to allow a user to select different heights, such as different levels of a parking garage. For example, areas with drivable areas at different heights may be highlighted and may be viewed in an exploded view.

[0098] In some embodiments, the vehicle locator element 805 displays both the position and orientation (or heading) of the vehicle. For example, the direction the vehicle is facing is indicated by the direction pointed to by the arrow of the vehicle locator element 805. The user locator element 809 indicates the position of the user. In some embodiments, the positions of other potential passengers are also displayed, for example, in another color (not shown). In some embodiments, the map 801 is centered around the vehicle locator element 805. In various embodiments, other data can be used as the center of the map 801, such as the original (or starting) position of the vehicle, the current position of the user, the closest reachable position of the current position of the user, etc.

[0099] In the example shown, the dialogue window 803 includes a text description such as "Press and hold to start, or tap the map to select a destination" to inform the user how to activate the summon feature. In some embodiments, the default action is to navigate the vehicle to the user. The default action is activated by selecting a "find me" button that is part of the dialogue window 803. In some embodiments, once the "find me" action is enabled, the selected path is displayed in the user interface ( Figure 8 When the vehicle navigates to the user's location, the vehicle locator element 805 is updated to reflect the new location of the vehicle. Similarly, as the user moves, the user locator element 809 is updated to reflect the new location of the user. In some embodiments, the track shows the changes in the user's location. In various embodiments, the user must keep in touch with the virtual heartbeat button to allow the automatic navigation to continue. Once the user releases the virtual heartbeat button, the automatic navigation will stop. In some embodiments, the heartbeat button is a "find me" button of the dialogue window 803. In some embodiments, separate forward and backward buttons are used as virtual heartbeat buttons to confirm the automatic navigation in the forward and backward directions respectively (not shown).

[0100] In some embodiments, an additional "find" function (not shown) may automatically navigate the vehicle to the location of a selected person, e.g., to pick up a passenger different from the user. In various embodiments, pre-identified locations that may be selected as destination targets may be displayed (not shown). For example, a location may be pre-identified as a valid parking or waiting location for picking up a passenger.

[0101] Fig. 9 is a diagram illustrating an embodiment of a user interface for automatically navigating a vehicle to a destination object. In some embodiments, Fig. 9 User interface for initiating and / or monitoring Figure 1-6 In some embodiments, Fig. 9 The user interface is the user interface of a smartphone application and / or Figure 7 Remote interface component 751. In some embodiments, Fig. 9The associated functions are Figure 2 201 is initiated to navigate the vehicle to a location specified by the user. For example, a user interacting with a user interface on a smart phone device would drop a fixture to specify a destination target for the vehicle to navigate to. In the illustrated example, the user interface 900 includes a user interface component map 901, a dialog window 903, a vehicle locator element 905, a user locator element 909, an active summoning area element 907, and a destination target element 911.

[0102] In some embodiments, user interface 900 displays map 901 with the vehicle location indicated at vehicle locator element 905 and the user location indicated at user locator element 909. The area that the vehicle can traverse when automatically navigating to the user is indicated by valid summon area element 907. In various embodiments, valid summon area element 907 is a circle that displays the maximum distance the vehicle is allowed to travel, and the user can select only a destination target within valid summon area element 907. In some embodiments, valid summon area element 907 takes into account the line of sight from the user to the vehicle, and only areas with line of sight are allowed for automatic navigation.

[0103] In the example shown, map 901 is a satellite map, although alternative maps may be used. In some embodiments, map 901 may be manipulated to view different locations, for example, by panning or zooming map 901. In various embodiments, map 901 includes a three-dimensional view (not shown) to allow a user to select different heights, such as different levels of a parking garage. For example, areas with drivable areas at different heights may be highlighted and may be viewed in an exploded view to select a destination target.

[0104] In some embodiments, vehicle locator element 905 displays both the position and orientation (or heading) of the vehicle. For example, the direction that the vehicle is facing is indicated by the direction pointed to by the arrow of vehicle locator element 905. User locator element 909 indicates the position of the user. In some embodiments, the positions of other potential passengers are also displayed, for example, in another color (not shown). In some embodiments, map 901 is centered on vehicle locator element 905. In various embodiments, other data can be used as the center of map 901, such as the original (or starting) position of the vehicle, the current position of the user, the closest reachable position of the current position of the user, the selected destination target represented by destination target element 911, etc.

[0105] In the example shown, the dialog window 903 includes a text description such as "Press and hold to start, or tap the map to select a destination" to inform the user how to activate the summoning function. The user can select a target destination by selecting a location within the valid summoning area element 907 on the map 901. The destination target element 911 is displayed at the selected valid location. In the example shown, a fixed part icon is used for the destination target element 911. In some embodiments, an icon (not shown) with an orientation (such as an arrow or a vehicle icon) is used as the destination target element 911. The icon of the destination target element 911 can be manipulated to select the final destination orientation. In various embodiments, the selected orientation is verified to confirm that the orientation is valid. For example, a destination orientation facing traffic may not be allowed on a one-way street. In order to clear the selected location associated with the destination target element 911, the user can select the "Clear Fixed Parts" dialog box of the dialog window 903. In order to initiate automatic navigation, the user selects the "Start" button of the dialog window 903. In some embodiments, once automatic navigation is enabled, the selected path will be displayed in the user interface ( Fig. 9 905 is updated to reflect the new location of the vehicle. Similarly, as the user moves, the user locator element 909 is updated to reflect the new location of the user. In some embodiments, the track shows the changes in the user's location. In various embodiments, the user must keep in touch with the virtual heartbeat button to allow the automatic navigation to continue. Once the user releases the virtual heartbeat button, the automatic navigation will stop. In some embodiments, the heartbeat button is the "start" button of the dialogue window 903. In some embodiments, separate forward and backward buttons are used as virtual heartbeat buttons to confirm the automatic navigation in the forward and backward directions respectively (not shown).

[0106] Although the foregoing embodiments have been described in some detail for the purpose of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways to implement the invention. The disclosed embodiments are illustrative rather than restrictive.

Claims

1. A system for navigating a vehicle, comprising: one or more sensors configured to generate sensor data by capturing a real-world environment surrounding the vehicle; a memory coupled to the processor and configured to provide instructions to the processor; The processor is configured to: receiving an identification of a geographic location associated with a target specified by a user remote from the vehicle; receiving the sensor data from the one or more sensors; generating a representation of the real-world environment around the vehicle using a machine learning model trained by training data, the representation of the real-world environment around the vehicle being generated based on a drivable space that is predicted by the machine learning model based on the sensor data, wherein the training data is prepared prior to generating the sensor data, and the drivable space is used to calculate one or more cost metrics associated with a cost of traversing corresponding locations in the real-world environment around the vehicle, wherein the representation of the real-world environment surrounding the vehicle comprises an occupancy grid comprising one or more drivability values ​​at each of a plurality of grid locations forming a respective portion of the occupancy grid, wherein each grid location corresponds to a location in the real-world environment, wherein the drivability at a grid location represents a probability that the corresponding location in the real-world environment is drivable, and the drivability is selected from a range of drivability values; as well as wherein the one or more cost metrics are calculated based on the one or more drivability values; Using the representation of the real-world environment around the vehicle, calculating at least a portion of a path to a target location corresponding to the received geographic location by: determining one or more potential paths to the target location, and selecting a potential path having an optimal cost value as a selected path to the target location, wherein the cost value of each potential path is determined based on cost values ​​of a set of occupancy grids including the corresponding potential path; and providing at least one command for automatically navigating the vehicle based on the path and updated sensor data from at least a portion of the one or more sensors of the vehicle; Wherein when the vehicle is autonomously navigating, the processor is configured to acquire new sensor data and update the one or more drivability values ​​included in the occupancy grid, and wherein based on the updated one or more drivability values, the processor is configured to adjust the path. 2 . The system of claim 1 , wherein the destination specified by the user is a dynamically updated current location of the user's mobile device.

3. The system of any one of claims 1 to 2, wherein the geographic location is based on a global positioning system location detected by a mobile device of the user. The system of claim 1 , wherein the target specified by the user indicates a target orientation of the vehicle. The system of claim 1 , wherein the destination is specified by the user on a map.

6. The system of claim 1, wherein the user selects the target within a limited geographic area that is within a threshold distance from a detected location of a device utilized by the user to specify the target.

7. The system of claim 1, wherein the user selects the destination within a limited geographic area associated with a current location of the vehicle. 8 . The system of claim 1 , wherein the ability for the user to specify the destination is enabled based on determining that a location of the user's device is within a threshold distance from a current location of the vehicle.

9. The system of any one of claims 1 to 2 and 4 to 8, wherein the automatic navigation of the vehicle is exited in response to determining that a heartbeat signal is not received from the user's device.

10. The system of any one of claims 1 to 2 and 4 to 8, wherein the automated navigation of the vehicle is suspended in response to detecting an obstacle using an ultrasonic sensor.

11. The system according to any one of claims 1 to 2 and 4 to 8, wherein the speed of the vehicle is limited to be equal to or less than a specified limit value during the automatic navigation.

12. The system of any one of claims 1 to 2 and 4 to 8, wherein the target location is dynamically updated based on a periodically updated geographic location of the user's device.

13. The system of any one of claims 1 to 2 and 4 to 8, wherein the one or more sensors of the vehicle include a plurality of cameras.

14. The system of any one of claims 1 to 2 and 4 to 8, wherein the representation of the at least a portion of the environment is generated using auxiliary sensor data acquired using one or more of: an ultrasonic sensor or a radar sensor.

15. The system of any one of claims 1 to 2 and 4 to 8, wherein the processor is further configured to receive a specified target time associated with the target specified by the user.

16. The system of any one of claims 1 to 2 and 4 to 8, wherein the representation of the at least a portion of the environment around the vehicle comprises a three-dimensional representation of the portion of the environment.

17. The system of any one of claims 1 to 2 and 4 to 8, wherein determining the target location comprises determining whether the geographic location of the designated target is a valid location to be occupied by the vehicle.

18. The system of any one of claims 1 to 2 and 4 to 8, wherein the target location is determined based on the sensor data from the one or more sensors.

19. A method for navigating a vehicle, comprising: generating sensor data by capturing a real-world environment around the vehicle using one or more sensors; receiving an identification of a geographic location associated with a target specified by a user remote from the vehicle; generating a representation of the real-world environment around the vehicle using the trained machine learning model, the representation of the real-world environment around the vehicle being generated based on a drivable space that was predicted by the machine learning model based on the sensor data, wherein the drivable space is used to calculate one or more cost metrics associated with a cost of traversing a corresponding location in the real-world environment around the vehicle, wherein the representation of the real-world environment surrounding the vehicle comprises an occupancy grid, the occupancy grid comprising one or more drivability values ​​at each of a plurality of grid locations constituting a respective portion of the occupancy grid, wherein each grid location corresponds to a location in the real-world environment, wherein the drivability value at a grid location represents a probability that the corresponding location in the real-world environment is drivable and is selected from a range of drivability values, and wherein the one or more cost metrics are calculated based on the one or more drivability values; Using the representation of the real-world environment around the vehicle, calculating at least a portion of a path to a target location corresponding to the received geographic location by: determining one or more potential paths to the target location, and selecting a potential path having an optimal cost value as a selected path to the target location, wherein the cost value of each potential path is determined based on cost values ​​of a set of occupancy grids including the corresponding potential path; and providing at least one command for automatically navigating the vehicle based on the path and updated sensor data from at least a portion of the one or more sensors of the vehicle; Wherein while the vehicle is autonomously navigating, new sensor data is acquired and the one or more drivability values ​​are updated based on the new sensor data, and wherein the path is adjusted based on the updated one or more drivability values.

20. A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for: receiving an identification of a geographic location associated with a target specified by a user remote from the vehicle; generating a representation of a real-world environment around the vehicle using a neural network that is part of a trained machine learning model, the representation of the real-world environment around the vehicle being generated based on a drivable space that is predicted by the trained machine learning model based on sensor data generated by one or more sensors, wherein the drivable space is used to calculate one or more cost metrics associated with a cost of traversing a corresponding location in the real-world environment around the vehicle, wherein the representation of the real-world environment around the vehicle comprises an occupancy grid, the occupancy grid comprising one or more drivability values ​​at each of a plurality of grid locations that constitute a corresponding portion of the occupancy grid, wherein each grid location corresponds to a location in the real-world environment, wherein the drivability value at a grid location represents a probability that the corresponding location in the real-world environment is drivable and is selected from a range of drivability values, and wherein the one or more cost metrics are calculated based on the one or more drivability values; Using the representation of the real-world environment around the vehicle, calculating at least a portion of a path to a target location corresponding to the received geographic location by: determining one or more potential paths to the target location, and selecting a potential path having an optimal cost value as a selected path to the target location, wherein the cost value of each potential path is determined based on cost values ​​of a set of occupancy grids including the corresponding potential path; and providing at least one command for automatically navigating the vehicle based on the path and updated sensor data from at least a portion of the one or more sensors of the vehicle, Wherein as the vehicle autonomously navigates, new sensor data is acquired and the one or more drivability values ​​included in the occupancy grid are updated, and wherein based on the one or more drivability values, the path is adjusted.

Images