Overall Pathfinding

By combining multi-sensor signals and providing client prompts, autonomous vehicles can identify and approach passengers, solving the problem of difficult pick-up in dense urban environments and improving operational efficiency and passenger convenience.

CN116472213BActive Publication Date: 2026-11-10WAYMO LLC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202180074202.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-10-29
Filing Date
2021-10-27
Publication Date
2026-11-10
Estimated Expiration
2041-10-27

AI Technical Summary

Technical Problem

Autonomous vehicles often struggle to accurately identify passenger locations, especially in dense urban environments, leading to inconvenience in pick-up operations and impacting passenger communication and vehicle operation efficiency.

Method used

By combining multiple sensor signal synthesis, proxy prediction, and situational awareness technologies, the system identifies passenger locations and provides navigation prompts through client devices, adjusting vehicle operations to approach passengers and achieving autonomous pick-up.

Benefits of technology

It improves the accuracy and efficiency of autonomous vehicles in identifying and picking up passengers in dense urban environments, reduces the difficulty of passengers finding their way, and enhances passengers' independence and freedom of movement.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116472213B_ABST
    Figure CN116472213B_ABST
Patent Text Reader

Abstract

This technology employs a holistic approach to passenger pick-up and other wayfinding scenarios. This includes identifying the position of the passenger (606) relative to the vehicle (602) and / or the pick-up location (604). Information synthesis from various sensors (238), an agent behavior prediction model (240), and real-time situational awareness are used to identify with sufficient confidence the likelihood that a passenger will be picked up at a given location at a specific time. The system can provide adaptive navigation by helping passengers understand their distance and direction to the vehicle, for example, using various prompts via an application on the person's device. Rider support tools (804) can be provided, enabling a remote agent to interact with passengers via the person's device, such as using the on-device camera to provide wayfinding support so that the person can locate their vehicle. Rider support can also utilize sensor information from the vehicle when providing wayfinding support.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit to U.S. Provisional Patent Application No. 63 / 107,180, filed October 29, 2020, and U.S. Application No. 17 / 511,634, filed October 27, 2021, the disclosure of which is incorporated herein by reference in its entirety. Background Technology

[0003] Autonomous vehicles, such as those that may not require a human driver in certain driving situations, can be used to assist in transporting passengers or goods from one location to another. Such vehicles can operate in a fully autonomous mode, where passengers can provide initial input such as the pick-up or destination location, and the vehicle maneuvers itself to that location. Autonomous vehicles are equipped with various types of sensors to detect objects in their surroundings. For example, autonomous vehicles may include sonar, radar, cameras, lidar, and other devices that scan, generate, and / or record data about the vehicle's surroundings. This information is important for the vehicle's computing systems to make appropriate driving decisions. However, identifying which person in the vehicle's environment is the passenger to be picked up can be challenging. This can adversely affect how the vehicle operates autonomously as it approaches the pick-up location and may lead to erroneous communication with the passenger or other nearby individuals. Summary of the Invention

[0004] This technology relates to a holistic approach to passenger pick-up and related wayfinding scenarios. This includes robust techniques for identifying passenger positions relative to the vehicle and / or pick-up location. The system can employ signal synthesis from various sensors, agent prediction, and situational awareness to identify with sufficient confidence the likelihood that a passenger to be picked up is at a given location at a specific point in time. Understanding this allows the system to provide adaptive navigation by helping passengers understand their distance and direction to the vehicle, for example, via an app on the person's device (such as a mobile phone or smartwatch), using visual, audible, and / or tactile cues. Rider support tools can be provided, enabling a remote agent to interact with passengers via the person's device, such as using on-device cameras to provide wayfinding support so that the person can locate their vehicle. When providing wayfinding support, rider support can also utilize sensor information from the vehicle.

[0005] According to one aspect, a method is provided for identifying a customer of a vehicle configured to operate in an autonomous driving mode. The method includes receiving, by one or more processors of the vehicle's computing system, perceived information about a set of objects in the vehicle's external environment; determining by the one or more processors that the set of objects in the external environment includes a plurality of pedestrian objects, each pedestrian object corresponding to a person; applying information from at least two different sensor source types to a localization model to obtain the probability that each pedestrian object is a customer; evaluating by the one or more processors the probability that each pedestrian object is a customer; and when determining that the probability of a given pedestrian object among the pedestrian objects indicates that the given pedestrian object may be a customer, performing at least one of the following: (i) sending a notification to the customer's client device regarding a pick-up location where the vehicle is encountered, (ii) adjusting the pick-up location, (iii) causing modification of the driving operation of the vehicle in autonomous driving mode, or (iv) transmitting information about the given pedestrian object to a remote management system.

[0006] In one example, at least two distinct sensor source types are selected from a group consisting of client device sources associated with the customer, vehicle sources onboard the vehicle, and fleet sources obtained from one or more other vehicles in the vehicle's fleet. Vehicle sources may include one or more sensors from the vehicle's perception system, positioning system sources, or communication system sources. The one or more sensors from the perception system may include one or more of lidar, cameras, radar, or acoustic sensors. Information from the at least two distinct sensor source types may include geolocation information from the customer's client device and perception information from one or more sensors from the vehicle's perception system.

[0007] Evaluating the likelihood that each pedestrian is a customer may include refining the list of pedestrians who could be customers. In one example, sending a notification to a customer's client device includes requesting the client device's current or historical geolocation information. In another example, sending a notification to a customer's client device includes providing updated walking directions or turn-by-turn directions. In yet another example, sending a notification to a customer's client device includes requesting the customer to perform a selected action. In yet another example, sending a notification to a customer's client device includes selecting the timing of sending the notification based on the distance of a given pedestrian to the pick-up location or vehicle. Sending a notification to a customer's client device may alternatively or additionally include selecting the level of specificity of the message given to the customer based on the distance of a given pedestrian to the pick-up location or vehicle.

[0008] Customers can be passengers dispatched by vehicles for their journeys. Adjusting the pick-up location can include moving it closer to the customer's current location to reduce the distance the customer will traverse. Transmitting information about a given pedestrian object to the remote management system can include requests for passenger support from the remote management system.

[0009] According to another aspect, the vehicle is configured to operate in an autonomous driving mode. The vehicle includes a perception system comprising one or more sensors configured to receive sensor data associated with objects in the vehicle's external environment. The vehicle also includes a driving system comprising a steering subsystem, an acceleration subsystem, and a deceleration subsystem for controlling the driving of the vehicle; a positioning system configured to determine the vehicle's current position; and a control system comprising one or more processors. The control system is operatively coupled to the driving system, the perception system, and the positioning system. The control system is configured to: receive perceived information from a perception system about a set of objects in the vehicle's external environment; determine that the set of objects in the external environment includes multiple pedestrian objects, each corresponding to a person; apply information from at least two different sensor source types to a positioning model to obtain the probability that each pedestrian object is a customer; evaluate the probability that each pedestrian object is a customer; and, when determining that the probability of a given pedestrian object among the pedestrian objects indicates that the given pedestrian object may be a customer, perform at least one of the following: (i) send a notification to the customer's client device about the pick-up location where the vehicle encounters them, (ii) adjust the pick-up location, (iii) modify the driving operation to be performed by the driving system in autonomous driving mode, or (iv) transmit information about the given pedestrian object to a remote management system. In one example, the vehicle is configured to deliver a package to a customer at the pick-up location.

[0010] Information from at least two different sensor source types may include geographic location information from the customer's client device and perceived information from one or more sensors in the perception system. Assessing the probability that each pedestrian is a customer may include refining the list of pedestrians who could be customers. The control system may also be configured to request updated location information from the customer's client device when it determines that the customer is within a threshold proximity to the vehicle or pick-up location. Furthermore, the control system may be configured to request sensor information from one or more other vehicles regarding their external environment and apply the provided sensor information to the positioning model. Attached Figure Description

[0011] Figure 1A-Figure 1B An example of a self-driving vehicle based on various aspects of this technology is shown.

[0012] Figures 1C-1EExample freight and delivery vehicles configured for use with aspects of this technology are shown.

[0013] Figure 2 The components of an autonomous vehicle according to various aspects of this technology are shown.

[0014] Figures 3A-3B This is a block diagram of a system for example cargo-type vehicles based on various aspects of this technology.

[0015] Figures 4A-4B The following diagram illustrates various aspects of the application of this technology.

[0016] Figure 5 A view of a sensor assembly according to various aspects of the present technology is shown.

[0017] Figures 6A-6B Examples of passenger transport according to various aspects of this technology are shown.

[0018] Figure 7 Another example of passenger pick-up according to various aspects of this technology is shown.

[0019] Figures 8A-8B Example systems based on various aspects of this technology are shown.

[0020] Figure 9 Example methods based on various aspects of this technology are shown. Detailed Implementation

[0021] Navigating in complex environments such as dense urban areas or during peak hours is a challenge for both disabled and non-disabled individuals, and is crucial for passengers or other customers using autonomous vehicle services. The holistic approach based on various aspects of this technology incorporates a broad set of features that work together to provide appropriate assistance to diverse passenger groups in various situations. For example, this allows disabled autonomous vehicle passengers to quickly and easily locate their vehicles during pick-up and drop-off, and to quickly and easily find their destination. Therefore, a variety of wayfinding features can help provide these passengers with greater independence and freedom of movement.

[0022] Autonomous vehicle systems consider safety, applicable traffic regulations, and other constraints when selecting where a vehicle should pull over, which can sometimes result in counterintuitive pick-up and drop-off locations for passengers or other customers. For example, the vehicle might pull over onto a road further away than intended, behind a building, or on the opposite side of the street from the passenger's or planned destination. Because passengers in fully autonomous vehicles cannot communicate with the human driver inside the vehicle or ask them to adjust the parking location, finding the vehicle or desired destination during pick-up or drop-off can be inconvenient or challenging. Furthermore, wayfinding can present particular challenges for passengers with visual and cognitive impairments, but it can also affect passengers with hearing and walking disabilities, passengers with no disabilities, or even customers receiving packages or other deliveries from autonomous vehicles.

[0023] There can be various advanced needs for vehicle navigation during pick-up or destination navigation during drop-off. Examples include the following: Predictability: Passengers want to know where the vehicle will pull over and be aware of them in advance before encountering any potential navigation challenges. Accessibility: Passengers may want the shortest possible walking distance to and from the vehicle (e.g., curbside pick-up / drop-off), although longer walking distances may be beneficial if they significantly improve the ETA (Estimated Time of Arrival) for pick-up and / or drop-off. Simplicity: When navigating, passengers may prefer to navigate through fewer road users and obstacles. Street intersections and large parking lots can be particularly difficult, while curbside may be the easiest and / or safest to manage. For example, prioritizing crossing other road users and obstacles, especially across streets or through unpredictable large parking lots, may be preferred. Flexibility: Passengers may not want a one-size-fits-all approach, so different tools may be necessary to cater to the different needs of different passengers in different situations. Patience: Passengers may want the vehicle to wait long enough during pick-up to allow them to find it, especially if finding their way might take additional time. Assistance: Passengers may want assistance available as a backup if needed, but they may prefer to navigate independently.

[0024] According to one aspect of this technology, the backend service layer can use the passenger's location (e.g., based on information from their mobile phone or smartwatch) to understand the distance and direction from the passenger to the vehicle. This can be coupled with Bluetooth Low Energy (BLE) and / or other sensor identification when the passenger is close enough to the vehicle. Theoretically, displaying the distance to the vehicle as a text field within the application is straightforward. However, the confidence level in the accuracy of the location and orientation is also important. For example, information about the distance to the vehicle could be provided in yards rather than feet to avoid implying higher accuracy than existing methods. Similarly, generating appropriate haptic vibrations in a handheld device is a minor technical challenge, but using them to effectively guide the user may be a larger user design challenge. Linking walking navigation to every turn in existing consumer mapping applications is likely much simpler. However, some mapping applications may not be optimized for autonomous vehicle use cases. For example, if the vehicle needs to update its location after the passenger has moved to another application, there may not be an efficient way to programmatically update the target location in the application's walking navigation.

[0025] Detecting a passenger's proximity to a vehicle or pick-up location with a chosen level of confidence can be significantly complex, as is employing helpful signals at the right time to assist with wayfinding. Therefore, one aspect of this technology includes various techniques for real-time identification of the location of a given passenger or other customer. This can include synthesizing signals from different sensors in the vehicle, the passenger's device, or other equipment; object prediction using behavioral models and sensor information; and situational awareness that can provide contextual cues about the passenger's location. Sufficient confidence in the passenger's real-time location allows the system to provide the passenger with various wayfinding information and assistance.

[0026] Example vehicle system

[0027] Figure 1A A perspective view of an example passenger vehicle 100, such as a minivan or SUV, is shown. Figure 1B A perspective view of another example passenger vehicle 120, such as a sedan or crossover, is shown. Passenger vehicles may include various sensors for obtaining information about the vehicle's external environment. Figures 1C-1D Example tractor-trailer type freight vehicle 140 is shown. Figure 1E Smaller freight vehicles 160 are shown, such as flatbed trucks used for local delivery.

[0028] Vehicles such as those described herein can be configured to operate in one or more different driving modes. For example, in manual driving mode, the driver can directly control acceleration, deceleration, and steering via inputs such as the accelerator pedal, brake pedal, and steering wheel. Vehicles can also operate in one or more autonomous driving modes, including, for example, semi-autonomous or partially autonomous driving modes, in which a person exercises a certain amount of direct or remote control over driving operations, or fully autonomous driving modes, in which the vehicle handles driving operations without direct or remote control by a person. These vehicles can be understood under various names, including, for example, autonomous driving vehicles, self-driving vehicles, etc.

[0029] The National Highway Traffic Safety Administration (NHTSA) and the Society of Automotive Engineers (SAE) each identify different levels of autonomy to indicate the degree to which a vehicle controls driving, although different organizations may classify the levels differently. Furthermore, such classifications can change over time (e.g., be updated).

[0030] As described herein, in semi-autonomous or partially autonomous driving modes, even when the vehicle assists with one or more driving actions (e.g., steering, braking, and / or acceleration to perform lane centering, adaptive cruise control, or emergency braking), the human driver is expected to be situationally aware of the vehicle's surroundings and supervise the assisted driving actions. Here, even if the vehicle can perform all driving tasks in certain situations, the human driver is expected to be responsible for taking control as needed. In contrast, in fully autonomous driving mode, the vehicle's control system performs all driving tasks and monitors the driving environment. This may be limited in certain situations, such as operating within specific service areas or under certain time or environmental restrictions, or it may cover driving in all conditions without limitations. In fully autonomous driving mode, the driver is not expected to take over control of any driving actions.

[0031] Unless otherwise indicated, the architectures, components, systems, and methods described herein can operate in semi-autonomous, partially autonomous, or fully autonomous driving modes. This technology can be used in vehicles configured to operate in autonomous driving modes in all modes, including vehicles transporting passengers or items such as food delivery, parcels, and cargo. While certain aspects of this disclosure are particularly useful in combination with specific types of vehicles, the vehicle can be any type of vehicle, including but not limited to automobiles, vans, motorcycles, freight vehicles, buses, recreational vehicles, emergency vehicles, etc.

[0032] For example, such as Figure 1AAs shown, the vehicle may include a roof-top housing unit (roof pod assembly) 102, which may include one or more lidar sensors and various cameras (e.g., optical or infrared), radar units, acoustic sensors (e.g., microphone or sonar type sensors, ultrasonic sensors, etc.), inertial sensors (e.g., accelerometers, gyroscopes, etc.) or other sensors (e.g., positioning sensors such as GPS sensors). The housing unit 102 may have any number of different configurations, such as dome, cylinder, "cake-top" shape, etc. The housing 104 located at the front of the vehicle 100 and the housings 106a, 106b located on the driver's side and passenger side of the vehicle may each incorporate lidar, radar, cameras, acoustic and / or other sensors. For example, housing 106a may be located along the quarter panel of the vehicle in front of the driver's side door. As shown, the passenger vehicle 100 also includes housings 108a, 108b located towards the rear roof portion of the vehicle for, for example, radar units, lidar and / or cameras. Additional lidar, radar units, and / or cameras (not shown) may be positioned elsewhere along vehicle 100. For example, arrow 110 indicates that sensor units (not shown) may be positioned along the rear of vehicle 100, such as on or adjacent to the bumper. Depending on the vehicle type and sensor housing configuration, acoustic sensors may be placed in any or all of these housings around the vehicle.

[0033] In this example, arrow 114 indicates that the roof pod 102 includes a base attached to the roof of the vehicle. Arrow 116 indicates that the roof pod 102 also includes an upper portion (e.g., having a dome, cylinder, or cake-top shape) that extends above the base portion. Each of the base and upper portion can house different sensor units configured to acquire information about objects and conditions in the environment surrounding the vehicle. The roof pod 102 and other sensor housings can also extend along... Figure 1B The vehicle 120 is placed. For example, each sensor unit may include one or more sensors of the types described above, such as lidar, radar, camera (e.g., optical or infrared), acoustic (e.g., passive microphone or active acoustic emission sonar type sensor), inertial (e.g., accelerometer, gyroscope, etc.) or other sensors (e.g., positioning sensors such as GPS sensors).

[0034] Figures 1C-1DThe example freight vehicle 140 is a tractor-trailer truck. The truck may include, for example, a single trailer, a double trailer, or a triple trailer, or it may be another medium or heavy-duty truck such as those in commercial weight classes 4 to 8. As shown, the truck includes a tractor unit 142 and a single cargo unit or trailer 144. Depending on the type of goods or other cargo to be transported, the trailer 144 may be fully enclosed, open (such as a flatbed), or partially open. In this example, the tractor unit 142 includes an engine and steering system (not shown) and a cab 146 for the driver and any passengers.

[0035] like Figure 1D As shown in the side view, trailer 144 includes a bolted connection 148 referred to as a pivot bolt, and a ground support device 150 for use when the trailer is detached from the tractor unit. The pivot bolt 148 is typically formed as a solid steel axle configured to pivotally attach to the tractor unit 142. Specifically, the pivot bolt 148 is attached to a trailer coupling 152, referred to as a half-trailer wheel, mounted at the rear of the cab. For dual or triple tractor-trailers, the second and / or third trailers can be simply bolted to the leading tractor unit. Alternatively, each trailer can have its own pivot bolt. In this case, at least the first and second trailers may include half-trailer wheel structures arranged to couple to the next trailer.

[0036] As shown in the figure, the tractor unit may have one or more sensor units 154 and 156 positioned along it. For example, sensor unit 154 may be positioned on the roof or top of the cab. Sensor unit 154 may be a sensor suite having an elongated central member 154a and side members 154b, the elongated central member 154a having one or more types of sensors (e.g., cameras and / or radar modules) located along the elongated central member 154a, and the side members 154b may include other types of sensors (e.g., short-range lidar modules capable of detecting objects within 10-25 meters of the vehicle and / or long-range lidar modules capable of detecting objects beyond 15-20 meters and up to 100-250 meters). Sensor unit 156 may be positioned on the left and / or right side of the cab. Sensor units may also be located along other areas of the cab, such as along the front bumper or hood area, at the rear of the cab, adjacent to the semi-trailer wheel, under the chassis, etc. Trailer 154 may also have one or more sensor units 158 placed along it, such as along one or two side panels, front, rear, top and / or underframe of trailer 154.

[0037] Figure 1E Perspective 160 shows an example flatbed truck or other vehicle that could be used for local delivery (e.g., groceries, meals, mail, or other parcels). Here, with Figure 1CIn contrast to the top housing unit 154 shown in -D, the truck 160 may have a pair of sensor assemblies in a housing 162 placed on either side of the vehicle.

[0038] as Figure 1A-Figure 1B Similar to the sensor units of passenger vehicles, each sensor unit of a freight vehicle may include one or more sensors, such as lidar, radar, cameras (e.g., optical or infrared), acoustic (e.g., microphone or sonar type sensors), inertial (e.g., accelerometers, gyroscopes, etc.) or other sensors, such as location-based (e.g., GPS) positioning sensors, force elements or pressure sensors (e.g., piezoelectric or mechanical), and inertial (e.g., accelerometers, gyroscopes, etc.).

[0039] like Figure 2 As shown in system diagram 200, a vehicle such as vehicle 100, 120 or 160 may have one or more computing devices, such as computing device 202 containing one or more processors 204, memory 206 and other components typically found in general-purpose computing devices.

[0040] Memory 206 stores information accessible by processor 204, including instructions 208 and data 210 that can be executed or otherwise used by processor 204. Memory 206 can be any type of memory capable of storing processor-accessible information, including computing devices or computer-readable media, or other media that store data readable by means of electronic devices, such as hard disk drives, memory cards, ROM, RAM, DVDs or other optical discs, and other writable and read-only memories. Systems and methods can include different combinations of the foregoing, whereby different portions of instructions and data are stored on different types of media.

[0041] Instructions 208 can be any set of instructions that are executed directly by the processor (such as machine code) or indirectly (such as scripts). For example, instructions can be stored as computing device code on a computing device-readable medium. In this regard, the terms "instruction" and "program" are used interchangeably herein. Instructions can be stored in an object code format for direct processing by the processor, or in any other computer device language that includes a collection of independent source code modules that are interpreted on demand or pre-compiled, or in scripts. The functions, methods, and routines of instructions are explained in more detail below.

[0042] Processor 204 can retrieve, store, or modify data 210 according to instruction 208. For example, although the claimed subject matter is not limited to any particular data structure, the data can be stored in a computing device register, as a table with multiple different fields and records in a relational database, in an XML document, or in a flat file. The data can also be formatted in any computing device-readable format.

[0043] One or more processors 204 can be any conventional processor, such as a commercial CPU, GPU, or TPU. Alternatively, one or more processors can include dedicated devices, such as ASICs or other hardware-based processors. Although Figure 2 While the processor, memory, and other elements of computing device 202 are shown functionally within the same frame, those skilled in the art will understand that a processor, computing device, or memory may actually include multiple processors, computing devices, or memories that may or may not be housed in the same physical enclosure. For example, memory may be a hard disk drive or other storage medium located in a different enclosure than processor 204. Therefore, references to processors or computing devices will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0044] The computing device 202 may include all components typically used in conjunction with a computing device, such as the processor and memory described above, and a user interface 212 having one or more user inputs 214 (e.g., one or more of buttons, mice, keyboards, touchscreens, gesture inputs, and / or microphones), various electronic displays 216 (e.g., monitors with screens or any other electronic devices operable to display information), and speakers 218 that provide information to passengers or other persons in the autonomous vehicle as needed. For example, the electronic display 216 may be located in the cabin of the autonomous vehicle 100, 120, or 160 and may be used by the computing device 202 to provide information to passengers or delivery personnel within the autonomous vehicle 100, 120, or 160.

[0045] The computing device 202 may also include a communication system 220 with one or more wireless connections to facilitate communication with other computing devices, such as client computing devices and server computing devices described in detail below. Wireless network connections may include short-range communication protocols such as Bluetooth, Bluetooth Low Energy (LE), cellular connections, and various configurations and protocols including the Internet, World Wide Web, intranet, virtual private network, wide area network, local area network, private networks using communication protocols proprietary to one or more companies, Ethernet, WiFi, and HTTP, as well as various combinations thereof.

[0046] The computing device 202 may be part of the autonomous control system of the autonomous vehicle 100, 120, or 160 and may be able to communicate with various components of the vehicle to control the vehicle in an autonomous driving mode. For example, the computing device 202 may communicate with various systems of the autonomous vehicle 100, 120, or 160, such as the deceleration system 222, the acceleration system 224, the steering system 226, the signal system 228, the planning system 230 (also known as the planning / trajectory module), the routing system 232, the positioning system 234 (for determining the vehicle's position, such as its pose, for example, its position and orientation along the road or the pitch, yaw, and roll of the vehicle chassis relative to the coordinate system), the perception system 236 with one or more sensors 238, the behavior modeling system 240 (also known as the behavior module), and the power system 242, in order to control the movement, speed, etc. of the autonomous vehicle 100, 120, or 160 in an autonomous driving mode according to instructions 208 from the memory 206.

[0047] As an example, computing device 202 can interact with deceleration system 222 and acceleration system 224 to control the speed of the vehicle. Similarly, computing device 202 can use steering system 226 to control the direction of autonomous vehicle 100, 120, or 160. For example, if autonomous vehicle 100, 120, or 160 is configured for use on a road, such as a car or truck, steering system 226 may include components that control the angle of wheels 244 to turn the vehicle. Some or all wheels / tires 244 are coupled to the deceleration, acceleration, and / or steering systems. Computing device 202 may be able to receive information about tire pressure, balance, and other factors that may affect driving in autonomous mode. Computing device 202 may also use signaling system 228 to signal the vehicle's intentions to other drivers or vehicles, such as by illuminating turn signals or brake lights when necessary.

[0048] Computing device 202 may use routing system 232 to generate a route to a destination using map information. Computing device 202 may also use planning system 230 to generate short-term trajectories that allow vehicles to follow the routes generated by the routing system. In this regard, planning system 230 and / or routing system 232 may store detailed map information, such as pre-stored, highly detailed maps identifying road networks, including roads, lane lines, intersections, pedestrian crossings, speed limits, traffic signals, buildings, signs, real-time traffic information (updated when received from a remote computing device such as the computing device or other computing devices discussed below), parking spots, the shape and height of vegetation, or other such objects and information.

[0049] Map information can be configured as a road map. A road map may include multiple graphical nodes and edges representing features such as pedestrian crossings, traffic lights, road signs, roads, or lane segments, which together constitute the road network of the map information. Each edge is defined by a starting graphical node with a specific geographic location (e.g., latitude, longitude, altitude), an ending graphical node with a specific geographic location (e.g., latitude, longitude, altitude), and a direction. This direction may refer to the direction that the autonomous vehicle 100 must move to follow the edge (i.e., the direction of traffic flow). Graphical nodes can be located at fixed or variable distances. For example, the spacing between graphical nodes can range from a few centimeters to a few meters and may correspond to the speed limits of the road where the graphical node is located. At this point, a greater speed may correspond to a greater distance between graphical nodes. Edges may represent driving along the same lane or changing lanes. Each node and edge may have a unique identifier, such as the latitude and longitude location of the node or the starting and ending locations of the edge or node. In addition to nodes and edges, the map can identify additional information, such as the type of maneuver required at different edges and which lanes are drivable.

[0050] The routing system 232 can use the aforementioned map information to determine a route from the current location (e.g., the location of the current node) to the destination. A cost-based analysis can be used to generate the route, attempting to select the lowest-cost route to the destination. Costs can be evaluated in any number of ways, such as time to reach the destination, distance traveled (each edge can be associated with a cost for crossing that edge), type of maneuver required, convenience for passengers or vehicles, etc. Each route can include a list of multiple nodes and edges that a vehicle can use to reach the destination. The route can be recalculated periodically as the vehicle travels to the destination.

[0051] The map information used for routing can be the same as or different from the map used for route planning. For example, map information for route planning requires not only information about individual lanes but also the nature of lane boundaries (e.g., solid white lines, dashed white lines, solid yellow lines, etc.) to determine where lane changes are permitted. However, unlike maps used for route planning, map information for routing does not need to include additional details such as the locations of pedestrian crossings, traffic lights, stop signs, etc., although some of this information may be useful for routing purposes. For example, between a route with numerous intersections with traffic controls (such as stop signs or traffic lights) and a route with few or no traffic controls, the latter route may have a lower cost (e.g., because it is faster) and is therefore preferred.

[0052] The computing device 202 may use the positioning system 234 to determine the vehicle's relative or absolute location on a map or the Earth. For example, the positioning system 234 may include a GPS receiver to determine the device's latitude, longitude, and / or altitude. Other positioning systems, such as laser-based positioning systems, inertial-assisted GPS, or camera-based positioning, may also be used to identify the vehicle's location. The vehicle's location may include absolute geographic location information such as latitude, longitude, and altitude, and the location of nodes or edges on a road map, as well as relative location information such as the location of other vehicles immediately surrounding the vehicle. Relative location information can typically be determined with less noise than absolute geographic location information.

[0053] The positioning system 234 may also include other devices, such as accelerometers, gyroscopes, or other orientation / velocity detection devices, that communicate with the computing device 202 to determine the vehicle's orientation and velocity, or changes thereof. By way of example only, an accelerometer device may determine its pitch, yaw, or roll (or changes thereof) relative to the direction of gravity or a plane perpendicular to the direction of gravity. The device may also track increases or decreases in velocity and the direction of such changes. The provision of position and orientation data by the devices described herein may be automatically provided to the computing device 110, other computing devices, and combinations thereof.

[0054] The perception system 236 includes one or more components (sensors 238) for detecting objects outside the vehicle, such as other road users (vehicles, pedestrians, cyclists, etc.), obstacles in the road, traffic signals, signs, trees, buildings, etc. For example, the sensors 238 of the perception system 236 may include lidar, sonar, radar, cameras, microphones (e.g., in an acoustic array arranged, for example, along the roof pod 102), pressure or inertial sensors, strain gauges, and / or any other detection devices that generate and / or record data that can be processed by the computing device of the computing device 202. In the case where the vehicle is a passenger vehicle such as a minivan 100 or a car 120, the vehicle may include lidar, cameras, and / or other sensors mounted on or near the roof, fenders, bumpers, or other convenient locations, such as... Figure 1A-Figure 1B As shown

[0055] The sensors of the perception system 236 can detect objects and their characteristics in the vehicle's external environment, such as their position relative to the road, orientation (pose), size, shape, type (e.g., vehicle, pedestrian, cyclist, etc.), direction, speed relative to the vehicle, and environmental conditions around the vehicle. The perception system 236 may also include other sensors within the vehicle to detect objects and conditions within the vehicle, such as those in the passenger compartment or storage compartment (e.g., luggage compartment). For example, such sensors can detect one or more people, pets, packages, etc., as well as conditions inside and / or outside the vehicle, such as temperature and humidity. Furthermore, the sensors 238 of the perception system 236 can measure the rotational rate of the wheels 244, the braking amount or type of the deceleration system 222, and other factors related to the vehicle's own equipment.

[0056] Raw data obtained from sensors (e.g., camera images, LiDAR point cloud data, radar return signals, acoustic information, etc.) can be processed by perception system 236 and / or periodically or continuously sent to computing device 202 for further processing as perception system 236 generates data. Computing device 202 can use positioning system 234 to determine the vehicle's location and use perception system 236 to detect objects and road information (e.g., signs or road markings) and respond to objects and road information as needed to safely reach the location, such as through adjustments made by planner / track module 230 (including in-operation adjustments to handle sensor occlusion and other issues).

[0057] In some cases, object characteristics can be input into the behavior prediction system software module of the behavior modeling system 240. This module uses various behavior models based on object type to output one or more predicted future behaviors of the detected object. The object trajectory can be a list of position and orientation or direction (e.g., pose) and other predicted characteristics, such as velocity, acceleration or deceleration, and the rate of change of acceleration or deceleration. In other cases, characteristics obtained from the perception system 236 can be input into one or more detection system software modules, such as a traffic light detection system software module configured to detect the state of a known traffic signal, a construction zone detection system software module configured to detect construction zones from sensor data generated by one or more sensors of a vehicle, and an emergency vehicle detection system configured to detect emergency vehicles from sensor data generated by sensors of a vehicle. Each of these detection system software modules can use various models to output the probability of a construction zone or the probability that the object is an emergency vehicle.

[0058] The detected objects, predicted future behaviors, various possibilities from the detection system software module, map information identifying the vehicle's environment, location information identifying the vehicle's position and orientation from the positioning system 234, the vehicle's destination location or node, and feedback from various other systems of the vehicle can be input into the planning system software module of the planner system 230. The planner system 230 can use this input to generate a trajectory that the vehicle will follow over a short period of time in the future, based on a route generated by the routing module of the routing system 232. At this point, the trajectory can define specific characteristics such as acceleration, deceleration, velocity, and direction to allow the vehicle to follow a route toward its destination. The control system software module of the computing device 202 can be configured to control the movement of the vehicle, for example, by controlling the vehicle's braking, acceleration, and steering, in order to follow the trajectory.

[0059] The computing device 202 can control the vehicle in one or more autonomous driving modes by controlling various components. For example, as an example, the computing device 202 can autonomously navigate the vehicle to a destination location using detailed map information and data from the planning system 230. The computing device 202 can use the positioning system 234 to determine the vehicle's position and use the perception system 236 to detect objects and respond to them when necessary to safely reach the location. Similarly, to do this, the computing device 202 and / or the planning system 230 can generate trajectories and make the vehicle follow these trajectories, for example, by accelerating the vehicle (e.g., by supplying fuel or other energy to the engine or power system 242 by the acceleration system 224), decelerating (e.g., by reducing the fuel supplied to the engine or power system 242, changing gears, and / or by applying braking by the deceleration system 222), changing direction (e.g., by turning the front or rear wheels of the autonomous vehicle 100, 120, or 160 by the steering system 226), and signaling such changes using the signal system 228 (e.g., by illuminating a turn signal). Therefore, the acceleration system 224 and the deceleration system 222 can be part of the transmission system, which includes various components between the vehicle's engine and wheels. Similarly, by controlling these systems, the computing device 202 can also control the vehicle's transmission system to autonomously maneuver the vehicle.

[0060] Figure 3A It shows a vehicle (e.g., Figures 1C-1D A block diagram 300 of various components and systems of a vehicle (140). As an example, the vehicle may be a truck, agricultural equipment, or construction equipment, configured to operate in one or more autonomous operating modes. As shown in block diagram 300, the vehicle includes a control system comprising one or more computing devices, such as a computing device 302, which includes one or more processors 304, a memory 306, and similar or equivalent components as described above. Figure 2Other components discussed in parts 202, 204, and 206. For example, data may include map-related information (e.g., road maps).

[0061] The control system can constitute the electronic control unit (ECU) of the tractor unit of a freight vehicle. Like instruction 208, instruction 308 can be any set of instructions executed directly (such as machine code) or indirectly (such as a script) by a processor. Similarly, data 310 can be retrieved, stored, or modified by one or more processors 304 according to instruction 308. Here, as described above, the system may include a user interface 312 with one or more user inputs, various electronic displays, and speakers.

[0062] In one example, computing device 302 can form an autonomous driving computing system integrated into vehicle 140. Similar to the above regarding... Figure 2 As discussed, the autonomous driving computing system of block diagram 300 may be able to communicate with various components of the vehicle to perform route planning and driving operations. Communication system 314 may provide one or more wireless connections in the manner described above for communication system 220. Additionally or alternatively, the communication system may include the vehicle's internal communication bus (e.g., a controller area network (CAN) bus or a FlexRay bus).

[0063] For example, computing device 302 can communicate with various systems of the vehicle, such as the driving system including deceleration system 316, acceleration system 318, steering system 320, signal system 322, and positioning system 324, each of which can be configured as described above. Figure 2 It operates as discussed.

[0064] The computing device 302 is also operatively coupled to a perception system 326 having one or more sensor components 328, and a powertrain 330. Some or all wheels / tires 332 are coupled to the driving system, and the computing device 302 may be able to receive information about tire pressure, balance, rotational speed, and other factors that may affect autonomous driving. Like the computing device 202, the computing device 302 can control the vehicle's direction and speed by controlling various components. As an example, the computing device 302 can autonomously navigate the vehicle to its destination using data from map information, a routing system 334, a planner system 336, and / or a behavior system 338. For example, similar to the above... Figure 2 In this manner, computing device 302 may employ the planner / track module of planning system 336, combined with positioning system 324, sensor component 328 of sensing system 326, and routing system 334, to detect objects and respond to them when necessary, in order to safely reach the location.

[0065] Similar to sensing system 236, sensing system 326 also includes one or more sensors or other components, such as those described above for detecting objects outside the vehicle, objects or conditions inside the vehicle, and / or the operation of certain vehicle equipment (such as wheels and driving systems). Each sensor assembly 328 may include one or more sensors. In one example, a pair of sensor assemblies 328 may be arranged as sensor towers integrated into side mirrors on trucks, agricultural equipment, construction equipment, etc. In another example, sensor assemblies 328 may also be positioned at different locations on the tractor unit 142 or trailer 144, as described above. Figures 1C-1D The computing device 302 can communicate with sensor assemblies located on both the tractor unit 142 and the trailer 144. Each assembly may have one or more sensors of the types described above.

[0066] Figure 3A The image also shows a coupling system 340 for connecting the tractor unit and the trailer. The coupling system 340 may include one or more power and / or pneumatic connections (not shown), and a semi-trailer wheel 342 at the tractor unit for mechanical connection to a central bolt on the trailer.

[0067] Figure 3B It shows things like Figures 1C-1D A block diagram 350 of a system for an example trailer 144 is provided. As shown in this example, the system includes a trailer ECU 352 with one or more computing devices, such as a computing device containing one or more processors 354, a memory 356, and other components typically found in general-purpose computing devices. The memory 356 stores information accessible by the one or more processors 354, including instructions 358 and data 360 that can be executed or otherwise used by the processors 354. Figure 2 and Figure 3A The description of the processor, memory, instructions, and data is applicable to Figure 3B These components.

[0068] In this example, the trailer ECU 352 is configured to receive information and control signals from the tractor unit, as well as information from various trailer components. The onboard processor 354 of the ECU 352 can communicate with various systems of the trailer, including a reduction system 362, a signal system 364, and a positioning system 366. The ECU 352 can also be operatively coupled to a sensing system 368, which has one or more sensors arranged in a sensor assembly 370 for detecting objects in the trailer's driving environment. The ECU 352 can also be operatively coupled to a power system 372 (e.g., battery power) to provide power to local components. Some or all of the trailer's wheels / tires 374 can be coupled to the reduction system 362, and the processor 354 may be able to receive information about tire pressure, balance, wheel speed, and other factors that may affect autonomous driving, and relay this information to the tractor unit's processing system. The deceleration system 362, signal system 364, positioning system 366, sensing system 368, power system 372, and wheels / tires 374 can be configured in accordance with the above-mentioned... Figure 2 and Figure 3A Operate in the manner described above.

[0069] The trailer also includes a landing gear set 376 and a coupling system 378. When detached from the tractor unit, the ground support device can provide a supporting structure for the trailer. It can be used as... Figure 3A Coupling system 378, a part of coupling system 340, provides a connection between the trailer and the tractor unit. Therefore, coupling system 378 may include a connection portion 380 (e.g., for communication, power, and / or pneumatic linking with the tractor unit). In this example, the coupling system also includes a pivot bolt 382 configured for connection to the semi-trailer wheel of the tractor unit.

[0070] Example Implementation

[0071] Figure 4AThis is example pick-up scenario 400, showing pick-up location 402 adjacent to road segment 404. In this example, when a passenger to be picked up by an autonomous vehicle schedules a trip, pick-up location 402 can be arranged, for example, using an application on their mobile phone or other device (e.g., tablet PC, smartwatch, etc.). The application can be used to schedule trips using an autonomous vehicle ride hailing service. The person scheduling the trip can select pick-up location 402, which can be adjusted by the passenger or the service before the actual pick-up time. The service can select which vehicle in the fleet to pick up the passenger. In this scenario, a vehicle (not shown) approaches pick-up location 402. Upon approach, the vehicle can detect the presence of different objects on or adjacent to the road, including stop sign 406 and pedestrian crossing sign 408, as well as different people 4101, 4102, and 4103. In this example, one or more sensors of a vehicle perception system, such as lidar and / or camera sensors, are used for detection.

[0072] However, in most cases, simply recognizing the presence of persons 4101-4103 may not be sufficient to determine which person is the passenger to be picked up by the assigned vehicle. For example, just because person 4102 is physically closest to the designated pick-up location 402 does not mean that person is the passenger. Or, when person 4101 may be identified as having a walking path away from the intersection in a crosswalk, this does not necessarily mean they are not the passenger. Here, person 4101 may be walking around to the other side of the vehicle to enter it. And although person 4103 may appear to be facing away from the designated pick-up location 402, this does not necessarily mean they are moving in a direction away from the pick-up location 402. Instead, they may be ensuring there is no oncoming traffic before crossing the street.

[0073] In some situations, location information from a passenger's client device can be used to identify their current location. This often helps to provide a rough estimate. Location information can be based on various sources (e.g., GPS, Wi-Fi or Bluetooth signals, cellular triangulation, etc.). Each source can have a finite granularity of approximately a few meters or tens of meters. The granularity of location information can be adversely affected by high obstacles such as skyscrapers or large trees with dense canopies, reflected radio frequency (RF) signals, signal attenuation, etc. The result may be a rough location covering some uncertain area. For example, Figure 4BView 420 shows that each person 4101-4103 has an uncertain region 4221, 4222, or 4223 based on location information from their respective client devices, as indicated by dashed, dotted, or dotted lines. As shown, the uncertain regions may overlap. Therefore, if the system is using location information obtained from the client devices and evaluating that location information relative to images of different people 410 in the scene, it may not be accurate enough to allow the vehicle to determine which person is the passenger.

[0074] To address these issues and achieve a high confidence level (e.g., 85%-90% or higher) that a person in the scenario is a passenger, the system can utilize additional information from various sensor sources. Sensor sources may include client device sources associated with the passenger, vehicle-mounted sources including sensing and positioning systems, fleet sources from other vehicles in the vehicle convoy, and / or other sources. Depending on the specific circumstances, information from each source may be more or less helpful. The approach may include using a trained passenger localization model to determine the probability that a particular person is a passenger. Other information, such as visit history, may also be employed. Here, if a passenger has previously visited a location several times and exhibits a pattern in their waiting locations, such information, combined with other described factors, can increase the confidence in locating / identifying the passenger.

[0075] As an example, client device sensor information may include location data from a GPS source, such as a GPS device integrated into a mobile phone, smartwatch, or other wearable device, tablet PC, etc. If location sharing is enabled, the system may perform background polling of GPS coordinates, such as every few seconds (e.g., every 1-10 seconds). If sharing permission is only available when the application is open, GPS coordinates may only be received when the application is in the foreground (actively in use). If sharing permission is not enabled, GPS coordinates will not be shared. Other client device sensor information may include Received Signal Strength Information (RSSI) associated with WiFi signals or even peer-to-peer connections such as Bluetooth. In the case of WiFi signals, knowing which WiFi access points, repeaters, or other devices can be detected by the client device and the relative signal strength can help determine how close the client device is to these devices. Similarly, knowing which Bluetooth devices are nearby can help pinpoint the location of the client device, especially if these Bluetooth devices are static (e.g., a wireless printer). As an example, the system can use all such information to create a heatmap or overlay map comparing GPS coordinate information, RSSI information, and the geographic locations of other "visible" WiFi or Bluetooth devices to identify one or more possible locations where the client device may be. In one example, the vehicle may receive updated location or other information from the passenger's client device every few seconds (e.g., every 1-5 seconds, more or less). The refresh rate used to receive updated location information can vary depending on the pick-up status. For example, when the time is within 1-2 minutes of the scheduled pick-up, the refresh rate may increase to once or more per second, while if the time is more than 5 minutes from the scheduled pick-up, the refresh rate may be at a lower (e.g., default) rate of every 2-10 seconds (more or less).

[0076] In addition to GPS information received from the passenger's client device, the vehicle can also use information from its onboard positioning system (e.g., Figure 2Location information (234) in the context of GPS. For example, a vehicle's GPS receiver or other location system can be used to determine the vehicle's latitude, longitude, and / or altitude. This information can be utilized, for example, by comparing satellites detected by the occupant's device with those detected by the vehicle's GPS system. Having a common set (or subset) of satellites can indicate that the vehicle and the occupant are located on the same side of the street. Alternatively, conversely, satellites in the set that do not overlap may indicate that the occupant is on the opposite side of the street, or that some intermediate obstacle is causing the two GPS modules to detect different satellites. Additionally or alternatively, if the occupant has multiple devices, each including a GPS or other location module, information from each device can be used to refine the occupant's location estimate (e.g., by averaging the locations from each module, or sorting the locations and selecting the most probable location). Thus, this information can be used to narrow the list of possible pedestrian candidates from the list of possible candidates detected by the vehicle's perception system.

[0077] As described above, the vehicle source may include devices from a sensing system and / or a positioning system. For example, as per [reference to...] Figure 1A Vehicle 100 and Figure 2 As discussed in the diagram, the sensors in the perception system can include radar units, lidar units, cameras, and acoustic sensors. Radar, lidar, and / or cameras can each have different fields of view (e.g., a given camera can have a field of view of 45°–135° along the front, side, or rear of the vehicle, while...). Figure 1A The rotating lidar unit on the upper part of the roof pod 102 can have a 360° field of view and an operating range (e.g., a short-range lidar configured to detect objects within 10-25 meters of the vehicle, a long-range lidar configured to detect objects beyond 15-20 meters and up to 100-250 meters, a lidar configured to detect objects 100-200 meters or more away, and a high-resolution camera configured to detect pedestrians and signs at a distance greater than 500 meters).

[0078] like Figure 5 As shown in the bottom view 500, the roof pod assembly ( Figure 1A102 in the diagram may include one or more sets of microphones 502. Specifically, the bottom plane of the roof pod assembly may include a location for a receptacle where microphones or other transducers 502 can be placed. In one example, at least three or four locations are used in the acoustic system, each with a pair of transducers (e.g., for local noise cancellation). Microphones or other transducers may be positioned along other parts of the vehicle. For directional microphones or other transducers, the system may have higher confidence in directionality and / or distance based on relative orientation (e.g., parallel versus perpendicular). Transducers may be configured to detect sounds within the typical range of human hearing, such as from 20 Hz to 20 kHz. Transducers may also be configured to detect sounds outside this typical range, such as frequencies below 20 Hz (e.g., as low as 1-15 Hz) or ultrasonic frequencies above 20 kHz (e.g., between 20 kHz and 30 kHz).

[0079] This acoustic subsystem may be able to provide some directional information about sounds detected by nearby vehicles. For example, if a passenger's device plays a tone or short melody, the subsystem can detect the direction (or orientation) from which it is coming relative to the vehicle's position and orientation (pose) along the road. Alternatively, the passenger can speak to a remote assistance person via an app, such as to get assistance locating the vehicle or pick-up location. As an example, the subsystem may be able to detect the orientation of a sound coming from within a 120-140° range relative to the front of the vehicle. The distance from the sound source to the vehicle can also be estimated, for example, if the passenger's device communicates the volume of the sound to the vehicle (e.g., the sound level emitted when playing a tone or melody, or the sound level detected when a person speaks to remote assistance). Here, knowing the loudness of the sound at the passenger's device and the loudness of the detected sound, the vehicle's system can estimate the approximate distance (e.g., within 25 meters, between 25 and 5 meters, over 50 meters, etc.) based on sound distance attenuation calculations. Knowing the estimated orientation and distance of the microphone or other transducers, the system can use this to focus on one or more pedestrian objects detected by other onboard sensors. In another example, a passenger's phone could transmit or ping a signal via an acoustic pulse. The vehicle's acoustic subsystem detects this signal and the subsequent echo return. This can be used to indicate to the passenger how close they are to the vehicle based on the time it takes for the acoustic pulse and echo to occur. This information can be conveyed to the passenger via tactile information such as one or more tactile patterns. The pitch of the acoustic pulse and echo can be ultrasonic, outside the range of human hearing. The length or pattern of the "pitch" can be used to inform the passenger how far / near they are from the vehicle (or pick-up location). Combined with images detected by one or more cameras on the vehicle, such a responsive acoustic pulse / echo method can help further clarify and guide the passenger toward the vehicle.

[0080] Communication systems (e.g., Figure 2The 200 in the model can also provide additional sources based on Bluetooth, WiFi, and / or cellular information detected by the vehicle. For example, Bluetooth and WiFi RF signals may have an effective range of several meters to hundreds of meters, while cellular signals may have an effective range of approximately several kilometers or more from the base station or gNodeB. In the case of Bluetooth or other point-to-point types of connections, once a connection has been established between the passenger's device and the vehicle, signal strength information (from the passenger's device and / or the vehicle) can help refine the location estimate for the passenger. Alternatively or additionally, the presence of other Bluetooth or WiFi-enabled devices detected by the passenger's device can be shared with the vehicle. Here, the vehicle's system can compare a list of such devices with its own list of detected devices. If there are common devices on each list, this can further narrow down the possible locations of the passenger's devices, especially when the devices on the list have known locations. A similar evaluation can be performed on cellular signals, for example, by comparing which base stations / gNodeBs (or which device sectors) are on the list. These analyses can be performed with or without considering signal strength information.

[0081] Combined, the information obtained or detected in this way can be used to identify specific individuals in the environment who are likely to be passengers, or to exclude other pedestrians from the list of potential passengers. For example, Figure 6A Example scenario 600 is shown, where vehicle 602 has arrived at pick-up location 604. In this example, three objects in the nearby environment have been identified as pedestrians, namely pedestrian objects 6061, 6062, and 6063. Based on location information provided by the passenger's client device, the dashed area 608 indicates the area where the passenger is likely to be, for example, with a 95% probability. However, this area encompasses all three pedestrian objects detected by the vehicle's perception system. As described above, using one or more pieces of information from the vehicle system, the vehicle can determine that the passenger is within the area defined by the straight dashed line 610. This allows the system to exclude pedestrian object 6061 as the passenger to be picked up. Other information can be used to further refine the list of possible passengers or to identify a single passenger candidate with the highest confidence value. This can include triangulation-type methods, where each different information source can be used to narrow down the area where the passenger might be. Note that in different situations, it may not be necessary to know exactly which pedestrian is the passenger, only that the passenger is one of a specific set of pedestrians. For example, the direction of travel is still helpful, even when the passenger is part of a group of people moving in the same direction. Furthermore, regardless of exactly which person in the group is the passenger, the vehicle can perform a modified pull-to-the-side parking maneuver, or even a proximityun lock, once it knows the passenger's location.

[0082] One aspect of this technology could use Bluetooth signals for navigation. For example, given the vehicle's determined location and beacon signals emitted from the vehicle (including RSSI information) detected by the passenger's device, trilateration can be used to calculate the current location of the client device. This can provide an estimate of the passenger's location of approximately 1-10 meters. Another aspect of this technology could use ultra-wideband RF signals for location refinement. The ultra-wideband approach does not rely on RSSI, which provides a coarse understanding of the location (approximately a few meters or more). Instead, systems using ultra-wideband signals use time-of-flight information to calculate distance. This method can be accurate to a coarse 50 centimeters when the passenger is within a range of 1-50 meters from the vehicle. While using the ultra-wideband approach may require relatively high power, it can be used sparingly at specific times to obtain more accurate location information. As an example, the system could use a hierarchical approach that uses the GPS location from the passenger's device for an overall location estimate, and then employs ultra-wideband and / or Bluetooth technology once the passenger is estimated to be within a certain threshold distance of the vehicle or pick-up location.

[0083] For example, another aspect of techniques for identifying the correct occupant with a high degree of certainty (e.g., above 80-90% certainty) involves historical information and behavior prediction. As mentioned above, pedestrian object characteristics can be input into the behavior prediction module of a vehicle behavior modeling system. The behavior modeling system uses various behavior models based on object type to output one or more predicted future behaviors of detected objects. Regarding pedestrians, object trajectories can be a list of the position and orientation or direction (e.g., pose) of a person walking in a scene, along with other predictive characteristics such as speed. According to one example, input can include pedestrian behavior in response to communication with their phone or other devices. For example, if a vehicle sends audio or video communication to a occupant's phone, and the pedestrian gestures or adjusts their gaze / gaze toward their device, this could be a contributing input in determining whether they are an occupant. Thus, in one aspect, the system can use behavior prediction in the method to model head-gazing behavior for occupancy. For example, a pedestrian who actively turns their head back and forth multiple times at a corner can be considered to want to cross or participate in traffic behavior. Here, knowing the pedestrian's pose, direction, and "key points" indicating posture and movement can also be useful indicators. Modeling such "I'm riding here" behavior can serve as an indicator of the rider (combined with other signals). Furthermore, if the rider is riding with a family member who requires devices such as child seats, booster seats, strollers, etc., then pedestrians carrying these objects are potential candidates.

[0084] Historical information and behavior prediction

[0085] The more robust the historical information of a given pedestrian object, the more likely the prediction of short-term behavior (e.g., within the next 5-10 seconds) will be accurate. This can include evaluating all nearby detected pedestrian objects, as well as using contextual information about riders. In one example, knowing the location of the rider's first hailing trip can be used to determine if someone is walking from that direction. For example, this could include using street addresses or geographic coordinates to determine where a person will emerge from a building. Vehicles can maintain historical information on a rolling (e.g., first-in, first-out) basis for 10, 30, 60 seconds, or longer. Alternatively, it can maintain historical information for objects within a certain threshold distance of the vehicle (e.g., 100-250 meters). How long or how far historical information can be maintained may vary for different object types. Therefore, because pedestrian objects tend to move slower than bicycle or vehicle objects, the system can maintain historical information for such slower-moving objects for a longer period during the pickup scenario.

[0086] When a trip is dispatched from a location near the pick-up location, such as a shopping mall, office building, or apartment building within 2-3 blocks or 250 meters of the pick-up location, the system can determine the passenger's expected walking path from the dispatched location to the selected pick-up location. This expected walking path can be derived based on road maps or other highly detailed map information indicating the locations of sidewalks, crosswalks, streetlights and street signs, and static obstacles (e.g., mailboxes or covered bus stops). It can also be based on whether a building has different entrances / exits (e.g., exits). Multiple paths can also be created depending on the number of possible routes that can be traversed (e.g., taking a shortcut through a grassy area instead of taking a longer route around a corner). Each expected walking path can be ranked from most likely to least likely. This ranking can be refined based on the current scene conditions. For example, a delivery truck might be parked in front of a crosswalk, so different walking paths avoiding the crosswalk might be ranked higher. Here, the vehicle's processing system can compare the expected walking paths with historical information about detected pedestrian objects.

[0087] When considering entrances and exits, the system can evaluate not only the raw location information but also the most likely entrances and exits and the trajectory from those points. Additionally or alternatively, the system can temporarily maintain historical GPS or other location information from the passenger's client device, such as location information equivalent to the past 30 seconds or 2 minutes. This location information can be limited to data that provides a rough approximation of the location (e.g., equivalent to the first standard deviation of the location data).

[0088] In one scenario, this information can be requested from the passenger's device only when it is determined that the passenger is within a certain range of the pick-up location (e.g., a distance in meters or a time in seconds or minutes), or when there is a sufficient probability that the vehicle detects from its sensor data that the passenger is in that area. In another scenario, the information can be requested from the passenger's device sometime before the planned pick-up time, for example, 1-3 minutes before the pick-up. Here, for vehicles that have already arrived at or are about to arrive at the pick-up location (e.g., within the next 30-60 seconds), receiving location-related information from the passenger's device before the passenger arrives can be used to focus the vehicle's perception system sensors on a specific area, allowing for rapid identification of the scene once the passenger arrives.

[0089] There's no need to receive "stale" information, such as location data from 10 minutes ago, as this is useless for predicting a passenger's walking path or behavior. Therefore, the vehicle can request the passenger's device to send only location information or other relevant data from a specific time frame (e.g., the last 20-30 seconds). The obtained information can be compared to the expected walking path or at least a segment of the path, such as a 25-100 meter segment, as this increases the confidence level of the person's actual path towards the pick-up location.

[0090] Figure 6B It shows the basis Figure 6A View 620 shows the scenario. Here, pedestrian object 6061 has been excluded as a candidate for a passenger and is therefore displayed in gray to indicate exclusion. The remaining candidates are pedestrian objects 6062 and 6063. In this example, pedestrian object 6062 has a walking path 6222, and pedestrian object 6063 has a walking path 6223 (the walking path of pedestrian object 6061 is no longer relevant). These predicted walking paths can be based on historical information and possible entrances / exits from building 624 for each pedestrian object.

[0091] As part of behavior prediction, a vehicle's sensors can detect the current gaze direction of potential pedestrian candidates. This may be impossible if someone is wearing sunglasses or eyeglasses, which could affect the camera sensors' ability to identify gaze direction. However, if the gaze direction is determined to be towards the vehicle or pick-up location when the person is within a certain threshold distance of the vehicle or location (e.g., within 10-15 meters, or more or less), the system can increase the confidence score that the pedestrian is likely a passenger.

[0092] Additional information about a scheduled ride can provide further contextual clues about which person is likely to be the rider. For example, when scheduling a ride, the person may have indicated that they want a specific type of vehicle (e.g., a minivan), or that they need to access a luggage compartment or other storage compartment. Therefore, if a pedestrian in the candidate list is detected pushing a stroller, walking with a service animal, or using a walker, the system can determine that this person is more likely to be the rider than other candidates in the list.

[0093] Based on available historical information and real-time sensor data from the vehicle environment, the system can predict how each pedestrian object of interest will move in the next few seconds (e.g., the next 2-10 seconds). Therefore, as... Figure 6B As shown, pedestrian object 6062 has a predicted path 6262, and pedestrian object 6063 has a predicted path 6263. As illustrated in this example, the predicted path 6263 of pedestrian object 6063 is moving towards the pick-up location 604, while the predicted path 6262 of pedestrian object 6062 is moving along the sidewalk opposite the pick-up location 604. Furthermore, someone walking in the middle of the street away from the curb, heading straight towards a vehicle, rather than near a crosswalk, would be a good example of how behavior prediction might guide a person to a vehicle door.

[0094] In addition to using information from the passenger's client device and vehicle, information from other vehicles in the convoy and / or from other sources can be employed in the methods described above. For example, another convoy vehicle may have recently, such as within the last 10-30 seconds, driven along the same road segment. Information from that vehicle's perception system can be used to enhance the information acquired by the vehicle assigned to pick up the passenger. This can include identifying distinct pedestrian objects (objects in the environment determined by the system to have a high probability of being pedestrians) and determining whether there are any pedestrian objects detected by other vehicles that may be near or approaching the pick-up location. This can be used to assist in situations where the pick-up vehicle is obscured by one or more pedestrian objects, for example, because a truck is parked on the side of the road, or because someone else is standing in front of the passenger at an intersection. Here, another convoy vehicle traveling in the same or opposite direction or crossing the same area (e.g., driving along an intersection) may have an unobstructed view from its lidar sensors, cameras, or other sensors. Sensor information from other vehicles can be used to supplement data from the vehicle's own sensor systems, for example, to effectively increase the field of view of the vehicle's sensors.

[0095] Figure 7Example 700 of this type of scenario is shown. As shown, vehicle 702 has arrived at a north-facing pick-up location 704 on the first road segment. Vehicle 702's perception system detects multiple pedestrian objects 7061...7063. In this example, there are other people in the scene who are not identified as pedestrian objects, for example, because their sensors are obstructed from vehicle 704. Here, another vehicle 708 in the convoy has just crossed the intersection westward. In this case, the other vehicle 708 detects pedestrian objects 7101 and 7102 as it passes by. Information about these pedestrian objects can be transmitted to vehicle 702. This can include the last known location of these objects, historical information about their travel paths, any predicted paths generated by vehicle 708, etc. Vehicle 702 can then use this information to update its list of pedestrian objects as potential pick-ups, or otherwise make any necessary pick-up-related adjustments.

[0096] Loading location adjustment and example scenarios

[0097] In some situations, the passenger, vehicle, or ride-hailing service may adjust the pick-up location before the passenger enters the vehicle. For example, this might be because another vehicle is parked or blocking the initially selected pick-up location. Alternatively, picking up the passenger on the other side of the street, around a corner, or elsewhere might be more convenient to reduce walking distance to the vehicle and avoid traffic congestion. When adjusting the pick-up location, the system can update the passenger's expected walking path to the adjusted location. Here, the timing of the adjustment can be compared with the behavior of any pedestrian objects in the potential passenger candidate list to see if any particular pedestrian object is detected changing their walking direction or otherwise moving towards the adjusted pick-up location.

[0098] There may be specific situations where identifying a particular pedestrian object detected by a vehicle's perception system as a passenger is particularly challenging. As an example, sidewalking for a location near a parking lot might involve detecting many different people walking towards the parking lot on their way to their own vehicles. Here, such as when a vehicle detects that more than a certain number of vehicles are parked in the parking lot (e.g., more than 3-5 vehicles, or when the parking lot is estimated to be more than 15-30% full), the vehicle or ride-hailing service may initiate adjustments to the pick-up location.

[0099] In other scenarios, multiple passengers may arrive at the pick-up location together (e.g., a family with children or business colleagues heading to a client meeting) or separately (e.g., friends meeting up and heading to dinner, a concert, or a sporting event). In the former case, the vehicle can use knowledge of how many passengers are scheduled for a multi-passenger trip, such as by identifying a group of pedestrians moving in the same direction at roughly the same pace. For example, if there is a trip with four people, and the vehicle's perception system identifies a set of four pedestrians moving toward the pick-up location, it can assign a very high probability (e.g., 90-95%) to that group as the set of passengers to be picked up. In such a case, the probability can be assessed as very high even if no location-related information is received from the passengers' client devices, fleet sources, or other sources. In the latter case, the various pedestrians detected in the scene can be evaluated to see if some of them are converging toward the pick-up location, the vehicle's current location, or some other location. For example, friends might meet at some intermediate location before heading to the pick-up point or the vehicle's current location, such as at the corner of an intersection. Determining whether multiple pedestrian objects are converging can be used to adjust the probability that each of these objects is one of the passengers to be picked up, and the vehicle's system can update its list of possible passenger candidates accordingly.

[0100] Training pedestrian object model

[0101] Machine learning models for pedestrian objects, which may include neural networks, can be trained on “baseline real” data associated with the pick-up location, occupant geolocation data, and perceptual information from the vehicle (or convoy) and / or the vehicle’s location itself. This can be done for different types of pick-up scenarios, both with other nearby objects (e.g., cars or other vehicles on the road, pedestrians or cyclists on the sidewalk, etc.) and without other nearby objects. Training can be based on collected real-world data (e.g., labeled according to road environment, intersection type, presence of sidewalks, signage such as stop or yield signs, etc.). Thus, one or more models can be developed and used in real time by the autonomous vehicle, developed and used after factual (e.g., post-processing) evaluation by the backend system, or both. This can include occupant or customer localization models for detected pedestrian objects, which are used to refine the estimated location of occupants in a specific geographic area. As an example, the model structure can be a deep network, where the exact structure and parameters can be searched by automated machine learning, for example, using a model of the Neural Architecture Search (NAS) type. Based on this, in-vehicle systems (e.g., planner systems, routing systems, and / or behavior systems of autonomous driving systems) can utilize models to evaluate detected pedestrian objects and handle pick-up (and drop-off) scenarios accordingly. This can include creating the probability of each such object being a passenger to be picked up across all detected or identified pedestrian objects.

[0102] In one approach, the system can use historical location data (e.g., the last 20-30 seconds of GPS location coordinates) for each pedestrian object in a Bayesian computation (e.g., maximum a posteriori or MAP) to create probabilities across all pedestrian objects. As mentioned above, location information from the passenger's client device may be inaccurate, so historical information (e.g., collected every 1-5 seconds) may be a series of noisy location data points. To train the pedestrian object (localization) model to achieve a high probability that a given pedestrian object is a passenger to be picked up, in addition to the passenger's geographic location information, the training input can include different sets of baseline real data. These other sets can include a selected pick-up location or any adjusted pick-up location, the vehicle's position at one or more time points before or at the time of pick-up, perception information about the area surrounding the pick-up location (e.g., lidar, cameras, radar, acoustic and / or other sensors from the vehicle's perception system), supplemental sensor-related information from fleet vehicles, other sensor or location information associated with the pick-up location, etc. Training data can also include highly detailed map information such as road map data, which can indicate other objects in the scene that may affect pedestrian movement (e.g., mailboxes on sidewalks, or stop lights, walk lights, or pedestrian beacons that can regulate when pedestrians can cross intersections or crosswalks). The model can be trained using any combination of such information.

[0103] Once trained, the model can be used by vehicles or backend systems in various ways. For example, a vehicle can use the model to predict how each pedestrian object of interest will move over the next few seconds (e.g., the next 2-10 seconds), such as by generating a set of one or more predicted paths for each pedestrian object of interest. Here, given a path observed by the perception system, the vehicle can determine the trajectory of the pedestrian object. This can include generating a probability value or composite score (e.g., a percentage value or a ranking of scores such as 0-4) of the person associated with that pedestrian object moving toward the car. Objects with scores below a threshold can be discarded from the set of possible occupants. Objects with scores above a threshold can be focused, for example by requesting additional information from the occupant's device, to help further narrow down the list of possible occupants.

[0104] Vehicles or backend systems can also use this model to perform back-evaluation, matching geographic location information to the most probable path. Here, knowing that a given pedestrian was in a specific location at a certain time, the system can work backward to determine where the object came from. If the process indicates that the object came from the location of the requested trip, then the given pedestrian is a passenger. Back-evaluation methods can also be used to identify how accurate the geographic location information received from the passenger's device is. Highly accurate information (e.g., within 1-2 meters or less) can provide a higher confidence level regarding the passenger's location than less accurate information (e.g., approximately 10-20 meters). Furthermore, given the past of a pedestrian, how likely they were to enter a vehicle, conditioned or unconditioned by GPS signal availability, can be useful, for example, when someone is standing near the curb in the middle of a street without a crosswalk.

[0105] User experience

[0106] Based on the above methods, the system can achieve a high level of confidence that a given person in a scenario is a passenger. As an example, this confidence level can indicate that the probability of a specific person being a passenger scheduled for a ride (or a group of people to be picked up) exceeds 85%-90%. Using this enhanced confidence, the backend system of a vehicle or autonomous vehicle ride-hailing service can modify the ride schedule or provide useful information to passengers or others nearby, which can improve the overall user experience.

[0107] Proximity unlock

[0108] Proximity unlocking is a feature that reassures the intended passenger. Knowing which person is the passenger, the vehicle can wait to unlock one or more doors until that person is within a threshold proximity of the vehicle. As an example, depending on the vehicle type and pick-up location, a door can be unlocked once the person is within 1-2 meters of the vehicle. Alternatively, the vehicle can unlock a door or open a van's sliding door when it detects that a person is reaching for a door handle or taking some other action indicating they are preparing to enter the vehicle (e.g., putting down a package or folding a portable bicycle). Here, close-in camera (or lidar, radar, or acoustic) sensors from the perception system can be positioned to detect objects within 2-3 meters of the vehicle.

[0109] When car doors are unlocked or opened can be adjusted by the vehicle based on how many people are around it. For example, the more crowded it is, the closer people must be to open the doors. There may also be considerations based on the time of day, allowing the vehicle to vary the timing of door openings. Late-night pick-ups might involve opening the doors when someone reaches for the handle, while morning pick-ups might involve opening the doors a few seconds before the person arrives. Knowing that the vehicle is locked and unaccessible before passengers arrive provides them with a sense of security and assurance, as they are unlikely to enter a car that may have been unlocked for some time.

[0110] In one scenario, the system can communicate with the occupant (e.g., via an app on the occupant's mobile device) to press a button on the device before unlocking. In another scenario, the vehicle doesn't need to unlock all doors. Instead, knowing which occupant is and which side of the vehicle they are approaching allows the vehicle to select which side's doors to unlock / open. In yet another scenario, the vehicle can use interior and / or exterior lights in conjunction with door unlocking to provide illumination. In yet another scenario, weather conditions might be a factor the vehicle uses to decide when to open doors. For example, instead of opening the sliding door 10 seconds before occupants arrive, if the ambient temperature is below a certain threshold (e.g., below 50°F), the door can remain closed until occupants are 2 seconds apart to keep the interior warm (e.g., around 70°F). Conversely, in heavy rain, the door can open earlier, e.g., 4-6 seconds before occupants arrive, so they can enter the vehicle without getting too wet. Furthermore, knowing which side the occupant is approaching from can be beneficial so the vehicle can communicate to them in a targeted manner that the door they are about to open is unlocked or is being unlocked. In addition to ensuring safety, occupants' awareness of unlock timing and lock status is meaningful for conveying security. In some situations, it may not be appropriate to widely broadcast the unlock status, or displaying a lockable sign on the opposite side of the vehicle to prevent unauthorized occupants from entering may be helpful.

[0111] Pickup location adjustment / early boarding

[0112] Knowing which pedestrian is associated with the passenger allows the vehicle to determine whether to adjust the selected pick-up location. For example, if the vehicle arrives at the scene and determines that the passenger is already at the pick-up location, assuming there is space for the vehicle to pull over, no adjustment is needed. However, when the passenger is not fully at the pick-up location, the vehicle can determine that performing the previous boarding procedure might be better by pulling over 50 meters away from the pick-up location and notifying the passenger of the adjusted pick-up location. This avoids the passenger having to unnecessarily walk an extra 50 meters.

[0113] Another aspect of this approach might involve altering the amount of time a vehicle will wait at the pick-up location. Here, knowing the passenger is not close to the location (e.g., more than 400 meters from the pick-up point), the vehicle could change from a default waiting time (e.g., 2 minutes or more, or less) to a longer waiting time (e.g., 5 minutes or more). Alternatively, the vehicle could be moved to a different location so it can wait for a longer period, for example, when the initial pick-up point is in an area with restricted idling.

[0114] In another scenario, if there are pedestrians nearby and the vehicle determines that they are not passengers and are simply crossing the street, the planner can allow the vehicle to remain stationary while waiting for the pedestrians to continue walking. However, using the technology described above, the vehicle can determine that the pedestrians are actually passengers and are simply walking to the other side of the vehicle to enter. Here, the vehicle will be able to use this information to switch to an appropriate boarding state, such as performing a proximity unlock operation.

[0115] gesture

[0116] There may be crowded situations, such as during peak hours, at the end of a concert, or at the end of a sporting event, where many other people are very close to the passenger. In this type of situation, knowing whether the passenger is part of a group of people helps to gauge how close they are to the pick-up point or vehicle. Here, if the vehicle detects a group of pedestrians all within a few meters of each other, the vehicle can request the passenger's gesture to signal to the vehicle, which will allow the vehicle to confirm that they are part of the group. This request can be made via an application on the client device. Similarly, the vehicle can choose different methods to respond to the passenger's signal indicating their location, such as by having the client device play a melody or other sound, or by turning the flashlight on and off.

[0117] Communicating with passengers and others

[0118] Having a very high confidence in knowing which detected pedestrian is a passenger (or multiple passengers) allows the vehicle to provide enhanced information to the passenger, which can speed up boarding, reduce waiting frustration, or otherwise make the ride more pleasant. As mentioned above, if the passenger is not exactly at the pick-up location, the vehicle can change the pick-up protocol to encourage faster boarding by modifying the pick-up location to be closer to their current location. Or, if the passenger is not nearby, the vehicle can bypass the traffic congestion if waiting would require blocking traffic. In either case, the vehicle can send the passenger one or more notifications about the ride, such as adjustments to the pick-up location, the vehicle's current or planned location, and the best way to enter the vehicle.

[0119] Vehicles can tailor user messages based on whether the system determines whether a passenger is nearby or not. For example, if a passenger is far from the pick-up location (e.g., more than 300 meters or more than 3 minutes away), the vehicle can send a message indicating when the passenger is expected to arrive or how long the vehicle might be able to wait at the selected pick-up location, or offer options to reschedule the vehicle to a location it can then determine to proceed or bypass (as opposed to unnecessarily waiting at that location while congested). However, if the passenger is nearby (e.g., within 25 meters), the message can indicate that the vehicle is very close and ready to let them on.

[0120] Furthermore, the timing of sending messages to passengers can vary depending on their distance from the vehicle. Here, the closer the person, the more frequently the messages may become. Alternatively, the closer the person, the more specific the messages become. This could include providing more granular directions for walking or each turn, identifying landmarks or objects on the road (e.g., “Turn left at the barber shop sign” or “Watch out for a damaged sidewalk within 20 feet”), details about the vehicle (e.g., “Blue van”), etc. For example, when passengers are 100 meters away, the vehicle could communicate that they should be there within 2 minutes after walking two more blocks in a straight line. And when passengers are within 20 meters or in the same block as the vehicle, the vehicle could choose to send messages indicating the exact location where they can be found (e.g., “Wait behind the red SUV about 10 meters to your left”). Alternatively, the vehicle could send audio, images, or short videos showing the vehicle or its position relative to the passengers, or otherwise instruct passengers on how to reach the vehicle (this could involve haptic feedback on the passenger's device to indicate directions). Therefore, although a vehicle may have a default set of messages and / or a default timing for sending those messages to passengers, the vehicle can modify the messages and / or timing based on the real-time conditions at the pick-up location.

[0121] Therefore, vehicles can communicate with passengers to modify their walking paths, such as avoiding obstacles or crowds. The application user interface on the passenger's device can present information (e.g., images or videos) from the passenger's perspective (what the person walking towards the pick-up location will see) or from the vehicle's perspective, the latter helping to give the passenger different perspectives to assist them in reaching the vehicle. The user interface can include captions, highlights, or audible information to explain what is being displayed. In one example, instead of pins on a map, there could be an audio description ("I parked near the big bushes or next to the brown sedan"). Alternatively, this can also be done via chat, such as using a chatbot with the passenger, which uses data between the vehicle and client data to inform the passenger of their location. In another example, the system can suggest a starting point for the direction of travel, or suggest one or more intermediate points in the direction of travel. These can include location / waypoints for people to pass through to more easily reach the pick-up location.

[0122] The vehicle can also communicate with people other than passengers. Sometimes communicating with others nearby can be helpful. For example, if someone who is not a passenger approaches the vehicle, the vehicle can provide them with an audible and / or visual notification that the vehicle is waiting for someone else. Here, the vehicle can remain locked until the designated passenger arrives. Communication with non-passengers can include presenting them with purposeful information. Simultaneously, the vehicle can also provide information to the actual passenger, such as boarding information displayed towards the passenger but not towards others around the vehicle.

[0123] As mentioned above, in some situations, multiple passengers may travel together but arrive separately. Here, the vehicle (or the ride-sharing system at the back end) can provide personalized information to each passenger. This could include walking separately or the direction of each turn. It could also include suggesting locations for some or all of the group to meet before arriving at the pick-up point. For example, the vehicle could suggest that each passenger meet at a specific corner of an intersection so that they can all walk together along the street and enter from one side of the vehicle.

[0124] Passenger support

[0125] Sometimes, passengers may require more assistance than the vehicle can provide. For example, ride-sharing services can provide passenger support via their backend systems. Here, human support personnel may be able to access image or video feeds from the vehicle to help locate the passenger, or even access images taken by the passenger's device. Images from the client device can be used to help locate the passenger, such as by comparing the acquired images with stored images of a specific area. This can be done by the backend system with or without passenger support personnel.

[0126] Passenger support can involve a support member instructing a passenger to perform a gesture (e.g., “Please raise your left hand”) or move in a certain way (e.g., “Please face left and look at the blue van”). Passengers can request passenger support via an app on their device. Alternatively, if the vehicle determines that passengers are having difficulty locating the vehicle or pick-up location, it can send a passenger support request to the backend system.

[0127] direct communication

[0128] In another scenario, once a passenger is close to the vehicle, the system can enable direct communication between the passenger's client device and the vehicle's in-vehicle communication system, such as via a point-to-point communication link (e.g., Bluetooth), a WiFi link, or cellular communication, without routing information through the backend ride-sharing service. This reduces potential communication latency that might occur when routing through the backend system. Direct communication could include informing the passenger that their vehicle is parked next to a red car in a parking lot, or requesting the client device to take some action (e.g., flashing lights or emitting a sound) to allow the vehicle's sensors to identify the person's current location with a very high degree of localization (e.g., within 1-3 meters).

[0129] Example System Architecture

[0130] Figure 8A and Figure 8B An example of a backend system for fleet type operations is shown. Specifically, Figure 8A and 8B These are schematic diagrams and functional diagrams of example system 800, which includes multiple computing devices 802, 804, 806, 808 and a storage system 808 connected via network 816. System 800 also includes vehicles 812 and 814 configured to operate in an autonomous driving mode, which can be configured to... Figures 1A-1EVehicles 100, 120, 140, and 160 are identical or similar. Vehicles 812 and / or 814 may be part of one or more fleets that provide rides to passengers or deliver food, groceries, goods, or other packages to customers. Although only a few vehicles and computing devices are depicted for simplicity, a typical system may include significantly more, such as dozens or hundreds of vehicles.

[0131] like Figure 8B As shown, each computing device 802, 804, 806, and 808 may include one or more processors, memory, data, and instructions. Such processors, memory, data, and instructions can be similar to those described above. Figure 2 or Figure 3A The configurations described are used to configure them.

[0132] Various computing devices and vehicles can communicate directly or indirectly via one or more networks (such as network 816). Network 816 and intermediate nodes can include a variety of configurations and protocols, including short-range communication protocols such as Bluetooth™, Bluetooth LE™, the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local area networks, private networks using communication protocols proprietary to one or more companies, Ethernet, WiFi, and HTTP, as well as various combinations thereof. Such communication can be facilitated by any device capable of transmitting and receiving data from other computing devices (such as modems and wireless interfaces).

[0133] In one example, computing device 802 may include one or more server computing devices (e.g., a load-balancing server cluster) having multiple computing devices that exchange information with different nodes in the network for the purpose of receiving, processing, and transmitting data to other computing devices. For example, computing device 802 may include one or more server computing devices capable of communicating with vehicles 812 and / or 814, as well as computing devices 804, 806, and 808, via network 816. For example, vehicles 812 and / or 814 may be part of an autonomous vehicle fleet that can be dispatched to various locations by the server computing devices. In this respect, computing device 802 can serve as a dispatch server computing system, which can be used to dispatch vehicles to different locations for picking up and dropping off passengers or picking up and delivering goods or other items. Furthermore, server computing device 802 can use network 816 to transmit and present information to users of one of the other computing devices or occupants in a vehicle. In this respect, computing devices 804, 806, and 808 can be considered client computing devices.

[0134] like Figures 8A-8BAs shown, each client computing device 804, 806, and 808 may be a personal computing device intended for use by the respective user 818 and has all the components typically used in conjunction with a personal computing device, including one or more processors (e.g., a central processing unit (CPU), graphics processing unit (GPU), and / or tensor processing unit (TPU)), memory for storing data and instructions (e.g., RAM and internal hard disk drives), a display (e.g., a monitor with a screen, touchscreen, projector, television, or other device operable to display information, such as a smartwatch display), and user input devices (e.g., a mouse, keyboard, touchscreen, or microphone). The client computing device may also include a camera for recording video streams, speakers, network interface devices, and all components for connecting these elements to each other.

[0135] While client computing devices may each comprise a full-size personal computing device, they may alternatively include mobile computing devices capable of wirelessly exchanging data with a server via a network such as the Internet. By way of example only, client computing devices 806 and 808 may be mobile phones or devices such as wireless-enabled PDAs, tablet PCs, wearable computing devices (e.g., smartwatches), or netbooks capable of accessing information via the Internet or other networks.

[0136] In some examples, the client computing device 804 can be a remote assistance workstation used by an administrator or operator to communicate with the passengers of a dispatched vehicle. Although in Figures 8A-8B Only a single remote assistance workstation 804 is shown, but any number of such workstations may be included in a given system. Furthermore, while the workstation is depicted as a desktop computer, it can include various types of personal computing devices, such as laptops, netbooks, tablets, etc. As an example, a remote assistance workstation can be used by technicians or other users to assist with sign-related processing, including labeling different types of signs.

[0137] Storage system 88 can be any type of computerized storage capable of storing information accessible by server computing device 802, such as hard disk drives, memory cards, ROM, RAM, DVDs, CD-ROMs, flash drives, and / or tape drives. Furthermore, storage system 88 can include a distributed storage system, wherein data is stored on multiple different storage devices that may be physically located in the same or different geographical locations. Figures 8A-8B It is shown to be connected to a computing device via network 816, and / or can be directly connected to or integrated into any computing device.

[0138] Storage system 810 can store various types of information. For example, storage system 810 can store autonomous vehicle control software, which will be used by vehicles such as vehicles 812 or 814 to operate such vehicles in autonomous driving mode. Storage system 810 can also store one or more models and data used to train the models, such as images, model parameter values, and data structures for different training information such as geographic location data. Storage system 810 can also store training subsystems for training models, as well as result information such as trained classifiers. Trained object classifiers can be shared with specific vehicles or across fleets as needed. They can be updated in real time, periodically, or offline when additional training input data is obtained. Storage system 810 can also include route information, weather information, etc. This information can be shared with vehicles 812 and 814, for example, to help operate the vehicles in autonomous driving mode.

[0139] Figure 9 A flowchart 900 according to one aspect of the present technology is shown, which provides a method for identifying customers of a vehicle configured to operate in an autonomous driving mode. Customers may be passengers, people receiving packages, food orders, or groceries, store employees loading vehicles, etc. In block 902, the method includes receiving perceptual information about a set of objects in the vehicle's external environment by one or more processors of the vehicle's computing system. In block 904, the method includes determining, by one or more processors, that the set of objects in the external environment includes a plurality of pedestrian objects, each corresponding to a person. In block 906, information from at least two different sensor source types is applied to a localization model to obtain the probability that each pedestrian object is a customer. In block 908, the method includes evaluating, by one or more processors, the probability that each pedestrian object is a customer. And in box 910, when determining the probability of a given pedestrian object among pedestrian objects indicates that the given pedestrian object may be a customer, the method includes performing at least one of the following: (i) sending a notification to the customer's client device about the pick-up location where the vehicle is encountered, (ii) adjusting the pick-up location, (iii) causing modification of the driving operation of the vehicle in autonomous driving mode, or (iv) transmitting information about the given pedestrian object to a remote management system.

[0140] While some of the use cases described above focus on passenger pick-up scenarios in the context of hailing a ride, the technology can be used in many other situations. This can include delivery scenarios, where the person approaching the vehicle could be a restaurant or shop employee loading food, groceries, prescriptions, or other packages onto the vehicle. Or the person could be a customer retrieving a package from the vehicle. Here, proximity unlocking can be particularly useful for opening doors or trunks when the person is very close to the vehicle. Similarly, in trucking scenarios, one or more people approaching the vehicle could be warehouse workers loading or unloading goods onto or from the truck, or warehouse workers who need to transfer goods to or from the truck (e.g., mail delivery trucks).

[0141] As an alternative to ride-sharing, this technology can also be used in situations involving personal car ownership. For example, an autonomous vehicle might have already dropped off its owner at a specific location (their office, gym, grocery store, etc.) and then parked at an available location. Here, the system can recognize when the owner walks towards the vehicle and signal them, open the door, etc., as described above. Furthermore, different protocols can be applied to ownership scenarios, where adjustments can be made based on the owner's preferences for approach unlocking or early boarding.

[0142] Finally, this technology provides enhanced flexibility for the pre-positioning of one or more vehicles in a fleet. This reduces the time each vehicle spends reaching its chosen destination for each appropriate pick-up. Furthermore, the fleet management system can instruct certain vehicles to position themselves at specific locations to improve sensor coverage across a group of vehicles. This can be particularly beneficial when multiple people leave a place after an event (e.g., after a movie or sporting event) and different people may use different vehicles to get home.

[0143] Although the techniques described herein have been referenced to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the technology. Therefore, it will be understood that various modifications can be made to the exemplary embodiments, and other arrangements can be designed, without departing from the spirit and scope of the technology as defined by the appended claims.

Claims

1. A method for identifying a customer of a vehicle configured to operate in an autonomous driving mode, the method comprising: The vehicle's computing system receives perceptual information about a set of objects in the vehicle's external environment from one or more processors. The set of objects in the external environment, determined by one or more processors, includes multiple pedestrian objects, each corresponding to a person; Obtain the probability that each pedestrian object is a customer, wherein obtaining the probability includes: The initial information from the customer's client device is applied to the positioning model; and The second information, from a sensor source type different from that of the customer's client device, is applied to the positioning model; One or more processors evaluate the probability that each pedestrian object is a customer; and When determining the probability of a given pedestrian object among pedestrian objects indicates that the given pedestrian object may be a customer, perform at least one of the following: (i) send a notification to the customer's client device about the pick-up location where the vehicle will meet, (ii) adjust the pick-up location, (iii) cause the driving operation of the vehicle in autonomous driving mode to be modified, or (iv) transmit information about the given pedestrian object to the remote management system.

2. The method according to claim 1, wherein, The sensor source type, which differs from that of the customer's client device, is selected from a group consisting of vehicle sources onboard to the vehicle and fleet sources obtained from one or more other vehicles in the vehicle's fleet.

3. The method according to claim 2, wherein, The vehicle source includes one or more sensors of the vehicle's perception system, a positioning system source, or a communication system source.

4. The method according to claim 3, wherein, The sensing system includes one or more sensors such as lidar, camera, radar, or acoustic sensors.

5. The method according to claim 1, wherein, The first information from the customer's client device includes the geographical location information of the customer's client device, and The second information, which comes from sensor sources different from those of the customer's client device, includes perception information from one or more sensors of the vehicle's perception system.

6. The method according to claim 1, wherein, Evaluating the likelihood that each pedestrian is a customer involves refining the list of pedestrians who could be customers.

7. The method according to claim 1, wherein, Sending notifications to a customer's client device may include requesting the client device's current or historical geolocation information.

8. The method according to claim 1, wherein, Sending notifications to customers' client devices includes providing updated walking directions or directions for each turn.

9. The method according to claim 1, wherein, Sending notifications to a customer's client device may include requesting the customer to perform a selected action.

10. The method according to claim 1, wherein, Sending notifications to a customer's client device includes selecting the timing of sending the notification based on the distance of a given pedestrian object to the pick-up location or vehicle.

11. The method according to claim 1, wherein, Sending notifications to a customer's client device involves selecting the level of specificity of the message to the customer based on the distance of a given pedestrian object to the pick-up location or vehicle.

12. The method according to claim 1, wherein, The customer is a passenger who is dispatched to be picked up by a vehicle for a journey.

13. The method according to claim 1, wherein, Adjusting the pick-up location includes moving it closer to the customer's current location to reduce the distance the customer will have to traverse.

14. The method according to claim 1, wherein, Transmitting information about a given pedestrian object to the remote management system includes requests for rider support from the remote management system.

15. The method according to claim 1, wherein, The first information from the customer's client device includes at least one of the following: Gestures received via the customer's client device; Sound from the customer's client device; and Activation of the flash on the customer's client device.

16. A vehicle configured to operate in an autonomous driving mode, the vehicle comprising: A perception system includes one or more sensors configured to receive sensor data associated with objects in the vehicle’s external environment; The driving system includes the steering subsystem, acceleration subsystem, and deceleration subsystem that control the driving of the vehicle; The positioning system is configured to determine the vehicle's current location; as well as A control system, including one or more processors, operatively coupled to a driving system, a perception system, and a positioning system, wherein the control system is configured to: Receives perceptual information about a set of objects in the vehicle's external environment from the perception system; The set of objects in the external environment is determined to include multiple pedestrian objects, each corresponding to a person; The system obtains the probability that each pedestrian object is a customer, wherein the control system is configured to obtain the probability in the following manner: The initial information from the customer's client device is applied to the positioning model; and The second information, from a sensor source type different from that of the customer's client device, is applied to the positioning model; Evaluate the likelihood that each pedestrian is a customer; as well as When determining the probability of a given pedestrian object among pedestrian objects indicates that the given pedestrian object may be a customer, perform at least one of the following: (i) send a notification to the customer's client device about the pick-up location where the vehicle will meet, (ii) adjust the pick-up location, (iii) modify the driving operation to be performed by the driving system in autonomous driving mode, or (iv) transmit information about the given pedestrian object to the remote management system.

17. The vehicle according to claim 16, wherein, The vehicle is configured to deliver packages to customers at the pick-up location.

18. The vehicle according to claim 16, wherein, The first information from the customer's client device includes the geographical location information of the customer's client device, and The second information, which comes from sensor sources different from those of the customer's client device, includes perception information from one or more sensors of the vehicle's perception system.

19. The vehicle according to claim 16, wherein, The evaluation of the probability that each pedestrian object is a customer includes refining the list of pedestrian objects that could be customers.

20. The vehicle according to claim 16, wherein, The control system is also configured to send a request for updated location information to the customer's client device when it determines that the customer is within a threshold proximity to the vehicle or pick-up location.

21. The vehicle according to claim 16, wherein, The control system is also configured to issue a request for sensor information from the external environment of one or more other vehicles, and to apply the provided sensor information to a positioning model.

Citation Information

Patent Citations

  • Identifying a customer of an autonomous vehicle

    US11904902B2

  • Vehicle passenger identification

    CN105676251A