Pickup and drop-off location identification for ride-sharing and delivery via augmented reality

Abstract

To present a pick-up location in a transportation service, a server device receives a request for a rideshare from a rider's current location to a delivery location. The server device broadcasts the request to the drivers and receives an indication from one of the drivers to accept the request. The server device then provides the rider and driver with the pick-up location presented in a map display of the rider and driver's client devices. When the rider or driver's current location is within a threshold distance of the pick-up location, the corresponding client device switches to an augmented reality mode and presents a camera view from a camera of the client device depicting a real-world image. The server device provides information to be presented in augmented reality mode overlaid on the real-world image to the corresponding client device to assist the rider or driver in finding the pick-up location.

Description

Technical Field

[0001] The present disclosure relates to ridesharing systems and, more particularly, to using augmented reality to provide accurate location information to ridesharing application users. Background Art

[0002] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. To the extent described in this background section, the work of the presently named inventors, as well as aspects of the description that may not be suitable as prior art at the time of filing, are neither explicitly nor implicitly admitted to be prior art with respect to the present disclosure.

[0003] Today, ride-sharing or carpooling providers connect drivers and passengers to arrange for the driver to transport the passenger to a destination. Typically, ride-sharing applications present pick-up locations to drivers and riders for picking up the riders.

[0004] A problem with ride-sharing services is that it is sometimes difficult for the rider and the driver of the vehicle to recognize each other. For example, the rider may be informed of the make, color, and license plate of the vehicle, and the rider may be given an estimated location of the vehicle on a map on the rider's mobile device. However, the make and color of the vehicle may not be particularly helpful in identifying the correct vehicle, as there may be dozens of vehicles of the same make and color nearby in an urban environment. The license plate may be difficult for the rider to read, especially when the vehicle is more than a few meters away from the rider. In addition, the estimated location of the vehicle may be too inaccurate to assist the rider. The estimated location is typically based on GPS measurements made by a mobile device in the driver's vehicle. Such GPS measurements may be inaccurate due to low signal strength of GPS signals in built-in or closed environments, or due to defects in the GPS receiver of the mobile device.

[0005] The driver typically receives even less information. For example, the driver may be given an approximate location of the rider and, depending on the ride-sharing provider, an image of the rider. The rider's approximate location is typically based on GPS measurements made by a mobile device owned by the rider, and therefore suffers from similar sources of inaccuracy as described in the context of the driver's location. Summary of the invention

[0006] To help drivers and passengers recognize each other, the ride-sharing augmented reality (AR) system switches the display of the ride-sharing application from a map display mode to an AR mode when the passenger or driver approaches the pick-up location. The AR mode presents AR features that are overlaid on a real-world imagery of an area captured by a camera of the passenger or driver client device. More specifically, in AR mode, the ride-sharing AR system may present a camera view of a real-world imagery in front of the passenger or driver client device. The ride-sharing AR system may then identify a location included in the camera view and determine whether the location includes a pick-up location and / or the current location of the passenger or driver. When the camera view includes the pick-up location, the ride-sharing AR system may present an indication of the pick-up location overlaid with the real-world imagery at a location within the camera view corresponding to the pick-up location. The indication may be a pin or other graphic placed at a location within the camera view corresponding to the pick-up location.

[0007] When the camera view includes the driver's vehicle, the ride-sharing AR system may present an indication of the vehicle overlaying the real-world image at a location within the camera view corresponding to the vehicle. For example, the ride-sharing AR system may highlight the vehicle within the real-world image by placing a pin or arrow above the vehicle, placing a circle, square, or other shape around the vehicle, rendering a particular color over the vehicle, placing a graphic over or above the vehicle, etc. Similarly, when the camera view includes an occupant, the ride-sharing AR system may present an indication of the occupant overlaying the real-world image at a location within the camera view corresponding to the occupant. For example, the ride-sharing AR system may highlight the occupant within the real-world image by placing a pin or arrow over the occupant, placing a circle, square, or other shape around the occupant, rendering a particular color over the occupant, placing a graphic over or above the occupant, etc.

[0008] One example embodiment of the technology disclosed herein is a method for providing a display of a pick-up location in a transportation service. The method includes receiving a request in a server device to transport a rider to a drop-off location, providing a pick-up location for the driver to pick up the rider to a rider's client device or a driver's client device, and receiving a real-world image from a field of view of the rider's client device or the driver's client device. The method also includes determining a location corresponding to the real-world image, comparing the determined location to the pick-up location, and providing information to be displayed in an augmented reality mode based on the comparison to assist the rider or the driver, the information overlaying the real-world image presented on the rider's client device or the driver's client device.

[0009] Another example embodiment is a method for identifying and presenting a pick-up location to a rider in a transportation service. The method includes sending a request in a client device of a user to transport the user to a drop-off location, and receiving a pick-up location at which a vehicle picks up the user. The method also includes providing, via a user interface, an interactive digital map of a geographic area including an indication of the pick-up location and an indication of a current location of the user in a map display mode. In response to determining that the current location of the user is within a threshold distance of the pick-up location, the method includes switching, via the user interface, to an augmented reality mode to assist the user, the augmented reality mode presenting a real-world image in a field of view of the client device along with information overlaying the real-world image.

[0010] Yet another example embodiment is a method for identifying and presenting a pick-up location to a driver in a delivery service. The method includes receiving a request in a driver's client device to deliver a passenger or item to a delivery location, accepting the request, and receiving the pick-up location. The method also includes providing, via a user interface, an interactive digital map of a geographic area including navigation directions from the driver's current location to the pick-up location in a map display mode. In response to determining that the driver's current location is within a threshold distance of the pick-up location, the method includes switching, via the user interface, to an augmented reality mode to assist the driver, the augmented reality mode presenting a real-world image in a field of view of the client device together with information overlaying the real-world image. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram of an example communication system in which a driver client device and a rider client device may operate to implement a ride-sharing AR system;

[0012] Figure 2 is Figure 1 A block diagram of an example passenger / driver client device that may operate in the system of FIG.

[0013] Figure 3 is an example message diagram illustrating example information communicated between a rider client device, a driver client device, and an augmented reality server to implement a ride-sharing AR system;

[0014] Figure 4A is an example display of a pick-up location in a map display mode of a ride-sharing application;

[0015] Figure 4B In AR mode for ride-sharing apps Figure 4A An example of a pickup location is shown;

[0016] Figure 5A is another example display of a pick-up location in a map display mode of a ride-sharing application;

[0017] Figure 5B In AR mode for ride-sharing apps Figure 5A An example of a pickup location is shown;

[0018] Figure 6 is an example display of a driver's vehicle in an AR mode of a ride-sharing application;

[0019] Figure 7 is another example display of a driver's vehicle in an AR mode of a ride-sharing application;

[0020] Figure 8 is a flow chart of an example method for providing a display of a pick-up location in a delivery service, which may be implemented in a server device; and

[0021] Fig. 9 is a flow chart of an example method for identifying and presenting a pick-up location to a rider in a transportation service, which may be implemented in a rider client device. DETAILED DESCRIPTION

[0022] Overview

[0023] The present disclosure provides technology for improved identification of passengers and drivers in transportation services. More specifically, the present disclosure uses augmented reality to help passengers and drivers identify each other when the passengers and drivers agree to share location and / or identity information. For example, augmented reality technology can be used to indicate the driver's vehicle in a display of real-world images around the passenger's client device (or vice versa) to reduce or eliminate the need for the passenger to use only vision to identify the vehicle. As another example, augmented reality technology can be used to guide the driver and / or passengers towards the pick-up location. The use of augmented reality can reduce reliance on GPS measurements when the passengers and drivers try to locate each other. Although these techniques are described herein with reference to the pick-up location, similar techniques can also be used to identify the delivery location by presenting AR features at or around the delivery location. In addition, the technology described herein can also be used in the context of shared vehicles or any other transportation service, such as scooters, bicycles, autonomous vehicles, ride-for-hire services, taxi services, or any other vehicle that can be used to transport one user from a pick-up location to a drop-off location, and another user can retrieve the vehicle at the drop-off location and travel to another drop-off location.

[0024] Example Hardware and Software Components

[0025] refer to Figure 1, an example communication system in which the techniques outlined above may be implemented includes a driver client device 10 and a rider client device 28, each of which may be a portable device configured to execute one or several ride sharing applications 44. In addition to the driver client device 10 and the rider client device 28, the communication system also includes a server device 110, such as an augmented reality server, which is configured to provide augmented reality features overlaid with real-world images to the driver client device 10 and the rider client device 28. The communication system also includes a navigation server device 120, which is configured to provide map displays and navigation data to the driver client device 10 and the rider client device 28. In addition, the communication system 100 includes a ride sharing server device 130 configured to provide ride service functions. The ride service functions may include receiving ride sharing requests from passengers, broadcasting ride sharing requests to drivers in the same area as the passengers, receiving messages from drivers accepting the requests, and providing drivers and passengers with a pick-up location for retrieving the passengers together with updated location information indicating the distance from the driver and the passengers to the pick-up location.

[0026] The driver client device 10, the passenger client device 28, the augmented reality server 110, the navigation server 120, and the ride sharing server 130 may be communicatively connected to each other via a network 100. The network 100 may be a public network such as the Internet, or a private network such as an intranet.

[0027] The augmented reality server 110 may be communicatively coupled to a database 136, which in an example implementation stores real-world images of various geographic areas. In this way, the augmented reality server 110 may compare the real-world image of the camera view from the driver client device 10 or the passenger client device 28 with the real-world image in the database 136 to identify the location and / or orientation of the client device 10, 28. The augmented reality server 110 may also be communicatively coupled to a database storing augmented reality features (such as shapes and graphics to be overlaid on real-world images). More generally, the server device 110 may communicate with one or more databases storing any type of information that can be used in an augmented reality environment. In some implementations, the server device 110 may include several communicatively connected server devices. For example, the passenger client device 28 may communicate with the ride-sharing server 130 to broadcast a request for a ride-sharing service to a destination to the driver, and receive a driver who accepts the request. The ride-sharing server 130 may then communicate with the navigation server 120 to identify the pick-up location and provide map data showing the pick-up location and the driver's current location, as well as navigation data presenting directions to the pick-up location. In addition, the ride-sharing server 130 may communicate with the augmented reality server to receive AR features and their corresponding locations within the camera view at which the AR features are to be presented in the AR mode of the ride-sharing application 44 on the rider client device 28.

[0028] In an example implementation, if Figure 2 As shown, for example, each of the driver client device 10 and the passenger client device 28 can be a smart phone or a tablet computer, and includes a memory 106, one or more processors 107, a graphics processing unit (GPU) 109, a network interface 118, a user interface (UI) 121, and one or more sensors 122. The memory 106 can be a non-transitory memory, and can include one or more suitable memory modules, such as a random access memory (RAM), a read-only memory (ROM), a flash memory, other types of permanent memory, etc. For example, the UI 121 can be a touch screen. More generally, the technology of the present disclosure can be implemented in other types of devices, such as laptop or desktop computers, devices embedded in vehicles (such as vehicle head units), wearable devices (such as smart watches or smart glasses, etc.).

[0029] Depending on the implementation, the one or more sensors 122 may include a camera, a global positioning system (GPS) module to detect the location of the client device 10, 28, a compass to determine the orientation of the client device 10, 28, a gyroscope to determine rotation and tilt, an accelerometer, etc.

[0030] The memory 106 stores an operating system (OS) 114, which can be any type of suitable mobile or general-purpose operating system. The OS 114 can include API functions that allow applications (such as the ride-sharing application 44 and other applications) to interface with each other or retrieve, for example, sensor readings. For example, a software application configured to execute on the client device 10, 28 may include instructions to call (invoke) the OS 114 API to retrieve the current location and orientation of the client device 10, 28 at that moment. The API can also return a quantitative indication of how certain the API's estimate is (e.g., as a percentage).

[0031] The memory 106 also stores a ride-sharing application 44 and other applications 104, and the ride-sharing application 44 is configured to provide ride-sharing services or other transportation services to passengers, or to identify passengers who need ride-sharing services or other transportation services for drivers. The ride-sharing application 44 can receive a request for a ride-sharing service or other transportation service from a passenger's current location to a destination, and can identify a pick-up location for the driver to retrieve the passenger. In addition, the ride-sharing application 44 can receive map data in raster (e.g., bitmap) or non-raster (e.g., vector graphics) format from a map database and / or navigation server 120. The ride-sharing application 44 can also display navigation directions from the driver or passenger's current location to the pick-up location and / or from the pick-up location to the destination. The navigation directions can include driving or walking directions.

[0032] The ride-sharing application 44 includes an AR display module 48 that is configured to switch the display of the ride-sharing application 44 from a map display mode that presents two-dimensional (2D) map data of a geographic area indicating the current location of the rider or driver, the pickup location, and / or the destination location to an AR mode that presents a camera view of the area in front of the client device 10, 28. In this way, the ride-sharing application 44 conserves data and battery power by presenting a map display until the rider or driver approaches the pickup location where the ride-sharing application 44 presents a camera view. In response to obtaining the pickup location, the ride-sharing application 44 presents a 2D map display of a geographic area indicating the current location of the rider or driver, the pickup location, and navigation directions from the current location to the pickup location. In other embodiments, the map display can be a three-dimensional (3D) or two and a half (2.5D) map display.

[0033] The ride-sharing application 44 continuously or periodically obtains location information indicating the current location of the driver and / or passenger, and presents the updated location of the driver and / or passenger in the map display. The AR display module 48 determines the distance from the current location of the user of the client device 10, 28 to the pick-up location, and compares the distance with the threshold distance. In response to determining that the distance is less than the threshold distance, the AR display module 48 switches the display from the map display mode to the AR mode of presenting the camera view of the area in front of the client device 10, 28. Then, the AR display module 48 obtains location information indicating the current location of the driver and / or passenger at a higher frequency than the ride-sharing application 44 obtains location information in the map display mode. For example, in the map display mode, the ride-sharing application 44 can obtain the location information of the driver and / or passenger once per second. In the AR mode, the AR display module 48 can obtain the location information of the driver and / or passenger twice or three times per second. In this way, the AR display module 48 has more up-to-date location information for presenting AR features at the accurate location in the camera view. For example, this may be particularly important for highlighting moving vehicles.

[0034] The camera view includes real-world images, such as buildings, streets, vehicles, and people within the field of view of the camera of the client device 10, 28. In addition, the AR mode can present AR features that overlay the real-world image, such as shapes, colors, and graphics that can be placed on or over specific vehicles, people, or locations that indicate a pickup location or highlight the driver's vehicle or a person who is a passenger in the camera view. In some implementations, the AR display module 48 can also present AR features that indicate the direction that the occupant or driver needs to turn to see the vehicle or passengers.

[0035] Note that although Figure 2 The ride-sharing application 44 is shown as a stand-alone application, but the functionality of the ride-sharing application 44 may also be provided in the form of an online service accessible via a web browser executing on the client device 10, 28, as a plug-in or extension to another software application executing on the client device 10, 28, etc. The ride-sharing application 44 may generally be provided in different versions for different respective operating systems. For example, the manufacturer of the client device 10, 28 may provide a software package for Android. TM Software Development Kit (SDK) for writing or building ride-sharing apps 44 for iOS TM Another SDK for the platform, etc.

[0036] Back to Figure 1, the augmented reality server 110 includes one or more processors 112, a network interface (not shown), and a memory 132. The memory 132 may be a tangible, non-transitory memory and may include any type of suitable memory module, including random access memory (RAM), read-only memory (ROM), flash memory, other types of permanent memory, etc. The memory 132 stores instructions executable on the processor 112, which may generate AR features overlaying real-world images to be displayed by the ride-sharing application 44 via an AR generation engine 134. The memory 132 or a memory in another server may similarly store instructions to generate a map display and navigation directions to a pickup location or destination to be displayed by the ride-sharing application 44. In some implementations, the ride-sharing server 130 may initiate a call to the augmented reality server 110 for an AR feature to overlay a real-world image presentation in an AR mode of the ride-sharing application 44 on the driver client device 10 and / or the passenger client device 28.

[0037] The AR generation engine 134 may receive a camera view or image from a camera of a driver client device 10 or a passenger client device 28, wherein the camera view depicts a real-world image, such as buildings, streets, vehicles, and people within the camera's field of view. The AR generation engine 134 may then determine a location corresponding to the camera view. More specifically, for example, the AR generation engine 134 may compare the camera view with a number of template camera views stored in the real-world image database 136. Each template camera view or image may store an indication of the camera viewpoint or location where the image was taken (e.g., a GPS location specifying latitude and longitude coordinates, a street address, etc.), an indication of the orientation of the camera when the image was taken (e.g., the camera faces north, thereby indicating that the camera view depicts an area north of the viewpoint), an indication of the scale or zoom level of the camera view, and / or an indication of the geographic area corresponding to the camera view, including an indication of the precise physical location at each location within the camera view based on the scale or zoom level. The orientation of the camera may include a three-axis orientation indicating the direction the camera is facing (eg, east, west, north, south, etc.), the tilt angle of the camera (eg, parallel to the ground), and whether the camera is in a horizontal or vertical position.

[0038] For example, for a template camera view, the camera viewpoint may be from the corner of Main Street and State Street, with the camera facing east. The template camera view may depict a width of 5 meters, a length of 7 meters, and a height of 3 meters. The AR generation engine 134 may then create a mapping of the exact location of each pixel or group of pixels in the template camera view based on the camera viewpoint, the camera orientation, and the size of the template camera view. For example, if the template camera view is 5m wide, the width of the image is 500 pixels, the orientation of the template camera view faces east, and the tilt angle is such that the camera is perpendicular to the ground, each pixel may represent a physical width of approximately 1 cm. The camera viewpoint of the template camera view may be determined from the GPS location of the client device that captured the camera view, and the orientation may be determined from a gyroscope and / or compass included in the client device that captured the camera view.

[0039] In any case, the AR generation engine 134 may compare the camera view from the camera of the driver client device 10 or the passenger client device 28 to the template camera view to determine the camera viewpoint, orientation, zoom or scale level of the camera view, and / or the precise physical location of various locations within the camera view based on the comparison. For example, the AR generation engine 134 may compare the camera view from the camera of the driver client device 10 or the passenger client device 28 to the template camera view using machine learning techniques such as random forests, boosting, nearest neighbors, Bayesian networks, neural networks, support vector machines, etc.

[0040] More specifically, the AR generation engine 134 can identify visual features of each template camera view by detecting stable regions within the template camera view, which are detectable regardless of blur, motion, distortion, orientation, lighting, scaling, and / or other changes in the camera's perspective. Stable regions can be extracted from the template camera view using scale-invariant feature transforms (SIFT), accelerated robust features (SURF), fast retinal keypoints (FREAK), binary robust invariant scalable keypoints (BRISK), or any other suitable computer vision technology. In some embodiments, keypoints may be located in high contrast areas of the template camera view, such as edges within the template camera view. A bounding box may be formed around the keypoint, and the portion of the template camera view created by the bounding box may be a feature.

[0041] The AR generation engine 134 may create numerical representations of the features to generate a template feature vector, such as the width and height of the feature, the RGB pixel value of the feature, the pixel position of the center point of the feature within the image, etc. The AR generation engine 134 may then use the feature vector as training data to create a machine learning model. For example, if the machine learning technique is nearest neighbor, the AR generation engine 134 may identify visual features of a camera view from a camera of a driver client device 10 or a passenger client device 28 to generate a feature vector. The AR generation engine 134 may then compare the feature vector of the camera view from the camera of the driver client device 10 or the passenger client device 28 with the template feature vector to identify a template camera view having a template feature vector that is closest to the feature vector of the camera view from the camera of the driver client device 10 or the passenger client device 28.

[0042] The AR generation engine 134 may then determine the precise physical locations of the various locations within the camera view and / or the camera viewpoint, orientation, scaling, or zoom level of the camera view based on the precise physical locations of the various locations within the identified template camera view and / or the camera viewpoint, orientation, scaling, or zoom level of the identified template camera view.

[0043] The AR generation engine 134 may then identify a location within the camera view for presenting an indication of the pick-up location based on the precise physical locations assigned to the various locations within the camera view. Additionally, when the camera view is from the camera of the rider client device 28, the AR generation engine 134 may obtain an indication of the current location of the driver client device 10. The AR generation engine 134 may identify a location within the camera view for presenting an indication of the driver's vehicle based on the current location of the driver client device 10 and the precise physical locations assigned to the various locations within the camera view.

[0044] In addition to identifying a location within the camera view for presenting an indication of the driver's vehicle based on the current location of the driver's client device 10, the AR generation engine 134 can use object recognition to analyze the camera view to identify the vehicle within the camera view. More specifically, the AR generation engine 134 can obtain several template objects from a database. In addition to other objects (such as buildings, street signs, etc.), the template object may also include a vehicle. The AR generation engine 134 can then identify the visual features of the template object by detecting a stable region within the template object, which is detectable regardless of blur, motion, distortion, orientation, lighting, telescoping, and / or other changes in the camera's perspective. Stable regions can be extracted from the template object using SIFT, SURF, FREAK, BRISK, or any other suitable computer vision technology.

[0045] The AR generation engine 134 can create a digital representation of the feature to generate a template feature vector, such as the width and height of the feature, the RGB pixel value of the feature, the pixel position of the center point of the feature within the object, etc. As also described above, the AR generation engine 134 can identify visual features of a camera view or a portion of a camera view (also referred to herein as an "object") from the camera of the occupant client device 28 to generate a feature vector. The AR generation engine 134 can then compare the feature vector of the camera view or the feature vector of the object within the camera view with the template feature vector to identify one or more template objects having a template feature vector that is closest to the feature vector of the camera view from the camera of the occupant client device 28. For example, the AR generation engine 134 can identify the first three template objects having the template feature vector that is closest to the feature vector of the camera view. If a majority of one or more identified template objects are vehicles, the AR generation engine 134 can identify the object in the camera view as a vehicle.

[0046] The AR generation engine 134 may also compare the location of the object in the camera view to the location corresponding to the current location of the driver client device 10. If the location of the object identified as a vehicle in the camera view is within a threshold distance of the location corresponding to the current location of the driver client device 10, the AR generation engine 134 may determine that the location of the object in the camera view is the location of the driver's vehicle.

[0047] If the current location of the driver client device 10 corresponds to a location within the camera view and / or the object is identified as a vehicle and is within a threshold distance of that location, the AR generation engine 134 may provide an indication of the driver's vehicle to the passenger client device 28 for display at or over the identified location.

[0048] Additionally, when the camera view is from the camera of the driver client device 10, the AR generation engine 134 may obtain an indication of the current location of the passenger client device 28. The AR generation engine 134 may identify locations within the camera view for presenting indications of the passenger based on the current location of the passenger client device 28 and the precise physical locations assigned to various locations within the camera view. In addition to identifying locations within the camera view for presenting indications of the passenger based on the current location of the passenger client device 28, the AR generation engine 134 may also analyze the camera view using facial detection and / or body detection to identify people within the camera view.

[0049] Similar to identifying a vehicle, the AR generation engine 134 can obtain several template objects from a database. In addition to other objects (such as vehicles, buildings, street signs, etc.), the template objects may also include people. The AR generation engine 134 can then identify the visual features of the template object by detecting stable regions within the template object, which are detectable regardless of blur, motion, distortion, orientation, lighting, telescoping, and / or other changes in the camera's perspective. Stable regions can be extracted from the template object using SIFT, SURF, FREAK, BRISK, or any other suitable computer vision technology.

[0050] The AR generation engine 134 can create a digital representation of features to generate a template feature vector, such as the skin color of a person, the width and height of a nose, a mouth, eyes and their respective positions relative to each other, the pixel position of the center point of the feature within the object, etc. As also described above, the AR generation engine 134 can identify visual features of a camera view from the camera of the driver client device 10 or an object in the camera view to generate a feature vector. The AR generation engine 134 can then compare the feature vector of the camera view or the feature vector of the object in the camera view with the template feature vector to identify one or more template objects having a template feature vector that is closest to the feature vector of the camera view from the camera of the driver client device 10. For example, the AR generation engine 134 can identify the first three template objects having the template feature vector that is closest to the feature vector of the camera view. If most of the one or more identified template objects are people, the AR generation engine 134 can identify the object in the camera view as a person.

[0051] The AR generation engine 134 may also compare the location of the object in the camera view to the location corresponding to the current location of the occupant client device 28. If the location of the object identified as a person in the camera view is within a threshold distance of the location corresponding to the current location of the occupant client device 28, the AR generation engine 134 may determine that the location of the object in the camera view is the location of the occupant.

[0052] If the current location of the occupant client device 28 corresponds to a location within the camera view and / or the object is identified as a person and within a threshold distance of that location, the AR generation engine 134 may provide an indication of the occupant to the driver client device 10 for display at or over the identified location.

[0053] For simplicity, Figure 1The augmented reality server 110 is shown as only one server instance. However, the augmented reality server 110 according to some implementations includes a group of one or more server devices, each of which is equipped with one or more processors and can operate independently of other server devices. The server devices operating in such a group can process requests from the driver client device 10 or the passenger client device 28 individually (e.g., based on availability) in a distributed manner or according to any other suitable technology, in which an operation associated with processing the request is performed on one server device, and another operation associated with processing the same request is performed on another server device. For the purposes of this discussion, the term "server device" can refer to a single server device, or a group of two or more server devices.

[0054] Sample message diagram

[0055] Figure 3 A messaging diagram 300 is shown that depicts an example sequence of events for the rider client device 28, the driver client device 10, and the augmented reality server 110 to implement a ride-sharing AR system. Although the messaging diagram 300 shows example communications between the rider client device 28, the driver client device 10, and the augmented reality server 110, the augmented reality server 110 may communicate with the ride-sharing server 130 to broadcast requests for ride-sharing services or other transportation services to destinations to drivers, and receive drivers who accept the requests.

[0056] In other implementations, the rider client device 28 and the driver client device 10 may communicate with the ride sharing server 130 to broadcast requests for ride sharing services or other transportation services to a destination to the driver and accept the requests. The ride sharing server 130 may communicate with the augmented reality server 110 to receive AR features and their corresponding locations within the camera view at which the AR features are to be presented in the AR mode of the ride sharing application 44 on the rider client device 28 and / or the driver client device 10.

[0057] In any case, the rider client device 28 requests 304 a ride service to a destination or delivery location, such as via a ride sharing application 44. The augmented reality server 110 broadcasts 306 the ride sharing request to drivers (which may include the driver client device 10) within a threshold radius of the rider's current location. The ride sharing request may include the rider's current location and the destination. In addition to the rider's request for ride service, the user may also provide a request for the driver to deliver an item to the delivery location. For example, a user may have food, merchandise, or other goods delivered from a pickup location to a delivery location via a ride sharing service. In this case, the user may be instructed to deliver the item from the item's current location to the pickup location.

[0058] The driver client device 10 then accepts 308 the ride request, and the augmented reality server 110 generates a pick-up location for retrieving the rider. The pick-up location may be within a threshold distance of the rider's current location. Additionally, the pick-up location may be selected from a list of predetermined pick-up locations stored in a database. The augmented reality server 110 provides 310, 312 the pick-up location to the rider client device 28 and the driver client device 10, respectively, along with identification information for the driver and rider. More specifically, the driver's identification information may include the driver's name, a photo of the driver, the make, model, and license plate number of the driver's vehicle, or any other suitable identification information. The rider's identification information may include the rider's name, a photo of the rider, or any other suitable identification information.

[0059] The pick-up location may then be presented in a map display view of the ride-sharing application 44 executed on the rider client device 28 and the driver client device 10. More specifically, the rider client device 28 may present a 2D map display of a geographic area indicating the rider's current location, the pick-up location, and navigation directions from the rider's current location to the pick-up location. The current location may be indicated by a dot or other suitable indicator, and the pick-up location may be indicated by a pin or other suitable indicator. Navigation directions may be indicated by highlighting the route from the rider's current location to the pick-up location in the 2D map display, such as by overlaying a dotted line on the route.

[0060] The driver client device 10 may present a 2D map display of a geographic area indicating the driver's current location, the pickup location, and navigation directions from the driver's current location to the pickup location. The current location may be indicated by a dot or other suitable indicator, and the pickup location may be indicated by a pin or other suitable indicator. Navigation directions may be indicated by highlighting the route from the driver's current location to the pickup location in the 2D map display, such as by overlaying a dotted line on the route.

[0061] The rider client device 28 continuously or periodically determines 314 its current location via GPS and presents the updated location of the rider in a map display. More specifically, the rider client device 28 may determine its current location at a first frequency. The rider client device 28 then determines the distance from the current location to the pick-up location and compares the distance to a threshold distance. In response to determining that the distance is less than the threshold distance, the rider client device 28 switches 316 the display from a map display mode to an AR mode that presents a camera view of the area in front of the rider client device 28. The rider client device 28 then obtains location information indicating the rider's current location at a second frequency that is higher than the first frequency. The location updates of the rider client device 28 may also be provided to the ride sharing server 130 and / or the augmented reality server 110 to be presented in the map display of the driver client device 10, and to identify the location of the AR feature to be overlaid in the camera view of the driver client device 10.

[0062] Likewise, the driver client device 10 continuously or periodically determines 322 its current location via GPS, and presents the driver's updated location in a map display. More specifically, the driver client device 10 may determine its current location at a first frequency. The driver client device 10 then determines the distance from the current location to the pick-up location, and compares the distance to a threshold distance. In response to determining that the distance is less than the threshold distance, the driver client device 10 switches the display 324 from the map display mode to an AR mode that presents a camera view of the area in front of the driver client device 10. The driver client device 10 then obtains location information indicating the driver's current location at a second frequency that is higher than the first frequency. The location updates of the driver client device 10 may also be provided to the ride-sharing server 130 and / or the augmented reality server 110 to be presented in the map display of the passenger client device 28, and to identify the location of the AR feature to be overlaid in the camera view of the passenger client device 28.

[0063] When the rider client device 28 switches the display to AR mode and presents a camera view of the area in front of the rider client device 28, the rider client device 28 sends 318 a camera view or image from the camera of the rider client device 28 to the augmented reality server 110, where the camera view depicts real-world images, such as buildings, streets, vehicles, and people within the camera's field of view. The augmented reality server 110 can then, for example, compare the camera view to a number of template camera views stored in the real-world image database 136. Each template camera view or image can store an indication of the camera viewpoint or location where the image was taken (e.g., a GPS location specifying latitude and longitude coordinates, a street address, etc.), an indication of the orientation of the camera when the image was taken (e.g., the camera was facing north, thereby indicating that the camera view depicts an area north of the viewpoint), an indication of the zoom or scale level of the camera view, and / or an indication of the geographic area corresponding to the camera view, including an indication of the precise physical location at each location within the camera view based on the zoom or scale level. The orientation of the camera may include a three-axis orientation indicating the direction the camera is facing (eg, east, west, north, south, etc.), the tilt angle of the camera (eg, parallel to the ground), and whether the camera is in a horizontal or vertical position.

[0064] When the augmented reality server 110 identifies a matching template camera view, the augmented reality server 110 determines the precise physical location of each location within the camera view and / or the camera viewpoint, orientation, scaling, or zoom level of the camera view based on the precise physical location and / or camera viewpoint, orientation, scaling, or zoom level of each location within the identified template camera view.

[0065] The augmented reality server 110 then identifies the AR features to overlay on the real-world image in the camera view. More specifically, the augmented reality server 110 may determine whether the pick-up location is included in the camera view based on the precise physical locations at various locations within the camera view. The augmented reality server 110 may also determine whether the driver's vehicle is included in the camera view based on the current location of the driver's client device 10, the precise physical locations at various locations within the camera view, and / or whether there is an object identified as a vehicle in the camera view.

[0066] If the pick-up location is not included in the camera view and / or the driver's vehicle is not included in the camera view, the augmented reality server 110 can determine the direction the rider client device 28 needs to face to view the pick-up location and / or the driver's vehicle. The augmented reality server 110 can also determine the tilt angle of the rider client device 28 to view the pick-up location and / or the driver's vehicle. The augmented reality server 110 can then determine the driver's orientation relative to the rider or the pick-up location based on the pick-up location or the driver's current location relative to the rider's current location. The augmented reality server 110 can also determine that the orientation of the rider client device 28 is different from the driver's orientation relative to the rider or the pick-up location.

[0067] For example, if the camera of the rider client device 28 is facing north, and the pick-up location is east of the camera of the rider client device 28, the augmented reality server 110 may determine that the rider needs to turn right. Additionally, if the camera of the rider client device 28 is tilted downward toward the ground, and the pick-up location is farther away from the rider client device 28, the augmented reality server 110 may determine that the rider needs to tilt the rider client device 28 upward. In yet another example, if the camera of the rider client device 28 is tilted toward the sky, and the pick-up location is in front of the rider client device 28, the augmented reality server 110 may determine that the rider needs to tilt the rider client device 28 downward.

[0068] In any case, the augmented reality server 110 may then generate and provide 320 an AR feature to be overlaid on the real-world image in the camera view. AR features may include shapes, colors, and graphics that may be placed on or over a particular vehicle, person, or location that indicates a pick-up location or highlights the driver's vehicle or a person who is a passenger in the camera view. AR features may also include arrows that direct the rider to turn or tilt the rider client device 28 to view the pick-up location or the driver's vehicle. In some embodiments, the augmented reality server 110 generates and provides 320 AR data to the rider client device 28, the AR data including a description of the AR feature and the location within the camera view at which the AR feature is to be rendered. The rider client device 28 then renders the AR feature at the indicated location. In the following Figure 4B , Figure 5B , Figure 6 and Figure 7 The AR features are shown in more detail in .

[0069] When the driver client device 10 switches the display to AR mode and presents a camera view of the area in front of the driver client device 10, the driver client device 10 sends 326 a camera view or image from a camera of the driver client device 10 to the augmented reality server 110, where the camera view depicts real-world images, such as buildings, streets, vehicles, and people within the camera's field of view. The augmented reality server 110 may then compare the camera view to a number of template camera views stored in the real-world image database 136, for example.

[0070] When the augmented reality server 110 identifies a matching template camera view, the augmented reality server 110 determines the precise physical location of each location within the camera view and / or the camera viewpoint, orientation, scaling, or zoom level of the camera view based on the precise physical location and / or camera viewpoint, orientation, scaling, or zoom level of each location within the identified template camera view.

[0071] The augmented reality server 110 then identifies AR features to overlay on the real-world image in the camera view. More specifically, the augmented reality server 110 may determine whether the pick-up location is included in the camera view based on the precise physical locations at various locations within the camera view. The augmented reality server 110 may also determine whether an occupant or item is included in the camera view based on the current location of the occupant client device 28, the precise physical locations at various locations within the camera view, and / or the presence or absence of an object identified as a person or item in the camera view.

[0072] If the pick-up location is not included in the camera view, and / or the passengers or items are not included in the camera view, the augmented reality server 110 can determine the direction the driver client device 10 needs to face to view the pick-up location and / or the passengers or items. The augmented reality server 110 can also determine the tilt angle of the driver client device 10 to view the pick-up location and / or the passengers or items.

[0073] The augmented reality server 110 may then generate and provide 328 an AR feature to be overlaid on the real-world image in the camera view. AR features may include shapes, colors, and graphics that may be placed on or over a particular vehicle, person, or location that indicates a pick-up location or highlights the driver's vehicle or person as a passenger in the camera view. AR features may also include arrows that direct the driver to turn or tilt the driver client device 10 to view the pick-up location, passengers, or items. In some embodiments, the augmented reality server 110 generates and provides 328 AR data to the driver client device 10, the AR data including a description of the AR feature and the location at which the AR feature is to be presented in the camera view. The driver client device 10 then renders the AR feature at the indicated location. In the following Figure 4B , Figure 5B , Figure 6 and Figure 7 The AR features are shown in more detail in .

[0074] Sample User Interface

[0075] Figure 4A-Figure 7 Example user interfaces are shown that may be presented on the rider client device 28 and / or the driver client device 10 via the ride sharing application 44 in both the map display mode and the AR mode. These are just a few example user interfaces that are included for ease of illustration only. Additional or alternative user interfaces may be included, as well as additional or alternative AR features presented in the user interfaces. For example, Figure 6 As shown, the driver's vehicle is highlighted by presenting a graphical indication of the vehicle above the driver's vehicle and a transparent color from the graphical indication to the driver's vehicle. In other implementations, the driver's vehicle can be highlighted with other AR features, such as a circle or rectangle around the driver's vehicle, a needle or arrow above the vehicle, rendering the vehicle in a solid bright color (such as yellow or pink, etc.).

[0076] In another example, when the camera view from the camera of the rider client device 28 or driver client device 10 does not include the pick-up location, the driver's vehicle, items, or riders, the user interface may include AR features that instruct the rider or driver to turn the rider client device 28 or driver client device 10 in a particular direction, or to tilt the rider client device 28 or driver client device 10 up or down, to view the pick-up location, the driver's vehicle, items, or riders in the camera view. These AR features may be arrows that point to the left or right to instruct the rider or driver to turn the rider client device 28 or driver client device 10 in a particular direction, or point up or down to instruct the rider or driver to tilt the rider client device 28 or driver client device 10 up or down.

[0077] in any case, Figure 4AAn example display 400 of a pick-up location in a map display mode of a ride-sharing application 44 is shown. The example display 400 may be presented on a rider client device 28 when a rider requests a ride service. The example display 400 includes a map display 410 of a geographic area that includes the rider's current location 402. The map display 410 also includes an indication of a pick-up location 404 in the form of a pin. The pick-up location is 915 Columbus Avenue in front of the Cobbs Comedy Club. As described above, the map display 410 may be presented when the rider's current location 402 is at least a threshold distance (e.g., 50 meters) from the pick-up location 404.

[0078] Figure 4B The AR mode of the ride sharing application 44 is shown. Figure 4A The example display 450 may be presented on the rider client device 28 when the rider's current location is within a threshold distance from the pickup location. The example display 450 includes a camera view 460 of a real-world image in front of the rider client device 28. The camera view 460 includes streets, trees, vehicles, buildings, etc. The example display 450 also includes an AR feature 462 overlaid on the real-world image at a location in the camera view 460 that corresponds to the pickup location. Figure 4A In that case, the indication of the pick-up location is pin 462 .

[0079] Figure 5A Another example display 500 of a pick-up location in a map display mode of a ride-sharing application 44 is shown. The example display 500 may be presented on the rider client device 28 when the rider requests a ride service. The example display 500 includes a map display 510 of a geographic area that includes the rider's current location 502. The map display 510 also includes an indication of a pick-up location 504 in the form of a pin. The pick-up location is 790 Lombard Street in front of Caffe Sapore. As described above, the map display 510 may be presented when the rider's current location 502 is at least a threshold distance (e.g., 50 meters) from the pick-up location 504.

[0080] Figure 5B The AR mode of the ride sharing application 44 is shown. Figure 5AThe example display 550 may be presented on the rider client device 28 when the rider's current location is within a threshold distance from the pickup location. The example display 550 includes a camera view 560 of a real-world image in front of the rider client device 28. The camera view 560 includes parking lots, vehicles, buildings, etc. The example display 550 also includes an AR feature 562 overlaid on the real-world image at a location in the camera view 560 that corresponds to the pickup location. Figure 5A In that case, the indication of the pick-up location is pin 562 .

[0081] Figure 6 An example display 600 of a driver's vehicle in an AR mode of a ride-sharing application 44 is shown. The example display 600 may be presented on the rider client device 28 when the rider's current location is within a threshold distance from a pickup location. The example display 600 includes a camera view 610 of a real-world image in front of the rider client device 28. The camera view 610 includes streets, vehicles, buildings, etc. The vehicle may include the driver's vehicle 602 that accepts the request to pick up the rider. An AR feature 604 is overlaid on the real-world image, which indicates the driver's vehicle 602. In the example display 600, the AR feature 604 is placed over a location in the camera view 610 corresponding to the driver's vehicle 602 to highlight the driver's vehicle 602. The AR feature 604 is a graphical indication that includes a description of the driver's vehicle 602, including the make, model, color, and license plate number, a photo of the driver's vehicle taken at a previous point in time, and a transparent color from the photo of the driver's vehicle to the driver's vehicle 602.

[0082] As shown in the camera view 610, the driver's vehicle 602 is currently moving because it appears to be turning right toward the rider client device 28. In order to highlight the moving vehicle, the augmented reality server 110 receives frequent updates of the driver's current location and determines the corresponding location at which to present the AR feature 604 based on the updated location information. The rider client device 28 receives the updated location for presenting the AR feature 604 and displays the AR feature 604 in the camera view 610 accordingly. In some embodiments, each time the camera view changes, the rider client device 28 can provide an updated camera view to the augmented reality server 110, and the augmented reality server 110 can analyze the updated camera view to identify the pick-up location, the driver's vehicle, or the rider in the camera view. For example, when the camera view does not include the pick-up location, the driver's vehicle, the item, or the rider, the user interface can include instructions for the rider to turn the rider client device 28 in a particular direction, or tilt the rider client device 28 up or down to view the AR feature of the pick-up location. As the rider turns the rider client device 28 , the rider client device 28 may provide an updated camera view to the augmented reality server 110 to determine whether the pick-up location is included in the camera view or whether the rider needs to continue turning.

[0083] Figure 7 Another example display 700 of a driver's vehicle in an AR mode of a ride-sharing application 44 is shown. The example display 700 may be presented on the rider client device 28 when the rider's current location is within a threshold distance from a pickup location. The example display 700 includes a camera view 710 of a real-world image in front of the rider client device 28. The camera view 710 includes streets, vehicles, buildings, etc. The vehicle may include the driver's vehicle 702 that accepts the request to pick up the rider. An AR feature 704 is overlaid on the real-world image, which indicates the driver's vehicle 702. In the example display 700, the AR feature 704 is placed over a location in the camera view 710 corresponding to the driver's vehicle 702 to highlight the driver's vehicle 702. The AR feature 704 is a graphical indication that includes a description of the driver's vehicle 702, including the make, model, color, and license plate number, a photo of the driver's vehicle taken at a previous point in time, and a transparent color from the photo of the driver's vehicle to the driver's vehicle 702.

[0084] Example method for providing display of a pickup location in a delivery service

[0085] Figure 8A flow chart of an example method 800 for providing a display of a pick-up location in a delivery service is shown. The method may be implemented with a set of instructions stored on a computer-readable memory and executable on one or more processors of an augmented reality server 110. For example, the method may be implemented by an AR generation engine 134.

[0086] At block 802, a rideshare or other transportation request including a current location and a destination of the rider is received from a rider client device 28. The augmented reality server 110 or rideshare server 130 may broadcast the request to drivers within a threshold distance of the rider's current location. A pick-up location is then determined and provided to the rider client device 28 and the driver client device 10 of the driver who accepted the request to transport the rider (block 804). The pick-up location may be within a threshold distance of the rider's current location. Additionally, the pick-up location may be selected from a list of predetermined pick-up locations stored in a database.

[0087] The rider client device 28 may then display an indication of the pick-up location and the rider's current location in a map display mode of the ride-sharing application 44. The rider client device 28 may then display a camera view in an AR mode depicting a real-world image in front of a camera of the rider client device 28 when the rider's current location is within a threshold distance of the pick-up location. At block 806, the augmented reality server 110 receives the camera view or image depicting the real-world image from the camera of the rider client device 28.

[0088] Thus, the augmented reality server 110 may provide information to be displayed in an AR mode overlaid with a real-world image to assist the rider or driver in finding the pick-up location. For example, the augmented reality server 110 may identify a location in the camera view that corresponds to the pick-up location (block 808). More specifically, for example, the augmented reality server 110 may compare the camera view to a number of template camera views stored in the real-world image database 136. Each template camera view or image may store an indication of the camera viewpoint or location at which the image was taken, an indication of the orientation of the camera when the image was taken, an indication of the zoom or scale level of the camera view, and / or an indication of a geographic area corresponding to the camera view including an indication of precise physical locations at various locations within the camera view based on the zoom or scale level.

[0089] When the augmented reality server 110 identifies a matching template camera view, the augmented reality server 110 determines the precise physical location of each location within the camera view and / or the camera viewpoint, orientation, scaling, or zoom level of the camera view based on the precise physical location and / or camera viewpoint, orientation, scaling, or zoom level of each location within the identified template camera view.

[0090] The augmented reality server 110 then identifies a location in the camera view that corresponds to the pick-up location by comparing the precise physical locations at various locations within the camera view to the pick-up location. At block 810, the augmented reality server 110 generates and provides an AR feature indicating the pick-up location to be presented at the identified location within the camera view of the rider client device 28. For example, the AR feature may be a pin that may be overlaid on the real-world image in the camera view.

[0091] In addition, at block 812, the augmented reality server obtains the current location of the driver's client device 10. In some embodiments, the augmented reality server 110 obtains continuous or periodic updates of the driver's current location. The frequency of updates may increase when the driver's current location is within a threshold distance of the pickup location.

[0092] In any case, the augmented reality server 110 identifies a location in the camera view that corresponds to the driver's vehicle (block 814). The augmented reality server 110 may identify a location corresponding to the driver's vehicle based on the driver's current location and / or an object identified as a vehicle in the camera view. At block 816, the augmented reality server 110 generates and provides an AR feature indicating the driver's vehicle to be presented at or above the identified location within the camera view of the occupant client device 28. The AR feature may be a graphical indication that may be overlaid on a real-world image in the camera view, for example, including a description of the driver's vehicle, a photo of the driver's vehicle taken at a previous point in time, and / or color.

[0093] Fig. 9 A flow chart of an example method 900 for identifying and presenting a pick-up location to a rider or driver in a transportation service is shown. The method may be implemented with a set of instructions stored on a computer-readable memory and executable on one or more processors of the rider client device 28. For example, the method may be implemented by the AR display module 48.

[0094] At block 902, the rider client device 28 sends a rideshare or other transportation request that includes the rider's current location and a destination. The augmented reality server 110 or rideshare server 130 may broadcast the request to drivers within a threshold distance of the rider's current location. A pick-up location is then determined and provided to the rider client device 28 and the driver client device 10 of the driver who accepted the request to transport the rider (block 904). The pick-up location may be within a threshold distance of the rider's current location. Additionally, the pick-up location may be selected from a list of predetermined pick-up locations stored in a database.

[0095] At block 906, the rider client device 28 presents a map display such as Figure 4A 4. The map display 410 is shown. The map display presents a geographic area including the rider's current location and the pickup location. The rider client device 28 then compares the rider's current location to the pickup location and determines whether the rider is within a threshold distance of the pickup location (block 908). If the rider is not within the threshold distance of the pickup location, the rider client device 28 continues to present the map display (block 906). Once the rider is within the threshold distance of the pickup location, the rider client device 28 switches to AR mode (block 910) and presents a camera view of a real-world image depicted in front of the camera of the rider client device 28. Thus, the rider client device 28 can present information overlaid on the real-world image in AR mode to assist the rider in finding the pickup location.

[0096] The rider client device 28 sends a camera view or image depicting a real-world image from a camera of the rider client device 28 to the augmented reality server 110 (block 912). In response, the augmented reality server 110 compares the camera view with several template camera views stored in the real-world image database 136, for example.

[0097] When the augmented reality server 110 identifies a matching template camera view, the augmented reality server 110 determines the precise physical location of each location within the camera view and / or the camera viewpoint, orientation, scaling, or zoom level of the camera view based on the precise physical location and / or camera viewpoint, orientation, scaling, or zoom level of each location within the identified template camera view.

[0098] The augmented reality server 110 then identifies AR features to overlay on the real-world image in the camera view. More specifically, the augmented reality server 110 can determine whether the pick-up location is included in the camera view based on the precise physical locations at various locations within the camera view. The augmented reality server 110 can also determine whether the driver's vehicle is included in the camera view based on the current location of the driver's client device 28, the precise physical locations at various locations within the camera view, and / or whether there is an object identified as a vehicle in the camera view. AR features can include shapes, colors, and graphics that can be placed on or above a specific vehicle, person, or location that indicates a pick-up location or highlights the driver's vehicle or a person who is a passenger in the camera view. In some implementations, the AR feature can also indicate the direction in which the passenger or driver needs to turn to see the vehicle or passenger.

[0099] At block 914, the rider client device 28 receives the AR feature and the corresponding location at which the AR feature is overlaid on the real-world image in the camera view. The rider client device 28 then presents the AR feature overlaid on the real-world image in the camera view (block 916).

[0100] Additional considerations

[0101] The following additional considerations apply to the foregoing discussion. In this description, multiple instances may implement components, operations or structures described as single instances. Although the individual operations of one or more methods are illustrated and described as individual operations, one or more individual operations may be performed simultaneously, and the operations are not required to be performed in the illustrated order. The structures and functions presented as individual components in the example configuration may be implemented as combined structures or components. Similarly, the structures and functions presented as individual components may be implemented as individual components. These and other variations, modifications, additions and improvements fall within the scope of the subject matter of the present disclosure.

[0102] In addition, certain embodiments are described herein as including logic or multiple components, modules, or mechanisms. Modules may constitute software modules (e.g., code stored on a machine-readable medium) or hardware modules. A hardware module is a tangible unit capable of performing certain operations and may be configured or arranged in a certain manner. In example embodiments, one or more computer systems (e.g., stand-alone, client, or server computer systems) or one or more hardware modules (e.g., a processor or a group of processors) of a computer system may be configured by software (e.g., an application or application portion) as a hardware module that operates to perform certain operations as described herein.

[0103] In various embodiments, hardware modules may be implemented mechanically or electronically. For example, a hardware module may include dedicated circuits or logic that are permanently configured (e.g., as a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC)) to perform certain operations. A hardware module may also include programmable logic or circuits (e.g., contained in a general purpose processor or other programmable processor) that are temporarily configured by software to perform certain operations. It will be appreciated that the decision to implement a hardware module mechanically in a dedicated and permanently configured circuit or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations.

[0104] Therefore, the term hardware should be understood to include a tangible entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a certain manner or perform certain operations described herein. As used herein, "hardware-implemented modules" refer to hardware modules. Considering embodiments in which hardware modules are temporarily configured (e.g., programmed), each hardware module does not need to be configured or instantiated at any one time. For example, in the case where the hardware module includes a general-purpose processor configured using software, the general-purpose processor can be configured as corresponding different hardware modules at different times. The software can configure the processor accordingly, for example, to constitute a specific hardware module at one time and to constitute different hardware modules at different times.

[0105] Hardware modules can provide information to other hardware and receive information from other hardware. Therefore, the described hardware modules can be considered to be communicatively coupled. When multiple such hardware modules exist simultaneously, communication can be achieved by signal transmission connecting the hardware modules (e.g., through appropriate circuits and buses). In embodiments in which multiple hardware modules are configured or instantiated at different times, for example, communication between such hardware modules can be achieved by storing and retrieving information in a memory structure accessible to multiple hardware modules. For example, a hardware module can perform an operation and store the output of the operation in a memory device that is communicatively coupled to it. Then, another hardware module can access the memory device at a later time to retrieve and process the stored output. Hardware modules can also initiate communication with input or output devices and can operate on resources (e.g., collections of information).

[0106] Methods 800 and 900 may include one or more functional blocks, modules, separate functions or routines in the form of tangible computer executable instructions stored in a non-transitory computer readable storage medium and executed using a processor of a computing device (e.g., a server device, a personal computer, a smart phone, a tablet computer, a smart watch, a mobile computing device, or other client computing device, as described herein). Methods 800 and 900 may include being part of any backend server (e.g., an augmented reality server, a ride sharing server, a map data server, a navigation server, or any other type of server computing device, as described herein), a client computing device module such as an example environment, or being part of a module external to such an environment. Although the figures may be described with reference to other figures for ease of explanation, methods 800 and 900 may be used with other objects and user interfaces. In addition, although the above explanation describes the steps of methods 800 and 900 performed by a specific device (such as an augmented reality server 110, a driver client device 10, or a passenger client device 28), this is for illustrative purposes only. The blocks of methods 800 and 900 may be performed by one or more devices or other parts of the environment.

[0107] The various operations of the example methods described herein may be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the related operations. Whether temporarily configured or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions. In some example embodiments, the modules mentioned herein may include processor-implemented modules.

[0108] Similarly, the methods or routines described herein may be implemented at least in part by a processor. For example, at least some of the operations of the method may be performed by one or more processors or a hardware module implemented by the processor. The execution of certain operations may be distributed in one or more processors, not only residing in a single machine, but also deployed on multiple machines. In some example embodiments, one or more processors may be located in a single location (e.g., in a home environment, in an office environment, or as a server cluster), while in other embodiments, the processor may be distributed in multiple locations.

[0109] One or more processors may also operate in a "cloud computing" environment or as a SaaS to support execution of the relevant operations. For example, as indicated above, at least some operations may be performed by a set of computers (as an example of a machine including a processor) accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs).

[0110] Furthermore, the drawings depict some embodiments of the example environment for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein.

[0111] Upon reading this disclosure, those skilled in the art will understand additional alternative structural and functional designs for providing a display of pick-up locations in a ride-sharing service through the principles disclosed herein. Therefore, although specific embodiments and applications have been shown and described, it should be understood that the disclosed embodiments are not limited to the precise structures and components disclosed herein. It will be clear to those skilled in the art that various modifications, changes and variations can be made in the arrangement, operation and details of the methods and apparatus disclosed herein without departing from the spirit and scope defined in the appended claims.

Claims

1. A method for providing a display of a pick-up location in a delivery service, the method comprising: Receiving, by one or more processors in the server device, a request to transport a rider to a delivery location; providing, by the one or more processors, to the client device of the rider or the client device of the driver a pick-up location for the driver to pick up the rider; At the one or more processors, receiving a real-world image from a field of view of a vehicle occupant's client device or a driver's client device; determining, by the one or more processors, a location corresponding to the real-world image; comparing, by the one or more processors, the determined location to a pickup location; as well as providing, by the one or more processors, information to be displayed in an augmented reality mode to assist an occupant or driver based on the comparison, the information overlaying a real-world image presented on the occupant's client device or the driver's client device; determining, by the one or more processors, an orientation of a client device of an occupant of the vehicle; obtaining, at the one or more processors, a current location of the occupant and a current location of the driver; identifying, by the one or more processors, a heading of the driver relative to the occupant based on the driver's current location relative to the occupant's current location; as well as determining, by the one or more processors, that an orientation of the occupant's client device is different than an orientation of the driver relative to the occupant; Wherein providing information to be displayed in the augmented reality mode overlaying the real-world image includes providing, by the one or more processors, an instruction for the occupant to turn to face the direction of the driver and displaying the driver's vehicle within the real-world image.

2. The method according to claim 1, wherein: Providing information to be displayed in the augmented reality mode includes providing, by the one or more processors, an indication of a pick-up location within the real-world image.

3. The method according to claim 2, wherein: Providing an indication of the pick-up location within the real-world image includes providing a pin at the pick-up location within the real-world image.

4. The method according to claim 1, further comprising: determining, by the one or more processors, that the real-world image includes a driver's vehicle; Wherein providing information to be displayed in the augmented reality mode includes highlighting, by the one or more processors, a vehicle within the real-world image.

5. The method according to claim 4, wherein: Determining that a real-world image includes a driver's vehicle includes: receiving, at the one or more processors, an indication of a current location of a driver; determining, by the one or more processors, a geographic area included within the real-world image; The indication of the driver's current location is compared, by the one or more processors, to the geographic area to determine whether the driver is located within the geographic area included in the real-world image.

6. The method according to claim 5, further comprising: identifying, by the one or more processors, an object at a current location of a driver within the real-world image; comparing, by the one or more processors, the object to one or more template objects representing vehicles to determine whether the object is a vehicle; as well as In response to determining that the object is a vehicle, determining, by the one or more processors, that the real-world image includes the driver's vehicle.

7. The method according to claim 1, wherein: Determining locations that correspond to real-world images involves: The real-world image provided by the client device is analyzed to determine the location of the client device.

8. A method for identifying and presenting a pick-up location to a rider in a transportation service, the method comprising: sending, by one or more processors in the user's client device, a request to transport the user to a delivery location; receiving, at the one or more processors, a pick-up location for a vehicle to pick up a user; providing, via the user interface, in a map display mode, an interactive digital map of the geographic area including an indication of the pickup location and an indication of the user's current location; receiving, at the one or more processors, a current location of a driver of a vehicle periodically at a first frequency; as well as In response to determining that the user's current location is within a threshold distance of the pickup location, periodically receiving, at the one or more processors, a current location of a driver at a second frequency higher than the first frequency; Switching to an augmented reality mode via the user interface to assist the user, the augmented reality mode presenting a real-world image in the field of view of the client device along with information overlaying the real-world image.

9. The method according to claim 8, further comprising: receiving, at the one or more processors, a current location of a driver of a vehicle periodically at a first frequency; as well as In response to determining that the driver's current location is within a threshold distance of the pick-up location, the driver's current location is periodically received at the one or more processors at a second frequency that is higher than the first frequency.

10. The method according to claim 9, further comprising: providing, by the one or more processors, a real-world image and an updated current location of the driver to a server device; as well as receiving, at the one or more processors, a real-world image including a driver's vehicle and an indication of a location of the vehicle within the real-world image; Wherein switching to an augmented reality mode that presents information overlaid on a real-world image to assist a user includes highlighting, by the one or more processors, a vehicle within the real-world image.

11. The method according to claim 8, wherein: Switching to an augmented reality mode that presents information overlaid on the real-world image to assist the user includes providing, by the one or more processors, an indication of the pick-up location within the real-world image.

12. The method according to claim 11, wherein: Providing an indication of the pick-up location within the real-world image includes providing a pin at the pick-up location within the real-world image.

13. The method according to claim 8, further comprising: determining, by the one or more processors, an orientation of a client device of a user; as well as obtaining, by the one or more processors, an indication of an orientation of the vehicle relative to the user based on a current location of the vehicle relative to a current location of the user; Wherein, switching to an augmented reality mode that presents information overlaying a real-world image to assist the user includes presenting, by one or more processors, overlaying the real-world image with instructions for the user to turn to face the vehicle so that the vehicle appears within the real-world image.

14. A method for identifying and presenting a pick-up location to a driver in a delivery service, the method comprising: Receiving, by one or more processors in the driver's client device, a request to transport the rider to a drop-off location; Accept the request; receiving a pick-up location at the one or more processors; providing, via the user interface, in a map display mode, an interactive digital map of the geographic area including navigation directions from the driver's current location to the pickup location; receiving, at the one or more processors, a current location of an occupant of a vehicle periodically at a first frequency; In response to determining that the driver's current location is within a threshold distance of the pickup location, periodically receiving, at the one or more processors, a current location of an occupant at a second frequency higher than the first frequency; Switching to an augmented reality mode via the user interface to assist the driver presents a real-world image in the field of view of the client device along with information overlaying the real-world image.

15. The method according to claim 14, further comprising: receiving, at the one or more processors, periodically at a first frequency, a current location of a rider requesting transportation; as well as In response to determining that the current location of the rider is within a threshold distance of the pick-up location, the current location of the rider is periodically received at the one or more processors at a second frequency that is higher than the first frequency.

16. The method according to claim 15, further comprising: providing, by the one or more processors, a real-world image and an updated current location of an occupant or item to a server device; as well as receiving, at the one or more processors, an indication that the real-world image includes an occupant or an item and a location of the occupant within the real-world image; Wherein switching to an augmented reality mode that presents information overlaid on a real-world image to assist the driver includes highlighting, by the one or more processors, occupants or objects within the real-world image.

17. The method according to claim 14, wherein: Switching to an augmented reality mode that presents information overlaid on a real-world image to assist the driver includes providing, by the one or more processors, an indication of a pick-up location within the real-world image.

18. The method according to claim 17, wherein: Providing an indication of the pick-up location within the real-world image includes providing a pin at the pick-up location within the real-world image.

19. The method according to claim 14, further comprising: determining, by the one or more processors, a heading of a driver's client device; as well as obtaining, by the one or more processors, an indication of an orientation of the occupant or item relative to the driver based on the current location of the occupant or item relative to the current location of the driver; Among them, switching to an augmented reality mode that presents information overlaying a real-world image to assist the driver includes one or more processors overlaying the real-world image with instructions for the driver to turn to face a passenger or object so that the passenger or object appears within the real-world image.

Images