Creating virtual parallax for a 3D appearance

Abstract

The present disclosure relates to "Creating Virtual Parallax for a Three-Dimensional Appearance". In some embodiments, a computing device may simulate virtual parallax to create a three-dimensional effect. For example, the computing device may obtain an image captured at a particular location. The captured two-dimensional image may be applied as a texture to a three-dimensional model of the capture location. In order to give the two-dimensional image a three-dimensional look and feel, the computing device may simulate moving the camera used to capture the two-dimensional image to different locations around the image capture location to generate different perspectives of the textured three-dimensional model, as if captured by multiple different cameras. Thus, the virtual parallax may be introduced into the generated image of the capture location. When the generated image is presented to a user on a display of the computing device, the generated image may have a three-dimensional look and feel even though it was generated from a single two-dimensional image.

Description

[0001] This application is a divisional application of invention patent application 202080039731.0, filed on May 28, 2020, and entitled “Creating Virtual Parallax for Three-Dimensional Appearance”. Technical Field

[0002] The present disclosure generally relates to generating realistic three-dimensional street scenes. Background Art

[0003] Navigation applications are common. Users of computing devices invoke navigation applications to present maps that display representations of streets, buildings, landmarks, and other points of interest. Users can use navigation applications to determine routes to various destinations and receive instructions for navigating the selected routes. However, sometimes users need more context for navigation. Therefore, in some navigation applications, users can provide input to cause the computing device to present a photo of the destination, point of interest (POI), or other location on a map. However, simple two-dimensional images may not provide sufficient context to help users navigate. Summary of the Invention

[0004] In some implementations, when presenting a map view on a display of a computing device, the computing device can present a dynamic street scene overlay. The dynamic street scene can be presented so that a user can clearly view the dynamic street scene overlay and the map view. The dynamic street scene can dynamically adjust in response to the user manipulating the map view to different positions. The dynamic street scene can be presented so that objects in the image of the dynamic street scene have a three-dimensional look and feel. The dynamic street scene can be presented so that the dynamic street scene does not block the user from viewing and interacting with the map view.

[0005] In some implementations, when presenting a three-dimensional animation on a display of a computing device, the computing device may synthesize images to improve image quality. For example, when presenting an image corresponding to a first location, the computing device may generate an animation based on images associated with a series of locations between the first location and a second location selected by the user, to generate a street scene animation that simulates moving from the first location to the second location. To generate the animation, the computing device may synthesize images captured at two different locations to generate an intermediate view associated with a location between the two different locations. The images may be synthesized in such a way that the good quality portions of each image are retained while removing the low quality portions of each image.

[0006] In some embodiments, a computing device may simulate virtual parallax to create a three-dimensional effect. For example, the computing device may obtain an image captured at a particular location. The captured two-dimensional image may be applied as a texture to a three-dimensional model of the capture location. To give the two-dimensional image a three-dimensional appearance and feel, the computing device may simulate moving the camera used to capture the two-dimensional image to different locations around the image capture location to generate different perspectives of the textured three-dimensional model, as if captured by multiple different cameras. Thus, the virtual parallax may be introduced into the generated image of the capture location. When the generated image is presented to a user on a display of the computing device, the generated image may have a three-dimensional appearance and feel, even though it was generated from a single two-dimensional image.

[0007] Certain embodiments provide at least the following advantages. A dynamic street scene overlay allows a user to interact with and / or view a realistic street-level view of a map location while viewing and / or interacting with a map depicting the corresponding map area or location. By accessing the dynamic street scene overlay and the map simultaneously, the user is able to better understand the context of a particular map location and reduce the likelihood that usage will be lost when using a corresponding navigation application. In addition, by selecting the best quality pixels from two different images depicting the same object and / or perspective when synthesizing an image, the quality of the resulting synthesized image can be improved, thereby improving the user experience when interacting with such synthesized image. By introducing a parallax effect into a single two-dimensional image based on the virtual position of a virtual image capture device, a user device can create a three-dimensional effect in a two-dimensional image without the expense of multiple real-life image capture devices and without the use of computing resources required to process and / or combine multiple images from multiple image capture devices (e.g., cameras).

[0008] The details of one or more specific implementations are set forth in the following figures and detailed description. Other features, aspects, and potential advantages will be apparent from the detailed description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram of an exemplary system for presenting dynamic street scene overlays, synthesizing images to improve image quality, and / or introducing virtual parallax to create a three-dimensional appearance.

[0010] Figure 2 An exemplary graphical user interface for presenting a map view on a display of a user device is shown.

[0011] Figure 3 An exemplary graphical user interface for presenting a dynamic street scene overlay is shown.

[0012] Figure 4is a diagram showing changes to the GUI when navigating a map view.

[0013] Figure 5 An exemplary graphical user interface for presenting a dynamic street scene overlay associated with a selected point of interest is shown.

[0014] Figure 6 An exemplary graphical user interface for presenting a minimized dynamic street scene overlay is shown.

[0015] Figure 7 An exemplary graphical user interface for presenting a maximized view of a sign is shown.

[0016] Figure 8 An exemplary graphical user interface for simultaneously presenting a maximized view of signage and a dynamic street scene overlay is shown.

[0017] Figure 9 An exemplary graphical user interface for presenting a maximized view of a dynamic street scene overlay is shown.

[0018] Figure 10 An exemplary graphical user interface for selecting a point of interest from a dynamic street scene overlay is shown.

[0019] Figure 11 is an illustration of an exemplary process for synthesizing images to improve image quality.

[0020] Figure 12 is a conceptual illustration of how the intermediate views are generated.

[0021] Figure 13 is an illustration of the process for synthesizing the converted images into intermediate views.

[0022] Figure 14 is a system interaction diagram illustrating an exemplary process for obtaining capture point data when generating a realistic animation that simulates moving a virtual device from one image capture point to another.

[0023] Figure 15 A conceptual illustration of generating virtual parallax to create a three-dimensional effect.

[0024] Figure 16 is a flowchart of an example process for rendering a dynamic street scene overlay.

[0025] Figure 17 is a flow chart of an exemplary process for synthesizing images to improve image quality.

[0026] Figure 18 is a flow chart of an exemplary process for generating virtual parallax to create a three-dimensional appearance.

[0027] Figure 19 It is achievable Figures 1 to 18 A block diagram of an exemplary computing device illustrating features and processes.

[0028] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION

[0029] Figure 1 is a block diagram of an exemplary system 100 for presenting dynamic street scene overlays, synthesizing images to improve image quality, and / or introducing virtual parallax to create a three-dimensional appearance.

[0030] System 100 may include a user device 102. For example, user device 102 may be a computing device such as a laptop, tablet, mobile device, smartphone, smartwatch, or other computing device.

[0031] User device 102 may include a navigation application 104. For example, navigation application 104 may present maps, routes, location metadata, and / or images (e.g., captured photos) associated with various geographic locations, points of interest, and the like. Navigation application 104 may obtain map data from a navigation server (e.g., navigation server 120), which may include data defining maps, map objects, routes, points of interest, images, and the like. For example, map data may be received as map tiles, each of which includes map data for the geographic area corresponding to the respective map tiles. Among other things, map data may include data defining roads and / or road segments, metadata for points of interest and other locations, three-dimensional models of buildings, infrastructure, and other objects found at various locations, and / or images captured at various locations. Navigation application 104 may request map data (e.g., map tiles) associated with locations frequently visited by user device 102 from a server device via a network 110 (e.g., a local area network, a cellular data network, a wireless network, the Internet, a wide area network, etc.). Navigation application 104 may store the map data in a map database 106. The navigation application 104 may use map data stored in the map database 106 and / or other map data received from the server device 120 to provide the navigation application features described herein (e.g., dynamic street scene overlays, composite images to improve image quality, and / or introducing virtual parallax to create a three-dimensional effect).

[0032] In some implementations, system 100 may include a server device 120. For example, as described herein, server device 120 may be one or more computing devices configured to store, generate, and / or provide map data to various user devices. For example, the functionality described herein with reference to server device 120 may be performed by a single computing device or may be distributed across multiple computing devices.

[0033] In some implementations, the server device 120 may include a navigation server 122. For example, the navigation server 122 may be a software server configured to obtain, generate, and / or store map data. For example, the navigation server 122 may obtain a lidar-generated point cloud (e.g., points defining locations on the surface of an object near the image capture location) for each location included in the map data. The navigation server 122 may use the corresponding point cloud for each location to generate a three-dimensional model (e.g., a three-dimensional mesh) for each of the locations. The navigation server 122 may obtain images captured at each location (e.g., a capture location) and use the images to add textures to the three-dimensional model, thereby generating a photorealistic three-dimensional image representing the corresponding location. For example, a captured image (e.g., a photograph, a panoramic photo, etc.) may be stretched on the surface of the three-dimensional model for a particular location to generate a photorealistic three-dimensional view of the particular location. The three-dimensional model and textures (e.g., the captured image, the stretched image, the image applied to the three-dimensional model, etc.) may be stored in a map database 124 on the server device 120 and provided to user devices (e.g., user device 102) to provide the various features and functionality described herein. As may be described below, the navigation server 122 may be configured to obtain, generate, and / or store other map data in the map database 124 .

[0034] Figure 2 An exemplary graphical user interface 200 is shown for presenting a map view on a display of a user device 102. For example, the graphical user interface (GUI) 200 may be generated by a navigation application 104 based on map data (eg, map tiles) stored in a map database 106.

[0035] In some implementations, the GUI 200 can include a map view 202. For example, the map view 202 can include a map that presents graphical representations of roads, buildings, points of interest (e.g., POIs 204-210), and / or other map data. While the map view 202 can present three-dimensional representations of objects (such as buildings, bridges, or other landmarks), the map view 202 does not include a photorealistic view of the various locations, points of interest, and the like presented by the map in the map view 202.

[0036] In some implementations, the GUI 200 may include a graphical object 212 for entering search criteria to find a place or address. For example, a user may type the name of a place (e.g., a business, a landmark, etc.) or an address in a text input box 214, causing the navigation application 104 to initiate a search for the place or address specified by the user. For example, the navigation application 104 may search the map database 106 for locations (e.g., places) that match the search criteria. The navigation application 104 may send a request to the navigation server 122, causing the navigation server 122 to search for locations that match the search criteria. After obtaining map data corresponding to the search criteria, the navigation application 104 may present a list of places that match the search criteria, and the user may select one of the places to have the place (e.g., an address, a point of interest, a landmark, etc.) presented on the map view 202.

[0037] In some implementations, the GUI 200 may include a graphical object 216 for invoking a dynamic street scene overlay. For example, the navigation application 104 may enter a dynamic street scene mode in response to receiving a user selection of the graphical object 216 and present a reference to the following. Figure 3 Described dynamic street scene overlay.

[0038] Figure 3 An exemplary graphical user interface 300 for presenting a dynamic street scene overlay is shown. For example, as described above, the GUI 300 can be presented by the navigation application 104 in dynamic street scene mode in response to receiving user input selecting the graphical object 216. The GUI 300 can be presented when no point of interest is currently selected. For example, the navigation application 104 can enter the dynamic street scene mode in response to receiving a user selection of the graphical object 216.

[0039] In some implementations, the GUI 300 can include a dynamic street scene overlay 302. For example, the overlay 302 can be a graphical object (e.g., a window, a view, a card, a graphical user interface element, etc.) that is presented on the map view 202. The overlay 302 can be presented by the navigation application 104 in response to receiving user input selecting the graphical object 216. For example, the overlay 302 can be animated to slide across the map view 202. For example, the overlay 302 can be animated to slide downward from the top of the GUI 300, as indicated by arrow 306. The size and position of the overlay 302 can be set so that when the user views the GUI 300, the user clearly sees the overlay 302 (and the image presented by the overlay 302) and the portion of the map view 202 that includes the currently selected location and the surrounding area. For example, on some displays, the size and position of the overlay 302 can be set so that the overlay 302 occupies approximately the top half of the display (e.g., 40%-60%), while the portion of the map view 202 including the currently selected location covers approximately the other half of the display (e.g., 40%-60%).

[0040] In some implementations, overlay 302 can include image 304. For example, image 304 can be a dynamic, interactive image of a map location corresponding to graphical object 312. Image 304 can be an image or a combination of multiple images (e.g., a panoramic image) captured at the map location identified or indicated by graphical object 312 presented on map view 202. Image 304 can represent or correspond to a particular perspective (e.g., scene, view, etc.) from the identified map location, as indicated by a viewing cone 314 extending from graphical object 312. Navigation application 104 can obtain map data for generating GUI 300 (and other GUIs described herein), including location metadata, imagery, three-dimensional models, etc., from map database 106 stored on user device 102 and / or from server device 120.

[0041] In some implementations, the overlay 302 can present a dynamic street scene. For example, when the navigation application 104 receives user input to the overlay 302, the navigation application 104 can dynamically change the image 304 to present a different perspective (e.g., a different view, scene, etc.) of the location corresponding to the graphical object 312. For example, when user input (e.g., a swipe touch input gesture) is received through the overlay 302, the navigation application 104 can pan the image 304 left, right, up, or down to present a different perspective of the current location of the graphical object 312 corresponding to the received input. In coordination with the change in perspective, the viewing cone 314 can rotate around the graphical object 312 to indicate the change in perspective or viewing direction represented by the image 304.

[0042] In some implementations, the overlay 302 can use virtual parallax to present the image 304 to create a three-dimensional effect. For example, as described below, the navigation application 104 can use the virtual parallax technology disclosed herein to present the image 304 with a three-dimensional look and feel.

[0043] In some implementations, user input received via overlay 302 can cause the navigation application 104 to present a new location on the map view 202 and / or overlay 302. For example, a user can provide input (e.g., a tap) selecting a distant location represented in image 304 to cause the navigation application 104 to move the graphical object 312 from its current location to a new location corresponding to the user input. A different image 304 can be presented in overlay 302 representing the new location. Thus, the map view 202, the location corresponding to the graphical object 312, and the image 304 can change based on the user input received via overlay 302.

[0044] In some implementations, the GUI 300 can include a graphical object 312 that represents the current dynamic street scene location. For example, the graphical object 312 can have an appearance (e.g., an icon, an image, etc.) that represents or indicates that the navigation application 104 is currently in dynamic street scene mode. When the GUI 300 is presented, the graphical object 312 can initially be located at the street-level image capture point (e.g., a map location corresponding to the real-world location where the street-level image was captured) near the center of the map view 202. The map within the map view 202 can move to present a map area surrounding the initial location, where the initial location is centered on the area in the lower portion of the GUI 300 that is not covered by the overlay 302.

[0045] In some implementations, although the graphical object 312 can be positioned at different locations relative to the map view 202, the graphical object 312 can remain at a static location in the map view 202 and / or the GUI 300. For example, the graphical object 312 will remain stationary as the map presented by the map view 202 moves beneath the graphical object 312 to place the graphical object 312 at a new map location. The graphical object 312 can, for example, remain in a centered location (e.g., horizontally and / or vertically) within the map view 202 as the map moves beneath the graphical object 312.

[0046] In some implementations, the navigation application 104 can receive user input regarding the map view 202 to change the current location presented on the GUI 300. For example, the user can provide input 316 (e.g., touch input) to the map view 202 to move, scroll, or change a portion of the map presented by the map view 202. For example, the navigation application 104 can detect user input 316 (e.g., touch and drag input) indicating that the user wishes to move the map left, right, up, or down. As the map is moved, the position of the graphical object 312 can change relative to the underlying map. When the graphical object 312 is above the desired location on the map, the user can cease user input. When the navigation application 104 detects the cessation of user input, the navigation application can update the overlay 302 with a different image 304 that represents the current map location corresponding to the graphical object 312 and / or the viewing cone 314.

[0047] In some implementations, the viewing cone 314 can, by default, point toward the top of the GUI 300. In some implementations, the viewing cone 314 can point in the direction of map movement that resulted in the current location being selected for presentation on the overlay 302. For example, if user input caused the graphical object 312 to move eastward on the map to its current map location, the viewing cone 314 can point eastward, and the overlay 302 can present a corresponding image 304 of an eastward view from the current location.

[0048] In some implementations, the overlay 302 can include a graphical object 310 for dismissing the overlay 302. For example, the navigation application 104 can hide the overlay 302 and present the GUI 200 in response to receiving a user selection of the graphical object 310.

[0049] In some implementations, the overlay 302 can include a graphical object 308 for switching between a partial screen version and a full screen version of the overlay 302. For example, when Figure 3 When in the partial screen view shown, receiving user input selecting the graphical object 308 may cause the navigation application 104 to present the following Figure 10 A full screen version of the overlay 302 is shown. Figure 10 When in the full screen view shown, receiving user input selecting the graphical object 308 may cause the navigation application 104 to present the following Figure 3 A partial screen version of overlay 302 is shown.

[0050] Figure 44 is an illustration 400 that represents changes to the GUI 300 when navigating the map view 202. For example, the illustration 400 includes device representations 402, 404, and 406 that each correspond to a single user device 102. Each of the device representations 402, 404, and 406 presents the GUI 300 as described above and presents a different portion of the map 408 in each map view 202.

[0051] Device representation 402 may represent a starting point or location presented by GUI 300. For example, graphical object 312 in device representation 402 may correspond to the starting location on map 408. As described above, image 410 (e.g., image 304) may present a realistic representation (e.g., a photograph, a realistic three-dimensional model, etc.) of the starting location. When a user provides input to map view 202 or overlay 302 to move map 408, the portion of map 408 presented in map view 202 may change (e.g., the map may scroll within map view 202 based on the user input), however, graphical object 312 will remain at its relative position within the graphical user interface and / or device display as the map is adjusted.

[0052] In some implementations, the representation of the graphical object 312 can be adjusted to indicate that a map manipulation is in progress. For example, the graphical object 312 can be enlarged, shaded, etc. so that the graphical object 312 appears to be rising or elevated from the map 408 in the map view 202, thereby indicating that the location of the graphical object 312 has not yet been selected. Additionally, as shown in the device representation 404, the viewing cone 314 can be removed. For example, because the graphical object 312 is animated to rise from the map 408 when the map 408 begins to move, the viewing cone 314 can be animated to retract back into the graphical object 312.

[0053] When manipulating a map, a graphical object 312 may be associated with a map location where an image capture point is unavailable (e.g., a location where a street scene image was captured). For example, as shown in device representation 404, the graphical object 312 may be presented above a building or other off-street location where street-level imagery is unavailable. To indicate that street-level imagery and / or map data for the current location of the graphical object 312 is unavailable, the appearance of the graphical object 312 may be dimmed or otherwise changed. Similarly, to indicate that street-level imagery and / or map data for the current location of the graphical object 312 is unavailable, the overlay 302 may be dimmed. In some implementations, when the overlay 302 is dimmed, the street-level image 412 may not be presented in the overlay 302. In some implementations, when the overlay 302 is dimmed, the street-level image 410 of the previous location may be presented in the overlay 302, or some default image 412 may be presented.

[0054] Once the adjustment of the map 408 in the map view 202 is complete, the graphical object 312 can return to its smaller size, so that the graphical object 312 appears to have been lowered back onto the map 402 in the map view 202, as shown in the device representation 406. Furthermore, when the navigation application 104 detects that the user input manipulating the map 408 has ceased, the navigation application 104 can determine the map location corresponding to the graphical object 312, as shown in the device representation 406, and present an image 414 corresponding to the determined map location in the undimmed overlay 302.

[0055] If the graphical object 312 is located over a map location where no street-level imagery exists (e.g., the graphical object 312 is not over the image capture point), the navigation application 104 can move the map 408 so that the graphical object 312 is placed over the nearest image capture point. If the image capture point is not within a threshold distance of the graphical object 312, the graphical object 312 can remain in an increased dimmed state (e.g., dimmed to 40%, 35%, etc.) until the user provides additional input to move the map 408 so that the graphical object 312 is located at or near the street-level image capture point (e.g., a location with map data that includes captured street-level imagery).

[0056] Figure 5 An exemplary graphical user interface 500 is shown for presenting a dynamic street scene overlay associated with a selected point of interest. For example, the graphical user interface (GUI) 500 can be presented by the navigation application 104 on a display of the user device 102. The GUI 500 can be presented when the navigation application 104 receives user input selecting a point of interest (e.g., POI 210) presented by the GUI 300 in the dynamic street scene mode.

[0057] In some implementations, when the navigation application 104 receives a selection to invoke a dynamic street scene mode (e.g., regarding a point of interest (e.g., POI 210)) when the point of interest (e.g., POI 210) is selected (e.g., has been selected), the navigation application 104 may further include a navigation window. Figure 2 GUI 500 may be presented upon user input from a navigation application 104 (e.g., as described herein). For example, navigation application 104 may receive user input selecting POI 210 when presenting GUI 200, and then receive user input selecting graphical object 215 to invoke dynamic street scene mode. Alternatively, navigation application 104 may receive user input to graphical object 212 (e.g., a search control) specifying search parameters for a desired point of interest. Navigation application 104 may receive a user selection of a point of interest from the search results, and then receive a user selection of graphical object 216 to enter dynamic street scene mode with respect to the point of interest (e.g., POI 210).

[0058] In some implementations, the navigation application 104 can modify the appearance of the selected POI 210 and the graphical objects associated therewith on the GUI 300 (e.g., highlight, enlarge, change color, etc.) so that the user can quickly visually distinguish the selected POI 210 from other points of interest (e.g., POI 208) presented on the GUI 500.

[0059] In some implementations, the GUI 500 can present a graphical object 312 indicating a street-level image capture point associated with the selected point of interest. For example, the GUI 500 can present the graphical object 312 at a map location corresponding to the real-world location at which the image for the selected POI 210 was captured. The GUI 500 can present a viewing cone 314 pointing toward the selected POI 210 in association with the graphical object 312 to indicate the relative location of the image capture point and the selected POI 210 and the angle at which the POI image was captured. The overlay 302 can initially present an image 304 (e.g., a photograph, a photorealistic image, etc.) representing the POI 210 from the map location of the graphical object 312 at an angle indicated by the viewing cone 314. The user can then select the image 304 as described above with reference to FIG. Figure 3 The user interacts with the image 304. For example, the user can manipulate (eg, pan) or interact with the image 304 to cause the map view 202 to present a different location (eg, a different POI).

[0060] In some implementations, the GUI 500 can present a placecard 502. For example, the placecard 502 can be presented at a minimum size and can present information related to the selected POI 210. The POI information can include a POI name (e.g., the name of a restaurant, building, park, bridge, landmark, etc.), a POI description (e.g., a POI type, such as a restaurant, landmark, park, etc.), a POI distance (e.g., from the current location of the user device 102 to the selected POI), and / or other POI information. Figure 5 As shown, the label 502 is in a minimized state at the lower portion of the GUI 500 .

[0061] Figure 6An exemplary graphical user interface 500 for presenting a minimized dynamic street scene overlay is shown. For example, the GUI 500 can be presented by the navigation application 104 in response to receiving user input to expand the sign 502 to a medium size and place it in a central display position on the GUI 500. For example, the user can provide input to drag the top edge of the minimized sign 502 presented on the GUI 500 upward (e.g., as indicated by the dashed arrow 602) to the central display position to expand the sign 502 and cause the sign 502 to present additional information (e.g., a picture, image, selectable graphical object, etc.) related to the selected POI 210.

[0062] In some implementations, the navigation application 104 can minimize the overlay 302 to make room for the expansion of the sign 502. For example, in response to receiving user input to expand the sign 502, the navigation application 104 can reduce the size of the overlay 302 and position the overlay 302 in an upper corner (e.g., upper right corner, upper left corner, etc.) of the GUI 600, as shown by minimized overlay 604. As described above, the overlay 604 can present the same image 304 for the selected POI 210, albeit in a smaller form. The navigation application 104 can present an animation that causes the larger overlay 302 to appear to shrink to the size of the smaller overlay 604. The animation can be synchronized with or coincide with the expansion of the sign 502.

[0063] In some implementations, the navigation application 104 can move the map in the map view 202 so that the selected POI 210 is not covered by the enlarged sign 502. For example, while the sign 502 is enlarged and the overlay 302 is minimized, the navigation application 104 can move the map toward the top of the GUI 600 so that the selected POI 210, graphical object 312, and / or viewing cone 314 are not covered or obscured by the sign 502 being enlarged upward into the GUI 600.

[0064] In some implementations, when the sign 502 is minimized, the navigation application 104 can enlarge the overlay 604. For example, the navigation application 104 can receive user input that drags the sign 502 back down toward the bottom of the GUI 600. As the sign 502 is animated downward toward the bottom of the GUI 600, the overlay 604 can be animated to become larger and become the overlay 302. Figure 5 As shown, while the placard 502 is minimized and the overlay 604 is expanded, the map view 202 may change to move the map so that the selected POI 210 , graphical object 312 , and viewing cone 314 are not obscured or covered by the larger overlay 302 .

[0065] Figure 7An exemplary graphical user interface 700 for presenting a maximized view of a sign is shown. For example, the GUI 700 may be presented by the navigation application 104 on the display of the user device 102. The GUI 700 may be presented in response to receiving user input to drag the sign 502 to the top of the display of the user device 102, as indicated by arrow 702. Figure 7 As shown, a maximized (eg, in some cases full screen) view of the signage 502 completely or substantially covers the map view 202 and / or the dynamic street scene overlay 302 / 604 .

[0066] like Figure 7 As shown, when the sign is maximized, the sign 502 may present additional information, such as images (e.g., pictures) 712-720, business hours 730, address 732, contact information 734 (e.g., email address, phone number, etc.), and / or website address 736. However, when the overlay 302 / 604 is obscured or hidden, the sign 502 may present the image 304 corresponding to the selected POI 210, which would have been presented on the overlay 302 / 604. As described above, the image 304 may be dynamic, in that the user may manipulate the image (e.g., provide input to pan the image), and the presentation of the image 304 may include a virtual parallax effect, described below, to make the image 304 appear three-dimensional.

[0067] As described above, when the user is finished viewing the maximized view of placard 502, the user can drag placard 502 back down to the center display position to cause navigation application 104 to present GUI 600, or drag placard 502 all the way down to the bottom of GUI 700 to cause navigation application 104 to present GUI 500. For example, when transitioning from GUI 700 to GUI 500, navigation application 104 can present GUI 600 and then present GUI 500 along with the animation described above for transitioning from GUI 600 to GUI 500.

[0068] Figure 8 An exemplary graphical user interface 800 is shown for simultaneously presenting a maximized view of a sign and a dynamic street scene overlay. For example, the GUI 800 can be presented by the navigation application 104 on a user device 102 having a large display area (e.g., such as a tablet, laptop, etc.). Because a large display allows more area for presenting content, the navigation application 104 can present a maximized view of the sign 502 and the dynamic street scene overlay 302 while still presenting enough of the map in the map view 202 so that the user can view the map area surrounding the currently selected point of interest and / or image capture point.

[0069] In some implementations, the GUI 800 can include a maximized view of the sign 502. For example, the maximized view of the sign 502 can allow the navigation application 104 to present more information about the selected point of interest (e.g., POI 210) than a medium-sized or minimized view of the sign 502. As described above, the POI information can include the POI name, POI description, POI distance, images 710-720 associated with the selected POI, business hours information 730, address information 732, contact information 734, and / or website information 736, among others. The maximized view of the sign 502 in the GUI 800 can be presented opposite the overlay 302 (e.g., in the upper left or upper right corner of the GUI 800).

[0070] In some implementations, the GUI 800 can include a dynamic street scene overlay 302. For example, when the overlay is presented in the GUI 800, the overlay 302 can include all of the configurations and behaviors described above and / or below with respect to the overlay 302. For example, the overlay 302 can include a dynamic interactive image 304. The GUI 800 can include a graphical object 312 that indicates a capture point location corresponding to the map location where the image 304 was captured. The GUI 800 can include a viewing cone 314 that indicates the viewing angle of the image 304 from the map location of the graphical object 312. The GUI 800 can include an emphasized (e.g., highlighted) representation of the selected POI 210, as described above.

[0071] Figure 9 An exemplary graphical user interface 900 is shown for presenting a maximized view of a dynamic street scene overlay 302. For example, as described above, in response to receiving a user selection of a graphical object 308 presented on GUI 300, GUI 500, or GUI 800, navigation application 104 may present GUI 900 on the display of user device 102. When the overlay is maximized, overlay 302 may cover most or all of the display of user device 102. For example, when no point of interest is currently selected, overlay 302 may cover the entire display area of ​​user device 102. When a point of interest is currently selected, overlay 302 may cover the entire display area of ​​user device 102; however, GUI 900 may also present signage 502 on a portion of overlay 302 to present information related to the selected point of interest, as described above.

[0072] In some implementations, the overlay 302 can be animated to expand to its maximum size. For example, when the overlay 302 is presented at its default size (e.g., covering approximately 30%-60% of the display of the user device 102), the navigation application 104 can, in response to receiving a user selection of the graphical object 308, present an animation that causes the overlay 302 to appear to grow or expand downward from its position at the top of the GUI 300 or GUI 500 to cover the entire display of the user device 102, as indicated by arrow 902. When the graphical object 308 is selected from the larger display GUI 800, the navigation application 104 can present an animation that causes the overlay 302 to appear to grow or expand downward and horizontally from its position at the top corner of the GUI 800 to cover the entire display of the user device 102.

[0073] Figure 10 An exemplary graphical user interface 1000 for selecting a point of interest from a dynamic street scene overlay is shown. For example, the GUI 1000 can be presented by the navigation application 104 on a display of the user device 102. The GUI 1000 can present, for example, a maximized view of the dynamic street scene overlay 302.

[0074] In some implementations, the dynamic street scene overlay 302 may include an image 304 depicting points of interest (e.g., points of interest or image capture points) near the currently selected map location. For example, the image 304 may include images of buildings, points of interest, roads, sidewalks, and other objects captured when the image capture device is located at a real-world location corresponding to the selected map location. When the navigation application is presented in the maximized overlay 302, the navigation application 104 may present labels identifying the various points of interest (e.g., POIs 1002-1008) included in the image 304.

[0075] In response to receiving user input selecting one of the POI tags (e.g., POI 1006), the navigation application 104 may present the user-selected POI 1006 as the currently selected POI in the various graphical user interfaces described above. For example, the navigation application 104 may present an animation in the overlay 302 that causes the image 304 to simulate moving from the current point of interest (e.g., POI 1002 or its associated image capture point) to the user-selected point of interest (e.g., POI 1006 or its associated image capture point). The animation may be generated using the image synthesis techniques disclosed below.

[0076] In some implementations, the navigation application 104 may receive user input selecting an image capture point via the overlay 302. For example, instead of receiving a user selection of one of the presented POIs 1002-1008, the navigation application 104 may receive a user selection (e.g., user input 1010, user input 1012, touch input, etc.) of a street, sidewalk, or location on an area other than the area associated with the POI tag or POI. In response to receiving the user input, the navigation application 104 may determine the image capture point closest to the map location associated with the user input and change the currently selected map location to the map location associated with the determined image capture point. The navigation application 104 may present an animation in the overlay 302 that causes the image 304 to simulate moving from the current point of interest (e.g., POI 1002 or the image capture point associated therewith) to the determined image capture point closest to the user input (e.g., inputs 1010 and 1012, etc.). The animation may be generated using the image synthesis techniques disclosed below.

[0077] Figure 11 1100 is an illustration of an exemplary process for synthesizing images to improve image quality. For example, when a virtual device is moved from a current image capture point to a newly selected image capture point (e.g., a destination location), the user device 102 may generate a realistic animation. To do so, the user device 102 may generate intermediate views of the environment along the route at intermediate locations along the route from the current image capture point to the destination image capture point. For example, the intermediate views and / or capture point views may correspond to images that present the perspective of a realistic three-dimensional model (e.g., a model textured with captured images) from the corresponding intermediate locations or capture point locations. The user device 102 may generate the realistic animation by presenting a sequence of frames including a series of intermediate views and image capture point views along the route. When generating images or views representing intermediate locations between two image capture points, the navigation application 104 may combine the least distorted, highest quality portions (e.g., image capture point views) of the images captured at the two image capture points to generate a high-quality composite image for each intermediate view (e.g., animation frame).

[0078] Illustration 1100 includes image capture points 1104 through 1110. For example, image capture points 1104 through 1110 may correspond to real-world locations where a map data collection system (e.g., a car with a camera and sensors) captures images and geometric data describing buildings and / or other objects near each image capture point. For example, at each image capture point 1104-1110, the map data collection system may capture images corresponding to different perspectives (e.g., perspectives or angles A, B, C, D, etc.). The map data collection system may also use sensors (e.g., lidar sensors) at image capture points 1104 through 1110 and / or locations therebetween to generate measurements (e.g., a point cloud indicating surface locations of nearby objects relative to the image capture points) that represent or correspond to the geometry of nearby objects 1140 through 1152 (e.g., buildings, trees, cars, lampposts, etc.). The map data collection system may also use a location sensor (e.g., a global satellite positioning system, a Wi-Fi-based positioning system, a cellular-based positioning system, etc.) to determine the locations of image capture points 1104 through 1110. The map data collection system may then send the data collected for image capture points 1104 through 1110 to server device 120 for processing.

[0079] In some implementations, server device 120 may process the map data collected for image capture points 1104 through 1110 to generate a three-dimensional model of the area near each image capture point. For example, server device 120 may generate a triangular mesh for image capture points 1104 through 1110 based on the point clouds generated at the corresponding image capture points. The triangular mesh may be generated to create a three-dimensional model of each of image capture points 1104 through 1110 and the points therebetween, as well as objects near image capture points 1104 through 1110 (e.g., surfaces of objects 1140 through 1152). For example, server device 120 may combine the point clouds generated at image capture points 1104 through 1110 and generate a triangular mesh to create a three-dimensional model of the map area covered by image capture points 1104 through 1110 and the points therebetween.

[0080] In some implementations, server device 120 may process the map data collected for each image capture point 1104-1110 to generate textures for the three-dimensional model generated for each image capture point. For example, server device 120 may combine the images captured for each viewpoint A, B, C, and D at image capture points 1104 through 1110 to generate a panoramic image (e.g., a 360-degree image) for each image capture point. This panoramic image, or portions thereof, corresponding to the image capture point may be applied to the three-dimensional model generated for that image capture point to add texture (e.g., an image representing an object, color, texture, etc.) to the three-dimensional model. Server device 120 may determine which portions of the captured image correspond to which surfaces of the three-dimensional model and store a mapping of the image portions to the model surfaces. Because the captured image corresponds to a specific viewpoint of the modeled surface from the specific image capture point at which the image was captured, when the captured image is applied (e.g., overlaid, painted, etc.) to the corresponding surface of the three-dimensional model, the captured image can be used to provide a textured view (e.g., a photorealistic view) of the model from the viewpoint of the corresponding image capture point. A textured view (eg, image) or photorealistic view (eg, image) of a three-dimensional model from the perspective of a particular image capture point may be referred to herein as an image capture point view (eg, image).

[0081] When sending map data for various locations to a user device (e.g., user device 102), server device 120 may send a map tile corresponding to each location, the map tile including the location (e.g., latitude and longitude) of each image capture point within the map tile, the corresponding texture image, and the corresponding three-dimensional model, as well as other data as may be described herein. In some implementations, the map tile may include data (e.g., a link, a reference, an address, etc.) indicating where user device 102 can obtain the image capture point data, the corresponding texture image, and the corresponding three-dimensional model, or portions thereof.

[0082] In some implementations, user device 102 can generate a realistic animation that simulates moving from one image capture point location to another. For example, as described above, user device 102 can present an image associated with a currently selected image capture point (e.g., a first image capture point) in a dynamic street scene overlay. User device 102 can receive user input selecting another image capture point or point of interest (e.g., a second image capture point). Instead of simply presenting an image corresponding to the first image capture point and then presenting an image corresponding to the second image capture point, user device 102 (e.g., navigation application 104) can generate an intermediate view at an intermediate location between the first image capture point and the second image capture point. For example, the intermediate view can be an image of a three-dimensional model textured (e.g., painted, colored, etc.) with a corresponding captured image from the perspective of a virtual device (e.g., a virtual camera) at the intermediate location. The intermediate view can be generated based on the capture point views corresponding to the first image capture point and / or the second image capture point (e.g., the captured image applied to the three-dimensional model) and transformed according to the virtual perspective of the virtual device at each intermediate location. User device 102 may present the intermediate views in sequence to present a realistic animation having the appearance of moving in the real world from the location of the first image capture point to the location of the second image capture point.

[0083] In some implementations, user device 102 can generate intermediate views based on the frame rate of user device 102. For example, if the frame rate of user device 102 is 120 frames per second (e.g., user device 102 can generate and / or present 120 video frames per second), user device 102 can generate 120 intermediate views per second. Thus, if the transition between image capture points 1006 and 1008 is configured to take one second, user device 102 can generate 120 intermediate views corresponding to 120 different virtual device positions along the path between image capture points 1006 and 1008 (e.g., the exemplary virtual device positions represented by diamonds 1112 through 1128).

[0084] In some implementations, the user device 102 can generate an intermediate view based on the direction of travel of the virtual device from the first image capture point to the second image capture point. For example, when the virtual device is virtually moved from the image capture point 1004 to the image capture point 1110, because the virtual device is at Figure 11 Moving upward and right, the user device 102 can use the capture point views or portions thereof corresponding to the images A and B captured at each image capture point 1004 to image capture point 1110 (for example, captured images pointing in the direction of travel) to generate intermediate views of the virtual device position 1112 to the virtual device position 1128.

[0085] When moving the virtual device between several different image capture points, user device 102 may use capture point views corresponding to the different image capture points to generate intermediate views 1112 through 1128. For example, the current image capture point may be image capture point 1104. The user may select image capture point 1110 as a new image capture point to be presented on the display of user device 102. As the virtual user device moves between image capture point 1104 and image capture point 1110, the virtual device also passes through or through transition image capture point 1106 and transition image capture point 1108. To generate the intermediate views between each image capture point, navigation application 104 may use the capture point view, or portion thereof, associated with the nearest image capture point. For example, to generate intermediate views 1112 through 1116, navigation application 104 may use the capture point views associated with image capture points 1104 and 1106. To generate intermediate views 1118 through 1122, navigation application 104 may use the capture point views associated with transition image capture points 1106 and 1108. To generate intermediate views 1124 through 1128, navigation application 104 may use the capture point views associated with image capture points 1108 and 1110.

[0086] Figure 12 is a conceptual illustration of how intermediate views are generated 1200. For example, the navigation application 104 on the user device 102 can transform and / or combine capture point views generated for capture points to generate intermediate views between the capture points.

[0087] Diagram 1200 includes image capture point 1202 and image capture point 1204. Diagram 1200 includes virtual device location 1206, where navigation application 104 can generate an intermediate view between image capture point 1202 and image capture point 1204. The intermediate view generated for virtual device location 1206 can include a street-level view from the perspective of virtual device location 1206 and / or based on the direction of travel of the virtual device. In this example, the direction of travel is from image capture point 1202 to image capture point 1204. Thus, the perspective of the virtual device at each intermediate location can be approximately in the direction of image capture point 1204. In the following example, the perspective of the virtual device at virtual device location 1206 (e.g., intermediate location 1206) can correspond to perspective cone 1208. Thus, the intermediate view generated for virtual device location 1206 can include three-dimensional model surfaces 1208 and 1210 (e.g., corresponding to buildings near virtual device location 1206).

[0088] When generating the intermediate view of the virtual device location 1206, the navigation application 104 can transform the capture point view corresponding to the capture point 1202 into a transformed view representing the perspective of the virtual device at the virtual device location 1206. For example, the user device 102 can obtain the capture point view corresponding to the capture point 1202. As the virtual device travels from the capture point 1202 to the capture point 1204, the user device 102 can obtain a portion of the capture point view corresponding to the perspective of the virtual device at the virtual device location 1206 (e.g., as shown by the viewing cone 1208). For example, the user device 102 can obtain a portion of the capture point view corresponding to the capture point 1202 that includes the three-dimensional model surfaces 1208 and 1210 and an applied portion of the image captured at the capture point 1202 (e.g., 1202A). The user device 102 may transform the three-dimensional model associated with the capture point view of surfaces 1208 and 1210 and other objects in the perspective of the virtual device so that it corresponds to or represents a view of surfaces 1208 and 1210 from the perspective of the virtual device at the virtual device position 1206.

[0089] User device 102 may apply a transformation similar to or the same as the transformation applied to the three-dimensional model to the image (e.g., texture image) captured to generate the capture point view at capture point 1202, such that after the three-dimensional model has been transformed to represent the perspective of the virtual device at virtual device position 1206, the image still covers the surface of the three-dimensional model. In some cases, when the transformation of the three-dimensional model increases the surface area of ​​surfaces 1208 and / or 1210, user device 102 may stretch (e.g., add pixels, multiply pixels, repeat pixels, etc.) the texture image to cover the enlarged surface of the transformed three-dimensional model. In other cases, when the transformation of the three-dimensional model decreases the surface area of ​​surfaces 1208 and / or 1210, user device 102 may compress (e.g., remove pixels) the texture image. In other cases, when the transformation of the three-dimensional model exposes a surface that is not represented in the capture point view corresponding to capture point 1202, user device 102 may add blank or black pixels to cover the exposed surface of the transformed three-dimensional model. Thus, user device 102 may generate a first transformed view representing the perspective of the virtual device at virtual device location 1206 based on the capture point view corresponding to capture point 1202 .

[0090] When generating the intermediate view of the virtual device location 1206, the navigation application 104 may transform the capture point view corresponding to the capture point 1204 into a transformed view that represents the perspective of the virtual device at the virtual device location 1206. For example, the user device 102 may obtain a capture point view corresponding to the capture point 1204. As the virtual device travels from the capture point 1202 to the capture point 1204, the user device 102 may obtain a portion of the capture point view that corresponds to the perspective of the virtual device at the virtual device location 1206 (e.g., as shown by the perspective cone 1208). For example, the user device 102 may obtain a portion of the capture point view corresponding to the capture point 1204 that includes the three-dimensional model surfaces 1208 and 1210 and an applied portion of the image captured at the capture point 1204 (e.g., 1204D, 1204C). The user device 102 may transform the three-dimensional model associated with the capture point view of surfaces 1208 and 1210 and other objects in the perspective of the virtual device so that it corresponds to or represents a view of surfaces 1208 and 1210 from the perspective of the virtual device at the virtual device position 1206.

[0091] User device 102 may apply a transformation similar to the transformation applied to the three-dimensional model to the image (e.g., a texture image) captured at capture point 1204 used to generate the capture point view, such that after the three-dimensional model has been transformed to represent the perspective of the virtual device at virtual device position 1206, the image still covers the surface of the three-dimensional model. In some cases, when the transformation of the three-dimensional model increases the surface area of ​​surfaces 1208 and / or 1210, user device 102 may stretch (e.g., add pixels, multiply pixels, repeat pixels, etc.) the texture image to cover the enlarged surface of the transformed three-dimensional model. In other cases, when the transformation of the three-dimensional model decreases the surface area of ​​surfaces 1208 and / or 1210, user device 102 may compress (e.g., remove pixels) the texture image. In other cases, when the transformation of the three-dimensional model exposes a surface that is not represented in the capture point view corresponding to capture point 1204 (e.g., surface 1208 relative to capture point 1204), user device 102 may add blank or black pixels to cover the exposed surface of the transformed three-dimensional model. Thus, user device 102 may generate a second transformed view representing the perspective of the virtual device at virtual device location 1206 based on the capture point view corresponding to capture point 1204 .

[0092] In some implementations, the transformed three-dimensional model represented in the first transformed view and the second transformed view can be the same. For example, since the same real-world objects (e.g., surface 1208, surface 1210, etc.) are measured when generating the point cloud and mesh used to generate the three-dimensional model, the perspective of the model from the virtual device position 1206 should be the same after the transformation from the perspectives of capture point 1202 and capture point 1204, regardless of the starting capture point. However, since the images of the model captured from different capture points 1202 and 1204 can be significantly different, the transformation of these images can produce images of widely varying quality, depending on how much the images have been stretched, compressed, or otherwise altered during the transformation to generate the corresponding transformed views.

[0093] As described above, when generating a transformed view based on the capture point view, the navigation application 104 may stretch portions of the texture image to cover corresponding surfaces of the three-dimensional model. For example, the further the perspective of the capture point view deviates from the object surface (e.g., surface 1210) by 90 degrees relative to the object surface in the three-dimensional model, the more the navigation application 104 may need to stretch portions of the corresponding texture image to cover the corresponding surface of the three-dimensional model when transforming the model. To stretch the image or portion thereof, the navigation application 104 may duplicate or multiply certain pixels when the corresponding texture image (e.g., the captured image) does not already include enough pixels to cover the surface of the three-dimensional model. For example, if the perspective from capture point 1202 to surface 1208 is not at a very extreme angle (e.g., deviating by approximately 90 degrees from surface 1208), and therefore, the texture image applied to surface 1208 may not require much stretching when transforming from the perspective of capture point 1202 to the perspective of the virtual device at virtual device position 1206. In contrast, the perspective from capture point 1202 to surface 1210 is at a fairly extreme angle (e.g., far from 90 degrees from surface 1210), and therefore, the texture image applied to surface 1210 may require significant stretching when converting from the perspective of capture point 1202 to the perspective of the virtual device at virtual device position 1206.

[0094] For each transformed view, the navigation application 104 may generate a quality score for each pixel of the transformed image in the transformed view. The quality score indicates the quality of the pixel in the transformed image. In some implementations, the navigation application 104 may determine the quality score based on the amount of stretching that the navigation application 104 must perform at the pixel location when generating the transformed view. For example, the more stretching performed, the lower the score. The less stretching performed, the higher the score. In some implementations, the navigation application 104 may determine the quality score based on the amount of compression that the navigation application 104 must perform at the pixel location when generating the transformed view. For example, the more compression performed, the lower the score. The less compression performed, the higher the score. In some implementations, the navigation application 104 may determine the quality score based on whether the pixel is a blank pixel or a black pixel. For example, a blank or black pixel may indicate that the pixel corresponds to a surface that is not visible in the capture point view on which the transformed view is based. Therefore, a blank or black pixel should not be used to generate the intermediate view of the virtual device location 1206. Therefore, a blank or black pixel may be assigned a quality score of zero, indicating that the pixel should not be used in the intermediate view.

[0095] In some implementations, a weighted pixel quality score can be calculated based on the distance between the virtual device location 1206 and the corresponding image capture point (e.g., image capture point 1202 or 1204). For example, the farther the image capture point is from the virtual device location, the greater the transformation required to convert the capture point view into the transformed view corresponding to the virtual device location. The greater the required transformation, the lower the quality of the resulting transformed view may be. Therefore, the transformed views and the image pixels therein can be weighted to give more weight to transformed views corresponding to image capture locations that are closer to the virtual device location. Thus, in diagram 1200, the transformed view and the pixels therein corresponding to capture point 1202, which is farther away from the virtual device location 1206 than the image capture point 1202, can be given a higher weight (e.g., a pixel quality score multiplier) than the transformed view and the pixels therein corresponding to capture point 1204, which is farther away from the virtual device location 1206 than the image capture point 1202.

[0096] Figure 13 1300 is a diagram of a process for compositing transition images into an intermediate view. For example, diagram 1300 depicts a mechanism for compositing or combining the first transition image and the second transition image (e.g., based on the capture point views at capture points 1202 and 1204) described above into an intermediate view corresponding to the perspective of the virtual device at virtual device location 1206.

[0097] Illustration 1300 includes a transformed view 1302 generated based on a capture point view corresponding to capture point 1202. For example, as described above, transformed view 1302 (e.g., an image) may include a transformed image representing surfaces 1208 and 1210. Each pixel in transformed view 1302 may have a corresponding quality score (e.g., an unweighted quality score, a weighted quality score, etc.), as shown in pixel quality graph 1304. For example, pixels in an area shaded by diagonal lines (e.g., on surface 1210) represent areas where the corresponding pixel quality score is relatively low (e.g., relative to transformed view 1306). Pixels in an area not shaded by diagonal lines represent areas where the corresponding pixel quality score is relatively high (e.g., relative to transformed view 1306).

[0098] Diagram 1300 includes a transformed view 1306 generated based on a capture point view corresponding to capture point 1204. For example, as described above, transformed view 1306 (e.g., an image) may include a transformed image representing surface 1208 and surface 1210. Each pixel in transformed view 1306 may have a corresponding quality score (e.g., an unweighted quality score, a weighted quality score, etc.), as shown in pixel quality graph 1308. For example, pixels in an area shaded by diagonal lines (e.g., on surface 1208) represent areas where the corresponding pixel quality score is relatively low (e.g., relative to transformed view 1302). Pixels in an area not shaded by diagonal lines represent areas where the corresponding pixel quality score is relatively high (e.g., relative to transformed view 1302).

[0099] In some implementations, the navigation application 104 can combine or composite the transformed view 1302 and the transformed view 1306 into an intermediate view 1310 of the virtual device location 1206 based on the pixel quality scores. For example, to select the best quality image or portion thereof for the intermediate view 1310, the navigation application can compare corresponding pixels from the transformed view 1302 and the transformed view 1306 (e.g., at the same location in the transformed view 1302 and the transformed view 1306) to determine which of the two corresponding pixels has the highest quality score. The navigation application 104 can then select the corresponding pixel with the highest quality to include in the intermediate view 1310. For example, the navigation application 104 can compare the pixel quality scores of each pixel in the transformed view 1302 and the transformed view 1306 and determine that the transformed view 1302 provides higher quality pixels for the surface 1208 than the transformed view 1306. Similarly, the navigation application 104 can compare the pixel quality scores for each pixel in the transformed view 1302 and the transformed view 1306 and determine that the transformed view 1306 provides higher quality pixels for the surface 1210 than the transformed view 1306. Based on these pixel quality determinations, the navigation application 104 can generate the intermediate view 1310 by including images / pixels from the transformed view 1302 corresponding to the surface 1208 and by including images / pixels from the transformed view 1306 corresponding to the surface 1210.

[0100] In some implementations, the navigation application 104 can generate blurred versions of the transformed view 1302 and the transformed view 1306. For example, when the pixel quality scores of corresponding pixels in the transformed view 1302 and the transformed view 1306 are both low (e.g., below a certain threshold), the navigation application 104 can use pixels from the blurred versions of the transformed view 1302 and the transformed view 1306 to generate the corresponding pixels in the intermediate view 1310. The navigation application 104 can select between the pixels of the blurred transformed view 1302 and the blurred transformed view 1306 based on the distance between the virtual device position and the position of the corresponding image capture point that generated the transformed view. For example, the pixel from the blurred transformed view corresponding to the image capture point closest to the virtual device position can be selected and included in the intermediate view 1310.

[0101] Figure 14 1 is a system interaction diagram illustrating an exemplary process 1400 for obtaining capture point data when generating a realistic animation that simulates moving a virtual device from one image capture point to another. For example, process 1400 may be performed to obtain captured images and three-dimensional models associated with capture points along a path between two selected capture points.

[0102] At step 1402, user device 102 may request map tiles. For example, user device 102 may request map tiles (e.g., map data defined by a geographic area) from server device 120 for the current location, frequently visited locations, and / or nearby locations of user device 102. The map tile request may include information identifying these locations.

[0103] At step 1404, user device 102 may receive a map tile. For example, user device 102 may receive a map tile corresponding to the requested location from server device 120. For example, server device 120 may determine which map tiles correspond to the requested location and send the determined map tiles to user device 102. The map tile may include image capture point data information corresponding to the image capture point locations within the geographic area of ​​the map tile. The image capture point data for each image capture point location may include the location of the image capture point, the image captured at the image capture point, a three-dimensional model of the area near the image capture point, and / or other data as may be described herein. In some embodiments, the image capture point data in the map tile may include a link, a reference, an address, etc., which indicates a location where user device 102 can obtain or download the corresponding image capture point data. For example, the image capture point data may include a link to an image and / or three-dimensional model data stored on a server device (e.g., server device 120) without including the image and / or three-dimensional model data.

[0104] At step 1406, user device 102 may receive input selecting a destination location. For example, user device 102 may present an image, point of interest data, etc. corresponding to the currently selected image capture point location. The user may wish to view a different image capture point or point of interest and may provide input selecting a new location to be viewed on the display of user device 102. User device 102 may determine the image capture point closest to the selected new location and select the determined image capture point as the destination location.

[0105] At step 1408, user device 102 may request view data for the image capture point location of the destination. For example, user device 102 may determine a map tile that includes the destination location and obtain information (e.g., a link, an address, etc.) from the map tile for downloading image capture point view data (e.g., a captured image, a 3D model, etc.) for the destination location. For example, user device 102 may request the image capture point view data for the destination location from server device 120.

[0106] At step 1410, user device 102 may request view data for transitional image capture point locations. For example, user device 102 may determine transitional image capture point locations between the current image capture point location and the destination image capture point location along a path between the current image capture point location and the destination image capture point location. User device 102 may determine a map tile that includes the transitional image capture point locations and obtain information (e.g., a link, an address, etc.) from the map tile for downloading image capture point view data (e.g., a captured image, a three-dimensional model, etc.) for each transitional image capture point location. For example, user device 102 may request image capture point view data for the transitional image capture point locations from server device 120 in order based on which transitional image capture point locations the virtual device will encounter first when traversing the path.

[0107] At step 1412, user device 102 may receive image capture point data of the destination location. For example, server device 120 may send image capture point data (e.g., captured images, texture images, three-dimensional models, etc.) of the requested destination location to user device 102.

[0108] At step 1414, user device 102 may initiate a movement animation from the current image capture point location to the destination image capture point location. For example, rather than waiting to receive image capture point view data for a first transitional image capture point location, user device 102 may begin generating an intermediate view for a first intermediate virtual device location based on the image capture point view data for the current image capture point location and the destination image capture point location. If no additional access point view data for other image access point locations is received before step 1414 is completed, user device 102 may present the generated intermediate view on the display of user device 102 to begin the movement animation.

[0109] At step 1416, the user device 102 may receive transition image capture point location view data. For example, the user device 102 may receive transition image capture point location view data for a first transition image capture point location.

[0110] At step 1418, the user device 102 may incorporate the transition image capture point position view data into the movement animation. For example, after receiving the transition image capture point position data for the first transition image capture point position, the user device 102 may generate an intermediate view sequence based on the view data for each position, which displays the movement between the current image capture point position and the first transition image capture point position in an animated manner. When the view data is received, the user device 102 may present the intermediate view generated at step 1418 instead of the intermediate view generated at step 1416, or before the intermediate view generated at step 1416. As described above, when view data for additional transition image capture point positions is received, the user device 102 may generate additional intermediate view sequences corresponding to intermediate virtual device positions based on the received view data. For example, as described above with reference to Figure 11 、 Figure 12 and Figure 13 As described above, user device 102 may generate a sequence of intermediate views of the virtual device location based on the available image capture point views of the image capture point closest to the virtual device location. The sequence of intermediate views may be presented sequentially according to a path traversed between the current image capture point location and the destination image capture point location to generate a realistic animation depicting travel along the path in the real environment. For example, as described above, user device 102 may present a movement animation in dynamic street scene overlay 302.

[0111] At step 1420, the user device 102 may present an image of the destination location upon completion of the movement animation. For example, as described herein, the user device 102 may present an image depicting the real-world environment near the destination image capture point location or point of interest. In some implementations, as described above, the user device 102 may present the image in a dynamic street scene overlay 302. In some implementations, the user device 102 may present the image using virtual parallax techniques described below.

[0112] Figure 15 15 is a conceptual illustration of generating virtual parallax to produce a three-dimensional effect. For example, user device 102 can simulate a parallax effect using a single two-dimensional image and a corresponding three-dimensional model to make the two-dimensional image appear three-dimensional to the user. For example, when presenting an image capture point view on a display of user device 102 (e.g., in a dynamic street scene overlay 302), user device 102 can generate, based on the image capture point view, transition views from the perspective of a virtual device (e.g., a virtual camera) at different virtual locations around the map location of the corresponding image capture point. By presenting an animation of the image capture point view that includes different perspectives (e.g., as applied to a captured image of a three-dimensional model associated with the image capture point, as described above), user device 102 can introduce parallax into the presentation of the image capture point view, thereby making the image capture point view appear three-dimensional.

[0113] In some implementations, the navigation application 104 can present an image capture point view from the perspective of the virtual device at the image capture point location 1502. As described above, the image capture point view can present an image generated from an image captured at a real-world location that corresponds to the image capture point location 1502 applied to the three-dimensional model. Figure 15 The three-dimensional model in the example of can model the surfaces of object 1504 (e.g., a lamp post, a tree, a person, etc.) and object 1506 (e.g., a building, a wall, a bridge, etc.). As shown, from the perspective of the virtual device positioned at capture point location 1502, object 1504 can be positioned in front of object 1506. Although the capture point view corresponding to image capture point location 1502 is generated based on the three-dimensional models of objects 1504 and 1506 textured with images (e.g., photographs) of objects 1504 and 1506, when user device 102 presents the capture point view on the display of user device 102, the capture point view will appear as a two-dimensional image.

[0114] To give the capture point view a three-dimensional appearance, user device 102 (e.g., navigation application 104) may generate a virtual parallax effect by simulating multiple virtual image capture devices around or near image capture point location 1502. For example, when user device 102 receives user input (e.g., the user input to overlay 302 described above) to pan the capture point view image presented on the display of user device 102, user device 102 may move the virtual position of the virtual device from location 1502 to nearby locations (e.g., location 1510 to location 1518) to the side and / or behind virtual location 1502. As the virtual device moves from location 1502, user device 102 may generate, based on the image capture point view (e.g., the captured image and the three-dimensional model) corresponding to location 1502, a transition image representing the changing perspective of the virtual device as the virtual device moves, thereby simulating different perspectives of different virtual devices (e.g., virtual cameras) captured at different locations.

[0115] In some implementations, the transition images can be generated based on the frame rate of the user device 102. As described above, the frame rate of the user device 102 can be 120 frames per second. If it takes one second to move the virtual device from position 1502 to position 1510, the user device 102 will generate 120 transition images representing the changes in the virtual device's perspective with respect to objects 1504 and 1506 as the virtual device traverses the 120 positions to arrive at position 1510. By sequentially presenting an animation comprising each of these transition images at the determined frame rate, the relative positions of objects 1504 and 1506 within each subsequent or adjacent transition image will change, thereby introducing a parallax effect and creating a three-dimensional appearance for the transition images or animation as the virtual device's perspective with respect to objects 1504 and 1506 changes with the movement.

[0116] In some implementations, the amount of horizontal movement of the virtual device may be based on the speed of the user input. For example, in order for user device 102 to generate and present the transition image, user device 102 may receive user input indicating that the user desires to pan the image corresponding to the capture point view. For example, user device 102 may detect touch input in the form of a left or right swipe gesture. As described above, user device 102 may move the virtual device from position 1502 to the left (e.g., toward position 1510) or to the right (e.g., toward position 1518) in response to detecting the touch input, and generate and present the transition image during the movement. The faster the swipe gesture, the greater the distance user device 102 will move the virtual device. For example, if user device 102 detects a slow left swipe gesture, user device 102 may slowly move the virtual device from position 1502 to the first diamond to the left of position 1502. If user device 102 detects a fast left swipe gesture, user device 102 may quickly move the virtual device from position 1502 to position 1510 or to the maximum horizontal distance from position 1502. When the maximum horizontal (e.g., lateral, translational, etc.) distance is reached, the virtual device may remain at the maximum horizontal distance until the user input ceases or until the user input causes the user device 102 to move the virtual device in a different horizontal direction. When the user device 102 no longer detects the user input, the user device 102 may move the virtual device back to position 1502 within a period of time (e.g., 1 second, 0.5 seconds, etc.).

[0117] In some implementations, the amount of backward movement of the virtual device may be based on the duration of the user input. For example, backward movement of the virtual device (e.g., opposite to the viewing direction) may accumulate over time until a maximum backward distance is reached. Thus, as the user provides translation input over an extended period of time, the virtual device may gradually move toward position 1514 until the maximum backward distance is reached. As the user input continues, the virtual device will remain at the maximum backward distance until the user device 102 no longer detects the user input. When the user device 102 no longer detects the user input, the user device 102 may move the virtual device back to position 1502 over a period of time (e.g., 1 second, 0.5 seconds, etc.).

[0118] In some implementations, user device 102 can dynamically adjust the parallax effect based on the distance between the virtual device and objects in the image capture point view. For example, when the image capture point view includes objects near the virtual device, user device 102 can move the virtual device a greater distance to increase the parallax effect of the nearby objects. When the image capture point view includes objects far away from the virtual device, user device 102 can move the virtual device a smaller distance to reduce the parallax effect of the distant objects.

[0119] Exemplary Process

[0120] In order to enable the reader to clearly understand the technical concepts described herein, the following process describes the specific steps performed in a specific order. However, one or more steps of a specific process can be rearranged and / or omitted while remaining within the expected scope of the technology disclosed herein. In addition, different processes and / or their steps can be combined, recombined, rearranged, omitted and / or performed in parallel to create different processing flows that are also within the expected scope of the technology disclosed herein. In addition, although the following process can omit or briefly summarize some details of the technology disclosed herein for the sake of clarity, the details described in the above paragraphs can be combined with the process steps described below to obtain a more complete and comprehensive understanding of these processes and the technology disclosed herein.

[0121] Figure 16 16 is a flow diagram of an exemplary process 1600 for presenting a dynamic street scene overlay. For example, a user device 102 may perform the process 1600 to present a dynamic street scene overlay that allows a user to interact with and / or view images presented in the dynamic street scene overlay while still allowing the user to interact with and / or view a corresponding map view.

[0122] At step 1602, the user device 102 may present a map view on a graphical user interface of the user device 102. For example, the map view may include a map of a geographic area.

[0123] At step 1604, the user device 102 may receive input to invoke a dynamic street scene overlay. For example, the user device 102 may receive the input described above with reference to Figure 2 The input of the selection graphic object 216 is described.

[0124] At step 1604, user device 102 may select a first map location. For example, user device 102 may automatically select an image capture point location that is closest to the center of the geographic area presented in the map view in response to receiving input to invoke a dynamic street scene overlay. If user device 102 has previously received user input selecting a map location (e.g., an image capture point location, a point of interest, etc.), user device 102 may select the image capture point location that is closest to the user-selected map location.

[0125] At step 1608, the user device 102 may present a dynamic street scene overlay in the first portion of the graphical user interface. For example, the size of the dynamic street scene overlay may be set so that the user can easily see the details of the image presented in the dynamic street scene overlay. For example, the size of the dynamic street scene overlay may be set to cover 25% or more of the display area of ​​the graphical user interface. The image presented by the dynamic street scene overlay may be an image representing the first map location. For example, the image may be a capture point view representing the perspective of the real-world environment from the first location. As described herein, the image may be dynamic, manipulable, and may appear three-dimensional.

[0126] At step 1610, user device 102 may move the map presented within the map view to present the first map location and the surrounding area in a second portion of the graphical user interface. For example, the second portion of the graphical user interface may be separate from the first portion of the graphical user interface so that the dynamic street scene overlay does not obstruct the user's view of the first map location and the surrounding area. For example, the map within the movable map view may move the display location (e.g., GUI location) of the first map location to the center of the second portion of the graphical user interface so that the user can view both the first map location and the dynamic street scene overlay on the map in the map view simultaneously. The first map location may be identified in the map view using a graphical object that indicates the first map location and the perspective (e.g., viewing angle) presented by the dynamic street scene overlay.

[0127] At step 1612, the user device 102 may receive user input to adjust the image presented in the dynamic street scene overlay. For example, the user device 102 may receive user input via the dynamic street scene overlay to change the perspective of a first location presented by the image. The user device 102 may receive user input via the dynamic street scene overlay to select a second location that appears in the image presented by the dynamic street scene overlay. As described above, the user device 102 may receive user input via the dynamic street scene overlay to pan the image presented by the dynamic street scene overlay, thereby triggering a three-dimensional rendering of the perspective presented by the image.

[0128] In some implementations, the user device 102 may receive user input through the map view that causes the user device to adjust the image presented in the dynamic street scene overlay. For example, as described above, the user device 102 may receive user input to scroll the map in the map view below the location indicator graphical object (e.g., graphical object 312) and select a second location. The user device 102 may adjust the image presented in the dynamic street scene overlay to represent the perspective of the real-life environment from the second location. As may be described elsewhere herein, the user device 102 may receive user input to adjust the image presented in the dynamic street scene overlay in other ways.

[0129] At step 1614, the user device 102 may modify the image presented in the dynamic street scene overlay. For example, while continuing to present the map view and the location indicator graphical object in the second portion of the graphical user interface, the user device 102 may modify the image presented in the dynamic street scene overlay. As may be described elsewhere herein, the modification may include presenting a different perspective of the first location, presenting a realistic animation of movement from the first location to the second location, presenting a realistic animation of the first location that introduces a three-dimensional effect, and / or presenting a perspective of the second location, etc.

[0130] Figure 17 is a flow chart of an exemplary process 1700 for synthesizing images to improve image quality. Figures 11 to 14 As described, user device 102 may perform process 1700 to synthesize images associated with a first image capture point and a second image capture point when generating an intermediate view of a virtual device position along a path from the first image capture point to the second image capture point.

[0131] At step 1702, user device 102 may obtain a first transformed image. For example, the first transformed image may be generated based on a first image capture point view corresponding to a first image capture point. The first image capture point view may transform a perspective of a portion of the photorealistic three-dimensional model from the first image capture point position to a perspective of the portion of the photorealistic three-dimensional model from an intermediate position between the first image capture point and the second image capture point.

[0132] At step 1704, user device 102 may obtain a second transformed image. For example, the second transformed image may be generated based on a second image capture point view corresponding to the second image capture point. The second image capture point view may transform a perspective of a portion of the photorealistic three-dimensional model from the second image capture point position to a perspective of the portion of the photorealistic three-dimensional model from an intermediate position between the first image capture point and the second image capture point.

[0133] At step 1706, the user device 102 may determine corresponding pixels in the first converted image and the second converted image. For example, for each pixel in the first converted image, the user device 102 may determine a corresponding second pixel in the second converted image.

[0134] At step 1708, user device 102 may compare the pixel quality score for the particular first pixel with the pixel quality score for the particular corresponding second pixel. For example, user device 102 may generate a quality score for each pixel in the first converted image and the second converted image. User device 102 may generate the quality score based on the amount of stretch or compression applied to the pixel location within the corresponding converted image. User device 102 may generate the quality score for the pixel based on whether the pixel corresponds to a surface of the corresponding three-dimensional model that is visible in the capture point view from which the converted image was generated. In some implementations, the quality score of the converted image may be weighted based on the distance between an intermediate location (e.g., a virtual device location) and the image capture point location associated with the converted image.

[0135] At step 1710, the user device 102 may compare the pixel quality score of a particular first pixel with the pixel quality score of a particular corresponding second pixel. For example, the user device 102 may compare the quality score of each pixel in the first converted image with the quality score of the corresponding pixel in the second converted image. For example, the corresponding pixel may be a particular pixel in the second converted image that occupies the same relative position in the second converted image as the particular pixel in the first converted image.

[0136] At step 1712, user device 102 may select between the particular first pixel and the particular corresponding second pixel based on the comparison at step 1710. For example, if the particular first pixel has a higher quality score than the particular corresponding second pixel, user device 102 may select the particular first pixel. However, if the particular corresponding second pixel has a higher quality score than the particular first pixel, user device 102 may select the particular first pixel. In some implementations, as described above, if both the quality scores of the particular first pixel and the particular corresponding second pixel are below a threshold, a blurred pixel may be selected. User device 102 may perform this pixel selection operation for each corresponding pair of pixels in the first converted image and the second converted image.

[0137] At step 1714, the user device 102 may generate a composite image based on the selected pixels from the first converted image and / or the second converted image. For example, the user device 102 may generate the composite image by including the selected pixels in the composite image according to their respective or relative positions in the first converted image and / or the second converted image. Thus, the composite image (e.g., the intermediate view) may represent the same view as the first converted image and the second converted image, but may include the highest quality portion of the first converted image and / or the second converted image, as described above.

[0138] Figure 18 1 is a flow chart of an exemplary process 1800 for generating virtual parallax to create a three-dimensional appearance. For example, user device 102 may generate process 1800 to generate a three-dimensional appearance for a two-dimensional image of one or more objects. Figure 15 As described, although a single two-dimensional image is used, the user device 102 may convert the two-dimensional image to simulate images of one or more objects captured at different virtual locations by different virtual image capture devices, thereby introducing virtual parallax and a corresponding three-dimensional effect or appearance into the two-dimensional image.

[0139] At step 1802, user device 102 may present a first view of one or more objects from a perspective of a first position corresponding to a first image capture point. For example, the first view may be a view (e.g., a two-dimensional image, an image capture point view, etc.) of a photorealistic three-dimensional model from the perspective of the first image capture point. The photorealistic three-dimensional model may include various objects, including buildings, trees, lampposts, cars, etc.

[0140] At step 1904, user device 102 may receive user input to modify the first view. For example, user device 102 may receive user input to pan the image presented in the first view left or right.

[0141] At step 1906, user device 102 may convert the first view into multiple second views of one or more objects from the perspective of multiple second positions near the first position. For example, user device 102 may determine multiple second positions (e.g., virtual device positions) along a path or trajectory near the first position of the image capture point in response to user input. For example, the path or trajectory may cause the virtual device to move left, right, and / or backward from the image capture point based on the type of user input received. As described above, user device 102 may generate converted views from the perspective of each of the multiple second positions based on the image capture point view of the first image capture point.

[0142] At step 1908, the user device 102 may present an animation including a first view and a plurality of second views. For example, when the virtual device moves from a first position of a first image capture point and returns to the first position of the first image capture point, the user device 102 may present a sequence of frames including the first view and a sequence of second views (e.g., transition views). When presenting the sequence of frames (e.g., a sequence of views), objects depicted in the animation frames may appear to be three-dimensional objects due to the change in perspective represented by each view.

[0143] Graphical User Interface

[0144] The present disclosure describes above various graphical user interfaces (GUIs) for implementing various features, processes, or workflows. These GUIs can be presented on various electronic devices, including but not limited to laptop computers, desktop computers, computer terminals, television systems, tablet computers, e-book readers, and smart phones. One or more of these electronic devices may include a touch-sensitive surface. The touch-sensitive surface can process multiple simultaneous input points, including processing data related to the pressure, degree, or position of each input point. Such processing can facilitate gestures performed with multiple fingers, including pinching and swiping.

[0145] When this disclosure refers to "selecting" a user interface element in a GUI, these terms are understood to include clicking or "hovering" over a user interface element with a mouse or other input device, or touching, tapping, or gesturing on a user interface element with one or more fingers or a stylus. A user interface element can be a virtual button, menu, selector, switch, slider, swipe, knob, thumbnail, link, icon, radio button, check box, and any other mechanism for receiving input from a user or providing feedback to a user.

[0146] privacy

[0147] As described above, one aspect of the present technology is to collect and use data available from various sources to improve the presentation of map-related data and / or imagery. The present disclosure contemplates that, in some instances, such collected data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, Twitter IDs, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0148] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit users. For example, the personal information data can be used to present locations and location data that are of interest to the user. In some cases, private information, such as faces, license plates, or other personally identifiable information, may be captured during the map data and / or image collection process. As described herein, when private information is captured in images and / or photos, the private information may be blurred or otherwise obscured to avoid sharing the private information when distributing the map data. In addition, the present disclosure also anticipates other uses for personal information data that are beneficial to users. For example, health and fitness data can be used to provide insights into the user's overall health, or can be used as positive feedback to individuals who use technology to pursue health goals.

[0149] This disclosure contemplates that entities responsible for collecting, analyzing, disclosing, transmitting, storing, or otherwise using such personal information will adhere to established privacy policies and / or practices. Specifically, such entities should implement and adhere to privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information. Such policies should be readily accessible to users and updated as the collection and / or use of data changes. Personal information collected from users should be used for the entity's legitimate and reasonable purposes and not shared or sold beyond those legitimate uses. Furthermore, such collection / sharing should be conducted with the user's informed consent. Furthermore, such entities should consider taking any necessary steps to safeguard and secure access to such personal information and ensure that others with access to the personal information adhere to their privacy policies and procedures. Furthermore, such entities may subject themselves to third-party assessments to demonstrate compliance with widely accepted privacy policies and practices. Furthermore, policies and practices should be tailored to the specific type of personal information collected and / or accessed and to applicable laws and standards, including jurisdictional considerations. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); whereas health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained for different types of personal data in each country.

[0150] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates providing hardware elements and / or software elements to prevent or block access to such personal information data. For example, when presenting map data, the technology of the present invention may be configured to allow users to “opt in” or “opt out” of participating in the collection of personal information data at any time during or after registration for the service. In addition to providing “opt-in” and “opt-out” options, the present disclosure contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then be reminded again just before the personal information data is accessed by the application.

[0151] Furthermore, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risk can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods, where appropriate.

[0152] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the absence of all or a portion of such personal information data. For example, map data may be presented based on non-personal information data or an absolute minimum amount of personal information (such as content requested by a device associated with a user, other non-personal information available to a mapping service, or publicly available information).

[0153] Exemplary system architecture

[0154] Figure 19 It is achievable Figures 1 to 18 19. A block diagram of an exemplary computing device 1900 illustrating features and processes of a computer system. Computing device 1900 may include a memory interface 1902, one or more data processors, an image processor, and / or a central processing unit 1904, and a peripheral device interface 1906. Memory interface 1902, one or more processors 1904, and / or peripheral device interface 1906 may be separate components or may be integrated into one or more integrated circuits. The various components in computing device 1900 may be coupled by one or more communication buses or signal lines.

[0155] Sensors, devices, and subsystems can be coupled to the peripherals interface 1906 to facilitate a variety of functions. For example, a motion sensor 1910, a light sensor 1912, and a proximity sensor 1914 can be coupled to the peripherals interface 1906 to facilitate orientation, lighting, and proximity functions. Other sensors 1916 can also be connected to the peripherals interface 1906, such as a global navigation satellite system (GNSS) (e.g., a GPS receiver), a temperature sensor, a biometric sensor, a magnetometer, or other sensing devices, to facilitate related functions.

[0156] Camera functions, such as taking photos and video clips, may be facilitated by a camera subsystem 1920 and an optical sensor 1922, such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) optical sensor. The camera subsystem 1920 and the optical sensor 1922 may be used to collect images of the user to be used during authentication of the user, for example, by performing facial recognition analysis.

[0157] Communication functionality may be facilitated by one or more wireless communication subsystems 1924, which may include radio frequency receivers and transmitters and / or optical (e.g., infrared) receivers and transmitters. The specific design and implementation of the communication subsystems 1924 may depend on the communication network over which the computing device 1900 is intended to operate. For example, the computing device 1900 may include a device designed to communicate over a GSM network, a GPRS network, an EDGE network, a Wi-Fi or WiMax network, and a Bluetooth® network. TM Network-operated communication subsystem 1924. Specifically, the wireless communication subsystem 1924 may include a hosting protocol so that the device 100 can be configured as a base station for other wireless devices.

[0158] The audio subsystem 1926 may be coupled to a speaker 1928 and a microphone 1930 to facilitate voice-enabled functions such as speaker recognition, voice replication, digital recording, and telephony. The audio subsystem 1926 may be configured to facilitate, for example, processing voice commands, voiceprint identification, and voice authentication.

[0159] I / O subsystem 1940 may include a touch surface controller 1942 and / or other input controller 1944. Touch surface controller 1942 may be coupled to touch surface 1946. Touch surface 1946 and touch surface controller 1942 may detect contact and its movement or interruption using, for example, any of a variety of touch-sensitive technologies including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch surface 1946.

[0160] The one or more other input controllers 1944 may be coupled to other input / control devices 1948, such as one or more buttons, rocker switches, a thumb wheel, an infrared port, a USB port, and / or a pointer device (such as a stylus). The one or more buttons (not shown) may include an up / down button for volume control of the speaker 1928 and / or the microphone 1930.

[0161] In one embodiment, pressing the button for a first duration can unlock touch surface 1946; and pressing the button for a second duration, longer than the first duration, can turn computing device 1900 on or off. Pressing the button for a third duration can activate a voice control or voice command module that enables a user to speak a command into microphone 1930 to cause the device to execute the spoken command. The user can customize the functionality of one or more buttons. For example, touch surface 1946 can also be used to implement virtual or soft buttons and / or a keyboard.

[0162] In some implementations, the computing device 1900 can render recorded audio and / or video files, such as MP3, AAC, and MPEG files. In some implementations, the computing device 1900 can include the functionality of an MP3 player, such as an iPod. TM .

[0163] The memory interface 1902 can be coupled to a memory 1950. The memory 1950 may include high-speed random access memory and / or non-volatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, and / or flash memory (e.g., NAND, NOR). The memory 1950 may store an operating system 1952, such as Darwin, RTXC, LINUX, UNIX, OSX, WINDOWS, or an embedded operating system (such as VxWorks).

[0164] The operating system 1952 may include instructions for handling basic system services and for performing hardware-related tasks. In some implementations, the operating system 1952 may be a kernel (e.g., a UNIX kernel). In some implementations, the operating system 1952 may include instructions for performing voice authentication. For example, the operating system 1952 may implement a dynamic street scene overlay feature, such as the one shown in FIG. Figures 1 to 18 As stated.

[0165] The memory 1950 may also store communication instructions 1954 to facilitate communication with one or more additional devices, one or more computers, and / or one or more servers. The memory 1950 may include graphical user interface instructions 1956 to facilitate graphical user interface processing; sensor processing instructions 1958 to facilitate sensor-related processing and functions; telephony instructions 1960 to facilitate telephony-related processes and functions; electronic message processing instructions 1962 to facilitate electronic message processing and functions; web browsing instructions 1964 to facilitate web browsing-related processes and functions; media processing instructions 1966 to facilitate media processing-related processes and functions; GNSS / navigation instructions 1968 to facilitate GNSS and navigation-related processes and instructions; and / or camera instructions 1970 to facilitate camera-related processes and functions.

[0166] Memory 1950 may store software instructions 1972 to facilitate other processes and functions, such as reference Figures 1 to 18 The dynamic street scene overlay process and function described.

[0167] The memory 1950 may also store other software instructions 1974, such as web video instructions that facilitate processes and functions related to web video; and / or online shopping instructions that facilitate processes and functions related to online shopping. In some implementations, the media processing instructions 1966 are divided into audio processing instructions and video processing instructions to facilitate processes and functions related to audio processing and processes and functions related to video processing, respectively.

[0168] Each of the instructions and applications identified above may correspond to an instruction set for performing one or more of the functions described above. These instructions need not be implemented as separate software programs, processes, or modules. Memory 1950 may include additional instructions or fewer instructions. Furthermore, various functions of computing device 1900 may be implemented in hardware and / or software, including in one or more signal processing and / or application specific integrated circuits.

[0169] To assist the Patent Office and any reader of any patent issuing upon this application in interpreting the appended claims, applicants wish to note that they do not intend any of the appended claims or claim elements to invoke 35 U.S.C. § 112(f) unless the words “means for” or “step for” are expressly used in a particular claim.

Claims

1. A method comprising: obtaining, by a computing device, a first captured image, the first captured image comprising a first perspective of an object captured from a first capture location; presenting, by the computing device, the first captured image on a display of the computing device; while presenting the first captured image, receiving, by the computing device, user input manipulating the first captured image; In response to receiving the user input: determining, by the computing device, a first virtual location within a threshold distance from the first capture location; transforming, by the computing device, the first captured image to depict a second perspective of the object from the first virtual position to generate a first transformed image; generating, by the computing device, an animation by transitioning from the first captured image to the first transitioned image and back to the first captured image, the animation simulating a parallax effect about the object, the parallax effect causing the object to appear three-dimensional; and The animation is presented by the computing device on a display of the computing device.

2. The method according to claim 1, further comprising: determining, by the computing device, a velocity associated with the user input; and determining, by the computing device, a lateral distance from the first capture location based on the velocity; and The first virtual position is determined based on the lateral distance.

3. The method according to claim 1, further comprising: determining, by the computing device, a duration associated with the user input; and determining, by the computing device, a backward distance from the first capture location to the object based on the duration; and The first virtual position is determined based on the rearward distance.

4. The method according to claim 1, further comprising: The threshold distance is adjusted based on the determined distance between the first capture location and the object.

5. The method according to claim 1, further comprising: obtaining, by the computing device, a three-dimensional model of the object; and The first captured image is transformed, by the computing device, based on the three-dimensional model to depict the second perspective of the object from the first virtual position.

6. The method according to claim 1, further comprising: generating, by the computing device, a plurality of transformed images corresponding to a plurality of virtual locations proximate to the first capture location, the plurality of transformed images including the first transformed image; generating, by the computing device, a photorealistic animation based on the plurality of intermediate images, the photorealistic animation simulating a parallax effect with respect to the object near the first capture location; and The photorealistic animation is presented by the computing device on a display of the computing device.

7. The method according to claim 6, further comprising: determining a frame rate corresponding to the computing device; and The plurality of virtual locations is determined based on the frame rate of the computing device.

8. A non-transitory computer-readable medium comprising one or more sequences of instructions which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

9. A system comprising: one or more processors; and A non-transitory computer-readable medium comprising one or more instruction sequences, which, when executed by the one or more processors, cause the processors to perform the method according to any one of claims 1-7.

10. A method comprising: obtaining, by a computing device, a first image and a second image; For each first pixel in the first image, determining, by the computing device, a corresponding second pixel in the second image; obtaining, by the computing device, a pixel quality score for each first pixel and corresponding second pixel; for one or more first pixels, comparing, by the computing device, a pixel quality score for the one or more first pixels with a pixel quality score for one or more corresponding second pixels; for each first pixel of the one or more first pixels, selecting, by the computing device, between the first pixel and a corresponding second pixel based on the comparison; and A composite image is generated by the computing device based on selected pixels from the first image and the second image.

11. The method according to claim 10, further comprising: capturing, by the computing device, a first image capture point view at the first image capture point; and The first image is generated by the computing device based on a perspective of a portion of the realistic three-dimensional model viewed from a position of the first image capture point to a perspective of a portion of the realistic three-dimensional model viewed from an intermediate position, the intermediate position being between the first image capture point and the second image capture point.

12. The method of claim 11, further comprising: The pixel quality scores for the one or more first pixels are weighted based on a distance between the intermediate position and the position of the first image capture point.

13. The method of claim 10, further comprising: capturing, by the computing device, a second image capture point view at the second image capture point; and The second image is generated by the computing device based on the perspective of a portion of the realistic three-dimensional model viewed from the position of the second image capture point to the perspective of a portion of the realistic three-dimensional model viewed from an intermediate position, wherein the intermediate position is between the second image capture point and the first image capture point.

14. The method according to claim 13, further comprising: The pixel quality score for one or more second pixels is weighted based on a distance between the intermediate position and the position of the second image capture point.

15. The method according to claim 10, further comprising: generating a pixel quality score for one or more first pixels based on an amount of stretching or compression applied to corresponding pixel locations within the first image; and A pixel quality score is generated for one or more second pixels based on an amount of stretching or compression applied to corresponding pixel locations within the second image.

16. The method according to claim 10, further comprising: generating a pixel quality score for one or more first pixels based on whether the corresponding pixel corresponds to a surface of the photorealistic three-dimensional model visible at the first image capture point; and A pixel quality score is generated for the one or more second pixels based on whether the corresponding pixel corresponds to a surface of the photorealistic three-dimensional model that is visible at the second image capture point.

17. A non-transitory computer-readable medium comprising one or more sequences of instructions that, when executed by one or more processors, cause the processors to perform the method of any one of claims 10-16.

18. A system comprising: one or more processors; and A non-transitory computer-readable medium comprising one or more instruction sequences, which, when executed by one or more processors, cause the processors to perform the method of any one of claims 10-16.

Images