Integrating media content into 3D scenes

Abstract

A method for integrating media content with a three-dimensional (3D) scene to provide an immersive view of a location includes obtaining a 3D scene of the location generated based on a plurality of images, receiving media content temporally associated with the location, integrating at least a portion of the media content with the 3D scene of the location, and providing an integrated 3D scene of the location having at least a portion of the media content integrated with the 3D scene of the location to represent a state of the location based on the temporal association of the media content with the location.

Description

[Technical field]

[0001] The present disclosure relates generally to providing an immersive view of a location. For example, the present disclosure relates to methods and systems for providing an immersive view of a location by integrating previously acquired images with media (e.g., images, sounds, etc.) to show the state of the location under various conditions. [Background technology]

[0002] A user uses a camera or a camera phone (e.g., a smartphone) to capture images of a point of interest, including a restaurant or a park. Although the images of the point of interest can be used to virtually navigate and experience a space, the images may not provide a viewer with an accurate representation of the condition of the point of interest. Additionally, because the images do not include people or dynamic objects, the sense of vibe or atmosphere of the point of interest may not be accurate. Summary of the Invention [Means for solving the problem]

[0003] Aspects and advantages of embodiments of the present disclosure are set forth in part in the description that follows, and may be learned from the description, or may be learned by practice of exemplary embodiments.

[0004] In one or more exemplary embodiments, a computer-implemented method is provided for integrating media content with a three-dimensional scene to provide an immersive view of a location. For example, the method includes acquiring a three-dimensional (3D) scene of the location generated based on a plurality of images, receiving media content temporally associated with the location, integrating at least a portion of the media content with the 3D scene of the location, and providing an integrated 3D scene of the location having at least a portion of the media content integrated with the 3D scene of the location to represent a state of the location based on the temporal association of the media content with the location.

[0005] In some implementations, the media content comprises user-generated media content captured by one or more users.

[0006] In some implementations, the user-generated media content comprises at least one of user-generated visual content or user-generated audio content.

[0007] In some implementations, the user-generated media content includes an image of a location including one or more real-world dynamic objects, and integrating at least a portion of the media content with a 3D scene of the location includes extracting the one or more real-world dynamic objects from the image of the location and determining one or more locations within the 3D scene for placing the one or more real-world dynamic objects.

[0008] In some implementations, the image of the location including one or more real-world dynamic objects includes a first image of the location including a first portion of the one or more real-world dynamic objects captured at the location by one or more users at a first time and a second image of the location including a second portion of the one or more real-world dynamic objects captured at the location by one or more users at a second time. Providing a 3D scene of the location having at least a portion of the media content integrated with the 3D scene of the location may include providing a first 3D scene of the location associated with a first time and having a first portion of the one or more real-world dynamic objects integrated with the first 3D scene of the location, and providing a second 3D scene of the location associated with a second time and having a second portion of the one or more real-world dynamic objects integrated with the second 3D scene of the location.

[0009] In some implementations, the user-generated media content includes first audio captured at the location by one or more users at a first time and second audio captured at the location by one or more users at a second time, and providing a 3D scene of the location having at least a portion of the media content integrated with the 3D scene of the location comprises providing a first 3D scene of the location having the first audio associated with the first time and integrated with the first 3D scene of the location, and providing a second 3D scene of the location having the second audio associated with the second time and integrated with the second 3D scene of the location.

[0010] In some implementations, the media content comprises machine-generated media content.

[0011] In some implementations, the machine-generated media content comprises at least one of machine-generated visual content or machine-generated audio content.

[0012] In some implementations, the machine-generated media content is synthesized based on user-generated media content captured by one or more users.

[0013] In some implementations, the user-generated media content includes an image of a location including one or more real-world dynamic objects. In some implementations, the method further comprises extracting the one or more real-world dynamic objects from the image and converting the one or more real-world dynamic objects into corresponding graphical representations of the one or more real-world dynamic objects, and integrating at least a portion of the media content with the 3D scene of the location includes determining one or more locations within the 3D scene for placing the corresponding graphical representations of the one or more real-world dynamic objects.

[0014] In some implementations, the one or more real-world dynamic objects include one or more humans, and the corresponding graphical representations of the one or more real-world dynamic objects include corresponding 3D digital avatars of the one or more humans.

[0015] In some implementations, the image of the location including one or more real-world dynamic objects includes a first image of the location including a first portion of the one or more real-world dynamic objects captured at the location by one or more users at a first time and a second image of the location including a second portion of the one or more real-world dynamic objects captured at the location by one or more users at a second time. Providing a 3D scene of the location having at least a portion of the media content integrated with the 3D scene of the location may include providing a first 3D scene of the location having a first graphical representation associated with a first time and corresponding to the first portion of the one or more real-world dynamic objects integrated with the first 3D scene of the location, and providing a second 3D scene of the location having a second graphical representation associated with a second time and corresponding to the second portion of the one or more real-world dynamic objects integrated with the second 3D scene of the location.

[0016] In some implementations, the user-generated media content includes a first audio captured at the location by one or more users at a first time and a second audio captured at the location by one or more users at a second time. In some implementations, the method further comprises synthesizing a first composite audio recording based on the first audio and synthesizing a second composite audio recording based on the second audio. Providing a 3D scene of the location having at least a portion of the media content integrated with the 3D scene of the location comprises providing a first 3D scene of the location having the first composite audio recording associated with a first time and integrated with the first 3D scene of the location, and providing a second 3D scene of the location having the second composite audio recording associated with a second time and integrated with the second 3D scene of the location.

[0017] In some implementations, the method further includes synthesizing machine-generated media content based on sensor data acquired by one or more sensors disposed at the location.

[0018] In some implementations, synthesizing the machine-generated media content based on sensor data acquired by one or more sensors positioned at the location comprises determining a number of people at the location at a first time based on the sensor data acquired by the one or more sensors; determining characteristics related to the people at the location at the first time based on the sensor data acquired by the one or more sensors; and generating a graphical representation of the people at the location according to the number of people at the location at the first time and according to the characteristics related to the people to indicate a state of the location at the first time, wherein integrating at least a portion of the media content with the 3D scene of the location includes determining one or more locations within the 3D scene to place the graphical representation of the people.

[0019] In one or more exemplary embodiments, a computing device (e.g., a laptop, a tablet, a smartphone, etc.) is provided. The computing device may include an input device, a display device, at least one memory for storing instructions, and at least one processor configured to execute the instructions to perform operations. The operations may include receiving a first input via the input device requesting a first immersive view of a location representative of a first state of the location at a first time, and providing, for display on the display device, the first immersive view of the location including a three-dimensional (3D) scene of the location generated based on a plurality of images and first media content integrated with the 3D scene of the location, the first media content representing the first state of the location at the first time. The operations may further include receiving a second input via the input device requesting a second immersive view of the location representing a second state of the location at a second time; and providing the second immersive view of the location for display on the display device, the second immersive view of the location including a 3D scene of the location generated based on the plurality of images and second media content integrated with the 3D scene of the location, the second media content representing the second state of the location at the second time.

[0020] In some implementations, the first media content comprises at least one of user-generated media content captured by one or more users or machine-generated media content, where the user-generated media content comprises at least one of user-generated visual content or user-generated audio content, and the machine-generated media content comprises at least one of machine-generated visual content or machine-generated audio content.

[0021] In some implementations, the operations further comprise receiving a third input via the input device to provide user-generated media content captured by a user of the computing device, the user-generated media content being temporally associated with the first time; transmitting the user-generated media content to at least one of a server computing system or a database configured to provide a first immersive view of the location representative of a first state of the location at the first time; receiving a fourth input via the input device requesting the first immersive view of the location representative of the first state of the location at the first time; and providing the first immersive view of the location for display on a display device, the first immersive view of the location including a 3D scene of the location generated based on the plurality of images and updated first media content integrated with the 3D scene of the location, the updated first media content representing the first state of the location at the first time and including at least a portion of the user-generated media content captured by the user of the computing device.

[0022] In one or more exemplary embodiments, a server system (e.g., a server) is provided. The server system may include at least one memory for storing instructions and at least one processor configured to execute the instructions to perform operations, the operations comprising: acquiring a three-dimensional (3D) scene of a location generated based on a plurality of images, receiving media content temporally associated with the location, integrating at least a portion of the media content with the 3D scene of the location, and providing the 3D scene of the location having at least a portion of the media content integrated with the 3D scene of the location to represent a state of the location based on the temporal association of the media content with the location.

[0023] In some implementations, the media content comprises at least one of user-generated media content captured by one or more users or machine-generated media content, where the user-generated media content comprises at least one of user-generated visual content or user-generated audio content, and the machine-generated media content comprises at least one of machine-generated visual content or machine-generated audio content.

[0024] In one or more exemplary embodiments, a computer-readable medium (e.g., a non-transitory computer-readable medium) is provided that stores instructions executable by one or more processors of a computing system. In some implementations, the computer-readable medium stores instructions that may include instructions for causing one or more processors to perform one or more operations associated with any of the methods described herein (e.g., operations of a server computing system and / or operations of a computing device). The computer-readable medium may store additional instructions for performing other aspects of the server computing system and computing device, and corresponding methods of operation, as described herein.

[0025] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description, drawings, and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain the associated principles.

[0026] A detailed description of exemplary embodiments, directed to those skilled in the art, is set forth herein with reference to the accompanying drawings. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 illustrates an example system, in accordance with one or more exemplary embodiments of the present disclosure. [Diagram 2] FIG. 1 illustrates an example block diagram of a computing device and a server computing system in accordance with one or more example embodiments of the present disclosure. [Diagram 3] 1 is a flow diagram of an exemplary, non-limiting, computer-implemented method according to one or more exemplary embodiments of the present disclosure. [Figure 4] FIG. 2 illustrates a user interface screen of a map application, in accordance with one or more exemplary embodiments of the present disclosure. [Figure 5A] A diagram showing an example of an immersive view from a video fly-through of a position, in accordance with one or more exemplary embodiments of the present disclosure. [Figure 5B] A diagram showing an example of an immersive view from a video fly-through of a position, in accordance with one or more exemplary embodiments of the present disclosure. [Figure 6] FIG. 1 illustrates an exemplary immersive view of a location including an integrated 3D scene of the location, in accordance with one or more exemplary embodiments of the present disclosure. [Figure 7] FIG. 1 illustrates an exemplary computing device having a user interface screen through which a user can upload user-generated content, in accordance with one or more exemplary embodiments of the present disclosure. [Figure 8A] 1A-1C illustrate example user interface screens through which a user can request an immersive view of a location associated with a particular time, in accordance with one or more exemplary embodiments of the present disclosure. [Figure 8B] 1A-1C illustrate example user interface screens through which a user can request an immersive view of a location associated with a particular time, in accordance with one or more exemplary embodiments of the present disclosure. [Figure 9] 1 is a flow diagram of an exemplary, non-limiting, computer-implemented method according to one or more exemplary embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Reference will now be made to the embodiments of the present disclosure, one or more examples of which are illustrated in the drawings, in which like reference numerals refer to like elements. Each example is provided for the purpose of illustrating the invention, and is not intended to limit the invention. Indeed, it will be apparent to those skilled in the art that various modifications and variations can be made to the present disclosure without departing from the scope or spirit of the disclosure. For example, features illustrated or described as part of one embodiment can be used with another embodiment to yield yet another embodiment. It is therefore intended that the present disclosure cover such modifications and variations as come within the scope of the appended claims and their equivalents.

[0029] The terms used herein are used to describe exemplary embodiments and are not intended to limit and / or restrict the disclosure. The singular forms "a", "an" and "the" are intended to include the plural unless the context clearly dictates otherwise. In this disclosure, terms such as "including", "having", "comprising" and the like are used to specify features, numbers, steps, operations, elements, components, or combinations thereof, but do not exclude the presence or addition of one or more of the features, elements, steps, operations, elements, components, or combinations thereof.

[0030] In this specification, terms such as first, second, third, etc. may be used to describe various elements, but it will be understood that the elements are not limited by these terms. Instead, these terms are used to distinguish one element from another element. For example, a first element may be referred to as a second element and a second element may be referred to as a first element without departing from the scope of the present disclosure.

[0031] The term "and / or" includes a combination of multiple associated list items or any items of multiple associated list items. For example, the scope of an expression or phrase "A and / or B" includes item "A," item "B," and the combination of items "A and B."

[0032] Additionally, the scope of an expression or phrase "at least one of A or B" is intended to include all of (1) at least one of A, (2) at least one of B, and (3) at least one of A and at least one of B. Similarly, the scope of an expression or phrase "at least one of A, B, or C" is intended to include all of (1) at least one of A, (2) at least one of B, (3) at least one of C, (4) at least one of A and at least one of B, (5) at least one of A and at least one of C, (6) at least one of B and at least one of C, and (7) at least one of A, at least one of B, and at least one of C.

[0033] Examples of this disclosure are directed to computer-implemented methods for integrating (e.g., embedding) three-dimensional (3D) scenes of media content and locations to represent the state of the location based on a temporal association between the media content and the location. For example, the state of a location under specified conditions (e.g., a particular time, particular weather conditions, etc.) may include the number of people at the location (e.g., crowded, empty, etc.), the general state of the atmosphere (e.g., cheerful, lively, calm, etc.), expected attire (e.g., formal wear, trendy, casual, sportswear, etc.), noise level (e.g., peaceful, noisy, etc.).

[0034] According to an exemplary embodiment, a navigation and mapping system may include an immersive view application to provide a user of a computing device with a way to explore a location through a multi-dimensional view of an area or point of interest, including landmarks, restaurants, and the like. The immersive view application may be part of a navigation application or a separate mapping application, or may be a standalone application. The immersive view may include an integrated 3D scene of a location, where the integrated 3D scene of the location is formed by integrating media content with an original or existing 3D scene of the location. The media content integrated with the 3D scene of a location may be temporally associated with the location such that the integrated 3D scene provides an accurate representation of the state of the location at a particular time or under other particular conditions. Additionally, an impression or feeling of the location at a particular time may be more accurately represented. In some implementations, the media content may be associated with the location in other aspects, including environmental aspects such as weather conditions, lighting conditions, and the like.

[0035] According to an example of the present disclosure, a server computing system can provide an integrated 3D scene of a location to a computing device to present on a display device of the computing device. The integrated 3D scene of a location may be provided dynamically (e.g., generated and transmitted in response to a request from the computing device), or the integrated 3D scene of a location may be provided by retrieving an integrated 3D scene of a location from a database. According to the conditions of the request, the integrated 3D scene of the location may be retrieved from the database. For example, if a user requests an immersive view of a restaurant at 6 p.m. on Friday, the server computing system may retrieve an integrated 3D scene of the location from the database that substantially matches the conditions entered by the user (e.g., an immersive view of the restaurant at a particular time).

[0036] The integrated 3D scene of a location may be generated by the server computing system by integrating media content, which may include user-generated media content, machine-generated media content, or a combination thereof. For example, the user-generated media content may include an image of a location including one or more real-world dynamic objects (e.g., people, animals, vehicles, etc.). The server computing system may be configured to integrate at least a portion of the media content with the 3D scene of the location by extracting the one or more real-world dynamic objects from the image of the location and determining one or more locations within the 3D scene to place the one or more real-world dynamic objects. For example, the server computing system may extract the dynamic objects via known image analysis techniques including image segmentation and may identify locations within the 3D scene to appropriately place the real-world dynamic objects.

[0037] In some implementations, the images of the location including one or more real-world dynamic objects include a first image of the location including a first portion of the one or more real-world dynamic objects captured at the location at a first time by one or more users (e.g., one or more computing devices) and a second image of the location including a second portion of the one or more real-world dynamic objects captured at the location at a second time by one or more users (e.g., one or more computing devices). That is, the user-generated media content may include images with different temporal relevance (e.g., images captured in the morning, midday, at night, etc.).

[0038] The server computing system may be configured to generate and provide a first 3D scene of the location associated with a first time and having a first portion of one or more real-world dynamic objects integrated with the first 3D scene of the location, and to generate and provide a second 3D scene of the location associated with a second time and having a second portion of one or more real-world dynamic objects integrated with the second 3D scene of the location.

[0039] In some implementations, the user-generated media content includes first audio captured at a first time by one or more users (e.g., one or more computing devices) at the location and second audio captured at a second time by one or more users (e.g., one or more computing devices) at the location. That is, the user-generated media content may include audio content with different temporal relevance (e.g., audio captured or recorded in the morning, midday, night, etc.). For example, the user-generated media content may include audio content with different environmental relevance (e.g., audio captured or recorded in sunny conditions, rainy conditions, windy conditions, etc.).

[0040] The server computing system may be configured to generate and provide a first 3D scene of the location associated with a first time and having first audio integrated with the first 3D scene of the location, and to generate and provide a second 3D scene of the location associated with a second time and having second audio integrated with the second 3D scene of the location.

[0041] In some implementations, the media content integrated with the 3D scene of the location may include machine-generated media content. For example, the user-generated media content may include an image of the location including one or more real-world dynamic objects. The server computing system may be configured to extract the one or more real-world dynamic objects from the image and convert the one or more real-world dynamic objects into corresponding graphical representations of the one or more real-world dynamic objects (i.e., into the machine-generated media content). Extracting the real-world dynamic objects from the image may comprise identifying an object type or characteristic of the real-world dynamic objects to identify the corresponding graphical representations of the real-world dynamic objects in a fast and efficient manner. In another implementation, the server computing system may be configured to first convert the image into the corresponding graphical representation and extract the graphical representations of the one or more real-world dynamic objects. The server computing system may be configured to integrate the graphical representations of the one or more real-world dynamic objects with the 3D scene of the location by determining one or more locations in the 3D scene to place the corresponding graphical representations of the one or more real-world dynamic objects. By using a graphical representation of a real-world dynamic object, integration with the 3D scene may be accomplished more quickly and efficiently, since the system only needs to identify the type or characteristics of the real-world dynamic object (e.g., a person) to generate an appropriate and accurate graphical representation. Thus, such a graphical representation provides an accurate prediction of the location state.

[0042] In some implementations, the one or more real-world dynamic objects include one or more humans (e.g., the object has a type of “human”), and the corresponding graphical representations of the one or more real-world dynamic objects include corresponding 3D digital avatars of the one or more humans.

[0043] In some implementations, the user-generated media content includes a first audio captured at a first time by one or more users (e.g., one or more computing devices) at the location and a second audio captured at a second time by one or more users (e.g., one or more computing devices) at the location. The server computing system may be configured to convert the first audio and / or the second audio into synthetic audio (i.e., machine-generated media content). Converting the audio into synthetic audio may comprise identifying a type or characteristic of the audio to identify the corresponding synthetic audio in a fast and efficient manner. Such synthetic audio provides an accurate prediction of the state of the location. For example, the server computing system may be configured to synthesize a first synthetic audio recording based on the first audio and synthesize a second synthetic audio recording based on the second audio. The server computing system may be configured to generate and provide a first 3D scene of the location having a first synthetic audio recording associated with a first time and integrated with the first 3D scene of the location, and to generate and provide a second 3D scene of the location having a second synthetic audio recording associated with a second time and integrated with the second 3D scene of the location. By using the synthetic audio instead of the captured audio itself, integration with the 3D scene may be achieved more quickly because the system only needs to identify a type or characteristic (e.g., genre) of the captured audio in order to generate an appropriate and accurate synthetic audio representation of the actual audio.

[0044] In some implementations, the server computing system may be configured to synthesize machine-generated media content based on sensor data acquired by one or more sensors disposed at a location associated with the 3D scene. For example, the server computing system may be configured to determine a number of people at the location at a first time based on the sensor data acquired by the one or more sensors, determine characteristics related to the people at the location at the first time based on the sensor data acquired by the one or more sensors, generate a graphical representation of the people at the location according to the number of people at the location at the first time and according to the characteristics related to the people to show an accurate representation of the state of the location at the first time, and integrate the graphical representation of the people at the location with the 3D scene of the location by determining one or more locations in the 3D scene to place the graphical representation of the people. For example, the sensor data may indicate that there are 20 people in a restaurant and the image may indicate that formal attire is worn at the restaurant, or external content obtained from a web page associated with the restaurant may indicate that formal attire is required at the restaurant. Thus, the server computing system may be configured to generate a graphical representation of people in the restaurant according to the number of people (approximately 20 people) and according to known or predicted characteristics of the people at the location at the first time (e.g., wearing formal attire) to accurately represent the state of the location at the first time. For example, the server computing system may integrate the graphical representation of the people in the restaurant with the 3D scene of the restaurant by determining one or more locations within the 3D scene to place the graphical representations of the people (e.g., at a table, at the bar, etc.).

[0045] One or more technical advantages of the present disclosure include allowing a user to easily and more accurately obtain an accurate representation of the state of a location under a particular situation or condition. For example, a user can easily and more accurately obtain an accurate representation of the state of an indoor or outdoor venue, including a restaurant or a park, at a particular time of day, time of day, season, etc. For example, a user can easily and more accurately obtain an accurate representation of the state of an indoor or outdoor venue, including a restaurant or a park, under particular environmental conditions (e.g., when it is sunny, when it is raining, when it is windy, etc.). The above method provides a user with an accurate representation of the state of the location virtually and via a display without the user having to actually go to the location. Additionally, the user can also be provided with an accurate prediction of the state of the location at a particular time or under particular conditions as defined by the user.

[0046] One or more technical advantages of the present disclosure also include integrating fresh media content (e.g., user-generated media content and / or machine-generated media content) associated with a location with an existing 3D scene of the location. For example, the media content may be acquired subsequent to images used to form a 3D model that forms the 3D scene of the location. Thus, the integrated 3D scene represents an accurate and updated state of the location. Furthermore, various integrated 3D scenes may be generated to accurately depict the location according to various conditions. For example, the server computing system is configured to select media content to integrate based on information associated with media content that matches a user's request for an immersive view. For example, an image of the interior of a restaurant taken in the morning when there are few customers will not be integrated into an integrated 3D scene generated for an immersive view of the restaurant at dinner time. Thus, the use of metadata and other descriptive content associated with the media content may be used to form an integrated 3D scene of the location in an accurate manner. Similarly, image segmentation techniques and machine learning resources may be implemented to place or install dynamic objects extracted from media content in appropriate locations within the integrated 3D scene of the location to accurately provide the state of the location.

[0047] Thus, in accordance with aspects of the present disclosure, technical advantages such as resource conservation and increased accuracy of the immersive view may be achieved.

[0048] Referring now to the drawings, FIG. 1 is an exemplary system according to one or more exemplary embodiments of the present disclosure. FIG. 1 illustrates an example of a system including a computing device 100, an external computing device 200, a server computing system 300, and external content 500 that may communicate with each other via a network 400. For example, the computing device 100 and the external computing device 200 may include any of a personal computer, a smartphone, a tablet computer, a global positioning service device, a smart watch, and the like. The network 400 may include any type of communication network, including a wired or wireless network, or a combination thereof. The network 400 may include a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a personal area network (PAN), a virtual private network (VPN), and the like. For example, wireless communications between elements of exemplary embodiments may be performed via wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), Ultra Wideband (UWB), Infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Radio Frequency (RF) signals, etc. For example, wired communications between elements of exemplary embodiments may be performed via cable pairs, coaxial cables, fiber optic cables, Ethernet cables, etc. Communications over the network may use a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0049] As described in more detail below, in some implementations, the computing device 100 and / or the server computing system 300 may form part of a navigation and mapping system that can provide a user of the computing device 100 with an immersive view of its location.

[0050] In some demonstrative embodiments, the server computing system 300 may obtain data from one or more of the user-generated content data store 350, the machine-generated content data store 360, the POI data store 370, the navigation data store 380, and the user data store 390 to implement various operations and aspects of the navigation and mapping system disclosed herein. The user-generated content data store 350, the machine-generated content data store 360, the POI data store 370, the navigation data store 380, and the user data store 390 may be provided integrally with the server computing system 300 (e.g., as part of one or more memory devices 320 of the server computing system 300) or separately (e.g., remotely). Furthermore, the user-generated content data store 350, the machine-generated content data store 360, the POI data store 370, the navigation data store 380, and the user data store 390 may be combined as a single data store (database) or may be multiple respective data stores. Data stored in one data store (e.g., POI data store 370) may overlap with some data stored in another data store (e.g., navigation data store 380). In some implementations, one data store (e.g., machine-generated content data store 360) may reference data stored in another data store (e.g., user-generated content data store 350).

[0051] User-generated content data store 350 can store media content captured by a user, for example, via computing device 100, external computing device 200, or some other computing device. User-generated media content may include user-generated visual content and / or user-generated audio content. For example, media content may be captured by a person operating a computing device or may be captured indirectly, for example, by a computing system monitoring a location (e.g., a security system, a surveillance system, etc.).

[0052] For example, the media content may be captured by a camera (e.g., image capturer 182) of a computing device and may include images of locations including restaurants, landmarks, businesses, schools, etc. The images may include various information (e.g., metadata, semantic data, etc.) that is useful for integrating the image (or a portion of the image) into a 3D scene of the location associated with the image. For example, the image may include information including the date the image was captured, the time the image was captured, location information indicating the location where the image was taken (e.g., GPS location), etc. For example, descriptive metadata may be provided with the image and may include keywords associated with the image, a title or name of the image, environmental information at the time the image was captured (e.g., lighting conditions including brightness levels, noise conditions including decibel levels, weather information including weather conditions such as temperature, wind, precipitation, cloudiness, humidity, etc.), etc. The environmental information may be obtained from sensors in the computing device used to capture the image or from another computing device.

[0053] For example, the media content may include audio that may be captured by a microphone (e.g., sound capturer 184) of a user computing device and associated with locations including restaurants, landmarks, businesses, schools, etc. The audio content may include various information (e.g., metadata, semantic data, etc.) that is useful for integrating the audio content (or a portion of the audio) with a 3D scene of the location associated with the audio content. For example, the audio content may include information including the date the audio was captured, the time the audio was captured, location information indicating the location where the audio was taken (e.g., GPS location), etc. For example, descriptive metadata may be provided with the audio and may include keywords associated with the audio, a title or name of the audio, environmental information at the time the audio was captured (e.g., lighting conditions including brightness levels, noise conditions including decibel levels, weather information including weather conditions such as temperature, wind, precipitation, cloudiness, humidity, etc.), etc. The environmental information may be obtained from sensors in the computing device used to capture the audio or from another computing device.

[0054] The machine-generated content data store 360 ​​can store machine-generated media content, which may be generated, for example, by the server computing system 300 (e.g., the machine-generated content generator 340) or some other computing device. The machine-generated media content may include machine-generated visual content and / or machine-generated audio content. For example, the machine-generated content stored in the machine-generated content data store 360 ​​can be generated based on user-generated media content captured by one or more users of a computing device and / or based on synthetic media content created by the machine-generated content generator 340 of the server computing system 300 or some other computing device.

[0055] In some implementations, the machine-generated content stored in the machine-generated content data store 360 ​​may be generated by the machine-generated content generator 340 by converting the user-generated media content into a common format (e.g., by converting a real-world image of a person positioned at a location into a two-dimensional or three-dimensional digital avatar representing that person) in order to anonymize the media content.

[0056] In some implementations, the machine-generated content stored in the machine-generated content data store 360 ​​may be generated by the machine-generated content generator 340 based on sensor data acquired by one or more sensors (which may form part of the external content 500) placed at a location. For example, the sensor data acquired by the one or more sensors may indicate how many people are at the location (e.g., based on the number of smartphones or other computing devices detected at the location). For example, the sensor data acquired by the one or more sensors may indicate various characteristics about the people at the location (e.g., clothing, facial expressions, etc., based on images captured by a camera). For example, the image synthesizer 342 may generate a graphical representation of people at the location according to the number of people at the location and according to characteristics about the people at the location to accurately represent the status of the location. As previously mentioned, the generation of such a graphical representation may be based on an identification of a type or characteristic of an object. For example, upon determining that the sensor data indicates that the object is a person, a graphical representation of a person may be generated. As another example, the sensor data may indicate that the object is a person wearing a hat, in which case a graphical representation of a person wearing a hat may be generated.

[0057] In some implementations, the machine-generated content stored in the machine-generated content data store 360 ​​may be generated by the machine-generated content generator 340 by creating new media content based on portions of user-generated media content. For example, the audio synthesizer 344 may generate audio content based on portions of recorded user-generated audio content (e.g., granular information) or other available sounds to create new audio content that represents the mood, atmosphere, impression, or feeling of the location at a particular time (e.g., time of day, day of the week, time of year, etc.). As previously mentioned, the generation of the audio content may be based on an identification of the audio type or characteristics of the portions of the recorded user-generated audio content, or other available sounds. For example, the user-generated audio content may have an audio type of "country music," in which case the generated audio content may also be country music to accurately represent the state of the location.

[0058] The POI data store 370 can store information about locations or points of interest, e.g., points of interest within an area or region associated with one or more geographic areas. Points of interest may include any destination or location. For example, points of interest may include restaurants, museums, sports venues, concert halls, amusement parks, schools, businesses, grocery stores, gas stations, theaters, shopping malls, lodging, etc. The points of interest data stored in the POI data store 370 may include any information associated with a POI. For example, the POI data store 370 may include location information of the POI, opening hours of the POI, a phone number of the POI, reviews about the POI, financial information associated with the POI (e.g., average cost of services offered and / or goods sold at the POI, such as meals, tickets, rooms, etc.), environmental information about the POI (e.g., noise level, description of ambience, traffic level, etc., which may be provided in real time or available by various sensors located at the POI), a description of the types of services offered and / or goods sold, the language spoken at the POI, the URL of the POI, image content associated with the POI, etc. For example, information about the POI may be obtainable from external content 500 (e.g., from a web page associated with the POI or from sensors located at the POI).

[0059] The navigation data store 380 may store or provide map data / geospatial data used by the server computing system 300. Exemplary geospatial data includes geographic imagery (e.g., digital maps, satellite imagery, aerial photographs, road-level photographs, synthetic models, etc.), tables, vector data (e.g., vector representations of roads, parcels, buildings, etc.), points of interest data, or other suitable geospatial data associated with one or more geographic areas. In some examples, the map data may include a series of sub-maps, each sub-map including data for a geographic area including objects (e.g., buildings or other static features), travel paths (e.g., roads, highways, public transportation, sidewalks, etc.), and other features of interest. The navigation data store 380 may be used by the server computing system 300 to provide navigation directions, perform searches for points of interest, provide location or categorization data for points of interest, determine distances, routes, or travel times between locations, or any other suitable use or task necessary or beneficial for performing the operations of the exemplary embodiments disclosed herein.

[0060] For example, the navigation data store 380 may store 3D scene images 382 including images associated with generating 3D scenes of various locations. In one example, the 3D scene generator 336 may be configured to generate a 3D scene based on multiple images of a location (e.g., an image of the interior of a restaurant, an image of a park, etc.). The multiple images may be captured and combined using known methods to create a 3D scene of the location. For example, images that overlap each other may be stitched together to create a 3D model of the scene. In some implementations, methods including structure from motion algorithms may be used to estimate the three-dimensional structure. In some implementations, machine learning resources may be implemented to generate camera-like images from any viewpoint within the location based on the captured images. For example, a video fly-through of the location may be generated based on the captured images. In some implementations, the initial 3D scene generated by the 3D scene generator 336 may be a static 3D scene lacking variable or dynamic (e.g., moving) objects. For example, an initial 3D scene of a park may include images of the park that do not include images of people, dogs, or non-static objects, but include images of trees, play equipment, picnic tables, etc. User-generated content may include images of variable or dynamic objects, and the images may be associated with different times and / or conditions (e.g., different times of the day, week, or year, different lighting conditions, different environmental conditions, etc.).

[0061] For example, the navigation data store 380 may store an integrated 3D scene image 384 including 3D scenes of various locations in which media content is integrated. In one example, the 3D scene integrator 338 may be configured to integrate user-generated content from the user-generated content data store 350 and / or machine-generated content from the machine-generated content data store 360 ​​with the 3D scene obtained from the 3D scene image 382. For example, the integrated 3D scene image 384 may include 3D scenes of various locations in which media content is integrated. The 3D scene generated based on multiple images of a location may be integrated with the media content using known methods to create an integrated 3D scene image 384 of the location. For example, the 3D scene integrator 338 may be configured to identify and extract one or more objects (e.g., one or more dynamic objects) from the image of the scene. For example, the 3D scene integrator 338 may be configured to place or install the extracted one or more objects in the 3D scene associated with the extracted one or more objects. For example, the 3D scene integrator 338 may select a 3D scene from the 3D scene images 382 that correspond to an image of a scene having one or more objects. For example, the 3D scene integrator 338 may select a 3D scene from the 3D scene images 382 that have the highest similarity (e.g., with respect to time of day, season, weather conditions, lighting conditions, etc.) to the image of the scene having one or more objects. For example, a user-generated image taken at noon in a park in sunny conditions may include several children playing on play equipment. The 3D scene integrator 338 may be configured to extract the children from the image using various techniques (e.g., image segmentation algorithms, machine learning resources, cropping tools, etc.). The 3D scene integrator 338 may be configured to select a 3D scene from the 3D scene images 382 that have similar characteristics to the image (e.g., similar time of day, season, sunny weather conditions, etc.).The 3D scene integrator 338 may be configured to place the extracted image of the child within the 3D scene to generate an updated or integrated 3D scene in which the child is placed within the scene (e.g., on a slide, on a see-saw, etc.) so that a user viewing the integrated 3D scene can be shown how the park generally feels at that time of day, e.g., in similar weather conditions, along with an accurate representation of the park's conditions at that time of day. In some implementations, audio content may also be integrated with the 3D scene. For example, the 3D scene integrator 338 may be configured to integrate sound with the 3D scene in a manner similar to integrating images. For example, audio recorded at the park at noon may include children's laughter, while audio recorded at the park at night may include wildlife sounds. The 3D scene integrator 338 may be configured to integrate the laughter with the 3D scene associated with a similar time frame of audio (e.g., using actual recordings taking into account privacy considerations or user permission, or machine-generated audio including synthesized laughter). For example, an integrated 3D scene in which children are placed in the scene (e.g., on a slide, on a see-saw, etc.) and has audio content played while the user views the immersive view may not only provide the user viewing the integrated 3D scene with an accurate representation of the condition of the 3D scene at that time of day, but may also provide a more accurate idea of ​​what the park may feel like at that time of day, e.g., under similar weather and noise conditions.

[0062] The media content, including user-generated content and / or machine-generated content, may include audio content and / or images of variable or dynamic objects, and the audio content and images may be associated with different times and / or conditions (e.g., different times of a day, week, or year, different lighting conditions, different environmental conditions, etc.). The 3D scene integrator 338 may be configured to integrate the user-generated content and / or machine-generated content with an initial 3D scene generated by the 3D scene generator 336, for example, according to time information associated with the media content. For example, a first integrated 3D scene of a location may be associated with a first time (e.g., a first hour of a day, a first hour of a year, etc.) based on media content captured at or associated with the first time, and a second integrated 3D scene of the location may be associated with a second time (e.g., a second hour of a day, a second hour of a year, etc.) based on media content captured at or associated with the second time.

[0063] In some exemplary embodiments, the user data store 390 may represent a single database. In some embodiments, the user data store 390 represents multiple different databases accessible to the server computing system 300. In some examples, the user data store 390 may include current user location and orientation data. In some examples, the user data store 390 may include information related to one or more user profiles, including various user data such as user preference data, user demographic data, user calendar data, user social network data, user historical travel data, and the like. For example, the user data store 390 may include, but is not limited to, text content, images, calendar information associated with an email or email data including contact information, social media data including comments, reviews, check-ins, likes, invitations, contacts, or reservations, calendar application data including dates, times, events, descriptions, or other content, virtual wallet data including purchases, e-tickets, coupons, or transactions, schedule data, location data, SMS data, or other suitable data associated with a user account. According to one or more examples of the present disclosure, data can be analyzed to determine user preferences for POIs, e.g., to automatically suggest or automatically provide immersive views of locations preferred by the user, where the immersive views are also associated with times preferred by the user (e.g., if a park is a user's favorite POI and the user data indicates that the user visits the park most often in the evening, provide an immersive view of the park in the evening). Data can be analyzed to determine user preferences for POIs, such as to determine user preferences for travel (e.g., mode of transportation, allowable time for travel, etc.), to determine possible recommendations for the user's POIs, and to determine possible travel routes and modes of transportation for the user to the POIs.

[0064] The user data store 390 is provided, in some embodiments, to indicate potential data that may be analyzed by the server computing system 300 to identify user preferences, recommend POIs, determine possible travel routes to a POI, determine the mode of transportation to be used to travel to a POI, determine an immersive view of the location to provide to a computing device associated with the user, and the like. However, such user data may not be collected, used, or analyzed unless the user consents after being informed of what data is collected and how such data is used. Additionally, in some embodiments, the user may be provided with tools (e.g., within the navigation application or via the user account) to revoke or modify the scope of the permissions. Additionally, certain information or data is processed in one or more ways before being stored or used, such that personally identifiable information is deleted or stored in an encrypted manner. Thus, certain user information stored in the user data store 390 may be accessible or inaccessible to the server computing system 300 based on the permissions given by the user, or such data may not be stored in the user data store 390 at all.

[0065] The external content 500 may be any form of external content, including news articles, web pages, video files, audio files, written descriptions, ratings, gaming content, social media content, photos, commercial offers, transportation, weather conditions, sensor data acquired by various sensors, or other suitable external content. The computing device 100, the external computing device 200, and the server computing system 300 may access the external content 500 via the network 400. The external content 500 may be searched by the computing device 100, the external computing device 200, and the server computing system 300 according to known search methodologies, and the search results may be ranked according to relevance, popularity, or other suitable attributes, including location-specific filtering or promotions.

[0066] Referring now to Figure 2, an exemplary block diagram of a computing device and a server computing system will be described in accordance with one or more exemplary embodiments of the present disclosure. Although a computing device 100 is shown in Figure 2, the features of the computing device 100 described herein are also applicable to an external computing device 200.

[0067] The computing device 100 may include one or more processors 110, one or more memory devices 120, a navigation and mapping system 130, a location determination device 140, an input device 150, a display device 160, an output device 170, and a capture device 180. The server computing system 300 may include one or more processors 310, one or more memory devices 320, a navigation and mapping system 330, and a machine-generated content generator 340.

[0068] For example, the one or more processors 110, 310 may be any suitable processing device that may be included in the computing device 100 or the server computing system 300. For example, the one or more processors 110, 310 may include one or more of a processor, processor core, controller and arithmetic logic unit, central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), image processor, microcomputer, field programmable array, programmable logic unit, application specific integrated circuit (ASIC), microprocessor, microcontroller, etc., as well as combinations thereof including any other device capable of responding to and executing instructions in a defined manner. The one or more processors 110, 310 may be a single processor or multiple processors operably connected, for example, in parallel.

[0069] The one or more memory devices 120, 320 may include read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), and one or more non-transitory computer readable storage media including flash memory, USB drives, random access memory (RAM), volatile memory devices including hard disks, floppy disks, Blu-ray disks, or optical media such as CD ROM disks and DVDs, and combinations thereof. However, examples of the one or more memory devices 120, 320 are not limited to the above description, and the one or more memory devices 120, 320 may be realized by various other devices and structures, as will be appreciated by those skilled in the art.

[0070] For example, the one or more memory devices 120 may store instructions that, when executed, cause the one or more processors 110 to execute an immersive view application 132 as well as to perform operations including receiving a first input via the input device 150 requesting a first immersive view of a location representing a first state of the location at a first time, and providing a first immersive view of the location for display on the display device 160, the first immersive view of the location including a three-dimensional (3D) scene of the location generated based on a plurality of images and first media content integrated with the 3D scene of the location, the first media content representing the first state of the location at the first time. The operations may further include, as described according to examples of the present disclosure, receiving a second input via input device 150 requesting a second immersive view of the location representing a second state of the location at a second time, and providing the second immersive view of the location for display on display device 160, the second immersive view of the location including a 3D scene of the location generated based on the plurality of images and second media content integrated with the 3D scene of the location, the second media content representing the second state of the location at the second time.

[0071] The one or more memory devices 120 may also include data 122 and instructions 124 that may be retrieved, manipulated, created, or stored by the one or more processors 110. In some demonstrative embodiments, such data is accessed as input and used to implement the immersive view application 132 to carry out instructions to perform operations including receiving a first input via the input device 150 requesting a first immersive view of a location representative of a first state of the location at a first time, and providing, for display on the display device 160, the first immersive view of the location including a three-dimensional (3D) scene of the location generated based on a plurality of images and first media content integrated with the 3D scene of the location, the first media content representing the first state of the location at the first time. The operations may further include, as described according to examples of the present disclosure, receiving a second input via input device 150 requesting a second immersive view of the location representing a second state of the location at a second time, and providing the second immersive view of the location for display on display device 160, the second immersive view of the location including a 3D scene of the location generated based on the plurality of images and second media content integrated with the 3D scene of the location, the second media content representing the second state of the location at the second time.

[0072] In some exemplary embodiments, computing device 100 includes a navigation and mapping system 130. For example, navigation and mapping system 130 may include an immersive view application 132 and a navigation application 134.

[0073] According to examples of the present disclosure, the immersive view application 132 may be performed by the computing device 100 to provide a user of the computing device 100 with a way to explore a location through a multi-dimensional view of an area or point of interest including landmarks, restaurants, etc. In some implementations, the immersive view application 132 may provide a video fly-through of a location to provide a user with an inside view of the location. The immersive view application 132 may be part of the navigation application 134 or a separate mapping application, or may be a standalone application.

[0074] In some examples, one or more aspects of the immersive view application 132 may be implemented by an immersive view application 332 of a server computing system 300, which may be remotely located, to provide the requested immersive view. In some examples, one or more aspects of the immersive view application 332 may be implemented by an immersive view application 132 of a computing device 100, to generate the requested immersive view.

[0075] According to examples of the present disclosure, for example, the navigation application 134 may be performed by the computing device 100 to provide a user of the computing device 100 with a way to navigate to a location. The navigation application 134 may provide navigation services to the user. In some examples, the navigation application 134 may facilitate a user's access to the server computing system 300 that provides navigation services. In some exemplary embodiments, the navigation services include providing directions to a particular location, such as a POI. For example, a user may input a destination location (e.g., an address or name of the POI). In response, the navigation application 134 may provide navigation information that enables the user to navigate to the destination location using locally stored map data for a particular geographic area and / or map data provided via the server computing system 300. For example, the navigation information may include turn-by-turn directions from a current location (or a provided origin or starting location) to the destination location. For example, the navigation information may include a travel time (e.g., an estimated or predicted travel time) from a current location (or a provided origin or starting location) to the destination location.

[0076] The navigation application 134 may provide a visual representation of a geographic area via the display device 160 of the computing device 100. The visual representation of the geographic area may include highlighting of one or more streets, one or more points of interest (including buildings, landmarks, etc.), and a planned route. In some examples, the navigation application 134 may also provide a location-based search option to identify one or more searchable points of interest within a given geographic area. In some examples, the navigation application 134 may include a local copy of relevant map data. In other examples, the navigation application 134 may access information at a server computing system 300, which may be remotely located, to provide the requested navigation services.

[0077] In some examples, navigation application 134 may be a specialized application designed specifically to provide navigation services, while in other examples, navigation application 134 may be a general application (e.g., a web browser) that may provide access to a variety of different services, including navigation services, over network 400.

[0078] In some exemplary embodiments, computing device 100 includes location determination device 140. Location determination device 140 can determine a current geographic location of computing device 100 and communicate such geographic location to server computing system 300 via network 400. Location determination device 140 can be any device or circuitry for analyzing the location of computing device 100. For example, location determination device 140 can determine an actual or relative location by using a satellite navigation positioning system (e.g., GPS system, Galileo positioning system, Global Navigation Satellite System (GLONASS), BeiDou satellite navigation and positioning system), an inertial navigation system, a dead reckoning system, based on an IP address, by using triangulation, and / or proximity to cellular telephone towers or Wi-Fi hotspots, and / or other suitable techniques for determining the location of computing device 100.

[0079] The computing device 100 may include an input device 150 configured to receive input from a user, and may include, for example, one or more of a keyboard (e.g., a physical keyboard, a virtual keyboard, etc.), a mouse, a joystick, a button, a switch, an electronic pen or stylus, a gesture recognition sensor (e.g., for recognizing a user's gestures including movements of a body part), an input sound device or a voice recognition sensor (e.g., a microphone for receiving voice input such as a voice command or a voice query), an output sound device (e.g., a speaker), a trackball, a remote control, a portable (e.g., cellular or smart) phone, a tablet PC, a pedal or foot switch, a virtual reality device, etc. The input device 150 may further include a haptic device for providing haptic feedback to the user. The input device 150 may be embodied by, for example, a touch-sensitive display having a touch screen capability. For example, the input device 150 may be configured to receive input from a user associated with the input device 150.

[0080] Computing device 100 may include a display device 160 that displays information viewable by a user (e.g., a map, an immersive view of locations, a user interface screen, etc.). For example, display device 160 may be a non-touch-sensitive display or a touch-sensitive display. Display device 160 may include, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, an active matrix organic light-emitting diode (AMOLED), a flexible display, a 3D display, a plasma display panel (PDP), a cathode ray tube (CRT) display, etc. However, the present disclosure is not limited to these exemplary displays and may include other types of displays. Display device 160 may be used by navigation and mapping system 130 installed on computing device 100 to display information related to an input to a user (e.g., information about locations of interest to the user, a user interface screen having user interface elements selectable by the user, etc.). The navigation information may include one or more of, but is not limited to, a map of the geographic region, an immersive view of the location (e.g., a three-dimensional immersive view, a fly-through immersive view of the location, etc.), a location of the computing device 100 within the geographic region, a route through the geographic region specified on the map, one or more navigation directions (e.g., turn-by-turn directions through the geographic region), a travel time for the route through the geographic region (e.g., from the location of the computing device 100 to a POI), and one or more points of interest within the geographic region.

[0081] Computing device 100 may include output device(s) 170 for providing output to a user, such as one or more of an audio device (e.g., one or more speakers), a haptic device (e.g., a vibration device) for providing haptic feedback to the user, a light source (e.g., one or more light sources such as LEDs that provide visual feedback to the user), a thermal feedback system, etc. According to various examples of the present disclosure, output device 170 may include a speaker that outputs a sound associated with a location in response to a user requesting an immersive view of the location.

[0082] According to various examples of the present disclosure, the computing device 100 may include a capture device 180 capable of capturing media content. For example, the capture device 180 may include an image capturer 182 (e.g., a camera) configured to capture an image (e.g., a photo, a video, etc.) of a location. For example, the capture device 180 may include a sound capturer 184 (e.g., a microphone) configured to capture sound or audio (e.g., an audio recording) of a location. The media content captured by the capture device 180 may be transmitted, for example, via the network 400 to the server computing system 300, the user generated content data store 350, the machine generated content data store 360, the POI data store 370, the navigation data store 380, and the user data store 390. For example, in some implementations, the images may be used to generate a 3D scene, and in some implementations, the media content may be integrated with an existing 3D scene.

[0083] According to the example embodiments described herein, the server computing system 300 may include one or more processors 310 and one or more memory devices 320 as described above. The server computing system 300 may also include a navigation and mapping system 330 and a machine-generated content generator 340.

[0084] For example, the navigation and mapping system 330 may include an immersive view application 332 that performs functions similar to those described above with respect to the immersive view application 132 .

[0085] For example, the navigation and mapping system 330 may include a 3D scene generator 336 configured to generate a 3D scene based on multiple images of a location (e.g., an image of the interior of a restaurant, an image of a park, etc.). The multiple images may be captured and combined using known methods to create a 3D scene of the location. For example, images that overlap each other may be stitched together and refined using methods including bundle adjustment to create a 3D model of the scene. In some implementations, methods including structure from motion algorithms may be used to estimate the three-dimensional structure. In some implementations, machine learning resources may be implemented to generate camera-like images from any viewpoint within the location based on the captured images. For example, a video fly-through of the location may be generated by the 3D scene generator 336 based on the captured images. In some implementations, the initial 3D scene generated by the 3D scene generator 336 may be a static 3D scene lacking variable or dynamic (e.g., moving) objects. For example, the initial 3D scene of a park may include an image of the park that includes images of trees, playground equipment, picnic tables, etc., without images of humans, dogs, or other moving objects.

[0086] For example, the navigation and mapping system 330 may include a 3D scene integrator 338 configured to integrate user-generated content from the user-generated content data store 350 and / or machine-generated content from the machine-generated content data store 360 ​​with a 3D scene obtained from the 3D scene image 382. For example, the integrated 3D scene image 384 may include 3D scenes of various locations with which media content is integrated. The 3D scenes stored in the 3D scene image 382 may also be categorized or classified according to time of day, season, weather conditions, lighting conditions, etc. The 3D scene generated based on multiple images of a location may be integrated with the media content using known methods to create an integrated 3D scene image 384 of the location. For example, the 3D scene integrator 338 may be configured to identify and extract one or more objects (e.g., one or more dynamic objects) from the images of the scene. The images may be user-generated images or machine-generated images. For example, the 3D scene integrator 338 may be configured to place or install the extracted one or more objects in the 3D scene associated with the extracted one or more objects. For example, the 3D scene integrator 338 may select a 3D scene from the 3D scene images 382 that correspond to an image of a scene having one or more objects. For example, the 3D scene integrator 338 may select a 3D scene from the 3D scene images 382 that have the highest similarity (e.g., with respect to time of day, season, weather conditions, lighting conditions, etc.) to the image of the scene having one or more objects. For example, a user-generated image taken at noon in a park in sunny conditions may include several children playing on play equipment. The 3D scene integrator 338 may be configured to extract the children from the image using various techniques (e.g., image segmentation algorithms, machine learning resources, cropping tools, etc.). The 3D scene integrator 338 may be configured to select a 3D scene from the 3D scene images 382 that have similar characteristics to the image (e.g., similar time of day, season, sunny weather conditions, etc.).The 3D scene integrator 338 may be configured to place the extracted image of the child within the 3D scene to generate an updated or integrated 3D scene in which the child is placed within the scene (e.g., on a slide, on a see-saw, etc.) so that a user viewing the integrated 3D scene can be shown how the park generally feels at that time of day, e.g., in similar weather conditions, along with an accurate representation of the park's conditions at that time of day. In some implementations, audio content may also be integrated with the 3D scene. For example, the 3D scene integrator 338 may be configured to integrate sound with the 3D scene in a manner similar to integrating images. For example, audio recorded at the park at noon may include children's laughter, while audio recorded at the park at night may include wildlife sounds. The 3D scene integrator 338 may be configured to integrate the laughter with the 3D scene associated with a similar time frame of audio (e.g., using actual recordings taking into account privacy considerations or user permission, or machine-generated audio including synthesized laughter). For example, an integrated 3D scene in which children are placed in the scene (e.g., on a slide, on a see-saw, etc.) and has audio content played while the user views the immersive view may not only provide the user viewing the integrated 3D scene with an accurate representation of the conditions in the park at that time of day, but also give them a more accurate idea of ​​what the park may feel like at that time of day under similar weather and noise conditions, for example.

[0087] For example, the navigation and mapping system 330 may include a machine-generated content generator 340 having an image synthesizer 342 and an audio synthesizer 344. For example, the machine-generated content stored in the machine-generated content data store 360 ​​may be generated by the machine-generated content generator 340 based on sensor data acquired by one or more sensors (forming part of the external content 500) located at a location.

[0088] For example, the image synthesizer 342 may generate a graphical representation of dynamic objects provided in user-generated images stored in the user-generated content data store 350. For example, the image synthesizer 342 may be configured to convert user-generated media content into a common format (e.g., by converting a real-world image of a person located at a location into a two-dimensional or three-dimensional digital avatar representing the person) to anonymize the media content. For example, sensor data acquired by one or more sensors may indicate how many people are at the location (e.g., based on the number of smartphones or other computing devices detected at the location). For example, the sensor data acquired by one or more sensors may indicate various characteristics about the people at the location (e.g., clothing, facial expressions, etc. based on images captured by a camera). For example, the image synthesizer 342 may generate a graphical representation of people at the location according to the number of people at the location and according to characteristics about the people at the location to accurately represent the location and portray an impression of the location. For example, an image of a crowd at a stadium may include various people wearing jerseys associated with the home team. The image synthesizer 342 may generate a graphical representation of people in the stadium wearing similar jerseys (as opposed to formal attire) according to the number of people in the stadium to accurately represent the location and portray an impression of the location.

[0089] For example, the audio synthesizer 344 may generate audio content based on portions of recorded user-generated audio content (e.g., granular information) or other available sounds to create new audio content that represents the state of the location at a particular time, as well as the mood, atmosphere, impression, or feeling of the location at a particular time (e.g., time of day, day of the week, time of year, etc.).

[0090] Examples of the present disclosure are also directed to computer-implemented methods for integrating media content with three-dimensional scenes. Figure 3 illustrates a flow diagram of an exemplary, non-limiting computer-implemented method according to one or more exemplary embodiments of the present disclosure.

[0091] 3, in an exemplary computer-implemented method 3000, at operation 3100, the method includes a server computing system 300 receiving a request for an immersive view of a location. For example, the request may be sent from a computing device 100. For example, the request for an immersive view may be associated with temporal conditions (e.g., an immersive view of a location at a particular time, including time of day, season, etc.) and / or other conditions including lighting conditions, weather conditions, etc. For example, a user of a computing device 100 may request an immersive view of a restaurant at 6 p.m. on Friday via an immersive view application 132, and the request may be sent to the server computing system 300.

[0092] In operation 3200, the server computing system 300 may obtain a 3D scene associated with the location. For example, the server computing system 300 may obtain a 3D scene from the 3D scene image 382. For example, the 3D scene of the location obtained from the 3D scene image 382 may be a 3D scene that corresponds to or is approximately associated with the conditions of the request (e.g., a 3D scene of a restaurant at 6 p.m. on a Friday, a 3D scene of a restaurant under similar lighting conditions expected at night or a similar time of day, etc.). As another example, if a user requests an immersive view of a park when it is raining, the 3D scene of the park retrieved from the 3D scene image 382 may be a 3D scene of the park when it is raining, rather than a 3D scene of the park when it is sunny.

[0093] In operation 3300, the server computing system 300 may retrieve media content associated with the location. For example, the server computing system 300 may retrieve media content from the user-generated content data store 350 and / or the machine-generated content data store 360. For example, the server computing system 300 may retrieve media content from the user-generated content data store 350 and / or the machine-generated content data store 360 ​​that corresponds to or is generally associated with the conditions of the request (e.g., media content captured in a restaurant at or around 6 p.m. on a Friday, media content captured in a restaurant under similar lighting conditions expected at night or a similar time of day, etc.). As another example, if a user requests an immersive view of a park when it is raining, the retrieved media content may include an image of the park when it is raining. Similarly, the retrieved media content may include audio content associated with the park captured when it is raining, or synthesized audio content reflecting a situation associated with rain in the park.

[0094] In operation 3400, the server computing system 300 may extract dynamic objects from the visual content (i.e., images). For example, the 3D scene integrator 338 may be configured to identify and extract one or more objects (e.g., one or more dynamic objects) from the image of the scene at the location. Extraction of the objects from the captured image may be performed using various techniques (e.g., image segmentation algorithms, machine learning resources, cropping tools, etc.). The extracted objects may include foreground objects identified as variable or dynamic objects, such objects providing an accurate representation of the state of the location at a particular time or under a particular circumstance. Furthermore, such objects may indicate or represent an impression or feeling of the location at a particular time or under a particular circumstance.

[0095] In operation 3500, the server computing system 300 may implement machine learning resources to integrate an object (e.g., a dynamic object) with the 3D scene of its location according to information (e.g., time information) associated with the object. For example, the 3D scene integrator 338 may be configured to integrate user-generated content from the user-generated content data store 350 and / or machine-generated content from the machine-generated content data store 360 ​​with the 3D scene obtained from the 3D scene image 382 in operation 3200. For example, the 3D scene integrator 338 may be configured to position extracted objects in the 3D scene to generate an updated or integrated 3D scene in which the objects are placed in the scene in a manner that is consistent with how those objects are positioned or placed in the image of the scene created by the user. For machine-generated objects, the 3D scene integrator 338 may be configured to place the machine-generated objects in the 3D scene in a manner consistent with how the corresponding real-world object would or would most likely be placed in the scene (e.g., via a neural network) in order to generate an updated or integrated 3D scene in which the machine-generated object is placed in the scene. The integration of the object is intended to provide a user viewing the integrated 3D scene with an accurate representation of the state of the location at a particular time or under other specified conditions. Additionally, the integration of the object may provide a sense of how the location generally feels at a particular time (e.g., time of day, time of year, etc.) or under other specified conditions (e.g., particular weather conditions, lighting conditions, etc.).

[0096] In operation 3600, the server computing system 300 may also integrate audio content with the 3D scene of the location according to information associated with the scene (e.g., time information). For example, the 3D scene integrator 338 may be configured to integrate sound with the 3D scene in a manner similar to integrating an image. For example, the 3D scene integrator 338 may be configured to integrate user-generated content from the user-generated content data store 350 and / or machine-generated content from the machine-generated content data store 360 ​​with the 3D scene obtained from the 3D scene image 382 in operation 3200. For example, the 3D scene integrator 338 may be configured to integrate audio content obtained in operation 3300 that matches the request received in operation 3100 (e.g., audio content recorded in a similar time frame to which the immersive view request is associated). The integration of audio content is intended to enable a user viewing the integrated 3D scene to be provided with a more accurate representation of the state of the location at a particular time or under other specified conditions. Additionally, the integration of audio content may provide an additional sense of what the location generally feels or sounds like at a particular time (e.g., time of day, season, etc.) or under other particular conditions (e.g., particular weather conditions, etc.).

[0097] At operation 3700, the server computing system 300 may be configured to provide a location update or an integrated 3D scene with integrated audio content and / or objects (e.g., dynamic objects) that fulfills the request received at operation 3100. For example, the location update or the integrated 3D scene may be transmitted from the server computing system 300 to the computing device 100. For example, the location update or the integrated 3D scene may be stored as an integrated 3D scene image 384.

[0098] In the example of FIG. 3, the server computing system 300 may dynamically generate an update or integrated 3D scene for the location in response to receiving the request in operation 3100. However, in some implementations, the update or integrated 3D scene for the location that satisfies the request received in operation 3100 may be pre-stored or may be pre-existing and stored as the integrated 3D scene image 384. Thus, in such cases, operations 3200, 3300, 3400, 3500, and 3600 may be omitted, and the operation of searching for an integrated 3D scene for the location in the integrated 3D scene image 384 that satisfies the conditions of the request may be performed as an intermediate operation between operations 3100 and 3700. Thus, the responsiveness of the server computing system 300 to the request may be faster as fewer operations are performed or required.

[0099] Examples of the present disclosure are also directed to user-facing aspects where a user can request an immersive view of a location. For example, Figures 4-8B show example user interface screens associated with obtaining an immersive view of a location in which media content is integrated with a three-dimensional scene, according to one or more example embodiments of the present disclosure.

[0100] 4 illustrates a user interface screen of a map application according to one or more exemplary embodiments of the present disclosure. In FIG. 4, user interface screen 4000 indicates that a user of computing device 100 is exploring a location in Westminster, specifically a building that includes Cinnamon Club 4100, and icon 4200 indicates that Cinnamon Club 4100 includes a restaurant. For example, user interface element 4300 may enable a user to obtain an immersive view of the location. For example, user interface element 4300 may be in the form of a symbol or selectable object that is overlaid on the location to indicate to the user that an immersive view of the location can be obtained. For example, in FIG. 4, user interface element 4300 is a white circle.

[0101] For example, Figures 5A and 5B show examples of immersive views from a video fly-through of a location, according to one or more exemplary embodiments of the present disclosure. For example, Figure 5A shows a first user interface screen 5000 showing a first view 5100 inside the Cinnamon Club, while a second user interface screen 5000' shows a second view 5100' inside the Cinnamon Club. The interior view of the Cinnamon Club is an example of a 3D scene of the location that can be stored as a 3D scene image 382. For example, the 3D scene of the location may be missing dynamic objects (e.g., customers, staff, etc.). Thus, displaying the 3D scene of the location alone or in combination as a fly-through video may not be an accurate representation of the state of the location. Furthermore, this may not provide a good impression of the restaurant or how it feels to be in the restaurant at any particular time.

[0102] For example, FIG. 6 illustrates an exemplary immersive view of a location including an integrated 3D scene of the location, according to one or more exemplary embodiments of the present disclosure. For example, FIG. 6 reflects the second view 5100′ of the Cinnamon Club interior of FIG. 5B, with dynamic objects and audio content integrated into the 3D scene of the location. For example, user interface screen 6000 illustrates a 3D scene 6100 of the location including integrated dynamic objects including a first dynamic object 6200 and a second dynamic object 6300, and integrated audio content 6400. For example, the first dynamic object 6200 may be a machine-generated graphical representation of a waiter (e.g., a 3D digital avatar representing the waiter), and the second dynamic object 6300 may be a machine-generated graphical representation of a customer (e.g., a 3D digital avatar representing the customer). In some implementations, the number of dynamic objects may reflect the average number of people in the restaurant at a particular time, day of the week, time of year, etc., which may be determined from statistical analysis of sensor data obtained from various sensors provided at the location. In some implementations, the dynamic object may be a graphical representation of a real-world object extracted from user-generated imagery that meets criteria associated with the request for an immersive view of a location (e.g., captured at a time and / or under some other conditions corresponding to the time and / or other conditions associated with the request for an immersive view of a location). In some implementations, the dynamic object may be a real-world object (which may be anonymized, if desired) extracted from user-generated imagery that meets criteria associated with the request for an immersive view of a location (e.g., captured at a time and / or under some other conditions corresponding to the time and / or other conditions associated with the request for an immersive view of a location).

[0103] In some implementations, according to one or more exemplary embodiments of the present disclosure, the immersive view may include a still image of the location including an integrated 3D scene of the location. In some implementations, according to one or more exemplary embodiments of the present disclosure, the immersive view may include multiple images of the location including an integrated 3D scene of the location. In some implementations, according to one or more exemplary embodiments of the present disclosure, the immersive view may be formed by combining multiple images of the location including an integrated 3D scene of the location to form a moving image (e.g., a fly-through of the location). In some implementations, the immersive view formed by the moving image of the integrated 3D scene of the location may include moving dynamic objects, which further bring the integrated 3D scene of the location to life and accurately represent the state of the location at a particular time and / or under particular conditions.

[0104] Referring to FIG. 6, the immersive view of the integrated 3D scene at the location may include a still image including the first dynamic object 6200 and the second dynamic object 6300 integrated with the 3D scene at the location, or an immersive view of the integrated 3D scene at the location including a plurality of images having the first dynamic object 6200 and the second dynamic object 6300 integrated with the 3D scene at the location, or an immersive view of the integrated 3D scene at the location including a plurality of images having the first dynamic object 6200 and the second dynamic object 6300 integrated with the 3D scene at the location, where the plurality of images are combined together to form a moving image.

[0105] In some implementations, an immersive view including multiple images may include a moving dynamic object (e.g., to provide dynamic movement of the dynamic object). For example, in FIG. 6, a first image including a first dynamic object 6200 and a second dynamic object 6300 may show the first dynamic object 6200 toward the back of the room, and a second image including the first dynamic object 6200 and the second dynamic object 6300 may show the first dynamic object 6200 located near the second dynamic object 6300 (toward the front of the room). An immersive view of an integrated 3D scene of a location (e.g., a restaurant) including multiple images with the first dynamic object 6200 and the second dynamic object 6300 integrated with the 3D scene of the location may be formed such that the multiple images are combined to form a moving image that appears to show the first dynamic object 6200 moving dynamically from one part of the location to another part of the location (e.g., from the back of the room to the front of the room). The immersive view created by the moving image (ie, video) may be created from multiple images captured by a user, multiple machine-generated images, or a combination thereof.

[0106] In some implementations, an image with the dynamic object may be generated, and the location of the dynamic object may be extrapolated from other images including the dynamic object to provide a smoother, more realistic depiction of the dynamic object moving within the scene (e.g., a movie, a fly-through, etc.). For example, a third image including the first dynamic object 6200 and the second dynamic object 6300 at the location may be generated (e.g., via extrapolation based on the first image and the second image, time information associated with the first image and the second image, etc.) to show the first dynamic object 6200 in the center of the room. For example, an immersive view of an integrated 3D scene of the location (e.g., a restaurant) including multiple images with the first dynamic object 6200 and the second dynamic object 6300 integrated with the 3D scene of the location may be formed such that the multiple images are combined to form a movie, the movie appearing to show the first dynamic object 6200 moving dynamically from one portion of the location to another portion of the location (e.g., from the back of the room to the center of the room and then to the front of the room). Additional images may be generated as needed to bring the integrated 3D scene of a location to life and provide smooth movement of dynamic objects (e.g., 24 frames per second, 30 frames per second, 60 frames per second, etc.) within an immersive view of the integrated 3D scene of the location to accurately represent the state of the location at a particular time and / or under particular conditions.

[0107] For example, the integrated audio content 6400 may be an audio recording that can be played via the output device 170 when an immersive view of the location is provided to the computing device 100. For example, the integrated audio content 6400 may be machine-generated or an actual recording. For example, the integrated audio content 6400 may provide an accurate representation of the state of the location at a particular time and / or under particular conditions. Additionally, the integrated audio content 6400 may indicate a mood, atmosphere, or impression of the location at a particular time and / or under particular conditions (e.g., weather conditions). For example, the selected audio content may meet criteria associated with a request for an immersive view (e.g., captured at a time and / or under some other conditions corresponding to the time and / or other conditions associated with the request for an immersive view of the location).

[0108] 7 illustrates an exemplary computing device 7000 (corresponding to computing device 100) having a user interface screen through which a user can upload user-generated content, according to one or more exemplary embodiments of the present disclosure. For example, FIG. 7 illustrates a user interface screen 7100 including various user interface elements, including a user interface element 7200 that enables a user to add photos to one or more of the server computing system 300, the user-generated content data store 350, the machine-generated content data store 360, the POI data store 370, the navigation data store 380, and the user data store 390, e.g., via the immersive view application 132 and the network 400. For example, a user interface element 7300 may be selectable by a user to obtain directions to a location, e.g., via the navigation application 134.

[0109] In some implementations, when the user interface element 7200 is selected, the image capturer 182 is launched for the user to capture an image of the location. In some implementations, when the user interface element 7200 is selected, the user can select a photo that is stored locally on the computing device 7000 or that is stored remotely, for uploading, for example, via the immersive view application 132 and the network 400, to one or more of the server computing system 300, the user generated content data store 350, the machine generated content data store 360, the POI data store 370, the navigation data store 380, and the user data store 390.

[0110] Although not shown in FIG. 7, other user interface elements may be provided. For example, a user interface element for adding audio content may also be provided. In some implementations, when a user interface element for adding audio content is selected, a sound capturer 184 is launched to allow a user to capture a sound at a location. In some implementations, when a user interface element for adding audio content is selected, a user may select a recording stored locally on the computing device 7000 or stored remotely for uploading, for example, via the immersive view application 132 and the network 400, to one or more of the server computing system 300, the user generated content data store 350, the machine generated content data store 360, the POI data store 370, the navigation data store 380, and the user data store 390.

[0111] Media content uploaded to the server computing system 300, the user generated content data store 350, the machine generated content data store 360, the POI data store 370, the navigation data store 380, and the user data store 390 may include a variety of information about the media content.

[0112] For example, media content captured by a camera (e.g., image capturer 182) may include an image of the location and various information (e.g., metadata, semantic data, etc.) that helps to integrate the image (or a portion of the image) into a 3D scene of the location associated with the image. For example, the image may include information including the date the image was captured, the time the image was captured, location information indicating the location where the image was taken (e.g., GPS location), etc. For example, descriptive metadata may be provided with the image and may include keywords associated with the image, a title or name of the image, environmental information at the time the image was captured (e.g., lighting conditions including brightness levels, noise conditions including decibel levels, weather information including weather conditions such as temperature, wind, precipitation, cloudiness, humidity, etc.), etc. Environmental information may be obtained from sensors in the computing device 7000 used to capture the image or from another computing device.

[0113] For example, media content captured by a microphone (e.g., sound capturer 184) may include audio associated with a location. The audio content may include various information (e.g., metadata, semantic data, etc.) that is useful for integrating the audio content (or a portion of the audio) with a 3D scene of the location associated with the audio content. For example, the audio content may include information including the date the audio was captured, the time the audio was captured, the audio type, and location information indicating the location where the audio was taken (e.g., GPS location), etc. For example, descriptive metadata may be provided with the audio and may include keywords associated with the audio, a title or name of the audio, the musical genre of the audio, environmental information at the time the audio was captured (e.g., lighting conditions including brightness levels, noise conditions including decibel levels, weather information including weather conditions such as temperature, wind, precipitation, cloudiness, humidity, etc.), etc. The environmental information may be obtained from sensors in the computing device 7000 used to capture the audio or from another computing device.

[0114] 8A and 8B illustrate example user interface screens through which a user may request an immersive view of a location associated with a particular time, according to one or more example embodiments of the present disclosure. For example, FIG. 8A illustrates a computing device 8000 displaying a user interface screen 8100 including various user interface elements, including a user interface element 8200 corresponding to a time slider widget that allows a user to request an immersive view of a location at various times of a day. For example, FIG. 8B illustrates a computing device 8000 displaying a user interface screen 8100' including various user interface elements, including a user interface element 8200' corresponding to a time slider widget that allows a user to request an immersive view of a location at various times of a day.

[0115] For example, user interface screen 8100 displays an immersive view of Westminster at 1:04 PM, while user interface screen 8100' displays an immersive view of Westminster at 9:15 PM. For example, the immersive view shown in FIG. 8A reflects the exact lighting conditions for the time of day, and the dynamic objects may include a vehicle 8300 integrated with the 3D scene for the location. For example, the immersive view shown in FIG. 8B reflects the exact lighting conditions for the time of day, and the dynamic objects may include a vehicle 8300' with its headlights on and integrated with the 3D scene for the location.

[0116] Although not shown in FIGS. 8A-8B, other user interface elements may be provided that allow a user to specify a request to obtain an immersive view of a location according to various conditions in order to accurately obtain a representation of the state of the location according to those conditions. This may further allow a user to accurately obtain an impression of the location according to those conditions. For example, a user interface element may be provided to identify weather conditions (e.g., a sunset view, a sunrise view, under sunny, cloudy, or rainy conditions, etc.). For example, a user of computing device 100 may request an immersive view of a park in the evening when it is raining. A user interface element for specifying the weather conditions (or any condition associated with the requested immersive view) may be in the form of a pull-down menu, a selectable user interface element, a text box, etc. For example, a user interface element may be provided to specify a crowding state (e.g., uncrowded, slightly crowded, crowded, very crowded, etc.). For example, a user of computing device 100 may request an immersive view of a park when the park is considered very crowded so that the user can properly understand the impression of the park when very crowded without actually going to the park. The integrated 3D scene of the park may include images of the park with a high level of visitors and / or audio content reflecting a very noisy environment showing a very crowded park. For example, a user interface element may be provided for specifying lighting conditions (e.g., normal ambient light, bright, dark, etc.). For example, a user of the computing device 100 may request an immersive view of a park that is considered very bright so that the user can understand the atmosphere of the park when it is very bright without actually going to the park. The integrated 3D scene of the park may include images of the park with integrated images showing the park under bright conditions. The user may specify the time of day in addition to or instead of the brightness level when requesting the immersive view.

[0117] Examples of the present disclosure are also directed to computer-implemented methods for integrating media content with three-dimensional scenes. Figure 9 illustrates a flow diagram of an exemplary, non-limiting computer-implemented method according to one or more exemplary embodiments of the present disclosure.

[0118] 9, in an example computer-implemented method 9000, at operation 9100, the method includes computer device 100 receiving (e.g., via input device 150) a request for a first input for a first immersive view of a location representing a first state of the location at a first time. For example, the first input may be provided via a selection of a user interface element provided on a user interface screen (e.g., user interface element 8200 as shown in FIG. 8A). For example, the first input may indicate a request for a first immersive view of the location representing what the location feels like at a particular time (e.g., what Westminster feels like at about 1:00 p.m. in April).

[0119] At operation 9200, the method includes computing device 100 providing, for display on display device 160, a first immersive view of a location including a 3D scene of the location generated based on the plurality of images and first media content integrated with the 3D scene of the location. For example, the first media content represents a first state of the location at a first time. For example, in FIG. 8A, computing device 100 provides a first immersive view of Westminster where the first media content may include dynamic objects including vehicle 8300.

[0120] At operation 9300, the method includes computing device 100 receiving (e.g., via input device 150) a request for a second input for a second immersive view of the location representing a second state of the location at a second time. For example, the second input may be provided via a selection of a user interface element provided on a user interface screen (e.g., user interface element 8200' as shown in FIG. 8B). For example, the second input may indicate a request for a second immersive view of the location that accurately represents the state of the location at another particular point in time. Furthermore, such a second immersive view may represent what the location feels like at another particular time (e.g., what Westminster feels like at about 9 p.m. in April).

[0121] At operation 9400, the method includes computing device 100 providing, for display on display device 160, a second immersive view of the location including a 3D scene of the location generated based on the plurality of images and second media content integrated with the 3D scene of the location. For example, the second media content represents a second state of the location at a second time. For example, in FIG. 8B, computing device 100 provides a second immersive view of Westminster where the second media content may include dynamic objects including a vehicle 8300′ with its headlights on.

[0122] As previously described herein, the first media content may include user-generated media content captured by one or more users, machine-generated media content, or a combination thereof. The user-generated media content may include user-generated visual content, user-generated audio content, or a combination thereof. The machine-generated media content may include machine-generated visual content, machine-generated audio content, or a combination thereof.

[0123] In some implementations, computing device 100 may transmit the user-generated media content to server computing system 300 or another database (e.g., user-generated content data store 350, machine-generated content data store 360, POI data store 370, navigation data store 380, and user data store 390). Updating the data stores with additional media content may result in future virtual renderings of locations associated with the user-generated content being updated and may include user-generated media content added by a user via computing device 100.

[0124] For example, a method implemented by computing device 100 may include receiving a third input via input device 150 and providing user-generated media content captured by a user of computing device 100, the user-generated media content being temporally associated with a first time (e.g., an image of Westminster captured at or around 1 PM in April, etc.). The method may further include transmitting the user-generated media content to server computing system 300, or another computing device including databases (e.g., user-generated content data store 350, machine-generated content data store 360, POI data store 370, navigation data store 380, and user data store 390) configured to provide a first immersive view of the location representing a first state of the location at the first time. The method may further include receiving a fourth input via input device 150 requesting a first immersive view of the location representing a first state of the location at a first time, and providing, for display on display device 160, the first immersive view of the location including a 3D scene of the location generated based on the plurality of images and updated first media content integrated with the 3D scene of the location, the updated first media content representing the first state of the location at the first time and including at least a portion of user-generated media content captured by a user of computing device 100. That is, the integrated 3D scene associated with the location at a particular time may be updated based on subsequent media content captured by computing device 100 associated with the location at a particular time. Thus, an accurate representation of the state of the location may continue to update over time based on recent experiences and media content captured by a user at the location.Additionally, the impression or feeling of a location can continue to update over time based on recent experiences and media content captured by the user at that location.

[0125] To the extent that general terms including "module" and "unit" are used herein, these terms may refer to, but are not limited to, software or hardware components or devices, such as field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs) that perform specific tasks. A module or unit may be configured to reside on an addressable storage medium and configured to execute on one or more processors. Thus, a module or unit may include, by way of example, components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, variables, such as software components, object-oriented software components, class components, and task components. The functionality provided for components and modules / units may be combined into fewer components and modules / units or may be further separated into additional components and modules.

[0126] Aspects of the exemplary embodiments described above may be recorded on a non-transitory computer-readable medium that includes program instructions for implementing various computer-implemented operations. The medium may also include, alone or in combination with program instructions, data files, data structures, and the like. Examples of non-transitory computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD ROM disks, Blue-Ray disks, and DVDs, magneto-optical media such as optical disks, and other hardware devices that are specifically configured to store and execute program instructions, such as semiconductor memory, read-only memory (ROM), random access memory (RAM), flash memory, USB memory, and the like. Examples of program instructions include both machine code, such as generated by a compiler, and files containing high-level code that may be executed by a computer using an interpreter. The program instructions may be executed by one or more processors. The hardware devices described above may be configured to function as one or more software modules to execute the operations of the embodiments described above, or vice versa. Furthermore, the non-transitory computer-readable storage medium may be distributed among computer systems connected via a network, and the computer-readable code or program instructions may be stored and executed in a distributed manner. Furthermore, the non-transitory computer-readable storage medium may also be embodied in at least one application specific integrated circuit (ASIC) or field programmable gate array (FPGA).

[0127] Each block in the flowchart diagrams may represent a unit, module, segment, or portion of code comprising one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions shown in the blocks may not be performed in order. For example, two blocks shown in succession may in fact be performed substantially in parallel (concurrently) depending on the functionality involved, or the blocks may be performed in the reverse order depending on the functionality involved.

[0128] Although the present disclosure has been described with respect to various exemplary embodiments, each example is provided for the purpose of illustration and not for the purpose of limiting the present disclosure. Those skilled in the art, upon understanding the foregoing, can easily create modifications, variations, and equivalents to such embodiments. Thus, the present disclosure is not intended to include such modifications, variations, and / or additions to the disclosed subject matter as would be readily apparent to one skilled in the art. For example, features illustrated or described as part of one embodiment can be used with another embodiment to obtain yet another embodiment. Thus, the present disclosure is intended to cover such modifications, variations, and equivalents. [Explanation of symbols]

[0129] 100 computing devices 110 Processor 120 Memory Devices 122 Data 124 Command 130 Navigation and Mapping Systems 132 Immersive View Applications 134 Navigation Applications 140 Location Determination Device 150 Input Devices 160 Display Devices 170 Output Devices 180 Capture Device 182 Image Capturer 184 Sound Capturer 200 External Computing Devices 300 Server Computing System 310 Processor 320 Memory Devices 330 Navigation and Mapping System 332 Immersive View Application 336 3D Scene Generator 338 3D Scene Integrator 340 Machine-Generated Content Generators 342 Image synthesis device 344 Audio Synthesiser 350 User Generated Content Data Store 360 Machine Generated Content Data Store 370 POI data store 380 Navigation Data Store 382 3D scene images 384 Integrated 3D Scene Images 390 User Data Store 400 Network 500 External Content 4000 User Interface Screen 4100 Cinnamon Club 4200 Icons 4300 User Interface Elements 5000 First User Interface Screen 5000' Second User Interface Screen 5100 1st View 5100' Second View 6000 User Interface Screens 6100 3D Scenes 6200 First Dynamic Object 6300 Second Dynamic Object 6400 Integrated Audio Content 7000 Computing Devices 7100 User Interface Screen 7200 User Interface Elements 7300 User Interface Elements 8000 Computing Devices 8100 User Interface Screen 8100' User Interface Screen 8200 User Interface Elements 8200' User Interface Elements 8300 vehicles 8300' Vehicle

Claims

1. Obtaining a three-dimensional (3D) scene of the location generated based on the plurality of images; receiving media content temporally associated with the location; integrating at least a portion of the media content with the 3D scene of the location to obtain an integrated 3D scene of the location; providing the integrated 3D scene of the location to represent a state of the location based on a temporal association of the media content with the location; and A method comprising:

2. The method of claim 1 , wherein the media content comprises user-generated media content captured by one or more users.

3. The method of claim 2 , wherein the user-generated media content comprises at least one of user-generated visual content or user-generated audio content.

4. the user-generated media content includes an image of the location including one or more real-world dynamic objects; The method of claim 2 or 3, wherein integrating at least the portion of the media content with the 3D scene of the location includes extracting the one or more real-world dynamic objects from the image of the location and determining one or more locations within the 3D scene for placing the one or more real-world dynamic objects.

5. the images of the location including the one or more real-world dynamic objects include a first image of the location including a first portion of the one or more real-world dynamic objects captured at a first time by the one or more users at the location, and a second image of the location including a second portion of the one or more real-world dynamic objects captured at a second time by the one or more users at the location; Providing the integrated 3D scene of the location, providing a first 3D scene of the location associated with the first time and having the first portion of the one or more real-world dynamic objects integrated with the first 3D scene of the location; providing a second 3D scene of the location associated with the second time and having the second portion of the one or more real-world dynamic objects integrated with the second 3D scene of the location; The method of claim 4 comprising:

6. the user-generated media content includes first audio captured by the one or more users at the location at a first time and second audio captured by the one or more users at the location at a second time; Providing the integrated 3D scene of the location providing a first 3D scene of the location having the first audio associated with the first time and integrated with the first 3D scene of the location; providing a second 3D scene of the location having the second audio associated with the second time and integrated with the second 3D scene of the location; 6. The method according to claim 2, comprising:

7. The method of claim 1 , wherein the media content comprises machine-generated media content.

8. The method of claim 7 , wherein the machine-generated media content comprises at least one of machine-generated visual content or machine-generated audio content.

9. The method of claim 7 or 8, wherein the machine-generated media content is synthesized based on user-generated media content captured by one or more users.

10. the user-generated media content includes an image of the location including one or more real-world dynamic objects, the method comprising: extracting the one or more real-world dynamic objects from the image; and converting the one or more real-world dynamic objects into a corresponding graphical representation of the one or more real-world dynamic objects; Further equipped with 10. The method of claim 9, wherein integrating at least the portion of the media content with the 3D scene of the location includes determining one or more locations within the 3D scene for placing the corresponding graphical representations of the one or more real-world dynamic objects.

11. the one or more real-world dynamic objects include one or more humans; The method of claim 10 , wherein the corresponding graphical representation of the one or more real-world dynamic objects comprises a corresponding 3D digital avatar of the one or more humans.

12. the images of the location including the one or more real-world dynamic objects include a first image of the location including a first portion of the one or more real-world dynamic objects captured at a first time by the one or more users at the location, and a second image of the location including a second portion of the one or more real-world dynamic objects captured at a second time by the one or more users at the location; Providing the integrated 3D scene of the location, providing a first 3D scene of the location having a first graphical representation corresponding to the first portion of the one or more real-world dynamic objects associated with the first time and integrated with the first 3D scene of the location; providing a second 3D scene of the location having a second graphical representation corresponding to the second portion of the one or more real-world dynamic objects associated with the second time and integrated with the second 3D scene of the location; The method according to claim 10 or 11, comprising:

13. the user-generated media content includes first audio captured by the one or more users at the location at a first time and second audio captured by the one or more users at the location at a second time, the method comprising: synthesizing a first composite audio recording based on the first audio and synthesizing a second composite audio recording based on the second audio; providing a 3D scene of the location having at least the portion of the media content integrated with the 3D scene of the location, providing a first 3D scene of the location having the first synthesized audio recording associated with the first time and integrated with the first 3D scene of the location; providing a second 3D scene of the location having the second synthesized audio recording associated with the second time and integrated with the second 3D scene of the location; 13. The method according to any one of claims 9 to 12, comprising:

14. 14. The method of claim 7, further comprising synthesizing the machine-generated media content based on sensor data acquired by one or more sensors disposed at the location.

15. synthesizing the machine-generated media content based on the sensor data acquired by the one or more sensors disposed at the locations; determining a number of people at the location at a first time based on the sensor data acquired by the one or more sensors; determining characteristics about the people present at the location at the first time based on the sensor data acquired by the one or more sensors; and generating a graphical representation of the people at the location according to the number of people at the location at the first time and according to the characteristics relating to the people to indicate a status of the location at the first time; Equipped with 15. The method of claim 14, wherein integrating at least the portion of the media content with the 3D scene of the location includes determining one or more locations within the 3D scene for placing the graphical representations of the people.

16. at least one memory for storing instructions; at least one processor configured to execute the instructions to perform the method according to any one of claims 1 to 15; A computing device comprising:

17. 16. A computer program comprising instructions which, when executed on at least one processor, cause said at least one processor to perform the method of any one of claims 1 to 15.

18. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 15.

19. An input device; A display device; at least one memory for storing instructions; at least one processor configured to execute the instructions to perform operations; and wherein the operation comprises: receiving a first input via the input device requesting a first immersive view of the location representing a first state of the location at a first time; providing a first immersive view of the location for display on the display device, the first immersive view of the location comprising: a three-dimensional (3D) scene of the location generated based on the plurality of images; and first media content integrated with the 3D scene of the location, the first media content representing the first state of the location at the first time; and receiving a second input via the input device requesting a second immersive view of the location representing a second state of the location at a second time; providing a second immersive view of the location for display on the display device, the second immersive view of the location comprising: the 3D scene of the location generated based on the plurality of images; second media content integrated with the 3D scene of the location, the second media content representing the second state of the location at the second time; and Including, A computing device comprising:

20. the first media content comprises at least one of user-generated media content captured by one or more users or machine-generated media content; the user-generated media content comprises at least one of user-generated visual content or user-generated audio content; 20. The computing device of claim 19, wherein the machine-generated media content comprises at least one of machine-generated visual content or machine-generated audio content.

21. The operation, receiving a third input via the input device to provide user-generated media content captured by a user of the computing device, the user-generated media content being temporally associated with the first time; and transmitting the user-generated media content to at least one of a server computing system or a database configured to provide the first immersive view of the location representative of the first state of the location at the first time; receiving a fourth input via the input device requesting the first immersive view of the location representing the first state of the location at the first time; providing a first immersive view of the location for display on the display device, the first immersive view of the location comprising: the 3D scene of the location generated based on the plurality of images; updated first media content integrated with the 3D scene of the location, the updated first media content representing the first state of the location at the first time and including at least a portion of the user-generated media content captured by the user of the computing device; and Including, 21. The computing device of claim 19 or 20, further comprising:

22. at least one memory for storing instructions; at least one processor configured to execute the instructions to perform operations; and wherein the operation comprises: Obtaining a three-dimensional (3D) scene of the location generated based on the plurality of images; receiving media content temporally associated with the location; integrating at least a portion of the media content with the 3D scene of the location to provide an integrated 3D scene of the location; providing the integrated 3D scene of the location to represent a state of the location based on the temporal association between the media content and the location; and A server computing system comprising:

23. the media content comprises at least one of user-generated media content captured by one or more users or machine-generated media content; the user-generated media content comprises at least one of user-generated visual content or user-generated audio content; 23. The server computing system of claim 22, wherein the machine-generated media content comprises at least one of machine-generated visual content or machine-generated audio content.

Images