Projecting existing user-generated content into immersive view

Abstract

To provide methods, systems, and devices for projecting a user-generated media content into three-dimensional immersive views.SOLUTION: A system can obtain a three-dimensional representation of a location generated based on a plurality of images. The system can access user-generated media content associated with the location. The system can receive path information representing a path through the three-dimensional representation of the location. The system can select one or more pieces of user-generated media content based on the path information. The system can integrate the one or more pieces of user-generated media content into the three-dimensional representation of the location based on the path information and a portion of the three-dimensional representation to be displayed to a user. The pieces of user-generated media content are presented within visual pop-outs in the three-dimensional representation. The system can provide the three-dimensional representation of the location for display to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates generally to providing immersive views of locations. For example, the present disclosure relates to methods and systems for integrating existing user-generated media content into a three-dimensional immersive representation of a location. [Background technology]

[0002] Geographic information systems (GIS) enable a variety of applications, including urban planning, navigation, environmental monitoring, and virtual tourism. With the advent of high-resolution imagery and the proliferation of location-aware devices, the amount of user-generated content available to enhance the realism and information value of virtual geographic representations is exponentially increasing.

[0003] However, integrating this user-generated content into 3D geographic models presents significant technical challenges. Specifically, the computational complexity associated with processing, selecting, and rendering large amounts of disparate data into a coherent, navigable 3D environment is substantial. This complexity is exacerbated when considering the need to maintain real-time interactivity and visual fidelity within these virtual environments.

[0004] As a result, a technical problem exists in the field of computerized geographic information systems as to how to efficiently and effectively project user-generated content into an immersive 3D view of a location while addressing the complex computations inherent in processing, selecting, and rendering in real time. Solving this technical problem would greatly improve the usability and realism of virtual geographic representations, providing users with a richer and more informative experience when exploring virtual environments. Summary of the Invention

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned by practice of exemplary embodiments.

[0006] In one or more exemplary embodiments, a computer-implemented method for updating a three-dimensional representation to include user-generated content is described. The method includes obtaining a three-dimensional representation of a location, the three-dimensional representation being generated based on a plurality of images. The method includes accessing user-generated media content associated with the location. The method includes receiving route information representing at least a portion of a route through the three-dimensional representation of the location. The method includes selecting one or more pieces of user-generated media content based at least in part on the route information. The method may include integrating one or more pieces of user-generated media content into the three-dimensional representation of the location based on the route information and portions of the three-dimensional representation displayed to the user, the pieces of user-generated media content being presented within one or more visual pop-outs within the three-dimensional representation. The method includes providing the three-dimensional representation of the location for display to the user.

[0007] Another exemplary aspect of the present disclosure is directed to a computing system. The system may include one or more processors and one or more non-transitory computer-readable media collectively storing instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include obtaining a three-dimensional representation of a location, the three-dimensional representation being generated based on a plurality of images. The operations may include accessing user-generated media content associated with the location. The operations may include receiving route information representing at least a portion of a route through the three-dimensional representation of the location. The operations may include selecting one or more pieces of user-generated media content based at least in part on the route information. The operations may include integrating one or more pieces of user-generated media content into the three-dimensional representation of the location based on the route information and portions of the three-dimensional representation displayed to the user, the pieces of user-generated media content being presented within one or more visual pop-outs within the three-dimensional representation. The operations may include providing the three-dimensional representation of the location for display to the user.

[0008] Another exemplary aspect of the present disclosure is directed to one or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations. The operations may include obtaining a three-dimensional representation of a location, the three-dimensional representation being generated based on a plurality of images. The operations may include accessing user-generated media content associated with the location. The operations may include receiving route information representing at least a portion of a route through the three-dimensional representation of the location. The operations may include selecting one or more pieces of user-generated media content based at least in part on the route information. The operations may include integrating one or more pieces of user-generated media content into the three-dimensional representation of the location based on the route information and portions of the three-dimensional representation displayed to the user, the pieces of user-generated media content being presented within one or more visual pop-outs within the three-dimensional representation. The operations may include providing the three-dimensional representation of the location for display to the user.

[0009] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description, drawings, and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain the relevant principles.

[0010] A detailed description of exemplary embodiments, directed to those skilled in the art, is set forth in the specification, which refers to the accompanying drawings. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is an exemplary system according to one or more exemplary embodiments of the present disclosure. [Figure 2] 1 includes an example block diagram of an electronic device and a server, according to one or more example embodiments of the present disclosure. [Figure 3]1 illustrates an exemplary system for integrating user-generated content into a three-dimensional representation of a location, according to an exemplary embodiment of the present disclosure. [Figure 4] 1 illustrates a user interface screen of a mapping application, according to one or more exemplary embodiments of the present disclosure. [Figure 5A] 1 illustrates an exemplary immersive three-dimensional representation of a location, according to one or more exemplary embodiments of the present disclosure. [Figure 5B] 1 illustrates an exemplary immersive three-dimensional representation of a location, according to one or more exemplary embodiments of the present disclosure. [Figure 6A] 1 illustrates an exemplary immersive three-dimensional representation of a location where a visual pop-out has been inserted, according to one or more exemplary embodiments of the present disclosure. [Figure 6B] 1 illustrates an exemplary immersive three-dimensional representation of a location where a visual pop-out has been inserted, according to one or more exemplary embodiments of the present disclosure. [Figure 6C] 1 illustrates an exemplary immersive three-dimensional representation of a location where a visual pop-out has been inserted, according to one or more exemplary embodiments of the present disclosure. [Figure 7A] 10A-10C show examples of three-dimensional representations of locations into which user-generated media content is seamlessly integrated, according to some embodiments of the present disclosure. [Figure 7B] 10 shows pieces of user-generated media content used to generate a 3D representation, according to some embodiments of the present disclosure. [Figure 8] 1 illustrates an exemplary flow diagram of a method for integrating user-generated media content into a three-dimensional representation of a location, according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Reference will now be made to the embodiments of the present disclosure, one or more example embodiments of which are illustrated in the drawings, wherein like reference numerals indicate like elements. Each example is provided as an explanation of the disclosure and is not intended to limit the disclosure. Indeed, it will be apparent to those skilled in the art that various modifications and variations can be made to the present disclosure without departing from the scope or spirit of the disclosure. For example, features illustrated or described as part of one embodiment can be used with other embodiments to yield still further embodiments. It is therefore intended that the present disclosure cover such modifications and variations as come within the scope of the appended claims and their equivalents.

[0013] The terms used herein are used to describe exemplary embodiments and are not intended to limit and / or restrict the present disclosure. The singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In this disclosure, terms such as "including," "having," "comprising," and the like are used to specify features, numbers, steps, operations, elements, components, or combinations thereof, but do not exclude the presence or addition of one or more of the features, elements, steps, operations, elements, components, or combinations thereof.

[0014] Terms such as first, second, and third may be used herein to describe various elements, but it will be understood that the elements are not limited by these terms. Instead, these terms are used to distinguish one element from another. For example, a first element may be referred to as a second element, and a second element may be referred to as a first element, without departing from the scope of the present disclosure.

[0015] The term "and / or" includes a combination of two or more associated listed items or any of two or more associated listed items. For example, the scope of an expression or phrase "A and / or B" includes item "A," item "B," and the combination of items "A and B."

[0016] Furthermore, the scope of the expression or phrase "at least one of A or B" is intended to include all of: (1) at least one of A, (2) at least one of B, and (3) at least one of A and at least one of B. Similarly, the scope of the expression or phrase "at least one of A, B, or C" is intended to include all of: (1) at least one of A, (2) at least one of B, (3) at least one of C, (4) at least one of A and at least one of B, (5) at least one of A and at least one of C, (6) at least one of B and at least one of C, and (7) at least one of A, at least one of B, and at least one of C.

[0017] The present disclosure is directed to systems and methods for integrating (e.g., embedding) user-generated media content within a three-dimensional (3D) representation of a location. More specifically, when a user selects to display an immersive three-dimensional representation of a location, a representation modification system can select one or more pieces of user-generated media content to be integrated into the immersive three-dimensional representation of the location. The one or more pieces of user-generated media content can be selected based on one or more of a particular category of media content, a particular subject or topic, a location associated with a particular three-dimensional representation (or portion thereof), subjects included in the piece of media content, a rating of the piece of user-generated media content, etc.

[0018] Selected pieces of user-generated media content can be inserted into the three-dimensional representation of the location. In some examples, the pieces of user-generated media content can be displayed as one or more visual pop-ups along a path through the three-dimensional representation of the scene. Each visual pop-up can be displayed on a particular piece of the three-dimensional representation, thereby displaying the piece of user-generated media content within the three-dimensional representation. A user can select a particular pop-up to receive a more detailed version of the piece of user-generated visual media.

[0019] For example, the three-dimensional representation of the location may be of a restaurant. When a user begins viewing the three-dimensional representation of the location, the display system may determine one or more pieces of user-generated media content to display in one or more pop-outs in the three-dimensional representation of the location. These pieces of user-generated content may be selected based on a particular location in three-dimensional space with which each piece of user-generated media content is associated and / or based on the content of the pieces of user-generated media content. Each pop-out may be populated with a single piece of user-generated media content (e.g., an image, video, or audio).

[0020] The user-presented view of the three-dimensional representation can be controlled to progress through the three-dimensional representation along a particular path. As the display follows the path through the three-dimensional representation, multiple visual pop-ups can be displayed. The user can select each pop-out (e.g., by clicking on the pop-out in the user interface). In response, the user interface can be updated to present a larger and more detailed view of the user-generated media content. In some examples, the user-generated content includes a location under specified conditions (e.g., a particular time of day, particular weather conditions, etc.), the number of people in the location (e.g., crowded, empty, etc.), the general atmosphere (e.g., lively, calm, etc.), recommended attire (e.g., formal, trendy, casual, sportswear, etc.), the noise level (e.g., quiet, noisy, etc.), etc.

[0021] More generally, this disclosure is directed to systems and methods for integrating (e.g., embedding) user-generated media content with three-dimensional (3D) representations of locations. The systems and methods may be performed by an application on a user computing device, a remote server system, or a combination of both. A server computing system may be any computing system configured to communicate with a user computing device (or other computing devices) over a network to provide information or services. When a server computing system is employed, the server computing system may receive a request from a user computing device to view a three-dimensional representation of a location. Data describing the three-dimensional representation may be transmitted to the user computing device for display to the user via an application on the user computing device.

[0022] A user computing device may be any computing device designed to be operated by an end user. For example, a user computing device may include, but is not limited to, a personal computer, a smartphone, a smart watch, a fitness band, a tablet computer, a laptop computer, a handheld navigation computing device, a wearable computing device, a gaming console, etc. In some examples, a user computing device may include one or more communication systems capable of communicating with a server system via a communication network.

[0023] Navigation and mapping systems may include immersive view applications that provide users of computing devices with a way to explore locations through multidimensional views of areas or points of interest, including landmarks, restaurants, stores, etc. The immersive view application may be part of a navigation application, a separate mapping application, or a standalone application. The immersive view application may receive multiple images of a particular location or point of interest. In some examples, the multiple images are generated specifically for the immersive view application. In this case, the multiple images are generated (or captured) by a user moving through the location and capturing images periodically (e.g., one image per second). In other examples, the multiple images may be sourced from existing images of the area. These existing images may be filtered to identify the most valuable images for generating a three-dimensional representation of the scene.

[0024] The multiple images can be provided to a machine-learned model trained to generate a three-dimensional representation of the location based on the multiple images. For example, the machine-learned model can use a neural radiance field process to generate the three-dimensional representation of the location based on the multiple images. The machine-learned model can output the three-dimensional representation of the location. The three-dimensional representation of the location can display an immersive view of a scene associated with the location. The content of the displayed immersive view can depend on the location and orientation of a viewing object (e.g., a simulated camera positioned within the three-dimensional representation).

[0025] The viewpoint object can be moved throughout the three-dimensional representation to view different portions of the scene and aspects of the location from different angles. In some examples, the movement of the viewpoint object can be predetermined based on one or more predetermined tracks or paths throughout the three-dimensional space. These predefined paths can be determined based on the path of a user who generated the images used to create the three-dimensional representation of the location. In other examples, the three-dimensional representation can be flexible so that the viewpoint object can move according to user input.

[0026] In some examples, the three-dimensional representation can be augmented with additional media content to provide additional information as a user views the three-dimensional representation. For example, the three-dimensional representation may include only fixed or static features (e.g., permanent or semi-permanent features) of a location. As a result, information associated with movable objects or transient phenomena, such as people, food, and the environment, including lighting and ambiance, may not be represented within the three-dimensional representation. This additional information can be provided through user-generated media content inserted into the 3D representation as a series of two-dimensional pop-outs or visual data display windows positioned throughout the 3D representation.

[0027] User-generated media content may include images, video, and audio captured by a user and made available to a representation generation system. For example, a user may post this information to a publicly accessible social media site and indicate that the information is available for generating a three-dimensional representation of a location. Thus, user-generated content may only be used with the user's explicit permission.

[0028] User-generated media content can provide information about non-permanent aspects or characteristics of a location, such as the location's overall atmosphere and feel, the services and meals offered, seasonal decorations, the estimated occupancy level of the location at a particular time, etc.

[0029] The representation generation system can determine the number and type of user-generated media content to display in one or more pop-outs within the representation. The representation generation system can determine the number of pop-outs to display within the representation based on several factors. For example, the representation generation system can determine the overall size of the 3D representation of the location. For example, the representation generation system can determine a path through the representation. The representation path can be determined based on the user's path when the image used to generate the representation was originally captured. In another example, the user can choose their own path within the representation, which can be used as the total distance of the user's path.

[0030] The representation generation system can determine the density of visual pop-ups displayed in the representation of the location. In some examples, the density can be based on how important the pop-ups are to the user's understanding of one or more non-stationary features of the location. For example, if many objects are present in the representation, the density can be less than a representation with fewer objects in the original view. In some examples, the estimated speed of the camera through the location can also be used to determine the density of the pop-outs. For example, if the camera associated with the user moves slowly, the density of the pop-outs can be high, and vice versa. The generative representation system can determine the number of pop-outs to display based on one or more factors.

[0031] In some examples, the representation generation system can determine the location of the pop-out based on existing features within the three-dimensional representation of the location. For example, a particular feature of the representation can be selected as an anchor point for the pop-out. In other examples, the location of the pop-up can be determined based on the highest-rated locations of the user-generated media content.

[0032] For example, each piece of user-generated media content may have an associated location. The associated location may be mapped to a representation of the location to determine where the media content is displayed within the representation. In some examples, the representation generation system may select which piece of user-generated media content to display based on one or more subjects determined to be relevant to the representation of the location. For example, for a particular location such as a restaurant, the subject of interest may be one of the environment, mood, ambiance, food, activity, etc. The subject of interest may be different in other locations, such as a mini-golf course. For example, the subject of interest at a mini-golf course may include user density, general weather, and general visitor atmosphere, which may be more interesting.

[0033] In some examples, particular pieces of user-generated media content may be selected based on their associated locations. For example, the representation generation system may determine that a particular location within the representation is suitable for a pop-out. Based on this determination, the three-dimensional representation modification system may select the highest-rated piece of content associated with that location. Similarly, suppose two pieces of highly rated user-generated content are near each other. In that case, the representation generation system may include only one of them in the representation of the location so that two pop-outs do not appear close together.

[0034] In some examples, the selected piece of user-generated content is chosen based on the time the piece of user-generated content was captured. For example, if a user chooses to view a three-dimensional representation of a location, they may select a specific time (e.g., 6:30 PM), a date (Friday night), or both (e.g., a specific date and a specific time). Based on the time and / or date selected by the user, the representation generation system can select user-generated content to insert into the three-dimensional representation of the location at the selected time / date. For example, assume a particular restaurant opens up space for a dance floor after 10:00 PM on Friday. In that case, a piece of user-generated content displayed at 10:30 PM on Friday may include content depicting the dance floor and the expected atmosphere that accompanies it. Similarly, before that time, the selected piece of user-generated media content would not be associated with a dance floor.

[0035] In some examples, a user may select a specific date or season (e.g., a holiday or other important date or season), and the presentation modification system may select pieces of user-generated media content appropriate for that date or season. For example, a user may see examples of the ambiance or lighting that can be expected at a particular location during the winter holiday season, or the pool ambiance available at a resort during the summer.

[0036] Once the presentation modification system determines the number of pieces of media content, the locations of the pieces of media content, the subjects of interest of the pieces of user-generated media content, and the times / dates of interest to the user, the presentation modification system can filter the candidate pieces of user-generated media content based on the specified criteria. Once the pieces of user-generated media content have been filtered, the presentation modification system can select one or more pieces of user-generated media content with the highest ratings.

[0037] In some examples, user-generated media content may be rated based on the degree to which it matches a particular set of criteria, including subject, location, time, etc. In some examples, the ratings are generated by a third-party rating service. In some examples, an expression modification system can generate the ratings.

[0038] In some examples, the presentation modification system can modify selected pieces of user-generated media content. For example, an image can be cropped or a video can be edited. In some examples, particular details can be removed or particular features can be highlighted. In some examples, text associated with the user-generated media content can also be displayed.

[0039] In some examples, the selected image can be inserted into the three-dimensional representation of the location as a visual pop-out. The visual pop-out can be displayed within the three-dimensional representation, allowing a user to view the visual pop-up as they navigate the three-dimensional representation of the location. For example, an application installed on the user's computing device can present the three-dimensional representation of the location to the user. The visual pop-out can be a two-dimensional image displayed in the context of a particular location with the three-dimensional representation. The visual pop-out can include a border (e.g., a white or black border) to distinguish it from other portions of the three-dimensional representation. Additionally, each visual pop-out can be associated with a particular location within the three-dimensional representation and can have a visual tail connecting the border to the particular location within the three-dimensional representation.

[0040] While navigating the three-dimensional representation of the location, the user can interact with or select one or more pop-outs to obtain more details about the user-generated content. For example, if the user clicks on a particular piece of user-generated visual content, the interface may update to enlarge that piece of user-generated content for the user to view.

[0041] According to examples of the present disclosure, a server computing system may provide a computing device with a three-dimensional representation of a location that can be presented on a display device of the computing device. The three-dimensional representation of the location may be provided dynamically (e.g., generated and transmitted in response to a request from the computing device), or the three-dimensional representation of the location may be provided by retrieving the three-dimensional representation of the location from a database. An integrated three-dimensional scene of the location may be retrieved from the database according to the terms of the request.

[0042] Therefore, the present disclosure provides techniques to address the computational complexity of processing, selecting, and rendering large amounts of heterogeneous user-generated data in real time and integrating them into a coherent, navigable 3D environment. Technical solutions include computer-implemented methods that include obtaining a 3D representation of a location based on multiple images, accessing associated user-generated media content, receiving route information within the 3D representation, selecting content based on the route information, and integrating the content into the 3D representation and presenting it in a visual pop-out. Specific technical implementations include using machine-learned models to generate the 3D representation, determining a route through the representation based on user input or a predefined route, and / or selecting user-generated content along the route that enhances understanding of non-permanent aspects of the location.

[0043] One or more technical advantages of the solutions described herein include enabling a user to easily and more accurately obtain an accurate representation of the state of a location under specific circumstances or conditions. For example, a user can easily and more accurately obtain an accurate representation of the state of an indoor or outdoor spot, including a restaurant or a park, at a specific time, season, etc. For example, a user can easily and more accurately obtain an accurate representation of the state of an indoor or outdoor spot, including a restaurant or a park, under specific environmental conditions (e.g., sunny weather, rainy weather, strong winds, etc.). The above method provides an accurate representation of the state of a location virtually via a display without the user having to physically visit the location. Furthermore, an accurate prediction of the state of a location at a specific time or under specific conditions specified by the user may also be provided.

[0044] One or more technical advantages of the solutions described herein also include integrating unused media content (e.g., user-generated media content) associated with a location with an existing three-dimensional representation of the location. For example, the media content may be acquired after the imagery used to form the three-dimensional representation of the location is acquired. Thus, the three-dimensional representation integrated with pieces of user-generated media content represents an accurate, up-to-date state of the location. Furthermore, various unique three-dimensional representations may be generated to accurately portray a location according to various conditions. For example, a server computing system may be configured to select and integrate pieces of user-generated media content based on information associated with the media content that matches a user's request for an immersive view. For example, images of a restaurant interior taken in the morning when few customers are present would not be integrated into an integrated 3D scene generated for an immersive view of the restaurant at dinnertime. Thus, metadata and other descriptive content associated with the media content may be used to accurately form a three-dimensional representation of a location. Similarly, image segmentation techniques and machine learning resources can be implemented to position or place dynamic objects extracted from media content at appropriate locations within a three-dimensional representation of a location to accurately provide the state of the location.

[0045] Thus, aspects of the proposed system and method represent a technical solution to the technical problem of augmenting existing three-dimensional representations of static content of a location with data representing dynamic aspects of the location. The system introduces a new way to select and integrate user-generated media content into existing three-dimensional representations representing non-static aspects of a location so that a user can grasp the atmosphere and / or environment of the location with minimal additional cost and time. Thus, it solves the problem of how to present dynamic data to a user when displaying a three-dimensional representation that includes static elements of a location.

[0046] FIG. 1 illustrates an exemplary system in accordance with one or more exemplary embodiments of the present disclosure. FIG. 1 illustrates an example system including a user computing device 100, an external computing device 200, a server computing system 300, and external content 500, which may communicate with each other through a network 400. For example, the user computing device 100 and the external computing device 200 may include any of a personal computer, a smartphone, a tablet computer, a global positioning service device, a smartwatch, etc. The network 400 may include any type of communication network, including a wired or wireless network, or a combination thereof. The network 400 may include a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a personal area network (PAN), a virtual private network (VPN), etc. For example, wireless communication between elements of exemplary embodiments may occur via wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), Ultra Wide Band (UWB), Infrared Data Communication (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), radio frequency (RF) signals, etc. For example, wired communication between elements of exemplary embodiments may occur via cable pairs, coaxial cables, fiber optic cables, Ethernet cables, etc. Communications over the network may use various types of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0047] As described in more detail below, in some embodiments, user computing device 100 and / or server computing system 300 may form part of a navigation and mapping system that can provide a user of user computing device 100 with an immersive view of a location.

[0048] In some demonstrative embodiments, server computing system 300 may retrieve data from one or more of user-generated content data store 350, POI data store 370, navigation data store 380, and user data store 390 to perform various operations and aspects of the navigation and mapping system disclosed herein. User-generated content data store 350, POI data store 370, navigation data store 380, and user data store 390 may be provided integrally with server computing system 300 (e.g., as part of one or more memory devices 320 of server computing system 300) or may be provided separately (e.g., remotely). Furthermore, user-generated content data store 350, POI data store 370, navigation data store 380, and user data store 390 may be combined into a single data store (database) or may be multiple respective data stores. Data stored in one data store (e.g., POI data store 370) may overlap with some data stored in another data store (e.g., navigation data store 380). In some implementations, one data store may reference data stored in another data store (eg, user-generated content data store 350).

[0049] User-generated content data store 350 can store media content captured by a user, for example, via user computing device 100, external computing device 200, or some other computing device. User-generated media content may include user-generated image, video, and / or user-generated audio content. For example, the media content may be captured by a person operating a user computing device (e.g., a smartphone) or may be captured indirectly, for example, by a computing system monitoring a location (e.g., a security system, a surveillance system, etc.).

[0050] For example, user-generated media content may be captured by a computing device's camera (e.g., image capture device 182 of FIG. 2 ) and may include images of locations including restaurants, landmarks, businesses, schools, etc. The images may include various information (e.g., metadata, semantic data, etc.) useful for integrating the image (or portions of the image) into a three-dimensional representation of the location associated with the image. For example, an image may include information including the date the image was captured, the time the image was captured, location information (e.g., GPS location) indicating where the image was taken, etc. For example, an image may be associated with descriptive metadata, which may include keywords associated with the image, a title or name of the image, environmental information at the time the image was captured (e.g., lighting conditions including brightness levels, noise conditions including decibel levels, weather information including weather conditions such as temperature, wind speed, precipitation, cloud cover, humidity, etc.). Environmental information may be obtained from sensors in the computing device used to capture the image or from another computing device.

[0051] For example, user-generated media content may be captured by a microphone (e.g., sound capture device 184) of a user computing device and may include audio associated with locations, including restaurants, landmarks, businesses, schools, etc. For example, audio content may include information including the date the audio was captured, the time the audio was captured, location information indicating the location where the audio was captured (e.g., GPS location), etc. For example, the audio may be provided with descriptive metadata, which may include keywords associated with the audio, a title or name of the audio, environmental information at the time the audio was captured (e.g., lighting conditions including brightness levels, noise conditions including decibel levels, weather information including weather conditions such as temperature, wind speed, precipitation, cloud cover, humidity, etc.), etc. Environmental information may be obtained from sensors in the computing device used to capture the audio or from another computing device.

[0052] POI data store 370 can store information about locations or points of interest, such as points of interest in areas or regions associated with one or more geographic areas. Points of interest may include any destination or location. For example, points of interest may include restaurants, museums, sports venues, concert halls, amusement parks, schools, businesses, grocery stores, gas stations, theaters, shopping malls, lodging, etc. Point of interest data stored in POI data store 370 may include any information associated with a POI. For example, the POI data store 370 may include location information of the POI, opening hours of the POI, a phone number of the POI, reviews about the POI, financial information associated with the POI (e.g., average cost of services offered at the POI, such as meals, tickets, rooms, etc., and / or goods sold at the POI), environmental information about the POI (e.g., noise levels, descriptions of the environment, traffic levels, etc., which may be provided in real time or obtained by various sensors located at the POI), descriptions of the types of services offered at the POI and / or goods sold at the POI, languages ​​spoken at the POI, the URL of the POI, image content associated with the POI, etc. For example, information about the POI may be obtainable from external content 500 (e.g., from a web page associated with the POI or from sensors located at the POI).

[0053] The navigation data store 380 may store or provide map data / geospatial data used by the server computing system 300. Exemplary geospatial data includes geographic imagery (e.g., digital maps, satellite imagery, aerial photographs, street-level photographs, synthetic models, etc.), tables, vector data (e.g., vector representations of roads, parcels, buildings, etc.), point-of-interest data, or other suitable geospatial data associated with one or more geographic areas. In some examples, the map data may include a series of submaps, each submap including data for a geographic area including objects (e.g., buildings or other static features), travel paths (e.g., roads, highways, transit lines, sidewalks, etc.), and other features of interest. The navigation data store 380 may be used by the server computing system 300 to provide navigation guidance, perform searches for points of interest, provide location or classification data for points of interest, determine distances, routes, or travel times between locations, or any other suitable application or task necessary or beneficial for performing the operation of the exemplary embodiments disclosed herein.

[0054] For example, the navigation data store 380 may store 3D scene images 382, ​​which include images associated with generating 3D scenes of various locations. For example, the 3D representation generator 336 may be configured to generate a 3D representation based on multiple images of a location (e.g., the interior of a restaurant, a park, etc.). Multiple images may be captured and combined using a machine-learned model (or other method) to create a 3D representation of the location. For example, the 3D representation generator 336 may use a neural radiance field method to create the 3D representation model. In some implementations, methods including structure from motion algorithms may be used to estimate three-dimensional structure. In some implementations, machine learning resources may be implemented to generate camera-like images from any viewpoint within a location based on the captured images. For example, a video fly-through of the location may be generated based on the captured images. In some implementations, the initial 3D representation generated by the 3D representation generator 336 may be a static 3D scene without variable or dynamic objects (e.g., moving objects). For example, an initial 3D scene of a park may include images of the park, including images of trees, playground equipment, picnic tables, etc., but not images of people, dogs, or non-static objects. User-generated content may include images of variable or dynamic objects, which may be associated with different times and / or conditions (e.g., different times of the day, day of the week, or season, different lighting conditions, different environmental conditions, etc.).

[0055] For example, navigation data store 380 may store integrated 3D scene image 384, which includes 3D scenes of various locations with integrated user-generated media content. In one example, representation modification system 338 may be configured to integrate user-generated content from user-generated content data store 350 with three-dimensional representations obtained from 3D scene image 382. For example, integrated 3D scene image 382 may include 3D scenes of various locations with integrated user-generated media content. The 3D scene generated based on multiple images of the location may be integrated with the media content using known methods to create integrated 3D scene image 384 of the location. For example, representation modification system 338 may be configured to identify and extract one or more objects (e.g., one or more dynamic objects) from the image of the scene.

[0056] For example, the expression modification system 338 may be configured to position or arrange one or more selected pieces of user-generated media content within a three-dimensional representation associated with the user-generated media content. For example, the expression modification system 338 may select a piece of user-generated media content from a database of user-generated media content (e.g., the user-generated media data store 342 of FIG. 3 ) that corresponds to a three-dimensional representation requested by a user. For example, the expression modification system 338 may select from among multiple pieces of user-generated content from the data (e.g., the user-generated media data store 342) the piece of user-generated content that has the highest similarity (e.g., in terms of time of day, season, weather conditions, lighting conditions, etc.) to the user's request. For example, a user-generated image taken at noon in a park on a sunny day may include people playing on playground equipment. The expression modification system 338 may be configured to extract children from the image using various techniques (e.g., image segmentation algorithms, machine learning resources, cropping tools, etc.). The representation modification system 338 may be configured to select pieces of 3D user-generated media content from a database that have similar characteristics as the image (e.g., same time of day, season, weather conditions, etc.). The representation modification system 338 may be configured to position the images of the people within the visual pop-up and generate an updated or integrated three-dimensional representation such that the images of the people are positioned within the scene (e.g., on or near a slide, on or near a see-saw, etc.), providing a user viewing the integrated three-dimensional representation with an accurate representation of the park's conditions at that time of day, as well as the overall atmosphere of the park at that time of day, e.g., under similar weather conditions.

[0057] Media content, including user-generated content and / or machine-generated content, may include audio content and / or images of variable or dynamic objects, and the audio content and images may be associated with different times and / or conditions (e.g., different times of day, days of the week, seasons, different lighting conditions, different environmental conditions, etc.). The representation modification system 338 may be configured to integrate the user-generated content and / or machine-generated content with the initial three-dimensional representation generated by the three-dimensional representation generator 336, for example, according to time information associated with the media content. For example, a first integrated three-dimensional representation of a location may be associated with a first time based on media content captured at or associated with the first time (e.g., a first time period, a first season, etc.), and a second integrated three-dimensional representation of the location may be associated with a second time based on media content captured at or associated with the second time (e.g., a second time period, a second season, etc.).

[0058] In some exemplary embodiments, user data store 390 may represent a single database. In some embodiments, user data store 390 represents multiple different databases accessible from server computing system 300. In some examples, user data store 390 may include current user location and heading data. In some examples, user data store 390 may include information related to one or more user profiles, including various user data, such as user preference data, user demographic data, user calendar data, user social network data, and user travel history data. For example, user data store 390 may include, but is not limited to, email data including text content, images, calendar information associated with an email, or contact information; social media data including comments, reviews, check-ins, likes, invitations, contacts, or reservations; calendar application data including dates, times, events, descriptions, or other content; virtual wallet data including purchases, e-tickets, coupons, or transactions; scheduling data; location data; SMS data; or other suitable data associated with a user account. According to one or more examples of the present disclosure, the data can be analyzed to determine a user's preferences regarding POIs, for example, to automatically suggest or provide immersive views of the user's preferred locations. The immersive views can also be associated with the user's preferred times of day (e.g., if user data indicates that a park is the user's preferred POI and is most frequently visited in the evening, an immersive view of the park in the evening is provided). For example, the data can be analyzed to determine the user's preferences regarding POIs to determine the user's travel preferences (e.g., mode of transportation, allowable travel time, etc.), potential POI recommendations for the user, the user's possible travel routes and modes of transportation to the POIs, etc.

[0059] The user data store 390, in some embodiments, is provided to represent potential data that may be analyzed by the server computing system 300 to identify user preferences, recommend points of interest (POIs), determine possible travel routes to a POI, determine the mode of transportation to use to travel to a POI, determine an immersive view of a location to provide to a computing device associated with the user, etc. However, such user data may not be collected, used, or analyzed unless the user consents after being informed of what data will be collected and how that data will be used. Furthermore, in some embodiments, the user may be provided with tools (e.g., in the navigation application or via a user account) to revoke or modify the scope of permissions. Furthermore, certain information or data may be processed in one or more ways before being stored or used, such that personally identifiable information is removed or the data is stored in an encrypted form. Thus, certain user information stored in the user data store 390 may or may not be accessible to the server computing system 300 based on the permissions given by the user, or such data may not be stored in the user data store 390 at all.

[0060] The external content 500 can be in any form, including news articles, web pages, video files, audio files, written descriptions, ratings, gaming content, social media content, photos, commercial offers, transportation information, weather conditions, sensor data acquired by various sensors, or other suitable external content. The user computing device 100, the external computing device 200, and the server computing system 300 can access the external content 500 through the network 400. The external content 500 can be searched by the user computing device 100, the external computing device 200, and the server computing system 300 using known search methods, and the search results can be ranked according to relevance, popularity, or other suitable attributes, including location-specific filtering or promotions.

[0061] 2 includes an exemplary block diagram of a computing device and a server computing system in accordance with one or more exemplary embodiments of the present disclosure. Although a user computing device 100 is depicted in FIG. 2, the features of the user computing device 100 described herein are also applicable to an external computing device 200.

[0062] The user computing device 100 may include one or more processors 110, one or more memory devices 120, a navigation and mapping system 130, a position determination device 140, an input device 150, a display device 160, an output device 170, and a capture device 180. The server computing system 300 may include one or more processors 322, one or more memory devices 320, and a navigation and mapping system 330.

[0063] For example, the one or more processors 110 may be any suitable processing device that may be included in the user computing device 100 or the server computing system 300. For example, the one or more processors 110 may include one or more of a processor, processor core, controller and arithmetic logic unit, central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), image processor, microcomputer, field programmable array, programmable logic unit, application specific integrated circuit (ASIC), microprocessor, microcontroller, etc., and combinations thereof, including any other device capable of responding in a defined manner and executing instructions. The one or more processors 110 may be a single processor or multiple processors operatively connected, e.g., connected in parallel.

[0064] The one or more memory devices 120 may include one or more non-transitory computer-readable storage media, including read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), flash memory, USB drives, volatile memory devices including random access memory (RAM), hard disks, floppy disks, Blu-ray disks, or optical media such as CD ROM disks and DVDs, and combinations thereof. However, examples of the one or more memory devices 120 are not limited to the above description, and the one or more memory devices 120 may be realized by a variety of other devices and structures, as will be appreciated by those skilled in the art.

[0065] For example, one or more memory devices 120 can store instructions 324 that, when executed, cause one or more processors 110 to execute immersive view application 132 and perform the following operations: receiving a first input via input device 150 requesting a first immersive view of a location representing a first state of the location at a first time; and providing the first immersive view of the location for presentation on display device 160, the first immersive view including a three-dimensional (3D) representation of the location generated based on the plurality of images and one or more pieces of user-generated media content included in a visual pop-out within the 3D representation of the location. The one or more pieces of user-generated media content can provide the user with additional information regarding the state of the location or one or more qualities associated therewith (e.g., the atmosphere of the location).

[0066] The one or more memory devices 120 may also include data 122 and instructions 124 that can be retrieved, manipulated, created, or stored by the one or more processors 110. In some demonstrative embodiments, such data can be accessed and used as input to execute instructions to cause the immersive view application 132 to perform the following operations: receive a first input via the input device 150 requesting a first immersive view representing a first state of a location at a first time; and provide the first immersive view of the location for presentation on the display device 160, the first immersive view including a three-dimensional (3D) scene of the location generated based on a plurality of images and first media content integrated with the 3D scene of the location, the first media content representing the first state of the location at the first time. As described according to an embodiment of the present disclosure, the operations may further include receiving, via input device 150, a second input requesting a second immersive view of the location representing a second state of the location at a second time, and providing the second immersive view of the location for presentation on display device 160, the second immersive view including a 3D scene of the location generated based on the plurality of images and second media content integrated with the 3D scene of the location, the second media content representing the second state of the location at the second time.

[0067] In some exemplary embodiments, user computing device 100 includes a navigation and mapping system 130. For example, navigation and mapping system 130 may include an immersive view application 132 and a navigation application 134.

[0068] According to examples of the present disclosure, immersive view application 132 may be executed by user computing device 100 to provide a user of user computing device 100 with a way to explore locations through multi-dimensional views of areas or points of interest, including landmarks, restaurants, etc. In some implementations, immersive view application 132 may provide a video fly-through of a location to provide a user with an interior view of the location. Immersive view application 132 may be part of navigation application 134, a separate mapping application, or a standalone application.

[0069] In some examples, one or more aspects of the immersive view application 132 may be implemented by an immersive view application 332 of a server computing system 300, which may be remotely located, to provide the requested immersive view. In some examples, one or more aspects of the immersive view application 332 may be implemented by an immersive view application 132 of a user computing device 100 to generate the requested immersive view.

[0070] According to examples of the present disclosure, the navigation application 134 may be executed by the user computing device 100 to provide a user of the user computing device 100 with a way to navigate to a location. The navigation application 134 may provide navigation services to the user. In some examples, the navigation application 134 may facilitate a user's access to the server computing system 300, which provides navigation services. In some exemplary embodiments, the navigation services include providing routes to a particular location, such as a point of interest (POI). For example, the user may input a destination (e.g., an address or name of the POI). In response, the navigation application 134 may provide navigation information for the user to navigate to the destination using locally stored map data for a particular geographic area and / or map data provided via the server computing system 300. For example, the navigation information may include turn-by-turn route directions from a current location (or a provided origin or starting point) to the destination. For example, the navigation information may include a travel time (e.g., an estimated or predicted travel time) from the current location (or a provided start or starting point) to the destination.

[0071] The navigation application 134 can visually depict a geographic area via the display device 160 of the user computing device 100. The visual depiction of the geographic area can include highlighting of one or more roads, one or more points of interest (including buildings, landmarks, etc.), and a planned route. In some examples, the navigation application 134 can also provide a location-based search option that identifies one or more searchable points of interest within a given geographic area. In some examples, the navigation application 134 can include a local copy of relevant map data. In other examples, the navigation application 134 can access information on a server computing system 300, which can be remotely located, to provide the requested navigation services.

[0072] In some examples, navigation application 134 may be a specialized application specifically designed to provide navigation services, while in other examples, navigation application 134 may be a general application (e.g., a web browser) that can provide access to a variety of different services, including navigation services, over network 400.

[0073] In some demonstrative embodiments, user computing device 100 includes positioning device 140. Positioning device 140 can determine the current geographic location of user computing device 100 and transmit such geographic location to server computing system 300 over network 400. Positioning device 140 can be any device or circuitry for analyzing the location of user computing device 100. For example, positioning device 140 can determine the actual or relative location by using a satellite navigation positioning system (e.g., GPS, Galileo Positioning System, Global Navigation Satellite System (GLONASS), BeiDou Satellite Navigation and Positioning System), an inertial navigation system, a dead reckoning system, based on an IP address, by using triangulation and / or proximity to cellular towers or WiFi hotspots, and / or other suitable techniques for determining the location of user computing device 100.

[0074] The user computing device 100 may include an input device 150 configured to receive input from a user, and may include, for example, one or more of a keyboard (e.g., a physical keyboard, a virtual keyboard, etc.), a mouse, a joystick, a button, a switch, an electronic pen or stylus, a gesture recognition sensor (e.g., recognizing a user's gestures including body movements), a voice input device or voice recognition sensor (e.g., a microphone receiving voice input such as voice commands or voice queries), an output sound device (e.g., a speaker), a trackball, a remote controller, a portable phone (e.g., a mobile phone or smartphone), a tablet PC, a pedal or foot switch, a virtual reality device, etc. The input device 150 may further include a haptic device that provides tactile feedback to the user. The input device 150 may also be embodied, for example, by a touch-sensitive display having touchscreen functionality. For example, the input device 150 may be configured to receive input from a user associated with the input device 150.

[0075] User computing device 100 may include a display device 160 that displays user-viewable information (e.g., a map, an immersive view of a location, a user interface screen, etc.). For example, display device 160 may be a non-touch-sensitive display or a touch-sensitive display. Display device 160 may include, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, an active matrix organic light-emitting diode (AMOLED), a flexible display, a 3D display, a plasma display panel (PDP), a cathode ray tube (CRT) display, etc. However, the present disclosure is not limited to these exemplary displays and may include other types of displays. Display device 160 may be used by navigation and mapping system 130 installed on user computing device 100 to display information related to an input to the user (e.g., information related to a location of interest to the user, a user interface screen with user-selectable interface elements, etc.). The navigation information may include, but is not limited to, a map of the geographic area, an immersive view of the location (e.g., a three-dimensional immersive view, a fly-through immersive view of the location, etc.), the location of the user computing device 100 in the geographic area, a route through the geographic area specified on the map, one or more navigation directions (e.g., turn-by-turn directions through the geographic area), travel time for the route through the geographic area (e.g., travel time from the location of the user computing device 100 to a POI), and one or more points of interest within the geographic area.

[0076] User computing device 100 may include output device(s) 170 for providing output to the user, such as one or more of an audio device (e.g., one or more speakers), a haptic device (e.g., a vibration device) that provides haptic feedback to the user, a light source (e.g., one or more light sources such as LEDs that provide visual feedback to the user), a thermal feedback system, etc. According to various examples of the present disclosure, output device 170 may include a speaker that outputs a sound associated with a location in response to a user requesting an immersive view of the location.

[0077] User computing device 100 may include capture device 180 capable of capturing media content, according to various examples of this disclosure. For example, capture device 180 may include image capture device 182 (e.g., a camera) configured to capture images (e.g., photos, videos, etc.) of a location. For example, capture device 180 may include sound capture device 184 (e.g., a microphone) configured to capture sound or audio (e.g., audio recordings) of a location. Media content captured by capture device 180 may be transmitted to one or more of server computing system 300, user-generated content data store 350, POI data store 370, navigation data store 380, and user data store 390, e.g., via network 400. For example, in some implementations, imagery is used to generate a 3D scene, and in some implementations, media content may be integrated with an existing 3D scene.

[0078] According to example embodiments described herein, the server computing system 300 may include one or more processors 322 and one or more memory devices 320 as previously described. The server computing system 300 may include a navigation and mapping system 330.

[0079] For example, navigation and mapping system 330 may include immersive view application 332, which performs functions similar to those described above with respect to immersive view application 132. Navigation and mapping system 330 may include generative map application 334, which performs functions similar to those described above with respect to generative map application 134.

[0080] For example, the navigation and mapping system 330 may include a three-dimensional representation generator 336 configured to generate a 3D representation based on multiple images of a location (e.g., the interior of a restaurant, park, etc.). Multiple images may be captured and combined using known methods to create a 3D scene of the location. For example, a neural radiance field method may be used to generate a three-dimensional representation of a location based on the multiple images. In some implementations, methods including structure from motion algorithms may be used to estimate three-dimensional structure. In some implementations, machine learning resources may be implemented to generate camera-like images from any viewpoint within the location based on the captured images. For example, a video fly-through of the location may be generated by the three-dimensional representation generator 336 based on the captured images. In some implementations, the initial three-dimensional representation generated by the three-dimensional representation generator 336 may be a static 3D representation without variable or dynamic objects (e.g., moving objects). For example, an initial 3D scene of a park may include an image of the park including images of trees, playground equipment, picnic tables, etc., but without images of people, dogs, or other moving objects.

[0081] For example, the navigation and mapping system 330 may include a representation modification system 338 configured to integrate user-generated content from the user-generated content data store 350 with the three-dimensional representations obtained from the three-dimensional representation generator 336. Additionally, the three-dimensional representations stored in the three-dimensional representation data store (e.g., the three-dimensional representation data store 340 of FIG. 3 ) may be categorized or classified according to time of day, season, weather conditions, and lighting conditions. The three-dimensional representation generated based on multiple images of a location may be integrated with media content using known methods to create an integrated three-dimensional representation of the location. For example, the representation modification system 338 may be configured to select appropriate user-generated content (e.g., one or more dynamic objects) for a particular location at a particular time or date.

[0082] 3 illustrates an exemplary system for integrating user-generated content into a three-dimensional representation of a location, according to an exemplary embodiment of the present disclosure. Representation modification system 338 includes receiving system 302, accessing system 304, media access system 306, selection system 308, modification system 310, display system 312, three-dimensional representation data store 340, and user-generated media data store 342.

[0083] The receiving system 302 can receive a request from a user. The request can be associated with a particular location. For example, a user can interact with a mapping application to view information about a particular location. In some examples, the request information can include a three-dimensional representation of the location generated using neural radiance field techniques. For example, a user can select a three-dimensional representation interface element on a particular location or building. Based on this interaction, a request can be generated. The request can be transmitted from the user computing device to the expression modification system 338 (or a server system associated with the expression modification system). The receiving system 302 can transmit the request to the accessing system 304.

[0084] The access system 304 can determine a particular three-dimensional representation of a location of interest to the user based on the request. For example, the request may include information identifying a particular location, building, or entity for which a three-dimensional location is requested. The access system 304 can access the three-dimensional representation data store 340. The three-dimensional representation data store 340 can store multiple three-dimensional representations for multiple different locations. In some examples, each three-dimensional representation is associated with a particular location and is generated based on previously captured media data. For example, a series of images captured by a photographer at a location can be used to generate a three-dimensional representation of the location using neural radiance field techniques.

[0085] The access system 304 can use information from the query to determine an appropriate three-dimensional representation to retrieve from the three-dimensional representation data store 340. For example, the query can have a location identifier based on a user's interaction with a navigation or mapping application. The location identifier can identify a particular location that the user is interested in viewing. In some examples, the user's query can also include information regarding times and / or dates of interest to the user. The three-dimensional representation data store 340 can include multiple three-dimensional representations for each location representing different times, dates, or situations.

[0086] Once the accessing system 304 accesses the correct three-dimensional representation from the location three-dimensional representation data store 340, the accessing system 304 can transmit the selected three-dimensional representation to the media access system 306. The media access system 306 can determine appropriate user-generated media content for the selected three-dimensional representation. In some examples, the media access system 306 can determine which piece of user-generated media content to access based on the location associated with the three-dimensional representation.

[0087] The selection system 308 can determine which pieces of user-generated media content to insert into the three-dimensional representation of the location. In some examples, the selection system 308 can determine the number of pieces of user-generated media content to insert. The number can be based on the length of a path through the user-generated media content and a target density of the pieces of user-generated media content. In some examples, the three-dimensional representation of the location includes a predetermined path through the location. For example, the predetermined path can trace the path used by the photographer who captured the initial issue image from which the three-dimensional representation is generated. In other examples, the user can use interactive controls to freely move through the location as a target. In this situation, the number of pieces of user-generated media content to insert can be based on the total size of the area and the target density.

[0088] In some examples, rather than determining the total amount of media content to insert, the selection system 308 can determine specific locations where media content is needed. This determination can be based on how much user-generated media content contributes to the user's understanding of a particular area of ​​the three-dimensional representation. For example, an area with tables and chairs can be augmented with images of users eating or enjoying themselves in that area. Other areas, such as the hallway leading to the restroom, may not require any user-generated media content because such content would not significantly improve the user's understanding of that location.

[0089] Once the selection system determines one or more locations where user-generated content needs to be inserted, the selection system can determine the type of social media-generated content to insert, or the subject of the user-generated content. In some examples, the subject or type of user-generated media content can be determined based on the location. For example, a restaurant or nightclub can be associated with a particular atmosphere or environment, and the selection system 308 can determine whether to include pieces of user-generated content associated with the particular atmosphere or environment. In other examples, pieces of user-generated content can be selected based on the time and / or date associated with the three-dimensional representation. For example, if the associated day is a holiday, the selection system 308 can select pieces of user-generated content associated with the holiday. Similarly, if the time is daytime, the selection system 308 can prioritize pieces of user-generated media content associated with daytime activities.

[0090] Once the selection system 308 determines the location and type of content, the selection system can select the highest-rated piece of user-generated content that meets certain criteria. For example, a third-party system can rate pieces of user content based on how well the pieces of user content match a particular topic subject, based on user feedback such as high ratings or comments, or based on other metrics. The selection system 308 can determine the metrics and select the best pieces of user-generated content based on the metrics. In some examples, the selection system determines multiple locations within the three-dimensional representation where pieces of user-generated media content will be inserted. The selection system 308 can select the piece of user-generated content associated with the location closest to where the user-generated content is displayed.

[0091] For example, the selection system 308 can modify the user-generated media content by cropping the user-generated media content, by editing details, or by providing one or more animation effects to the media content. In some examples, portions of an image or video can be associated with a particular subject or topic, while other portions of the image cannot. The selection system 308 can edit or crop sections of the media content that are not associated with the subject that the selection system 308 has determined should be displayed.

[0092] Once the selection system 308 selects one or more pieces of user-generated media content, the selected pieces of user-generated content can be transmitted to the modification system 310. The representation modification system can modify the three-dimensional representation of the location to include one or more visual pop-ups that display the selected pieces of user-generated content at a predetermined location. The visual pop-outs can be a part of the user interface that is distinct from other portions of the three-dimensional representation. For example, the visual pop-out can be a two-dimensional image surrounded by a white border to offset the two-dimensional image from other portions of the three-dimensional representation. In some examples, the visual pop-out can be associated with a particular portion of the location shown in the three-dimensional representation. For example, the visual pop-out can have a visual tail or route that connects the two-dimensional image to a particular location in the three-dimensional representation.

[0093] A visual pop-out can be a two-dimensional element of a user interface (e.g., a window) that displays user-generated media content within a three-dimensional representation. For example, a visual pop-out can include a stem extending from a location and a white border around the piece of user-generated content that distinguishes the visual pop-out from the surrounding content. Other designs can be used to display user-generated content within a 3D representation.

[0094] In some examples, the visual pop-outs may be selectable. Thus, when viewing the three-dimensional representation, a user can select (e.g., click or touch) a visual pop-out associated with particular user-generated media content. When a particular visual pop-up is selected, the user interface may be updated to provide an enlarged or more detailed view of the user-generated media content.

[0095] Once the modification system 310 has inserted the user-generated media content into one or more visual pop-ups, the visual representation can be transmitted to the user computing device by a transmission system. In some examples, the three-dimensional representation includes the user-generated content and the visual pop-up that can be displayed to the user.

[0096] FIG. 4 illustrates a user interface screen of a mapping application according to one or more exemplary embodiments of the present disclosure. In FIG. 4, user interface screen 402 indicates that a user of user computing device 100 is exploring a location in Westminster, specifically a building containing Cinnamon Club 410, and icon 420 indicates that Cinnamon Club 410 includes a restaurant. For example, user interface element 430 may enable the user to obtain an immersive view of the location. For example, user interface element 430 may be in the form of a symbol or selectable object overlaid on the location to indicate to the user that an immersive view of the location can be obtained. For example, in FIG. 4, user interface element 430 is a white circle.

[0097] 5A-5B illustrate an exemplary immersive three-dimensional representation 510 of a location, according to one or more exemplary embodiments of the present disclosure. In this example, a user interface 502 displays a particular view from a particular portion of a road within the three-dimensional representation. In some examples, the three-dimensional representation includes a particular viewpoint object that moves through the three-dimensional location based on a predefined path. In other examples, a user can use controls to direct how the viewpoint object moves through the three-dimensional representation. In some examples, if a predefined path has been established, the path may be determined based on a path taken by a person who collected the images used to generate the three-dimensional representation, for example, using neural radiance field techniques.

[0098] In some examples, the user interface can start with the view (504) shown in Figure 5A and move seamlessly to a second view. In this way, the viewpoint object can move smoothly and uninterrupted through space, and the quality and type of views available maintain the same level of quality and fidelity.

[0099] Figure 5B represents a second snapshot of the view 506 available within the three-dimensional representation of the location. For example, a user has controlled the viewpoint object to move from a position resulting in the view shown in Figure 5A to a position resulting in the view shown at 506A in Figure 5B. Although not shown, multiple views exist between the views shown in Figures 5A and 5B as the viewpoint object moves along a path defined pre-defined or by user input.

[0100] 6A-6C illustrate exemplary immersive three-dimensional representations of a location 600 with inserted visual pop-outs, according to one or more exemplary embodiments of the present disclosure. FIG. 6A illustrates a user interface 602 displaying an initial viewpoint 604 of the three-dimensional representation, including permanent portions of the location (e.g., not movable aspects such as people) and multiple visual pop-outs. For example, visual pop-out 606 is a visual pop-out that provides additional context for the location, including dynamic (e.g., non-permanent) aspects of the location. Dynamic feature aspects may include people, services such as food, and other aspects of the location's mood or atmosphere. Visual pop-outs, including visual pop-out 606, can display user-generated media content that is provided to a representation modification system (e.g., representation modification system 338 of FIG. 3 ) that generates the three-dimensional representation of the location. For example, a user generating this content may specify specific pieces of user-generated media content that are available when generating the three-dimensional representation.

[0101] Figure 6B shows a second viewpoint 610 of the three-dimensional representation. To get from the viewpoint displayed in Figure 6A to the viewpoint displayed in Figure 6B, the camera or viewpoint object moves along a path that ensures that the 3D representation is always clearly visible. In some examples, the path from the position of the viewpoint object in Figure 6A to the position of the viewpoint object in Figure 6B is predefined within the three-dimensional representation. For example, the three-dimensional representation can be generated using one or more fixed or predetermined paths through the three-dimensional representation. In other examples, a user can control the position of the viewpoint object or camera to move it to any point within the 3D representation.

[0102] 6B also includes several additional visual pop-outs. As can be seen, some of the visual pop-outs, including visual pop-up 612, include user-generated media content positioned near the location where the user-generated media was captured. For example, visual pop-out 612 shows a woman sitting at a table eating a meal, and visual pop-out 612 is positioned near the table where the woman is pictured in the visual media content. In some examples, the particular user-generated media content shown in each pop-out can be determined by location, subject, and overall quality rating.

[0103] FIG. 6C illustrates a piece of user-generated media content 622 that is displayed in an updated interface 620 upon selection by a user, according to one or more exemplary embodiments of the present disclosure. In some examples, a user can interact with a particular piece of user-generated content displayed in a visual pop-up. For example, a user can tap or select a particular visual pop-up. In response, the user interface can update to display the piece of user-generated media content displayed in the visual pop-out in a larger format or in more detail. For example, a user selects or interacts with user-generated content in a visual pop-out (e.g., visual pop-out 612 displayed in FIG. 6B). As a result, the user interface 620 is updated to display the piece of user-generated content in a larger format for closer inspection. In some examples, the user-generated content displayed in a visual pop-out may be edited or cropped to fit the format of the visual pop-out. In that case, selecting the visual pop-out may display the entire piece of user-generated content.

[0104] 7A displays an example of a three-dimensional representation 700 of a location into which user-generated media content has been seamlessly integrated, according to some embodiments of the present disclosure. In this example, the three-dimensional representation includes non-permanent portions of the location. For example, the three-dimensional representation is displayed in a user interface 702 of an application. The displayed portion of the three-dimensional representation includes a bartender. To achieve this effect, an expression modification system (e.g., expression modification system 338 of FIG. 3 ) can identify specific pieces of user-generated content associated with the specific location represented within the three-dimensional representation and a specific subject theme or ambiance target. In this example, the employees at this location may portray an ambiance of being friendly, cool, and helpful.

[0105] As can be seen from the illustration, the pieces of user-generated media content are seamlessly inserted into the three-dimensional representation of the location, rather than being inserted into a visual pop-out. In some instances, a particular piece of user-generated media content is inserted only with the permission of the user depicted therein and the user who generated the piece of user-generated media content. In some instances, the seamlessly integrated piece of user-generated media content may only be displayed from a particular angle or along a particular route through the three-dimensional user-generated content. This is because there is insufficient information about the piece of user-generated content to reliably generate a complete three-dimensional representation of the user-generated content.

[0106] FIG. 7B displays a piece of user-generated media content 720 used to generate the 3D representation, as seen in FIG. 7A. In this example, the user-generated media content is an image of a bartender at a bar in the location associated with the three-dimensional representation shown in FIG. 7A. As can be seen, the image is a single image and, therefore, may not be suitable for full integration into the three-dimensional representation as a fully realized three-dimensional model. Instead, pieces of user-generated media content may be integrated from particular angles, along particular routes, or in a way that is seamless in effect, although such seamless integration may not be possible from all potential locations within the three-dimensional representation. Thus, when a user selects a 3D representation of a location to display, the system can determine the particular path the user will take and determine whether any user-generated media content can be integrated into the three-dimensional representation. If so, the system can determine whether the piece of user-generated media content will be properly integrated into the scene along the route for the user.

[0107] If so, the representation modification system can modify the three-dimensional representation to include the user's pieces of user-generated media content, so that it appears as a seamless piece of the three-dimensional representation while the user is progressing along a predetermined path. In some examples, there is enough data in the user-generated media content (or some pieces of user-generated media content) that the user-generated media content can be fully integrated into the three-dimensional representation sensor and viewed from all potential angles along all potential paths.

[0108] FIG. 8 illustrates an exemplary flow diagram of a method for integrating user-generated media content into a three-dimensional representation of a location, according to an exemplary embodiment of the present disclosure. One or more portions of the method may be implemented by one or more computing devices, such as, for example, the computing devices described herein. Additionally, one or more portions of the method may be implemented as an algorithm on a hardware component of the device(s) described herein. FIG. 8 illustrates elements in a particular order for purposes of illustration and explanation. Those skilled in the art will appreciate, using the disclosure provided herein, that elements of any of the methods described herein may be adapted, rearranged, extended, omitted, combined, and / or modified in various manners without departing from the scope of the present disclosure. The method may be implemented by one or more computing devices, such as one or more computing devices shown in FIGS. 1, 2, and 3.

[0109] A computing system (e.g., the user computing device 102 of FIG. 1 ) may include one or more processors, memory, and one or more communication systems. The one or more communication systems enable the computing system to transmit data to other computing systems over a communication network. The user computing device 102 (e.g., the user computing device 102 of FIG. 1 ) may include other components. These components, in conjunction with one another, enable the user computing device 102 (e.g., the user computing device 102 of FIG. 1 ) to obtain a three-dimensional representation of a location, at 802, where the representation is generated based on multiple images. In some examples, the three-dimensional representation may be a virtual representation of a physical location.

[0110] The virtual representation may include a virtual camera that simulates a person moving through a space. The virtual camera may have a location and an orientation. Portions of the three-dimensional representation captured by the virtual camera (e.g., based on its location and orientation) may be displayed to a user viewing the three-dimensional representation of the location. The virtual camera (or other display mechanism) may move through the three-dimensional representation, and the portions of the three-dimensional representation that are displayed may transition seamlessly as the virtual camera moves. The movement of the virtual camera may be based on a path or path information.

[0111] The expression modification system can access user-generated media content associated with the location at 804. The media content includes user-generated media content captured by one or more users. In some examples, the user-generated media content includes at least one of user-generated visual content, user-generated audio content, or user-generated text content. In some examples, a user can make media content available to the expression modification system for use within the augmented three-dimensional representation of the location. As a policy, the expression modification system can access only user-generated media content that is explicitly made available for use in this manner.

[0112] The representation modification system can receive 806 path information representing at least a portion of a path through the three-dimensional representation of the location. In some examples, the path information can be determined based on a user's path as the user navigates the three-dimensional representation of the location. For example, a user can walk through a location while periodically capturing images of the location. The three-dimensional representation of the location can be generated by accessing a series of two-dimensional images of the location, the images captured by a camera (held by a user) that moves through space and periodically captures one or more two-dimensional images. The representation generation system can provide the series of images to a machine-learned model trained to generate the three-dimensional representation as output.

[0113] In some examples, the received path information can be generated based on the path of a camera moving through space to capture one or more two-dimensional images and provided to the representation modification system. The path information can be pre-generated to follow the path of a user who captured the series of two-dimensional images used to generate the three-dimensional representation of the location. In some examples, the user may move multiple distinct paths through the location while capturing the two-dimensional images. The three-dimensional representation can move through any of multiple predetermined paths through the three-dimensional representation based on the path of the user who captured the images.

[0114] In some examples, the path information is generated based on inputs from a user to manipulate a virtual camera within the three-dimensional representation of the location so that different portions of the three-dimensional representation are seamlessly visible. For example, a user viewing the three-dimensional representation of the location is provided with controls (e.g., touch input controls or other controls) that enable the user to specify a direction (e.g., forward, backward, left, right, etc.) in which to move the virtual camera. In this manner, the user can determine a path through the three-dimensional representation of the location in real time, as desired. Each input from the user can be provided to the representation modification system as path information.

[0115] One or more pieces of user-generated media content include imagery of a location that includes one or more real-world dynamic objects. For example, the images may include users, movable or non-permanent objects, services offered at the location (e.g., food, beverages, or other services), temporary decorations or conditions, time-specific features of the location, etc.

[0116] At 808, the representation modification system can select one or more pieces of user-generated media content based on a path through the three-dimensional representation of the location. In some examples, the representation modification system can determine one or more categories of content to be displayed in the three-dimensional representation. The representation modification system can select one or more pieces of user-generated media based on the one or more content categories.

[0117] The representation modification system can determine a target density of pop-out images along a path through the environment. The representation modification system can select a number of images based on the length of the path and the target density. In some examples, the representation modification system can determine an associated location within the three-dimensional representation for each candidate image of a plurality of candidate images.

[0118] In some examples, the representation modification system can receive a rating of the content of each candidate image. In some examples, the representation modification system can determine that a pop-out should be displayed at each position within the three-dimensional representation of the location. The representation modification system can select an image for each position from the candidate images based on the location associated with each candidate image and the rating of each candidate image. In some examples, the piece of user media content is selected based at least in part on a time associated with the user media.

[0119] In some examples, the representation modification system can integrate 810 one or more pieces of user-generated media content into a three-dimensional representation of a location along a path through the location, with the integrated pieces of user-generated media content being represented within a visual pop-out within the three-dimensional representation.

[0120] In some examples, the representation modification system can provide 812 an integrated 3D scene of the location to represent the state of the location based on the temporal association of the media content with the location.

[0121] When general terms including "module" and "unit" are used herein, these terms may refer to software or hardware components or devices, such as, but not limited to, a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC), that perform a specific task. A module or unit may be configured to reside on an addressable storage medium and to execute on one or more processors. Thus, a module or unit may include multiple components, including, by way of example, software components, object-oriented software components, class and task components, processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided in the components and modules / units may be combined into fewer components and modules / units or further divided into additional components and modules.

[0122] Aspects of the above-described exemplary embodiments may be recorded on a non-transitory computer-readable medium containing program instructions for performing various computer-implemented operations. The medium may also include, alone or in combination, program instructions, data files, data structures, etc. Examples of non-transitory computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM disks, Blue-Ray disks, and DVDs; magneto-optical media such as optical disks; and other hardware devices specially configured to store and execute program instructions, such as semiconductor memory, read-only memory (ROM), random access memory (RAM), flash memory, USB memory, etc. Examples of program instructions include both machine code, such as that produced by a compiler, and files containing higher-level code that can be executed by a computer using an interpreter. The program instructions may be executed by one or more processors. The described hardware devices may be configured to function as one or more software modules to perform the operations of the above-described embodiments, or vice versa. Furthermore, the non-transitory computer-readable storage medium may be distributed among computer systems connected via a network, and the computer-readable code or program instructions may be stored and executed in a decentralized manner. Furthermore, the non-transitory computer-readable storage medium may also be embodied in at least one application-specific integrated circuit (ASIC) or field-programmable gate array (FPGA).

[0123] Each block in the flowchart diagrams may represent a unit, module, segment, or portion of code that includes one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of order. For example, two blocks shown in succession may in fact be executed substantially (concurrently), or the blocks may sometimes be executed in the reverse order, depending on the functionality involved.

[0124] While the present disclosure has been described in terms of various exemplary embodiments, each example is provided for purposes of explanation and not limitation of the present disclosure. Those skilled in the art, once they arrive at the foregoing understanding, can readily produce modifications, variations, and equivalents of such embodiments. Accordingly, the present disclosure does not exclude inclusion of such modifications, variations, and / or additions to the disclosed subject matter as would be readily apparent to one skilled in the art. For example, features illustrated or described as part of one embodiment can be used with other embodiments to yield still other embodiments. Accordingly, the present disclosure is intended to cover such modifications, variations, and equivalents.

Claims

1. 1. A computer-implemented method comprising: obtaining a three-dimensional representation of a location, the three-dimensional representation being generated based on a plurality of images; accessing user-generated media content associated with the location; and receiving route information representing at least a portion of a route through the three-dimensional representation of the location; selecting one or more pieces of user-generated media content based at least in part on the path information; integrating one or more pieces of the user-generated media content into the three-dimensional representation of the location based on the route information and portions of the three-dimensional representation to be displayed to the user, wherein the pieces of user-generated media content are presented within one or more visual pop-outs within the three-dimensional representation; providing the three-dimensional representation of the location for display to a user; A method comprising:

2. The computer-implemented method of claim 1 , wherein the user-generated media content is captured by one or more users.

3. The computer-implemented method of claim 2 , wherein the user-generated media content includes at least one of user-generated visual content, user-generated audio content, and user-generated textual content.

4. 2. The computer-implemented method of claim 1, wherein the path information is generated based on input from a user for navigating a virtual camera through the three-dimensional representation of the location to seamlessly view different portions of the three-dimensional representation.

5. The three-dimensional representation is accessing a series of two-dimensional images, the two-dimensional images being captured by a camera moving through the location and periodically capturing one or more two-dimensional images of the location; providing the set of images to a machine-learned model trained to generate the three-dimensional representation as an output; 2. The computer-implemented method of claim 1, wherein the method is generated by:

6. 6. The computer-implemented method of claim 5, wherein the path information is generated based on a path of the camera as it moved through the location to capture the two-dimensional image of the location for use in generating the three-dimensional representation.

7. The computer-implemented method of claim 1 , wherein one or more pieces of user-generated media content include an image of the location that includes one or more real-world dynamic objects.

8. Selecting one or more pieces of user-generated media content based on the path information further comprises: determining one or more categories of content to be displayed in said three-dimensional representation; selecting one or more pieces of the user-generated media content based on one or more categories of the content; 10. The computer-implemented method of claim 1, comprising:

9. Selecting one or more pieces of user-generated media content based on the path information further comprises: determining a target density of visual pop-outs within the three-dimensional representation of the location; selecting a number of images based on the portion of the three-dimensional representation of the location to be displayed to a user; 10. The computer-implemented method of claim 1, comprising:

10. Selecting one or more pieces of user-generated media content based on the path information further comprises: determining that a visual pop-out should be displayed at each position within the three-dimensional representation of the location; determining, for each candidate piece of user-generated media content among a plurality of candidate pieces of user-generated media content, an associated location within the three-dimensional representation; receiving a content rating for each candidate piece of user-generated media content; selecting a piece of user-generated media content for the respective location from the candidate pieces of user-generated media content based on the location associated with each candidate piece of user-generated media content and the rating for each candidate piece of user-generated media; 10. The computer-implemented method of claim 1, comprising:

11. The computer-implemented method of claim 10 , wherein the piece of user-generated media content is selected based at least in part on a temporal relevance of the user-generated media content to the location.

12. The computer-implemented method of claim 11 , wherein the piece of user-generated media content is selected based at least in part on a time period associated with the piece of user-generated media content.

13. The computer-implemented method of claim 11 , wherein the piece of user-generated media content is selected based at least in part on a date associated with the piece of user-generated media content.

14. The computer-implemented method of claim 10 , further comprising accessing user preference data, wherein the piece of user-generated media content is selected based at least in part on the user preferences.

15. Each piece of user-generated media content among the plurality of pieces of user-generated media content has an associated media viewpoint, and selecting the piece of user-generated media content from the candidate pieces of user-generated media content for the respective position based on the location associated with each candidate piece of user-generated media content and the rating for each piece of user-generated media content image further includes: determining a user viewpoint associated with a portion of the three-dimensional representation of the location to be displayed; selecting one or more pieces of user-generated media content based at least in part on the associated media viewpoint of each piece of the user-generated media content and a user viewpoint associated with a portion of the three-dimensional representation of the location to be displayed; 11. The computer-implemented method of claim 10, comprising:

16. determining a semantic label associated with each of the positions within the three-dimensional representation of the location; selecting, for each position, a piece of user-generated media content from the candidate pieces of user-generated media content based at least in part on the semantic label associated with the respective position within the three-dimensional representation of the location; The computer-implemented method of claim 10 further comprising:

17. while displaying a portion of the three-dimensional representation of the location to the user; receiving user input indicating a selection of the piece of user-generated media content displayed in the visual pop-out; updating a user interface to display the selected piece of user-generated media content in greater detail; The computer-implemented method of claim 1 further comprising:

18. while displaying to the user a portion of the three-dimensional representation of the location determined based on a position and orientation of a virtual camera within the three-dimensional representation of the location; receiving user input indicating a selection of the piece of user-generated media content displayed in the visual pop-out; updating the orientation of the virtual camera within the three-dimensional representation of the location to provide additional detail of the selected piece of user-generated media content within a user interface; The computer-implemented method of claim 1 further comprising:

19. 1. A computing device comprising: An input device; A display device; at least one memory for storing instructions; at least one processor configured to execute the instructions to perform operations, the operations comprising: obtaining a three-dimensional representation of a location, the three-dimensional representation being generated based on a plurality of images; accessing user-generated media content associated with the location; and receiving route information representing at least a portion of a route through the three-dimensional representation of the location; selecting one or more pieces of user-generated media content based at least in part on the path information; integrating one or more pieces of the user-generated media content into the three-dimensional representation of the location based on the route information and portions of the three-dimensional representation to be displayed to the user, wherein the pieces of user-generated media content are presented within one or more visual pop-outs within the three-dimensional representation; providing the three-dimensional representation of the location for display to a user; a computing device,

20. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations including: obtaining a three-dimensional representation of a location, the three-dimensional representation being generated based on a plurality of images; accessing user-generated media content associated with the location; and receiving route information representing at least a portion of a route through the three-dimensional representation of the location; selecting one or more pieces of user-generated media content based at least in part on the path information; integrating one or more pieces of the user-generated media content into the three-dimensional representation of the location based on the route information and portions of the three-dimensional representation to be displayed to the user, wherein the pieces of user-generated media content are presented within one or more visual pop-outs within the three-dimensional representation; providing the three-dimensional representation of the location for display to a user; 1. One or more non-transitory computer-readable media, including:

Images