Location-specific 3D models in response to location queries

Abstract

Generating a location-specific three-dimensional model in response to a location query can provide a user with a better understanding of the location by providing better interactivity, better visibility, and better understanding of dimensionality. Generating the model can be made possible by leveraging a three-dimensional asset database and segmentation methods. The location-specific model can provide further utility by further including situation-specific simulated results, such as simulated weather or traffic.

Description

[Technical field]

[0001] Related Applications This application claims priority to and the benefit of U.S. Non-provisional Patent Application No. 17 / 227,871, filed April 12, 2021. U.S. Non-provisional Patent Application No. 17 / 227,871 is incorporated herein by reference in its entirety.

[0002] The present disclosure relates generally to providing a three-dimensional model of a location in response to a user query related to or requesting information about a location. More particularly, the present disclosure relates to providing location-specific segments of a three-dimensional model of a location in response to a query about a location that requests information associated with the location, such as, for example, a search query, a navigation query, a mapping query, a weather query, a review query, a visual query, and / or other user request for information about a location. [Background technology]

[0003] When searching for a location via a search engine, some existing systems may return results in the form of hyperlinks and / or generated images that may include text excerpts, photographs, or maps. For monuments and landmarks, these results often fail to portray the actual view and size of the monument or landmark. Furthermore, while an image may provide a perspective of what the location looks like, an image alone may fail to capture the dimensionality of the location. Furthermore, videos of the location may also be limited for a variety of reasons. In particular, search results lack interactivity, which may make it difficult to find other aspects of the location (e.g., the context of the location or different perspectives of the location). For example, trying to see different views close-up or to see different parts of the location may require tedious searching, rather than being provided in an intuitive, interactive interface.

[0004] Knowing the weather, traffic, and crowd conditions at monuments and landmarks can help tourists and other visitors better understand and prepare for visiting the monument. Maps alone fail to take into account traffic or crowd conditions. Furthermore, the visibility of a monument or landmark can be affected by crowd conditions or weather. Knowledge of the effects of these factors on visibility is often limited to sweeping literature on the object. However, many people consider themselves to be visual learners. Summary of the Invention [Means for solving the problem]

[0005] Aspects and advantages of embodiments of the disclosure will be set forth in part in the description that follows, or can be learned from the description, or may be learned by practice of the embodiments.

[0006] An exemplary aspect of the present disclosure is directed to a computer-implemented method for providing a three-dimensional modeling to a user. The method includes obtaining, by a computing system, which may include one or more computing devices, a location query associated with the user computing device. The method can include processing, by the computing system, the location query to determine a location. In some embodiments, the method can include accessing, by the computing system, a three-dimensional asset database to obtain a three-dimensional model that models the location in combination with one or more other distinct locations. The method can include segmenting, by the computing system, the three-dimensional model to generate a location-specific segment of the three-dimensional model that corresponds to the location. In some embodiments, the location-specific segment can model the location in isolation from one or more other distinct locations. The method can include providing, by the computing system, to the user computing device, the location-specific segment of the three-dimensional model or one or more renderings of the location-specific segment of the three-dimensional model.

[0007] As described in more detail below, segmenting the three-dimensional model to generate location-specific segments can provide a number of technical advantages. In particular, location-specific segments can be provided to a user computing device, rather than the three-dimensional model as a whole. Doing so can reduce bandwidth requirements for sending relevant data to the user device, and can reduce computational resource requirements at the user device for storing and rendering the location-specific model. In other words, the steps involved in segmenting the three-dimensional model can reduce data that needs to be transmitted to the user computing device, thereby enabling a more computationally efficient method for providing the three-dimensional modeling to the user.

[0008] Another exemplary aspect of the present disclosure is directed to a computing system. The computing system may include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include obtaining a location query from a user computing device. The operations may include processing the location to determine the location. In some embodiments, the operations may include accessing a three-dimensional asset database to obtain a three-dimensional model associated with the location. The operations may include generating a location-specific model by segmenting the three-dimensional model to separate the location from other objects in the three-dimensional model, and providing the location-specific model to the user computing device.

[0009] Another exemplary aspect of the present disclosure is directed to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations. The operations can include obtaining a user input specifying a location query for a location. The operations can include providing the location query to a server system over a network. In some embodiments, the operations can include receiving a location-specific segment of a three-dimensional model, or one or more renderings of the location-specific segment of the three-dimensional model, from the server system over the network. The three-dimensional model can model a location in combination with one or more other different locations, and the location-specific segment of the three-dimensional model can be generated by segmenting the three-dimensional model to isolate the location from one or more other different locations. The operations can include providing a display of at least a first rendering of the location-specific segment of the three-dimensional model.

[0010] Other aspects of the present disclosure are directed to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0011] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain relevant principles.

[0012] A detailed discussion of embodiments directed to those skilled in the art is set forth herein with reference to the accompanying figures. [Brief description of the drawings]

[0013] [Figure 1]FIG. 1 is a block diagram of an exemplary computing system for performing location-based model searching in accordance with an exemplary embodiment of the present disclosure. [Diagram 2] FIG. 1 is a block diagram of an exemplary model retrieval system according to an exemplary embodiment of the present disclosure. [Diagram 3] FIG. 1 is a block diagram of an exemplary context-aware model search system according to an exemplary embodiment of the present disclosure. [Figure 4] FIG. 2 is a block diagram of an exemplary three-dimensional rendering according to an exemplary embodiment of the present disclosure. [Diagram 5] 1 is an exemplary search results page in accordance with an exemplary embodiment of the present disclosure. [Figure 6] 1 is a flow diagram of an example method for performing location-based model search according to an example embodiment of the present disclosure. [Figure 7] 1 is a flow diagram of an example method for performing location-based model search according to an example embodiment of the present disclosure. [Figure 8] 1 is a flow diagram of an exemplary method for implementing creation of a three-dimensional model database according to an exemplary embodiment of the present disclosure. [Figure 9A] FIG. 2 illustrates an exemplary three-dimensional model according to an exemplary embodiment of the present disclosure. [Figure 9B] FIG. 2 illustrates an exemplary context-aware three-dimensional model in accordance with an exemplary embodiment of the present disclosure. [Figure 10] FIG. 1 illustrates an exemplary augmented reality experience according to an exemplary embodiment of the present disclosure. [Figure 11] FIG. 2 illustrates an exemplary rendering display according to an exemplary embodiment of the present disclosure. [Figure 12] FIG. 1 illustrates an exemplary route overlay according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Reference numerals repeated among the figures are intended to identify like features in the various embodiments.

[0015] overview In general, the present disclosure is directed to providing location-specific segments of a three-dimensional model of a location in response to a query about a location that seeks information associated with that location, such as, for example, a search query, a navigation query, a mapping query, a weather query, a review query, a visual query, and / or other user request for information. The systems and methods disclosed herein can utilize a database of three-dimensional models to provide location-based renderings and / or location-specific models in response to a query about a location ("location query"). In some embodiments, the systems and methods can include obtaining a location query. The location query can be obtained from or triggered by a user's computing device. The systems and methods can include processing the location query to determine the location. In some embodiments, the systems and methods can include accessing a three-dimensional asset database to obtain a three-dimensional model associated with the location. The three-dimensional model and the location can be used to generate a location-specific model. Generating the location-specific model can include segmenting the three-dimensional model to separate the location from other objects in the three-dimensional model. The systems and methods can include providing the location-specific model to a computing device. For example, the location-specific model can be provided to a user computing device. In some embodiments, the location-specific model may include architectural structures (eg, landmarks, monuments, and / or buildings).

[0016] Obtaining a location query may be facilitated by a user interface configured to receive input from a user. Alternatively and / or additionally, a location query may be a query generated and obtained by a computing system in response to an event triggered by a user computing device. For example, a user may scroll through a list of attractions and select a particular attraction to learn more about the attraction. In response to the selection, a location query for the attraction may be generated and obtained for processing. In other examples, a user may request to view the location on a map, request navigational directions to the location, request information about points of interest associated with the location (e.g., user-generated reviews), submit a textual or visual search query from which the location is obtained, and / or perform other actions indicating a request for information about the location. Each of these exemplary actions may trigger a location query for processing by the proposed system.

[0017] In some embodiments, a location query can include a set of terms or one or more images associated with a particular location. The terms can describe the location, can include the name of the location, can be the address of the location, or can be any other form of indirectly or directly related information that implies or results in the location being relevant. In some examples, a location query can include processing and determining where a photo was taken or what is depicted in the image.

[0018] After the location is determined, a three-dimensional asset database may be accessed and searched to determine whether that particular location has any associated three-dimensional models. The three-dimensional model may be a constructed three-dimensional model of a particular structure or structures found at that location. In particular, in some embodiments, the three-dimensional asset database may include a general or global three-dimensional model spanning a significant portion of the Earth's surface, or of an individual country or city located therein. For example, various algorithms may be applied to automatically generate one or more very large three-dimensional models spanning various addresses, streets, neighborhoods, cities, and / or the like from available imagery (e.g., satellite and / or street-level imagery).

[0019] To provide location-specific models, the system and method can isolate locations in a general three-dimensional model to generate the location-specific models. Specifically, an exemplary system of the present disclosure can segment portions of a general three-dimensional model into multiple segments. In some embodiments, the system and method can perform three-dimensional segmentation to extract location-specific segments of the three-dimensional model to model one or more specific locations. The location-specific models can be generated using image segmentation. In this manner, smaller models that include the required locations can be obtained. Smaller models can require less processing power to store and render, and can also require less bandwidth to transmit the models to a user's computing device. Furthermore, the location-specific models can be easier for users to access and understand than the general three-dimensional models. Thus, the location-specific models can enable improved interaction for users.

[0020] After the location-specific segments of the model are generated, the location-specific segments of the three-dimensional model may be rendered and displayed to a user in a variety of situations. As an example, the server computing system may transmit or provide the location-specific segments of the three-dimensional model to a user computing device, and the user computing device may show a rendering of the location with the location-specific model. Alternatively and / or in addition, the server computing system may perform a rendering of the location-specific segments of the three-dimensional model and transmit or provide the rendering to the user computing device.

[0021] In some embodiments, location-specific segments of the three-dimensional model can be generated by image segmentation using multiple images associated with the location. Further, the multiple images can include one or more aerial images. Alternatively and / or in addition, in some embodiments, the model can be generated using Light Detection and Ranging (LiDAR) data or other sensor data (e.g., spectral sensor data generated using a spectral sensor that can be used to segment trees or other objects, etc.). In particular, the LiDAR data or other sensor data can be used to determine what to segment.

[0022] The location-specific segments of the three-dimensional model may be provided in a variety of ways, via a variety of media. In some embodiments, the location-specific models may be provided as augmented reality assets rendered in the user's environment via an augmented reality experience. Alternatively and / or in addition, the location-specific models may be provided in a virtual reality experience. In some embodiments, the user computing device may include an interactive virtual environment that may include the location-specific models. Additionally, the interactive environment may include one or more two-dimensional or three-dimensional maps. The location-specific segments of the three-dimensional model may be provided in a pre-rendered video, or a set of pre-rendered videos, where each video may depict a different view of the location generated based on the location-specific segments.

[0023] In some embodiments, the location-specific model may comprise other information or data related to or associated with the location. For example, the system and method may include obtaining an address for the location and providing the address of the location with a location-specific model. In some embodiments, the system and method may include obtaining location information data. The location information data may include information related to the location (e.g., information about the history of the location). The location information data may be provided to the user computing device with the location-specific model. In some embodiments, the location-specific model may comprise a plurality of images of the location. Further, in some embodiments, the system and method may include generating a map including the location-specific model and providing the map to the user computing device. Further, in some embodiments, the system and method may include obtaining location data for the user computing device. The location data may be descriptive of the user's location. The system and method may include obtaining an address for a location associated with the location-specific model. The address and the user's location may be used to determine a route the user will use to get from the user's location to the address. A visual route overlay may be generated based on the route, and the visual route overlay and the map may be used to generate an augmented map with the visual route overlay overlaid on the map. The augmented map may then be provided to a user computing device, which has the technical effect of providing an efficient and effective means for navigating to the location.

[0024] Alternatively and / or in addition, the system and method may include obtaining situation data. The situation data may represent time-specific situations associated with the location. The system and method may include processing the situation data to determine each simulation asset and obtaining each simulation asset. Each simulation asset and location-specific model may be used to generate a simulated event including time-specific situations occurring at the location. In some embodiments, the situation may be weather conditions, as well as the location of various physical items such as trains, airplanes, buses, taxis, and / or the like, crowding at or near the location, and / or traffic. For example, the time-specific situation may be rain, and the generated simulated event may be a rendering of a location-specific segment of a three-dimensional model with simulated rain. More generally, a simulated event including time-specific situations occurring at the location may provide technical effects beyond the embodiment of the simulation itself. Specifically, the simulated event may enable a user to visualize the location in various time-specific situations, allowing the user to gain a better understanding of the location, thereby enabling the user to navigate to the location themselves in real life.

[0025] In some embodiments, providing the location-specific model may include providing the location-specific model with one or more search results via a web page (e.g., a web page returned in response to a textual or visual location query, etc.). Alternatively and / or in addition, the location-specific model may be provided and displayed in a section or location on the web page separate from other search results. For example, in some embodiments, the location-specific model may be provided in a separate panel along with other data at the location associated with the model (e.g., the name of the location associated with the location-specific model, the address of the location, and / or information about the location, etc.). In some embodiments, mesh segmentation may allow for compact responses to queries. In particular, segments of the mesh may be rendered more quickly (e.g., in a panel) than rendering the entire region. Rendering the segmentation may thus aid in providing search results without the heavy strain on bandwidth that rendering a full model may cause.

[0026] The systems and methods disclosed herein can be utilized in various fields and industries. For example, the systems and methods disclosed herein can be utilized to provide an improved user experience for map applications. More specifically, the systems and methods disclosed herein can be used to provide a user with a map having a location-specific model in response to a location query. The map application can provide a visual route overlay to provide a realistic rendering of the area to be navigated through. In some embodiments, the map application can provide several options to understand an area. The map application can provide a street view, an overhead map, an aerial view map, or an interactive three-dimensional model of the location. In this manner, the map application can provide the user with a more efficient way to navigate to the location.

[0027] Another example can include implementing the systems and methods into video games and / or augmented or virtual reality experiences. For example, the systems and methods disclosed herein can enable the creation of open-world games, where a user can travel to various areas and, upon reaching an area, can send a location query and receive a location-specific model of the area in response. More specifically, the game can be set in France and, when a character travels to a location, can send a location query for that location to a server and receive a location-specific model of the location in return (e.g., a character can travel to the Eiffel Tower and can send a location query for the Eiffel Tower and receive a location-specific model of the Eiffel Tower in return).

[0028] The three-dimensional asset database utilized by the systems and methods disclosed herein may be generated using a variety of methods or processes. The process may begin with the creation of three-dimensional content. The three-dimensional environment may be constructed by common three-dimensional reconstruction methods. The reconstruction may generate a three-dimensional mesh with attached texture (e.g., a two-dimensional image representing the color of the mesh, etc.). The next step of the process may include image segmentation. The segmentation may be used to understand the mesh and texture to find the underlying semantic structure of the environment (e.g., the semantic structure of a reconstructed city, etc.). In some embodiments, the process may utilize semantic diversity for image segmentation. Image segmentation may include using machine learning models to segment multiple two-dimensional images into portions. The segmented images may then be projected into a three-dimensional mesh. The process may further include cleaning the generated three-dimensional environment by removing blobs and cleaning the mesh. For example, trees, streets, sidewalks, and other objects may be removed. The cleaning step may be completed manually or may be automated. The resulting 3D assets can be compressed and stored in a database. The process can be repeated iteratively to update the database with the latest structures from cities, towns, etc.

[0029] The systems and methods of the present disclosure provide several technical advantages and benefits. As an example, the systems and methods can provide location-specific segments of a three-dimensional model in response to a location query. The systems and methods can further be used to provide models of locations (e.g., landmarks or monuments, etc.) with real-time traffic, crowding, weather, and / or other context or situation-specific information. Additionally, the systems and methods can enable the three-dimensional model for map applications.

[0030] Another technical benefit of the systems and methods of the present disclosure is the dimensionally accurate modeling and interactivity of the renderings provided. The models can provide a dimensionally accurate representation of a location, allowing potential visitors to decide whether they want to actually visit the location. Furthermore, the dimensionally accurate representation allows users to more effectively navigate to the location.

[0031] Another exemplary technical effect and benefit relates to improved computational efficiency in computing systems, and increased functionality. For example, some existing systems allow a user to view a rendering of a global or general three-dimensional model. Processing and rendering the entire global or general three-dimensional model requires significant computational resources. In contrast, by extracting and then rendering location-specific segments of the three-dimensional model, the proposed systems and methods can save computational resources, such as processor usage, memory usage, and / or network bandwidth. In particular, the location-specific segments can require less bandwidth to be transmitted to a user device compared to transmitting a general three-dimensional model. Furthermore, the location-specific segments can require less memory usage in the user device, resulting in less processor usage to render the location-specific segments.

[0032] Referring now to the figures, exemplary embodiments of the present disclosure will be discussed in further detail.

[0033] Exemplary Devices and Systems 1 illustrates an example computing system 100 that may be used to implement location-based model searching according to aspects of the present disclosure. The system 100 has a client-server architecture that includes a server 110 that communicates with one or more client devices 130 over a network 160. However, the present disclosure may be implemented using other suitable architectures, such as a single computing device that is not connected to a network.

[0034] The system 100 includes a server 110, such as, for example, a web server. The server 110 may be one or more computing devices implemented as a parallel or distributed computing system. In particular, multiple computing devices may operate together as a single server 110. The server 110 may have one or more processors 112 and memory 114. The server 110 may also include a network interface used to communicate with one or more remote computing devices (e.g., client devices) 130 over a network 160.

[0035] The processor 112 may be any suitable processing device, such as a microprocessor, a microcontroller, an integrated circuit, or other suitable processing device. The memory 114 may include any suitable computing system or medium, including but not limited to a non-transitory computer readable medium, a RAM, a ROM, a hard drive, a flash drive, or other memory device. The memory 114 may store information accessible by the processor 112, including instructions 116 that may be executed by the processor 112. The instructions 116 may be any set of instructions that, when executed by the processor 112, causes the processor 112 to provide a desired functionality.

[0036] In particular, the instructions 116 may be executed by the processor 112 to perform a three-dimensional model search 120. The three-dimensional model search 120 may be configured to search for a three-dimensional model associated with a location and to segment and output a location-specific three-dimensional model. In some embodiments, the three-dimensional model search 120 may be configured to access a three-dimensional asset database 180 to obtain a three-dimensional model, and the three-dimensional model may be segmented using an image segmentation element 122 stored in the memory 114 of the server computing system 110.

[0037] It will be understood that the term "element" can refer to computer logic used to provide a desired functionality. Thus, any element, function, and / or instruction can be implemented in hardware, application specific circuitry, firmware, and / or software controlling a general purpose processor. In one embodiment, an element or function can be provided from a computer program product, which is a program code file stored in a storage device, loaded into memory, and executed by a processor, or which is, for example, computer executable instructions stored in a tangible computer readable storage medium, such as a RAM, a hard disk, or an optical or magnetic medium.

[0038] The memory 114 may also include geographic data 118, which may be obtained, manipulated, created, or stored by the processor 112. The geographic data 118 may include geographic imagery (e.g., digital maps, satellite imagery, aerial photographs, street-level photographs, synthetic models, etc.), tables, vector data (e.g., vector representations of roads, parcels, buildings, etc.), point of interest data (e.g., provinces such as islands, cities, restaurants, hospitals, parks, hotels, schools, etc.), or other suitable geospatial data or related information. By way of example, the geographic data 118 may be used to access information and data associated with locations and to generate renderings of a three-dimensional model of the Earth's surface.

[0039] The geographic data 118 may be stored in one or more databases. The one or more databases may be connected to the server 110 by a high bandwidth LAN or WAN, or may also be connected to the server 110 through a network 160. The one or more databases may be split so as to be located locally in multiple locations.

[0040] The server 110 can exchange data with one or more client devices 130 over a network 160. Although two clients 130 are shown in FIG. 1, any number of client devices 130 can be connected to the server 110 over the network 160. The client devices 130 can be any suitable type of computing device, such as a general purpose computer, a special purpose computer, a navigation device, a laptop, a desktop, an integrated circuit, a mobile device, a smartphone, a tablet, a wearable computing device, a display coupled with one or more processors and / or embedded, or other suitable computing device. Additionally, the client devices 130 can be multiple computing devices that operate together to perform an operation or computing action.

[0041] Similar to the server 110, the client device 130 may include a processor 132 and a memory 134. The memory 134 may store information accessible by the processor 132, including instructions and data executable by the processor. By way of example, the memory 134 may store a browser component 140 and an application component 142.

[0042] The browser component 140 can provide instructions for implementing a browser. In particular, a user of the client device 130 can use the browser to exchange data with the server 110 to visit a website accessible at a particular web address. The 3D model search of the present disclosure can be provided as an element of the website's user interface.

[0043] The application element 142 can provide instructions for running specialized applications on the client device 130. In particular, the specialized applications can be used to exchange data with the server 110 over the network 160. The application element 142 can include client device readable code for providing and implementing aspects of the present disclosure. For example, the application element 142 can provide instructions for implementing a mapping application, or a virtual earth application.

[0044] Client device 130 may include various user input devices 150 for receiving information from a user, such as a touch screen, a touch pad, data entry keys, a speaker, a mouse, a motion sensor, and / or a microphone suitable for voice recognition. Additionally, client device 130 may have a display 146 for presenting information, such as a rendering of a three-dimensional model in a three-dimensional environment.

[0045] The client device 130 may also include a positioning system 148 that may be used to determine the location of the client device 130. The positioning system 148 may optionally be used by a user to monitor the user's location relative to the rendering. The positioning system 148 may be any device or circuit for monitoring the location of the client device 130. For example, the positioning device 148 may determine an actual or relative location using a satellite navigation positioning system (e.g., a GPS system, a Galileo positioning system, a Global Navigation Satellite System (GLONASS), a Beidou satellite navigation and positioning system, etc.), an inertial navigation system, a dead reckoning system, based on an IP address, by using triangulation and / or proximity to cellular towers or WiFi hotspots, and / or by using other suitable techniques for determining location.

[0046] Client device 130 may further include a graphics processing unit 152. Graphics processing unit 152 may be used by processor 132 to render or render three-dimensional images. In some embodiments, client device 130 performs any or all of the three-dimensional rendering necessary to provide a three-dimensional environment.

[0047] The client device 130 may include a network interface 154 for communicating with the server 110 over the network 160. The network interface 154 may include any component or configuration suitable for communicating with the server 110 over the network 160, including, for example, one or more ports, transmitters, wireless cards, controllers, physical layer components, or other items for communicating according to any now known or future developed communication protocol or technology.

[0048] Network 160 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof. Network 160 may also include a direct connection between client device 130 and server 110. In general, communications between server 110 and client device 130 may be conveyed over a network interface using any type of wired and / or wireless connection, and with a variety of communications protocols (e.g., TCP / IP, HTTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0049] Example Model Configuration 2 illustrates a block diagram of an exemplary model search system 200 according to an exemplary embodiment of the present disclosure. In some embodiments, the model search system 200 is trained to receive a set of input data 202 representing a search query and provide output data 206 including one or more search results as a result of receiving the input data 202. Thus, in some embodiments, the model search system 200 may include a search engine 204 operable to process the search query and determine intent.

[0050] The exemplary model retrieval system 200 may include a search engine 204 that takes a search query 202 as input and outputs search results 206 that may include one or more location-specific models. The search query 202 may be a location search query associated with a location. The search engine 204 may process the query to determine a location associated with the search query 202. The search engine may then access a location database 208 to retrieve data related to the location and may access a three-dimensional asset database 210 to determine whether the location has an associated three-dimensional model. If a three-dimensional model is found, the three-dimensional model may be segmented to generate a location-specific model, which may be output as the search result 206. In some embodiments, the search result 206 may further include one or more links based on the search query 202 and may include location data retrieved from the location database 208.

[0051] 3 illustrates a block diagram of an exemplary context-aware model retrieval system 300 according to an exemplary embodiment of the present disclosure. The context-aware model retrieval system 300 is similar to the model retrieval system 200 of FIG. 2, except that the context-aware model retrieval system 300 further includes context-aware retrieval and rendering.

[0052] An exemplary context-aware model search system 300 may include a search engine 304 that takes a search query 302 as input and outputs search results 306 that may include one or more location-specific models. The search query 302 may be a location search query associated with a location. The search engine 304 may process the query to determine a location associated with the search query 302. The search engine may then access a location database 308 to obtain data related to the location and may access three-dimensional asset data 310 to determine whether the location has an associated three-dimensional model. If a three-dimensional model is found, the three-dimensional model may be segmented to isolate the location to generate a location-specific model, which may be output as a search result 306. In some embodiments, the search result 306 may further include one or more links based on the search query 302 and may include location data obtained from the location database 308. In some embodiments, the search engine may further access time-specific context data associated with the location using context query processing 312, which may be used to determine the time-specific context. Once the time-specific conditions are determined, the simulation asset database 314 may be accessed to obtain simulation rendering results that represent the determined conditions. The location-specific models and simulation rendering results may be used to generate simulated events that are output as search results. The simulated events may include a three-dimensional model of the location with the conditions rendered into the model. For example, a search query may relate to the Washington Monument and the time-specific conditions may include current weather conditions including rain. Thus, the simulated event may be a model of the Washington Monument and the rain is rendered into the model.

[0053] 4 illustrates a block diagram of an exemplary results page 400 according to an exemplary embodiment of the present disclosure. In some embodiments, the results page 400 includes a search query input bar 402, and entering a search query into the search query input bar 402 may result in providing one or more search results 404, which may include links, text, and / or media, in response to the search query input. In some embodiments, the results page 400 may include a knowledge graph 410 operable to provide a collection of data related to the one or more search results.

[0054] The example results page 400 of FIG. 4 may be output in response to a search query. The search query may be a location search query or may be a search query entered into a web search service. The search results page 400 may include a search query input bar 402 for entering a new query or refining a current query. In some embodiments, the search query input bar 402 may continue to display the last search query or may be blank upon retrieval of search results. The search results page 400 may further include one or more search results 404. The search results 404 may include links, images, excerpts, maps, videos, and / or documents related to the determined search intent. The search results page 400 may include a knowledge graph 410, which may include one or more media items 412, a location-specific model 414, a name of the location 416, and location information 418. The knowledge graph 410 may be generated based on a predefined function associated with the location, one or more machine learning models, and / or a predefined knowledge graph. The one or more media items 412 can be images, videos, graphs, maps, or audio data related to the location. Additionally, a location-specific model 414 can be generated by obtaining and segmenting a three-dimensional model in response to obtaining a search query. The name of the location 416 can be the official name of the location determined from the search query, or a paraphrased name. Finally, the location information 418 can include various information regarding the population, history, architecture, or size of the location. The location information 418 can be extracted from one or more search results or can be data paired with the name of the location in a location database.

[0055] 5 illustrates a diagram of an exemplary three-dimensional rendering 500 according to an exemplary embodiment of the present disclosure. In some embodiments, the three-dimensional rendering 500 can provide various perspective views, which can include an aerial view.

[0056] The exemplary three-dimensional rendering 500 of FIG. 5 displays one view of a location. The rendering 500 may be displayed in response to a search query and may be rendered based on a location-specific model that is provided. The location-specific model may be provided in an interactive environment to provide a user with the ability to navigate the location-specific model to view different perspectives of the location, including an aerial view 502, or various side views 504 and 506. The rendering of the different perspectives may be based on the location-specific model and may occur on either a server computing system or a client computing system. Alternatively and / or in addition, the location-specific model may be provided as part of a pre-rendered media clip in which the three-dimensional rendering 500 and the various perspectives 502, 504, and 506 may be part of a video, slideshow, or other display media. In some embodiments, the location-specific model may be provided as part of a virtual reality experience, and thus the three-dimensional rendering 500 may be one of multiple renderings that the user views while navigating a virtual space. Generating location-specific 3D models can utilize geographic extraction and also use the Unreal Engine to display the models.

[0057] FIG. 9A illustrates a diagram of an exemplary three-dimensional rendering 900 according to an exemplary embodiment of the present disclosure.

[0058] The three-dimensional rendering 900 may be based on a three-dimensional model and may be any perspective rendering that may include one or more overhead views. The three-dimensional rendering 900 of FIG. 9 is based on a location-specific model of the Space Needle in Seattle. The rendering 900 includes one perspective, but alternative renderings may be generated based on the location-specific model, including but not limited to an aerial view 902 or multiple side views 904 and 906. In some embodiments, the rendering 900 may include a situation-specific rendering that provides a situation-specific simulation. For example, this specific rendering 900 displays the Space Needle during an overcast day. However, Seattle is not always precipitation-free, and alternatively, it is not always cloudy. A user may want a simulation of what the Space Needle looks like during various weather conditions. In some embodiments, the location-specific model may be provided to a user computing device without including external features or objects. The location-specific model may then be viewed completely independently, or external environmental features may be added by the user computing device based on user input.

[0059] 9B illustrates a diagram of example three-dimensional renderings 910, 920, and 930 according to an example embodiment of the present disclosure. The three-dimensional renderings 910, 920, and 930 are similar to the three-dimensional rendering 900 of FIG. 9A, except that the three-dimensional renderings 910, 920, and 930 further include a situation-based simulation.

[0060] The systems and methods disclosed herein can be used to obtain or generate simulation data, which can be used to obtain simulation assets related to the situation described by the situation data. The simulation assets can be used to generate simulated events using location-specific models, which can include simulation of a particular situation. For example, FIG. 9B displays three simulated weather conditions generated using location-specific models and simulation assets. Three-dimensional renderings are displayed in the user interface with the location and weather conditions. A partly cloudy three-dimensional rendering 910 can provide a simulated partly cloudy weather condition for the location, which can provide the user with a rendering that accurately depicts the location with the particular weather condition. Additionally, the systems and methods disclosed herein can provide previews of the location in various other weather conditions, such as a rainy weather condition 920, or a heavy rainy weather condition 930. These renderings can be time-specific to a time entered by the user, can be based on real-time situation conditions, or can be manually selected situation conditions. Alternatively and / or in addition, the conditions may be crowding conditions, traffic conditions, and / or various other conditions that may be determined and simulated for the user.

[0061] 10 illustrates an example augmented reality experience diagram according to an example embodiment of the present disclosure. In some embodiments, the augmented reality experience 1020 is trained to receive a set of data representing a three-dimensional model and provide output data 1024, which may be a rendering of the three-dimensional model of the location as a result of receiving the input data. Thus, in some embodiments, the augmented reality experience 1020 may include a preview 1012 of the location that can be used to depict the location before the augmented reality experience is selected 1016.

[0062] The augmented reality experience 1020 can be one of many mediums for providing the location-specific model. In this embodiment, the user computing device is provided with a user interface that allows the user to view the location-specific model 1012 in the context 1010 or as an augmented reality rendering 1024. In this embodiment, the user interface provides the location-specific model in the context 1010 with the name of the location and a text excerpt 1014 with information about the location. The user interface can include a selectable icon 1016 for switching to an augmented reality experience 1020 that can use the location-specific model 1012 to render the location into the user's environment 1022. The augmented reality rendering 1024 can be placed in the user's environment 1022 and viewed as a scaled model of the location via the augmented reality experience 1024. Additionally, the user interface can provide a selectable icon 1026 to exit the augmented reality experience 1020 and return to the location-specific model 1012 in the context 1010.

[0063] 11 illustrates a diagram of an exemplary media results page 1100 in accordance with an exemplary embodiment of the present disclosure. In some embodiments, the media results page 1100 can include a set of images 1104, 1106, and 1108 of a location, as well as a location-specific three-dimensional model 1110. Additionally, in some embodiments, the media results page 1100 can include an interactive environment that can be used to view and interact with the location-specific three-dimensional model 1110.

[0064] The media results page 1100 in FIG. 11 includes multiple image results 1104, 1106, and 1108, a search query 1102, a three-dimensional model of the location, a map of the location, and a name of the location 1112. In the illustrated embodiment, the search query 1102 includes the name of the location, but in other embodiments, the search query can include an address, a descriptor of the location, or any other query that can be associated with the location. The multiple images 1104, 1106, and 1108 can be displayed as a side panel with selectable tabs at the top of the panel to refine the results or to show different forms of media. Additionally, the media results page 1100 can provide links to source data or other pages by selecting the media or various selectable icons.

[0065] The location-specific three-dimensional model 1110 can be provided in a main pane, a separate pane, or a side pane. The location-specific model 1110 can be provided in an interactive environment or in a pre-rendered media form. In some embodiments, the media results page 1100 can include location information or location-specific data overlaid on one or more media items. For example, in this embodiment, a map of the location and its surrounding area, the name of the location 1112, and information about the location are overlaid on the three-dimensional model 1110.

[0066] 12 illustrates a diagram of an example route overlay model 1200 according to an example embodiment of the present disclosure. In some embodiments, the route overlay model 1200 can include a location-specific model 1204 overlaid on a map and a visual route overlay 1208 showing a suggested navigation route to reach the location.

[0067] The route overlay model 1200 can utilize the location-specific model 1204, the map database, and the determined navigation route to provide an enhanced map application experience to the user. The route overlay model can include generating the location-specific model 1204, obtaining a user computing device location, obtaining a location address, and obtaining a map. The obtained user location and the obtained location address along with the obtained map can be used to determine a route to go from the user's location to the address. The determined route can be used to generate a visual route overlay 1208 representing the navigation route. The map, the location-specific model 1204, and the visual route overlay 1208 can be used to generate an augmented map representing the location-specific model 1204 and the visual route overlay 1208 overlaid on the map to provide the user with navigation directions with visual nuances added to the three-dimensional model.

[0068] In this exemplary embodiment, the route overlay model 1200 may be part of a map application that may further include various features such as a route preview 1202, an ordered direction list, location labels (e.g., transit locations, restaurants, overnight locations, etc.), location markers (e.g., waypoints, final destination, etc.). The illustrated route overlay model 1200 includes location-specific models (i.e., transit stations) overlaid on a map with a visual route overlay 1208 that provides directions to location markers 1206. The route overlay model may be provided via a user interface, which may include a two-dimensional map and directions 1210 in conjunction with a three-dimensional representation.

[0069] Exemplary Methods 6 illustrates a flow diagram of an exemplary method for performing in accordance with an exemplary embodiment of the present disclosure. Although FIG. 6 illustrates steps performed in a particular order for purposes of explanation and discussion, the method of the present disclosure is not limited to the order or configuration specifically shown. Various steps of method 600 may be removed, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0070] At 602, the computing system can obtain a location query from a user computing device. The location query can be a search query entered by a user into a user interface.

[0071] At 604, the computing system may process the location query to determine a location.

[0072] At 606, the computing system can access a three-dimensional asset database to obtain a three-dimensional model associated with the location. The three-dimensional model can include one or more architectural structures and can include the determined location.

[0073] At 608, the computing system can generate a location-specific model by segmenting the three-dimensional model to separate the location from other objects in the three-dimensional model. The segmentation can be three-dimensional segmentation or image segmentation that removes trees, roads, and other structures from the three-dimensional model to generate the location-specific model. The segmentation can include utilizing multiple images of the location to segment the three-dimensional model, and the multiple images can include one or more aerial images. In some embodiments, the location-specific model can include models of architectural structures (e.g., landmarks, monuments, historical structures, etc.).

[0074] At 610, the computing system can provide the location-specific model to a user computing device. The location-specific model can be provided as part of an interactive environment, a virtual reality experience, an augmented reality experience, and / or a pre-rendered media item. In some embodiments, the location-specific model can comprise location information, which can include a name of the location, an address of the location, an image of the location, and / or an actuality of the location.

[0075] 7 illustrates a flow diagram of an exemplary method performed in accordance with an exemplary embodiment of the present disclosure. Although FIG. 7 illustrates steps performed in a particular order for purposes of explanation and discussion, the method of the present disclosure is not limited to the order and configuration specifically shown. Various steps of method 700 may be removed, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0076] At 702, a computing system may obtain a location query. The location query may include one or more search terms and / or one or more images.

[0077] At 704, the computing system may process the location query to determine a location. The location may be determined using a search engine, a location tag, and / or other determination methods.

[0078] At 706, the computing system can access a three-dimensional asset database to obtain the three-dimensional model and generate location-specific segments. The three-dimensional asset database can utilize a large database of images to generate multiple three-dimensional models that can be associated with locations and easily searchable. Generating location-specific segments can include removing objects from the three-dimensional model that are not part of a particular determined location. In some embodiments, the location-specific segments can be generated by a segmentation process that isolates the location-specific model by removing portions of the three-dimensional model.

[0079] At 708, the computing system can obtain situational data. The situational data can represent time-specific conditions associated with the determined location. The time-specific conditions can be weather conditions, traffic levels, crowding, and / or train location.

[0080] At 710, the computing system can process the situation data to determine each simulation asset. Each simulation asset can be a set of data that enables a particular situation to be rendered into a simulated event.

[0081] At 712, the computing system may obtain each simulation asset. The simulation assets may be obtained from a simulation asset database that stores a plurality of situation-specific simulation assets.

[0082] At 714, the computing system can generate a simulation event. The simulation event can include location-specific segments and location- and time-specific conditions based on each simulation asset. For example, the location can be the Golden Gate Bridge and the time-specific conditions can be heavy traffic and rain. The location-specific segment can be a three-dimensional model of the Golden Gate Bridge and the simulated event can be the Golden Gate Bridge with heavy traffic and rain rendered into the model.

[0083] 8 illustrates a flow diagram of an exemplary method performed in accordance with an exemplary embodiment of the present disclosure. Although FIG. 8 illustrates steps performed in a particular order for purposes of explanation and discussion, the method of the present disclosure is not limited to the order and configuration specifically shown. Various steps of method 800 may be eliminated, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0084] At 802, a computing system can reconstruct a three-dimensional environment to generate a three-dimensional mesh and textures. The reconstruction can include a database of collected map data, image data, and other location data.

[0085] At 804, the computing system may acquire a plurality of images of the environment.

[0086] At 806, the computing system can process the images using a machine learning model to segment the images into parts. The segmentation can be used to understand meshes and textures to discover the underlying semantic structure of the environment. The segmentation can exploit semantic diversity for image segmentation. A graph cut algorithm can be applied to truly segment the mesh into parts.

[0087] At 808, the computing system can project the image portion onto a three-dimensional mesh.

[0088] At 810, the computing system may remove one or more objects from the reconstructed environment to generate the augmented environment. The one or more objects may include other building structures, trees, walkways, and other obstacles. The removing may be completed automatically, or may include obtaining user input to remove one or more features in the mesh.

[0089] The computing system may store the augmented environment at 812. The augmented environment may be compressed before being stored and may be stored in a three-dimensional asset database.

[0090] Further disclosure The techniques discussed herein make reference to servers, databases, software applications, and other computer-based systems, and to actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and division of tasks and functions therein between and among components. For example, the processes discussed herein can be implemented using a single device or component, or multiple devices or components operating in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0091] Although the present subject matter has been described in detail with respect to various specific exemplary embodiments thereof, each example is provided by way of explanation and not of limitation of the present disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, will be able to readily make modifications, variations, and equivalents to such embodiments. Thus, the present disclosure does not exclude the inclusion of such changes, modifications, and / or additions to the present subject matter as would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment can be used with another embodiment to yield a further embodiment. Thus, the present disclosure is intended to encompass such modifications, variations, and equivalents. [Explanation of symbols]

[0092] 100 Computing Systems 110 Servers, server computing systems 112 processors 114 Memory 116 Command 118 Geographic Data 120 3D Model Search 122 Image Segmentation Elements 130 client devices 132 processors 134 Memory 140 Browser Elements 142 Application Elements 146 Display 148 Positioning System 150 User Input Devices 152 Graphics Processing Unit 154 Network Interface 160 Network 180 3D Asset Database 200 Model Search System 202 Input Data 204 Search Engines 206 Output, Search results 208 Location Database 210 3D Asset Database 300 Situational Awareness Model Search System 302 Search Queries 304 Search Engines 306 Results 308 Location Database 310 3D Asset Database 312 Status Query Processing 314 Simulation Asset Database 400 result page 402 Search Query Input Bar 404 Results 410 Knowledge Graph 412 Media Item 414 Location Specific Models 416 Location Name 418 Location Information 500 3D renderings 502 Aerial View 504 Side View 506 Side View 600 ways 700 methods 800 ways 900 3D Rendering 902 Aerial View 904 Side View 906 Side View 910 3D Rendering 920 3D rendering, cloudy weather conditions 930 3D rendering of heavy rain weather conditions 1010 Context 1012 Location preview, location specific models 1014 Text Excerpt 1016 selectable icons 1020 Augmented Reality Experience 1022 User's environment 1024 output data, augmented reality rendering 1026 selectable icons 1100 Media Results Page 1102 Search Queries 1104 Location images, location specific models 1106 Location Images 1108 Location Images 1110 Location-specific 3D models 1112 Location Name 1200 Route Overlay Model 1202 Route Preview 1204 Location Specific Models 1206 Location Marker 1208 Visual Route Overlay 1210 direction

Claims

1. 1. A computer-implemented method for providing a three-dimensional modeling to a user, comprising: obtaining, by a computing system comprising one or more computing devices, a location query associated with a user computing device; processing, by the computing system, the location query to determine a location; accessing, by the computing system, a three-dimensional asset database to obtain a three-dimensional model that models the location in combination with one or more other different locations; segmenting, by the computing system, the three-dimensional model to generate a location-specific segment of the three-dimensional model that corresponds to the location, the location-specific segment modeling the location in isolation from the one or more other distinct locations; acquiring, by the computing system, context data, the context data representing a time-specific context associated with the location; processing, by the computing system, the situation data to determine each simulation asset; acquiring, by the computing system, each of the simulation assets; generating, by the computing system, a simulated event comprising the location and the time-specific situation based on the location-specific segments of the three-dimensional model and each of the simulation assets; providing, by the computing system, the simulated event to the user computing device; A computer-implemented method comprising:

2. The method of claim 1 , wherein the location-specific segments of the three-dimensional model are generated by image segmentation using a plurality of images associated with the location.

3. The method of claim 1 or 2, wherein the location-specific segments of the three-dimensional model are generated based at least in part on sensor data representative of the location.

4. The method of claim 1 , wherein the location-specific segments of the three-dimensional model include architectural structures or landmarks.

5. The method of claim 1 , wherein the location-specific segments of the three-dimensional model are augmented reality assets.

6. 6. The method of claim 1, further comprising providing a virtual reality experience including the simulated event.

7. The method of claim 1 , wherein the simulated events are provided in an interactive environment.

8. The method of claim 7 , wherein the interactive environment includes the simulated events in combination with one or more two-dimensional maps.

9. The method of claim 7 or 8, wherein the interactive environment comprises a portion of a search result webpage that includes further search results responsive to the location query.

10. The method of claim 1 , wherein the simulated events are provided in a pre-rendered video.

11. obtaining, by the computing system, an address of the location; providing, by the computing system, the address of a location having the simulated event; 11. The method of any one of claims 1 to 10, further comprising:

12. the situation data includes traffic data; The method of claim 1 , wherein each of the simulation assets comprises simulated traffic.

13. The situation data includes crowd data, The method of claim 1 or 12, wherein each of the simulation assets includes a simulated crowd of people whose size is based on the crowd data.

14. The condition data includes weather data. The method of claim 1 , 12 or 13, wherein each of the simulation assets includes a simulated weather condition.

15. 1. A computing system comprising: one or more processors; one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations including: receiving a location query from a user computing device; processing the location query to determine a location; accessing a three-dimensional asset database to obtain a three-dimensional model associated with the location; generating a location-specific model by segmenting the three-dimensional model to separate the location from other objects in the three-dimensional model; acquiring situation data, the situation data representing a time-specific situation associated with the location; processing the situation data to determine each simulation asset; acquiring each of the simulation assets; generating a simulated event including the location-specific and time-specific conditions based on the location-specific segments of the three-dimensional model and each of the simulation assets; and providing said simulated event to said user computing device. One or more non-transitory computer-readable media, A computing system comprising:

16. The operation includes: generating a map including the location-specific model; providing said map to said user computing device; The computing system of claim 15 further comprising:

17. The operation includes: obtaining location data for the user computing device, the location data representing a user location; obtaining an address for the location associated with the location-specific model; a user determining a route to reach said address from said user location; generating a visual route overlay based on the route; generating an augmented map, the augmented map including the visual route overlay and the map; providing said augmented map to a user computing device; The computing system of claim 16 further comprising:

18. One or more non-transitory computer-readable storage media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations including: obtaining user input specifying a location query relating to a location; providing the location query to a server system over a network; receiving a location-specific segment of a three-dimensional model, or one or more renderings of the location-specific segment of the three-dimensional model, from the server system over the network, the three-dimensional model modeling the location in combination with one or more other distinct locations, the location-specific segment of the three-dimensional model being generated by segmenting the three-dimensional model to separate the location from the one or more other locations; acquiring situation data, the situation data representing a time-specific situation associated with the location; processing the situation data to determine each simulation asset; acquiring each of the simulation assets; generating a simulated event including the location-specific and time-specific conditions based on the location-specific segments of the three-dimensional model and each of the simulation assets; providing a display of at least a first rendering of the simulated event; One or more non-transitory computer-readable storage media,

19. One or more non-transitory computer-readable recording media as described in claim 18, wherein the step of providing a display of at least the first rendering of the simulated event includes a step of providing a display of at least the first rendering in a search results webpage.

Images