Location-specific three-dimensional models responsive to location-related queries

Abstract

To provide better understanding of a location by providing better interactivity, better visibility and better understanding of dimensionality about the generation of a location-specific three-dimensional model in response to a location query.SOLUTION: Generation of a model can be enabled by utilizing a three-dimensional asset database and a segmentation method. A location-specific model can provide further practicability by further including situation-specific simulated results, such as simulated weather or traffic.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] Related Applications This application claims the benefit and priority of U.S. Non-Provisional Application No. 17 / 227,871, filed on Apr. 12, 2021. U.S. Non-Provisional Application No. 17 / 227,871 is hereby incorporated by reference in its entirety.

[0002] The present disclosure generally relates to providing a three-dimensional model of a location in response to a user query related to or seeking information about a location. More particularly, the present disclosure relates to providing location-specific segments of a three-dimensional model of a location in response to a location-related query seeking information associated with that location, such as, for example, a search query, a navigation query, a mapping query, a weather query, a review query, a visual query, and / or other user requests for information about a location.

Background Art

[0003] When searching for a location via a search engine, some existing systems can return results in the form of hyperlinks and / or generated images that can include excerpts of text, photos, or maps. For monuments and landmarks, these results often fail to depict the actual appearance and size of the monument or landmark. Further, while an image can provide a perspective of how the location looks, an image alone may fail to capture the dimensionality of the location. Further, videos of the location can also be limited for various reasons. In particular, search results lack interactivity, which can make it difficult to find another aspect of the location (e.g., the context of the location or various perspectives of the location). For example, trying to view close-ups of various views or different parts of the location may require a tiresome search rather than being provided through an intuitive, interactive interface.

[0004] Knowing the weather, traffic, and crowd conditions at monuments and landmarks can help travelers and other visitors better understand and prepare for visiting the monument. A map alone fails to take into account traffic or crowd conditions. Further, the visibility of a monument or landmark can be affected by crowd conditions or weather. Knowledge of the impact of these factors on visibility is often limited to scouring the literature on the object. However, many people consider themselves visual learners. SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description that follows, or can be learned from the description, or can be learned through practice of the embodiments.

[0006] One exemplary aspect of the present disclosure is directed to a computer-implemented method for providing a user with a three-dimensional (3D) model. The method includes obtaining, by a computing system that can include one or more computing devices, a location query associated with a user computing device. The method can include determining, by the computing system, a location by processing the location query. In some embodiments, the method can include obtaining, by the computing system, a 3D model that models the location in combination with one or more other different locations by accessing a 3D asset database. The method can include segmenting, by the computing system, the 3D model to generate a location-specific segment of the 3D model corresponding to the location. In some embodiments, the location-specific segment can model the location separated from one or more other different locations. The method can include providing, by the computing system, the location-specific segment of the 3D model, or one or more renderings of the location-specific segment of the 3D model, to the user computing device.

[0007] As described in more detail below, the step of segmenting a 3D model to generate location-specific segments can provide a number of technical advantages. In particular, instead of providing the 3D model as a whole, location-specific segments can be provided to a user computing device. Doing so can reduce the bandwidth requirements for sending relevant data to the user device, and can also reduce the computational resource requirements for storing and rendering the location-specific model on the user device. In other words, the steps related to segmenting a 3D model can reduce the amount of data that needs to be sent to the user computing device, thereby enabling a more computationally efficient way of supplying 3D modeling to the user.

[0008] Another exemplary aspect of the present disclosure is directed to a computing system. The computing system can include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations can include obtaining a location query from a user computing device. The operations can include processing the location to determine the location. In some embodiments, the operations can include accessing a 3D asset database to obtain a 3D model associated with the location. The operations can include generating a location-specific model by segmenting the 3D model to isolate the location from other objects in the 3D model, and providing the location-specific model to the user computing device.

[0009] Another exemplary aspect of the present disclosure is directed to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations. The operations can include obtaining user input that specifies a location query regarding a location. The operations can include providing the location query to a server system via a network. In some embodiments, the operations can include receiving, via the network, from the server system, a location-specific segment of a 3D model, or one or more renderings of a location-specific segment of the 3D model. The 3D model can model the location in combination with one or more other different locations, and the location-specific segment of the 3D model can be generated by segmenting the 3D model to isolate the location from one or more other different locations. The operations can include providing a display of at least a first rendering of the location-specific segment of the 3D model.

[0010] Other aspects of the present disclosure are directed to various systems, devices, non-transitory computer-readable media, user interfaces, and electronic devices.

[0011] These and other features, aspects, and advantages of the various embodiments of the present disclosure will be better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated herein and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain the relevant principles.

[0012] A detailed discussion of embodiments directed to those of ordinary skill in the art is set forth in this specification with reference to the accompanying figures.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9A

Figure 9B

Figure 10

Figure 11

Figure 12

DETAILED DESCRIPTION OF THE INVENTION

[0014] Reference numerals repeated throughout multiple figures are intended to identify the same functions in various embodiments.

[0015] Summary Generally, the present disclosure is directed to providing location-specific segments of a three-dimensional model of a location in response to a query regarding the location, such as, for example, a search query, a navigation query, a mapping query, a weather query, a review query, a visual query, and / or other user requests for information. The systems and methods disclosed herein can utilize a database of three-dimensional models to provide location-based rendering and / or location-specific models in response to a query regarding a location (a "location query"). In some embodiments, the systems and methods can include obtaining a location query. The location query can be obtained from a user's computing device or can be triggered by a user's computing device. The systems and methods can include processing the location query to determine the location. In some embodiments, the systems and methods can include accessing a three-dimensional asset database to obtain a three-dimensional model associated with the location. The three-dimensional model and the location can be used to generate a location-specific model. Generating a location-specific model can include segmenting the three-dimensional model to isolate the location from other objects in the three-dimensional model. The systems and methods can include providing the location-specific model to a computing device. For example, the location-specific model can be provided to a user computing device. In some embodiments, the location-specific model can include architectural structures (e.g., landmarks, monuments, and / or buildings).

[0016] Obtaining a location query can be easily done by a user interface configured to receive input from a user. Alternatively, and / or in addition, the location query can be a query generated and obtained by a computing system in response to an event triggered by a user computing device. For example, a user can scroll through a list of attractions, select a particular attraction, and learn more about that attraction. In response to the selection, a location query for the attraction can be generated and obtained for processing. In other examples, a user can request to view that location on a map, request navigational instructions to that location, request information about points of interest associated with the location (e.g., reviews generated by the user), present a textual or visual search query that results in the location being obtained, and / or perform other actions indicating a request for information about that location. Each of these exemplary actions can trigger a location query for processing by the proposed system.

[0017] In some embodiments, a location query can include a series of terms or one or more images associated with a particular location. The terms can include the name of the location that can describe the location, can be the address of the location, or can be any other form of information that is indirectly or directly related and implies or results in the location being relevant. In some examples, a location query can include an image that processes and determines where a photo was taken or what is depicted in the image.

[0018] After a location has been determined, a three-dimensional asset database can be accessed and searched to determine whether that particular location has any associated three-dimensional model. The three-dimensional model can be a three-dimensional model of a particular structure or structures found at that location. In particular, in some embodiments, the three-dimensional asset database can include general or global three-dimensional models covering a significant portion of the Earth's surface or of an individual country or city located therein. For example, various algorithms can be applied to automatically generate one or more very large three-dimensional models covering various addresses, streets, neighborhoods, cities, and / or the like from available images (e.g., satellite images and / or street-level images).

[0019] To provide a location-specific model, the system and method can separate the location in the general three-dimensional model to generate a location-specific model. Specifically, an exemplary system of the present disclosure can segment a portion of the general three-dimensional model into a plurality of segments. In some embodiments, the system and method can perform three-dimensional segmentation to extract the location-specific segments of the three-dimensional model to model one or more particular locations. The location-specific model can be generated using segmentation of an image. In this way, a smaller model including the required location can be obtained. The smaller model can reduce the processing power required for storage and rendering, and can also reduce the bandwidth required to transmit the model to the user's computing device. Further, the location-specific model can be easier for the user to access and understand with respect to the general three-dimensional model. Thus, the location-specific model can enable an improved interaction with the user.

[0020] After the location-specific segments of the model are generated, the location-specific segments of the three-dimensional model can be rendered and presented to the user in various situations. As an example, the computing system of the server can send or supply the location-specific segments of the three-dimensional model to the user computing device, and the user computing device can display the rendering of that location using the location-specific model. Alternatively, and / or in addition, the server computing system can perform the rendering of the location-specific segments of the three-dimensional model and send or supply that rendering to the user computing device.

[0021] In some embodiments, the location-specific segments of the three-dimensional model can be generated by image segmentation using a plurality of images associated with that location. Further, the plurality of images can include one or more aerial images. Alternatively, and / or in addition, in some embodiments, the model can be generated using light detection and ranging (LiDAR) data, or other sensor data (e.g., spectral sensor data generated using a spectral sensor that can be used to segment trees or other objects). In particular, the LiDAR data or other sensor data can be used to determine what to segment.

[0022] Location-specific segments of the 3D model can be provided in various ways via various media. In some embodiments, the location-specific model can be provided as an augmented reality asset rendered in the user's environment via an augmented reality experience. Alternatively, and / or in addition, the location-specific model can be provided in a virtual reality experience. In some embodiments, the user computing device can comprise an interactive virtual environment that can include the location-specific model. Further, the interactive environment can include one or more 2D or 3D maps. Location-specific segments of the 3D model can be provided by a pre-rendered video, or a set of pre-rendered videos, where each video can depict various views of the location generated based on the location-specific segments.

[0023] In some embodiments, the location-specific model can comprise other information or data related to or associated with the location. For example, the system and method can include obtaining the address of the location and providing the address of the location that has the location-specific model. In some embodiments, the system and method can include obtaining location information data. The location information data can include information related to the location (e.g., information regarding the history of the location). The location information data can be provided to the user computing device using the location-specific model. In some embodiments, the location-specific model can comprise a plurality of images of the location. Further, in some embodiments, the system and method can include generating a map that includes the location-specific model and providing the map to the user computing device. Further, in some embodiments, the system and method can include obtaining location data for the user computing device. The location data can describe the user's location. The system and method can include obtaining the address for the location associated with the location-specific model. The address and the user's location can be used to determine the route that the user uses to go from the user's location to the address. A visual route overlay can be generated based on the route, and the visual route overlay and the map can be used to generate an extended map using the visual route overlay superimposed on the map. The extended map can then be provided to the user computing device. Such an extended map has the technical effect of providing an efficient and effective means for navigating to that location.

[0024] Alternatively, and / or in addition, the system and method can include obtaining situational data. The situational data can represent time-specific situations associated with a location. The system and method can include processing the situational data and obtaining each simulation asset to determine each simulation asset. Each simulation asset and location-specific model can be used to generate simulated events that include time-specific situations occurring at that location. In some embodiments, the situation can include, in addition to weather conditions, the location of various physical items such as trains, airplanes, buses, taxis, and / or the like, the congestion of people at or near that location, and / or traffic. For example, the time-specific situation can be rain, and the generated simulated event can be the rendering of a location-specific segment of a 3D model with simulated rain. More generally, simulated events that include time-specific situations occurring at that location can provide a technical effect beyond the embodiments of the simulation itself. Specifically, the simulated events enable the user to visualize the location in various time-specific situations, enabling the user to gain a better understanding of the location, thereby enabling the user to navigate to that location in real life.

[0025] In some embodiments, providing a location-specific model can include providing a location-specific model having one or more search results via a web page (e.g., a web page returned in response to a textual or visual location query). Alternatively and / or in addition, the location-specific model can be provided and displayed in a section or location on a web page separate from other search results. For example, in some embodiments, the location-specific model can be provided in a separate panel along with other data related to the location associated with the model (e.g., the name of the location associated with the location-specific model, the address of the location, and / or information about the location). In some embodiments, mesh segmentation can be used to compactly respond to a query. Specifically, the segments of the mesh are made to be renderable more quickly than a rendering of the entire area (e.g., in a panel). The rendering of the segmentation can thus assist in providing search results without imposing a heavy burden on the bandwidth where a rendering of the complete model might occur.

[0026] The systems and methods disclosed herein can be utilized in various fields and industries. For example, the systems and methods disclosed herein can be used to provide an improved user experience for map applications. More specifically, the systems and methods disclosed herein can be used to provide a map with location-specific models to a user in response to a location query. The map application can provide a visual route overlay to provide a realistic rendering of the area to be navigated through. In some embodiments, the map application can provide several options for understanding an area. The map application can provide a street view of the location, an overhead map, an aerial view map, or an interactive 3D model. In this way, the map application can enable the user to move to that location more effectively.

[0027] Another example can include implementing the systems and methods into video games, and / or augmented or virtual reality experiences. For example, the systems and methods disclosed herein can enable the generation of open-world games, where the user can travel to various areas and, upon reaching an area, can send a location query and receive a location-specific model of that area in response. More specifically, the game can be set in France, and when a character travels to a location, it can send a location query for that location to the server and receive a location-specific model of that location in return (e.g., the character can travel to the Eiffel Tower, send a location query for the Eiffel Tower, and receive a location-specific model of the Eiffel Tower in return).

[0028] The 3D asset databases utilized by the systems and methods disclosed herein can be generated using various methods or processes. The process can start with the creation of 3D content. The 3D environment can be constructed by common 3D reconstruction methods. The reconstruction can generate a 3D mesh with attached textures (e.g., 2D images representing the color of the mesh, etc.). The next step of the process can include the segmentation of the images. Segmentation can be used to understand the mesh and textures to find the underlying semantic structure of the environment (e.g., the semantic structure of a reconstructed city, etc.). In some embodiments, the process can utilize the semantic diversity for the segmentation of the images. The segmentation of the images can include using a machine learning model to segment a plurality of 2D images into parts. The segmented images can then be projected onto the 3D mesh. The process can further include cleaning the generated 3D environment by cleaning the mesh, except for blobs. For example, trees, streets, sidewalks, and other objects can be removed. The cleaning step can be completed manually or automated. The resulting 3D assets can be compressed and stored in a database. The process can be repeated iteratively to update the database with the latest structures from cities, towns, etc.

[0029] The systems and methods of the present disclosure provide several technical effects and benefits. As an example, the systems and methods can provide location-specific segments of 3D models in response to location queries. The systems and methods can be further used to provide models of locations (e.g., landmarks or monuments, etc.) with real-time traffic, crowding, weather, and / or other context or situation-specific information. Further, the systems and methods can enable 3D models for map applications.

[0030] Another technical benefit of the systems and methods of the present disclosure is the dimensionally accurate modeling and interactivity of the rendering provided. The model can provide a dimensionally accurate representation of the location, enabling potential visitors to determine whether they desire to actually visit that location. Further, the dimensionally accurate representation enables the user to more effectively navigate to that location.

[0031] Another exemplary technical effect and benefit relate to improved computational efficiency and functionality in a computing system. For example, some existing systems enable a user to view a rendering of a global or general 3D model. Processing and rendering an entire global or general 3D model requires significant computational resources. In contrast, by extracting and then rendering location-specific segments of a 3D model, the proposed systems and methods can conserve computational resources such as processor usage, memory usage, and / or network bandwidth. In particular, the location-specific segments can reduce the required bandwidth for transmission thereof to a user device as compared to transmitting a global 3D model. Further, the location-specific segments can reduce the memory usage in a user device and result in a reduction in processor usage to render the location-specific segments.

[0032] Reference will now be made to the figures to discuss exemplary embodiments of the present disclosure in further detail.

[0033] Exemplary Devices and Systems FIG. 1 shows an exemplary computing system 100 that can be used to perform location-based model search according to aspects of the present disclosure. System 100 has a client-server architecture that includes a server 110 that communicates with one or more client devices 130 via a network 160. However, the present disclosure can also be implemented using other suitable architectures, such as a single computing device not connected to a network.

[0034] System 100 includes a server 110, such as a web server for example. Server 110 can be one or more computing devices implemented as a parallel or distributed computing system. In particular, multiple computing devices can operate together as a single server 110. Server 110 can have one or more processors 112 and a memory 114. Server 110 can also include a network interface used to communicate with one or more remote computing devices (e.g., client devices) 130 via network 160.

[0035] Processor 112 can be any suitable processing device, such as a microprocessor, a microcontroller, an integrated circuit, or other suitable processing device. Memory 114 can include any suitable computing system or medium, including but not limited to a non-transitory computer-readable medium, RAM, ROM, a hard drive, a flash drive, or other memory devices. Memory 114 can store information accessible by processor 112, including instructions 116 that can be executed by processor 112. Instructions 116 can be any set of instructions that, when executed by processor 112, cause processor 112 to provide desired functionality.

[0036] In particular, instruction 116 can be executed by processor 112 to perform a 3D model search 120. The 3D model search 120 can be configured to search for 3D models associated with a location, segment and output location-specific 3D models. In some embodiments, the 3D model search 120 can be configured to access a 3D asset database 180 to obtain 3D models, and the 3D models can be segmented using image segmentation element 122 stored in memory 114 of server computing system 110.

[0037] It will be appreciated that the term "element" can refer to computer logic used to provide desired functionality. Thus, any element, function, and / or instruction can be implemented in hardware that controls a general purpose processor, application specific circuitry, firmware, and / or software. In one embodiment, an element or function is a program code file stored in a storage device, loaded into memory, and executed by a processor, or provided from a computer program product stored in a tangible computer-readable storage medium such as RAM, a hard disk, or an optical or magnetic medium, e.g., computer-executable instructions.

[0038] Memory 114 can also include geographic data 118 that can be obtained, manipulated, created, or stored by processor 112. The geographic data 118 can include geographic images (e.g., digital maps, satellite images, aerial photographs, street-level photographs, synthetic models, etc.), tables, vector data (e.g., vector representations of roads, plots, buildings, etc.), data of points of interest (e.g., localities such as islands, cities, restaurants, hospitals, parks, hotels, and schools, etc.), or other suitable geospatial data or related information. By way of example, the geographic data 118 can be used to access information and data associated with a location and generate a rendering of a 3D model of the earth's surface.

[0039] The geographical data 118 can be stored in one or more databases. The one or more databases can be connected to the server 110 by a high-bandwidth LAN or WAN, or can also be connected to the server 110 via the network 160. The one or more databases can be divided to be locally located at multiple locations.

[0040] The server 110 can exchange data with one or more client devices 130 via the network 160. Two clients 130 are shown in FIG. 1, but any number of client devices 130 can be connected to the server 110 via the network 160. The client device 130 can be any suitable type of computing device such as a general-purpose computer, a dedicated computer, a navigation device, a laptop, a desktop, an integrated circuit, a mobile device, a smartphone, a tablet, a wearable computing device, a display combined with and / or embedded with one or more processors, or other suitable computing devices. Further, the client device 130 can be a plurality of computing devices that operate together to perform an operation or a computing action.

[0041] Similar to the server 110, the client device 130 can include a processor 132 and a memory 134. The memory 134 can store information accessible by the processor 132, including instructions and data executable by the processor. By way of example, the memory 134 can store a browser element 140 and an application element 142.

[0042] The browser element 140 can provide instructions for implementing a browser. In particular, a user of the client device 130 can exchange data with the server 110 by using the browser to visit a website accessible at a specific web address. The 3D model search of the present disclosure can be provided as an element of the user interface of the website.

[0043] The application element 142 can provide instructions for operating a dedicated application on the client device 130. In particular, the dedicated application can be used to exchange data with the server 110 via the network 160. The application element 142 can include client device-readable code for providing and implementing aspects of the present disclosure. For example, the application element 142 can provide instructions for implementing a mapping application or a virtual earth application.

[0044] The client device 130 can include various user input devices 150 for receiving information from a user, such as a touch screen, a touch pad, data input keys, a speaker, a mouse, a motion sensor, and / or a microphone suitable for voice recognition. Further, the client device 130 can have a display 146 for presenting information, such as rendering a 3D model in a 3D environment.

[0045] The client device 130 can also include a positioning system 148 that can be used to identify the location of the client device 130. The positioning system 148 can optionally be used by the user to monitor the user's position relative to the rendering. The positioning system 148 can be any device or circuitry for monitoring the location of the client device 130. For example, the positioning device 148 can determine an actual or relative position by using a satellite navigation positioning system (e.g., GPS system, Galileo positioning system, Global Navigation Satellite System (GLONASS), Beidou satellite navigation and positioning system, etc.), an inertial navigation system, a dead reckoning system, based on an IP address, by using triangulation and / or proximity to a cellular tower or WiFi hotspot, and / or by using other suitable techniques for determining a position.

[0046] The client device 130 can further include a graphics processing unit 152. The graphics processing unit 152 can be used by the processor 132 to render or depict a three-dimensional image. In some embodiments, the client device 130 performs any or all of the three-dimensional rendering required to provide a three-dimensional environment.

[0047] The client device 130 can include a network interface 154 for communicating with the server 110 via the network 160. The network interface 154 can include, for example, one or more ports, transmitters, wireless cards, controllers, physical layer components, or other items for communicating according to any currently known or future developed communication protocol or technology, and can include any component or configuration suitable for communicating with the server 110 via the network 160.

[0048] The network 160 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof. The network 160 can also include a direct connection between the client device 130 and the server 110. Generally, communication between the server 110 and the client device 130 can use any type of wired and / or wireless connection and be transmitted via a network interface using various communication protocols (e.g., TCP / IP, HTTP), encoding or formatting (e.g., HTML, XML), and / or protection methods (e.g., VPN, secure HTTP, SSL).

[0049] Exemplary model configuration FIG. 2 shows a block diagram of an exemplary model search system 200 according to an exemplary embodiment of the present disclosure. In some embodiments, the model search system 200 is trained to receive a set of input data 202 representing a search query and, as a result of receiving the input data 202, provide output data 206 including one or more search results. Thus, in some embodiments, the model search system 200 can include a search engine 204 operable to process the search query and determine the intent.

[0050] An exemplary model search system 200 can include a search engine 204 that obtains a search query 202 as input and outputs a search result 206 that can include one or more location-specific models. The search query 202 can be a location search query associated with a certain location. The search engine 204 can process the query to determine the location associated with the search query 202. The search engine can then access a location database 208 to obtain data related to that location, and can also access a three-dimensional asset database 210 to determine whether that location has an associated three-dimensional model. If a three-dimensional model is found, the three-dimensional model can be segmented to generate a location-specific model, which can be output as the search result 206. In some embodiments, the search result 206 can further include one or more links based on the search query 202 and can include location data obtained from the location database 208.

[0051] FIG. 3 shows a block diagram of an exemplary situation-aware model search system 300 according to an exemplary embodiment of the present disclosure. The situation-aware model search system 300 is similar to the model search system 200 of FIG. 2, except that it further includes situation-aware search and rendering.

[0052] Exemplary situation - aware model search system 300 can include a search engine 304 that takes as input a search query 302 and outputs a search result 306 that can include one or more location - specific models. The search query 302 can be a location search query associated with a location. The search engine 304 can process the query to determine the location associated with the search query 302. The search engine can then access a location database 308 to obtain data related to that location and access 3D asset data 310 to determine whether that location has an associated 3D model. If a 3D model is found, the 3D model can be segmented to generate a location - specific model for separating the location, which can be output as the search result 306. In some embodiments, the search result 306 can further include one or more links based on the search query 302 and can include location data obtained from the location database 308. In some embodiments, the search engine can further access time - specific situation data related to the location using situation query processing 312, which can be used to determine the time - specific situation. Once the time - specific situation is determined, a simulation asset database 314 can be accessed to obtain a simulation rendering result representing the determined situation. The location - specific model and the simulation rendering result can be used to generate a simulated event that is output as the search result. The simulated event can include a 3D model of the location in a state where the situation is rendered on the model. For example, the search query can be related to the Washington Monument, and the time - specific situation can include the current weather condition including rain. Thus, the simulated event can be a model of the Washington Monument with rain rendered on the model.

[0053] FIG. 4 shows a block diagram of an exemplary result page 400 according to an exemplary embodiment of the present disclosure. In some embodiments, the result page 400 includes a search query input bar 402, and in response to inputting a search query into the search query input bar 402, one or more search results 404 that can include links, text, and / or media can be provided according to the search query input. In some embodiments, the result page 400 can include a knowledge graph 410 operable to provide a set of data related to one or more search results.

[0054] The exemplary result page 400 of FIG. 4 can be output in response to a search query. The search query can be a location search query or a search query input into a web search service. The search result page 400 can include a search query input bar 402 for entering a new query or refining the current query. In some embodiments, the search query input bar 402 can continue to display the last search query or can be blanked out upon obtaining search results. Further, the search result page 400 can further include one or more search results 404. The search results 404 can include links, images, excerpts, maps, videos, and / or documents related to the determined search intent. The search result page 400 can include a knowledge graph 410, which can include one or more media items 412, a location-specific model 414, a name of the location 416, and location information 418. The knowledge graph 410 can be generated based on predefined functions associated with the location, one or more machine learning models, and / or a predefined knowledge graph. The one or more media items 412 can be images, videos, graphs, maps, or audio data related to the location. Further, the location-specific model 414 can be generated by obtaining and segmenting a 3D model in response to obtaining a search query. The name of the location 416 can be the official name of the location determined from the search query or an alternative name. Finally, the location information 418 can include various information about the population, history, architecture, or size of the location. The location information 418 can be extracted from one or more search results or can be data paired with the name of the location in a location database.

[0055] FIG. 5 shows a diagram of an exemplary 3D rendering 500 according to an exemplary embodiment of the present disclosure. In some embodiments, the 3D rendering 500 can provide various perspective views that can include an aerial view.

[0056] Exemplary 3D rendering 500 of FIG. 5 displays one view of a location. The rendering 500 can be displayed in response to a search query and can be represented based on a location-specific model provided. The location-specific model can be provided in an interactive environment to enable the user to navigate the location-specific model to view different perspectives of the location, including an aerial view 502 or various side views 504 and 506. The rendering of different perspectives can be based on the location-specific model and can be performed on either a server computing system or a client computing system. Alternatively and / or in addition, the location-specific model can be provided as part of a pre-rendered media clip where the 3D rendering 500 and various perspectives 502, 504, and 506 can be part of a video, slide show, or other display medium. In some embodiments, the location-specific model can be provided as part of a virtual reality experience, and thus the 3D rendering 500 can be one of a plurality of renderings viewed by the user while navigating through a virtual space. Generating a location-specific 3D model can utilize geographical extraction and can also use the Unreal Engine to display the model.

[0057] FIG. 9A shows a diagram of an exemplary 3D rendering 900 according to an exemplary embodiment of the present disclosure.

[0058] The 3D rendering 900 can be based on a 3D model and can be any perspective rendering that can include one or more overhead views. The 3D rendering 900 of FIG. 9 is based on a location-specific model of the Space Needle in Seattle. The rendering 900 includes one perspective, but alternative renderings can be generated based on the location-specific model that include, but are not limited to, an aerial view 902 or multiple side views 904 and 906. In some embodiments, the rendering 900 can include a situation-specific rendering that provides a situation-specific simulation. For example, this specific rendering 900 shows the Space Needle during a cloudy day. However, Seattle is not always precipitation-free, nor is it always cloudy alternatively. A user may desire a simulation of how the Space Needle would look during various weather conditions. In some embodiments, the location-specific model can be provided to the user computing device without including external features or objects. The location-specific model can then be viewed completely independently, or the outer environmental features can be added by the user computing device based on user input.

[0059] FIG. 9B shows diagrams of exemplary 3D renderings 910, 920, and 930 according to an exemplary embodiment of the present disclosure. The 3D renderings 910, 920, and 930 are similar to the 3D rendering 900 of FIG. 9A except that the 3D renderings 910, 920, and 930 further include a situation-based simulation.

[0060] The systems and methods disclosed herein can be used to obtain or generate simulation data, which can be used to obtain simulation assets related to a situation described by situation data. The simulation assets can be used to generate events simulated using location-specific models, which can include simulations of specific situations. For example, FIG. 9B shows three simulated weather states generated using a location-specific model and simulation assets. A three-dimensional rendering is displayed on a user interface along with the location and weather state. The partially cloudy three-dimensional rendering 910 can provide a simulated partially cloudy weather state for that location, which can provide the user with a rendering that accurately depicts that location with that specific weather state. Further, the systems and methods disclosed herein can provide a preview of that location in various other weather states, such as a rainy weather state 920, or a heavy rain weather state 930. These renderings can be time-specific for the time input by the user, can be based on real-time situation states, or can be manually selected situation states. Alternatively, and / or in addition, the situation can be a human congestion situation, a traffic situation, and / or various other situations determined and simulated for the user.

[0061] FIG. 10 shows a diagram of an exemplary augmented reality experience according to an exemplary embodiment of the present disclosure. In some embodiments, the augmented reality experience 1020 is trained to receive a set of data representing a three-dimensional model and provide output data 1024 that can be a rendering of the three-dimensional model of the location as a result of receiving the input data. Thus, in some embodiments, the augmented reality experience 1020 can include a preview 1012 of the location that can be used to depict that location before the augmented reality experience is selected 1016.

[0062] The augmented reality experience 1020 can be one of many media that provide location-specific models. In this embodiment, the user computing device is provided with a user interface that enables the user to view the location-specific model 1012 in context 1010 or as an augmented reality rendering 1024. In this embodiment, the user interface provides the location-specific model in context 1010 with the name of the location and an excerpt 1014 of text having information about the location. The user interface can include a selectable icon 1016 for switching to an augmented reality experience 1020 that can use the location-specific model 1012 to render that location into the user's environment 1022. The augmented reality rendering 1024 can be placed in the user's environment 1022 via the augmented reality experience 1024 and viewed as a scaled model of that location. Further, the user interface can provide a selectable icon 1026 to exit the augmented reality experience 1020 and return to the location-specific model 1012 in context 1010.

[0063] FIG. 11 shows a diagram of an exemplary media results page 1100 according to an exemplary embodiment of the present disclosure. In some embodiments, the media results page 1100 can include a set of images 1104, 1106, and 1108 of a location, as well as a location-specific three-dimensional model 1110. Further, in some embodiments, the media results page 1100 can include an interactive environment that can be used to view and interact with the location-specific three-dimensional model 1110.

[0064] In FIG. 11, the media results page 1100 includes a plurality of image results 1104, 1106, and 1108, a search query 1102, a three-dimensional model of the location, a map of the location, and a name 1112 of the location. In the illustrated embodiment, the search query 1102 includes the name of the location, but in other embodiments, the search query can include an address, a descriptor of the location, or any other query that can be associated with that location. The plurality of images 1104, 1106, and 1108 can be displayed as a side panel with selectable tabs at the top of the panel to improve the results or to show different forms of media. Further, the media results page 1100 can provide links to source data or other pages by selecting the media or various selectable icons.

[0065] The location-specific three-dimensional model 1110 can be provided in the main pane, a single pane, or a side pane. The location-specific model 1110 can be provided in an interactive environment or in a pre-rendered media form. In some embodiments, the media results page 1100 can include location information or location-specific data overlaid on one or more media items. For example, in this embodiment, the map of the location and the surrounding area, the name 1112 of the location, and information about the location are overlaid on the three-dimensional model 1110.

[0066] FIG. 12 shows a diagram of an exemplary route overlay model 1200 according to an exemplary embodiment of the present disclosure. In some embodiments, the route overlay model 1200 can include a location-specific model 1204 overlaid on a map and a visual route overlay 1208 showing a proposed navigation route to reach the location.

[0067] The route overlay model 1200 can provide an improved map application experience to the user by utilizing the location-specific model 1204, the map database, and the determined navigation route. The route overlay model can include the generation of the location-specific model 1204, the acquisition of the user computing device location, the acquisition of the location address, and the acquisition of the map. Along with the acquired map, the acquired user location and the address of the acquired location can be used to determine the route from the user's location to that address. The determined route can be used to generate a visual route overlay 1208 representing the navigation route. Using the map, the location-specific model 1204, and the visual route overlay 1208, an extended map representing the location-specific model 1204 and the visual route overlay 1208 superimposed on the map can be generated to provide the user with the navigation direction by adding visual nuances to the 3D model.

[0068] In this exemplary embodiment, the route overlay model 1200 can be part of a map application that can further include various features such as a route preview 1202, an ordered list of directions, location labels (e.g., transfer locations, restaurants, locations to stay, etc.), location markers (e.g., intermediate points, final destinations, etc.). The shown route overlay model 1200 includes a location-specific model (i.e., a transfer station) superimposed on the map having a visual route overlay 1208 that provides directions to the location marker 1206. The route overlay model can be provided via a user interface, which can include a 2D map and directions 1210 in conjunction with a 3D representation.

[0069] Exemplary method FIG. 6 shows a flowchart of an exemplary method for implementing in accordance with an exemplary embodiment of the present disclosure. FIG. 6 shows steps that are performed in a particular order for purposes of explanation and discussion, but the methods of the present disclosure are not limited to the particular order or configuration shown. The various steps of method 600 may be deleted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0070] At 602, the computing system can obtain a location query from a user computing device. The location query can be a search query entered by the user into the user interface.

[0071] At 604, the computing system can process the location query to determine a location.

[0072] At 606, the computing system can access a three-dimensional asset database to obtain a three-dimensional model associated with that location. The three-dimensional model can include one or more architectural structures and can include the determined location.

[0073] At 608, the computing system can generate a location-specific model by segmenting the three-dimensional model to separate that location from other objects in the three-dimensional model. The segmentation can be a three-dimensional segmentation or an image segmentation that removes trees, roads, and other buildings from the three-dimensional model to generate a location-specific model. The segmentation can include using a plurality of images of the location to segment the three-dimensional model, and the plurality of images can include one or more aerial images. In some embodiments, the location-specific model can include a model of an architectural structure (e.g., a landmark, monument, historical building, etc.).

[0074] At 610, the computing system can provide a location-specific model to the user computing device. The location-specific model can be provided as part of an interactive environment, a virtual reality experience, an augmented reality experience, and / or a pre-rendered media item. In some embodiments, the location-specific model can include location information, which can include the name of the location, the address of the location, an image of the location, and / or the actuality of the location.

[0075] FIG. 7 shows a flowchart of an exemplary method implemented in accordance with an exemplary embodiment of the present disclosure. Although FIG. 7 shows steps implemented in a particular order for purposes of explanation and discussion, the methods of the present disclosure are not limited to the particular order and configuration shown. The various steps of method 700 can be deleted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0076] At 702, the computing system can obtain a location query. The location query can include one or more search terms and / or one or more images.

[0077] At 704, the computing system can process the location query to determine a location. The location can be determined using a search engine, a location tag, and / or other determination methods.

[0078] At 706, the computing system can access a three-dimensional asset database, obtain a three-dimensional model, and generate a location-specific segment. The three-dimensional asset database can utilize a large database of images to generate a plurality of three-dimensional models that are associated with a location and can be easily searchable. Generating a location-specific segment can include removing objects that are not part of a particular determined location from the three-dimensional model. In some embodiments, the location-specific segment can be generated by a segmentation process that isolates the location-specific model by removing portions of the three-dimensional model.

[0079] At 708, the computing system can obtain situation data. The situation data can represent a time-specific situation associated with a determined location. The time-specific situation can be a weather condition, a traffic level, a crowd density, and / or a train location.

[0080] At 710, the computing system can process the situation data to determine each simulation asset. Each simulation asset can be a set of data that enables rendering a particular situation into a simulated event.

[0081] At 712, the computing system can obtain each simulation asset. The simulation asset can be obtained from a database of simulation assets that stores a plurality of situation-specific simulation assets.

[0082] At 714, the computing system can generate simulation events. The simulation events can include location- and time-specific situations based on location-specific segments and each simulation asset. For example, the location can be the Golden Gate Bridge, and the time-specific situation can be heavy traffic and rain. The location-specific segment can be a 3D model of the Golden Gate Bridge, and the simulated event can be the Golden Gate Bridge with heavy traffic and rain rendered onto the model.

[0083] FIG. 8 shows a flowchart of an exemplary method implemented in accordance with an exemplary embodiment of the present disclosure. Although FIG. 8 shows steps implemented in a particular order for explanation and discussion, the methods of the present disclosure are not limited to the particular order and configuration shown. The various steps of method 800 can be deleted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0084] At 802, the computing system can reconstruct a 3D environment to generate a 3D mesh and textures. The reconstruction can include a database of collected map data, image data, and other location data.

[0085] At 804, the computing system can acquire a plurality of images of the environment.

[0086] At 806, the computing system can process the plurality of images using a machine learning model and segment the plurality of images into parts. The segmentation can be used to understand the mesh and textures and find the underlying semantic structure of the environment. The segmentation can utilize the semantic diversity for image segmentation. A graph cut algorithm can be applied to truly segment the mesh into parts.

[0087] At 808, the computing system can project the image portion onto a 3D mesh.

[0088] At 810, the computing system can remove one or more objects from the reconstructed environment to generate an extended environment. The one or more objects can include other building structures, trees, sidewalks, and other obstacles. The removal can be completed automatically or can include obtaining user input to remove one or more features in the mesh.

[0089] At 812, the computing system can store the extended environment. The extended environment can be compressed before being stored and can be stored in a 3D asset database.

[0090] Further Disclosure The techniques discussed herein are referenced with respect to servers, databases, software applications, and other computer-based systems, as well as actions performed and information sent between such systems. The inherent flexibility of computer-based systems allows for a very wide variety of possible configurations, combinations, and divisions of tasks and functions among the components. For example, the processes discussed herein can be implemented using a single device or component, or multiple devices, or components operating in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. The distributed components can operate sequentially or in parallel.

[0091] Although the subject matter has been described in detail with respect to its various specific exemplary embodiments, each example is provided for illustrative purposes only and is not intended to limit the disclosure. Those skilled in the art will be able to readily create modifications, variations, and equivalents to such embodiments when the understanding of the foregoing has been achieved. Accordingly, the disclosure is not intended to exclude such changes, variations, and / or additions to the subject matter as will be readily apparent to those skilled in the art. For example, features shown or described as part of one embodiment can be used with another embodiment to yield a further embodiment. Accordingly, the disclosure is intended to embrace such modifications, variations, and equivalents.

Explanation of Signs

[0092] 100 Computing system 110 Server, server computing system 112 Processor 114 Memory 116 Instructions 118 Geographic data 120 3D model search 122 Image segmentation element 130 Client device 132 Processor 134 Memory 140 Browser element 142 Application element 146 Display 148 Positioning system 150 User input device 152 Graphics processing unit 154 Network interface 160 Network 180 3D asset database 200 Model search system 202 Input data 204 Search engine 206 Output, search results 208 Location Database 210 3D Asset Database 300 Situation Recognition Model Search System 302 Search Query 304 Search Engine 306 Search Results 308 Location Database 310 3D Asset Database 312 Situation Query Processing 314 Simulation Asset Database 400 Result Page 402 Search Query Input Bar 404 Search Results 410 Knowledge Graph 412 Media Item 414 Location-Specific Model 416 Location Name 418 Location Information 500 3D Rendering 502 Aerial View 504 Side View 506 Side View 600 Method 700 Method 800 Method 900 3D Rendering 902 Aerial View 904 Side View 906 Side View 910 3D Rendering 920 3D Rendering, Cloudy Weather Condition 930 3D Rendering, Heavy Rain Weather Condition 1010 Context 1012 Location Preview, Location-Specific Model 1014 Text Excerpt 1016 Selectable Icon 1020 Augmented Reality Experience 1022 User Environment 1024 Output Data, Augmented Reality Rendering 1026 Selectable Icon 1100 Media Results Page 1102 Search Query 1104 Location Image, Location-Specific Model 1106 Location Image 1108 Location Image 1110 Location-Specific 3D Model 1112 Location Name 1200 Route Overlay Model 1202 Route Preview 1204 Location-Specific Model 1206 Location Marker 1208 Visual Route Overlay 1210 Direction

Claims

1. 1. A computer-implemented method for providing a three-dimensional modeling to a user, comprising: obtaining, by a computing system comprising one or more computing devices, a location query associated with a user computing device; processing, by the computing system, the location query to determine a location; accessing, by the computing system, a three-dimensional asset database to obtain a three-dimensional model that models the location in combination with one or more other different locations; segmenting, by the computing system, the three-dimensional model to generate a location-specific segment of the three-dimensional model that corresponds to the location, the location-specific segment modeling the location in isolation from the one or more other distinct locations; providing, by the computing system, the location-specific segment of the three-dimensional model, or one or more renderings of the location-specific segment of the three-dimensional model, to the user computing device; A computer-implemented method comprising:

2. The method of claim 1 , wherein the location-specific segments of the three-dimensional model are generated by image segmentation using a plurality of images associated with the location.

3. The method of claim 1 or 2, wherein the location-specific segments of the three-dimensional model are generated based at least in part on sensor data representative of the location.

4. The method of claim 1 , wherein the location-specific segments of the three-dimensional model include architectural structures or landmarks.

5. The method of claim 1 , wherein the location-specific segments of the three-dimensional model are augmented reality assets.

6. The method of claim 1 , further comprising providing a virtual reality experience including the location-specific segment of the three-dimensional model.

7. The method of claim 1 , wherein the location-specific segments of the three-dimensional model are provided in an interactive environment.

8. The method of claim 7 , wherein the interactive environment includes the one or more renderings of the location-specific segments of the three-dimensional model combined with one or more two-dimensional maps.

9. The method of claim 7 or 8, wherein the interactive environment comprises a portion of a search result webpage that includes further search results responsive to the location query.

10. The method of claim 1 , wherein the location-specific segments of the three-dimensional model are provided in a pre-rendered video.

11. obtaining, by the computing system, an address of the location; providing, by the computing system, the address of a location having the location-specific segment of the three-dimensional model or the one or more renderings of the location-specific segment of the three-dimensional model; 11. The method of any one of claims 1 to 10, further comprising:

12. acquiring, by the computing system, context data, the context data representing a time-specific context associated with the location; processing, by the computing system, the situation data to determine each simulation asset; acquiring, by the computing system, each of the simulation assets; generating, by the computing system, a simulated event including the location-specific and the time-specific conditions based on the location-specific segments of the three-dimensional model and each of the simulation assets; 12. The method of claim 1, further comprising:

13. the situation data includes traffic data; The method of claim 12 , wherein each of the simulation assets comprises simulated traffic.

14. The situation data includes crowd data, The method of claim 12 or 13, wherein each said simulation asset includes a simulated crowd of people whose size is based on the crowd data.

15. The condition data includes weather data. The method of claim 12, 13, or 14, wherein each simulation asset includes a simulated weather condition.

16. 1. A computing system comprising: one or more processors; one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations including: receiving a location query from a user computing device; processing the location query to determine a location; accessing a three-dimensional asset database to obtain a three-dimensional model associated with the location; generating a location-specific model by segmenting the three-dimensional model to separate the location from other objects in the three-dimensional model; and providing the location-specific model to the user computing device. One or more non-transitory computer-readable media, A computing system comprising:

17. The operation includes: generating a map including the location-specific model; providing said map to said user computing device; The computing system of claim 16 further comprising:

18. The operation includes: obtaining location data for the user computing device, the location data representing a user location; obtaining an address for the location associated with the location-specific model; a user determining a route to reach said address from said user location; generating a visual route overlay based on the route; generating an augmented map, the augmented map including the visual route overlay and the map; providing said augmented map to a user computing device; The computing system of claim 17 further comprising:

19. One or more non-transitory computer-readable storage media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations including: obtaining user input specifying a location query relating to a location; providing the location query to a server system over a network; receiving a location-specific segment of a three-dimensional model, or one or more renderings of the location-specific segment of the three-dimensional model, from the server system over the network, the three-dimensional model modeling the location in combination with one or more other distinct locations, the location-specific segment of the three-dimensional model being generated by segmenting the three-dimensional model to separate the location from the one or more other locations; providing a display of at least a first rendering of the location-specific segment of the three-dimensional model; One or more non-transitory computer-readable storage media,

20. 20. The one or more non-transitory computer-readable storage media of claim 19, wherein providing a display of at least the first rendering of the location-specific segment of the three-dimensional model comprises providing a display of at least the first rendering in a search results webpage.

Images