Location-specific 3D models in response to location queries

Abstract

To provide better understanding of a location by providing better interactivity, better visibility and better understanding of dimensionality about the generation of a location-specific three-dimensional model in response to a location query.SOLUTION: Generation of a model can be enabled by utilizing a three-dimensional asset database and a segmentation method. A location-specific model can provide further practicability by further including situation-specific simulated results, such as simulated weather or traffic.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] Related Applications This application claims the priority and benefit of U.S. Non-Provisional Patent Application No. 17 / 227,871, filed on April 12, 2021. U.S. Non-Provisional Patent Application No. 17 / 227,871 is hereby incorporated by reference in its entirety.

[0002] The present disclosure generally relates to providing a three-dimensional model of a location in response to a user query related to or seeking information about a location. More particularly, the present disclosure provides location-specific segments of a three-dimensional model of a location in response to a query regarding the location that seeks information associated with the location, such as, for example, a search query, a navigation query, a mapping query, a weather query, a review query, a visual query, and / or other user requests seeking information about the location.

Background Art

[0003] When searching for a location via a search engine, some existing systems can return results in the form of hyperlinks and / or generated images that may include text excerpts, photographs, or maps. For monuments and landmarks, these results often fail to depict the actual appearance and size of the monument or landmark. Furthermore, while images can provide a perspective of what the location looks like, images alone may fail to capture the dimensionality of the location. Additionally, videos of the location may also be limited for various reasons. In particular, search results lack interactivity, which can make it difficult to discover other aspects of the location (e.g., the context of the location or various perspectives of the location). For example, trying to see close-ups of different views or different parts of the location may require a tedious search rather than being provided through an intuitive, interactive interface.

[0004] Knowing the weather, traffic, and crowd levels at monuments and landmarks can help travelers and other visitors better understand and prepare for their visit. Maps alone fail to take traffic or crowd levels into account. Furthermore, the visibility of a monument or landmark can be affected by crowd levels or weather. Knowledge about the impact of these factors on visibility is often limited to sweeping through the literature on the object. However, many people consider themselves to be visual learners. [Overview of the project] [Means for solving the problem]

[0005] Aspects and advantages of the embodiments of this disclosure are, in part, described below, can be learned from that description, or can be learned through the practice of the embodiments.

[0006] An exemplary aspect of this disclosure relates to a computer-implemented method for providing a user with a three-dimensional model. The method includes the step of obtaining a location query associated with a user computing device by a computing system which may include one or more computing devices. The method may include the computing system processing the location query and determining a location. In some embodiments, the method may include the computing system accessing a three-dimensional asset database and obtaining a cubic model that models a location in combination with one or more other different locations. The method may include the computing system segmenting a three-dimensional model to generate location-specific segments of the three-dimensional model corresponding to a location. In some embodiments, the location-specific segments may be separated from one or more other different locations to model that location. The method may include the computing system providing a user computing device with a location-specific segment of the three-dimensional model, or one or more renderings of a location-specific segment of the three-dimensional model.

[0007] As will be discussed in more detail below, the step of segmenting a 3D model to generate location-specific segments can offer numerous technical advantages. In particular, location-specific segments, rather than the entire 3D model, can be provided to the user computing device. This reduces the bandwidth requirements for sending relevant data to the user device and also reduces the computational resource requirements on the user device for storing and rendering the location-specific model. In other words, the step of segmenting a 3D model reduces the amount of data that needs to be sent to the user computing device, thereby enabling a more computationally efficient way of providing the 3D model to the user.

[0008] Another exemplary aspect of this disclosure relates to a computing system. The computing system may include one or more processors and one or more non-temporary computer-readable media that collectively store instructions causing the computing system to perform an action when executed by the one or more processors. The action may include the step of obtaining a location query from a user computing device. The action may include the step of processing a location to determine the location. In some embodiments, the action may include the step of accessing a three-dimensional asset database to obtain a three-dimensional model associated with that location. The action may include the step of generating a location-specific model by segmenting the three-dimensional model to isolate the location from other objects in the three-dimensional model, and the step of providing the location-specific model to a user computing device.

[0009] Another exemplary aspect of this disclosure relates to one or more non-temporary computer-readable media that collectively store instructions causing a computing system to perform an action when executed by one or more processors. The action may include the step of obtaining user input specifying a location query about a location. The action may include the step of providing a location query to a server system over a network. In some embodiments, the action may include the step of receiving a location-specific segment of a 3D model, or one or more renderings of location-specific segments of a 3D model, from the server system over a network. The 3D model may model a location in combination with one or more other different locations, and location-specific segments of a 3D model may be generated by segmenting the 3D model to isolate that location from one or more other different locations. The action may include the step of providing a display of at least a first rendering of a location-specific segment of a 3D model.

[0010] Other aspects of this disclosure cover a variety of systems, apparatus, non-temporary computer-readable media, user interfaces, and electronic devices.

[0011] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood by referring to the following description and the appended claims. The appended drawings incorporated herein and constituting part thereof illustrate exemplary embodiments of this disclosure and, together with this description, serve to illustrate the relevant principles.

[0012] A detailed discussion of embodiments for those skilled in the art is given herein with reference to the accompanying drawings. [Brief explanation of the drawing]

[0013] [Figure 1]This is a block diagram of an exemplary computing system that performs location-based model search according to an exemplary embodiment of the present disclosure. [Figure 2] This is a block diagram of an exemplary model search system according to an exemplary embodiment of the present disclosure. [Figure 3] This is a block diagram of an exemplary situational model search system according to an exemplary embodiment of the present disclosure. [Figure 4] This is a block diagram of an exemplary three-dimensional rendering according to an exemplary embodiment of the present disclosure. [Figure 5] This is a diagram of an exemplary search results page according to an exemplary embodiment of the present disclosure. [Figure 6] This is a flowchart illustrating an exemplary method for performing location-based model search according to exemplary embodiments of the present disclosure. [Figure 7] This is a flowchart illustrating an exemplary method for performing location-based model search according to exemplary embodiments of the present disclosure. [Figure 8] This is a flowchart illustrating an exemplary method for creating a three-dimensional model database according to an exemplary embodiment of the present disclosure. [Figure 9A] This figure shows an exemplary three-dimensional model according to an exemplary embodiment of the present disclosure. [Figure 9B] This figure shows an exemplary situation-responsive 3D model according to an exemplary embodiment of the present disclosure. [Figure 10] This figure shows an exemplary augmented reality experience according to an exemplary embodiment of the present disclosure. [Figure 11] This figure shows an exemplary rendering display according to an exemplary embodiment of the present disclosure. [Figure 12] This figure shows an exemplary path overlay according to an exemplary embodiment of the present disclosure. [Modes for carrying out the invention]

[0014] The reference figures that appear repeatedly across multiple figures are intended to identify the same function in various embodiments.

[0015] overview In general, this disclosure covers providing location-specific segments of a 3D model of a location in response to location-related queries that seek information associated with that location, such as search queries, navigation queries, mapping queries, weather queries, review queries, visual queries, and / or other user requests seeking information. The systems and methods disclosed herein can utilize a database of 3D models to provide location-based renderings and / or location-specific models in response to location-related queries ("location queries"). In some embodiments, the systems and methods may include obtaining location queries. Location queries may be obtained from or triggered by a user's computing device. The systems and methods may include processing location queries to determine a location. In some embodiments, the systems and methods may include accessing a 3D asset database to obtain a 3D model associated with a location. The 3D models and locations may be used to generate location-specific models. Generating location-specific models may include segmenting the 3D model to isolate the location from other objects in the 3D model. The systems and methods may include providing location-specific models to a computing device. For example, location-specific models may be provided to a user's computing device. In some embodiments, location-specific models may include architectural structures (e.g., landmarks, monuments, and / or buildings).

[0016] Obtaining a location query can be easily done by a user interface configured to receive input from a user. Alternatively, and / or in addition, the location query can be a query generated and obtained by a computing system in response to an event triggered by a user computing device. For example, a user can scroll through a list of attractions, select a particular attraction, and learn more about that attraction. In response to the selection, a location query for the attraction can be generated and obtained for processing. In other examples, a user can request to view that location on a map, request navigational instructions to that location, request information about points of interest associated with the location (e.g., reviews generated by the user), present a textual or visual search query that yields that location, and / or perform other actions indicative of a request for information about that location. Each of these exemplary actions can trigger a location query to be processed by the proposed system.

[0017] In some embodiments, a location query can include a series of terms or one or more images associated with a particular location. The terms can include the name of the location that can describe the location, can be the address of the location, or can be any other form of information that is indirectly or directly related and implies or results in a location being relevant. In some examples, a location query can include an image that processes and determines where a photo was taken or what is depicted in the image.

[0018] After a location has been determined, a three-dimensional asset database can be accessed and searched to determine whether that particular location has any associated three-dimensional model. The three-dimensional model can be a three-dimensional model of a particular structure or structures found at that location. In particular, in some embodiments, the three-dimensional asset database can include general or global three-dimensional models that cover a significant portion of the Earth's surface, or of an individual country or city located therein. For example, various algorithms can be applied to automatically generate one or more very large three-dimensional models spanning various addresses, streets, neighborhoods, cities, and / or the like from available images (e.g., satellite images and / or street-level images).

[0019] To provide a location-specific model, the system and method can separate the location in the general three-dimensional model to generate a location-specific model. Specifically, an exemplary system of the present disclosure can segment a portion of the general three-dimensional model into a plurality of segments. In some embodiments, the system and method can perform three-dimensional segmentation to extract location-specific segments of the three-dimensional model to model one or more particular locations. The location-specific model can be generated using segmentation of an image. In this way, a smaller model containing the required location can be obtained. The smaller model can reduce the processing power required for storage and rendering, and can also reduce the bandwidth required to transmit the model to the user's computing device. Furthermore, the location-specific model can be easier for the user to access and understand with respect to the general three-dimensional model. Thus, the location-specific model can enable improved interaction with the user.

[0020] After location-specific segments of a model are generated, these segments can be rendered and displayed to the user in various situations. For example, a server computing system can transmit or supply location-specific segments of a 3D model to a user computing device, and the user computing device can then display a rendering of that location using the location-specific model. Alternatively, and / or in addition, the server computing system can perform the rendering of location-specific segments of a 3D model and transmit or supply the rendering to the user computing device.

[0021] In some embodiments, location-specific segments of a 3D model can be generated by image segmentation using multiple images associated with that location. These multiple images may include one or more aerial images. Alternatively, and / or in addition, in some embodiments, the model can be generated using light detection and ranging (LiDAR) data or other sensor data (e.g., spectral sensor data generated using a spectral sensor, which can be used to segment trees or other objects). In particular, LiDAR data or other sensor data can be used to determine which segments should be segmented.

[0022] Location-specific segments of a 3D model can be provided in various ways and through various media. In some embodiments, a location-specific model may be provided as an augmented reality asset rendered in the user's environment via an augmented reality experience. Alternatively, and / or in addition, a location-specific model may be provided in a virtual reality experience. In some embodiments, a user computing device may have an interactive virtual environment that can contain a location-specific model. Furthermore, the interactive environment may include one or more 2D or 3D maps. A location-specific segment of a 3D model may be provided as a pre-rendered video, or a set of pre-rendered videos, in which case each video may depict different appearances of the location generated based on the location-specific segment.

[0023] In some embodiments, a location-specific model may include other information or data related to or associated with a location. For example, the system and method may include obtaining the address of the location and providing the address of the location having a location-specific model. In some embodiments, the system and method may include obtaining location information data. The location information data may include information related to the location (e.g., information about the location's history). The location information data may be provided to the user computing device using the location-specific model. In some embodiments, a location-specific model may include multiple images of the location. Furthermore, in some embodiments, the system and method may include generating a map containing the location-specific model and providing the map to the user computing device. Furthermore, in some embodiments, the system and method may include obtaining location data for the user computing device. The location data may describe the user's location. The system and method may include obtaining the address for a location associated with the location-specific model. The address and the user's location may be used to determine the route the user takes to get from the user's location to that address. A visual route overlay may be generated based on the route, and the visual route overlay and map may be used to generate an augmented map using the visual route overlay superimposed on the map. The extended map can then be provided to the user's computing device. Such an extended map has the technical effect of providing an efficient and effective means of navigating to its location.

[0024] Alternatively, and / or in addition, the system and method may include acquiring situational data. Situational data may represent time-specific conditions associated with a location. The system and method may include processing situational data and acquiring each simulation asset in order to determine each simulation asset. Each simulation asset and location-specific model can be used to generate simulated events that include time-specific conditions occurring at that location. In some embodiments, conditions may include weather conditions, as well as the locations of various physical items such as trains, aircraft, buses, taxis, and / or similar, the level of human traffic at or near the location, and / or traffic. For example, the time-specific condition may be rain, and the generated simulated event may be a rendering of a location-specific segment of a 3D model with simulated rain. More generally, simulated events that include time-specific conditions occurring at a location can provide technical effects beyond the embodiment of the simulation itself. Specifically, simulated events allow users to visualize the location in various time-specific conditions, enabling users to gain a better understanding of the location, thereby enabling them to navigate to the location themselves in real life.

[0025] In some embodiments, providing a location-specific model may include providing a location-specific model with one or more search results via a web page (e.g., a web page returned in response to a textual or visual location query). Alternatively, and / or in addition, a location-specific model may be provided and displayed in a section or location on a separate web page from other search results. For example, in some embodiments, a location-specific model may be provided in a separate panel along with other data at the location associated with the model (e.g., the name of the location associated with the location-specific model, the address of the location, and / or information about the location). In some embodiments, mesh segmentation can make the response to a query more compact. Specifically, a mesh segment can be rendered more quickly than rendering the entire region (e.g., in a panel). Rendering a segmented area can therefore help provide search results without the heavy bandwidth burden that rendering the complete model might incur.

[0026] The systems and methods disclosed herein can be used in a variety of fields and industries. For example, the systems and methods disclosed herein can be used to provide an improved user experience for map applications. More specifically, the systems and methods disclosed herein can be used to provide a user with a map having a location-specific model in response to a location query. The map application can provide a visual path overlay to provide a realistic rendering of the area to be navigated through. In some embodiments, the map application can provide several options for understanding a given area. The map application can provide a street view, overhead map, aerial view map, or interactive 3D model of the location. In this way, the map application makes it easier for the user to get to that location.

[0027] Another example could include implementing the system and method into a video game and / or an augmented or virtual reality experience. For example, the system and method disclosed herein could enable the creation of an open-world game in which a user can travel to various areas, and upon reaching an area, can send a location query and receive a location-specific model of that area accordingly. More specifically, the game could be set in France, and when a character travels to a location, can send a location query to the server for that location and receive a location-specific model of that location in return (for example, the character can travel to the Eiffel Tower, send a location query for the Eiffel Tower, and receive a location-specific model of the Eiffel Tower in return).

[0028] The 3D asset databases used by the systems and methods disclosed herein can be generated using a variety of methods or processes. The process can begin with the creation of 3D content. A 3D environment can be constructed using a general 3D reconstruction method. Reconstruction can generate a 3D mesh with accompanying textures (e.g., a 2D image representing the mesh's color). The next step in the process may include image segmentation. Segmentation can be used to understand the mesh and textures and find the underlying semantic structure of the environment (e.g., the semantic structure of a reconstructed city). In some embodiments, the process can leverage semantic diversity for image segmentation. Image segmentation may include segmenting multiple 2D images into parts using a machine learning model. The segmented images can then be projected onto a 3D mesh. The process may further include cleaning the generated 3D environment by cleaning the mesh, removing blobs. For example, trees, streets, sidewalks, and other objects can be removed. The cleaning step can be completed manually or automated. The resulting 3D assets can be compressed and stored in a database. The process can be iterated over and over to update the database with the most up-to-date structures, such as cities and towns.

[0029] The systems and methods of this disclosure offer several technical effects and benefits. For example, the systems and methods can provide location-specific segments of a 3D model in response to location queries. The systems and methods can further be used to provide models of locations (e.g., landmarks or monuments) that have real-time traffic, pedestrian congestion, weather, and / or other background or situation-specific information. Furthermore, the systems and methods can enable 3D models for mapping applications.

[0030] Another technical benefit of the systems and methods of this disclosure is the dimensionally accurate modeling and interactivity of the renderings provided. The models can provide dimensionally accurate representations of locations, allowing potential visitors to decide whether they wish to actually visit that location. Furthermore, dimensionally accurate representations enable users to navigate to that location more effectively.

[0031] Another exemplary technical effect and benefit relates to improved computational efficiency and enhanced functionality in computing systems. For example, some existing systems allow users to view renderings of global or general 3D models. Processing and rendering an entire global or general 3D model requires significant computational resources. In contrast, by extracting and then rendering location-specific segments of a 3D model, the proposed systems and methods can conserve computational resources such as processor usage, memory usage, and / or network bandwidth. In particular, location-specific segments require less bandwidth to be transmitted to the user device compared to transmitting a general 3D model. Furthermore, location-specific segments can reduce memory usage on the user device and reduce processor usage to render the location-specific segments.

[0032] Exemplary embodiments of the present disclosure will be discussed in further detail with reference to the following figures.

[0033] Exemplary devices and systems Figure 1 shows an exemplary computing system 100 that may be used to perform location-based model search according to various aspects of the present disclosure. The system 100 has a client-server architecture that includes a server 110 that communicates with one or more client devices 130 over a network 160. However, the present disclosure may also be implemented using other suitable architectures, such as a single computing device that is not connected to a network.

[0034] System 100 includes, for example, a server 110, such as a web server. Server 110 can be one or more computing devices implemented as a parallel or distributed computing system. In particular, multiple computing devices can operate together as a single server 110. Server 110 may have one or more processors 112 and memory 114. Server 110 may also include a network interface used to communicate with one or more remote computing devices (e.g., client devices) 130 via a network 160.

[0035] The processor 112 can be any suitable processing device, such as a microprocessor, microcontroller, integrated circuit, or other suitable processing device. The memory 114 can include any suitable computing system or medium, but is not limited to non-temporary computer-readable media, RAM, ROM, hard drives, flash drives, or other memory devices. The memory 114 can store information accessible by the processor 112, including instructions 116 that can be executed by the processor 112. The instructions 116 can be any set of instructions that, when executed by the processor 112, cause the processor 112 to provide desired functionality.

[0036] In particular, instruction 116 can be executed by processor 112 to perform a 3D model search 120. The 3D model search 120 may be configured to search for 3D models associated with a location, segment the location-specific 3D models, and output them. In some embodiments, the 3D model search 120 may be configured to access a 3D asset database 180 to retrieve 3D models, and the 3D models may be segmented using image segmentation elements 122 stored in the memory 114 of the server computing system 110.

[0037] It will be understood that the term “element” can refer to computer logic used to provide desired functionality. Therefore, any element, function, and / or instruction may be implemented in hardware controlling a general-purpose processor, application-specific circuitry, firmware, and / or software. In one embodiment, an element or function may be provided from a computer program product, such as a program code file stored in a storage device, loaded into memory, and executed by the processor, or a computer executable instruction stored in a tangible computer-readable storage medium such as RAM, a hard disk, or an optical or magnetic medium.

[0038] Memory 114 may also contain geographic data 118 that can be acquired, manipulated, created, or stored by the processor 112. Geographic data 118 may include geographic images (e.g., digital maps, satellite imagery, aerial photographs, street-level photographs, composite models, etc.), tables, vector data (e.g., vector representations of roads, parcels, buildings, etc.), data of points of interest (e.g., regions such as islands, cities, restaurants, hospitals, parks, hotels, and schools, etc.), or other appropriate geospatial data or related information. For example, geographic data 118 can be used to access information and data associated with locations and to generate renderings of three-dimensional models of the Earth's surface.

[0039] The geographic data 118 may be stored in one or more databases. One or more databases may be connected to server 110 via a high-bandwidth LAN or WAN, or they may be connected to server 110 via network 160. One or more databases may be divided to be locally located in multiple locations.

[0040] The server 110 can exchange data with one or more client devices 130 via the network 160. Although two clients 130 are shown in Figure 1, any number of client devices 130 can be connected to the server 110 via the network 160. The client devices 130 can be any suitable type of computing device, such as a general-purpose computer, a dedicated computer, a navigation device, a laptop, a desktop, an integrated circuit, a mobile device, a smartphone, a tablet, a wearable computing device, a display with one or more processors combined and / or embedded, or other suitable computing device. Furthermore, the client devices 130 can be multiple computing devices that work together to perform operations or computing actions.

[0041] Similar to the server 110, the client device 130 may include a processor 132 and memory 134. Memory 134 can store information accessible by the processor 132, including instructions and data that the processor can execute. For example, memory 134 can store browser elements 140 and application elements 142.

[0042] Browser element 140 can provide instructions for performing a browser. In particular, a user of client device 130 can exchange data with server 110 by using the browser to visit a website accessible at a specific web address. The 3D model search of this disclosure may be provided as an element of the user interface of a website.

[0043] Application element 142 can provide instructions for running a dedicated application on the client device 130. In particular, the dedicated application can be used to exchange data with the server 110 over the network 160. Application element 142 may include client device-readable code for providing and implementing aspects of the present disclosure. For example, application element 142 can provide instructions for implementing a mapping application or a virtual earth application.

[0044] The client device 130 may include various user input devices 150 for receiving information from the user, such as a touchscreen, touchpad, data entry keys, speaker, mouse, motion sensor, and / or microphone suitable for voice recognition. Furthermore, the client device 130 may have a display 146 for presenting information, such as rendering a 3D model in a 3D environment.

[0045] The client device 130 may also include a positioning system 148 that can be used to determine the location of the client device 130. The positioning system 148 may optionally be used by the user to monitor the user's location relative to the rendering. The positioning system 148 can be any device or circuit for monitoring the location of the client device 130. For example, the positioning device 148 may determine its actual or relative location by using satellite navigation positioning systems (e.g., GPS system, Galileo positioning system, Global Navigation Satellite System (GLONASS), Beidou satellite navigation and positioning system, etc.), inertial navigation systems, dead reckoning systems, triangulation and / or proximity to a cellular tower or WiFi hotspot based on an IP address, and / or by using other appropriate techniques for determining location.

[0046] The client device 130 may further include a graphics processing unit 152. The graphics processing unit 152 is used by the processor 132 to render or depict three-dimensional images. In some embodiments, the client device 130 performs any or all of the three-dimensional rendering necessary to provide a three-dimensional environment.

[0047] The client device 130 may include a network interface 154 for communicating with the server 110 via the network 160. The network interface 154 may include any components or configurations suitable for communicating with the server 110 via the network 160, such as one or more ports, transmitters, wireless cards, controllers, physical layer components, or other items for communication according to any currently known or future-developed communication protocols or technologies.

[0048] Network 160 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or any combination thereof. Network 160 can also include a direct connection between client devices 130 and server 110. Generally, communication between server 110 and client devices 130 can be transmitted over the network interface using any type of wired and / or wireless connection, with various communication protocols (e.g., TCP / IP, HTTP), encoding or formatting (e.g., HTML, XML), and / or protection methods (e.g., VPN, Secure HTTP, SSL).

[0049] Exemplary model configuration Figure 2 shows a block diagram of an exemplary model search system 200 according to an exemplary embodiment of the present disclosure. In some embodiments, the model search system 200 receives a set of input data 202 representing a search query and is trained to provide output data 206 containing one or more search results as a result of receiving the input data 202. Thus, in some embodiments, the model search system 200 may include a search engine 204 that can process the search query and determine intent.

[0050] An exemplary model search system 200 may include a search engine 204 that takes a search query 202 as input and outputs a search result 206 which may include one or more location-specific models. The search query 202 may be a location search query associated with a certain location. The search engine 204 may process the query to determine the location associated with the search query 202. The search engine may then access a location database 208 to retrieve data related to that location, and may also access a 3D asset database 210 to determine whether that location has an associated 3D model. If a 3D model is found, the 3D model may be segmented to generate location-specific models, which may be output as a search result 206. In some embodiments, the search result 206 may further include one or more links based on the search query 202 and may include location data retrieved from the location database 208.

[0051] Figure 3 shows a block diagram of an exemplary context-aware model search system 300 according to an exemplary embodiment of the present disclosure. The context-aware model search system 300 is similar to the model search system 200 of Figure 2, except that it further includes context-aware search and rendering.

[0052] An exemplary context-aware model search system 300 may include a search engine 304 that takes a search query 302 as input and outputs a search result 306 which may include one or more location-specific models. The search query 302 may be a location search query associated with a location. The search engine 304 may process the query to determine the location associated with the search query 302. The search engine may then access a location database 308 to retrieve data associated with that location and access 3D asset data 310 to determine whether that location has an associated 3D model. If a 3D model is found, the 3D model may be segmented to isolate the location and generate a location-specific model which may be output as the search result 306. In some embodiments, the search result 306 may further include one or more links based on the search query 302 and may also include location data retrieved from the location database 308. In some embodiments, the search engine may use context query processing 312 to further access time-specific context data associated with the location which can be used to determine the time-specific context. Once time-specific conditions are determined, the simulation asset database 314 may be accessed to obtain simulation rendering results representing the determined conditions. Location-specific models and simulation rendering results may be used to generate simulated events that are output as search results. Simulated events may include a 3D model of the location in which the conditions are rendered in the model. For example, the search query may be related to the Washington Monument, and the time-specific conditions may include current weather conditions, including rain. Thus, the simulated event may be a model of the Washington Monument, with rain rendered in that model.

[0053] Figure 4 shows a block diagram of an exemplary results page 400 according to an exemplary embodiment of the present disclosure. In some embodiments, the results page 400 includes a search query input bar 402, and as a result of entering a search query into the search query input bar 402, it can provide one or more search results 404 which may include links, text, and / or media, depending on the search query input. In some embodiments, the results page 400 may include a knowledge graph 410 which is operable to provide a set of data relating to one or more search results.

[0054] The exemplary results page 400 in Figure 4 may be output depending on the search query. The search query can be a location search query, or a search query entered into a web search service. The search results page 400 may include a search query input bar 402 for entering a new query or refining the current query. In some embodiments, the search query input bar 402 may continue to display the last search query, or it may be blank when search results are obtained. Furthermore, the search results page 400 may further include one or more search results 404. The search results 404 may include links, images, excerpts, maps, videos, and / or documents related to the determined search intent. The search results page 400 may include a knowledge graph 410, which may include one or more media items 412, a location-specific model 414, a location name 416, and location information 418. The knowledge graph 410 may be generated based on a predefined function associated with that location, one or more machine learning models, and / or a predefined knowledge graph. One or more media items 412 can be images, videos, graphs, maps, or audio data related to the location. Furthermore, a location-specific model 414 may be generated by obtaining and segmenting a three-dimensional model in response to a search query. The location name 416 can be the official name of the location determined from the search query, or a paraphrased name. Finally, the location information 418 can include various information about population, history, architecture, or the size of the location. The location information 418 can be extracted from one or more search results, or it can be data paired with the location name in a location database.

[0055] Figure 5 shows an exemplary three-dimensional rendering 500 according to an exemplary embodiment of the present disclosure. In some embodiments, the three-dimensional rendering 500 can provide a variety of perspective views, which may include an aerial view.

[0056] The exemplary three-dimensional rendering 500 in Figure 5 displays one view of a location. The rendering 500 can be displayed in response to a search query and may be represented based on a location-specific model provided. The location-specific model may be provided in an interactive environment to give the user the ability to navigate the location-specific model to view each different perspective view of the location, including an aerial view 502 or various side views 504 and 506. Rendering of different perspective views may be based on the location-specific model and may be done on either a server computing system or a client computing system. Alternatively, and / or in addition, the location-specific model may be provided as part of a pre-rendered media clip, which may be part of a video, slideshow, or other display medium, along with the three-dimensional rendering 500 and the various perspective views 502, 504, and 506. In some embodiments, the location-specific model may be provided as part of a virtual reality experience, and thus the three-dimensional rendering 500 may be one of several renderings seen by the user while navigating the virtual space. Generating location-specific 3D models can utilize geographical extraction, and Unreal Engine can also be used to display the models.

[0057] Figure 9A shows an exemplary three-dimensional rendering 900 according to an exemplary embodiment of the present disclosure.

[0058] The 3D rendering 900 can be based on a 3D model and can be a rendering of any perspective view that may include one or more overhead views. The 3D rendering 900 in Figure 9 is based on a location-specific model of the Space Needle in Seattle. The rendering 900 includes one perspective view, but alternative renderings may be generated based on a location-specific model that include, but are not limited to, an aerial view 902 or multiple side views 904 and 906. In some embodiments, the rendering 900 may include a situation-specific rendering that provides a situation-specific simulation. For example, this specific rendering 900 displays the Space Needle during a cloudy day. However, Seattle is not always free of precipitation, nor is it always cloudy. The user may want a simulation of how the Space Needle looks during various weather conditions. In some embodiments, the location-specific model may be provided to the user computing device without including external features or objects. The location-specific model may then be viewable entirely independently, or external environmental features may be added by the user computing device based on user input.

[0059] Figure 9B shows exemplary three-dimensional renderings 910, 920, and 930 according to exemplary embodiments of the present disclosure. Three-dimensional renderings 910, 920, and 930 are similar to three-dimensional rendering 900 in Figure 9A, except that three-dimensional renderings 910, 920, and 930 further include situation-based simulations.

[0060] The systems and methods disclosed herein can be used to acquire or generate simulation data, which can be used to acquire simulation assets relating to situations described by situational data. The simulation assets can be used to generate simulated events using location-specific models, which may include simulations of specific situations. For example, Figure 9B displays three simulated weather conditions generated using location-specific models and simulation assets. Three-dimensional renderings are displayed in the user interface along with the location and weather conditions. The partially cloudy three-dimensional rendering 910 can provide a simulated partially cloudy weather condition for its location, which can provide the user with a rendering that accurately depicts its location having that particular weather condition. Furthermore, the systems and methods disclosed herein can provide previews of its location in various other weather conditions, such as rainy weather conditions 920 or heavy rain weather conditions 930. These renderings can be time-specific to a time entered by the user, based on real-time situational conditions, or based on manually selected situational conditions. Alternatively, and / or in addition, the situation can be a variety of other situations that are determined and can be simulated for the user, such as crowding, traffic conditions, and / or other factors.

[0061] Figure 10 shows a diagram of an exemplary augmented reality experience according to an exemplary embodiment of the present disclosure. In some embodiments, the augmented reality experience 1020 is trained to receive a set of data representing a three-dimensional model and, as a result of receiving the input data, provide output data 1024 which may be a rendering of the three-dimensional model of the location. Thus, in some embodiments, the augmented reality experience 1020 may include a preview 1012 of the location that is available to depict the location before the augmented reality experience is selected 1016.

[0062] The augmented reality experience 1020 can be one of many media that provide a location-specific model. In this embodiment, the user computing device is provided with a user interface that allows the user to view the location-specific model 1012 in context 1010 or as an augmented reality rendering 1024. In this embodiment, the user interface provides the location-specific model in context 1010 with the name of the location and a text excerpt 1014 containing information about the location. The user interface may include a selectable icon 1016 for switching to an augmented reality experience 1020 that can use the location-specific model 1012 to render that location into the user's environment 1022. The augmented reality rendering 1024 can be placed in the user's environment 1022 via the augmented reality experience 1024 and viewed as a scaled model of that location. Furthermore, the user interface may provide a selectable icon 1026 to exit the augmented reality experience 1020 and return to the location-specific model 1012 in context 1010.

[0063] Figure 11 shows an exemplary media results page 1100 according to an exemplary embodiment of the present disclosure. In some embodiments, the media results page 1100 may include a set of images 1104, 1106, and 1108 of a location, as well as a location-specific three-dimensional model 1110. Furthermore, in some embodiments, the media results page 1100 may include an interactive environment that can be used to view and interact with the location-specific three-dimensional model 1110.

[0064] The media results page 1100 in Figure 11 includes multiple image results 1104, 1106, and 1108, a search query 1102, a 3D model of the location, a map of the location, and the name of the location 1112. In the shown embodiment, the search query 1102 includes the name of the location, but in other embodiments, the search query may include an address, a location descriptor, or any other query that can be associated with that location. The multiple images 1104, 1106, and 1108 can be displayed as a side panel with selectable tabs at the top of the panel to improve the results or to show different forms of media. Furthermore, the media results page 1100 can provide links to source data or other pages by selecting media or various selectable icons.

[0065] The location-specific 3D model 1110 can be provided in the main pane, a standalone pane, or a side pane. The location-specific model 1110 may be provided in an interactive environment or in a pre-rendered media form. In some embodiments, the media results page 1100 may include overlaid location information or location-specific data on one or more media items. For example, in this embodiment, a map of the location and its surrounding area, the name of the location 1112, and information about the location are overlaid on the 3D model 1110.

[0066] Figure 12 shows an exemplary route overlay model 1200 according to an exemplary embodiment of the present disclosure. In some embodiments, the route overlay model 1200 may include a location-specific model 1204 overlaid on a map and a visual route overlay 1208 showing a proposed navigation route to reach the location.

[0067] The route overlay model 1200 can provide users with an enhanced map application experience by utilizing a location-specific model 1204, a map database, and a determined navigation route. The route overlay model can include generating the location-specific model 1204, obtaining the user computing device location, obtaining the location address, and obtaining the map. Along with the obtained map, the obtained user location and the obtained location address can be used to determine a route from the user's location to that address. The determined route can be used to generate a visual route overlay 1208 that represents the navigation route. Using the map, the location-specific model 1204, and the visual route overlay 1208, an augmented map can be generated that represents the location-specific model 1204 and the visual route overlay 1208 superimposed on the map, providing users with navigation directions with subtle visual differences in the 3D model.

[0068] In this exemplary embodiment, the route overlay model 1200 can be part of a map application that may further include various features such as a route preview 1202, an ordered list of directions, location labels (e.g., transfer locations, restaurants, accommodation locations, etc.), and location markers (e.g., intermediate points, final destinations, etc.). The shown route overlay model 1200 includes a location-specific model (i.e., transfer stations) overlaid on a map, having a visual route overlay 1208 that provides directions to location markers 1206. The route overlay model may be provided via a user interface that may include a two-dimensional map and directions 1210 in conjunction with a three-dimensional representation.

[0069] Exemplary Method Figure 6 shows a flowchart of an exemplary method for carrying out an exemplary embodiment of the present disclosure. While Figure 6 shows the steps to be carried out in a specific order for illustrative and discussion purposes, the methods of the present disclosure are not limited to the order or configuration shown. Various steps of Method 600 can be deleted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0070] In 602, the computing system can obtain a location query from a user computing device. The location query can be a search query entered by the user into the user interface.

[0071] In version 604, the computing system can process location queries and determine the location.

[0072] In 606, the computing system can access a 3D asset database to retrieve 3D models associated with its locations. These 3D models may include one or more building structures and may include their determined locations.

[0073] In 608, a computing system can generate a location-specific model by segmenting a 3D model in order to isolate its location from other objects. Segmentation can be 3D segmentation or image segmentation, in which trees, roads, and other structures are excluded from the 3D model in order to generate a location-specific model. Segmentation may involve using multiple images of the location to segment the 3D model, and these multiple images may include one or more aerial images. In some embodiments, the location-specific model may include a model of architectural structures (e.g., landmarks, monuments, historical buildings, etc.).

[0074] In 610, the computing system may provide a location-specific model to the user computing device. The location-specific model may be provided as part of an interactive environment, a virtual reality experience, an augmented reality experience, and / or a pre-rendered media item. In some embodiments, the location-specific model may include location information, which may include the name of the location, the address of the location, an image of the location, and / or the actual location.

[0075] Figure 7 shows a flowchart of an exemplary method to be carried out according to an exemplary embodiment of the present disclosure. While Figure 7 shows the steps to be carried out in a particular order for explanatory and discussion purposes, the methods of the present disclosure are not limited to the order and configuration shown. Various steps of Method 700 may be deleted, rearranged, combined and / or adapted in various ways without departing from the scope of the present disclosure.

[0076] In 702, the computing system can obtain a location query. The location query may include one or more search items and / or one or more images.

[0077] In 704, the computing system can process location queries and determine a location. The location can be determined using a search engine, location tags, and / or other determination methods.

[0078] In 706, the computing system can access a 3D asset database to retrieve 3D models and generate location-specific segments. The 3D asset database can utilize a large database of images to generate multiple 3D models that are associated with a location and can be easily searchable. Generating location-specific segments may include removing objects from the 3D model that are not part of a particular determined location. In some embodiments, location-specific segments may be generated by a segmentation process that isolates location-specific models by removing portions of the 3D model.

[0079] In 708, the computing system can acquire situational data. Situational data can represent time-specific conditions associated with a determined location. Time-specific conditions can include weather conditions, traffic levels, crowd levels, and / or train locations.

[0080] In 710, the computing system can process situational data to determine each simulation asset. Each simulation asset can be a set of data that allows a specific situation to be rendered into a simulated event.

[0081] In 712, the computing system can acquire each simulation asset. Simulation assets can be retrieved from a database of simulation assets that stores multiple situation-specific simulation assets.

[0082] In 714, the computing system can generate simulation events. Simulation events can include location-specific segments and location- and time-specific conditions based on each simulation asset. For example, the location could be the Golden Gate Bridge, and the time-specific conditions could be heavy traffic and rain. The location-specific segment could be a 3D model of the Golden Gate Bridge, and the simulated event could be the Golden Gate Bridge rendered into a model with heavy traffic and rain.

[0083] Figure 8 shows a flowchart of an exemplary method to be carried out according to an exemplary embodiment of the present disclosure. While Figure 8 shows the steps to be carried out in a particular order for explanatory and discussion purposes, the methods of the present disclosure are not limited to the order and configuration shown. Various steps of Method 800 can be deleted, rearranged, combined and / or adapted in various ways without departing from the scope of the present disclosure.

[0084] In 802, the computing system can reconstruct a three-dimensional environment to generate a three-dimensional mesh and textures. The reconstruction may include a database of collected map data, image data, and other location data.

[0085] In 804, the computing system can acquire multiple images of the environment.

[0086] In 806, the computing system can process multiple images using machine learning models and segment them into parts. Segmentation can be used to understand meshes and textures and to find the underlying semantic structure of the environment. Segmentation can leverage the semantic diversity of image segmentation. Graph cut algorithms can be applied to truly segment the mesh into parts.

[0087] With the 808, the computing system can project the image portion onto a three-dimensional mesh.

[0088] In 810, the computing system can generate an extended environment by removing one or more objects from the reconfigured environment. These one or more objects may include other building structures, trees, walkways, and other obstacles. The removal may be completed automatically, or it may involve taking user input to remove one or more features in the mesh.

[0089] In 812, the computing system can store an extended environment. This extended environment can be compressed before being stored and then stored in a 3D asset database.

[0090] Further disclosure The technologies discussed herein refer to servers, databases, software applications, and other computer-based systems, as well as actions performed and information transmitted to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among their components. For example, the processes discussed herein can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0091] While the subject matter has been described in detail with respect to various specific exemplary embodiments, each example is provided for illustrative purposes only and does not limit the disclosure. Those skilled in the art, once an understanding of the foregoing is achieved, will readily be able to create modifications, variations, and equivalent forms of such embodiments. Therefore, the disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter, as will be readily apparent to those skilled in the art. For example, features shown or described as part of one embodiment can be used in conjunction with another embodiment to produce further embodiments. Thus, the disclosure is intended to encompass such modifications, variations, and equivalents. [Explanation of symbols]

[0092] 100 Computing Systems 110 Servers, Server Computing Systems 112 processors 114 memory 116 Command 118 Geographic Data 120 3D Model Search 122 Image Segmentation Elements 130 client devices 132 processors 134 memory 140 Browser Elements 142 Application Elements 146 displays 148 Positioning Systems 150 User Input Devices 152 graphics processing units 154 Network Interfaces 160 Networks 180 3D Asset Databases 200 Model Search System 202 Input Data 204 Search Engines 206 Outputs, Search Results 208 Location Database 210 3D Asset Database 300 Situational Awareness Model Search System 302 Search Queries 304 Search Engine 306 search results 308 Location Database 310 3D Asset Database 312 Status Query Processing 314 Simulation Asset Database 400 Results Page 402 Search query input bar 404 Search Results 410 Knowledge Graph 412 Media Item 414 Location-Specific Models 416 Location Name 418 Location Information 500 3D rendering 502 Aerial View 504 Side View 506 Side View 600 ways 700 methods 800 ways 900 3D rendering 902 Aerial View 904 Side View 906 Side View 910 3D rendering 920 3D rendering, cloudy weather conditions 930 3D rendering, heavy rain weather conditions 1010 Context 1012 Location preview, location-specific model 1014 Excerpt from the text 1016 selectable icons 1020 Augmented Reality Experience 1022 User Environment 1024 output data, augmented reality rendering 1026 selectable icons 1100 Media Results Page 1102 Search Queries 1104 Location images, location-specific models 1106 Location Image 1108 Location Image 1110 Location-specific 3D models 1112 Location Name 1200 Path Overlay Model 1202 Route Preview 1204 Location-Specific Models 1206 Location Marker 1208 Visual Path Overlay 1210 direction

Claims

1. A method performed by a computer, A computing system having one or more processors obtains a 3D model that models multiple locations, The computing system provides a step of generating one or more three-dimensional meshes of a specific location among the plurality of locations based on the three-dimensional model, The computing system performs the steps of processing one or more images of the environment in order to generate one or more segmented images of the specific location. The computing system comprises the steps of projecting the one or more segmented images onto one or more three-dimensional meshes in order to generate location-specific segments, wherein the location-specific segments include a reconstructed environment associated with the particular location. The steps include: generating a location-specific video of a particular location based on a location-specific segment using the computing system, wherein the location-specific video includes a plurality of different perspective views of the location-specific segment; The computing system includes the steps of storing location-specific videos of the specific location in a 3D asset database. A method that includes this.

2. The method according to claim 1, further comprising the step of extending the reconstructed environment by removing one or more objects from the reconstructed environment using the computing system in order to generate the location-specific segment, wherein the location-specific segment describes a three-dimensional representation of the extended environment.

3. The step of generating a location-specific video of a particular location based on the location-specific segment using the computing system is: The computing system generates multiple renderings of the location-specific segments, The computing system generates location-specific video based on the plurality of renderings. The method according to claim 1, including the method described in claim 1.

4. The method according to claim 1, wherein the specific location is associated with a specific structure.

5. The computing system further includes the step of obtaining data associated with the specific location, The method according to claim 1, wherein the three-dimensional model is obtained based on the data associated with the specific location.

6. The computing system further includes the step of obtaining data associated with the specific location, The method according to claim 1, wherein the one or more three-dimensional meshes are generated based on the data associated with the specific location.

7. The computing system acquires data associated with the specific location, The computing system acquires one or more images based on the data associated with the specific location. It further includes, The method according to claim 1, wherein the one or more segmented images are segmented based on the data associated with the specific location.

8. The computing system obtains a search query associated with the user computer device. The computing system processes the search query and determines that the search query is associated with the specific location. The computing system performs the steps of obtaining location-specific video from the 3D asset database based on the fact that the search query is associated with the specific location, The computing system provides the location-specific video for display. The method according to claim 1, further comprising:

9. The method according to claim 1, wherein the three-dimensional model includes LiDAR data generated by a light detection and ranging sensor.

10. The method according to claim 1, further comprising the step of storing the location-specific segment together with the location-specific video in the three-dimensional asset database.

11. A computing system, One or more processors, One or more non-temporary computer-readable media that, when executed by the one or more processors, collectively store instructions that cause the computing system to perform an operation, wherein the operation is Steps include obtaining a 3D model that models multiple locations, The steps include generating one or more three-dimensional meshes for a specific location among the plurality of locations based on the three-dimensional model, The steps include processing one or more images of the environment in order to generate one or more segmented images of the specific location, A step of projecting one or more segmented images onto one or more three-dimensional meshes in order to generate location-specific segments, wherein the location-specific segments include a reconstructed environment associated with the particular location. A step of generating a location-specific video of a particular location based on the location-specific segment, wherein the location-specific video includes a plurality of different perspective views of the location-specific segment. The steps include storing location-specific videos of the specific location in a 3D asset database. including one or more non-temporary computer-readable media and A computing system equipped with the following features.

12. The computing system according to claim 11, wherein the aforementioned specific location is associated with a building structure.

13. The computing system according to claim 12, wherein the location-specific video includes a plurality of different perspective views of the building structure.

14. The computing system according to claim 11, wherein the one or more three-dimensional meshes include vector data.

15. The computing system according to claim 14, wherein the vector data includes one or more vector representations of one or more buildings associated with the specific location.

16. The one or more three-dimensional meshes are generated based on geospatial data. The computing system according to claim 11, wherein the geospatial data includes geographical images, vector data, and data of points of interest.

17. One or more non-temporary computer-readable recording media that, when executed by one or more computing devices, collectively store instructions causing the one or more computing devices to perform an action, wherein the action is Steps include obtaining a 3D model that models multiple locations, The steps include generating one or more three-dimensional meshes for a specific location among the plurality of locations based on the three-dimensional model, The steps include processing one or more images of the environment in order to generate one or more segmented images of the specific location, A step of projecting one or more segmented images onto one or more three-dimensional meshes in order to generate location-specific segments, wherein the location-specific segments include a reconstructed environment associated with the particular location. A step of generating a location-specific video of a particular location based on the location-specific segment, wherein the location-specific video includes a plurality of different perspective views of the location-specific segment. The steps include storing location-specific videos of the specific location in a 3D asset database. One or more non-temporary computer-readable recording media, including [the specified element].

18. The one or more non-temporary computer-readable recording media according to claim 17, wherein the one or more images include satellite images and street-level images.

19. The three-dimensional model comprises one or more non-temporary computer-readable recording media according to claim 17, spanning multiple addresses.

20. The three-dimensional model is one or more non-temporary computer-readable recording media according to claim 17, which spans multiple directions.

Images