State-aware generation of panoramic images

Abstract

Systems and methods for generating panoramic images are provided. One exemplary method is executable by one or more processors and may include acquiring a first panoramic image showing a geographic area. The method also includes acquiring an image showing one or more physical objects not present in the first panoramic image. Further, the method includes converting the first panoramic image into a second panoramic image showing the one or more physical objects and including at least a portion of the first panoramic image.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates to panoramic imagery, and more particularly, to generating panoramic imagery based on content of other images. [Background technology]

[0002] The background description provided herein is intended to generally present the contents of the present disclosure. The inventors' work within the scope described in the Background Art section, as well as aspects of the Background Art that may not otherwise qualify as prior art at the time of filing, are not admitted, either explicitly or implicitly, as prior art to the present disclosure.

[0003] Today, users access panoramic imagery from geographic services that provide panoramic or "360" viewers of various geographic locations. Users can also navigate through a series of images, if such a series is available, to visually explore the location. For example, a user can virtually "drive" or "walk" down a road or street and see the surroundings approximately from a driver's or pedestrian's perspective. However, existing panoramic imagery only reflects the conditions of the geographic location at the time the imagery was captured. For example, even when a user views a panoramic image of a location in snowy conditions in January, the panoramic imagery may only show this location in clear conditions in July.

[0004] Furthermore, some locations may be far from or otherwise difficult for a vehicle equipped with a specialized camera or camera array to reach, and as a result, the panoramic imagery is updated infrequently and some changes to the environment are not reflected in the imagery for an extended period of time. Summary of the Invention [Means for solving the problem]

[0005] A first aspect of the technique of the present disclosure is a method for generating a panoramic image. The method is executable by one or more processors and includes acquiring a first panoramic image showing a geographic area. The method also includes acquiring an image showing one or more physical objects not in the first panoramic image. In other words, one or more physical objects shown in the image are not shown in the first panoramic image. Furthermore, the method includes transforming the first panoramic image into a second panoramic image showing the one or more physical objects and including at least a portion of the first panoramic image. This provides the technical effect of generating a new image based on an existing panoramic image and other images. Specifically, a new panoramic image showing a geographic area can be generated by transforming an existing panoramic image. The existing panoramic image is transformed such that the image shows features not previously shown in the existing panoramic image, thereby resulting in a new panoramic image. Such new features are acquired from the other images. In some embodiments, these images are non-panoramic images. Thus, a new panoramic image showing a geographic area can be acquired based on an existing panoramic image and other existing images without requiring a new panoramic image to be captured using a specialized panoramic camera. A new panoramic image may include features that were not present in the existing panoramic image. In other words, at the time the first panoramic image was captured, one or more features may not have been present in the geographic area shown. However, at a later time, new features may be present in the geographic area (including previous features being removed or replaced, or new features being added). In conventional techniques, obtaining an updated panoramic image (showing new features) of a geographic area requires visiting the area with specialized equipment suitable for obtaining panoramic images. However, the present technique allows an updated panoramic image to be created based on already captured images, without the need for a new panoramic image to be captured. In this manner, an improved means for obtaining an updated panoramic image is provided.

[0006] In some embodiments, the image is user-generated content (UGC). In other words, the image showing one or more features not present in the first panoramic image may be any form of image captured by a user. In some examples, a conventional camera may be used to capture an image, which is then used to convert the first panoramic image to capture an updated panoramic image. In this way, access to updated imagery is increased because the source of the first image may be a wide variety of sources (not just specialized panoramic image capture equipment).

[0007] In some embodiments, the image shows at least a portion of the geographic area shown in the first panoramic image. In other words, the image is not the same panoramic image as the first panoramic image (showing the same geographic area), but the image shows at least a portion of the area shown in the first panoramic image. This can be used to identify the location of new features in the new panoramic image (based on that portion of the area shown in the image).

[0008] A second aspect of the technique of the present disclosure is a method for generating a panoramic image, executable by one or more processors, and including the step of acquiring a first panoramic image. The method also includes the step of acquiring a non-panoramic image showing one or more features not present in the first panoramic image. The method further includes the step of applying the non-panoramic image to the first panoramic image to show the one or more features in the second panoramic image. In these embodiments, the one or more features shown in the non-panoramic image are not shown in the first panoramic image. Similar to the advantages described above, this allows an updated panoramic image to be created based on already captured panoramic images and other non-panoramic images without requiring a new panoramic image to be captured. In this way, a simpler approach is provided for acquiring an updated panoramic image that does not require specialized equipment to capture a new panoramic image.

[0009] A third aspect of the techniques of this disclosure is a computing device configured to implement some of the methods described herein, including processing hardware.

[0010] A fourth aspect of the techniques of this disclosure is a (optionally non-transitory) computer-readable medium carrying instructions that, when executed by one or more processors, cause the one or more processors to perform any one of the methods disclosed herein. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram of an example system in which techniques of the present disclosure for generating panoramic imagery may be implemented. [Diagram 2] 1A-1C are diagrams illustrating example panoramic images that may be generated using techniques of this disclosure based on an existing panoramic image and a non-panoramic image that includes physical objects not present in the existing panoramic image. [Diagram 3] A diagram showing an example panoramic image that may be generated using techniques of this disclosure based on an existing panoramic image and a non-panoramic image that exhibits features not present in the existing panoramic image due to different environmental conditions. [Figure 4A] FIG. 1 illustrates an example user interface displaying an existing panoramic image and a notification that the existing panoramic image is out of date. [Figure 4B] 4B illustrates an exemplary user interface displaying a panoramic image generated based on the existing panoramic image of FIG. 4A and including a notification that the panoramic image has been generated. [Figure 5A] FIG. 1 illustrates an example user interface displaying existing panoramic images showing a geographic location and user-selectable options for accessing new panoramic images showing the geographic location under different environmental conditions. [Figure 5B]5B illustrates an exemplary user interface displaying a panoramic image generated based on the existing panoramic image of FIG. 5A and user-selected environmental conditions. [Figure 6A] FIG. 1 is a combined block and logic diagram illustrating training of a generative adversarial network configured to generate panoramic imagery. [Figure 6B] FIG. 6B is a combined block and logic diagram illustrating the generation of a panoramic image using the trained generative adversarial network of FIG. 6A. [Figure 7] 2 is a flow diagram of an example method for generating a panoramic image that may be implemented within the computing device of FIG. 1. [Figure 8] 4 is a flow diagram of another exemplary method for generating a panoramic image that may be implemented within the computing device of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] overview In general, the systems and methods of the present disclosure generate new panoramic images for a geographic location such that the new panoramic image includes features and / or objects not present in an existing panoramic image corresponding to the same geographic location. The generated panoramic image may reflect a user-specified state and / or may be an updated panoramic image that reflects changes to the location. For example, existing panoramic images available for a location may show the location during daylight hours, during the summer. A user may request a panoramic image showing the location as it appears at night, during the winter. The system may generate a panoramic image showing the location for the requested day and season, and transmit the generated panoramic image to a client device for display. As another example, the system may receive a non-panoramic image for a location and determine that the appearance of the location has changed relative to an existing panoramic image. The non-panoramic image may include, for example, a new storefront on a street. If a user requests a panoramic image showing the location, the system may generate an updated panoramic image of the location including the new storefront. Importantly, the updated panoramic image is generated based on already captured images as described above, and does not require any new images to be captured using an image capture means, such as a camera, specifically a panoramic camera (which may require multiple cameras).

[0013] Depending on the implementation, the system may generate a new panoramic image based on an existing panoramic image and a "source" panoramic or non-panoramic image showing features or objects not present in the existing panoramic image. The source image may show at least a portion of the same geographic location as the existing panoramic image, or a different geographic location. The system may generate the panoramic image using a machine learning model. In some implementations, the system may utilize a generative adversarial network to generate the panoramic image and evaluate the quality of the generated panoramic image (i.e., whether the generated panoramic image is perceived by humans as real or artificial). In this way, the generative adversarial network (GAN) is used in a specific technical application of generating and evaluating panoramic imagery showing a geographic area. In other words, the disclosed GAN serves a technical purpose for generating and evaluating panoramic imagery showing a geographic area.

[0014] The user may specify what conditions the generated panoramic imagery should include. For example, the user-specified conditions may include weather conditions (e.g., snow, rain, fog, cloudy, clear, etc.), time of day or amount of daylight (e.g., sunrise, midday, sunset, night, etc.), season (e.g., summer, spring, fall, winter, etc.), crowding level (e.g., crowded, no pedestrians, etc.), natural disaster (e.g., flood, post-fire, post-earthquake, etc.), latest view, etc. Using the techniques of this disclosure, the system may generate panoramic imagery showing some user-specified conditions and / or changes to a geographic location without having to capture a new panoramic imagery showing the actual conditions and / or changes at the geographic location. In this manner, accessibility to various types of imagery is improved, as such images showing different conditions or features are easily obtained without the need for images showing those conditions or features to actually be captured. It is also possible to generate such images in response to a request from a user (described in more detail below). In these embodiments, images may simply be generated in response to a user request, rather than being stored and retrieved in response to a request. In this way, a technical effect is achieved that improves the use of storage space, since the various images accessible by the user do not occupy a corresponding amount of storage (because they are created temporarily in real time as they are requested). When the user no longer requires the image, the image may be removed from the temporary storage device, thereby avoiding the need to store extraneous amounts of data relating to images showing geographic areas under different conditions.

[0015] Exemplary Hardware and Software Components 1, an exemplary panoramic image generation system 100 includes a client computing device 102 (also referred to herein as a "client device") coupled to a panoramic image generation server 130 (also referred to herein as a "server 130") via a network 120. Network 120 may generally include one or more wired and / or wireless communication links, and may include, for example, a wide area network (WAN) such as the Internet, a local area network (LAN), a cellular telephone network, or another suitable type of network.

[0016] The client device 102 may be, for example, a portable device such as a smartphone or tablet computer. The client device 102 may also be a laptop computer, a desktop computer, a personal digital assistant (PDA), a wearable device such as smart glasses, or other suitable computing device. The client device 102 may include a memory 106, one or more processors (CPUs) 104, a global positioning system (GPS) module 112 or another suitable positioning module, a network interface 114, a user interface 116, and an input / output (I / O) interface 118. The client device 102 may include components not shown in FIG. 1, such as a graphics processing unit (GPU).

[0017] Network interface 114 may include one or more communications interfaces, such as hardware, software, and / or firmware, for enabling communication over a cellular network, a WiFi network, or any other suitable network, such as network 120. User interface 116 may be configured to provide information, such as a panoramic image, to a user. I / O interface 118 may include various I / O components (e.g., ports, capacitive or resistive touch-sensitive input panels, keys, buttons, lights, LEDs). For example, I / O interface 118 may be a touch screen.

[0018] The memory 106 may be non-transitory memory and may include one or more suitable memory modules, such as random access memory (RAM), read only memory (ROM), flash memory, other types of persistent memory, etc. The memory 106 may store machine-readable instructions executable on one or more processors 104 and / or specialized processing units of the client device 102. The memory 106 also stores an operating system (OS) 110, which may be any suitable mobile or general-purpose OS. In addition, the memory may store one or more applications that communicate data over the network 120, including a panoramic image application 108. Communicating data may include sending data, receiving data, or both. The OS 110 may include application programming interface (API) functions that allow applications to access information from the GPS module 112 or other components of the client device 102. For example, the panoramic image application 108 may include instructions that invoke the OS 110 API to retrieve the current geographic location of the client device 102.

[0019] Depending on the implementation, the panoramic image application 108 can display panoramic images of geographic locations, provide user controls for exploring geographic locations by navigating through panoramic images, display interactive digital maps showing geographic locations for which panoramic images (real or generated) are available, request and receive generated panoramic images, provide user controls for requesting panoramic images reflecting user-specified states, provide various geolocation content, etc. Although FIG. 1 shows the panoramic image application 108 as a standalone application, the functionality of the panoramic image application 108 may be provided in the form of an online service accessible via a web browser running on the client device 102, as a plug-in or extension for another software application running on the client device 102, etc. The panoramic image application 108 may generally be provided in different versions for each different operating system. For example, a manufacturer of the client device 102 may provide a software development kit (SDK), including the panoramic image application 108 for the Android® platform, another for the iOS® platform, etc.

[0020] The server 130 may be configured to receive requests for panoramic images, generate panoramic images, and transmit the generated panoramic images to the client device 102. The server 130 includes one or more processors 132 and memory 134. The memory 134 may be tangible, non-transitory memory and may include any type of suitable memory module, including random access memory (RAM), read-only memory (ROM), flash memory, other types of persistent memory, and the like. The memory 134 stores instructions executable on the processor 132 that configure a panoramic image generation module 136, which may process requests for panoramic images and generate the panoramic images. In some implementations, the panoramic image generation module 136 may generate panoramic images and cache the generated images for later retrieval before receiving a request from the client device 102 for such images.

[0021] The panoramic image generation module 136 may train and store a machine learning model operable to generate a panoramic image. In some implementations, the machine learning model is a generative adversarial network (GAN) that includes two neural networks, a generator network and a discriminator network, as discussed in more detail with reference to FIGS. 6A-6B.

[0022] The server 130 may be communicatively coupled to databases 140, 142, and 144 that store geospatial information, existing panoramic images, and non-panoramic images that may be used as training data for the machine learning models. The panoramic image database 140 includes panoramic images that may be 360-degree panoramas. The panoramic images may be captured by an entity associated with the server 130 using a 360-degree imaging device. The panoramic image application 108 may request existing panoramic images from the database 140, either directly or via the server 130. More specifically, the panoramic image database 140 may include roadside imagery that is comprised of panoramic images stitched together to form a virtual environment in which the panoramic images are captured from the perspective of a person or vehicle moving along a path or road. For example, a series of such panoramic images may be stitched together such that a user may navigate through the series of panoramic images to virtually recreate the experience of moving along a path or road.

[0023] The panoramic image application 108 may be configured to operate in a roadside mode that allows a user to navigate a virtual environment formed by the panoramic imagery. For example, the panoramic image application 108 allows a user to navigate through a virtual environment formed by the roadside imagery such that the user virtually experiences traveling along a route or road along which the roadside imagery was captured.

[0024] The user-generated content (UGC) database 142 includes images that may be non-panoramic or panoramic. The panoramic images are 360-degree panoramas. The images in the UGC database 142 include crowdsourced images that are tagged (e.g., with metadata) with the time and location the image was captured. These images can be provided from a wide variety of sources, such as user smart devices. In this way, new panoramic imagery can be generated using such widely available IGC imagery, facilitating the creation of new panoramic imagery without the need for a panoramic camera.

[0025] Map database 144 may include general satellite data that stores street and road information, topographical data, satellite imagery, information about public transportation routes, information about businesses or other points of interest (POIs), navigational data such as directions for various modes of transportation, and the like. Images that server 130 receives from databases 140 and 142 may include metadata indicating the time and / or location at which the image was captured. Server 130 may receive information from map database 144 (e.g., in response to a request from server 130 to map database 144 for information about a particular location) and transmit this information to client device 102. Panoramic image application 108 may use this information to display an interactive digital map that indicates the geographic locations at which panoramic imagery is available. As an example, panoramic image application 108 operating in roadside mode may also display roadside imagery for a location and a map that indicates the location and viewpoint from which the roadside imagery was captured. By interacting with panoramic image application 108, a user may navigate through roadside imagery for a location indicated by the map.

[0026] Generally, server 130 may receive information regarding the geographic locations from any number of suitable databases, web services, etc. For example, server 130 may be coupled to a weather database (not shown) including current or average weather data in various geographic areas, a natural disaster database (not shown) including current or common natural disasters occurring in various geographic areas, and / or a traffic database including current or average vehicles and / or pedestrians on roads, routes, or areas.

[0027] Although examples of the present disclosure primarily reference server 130 generating panoramic images, client device 102 may generate the panoramic image instead of requesting the generated panoramic image from server 130. Panoramic image application 108 may include, for example, panoramic image generation module 136. Client device 102 may be communicatively coupled to databases 140, 142, 144 with reference to server 130 such that client device 102 can provide the functionality of server 130.

[0028] Exemplary Techniques for Producing Panoramic Images FIG. 2 illustrates an example panoramic image that may be generated using techniques of this disclosure based on an existing panoramic image and a non-panoramic image that includes physical objects not present in the existing panoramic image. In the example scenario of FIG. 2, a user may use a client device 102 to interact with a panoramic image application 108 to search for a panoramic image showing a given location, referred to herein as "location A." In response to receiving a request from the client device 102 for a panoramic image for location A, the server 130 may retrieve a panoramic image including a panoramic image 202 from the panoramic image database 140 and provide the panoramic image to the client device 102. The panoramic image 202 is thus an existing panoramic image of location A. The panoramic image 202 may be a 360-degree image such that a user interacting with the image can rotate the image to view the scene at location A in 360 degrees. The panoramic image 202 includes a sign 204 for a business located at location A. 2 shows a single panoramic image 202, server 130 may retrieve a series of panoramic images corresponding to roadside imagery at location A and adjacent locations from panoramic image database 140. Server 130 may generate new roadside imagery for location A to show new objects and / or features such as snow, rain, etc., so that a user can virtually explore location A under various conditions.

[0029] More specifically, before or after providing the panoramic image 202 to the client device 102, the server 130 may determine that the panoramic image 202 may not reflect the current conditions at location A. The server 130 may make this determination based on the time a user submits a request from a client device, a time the user explicitly specifies in the request (e.g., "Show me the roadside statues at 6 p.m."), the time the panoramic image 202 was captured, etc. Additionally or alternatively, the server 130 may determine that the panoramic image 202 is out of date based on the content of the panoramic image 202. Specifically, the server 130 may retrieve an image from the UGC database 142 that indicates location A that was captured (e.g., based on the image's location tag) near when the request for the panoramic image 202 was made. For example, the tag of the panoramic image 202 may specify a capture time of approximately one year before the user request, while the tag of the retrieved image that indicates location A may specify a capture time as approximately two weeks before the user request. Server 130 may compare the content of the retrieved image showing location A to panoramic image 202 to determine whether the retrieved image shows physical objects not included in panoramic image 202. If so, server 130 may determine that panoramic image 202 is stale and identify the retrieved image as an image to use to generate a new panoramic view that reflects the current conditions at location A.

[0030] In the example scenario of FIG. 2, server 130 retrieves image 206 from UGC database 142. Image 206 also shows an image at location A (or at least some of the images at location A), but has a later capture time than panoramic image 202. In this particular example, image 206 includes a new sign 208 for a business that has apparently replaced the business indicated by sign 204. In the example of FIG. 2, retrieved image 206 is a non-panoramic image. However, in some implementations, retrieved image 206 may be a panoramic image. Furthermore, in this example, new sign 208 is an object that is present in image 206 but not in panoramic image 202, whereas in other scenarios, some objects may be present in panoramic image 202 but not in newer image 206 (e.g., a municipality may remove a traffic sign at location A after panoramic image 202 is captured, and newer image 206 may reflect this change). Still further, an object in image 206 need not replace another object in panoramic image 202 at the same location, and may generally appear anywhere in the image. In general, one skilled in the art will appreciate that an object shown in the previously taken panoramic image 202 may be replaced or may be absent from retrieved image 206, and / or an object shown in retrieved image 206 may be absent from panoramic image 202. In either case, retrieved image 206 will show features not shown in panoramic image 202 (e.g., new objects, and / or replaced objects, and / or removed objects).

[0031] In some implementations, the server 130 provides the panoramic image 202 to the client device 102, and the panoramic image application 108 displays the panoramic image 202 to the user. The panoramic image application 108 may also indicate to the user that an existing panoramic image showing location A is out of date based on information received from the server 130, and ask the user whether the user would like to receive a newer panoramic image generated. In response to receiving a confirmation from the user, the panoramic image application 108 may request a newer panoramic image from the server 130. The user may also request a newer panoramic image via the panoramic image application 10 that has not been prompted by the panoramic image application 108. Additional details regarding user interaction elements are discussed below with reference to Figures 4A-4B.

[0032] In some implementations, the server 130 may not provide the panoramic image 202 to the client device 102. Instead, the server 130 may automatically identify the panoramic image 202 as out of date based on the time the panoramic image 202 was captured and / or based on the content of the UGC images of location A, generate an updated panoramic image, and provide the generated panoramic image to the client device 102.

[0033] Additionally, in various implementations, server 130 may receive a request to update an existing panoramic image of a location from a third party, not necessarily a panoramic image requested by a user, via client device 102. For example, a representative of a new business at location A may submit image 206 to UGC database 142, indicating to server 130 that the existing panoramic image of location A should be updated to reflect new signage 208 for the new business.

[0034] Although the above discussion refers to the server 130 retrieving the images 206 from the UGC database 142 , in some implementations, the server 130 can retrieve the images 206 from the panoramic image database 140 .

[0035] In either case, server 130 may perform a transformation 210 on panoramic image 202 to generate a new panoramic image 212 based on panoramic image 202 and image 206. The generated panoramic image 212 shows location A but includes the new sign 208 from image 206. In this example, image 206 showing location A is a non-panoramic image, but in other scenarios, image 206 may be a panoramic image and transformation 210 may include transforming panoramic image 202 into the generated panoramic image 212 based on the panoramic image but not the non-panoramic image.

[0036] Transformation 210 may include determining whether panoramic image 202 and image 206 show the same geographic location or at least a portion of the same geographic location, location A shown in Figure 2. Server 130 may make this determination based on location data contained in the metadata of images 202 and 206, or using suitable image processing techniques to identify and compare features in the two images.

[0037] If the panoramic image 202 and the image 206 show the same geographic location, the server 130 may compare the panoramic image 202 and the image 206 to determine whether the images 202, 206 show different content. Specifically, the server 130 may compare the images 202, 206 to determine whether the images show different physical objects and / or whether the image 206 shows a physical object that is not in the image 202. For example, the server 130 may determine whether the images 202, 206 show different physical objects at the same location in space. In the panoramic image 202, the sign 204 is located near the top of the building facing the street above a window. The server 130 may identify this location in the space as shown in the image 206 and determine whether a different physical object is located at that location. In the image 206, a new sign 208 is located at the same (or at least similar) location as the sign 204, and the sign 204 is not present in the image 206. Based on this comparison, server 130 may determine that sign 204 has been replaced by new sign 208 at location A. To compare images 202, 206, server 130 applies, for example, machine learning models or heuristic algorithms to detect the presence and location of objects (e.g., objects within known object classes) in images 202, 206.

[0038] If the panoramic image 202 and the image 206 show different geographic locations (unlike the example shown in FIG. 2 ), the server 130 may still identify physical objects in the image 206 for extraction and insertion into the panoramic image 202. For example, a third party or another user may indicate that the image 206 includes a physical object to be reflected in the panoramic image 202 and submit the image 206 to the server 130 via the panoramic image application 108. For example, a business owner may submit an image 206 of a different location of the business and indicate that a similar sign is currently located at location A and should be reflected in the panoramic image of location A. As another example, a machine learning model may identify a traffic sign, traffic light, mailbox, or other repeating object. The server 130 may then use the panoramic image 202 to reference a descriptor of the location (e.g., GPS coordinates) and object (e.g., a “stop sign”) and select an image of the object from a database and insert the image at the referenced location to generate a new panoramic image 212. In this way, the updated imagery may include traffic signs or signals that are actually present in the geographic location but are not shown in the original panoramic image of the area. In this manner, the disclosed techniques provide a more accurate description of the geographic area.

[0039] The server 130 may extract physical objects contained in the image 206, but not in the panoramic image 202, and prepare these physical objects for insertion into the panoramic image 202. This preparation may include mapping the extracted physical objects to a projection corresponding to the panoramic image 202, scaling the extracted physical objects to the scale of the panoramic image 202, and aligning the physical objects within the panoramic image 202, which may be described as mapping, scaling, and aligning the physical objects. In some implementations, some or all of these preparation steps may be performed on the image 206 first, and the physical objects may be extracted from a prepared version of the image 206, or the like, before being inserted into the panoramic image 202. Certain techniques of mapping, scaling, and alignment, which are described in more detail below, help to produce an accurate up-to-date panoramic image. In particular, these techniques are used to ensure that features present in the image 206 are accurately included in the panoramic image by ensuring that these features appear in the correct location and that these features have the correct appearance. In other words, these techniques may be used to ensure that the generated panoramic image accurately depicts a geographic area, which helps to improve the generated panoramic image and provides the technical effect of generating a panoramic image that accurately depicts a geographic area.

[0040] If the image 206 is a non-panoramic image, the transformation 210 includes mapping the extracted physical objects to a projection corresponding to the panoramic image 202. The mapping process refers to the mathematical transformation of the physical objects or the retrieved image 206 to the space of the panoramic image 202. The server 130 may identify a projection type of the panoramic image 202. For example, the projection type may be a spherical projection, an equirectangular projection, a cubic projection, or a cylindrical projection. The server 130 may identify the projection type by analyzing image properties or metadata of the image, which may indicate the projection type. Based on the projection type, the server 130 may map the extracted physical objects to a coordinate system of the projection type. The coordinate system is a projected coordinate system defined on a two-dimensional surface (e.g., for display on a two-dimensional screen) that corresponds to the projection type of the panoramic image 202, such that coordinates of the coordinate system map to positions within the panoramic image 202.

[0041] If image 206 is a panoramic image but of a different projection type than panoramic image 202, transformation 210 may include mapping the extracted physical objects to a projection corresponding to panoramic image 202, as discussed above. If image 206 is a panoramic image of the same projection type as panoramic image 202, transformation 210 may not include a mapping to a different projection type.

[0042] The transformation 210 may include scaling the extracted physical objects to the scale of the panoramic image 202. For example, if the image 206 is captured from a perspective closer to the building than the perspective of the panoramic image 202, the size of the extracted physical objects may be scaled down before being inserted into the panoramic image 202. The scaling factor may be identified based on, for example, the relative sizes of objects appearing in both the panoramic image 202 and the image 206. The scaling factor may also be identified based on scaling factors that produce realistic images, as determined by a machine learning model, discussed in more detail with reference to FIGS. 6A-6B.

[0043] The extracted physical objects and / or image 206 may be aligned within panoramic image 202 such that the extracted physical objects are placed in a suitable location within panoramic image 202. In the example shown in FIG. 2, new landmark 208 is aligned within panoramic image 202 such that it is placed where landmark 204 was previously located. How image 206 and / or physical objects extracted from image 206 are aligned within panoramic image 202 may be based on a comparison of image 206 and panoramic image 202 and the relative positions of objects appearing in both images 206, 202.

[0044] Merging the scaled and registered extracted physical objects into the panoramic image may require additional processing. In some scenarios, the server 140 may need to remove other objects from the panoramic image 202, such as the sign 204 that has been replaced by the new sign 208. The server 130 may then insert the extracted physical objects into the panoramic image 202. In some scenarios, the server 130 may insert the extracted physical objects into the panoramic image 202 as an overlay covering a portion of the panoramic image 202. In addition, the server 130 may make further modifications to the panoramic image 202 to enhance the realism of the new panoramic image 212. In other words, the server 130 may make further modifications to the panoramic image 202 to make the new panoramic image look more realistic, as if the new panoramic image had been captured using a panoramic camera (rather than a combination of the panoramic image 202 and another image 206). For example, server 130 may blend edges of extracted physical objects to make the objects appear more realistic in new panoramic image 212. Server 130 may need to add shadows to new physical objects and remove or modify existing shadows when generating new panoramic image 212. As mentioned above, the above techniques may be used to make the resulting generated image more accurate, meaning that the generated image accurately represents a geographic area.

[0045] The server 130 may use a machine learning model to implement some or all of the steps of the transformation 210, as discussed with reference to FIGS. 6A-6B. The machine learning model may be trained, for example, to extract physical objects from an image, map the extracted physical objects to the coordinate system of the panoramic image, and merge the extracted and mapped physical objects to generate a new panoramic image. In this way, the machine learning model is specifically adapted for the technical purpose of generating a new panoramic image based on existing images and other images showing physical objects. Depending on the implementation, multiple machine learning models may perform these functions. For example, a first machine learning model may extract physical objects from an image, and a second machine learning model may generate a new panoramic image using the output from the first machine learning model.

[0046] Additionally, after generating panoramic image 212, server 130 may generate additional panoramic images (e.g., similar to and / or based on panoramic image 212) showing location A, or locations near location A, and connect or "stitch" these images together to form an interactive roadside view. More specifically, server 130 may generate the roadside view to provide a virtual environment through which a user may navigate as if the user were traveling along the route or road from which the panoramic image was captured.

[0047] Additionally, as mentioned above, in some implementations, the client device 102, rather than the server 130, may perform the conversion 210. For example, the client device 102 may receive the panoramic image 202 from the server 130 and generate the panoramic image 212 in response to a request from a user received by the panoramic image application 108.

[0048] With reference to FIG. 3, the techniques of this disclosure may also include generating a panoramic image based on an existing panoramic image and one or more other images that do not necessarily correspond to the same geographic location as the existing panoramic image. In other words, the one or more other images may not show any portion of the geographic location shown in the panoramic image. The generated panoramic image may include features that are shown in the one or more other images but not in the existing panoramic image. In the example scenario of FIG. 3, a user may interact with a panoramic image application 108 using a client device 102 to search for a panoramic image showing a given location, referred to herein as “location B.” In response to receiving a request from the client device 102 for a panoramic image for location B, the server 130 may retrieve a panoramic image including a panoramic image 302 from the panoramic image database 140 and provide the panoramic image to the client device 102. The panoramic image 302 is an existing panoramic image of location B. The panoramic image 302 is a 360-degree image such that a user interacting with the image can rotate the image to view location B in 360 degrees. 3 shows a single panoramic image 302, server 130 may retrieve a series of panoramic images from panoramic image database 140 that correspond to roadside imagery of location B. Server 130 may generate new images that correspond to new roadside imagery of location B so that a user can virtually explore location B.

[0049] After the server 130 provides the panoramic image 302 to the client device, the panoramic image application 108 may display the panoramic image 302 to the user. The panoramic image application 108 may ask the user whether he / she would like to receive a generated panoramic image showing location B with different characteristics (e.g., showing location B under different conditions) and provide the user with an option to select the given characteristics. In response to receiving a selection from the user, the panoramic image application 108 may request a new panoramic image including the requested characteristics from the server 130. In some implementations, the server 130 may pre-generate panoramic images under different conditions and store the generated panoramic images so that the panoramic images are generated before receiving a specific user request. In other implementations, the generated panoramic images may be generated in response to a user request. In this way, the generated images may not be stored permanently in memory, but may be temporarily stored for the duration of the generated panoramic images requested by the user, thereby making better use of storage space.

[0050] The server 130 may perform a transformation 310 on the panoramic image 302 to generate a new panoramic image 312 based on the panoramic image 302 and other images that include features not present in the panoramic image 302. The features may include physical objects depicted in the features or characteristics of the images more generally. For example, the features may include patterns or visual manifestations of environmental conditions, including weather conditions (e.g., rain, snow, fog, ice, cloudy, sunny), daylight conditions (e.g., sunrise, daytime, sunset, night), and seasons (e.g., winter, spring, summer, fall). In addition, the features may include patterns or visual manifestations of other transient conditions, such as crowding levels (e.g., no people in the image, low crowding, or high crowding levels), natural disasters and / or damage from natural disasters (e.g., floods, earthquake damage, tornadoes, hurricanes). As discussed with reference to FIG. 4B, what features the transformation 310 is relevant to may depend on user-specified conditions (e.g., a user may request a panoramic image of snow at location B).

[0051] The transformation 310 includes generating a new panoramic image 312 based on the panoramic image 302 and the image 306. The image 306 includes features not present in the panoramic image 302. In the example shown in FIG. 3, the image 306 shows a snowy environment. Although the image 306 is a non-panoramic image, the image 306 may be a panoramic image. The server 130 may retrieve the image 306 from the panoramic image database 140 or from the UGC database 142. In addition, although only one image 306 is shown in FIG. 3, the transformation 310 may be based on multiple images with similar features as the image 306. Furthermore, although the image 306 shows a different location than the panoramic image 302, in some implementations, the image 306 may show snow conditions at the same geographic location.

[0052] In general, the steps involved in transformation 310 depend on whether image 306 is panoramic or non-panoramic, whether the image shows the same geographic location as panoramic image 302 or a different geographic location, and what features are to be reflected in new panoramic image 312. In the example of Figure 3, transformation 310 involves transforming panoramic image 302 showing the same geographic location, location B, when it is covered in snow.

[0053] If panoramic image 302 and image 306 show the same geographic location (or at least a portion of the same geographic location), server 130 may compare panoramic image 302 and image 306 to determine what features differ between panoramic image 302 and image 306. Based on this comparison, server 130 may identify patterns in image 306 that correspond to features absent from panoramic image 302.

[0054] 3, panoramic image 302 and image 306 may show different geographic locations. Image 306 may be marked or annotated with features shown in the image. Server 130 may extract visual manifestations and / or patterns of those features from image 306 and prepare the extracted features for insertion into or overlaying on panoramic image 302.

[0055] As an example, server 130 may identify whether weather features are included in image 306, such as one or more patterns indicative of weather conditions. Image 306 includes a pattern indicative of snow. Server 130 may use a machine learning model trained to identify patterns indicative of different weather conditions, as discussed with reference to FIGS. 6A-6B.

[0056] As another example, server 130 may identify whether weather features, such as one or more light conditions indicative of an amount of daylight (e.g., sunrise, midday, sunset, night, etc.), are included within image 306. Server 130 may use a machine learning model trained to identify patterns indicative of different lighting conditions, as described with reference to FIGS.

[0057] The server 130 may extract features from the images 306 that are not in the panoramic image 302 and prepare those features for insertion into the panoramic image 302. The preparation may be similar to the preparation discussed above with respect to physical objects. Thus, the preparation may include mapping the extracted features to a projection corresponding to the panoramic image 302, scaling the extracted features to the scale of the panoramic image 302, and aligning the features to appropriate (e.g., as "appropriate" as determined by how realistic the inserted features appear, which may be determined by a discriminator network) locations in the panoramic image 302. The mapping process refers to performing a mathematical transformation of the physical object or extracted image into the space of the panoramic image 302.

[0058] Similar to transformation 210, how the extracted features are mapped depends on whether image 306 is a non-panoramic image, a panoramic image of the same projection type as panoramic image 302, or a panoramic image of a different projection type than panoramic image 302. If image 306 is a non-panoramic image (as in the case of FIG. 3), transformation 310 involves mapping the extracted features to a projection corresponding to panoramic image 302. In a scenario where image 306 is an image of a different projection type than panoramic image 302, transformation 310 may involve mapping the extracted features to a projection corresponding to panoramic image 302. In a scenario where image 306 is an image of the same projection type as panoramic image 302, transformation 310 may not involve mapping to a different projection type.

[0059] Transformation 310 may also include scaling the features to the scale of the panoramic image 302 and aligning the extracted features in the panoramic image 302 so that the extracted features are located in a suitable location in the panoramic image 302. In the example shown in FIG. 3, the extracted snow feature is placed in the appropriate location of the snow in the panoramic image 302. How to align the extracted features may be based on a comparison of image 306 and panoramic image 302. A machine learning model may be trained for various features to identify their appropriate location so that any resulting generated panoramic imagery appears realistic.

[0060] Merging the scaled and aligned extracted features into the panoramic image may require additional processing. In some scenarios, other features may need to be removed from the panoramic image 302 or may need to be modified. In an example where the transformation 310 includes inserting the extracted lighting conditions into the panoramic image 302, the existing lighting conditions in the panoramic image 302 are transformed into the extracted lighting conditions. In some scenarios, the extracted features may be inserted into the panoramic image 302 as an overlay covering a portion of the panoramic image 302. For example, a snow pattern extracted from the image 306 may be overlaid on a portion of the panoramic image 302. Applying the transformation 310 to the panoramic image 302 produces a new panoramic image 312 that shows location B to be covered in snow.

[0061] Server 130 may use a machine learning model to perform some or all of the steps of transformation 310. For example, server 130 may use a machine learning model, such as a generative machine learning model as discussed with reference to Figures 6A-6B, to identify features, such as patterns that correspond to weather or light conditions, extract such features, and merge those features into the panoramic imagery. In this manner, the machine learning model is specifically adapted for the technical objective of generating realistic and accurate panoramic imagery that accurately reflects different environmental conditions.

[0062] Additionally, the transformation of a panoramic image may include a combination of transform 210 and transform 310. For example, panoramic image 202 may be transformed to include both physical objects from image 206 using steps of transform 210 and weather conditions using steps of transform 310.

[0063] In addition, after panoramic image 312 is generated, additional panoramic images showing location B, or locations near location B, may be generated (e.g., in the same manner as and / or based on panoramic image 312) and stitched together to form roadside images. The roadside images form a virtual environment through which a user may navigate as if the user were traveling along the route or road from which the panoramic image was captured.

[0064] As discussed above with respect to transformation 210, in some implementations, client device 102 may perform transformation 310 rather than server 130. For example, client device 102 may receive panoramic image 302 from server 130 and generate panoramic image 312 in response to a request from a user received by panoramic image application 108.

[0065] FIG. 4A illustrates an exemplary user interface 400 that displays an existing panoramic image and a notification that the existing panoramic image is out of date. The user interface 400 may be displayed via a display 402 (e.g., of the user interface 116) of the client device 102. The user interface 400 may be a user interface of the panoramic image application 108. In response to a request from a user, the panoramic image application 108 may display panoramic imagery available for a location. In the example illustrated in FIG. 4A, the panoramic image application 108 operates in a roadside mode that allows a user to navigate through a virtual environment formed by the panoramic images. The panoramic imagery available for the location includes the panoramic image 202.

[0066] In addition to the panoramic image 202, the user interface 400 may include user selectable options and information regarding the location. User interface element 404 includes an indication of the location where the user is virtually located and user selectable options, such as an option to navigate to a map of the location or to view older roadside imagery of the location. User interface element 406 may include a map of the location and may include an icon 408 within the map. The icon 408 indicates where the user is located and the perspective from which the user is looking (i.e., where the roadside imagery was captured, the direction the user is looking within the virtual environment). Additionally, user interface element 410 includes options for selecting a zoom level, rotating the view, or viewing additional imagery available at or near the location.

[0067] As discussed with reference to FIG. 2, the panoramic image application 108 may determine that an available panoramic image 202 may be out of date (e.g., based on a timestamp of the panoramic image 202, based on an image captured at the location and stored in the UGC database 142 showing an object not present in the panoramic image 202, based on a flag for the image location indicating that the image or the appearance of the location has changed, etc.). The panoramic image application 108 may display a notification 412 indicating that the panoramic image is out of date. The notification 412 may also ask whether the user would like a new panoramic image to be generated. The user may select the notification 412 to indicate to the panoramic image application 108 to generate or request a panoramic image to be generated for the location. In response, the panoramic image application 108 may generate or send a request to the server 130 to generate a new panoramic image 212. The new panoramic image 212 may be generated via a transformation 210, as discussed with reference to FIG. 2. After generation, the new panoramic image may be temporarily stored by the panoramic image application for viewing by the user, and may later be deleted when no longer needed. In this way, storage space is conserved because generated images are generated only on demand and not permanently stored.

[0068] Additionally, in some cases, the panoramic image application 108 may automatically generate and present the new panoramic image 212 to the user without first presenting the notification 412 and / or the panoramic image 202. The panoramic image application 108 may choose this implementation, for example, if the quality of the new panoramic image 212 is high (e.g., the new panoramic image 212 is realistic as measured by the discriminator network) or if there are a large number of images in the UGC database 142 for that location. The panoramic image application 108 may display a notification to the user that the displayed image has been generated and may display a user-selectable option to go back and view the uncomposite panoramic image 202.

[0069] Referring to FIG. 4B, user interface 420 displays generated panoramic image 212. User interface 420 may be similar to user interface 400, but includes new panoramic image 212. Client device 102 may display user interface 420 after retrieving new panoramic image 212. User interface 420 may include notification 416 indicating that panoramic image 212 is a generated image. As the user navigates within the virtual roadside environment, panoramic image application 108 may continue to display the generated panoramic image and may continue to display notification 416. New generated panoramic images for a location may be generated as the user navigates, or may be pre-cached for a location ahead of or near the user's virtual location. If the quality of the generated images (e.g., quality as measured by an image quality standard or as assessed by a discriminator, as discussed with reference to FIGS. 6A-6B) decreases, panoramic image application 108 may stop displaying the generated panoramic images and return to displaying panoramic images available from panoramic image database 140. In such a case, the panoramic image application 108 may display a notice to the user explaining that the quality of the generated image has been reduced and is no longer available. In this way, the user is always provided with high quality images rather than images that may be potentially inaccurate (i.e., do not accurately represent a given geographic area).

[0070] 5A illustrates an exemplary user interface 500 illustrating an existing panoramic image showing a geographic location and a user-selectable option to access a new panoramic image showing the geographic location under different environmental conditions. User interface 500 may be displayed via a display 502 of client device 102 (e.g., of user interface 116). User interface 500 is generally similar to user interface 400, but illustrates panoramic image 302 and a tool 512. User interface 500 also includes user interface elements 504, 506, 508, and 510, which may be similar to user interface elements 404, 406, 408, and 410, respectively. Panoramic image 302 showing a different location from panoramic image 202 illustrates the location during summer and / or when snow is not present.

[0071] The tool 512 includes user-selectable options for converting the panoramic image 302 into a panoramic image showing the location under different conditions. For example, the tool 512 may include options such as different lighting conditions (e.g., sunrise, day, sunset, night) and / or weather conditions (e.g., rain, snow, fog). Depending on what condition the user selects using the tool 512, the panoramic image application 108 can generate or send a request to the server 130 to generate (e.g., via the conversion 310) a new panoramic image showing the location under that condition. Depending on the implementation and / or scenario, the panoramic image application 108 may display a combination of the tool 512 and the notification 412 to allow the user to request an updated image that also shows the location under different conditions. In addition, the user may select multiple conditions of the tool 512. Furthermore, the panoramic image application 108 may display different options within the tool 512 depending on the location. For example, the panoramic image application 108 may determine (e.g., based on information from the map database 144) that it frequently snows at the location and include "snow" in the tool 512. As another example, the panoramic image application 108 may determine (e.g., based on information from the map database 144) that visitors travel to or take pictures of the location at certain times of day and include those times as suggestions (e.g., sunset) in the tool 512. As a further example, the panoramic image application 108 may determine (e.g., based on information from the map database 144) that the location is frequently affected by some particular weather pattern, including severe weather, and include a suggestion to view a generated panoramic image of the location that is likely to occur under these conditions.

[0072] In the example shown in FIG. 5B, the user selects the option "snow" from tools 512. In response, client device 102 retrieves and displays new panoramic image 312 as shown in user interface 520. New panoramic image 312 shows the location that will appear covered in snow. Tool 514 is similar to tool 512, but indicates that the user selected the "snow" option. Additionally, user interface 520 includes notification 516, which may be similar to notification 416, indicating that panoramic image 312 will be generated. As described with reference to FIG. 4B, as the user navigates within the virtual roadside environment, panoramic image application 108 may continue to display the panoramic image generated under the selected conditions and may continue to display notification 516. New generated panoramic images may be generated as the user navigates, or may be pre-cached for the location in front of or near the user's virtual location. Additionally, panoramic image application 108 may monitor the quality of the generated panoramic image and stop displaying the generated panoramic image if the quality degrades.

[0073] As described above, transformations such as transform 210, transform 310, and the combination of transforms 210 and 310 may be performed utilizing machine learning models. A generative adversarial network is an example of a machine learning model that can be trained and utilized to generate panoramic imagery including physical objects and / or features. FIGS. 6A-6B illustrate training a machine learning model and applying a trained machine learning model to generate panoramic imagery, respectively. Some of the blocks in FIGS. 6A-6B represent data structures or memories, registers, or state variables that store data structures (e.g., blocks 606a-n, 608a-n, 622, 626, 628), other blocks represent hardware and / or software components (e.g., blocks 610, 612, 616), and other blocks represent output data (e.g., blocks 614a-n, 634, 636). Input signals are represented by arrows labeled with the corresponding signal names.

[0074] Generally, as described above, panoramic image database 140 stores panoramic imagery, such as 360-degree panoramas and roadside imagery. When a user requests a panoramic imagery showing a geographic location under given conditions (e.g., current environmental conditions) that are not reflected in existing panoramic imagery stored in panoramic image database 140, panoramic image generation module 136 may generate a new panoramic imagery showing the geographic location under the given conditions.

[0075] FIG. 6A illustrates generally how the server 130 may train the panoramic image generation module 136 of FIG. 1 to generate a panoramic image. The panoramic image generation module 136 may include a generative machine learning engine 610 (which may be a GAN) for generating new panoramic images based on a generative machine learning model approach. A generative machine learning model approach, broadly defined, involves training an engine to learn regularities and / or patterns within a set of input data so that the generative engine may generate new examples of the input data. As the generative engine is trained on more inputs, the generated new examples of the engine may have increasing similarity to the input data. Thus, the goal of a generative machine learning model approach is to enable the generation of new examples of input data that are similar to the original input data. Although this disclosure describes generating panoramic images primarily with reference to GANs, the generative machine learning engine 610 may include other types of machine learning models. Additionally, as discussed above, different machine learning models may perform different steps of a panoramic image generation sequence.

[0076] To generate the panoramic imagery, the generative machine learning engine 610 receives training data from various sources. The training data includes a first set of panoramic images 606a for a first geographic area retrieved from the panoramic image database 140. The first set of panoramic images 606a may include panoramic images and roadside images. For example, the first set of panoramic images 606a may be roadside images of a first geographic location captured from the perspective of a particular route. The panoramic images of the first set of panoramic images 606a may show the first geographic area under different conditions. For example, the panoramic images may show the first geographic area under various weather conditions, under various lighting conditions, in different seasons, before and after changes to the geographic area (e.g., at an earlier time, at a later time, before a natural disaster, after a natural disaster), and / or under various combinations of conditions. In addition, the training data includes a first set of images 608a for the first geographic area retrieved from the UGC database 142. The images from the first set of images 608a may be panoramic images and / or non-panoramic images.

[0077] The training data also includes a second set 606b of panoramic images for a second geographic area that may be similar to the first set 606a of panoramic images, but for a second geographic area, retrieved from the panoramic image database 140. Additionally, the training data may include a second set 608b of images for a second geographic area that may be similar to the first set 608a of images, but for a second geographic area, retrieved from the UGC database. Similarly, the training data includes an nth set 606n of panoramic images for an nth geographic area, with an nth set 608n of images for the nth geographic area. The sets 606a-606n of panoramic images may represent available panoramic images for the respective geographic areas from the panoramic image database 140.

[0078] Upon retrieving the training data, the generative machine learning engine 610 may utilize the training data to train the generator 612 and the discriminator 616. For example, the generative machine learning engine 610 may pass the set of panoramic images 606a-606n and the set of images 608a-608n through the generator 612 to create sets of generated panoramic images 614a-614n. Each set of generated panoramic images 614a-614n may include any number of generated panoramic images. The generator 612 may stitch the generated panoramic images together to create roadside images.

[0079] The generator 612 may then pass the generated panoramic images 614a-614n to a discriminator 616. The discriminator 616 may also receive the training panoramic images 606a-606n. Using both sets of data (e.g., 606a-606n and 614a-614n), the discriminator 616 may attempt to determine which images are not images in the set of panoramic images 606a-606n. In other words, the discriminator 616 classifies which images in the generated panoramic images 614a-614n are "real," i.e., photographs captured by one or more cameras, or "fake," i.e., generated images.

[0080] For example, the discriminator 616 may analyze each image included in the first set of panoramic images 606a and each image included in the first set of generated panoramic images 614a. The discriminator 616 may determine consistent image characteristics, such as road placement, landmark placement, or other content of the images that indicate a geographic location. In addition to content, the characteristics may include image parameters such as size, aspect ratio, projection type, etc. In some implementations, the discriminator 616 may also use the images 608a-608n to determine consistent characteristics, such as consistent characteristics of the images that indicate a geographic location. In such implementations, if the first set of images 608a includes non-panoramic images, the discriminator 616 will predict that some image parameters between the first set of images 608a, the first set of panoramic images 606a, and the first set of generated panoramic images 614a, for example, will be different.

[0081] If the discriminator 616 determines that a particular image is not an actual image (e.g., because the particular image includes characteristics that deviate from the consistent characteristics), the discriminator 616 may flag the particular image as a generated image. The discriminator 616 may then return the flagged image to the generator 612 and / or may indicate to the generator 612 that an image from the set of otherwise generated panoramic images 614a-614n is not sufficiently similar to an actual image, such as the panoramic images 606a-606n (e.g., the output may indicate whether the input image is "real" or "generated"). The generator 612 may analyze the flagged image to determine the characteristics of the flagged image that resulted in the flagged image. The discriminator 616 classification may thus serve as further training data for the generator 612. Thus, in subsequent iterations of the generative machine learning process, the generator 612 may modify the panoramic image generation process to avoid similar flagging results from the discriminator 616. In this manner, the generative machine learning engine 610 can progressively generate panoramic images that more closely correspond to the panoramic images 606 a-606 n, which in turn allows accurate panoramic images to be generated that accurately depict a geographic area.

[0082] The server 130 may train the generative machine learning engine 610 to learn a transformation from a first panoramic image of a geographic location to a second panoramic image of the geographic location that shows the geographic location with different characteristics. Such characteristics include different conditions, such as weather conditions or light conditions, or different physical objects. In particular, the generative machine learning engine 610 may be trained to perform the transformation 210, the transformation 310, and a combination of the transformations 210 and 310.

[0083] For example, the server 130 may train the generator 616 to insert physical objects from an image of a geographic location captured at a later time into a panoramic image captured at an earlier time to create a "latest" image of the geographic location. The generator 616 may be trained to perform the steps discussed above with respect to the transformation 210, including extracting physical objects from one or more images, mapping the extracted physical objects into a coordinate system of the projection type of the source panoramic image, and merging the extracted physical objects into the source panoramic image to generate a new panoramic image. To merge the extracted physical objects, the generator 616 may be trained to identify scaling factors for the sizes of the different images and how to align the physical objects into the existing panoramic image. The generator 616 may also be trained to blend the physical objects into the existing panoramic image in a way that looks realistic.

[0084] Additionally, the server 130 may train the generator 616 to insert features from the images into the panoramic image. The features may be associated with weather conditions, lighting conditions, or other transient conditions that may not be reflected in the available panoramic or roadside imagery. The generator 616 may be trained to perform the steps discussed above with respect to the transformation 310, including identifying features from one or more images, mapping the extracted features into a coordinate system of the projection type of the source panoramic image, and merging the extracted features into the source panoramic image to generate a new panoramic image. How the generator 616 identifies different features may vary from feature to feature. For example, the generator 616 may identify one or more patterns associated with different weather conditions, one or more lighting conditions associated with different amounts of daylight, one or more patterns indicative of different seasons, etc. The generator 616 may also be trained to generate roadside imagery based on the generated panoramic imagery, which may be navigated by a user. The generator 616 may be trained to generate roadside imagery by stitching the generated panoramic images together.

[0085] The trained generative machine learning engine 610 may be used to generate panoramic imagery that reflects specific conditions for a particular geographic location. To illustrate, as shown in the example scenario of FIG. 6B, the generative machine learning engine 610 may receive data indicating a request 622 to generate a new panoramic imagery. The request 622 may indicate the requested conditions to be reflected in the generated panoramic imagery and may indicate the geographic location to be shown in the generated panoramic imagery. The requested conditions may be expressed as text (e.g., "snow covered") or an image (e.g., an image of the location under snowy conditions). In addition, the request 622 may indicate a viewpoint or vantage point from which a panoramic imagery is requested (e.g., from a particular street or route). The data in the request 622 may be indicated by a user interacting with the panoramic imagery application 108, and the new panoramic imagery may be generated in real time or may be generated and cached for the user. In other implementations, the request 622 may not be associated with a particular user, but may be associated with a request for a batch job for panoramic imagery that reflects different conditions of one or more geographic locations.

[0086] For example, in implementations where the request 622 is associated with a particular user, the user may be interacting with the panoramic image application 108. The panoramic image application 108 may request a panoramic image for a location from the server 130, which may provide an available panoramic image for the geographic location, referred to in the context of FIG. 6B as the source panoramic image 626, for display on the client device 102. In response, the user may request an updated panoramic image that reflects the geographic location at a more recent time or that reflects a different condition. In some embodiments, the server 130 and / or the panoramic image application 108 may automatically determine that the source panoramic image 626 is out of date (e.g., by comparing a timestamp of the source panoramic image 626 to the time of the user's request for a panoramic image or by comparing the source panoramic image 626 to a recently retrieved image from the UGC database 642) and automatically request a generated panoramic image. The panoramic image application 108 may ask the user if he or she would like to receive an updated panoramic image. Similarly, in some embodiments, server 130 and / or panoramic image application 108 may automatically determine that source panoramic image 626 exhibits certain conditions and automatically request a generated panoramic image that reflects different conditions. Panoramic image application 108 may ask the user whether the user would like to receive a panoramic image of the geographic location under different conditions.

[0087] In either case, the engine 610 retrieves a source panoramic image 626 for the geographic location from the panoramic image database 140. The source panoramic image 626 may be a roadside image consisting of multiple panoramic images stitched together. Correspondingly, any generated panoramic image may also be a roadside image. In addition, the engine 610 also retrieves one or more source UGC images 628 that are UGC images (panoramic and / or non-panoramic) available for the geographic location from the UGC database 142 or from different geographic locations that indicate the status included in the request 622. In some scenarios, the source UGC image 628 may be unavailable (e.g., no UGC images may exist for the geographic location). Once the engine 610 receives the request 622 and both the source panoramic image 626 and the source UGC image 628, the engine 610 may proceed with the panoramic image generation process.

[0088] The engine 610 may first pass the request 622 and both the source panoramic image 626 and the source UGC image 628 (if available) to the generator 612. The generator 612 may then analyze the source panoramic image 626 and the conditions included in the request 622 in conjunction with the source UGC image 628 to generate a new panoramic image 634. The generated panoramic image 634 may be one or more panoramic images, and may be a roadside image representing panoramic images stitched together. With reference to the example of FIG. 2, the generator 612 may generate the latest panoramic image 212 based on the existing panoramic image 202 (e.g., the source panoramic image 626) and a more recent image 206 (e.g., the source UGC image 628) even though the generator 612 does not have a panoramic image that includes the physical object shown in the image 206. With reference to the example of FIG. 3, the generator 612 may generate the panoramic image 312 based on the existing panoramic image 302 (e.g., the source panoramic image 626). In such an example, images of the location may not be available for the conditions included in the request 622, but the generator 612 may still generate the panoramic image 312 after being trained using images from other locations that exhibit those conditions.

[0089] The generator 612 can pass the generated panoramic image 634 to the discriminator 616 along with the source panoramic image 626. The generator 612 can also pass other training panoramic images 606 (e.g., from the panoramic images 606a-606n) to the discriminator 616. The discriminator can then attempt to determine which of the received images is not the source panoramic image 626 or the training panoramic image 606. If the discriminator 616 determines that the generated panoramic image 634 contains characteristics that deviate beyond what the discriminator 616 expects a panoramic image for the geographic location to have, the discriminator 616 can flag the generated panoramic image 634 as described above. However, if the discriminator 616 does not flag the generated panoramic image 634, the engine 610 can determine that the generated panoramic image should be sent to a user for display (or stored for later transmission to a user). Thus, the engine 610 may designate the "passing" generated panoramic image 634 as the generated panoramic image 636 and transmit the image to the client device 102 for display (or may store the generated panoramic image 636 in the panoramic image database 140 or another similar database for later display on the client device). In this manner, the GAN is specifically adapted to generate panoramic images and determine the accuracy of the generated images such that only generated panoramic images that show the geographic area sufficiently accurately are provided to the user. In other words, the GAN is specifically adapted for the technical purpose of generating and providing sufficiently accurate panoramic images to the user. In some implementations, the generative machine learning engine 610 may not pass the generated panoramic image 634 to the discriminator 616 before designating the generated panoramic image as the "passing" generated panoramic image 636. Thus, the generative machine learning engine 610 may use the discriminator 616 to train the generator 612 after the generator 612 is deployed, but the images may not pass through the discriminator 616 before being displayed to the user.

[0090] 7 is a flow diagram of an example method 700 for generating a panoramic image that may be implemented within the computing device of FIG. 1. In a scenario in which the server 130 implements the panoramic image generation module 136, the method 700 may be performed by the server 130. In a scenario in which the client device 102 implements the panoramic image generation module 136, the method 700 may be performed by the client device 102. In some implementations, the method 700 may begin in response to a user request for a panoramic image of a geographic area (e.g., a request made via a panoramic image application 108 implemented at the client device 102). In other implementations, the method 700 may be part of a processing job to generate and cache a new panoramic image for a geographic location or region such that the panoramic image is available for retrieval in response to a later user request.

[0091] At block 702, a computing device obtains a first panoramic image (e.g., panoramic image 202) showing a geographic area. The computing device may retrieve the first panoramic image from panoramic image database 140. Additionally, the first panoramic image may include multiple panoramic images that may be stitched together or stitched together to form a roadside image.

[0092] At block 704, the computing device acquires an image showing one or more physical objects not in the first panoramic image. The image may be a panoramic image that the computing device retrieves from the panoramic image database 140, or a panoramic or non-panoramic image that the computing device retrieves from the UGC database 142. The computing device may compare the image to the first panoramic image to determine whether the image includes a physical object not in the panoramic image. The image may show the same geographic area as the first panoramic image (or may show at least a portion of the same geographic area) or may show a different geographic area. In some implementations, block 704 may include acquiring a plurality of images showing one or more physical objects not in the first panoramic image.

[0093] At block 706, the computing device transforms (e.g., transform 210) the first panoramic image into a second panoramic image that shows one or more physical objects and includes at least a portion of the first panoramic image.

[0094] The second panoramic image may show a geographic area from the first panoramic image, but includes one or more physical objects. For example, the image acquired in block 704 may show a more recent geographic area than the first panoramic image. Physical objects may have been added to the geographic area or may have replaced other physical objects in the geographic area since the first panoramic image was captured. Thus, the second panoramic image shows the geographic area at the time the image was captured. If the image and the first panoramic image show the same or similar geographic area, physical objects that are not in the first panoramic image may be identified by comparing the images. If the image and the first panoramic image show different geographic locations, transforming the first panoramic image may include applying a machine learning model (e.g., generative machine learning engine 610) trained to identify a predetermined object, identifying that the predetermined object is in the image and not in the first panoramic image, and merging the predetermined object into the first panoramic image.

[0095] Transforming the first panoramic image into the second panoramic image may include extracting physical objects from the image. This transformation may also include performing a mathematical transformation on the image or the extracted physical objects to map the image or the extracted physical objects into the space of the first panoramic image. If the image is non-panoramic, the mathematical transformation may include identifying a projection type of the first panoramic image and mapping one or more physical objects into a coordinate system of the projection type. The mapped one or more physical objects may then be merged into the first panoramic image to generate the second panoramic image. Merging the physical objects may include inserting or overlaying the physical objects into the first panoramic image and may further include processing the first panoramic image to blend the physical objects into the first panoramic image (e.g., by removing existing physical objects, adjusting shadows, etc.).

[0096] Transforming the first panoramic image may include applying a generator network (e.g., generator 612) of a GAN (e.g., generative machine learning engine 610) including a generator network and optionally a discriminator network (e.g., discriminator 616) to perform one or more steps of the transformation described above (e.g., to extract physical objects from the image and merge the physical objects into the first panoramic image to generate a second panoramic image). Transforming the first panoramic image may also include applying the discriminator network to the second panoramic image to classify the second panoramic image as real or artifact. If the discriminator network classifies the second panoramic image as artifact, method 700 may include applying the generator network to the first panoramic image and the image to generate a third panoramic image showing a panoramic image showing one or more physical objects. The third panoramic image may be passed to the discriminator.

[0097] Further, transforming the first panoramic image may include inserting other features in addition to the physical objects (e.g., elements of transform 310). For example, a generator network may be applied to the first panoramic image to insert one or more patterns indicative of weather conditions not present in the first panoramic image. The generator network may be trained to identify one or more patterns indicative of weather conditions using a plurality of training panoramic images and a plurality of training images (e.g., images 606a-606n, 608a-608n). As another example, a generator network may be applied to the first panoramic image to insert one or more lighting conditions indicative of an amount of sunlight into the first panoramic image, where the amount of sunlight is different than that shown in the first panoramic image. The generator network may be trained to identify one or more lighting conditions using a plurality of training panoramic images and a plurality of training images (e.g., images 606a-606n, 608a-608n).

[0098] Transforming the first panoramic image into the second panoramic image may include generating a plurality of panoramic images and stitching the panoramic images together to form a roadside image. A generator network may be trained to perform such stitching.

[0099] If not generated at the client device 102, the second panoramic image may be transmitted to the client device 102. The client device 102 may then display the second panoramic image to the user.

[0100] 8 is a flow diagram of another exemplary method for generating a panoramic image that may be implemented within the computing device of FIG. 1. Similar to method 700, in a scenario in which server 130 implements panoramic image generation module 136, method 800 may be performed by server 130. In a scenario in which client device 102 implements panoramic image generation module 136, method 800 may be performed by client device 102. In some implementations, method 800 may begin in response to a user request for a panoramic image of a geographic area (e.g., a request made via panoramic image application 108 implemented at client device 102). In other implementations, method 800 may be part of a processing job for generating and caching a new panoramic image for a geographic location or region such that the panoramic image is available for retrieval in response to a later user request.

[0101] At block 802, a computing device acquires a first panoramic image (e.g., panoramic image 302). The computing device may retrieve the first panoramic image from panoramic image database 140. Additionally, the first panoramic image may include multiple images that can be stitched together or stitched together to form a roadside image.

[0102] In block 804, the computing device acquires a non-panoramic image showing one or more features not present in the first panoramic image. The non-panoramic image may be retrieved from the UGC database 142. The computing device may compare the image to the first panoramic image to determine if the image includes features not present in the panoramic image. The image may show the same geographic area (or at least a portion of the same geographic area) as the first panoramic image or a different geographic area. In some implementations, block 804 may include acquiring a plurality of images showing the features not present in the first panoramic image. The computing device may retrieve the non-panoramic image (or images) including some features based on a state indicated in the user request (e.g., request 622).

[0103] In block 806, the computing device applies the non-panoramic image to the first panoramic image to show one or more features in the second panoramic image. Applying the non-panoramic image to the first panoramic image may include steps similar to those discussed with reference to block 706. For example, applying the non-panoramic image to the first panoramic image may include extracting one or more features, identifying a projection type of the first panoramic image, mapping the one or more features to a coordinate system of the projection type, and merging the mapped features to the first panoramic image to generate the second panoramic image. If the features include physical objects, merging the features may include inserting the mapped physical objects into the first panoramic image. If the features include a manifestation of an environmental condition, merging the features may include overlaying the mapped manifestation on the first panoramic image. As discussed with reference to block 706, applying the non-panoramic image to the first panoramic image may be performed using a GAN.

[0104] Transforming the first panoramic image into the second panoramic image may include generating a plurality of panoramic images and stitching the panoramic images together to form a roadside image. A generator network may be trained to perform such stitching.

[0105] If not generated at the client device 102, the second panoramic image may be transmitted to the client device 102. The client device 102 may then display the second panoramic image to the user.

[0106] By way of example and not by way of limitation, the disclosure herein contemplates the following aspects.

[0107] 1. A method for creating a panoramic image, the method comprising: acquiring, by one or more processors, a first panoramic image showing a geographic area; acquiring, by one or more processors, images showing one or more physical objects not in the first panoramic image; and converting, by the one or more processors, the first panoramic image into a second panoramic image showing the one or more physical objects and including at least a portion of the first panoramic image.

[0108] 2. The method of aspect 1, wherein the step of transforming the first panoramic image includes applying a generator network of a generative adversarial network (GAN) including a generator network and a discriminator network to the first panoramic image and the image to extract one or more physical objects from the images and merge the one or more physical objects into the first panoramic image to generate the second panoramic image.

[0109] 3. The method of aspect 2, wherein the step of transforming the first panoramic image includes applying a discriminator network to the second panoramic image to classify the second panoramic image as real or artifact, and if the discriminator network classifies the second panoramic image as artifact, applying a generator network to the first panoramic image and the image to generate a third panoramic image showing one or more physical objects.

[0110] 4. The method of aspect 2 or 3, further comprising: training a generator network to use the plurality of panoramic images and the plurality of images to extract physical objects from the plurality of images and merge the physical objects into the plurality of panoramic images; and training a discriminator network to use the plurality of panoramic images and the plurality of generated panoramic images generated by the generator network to classify the plurality of generated panoramic images as artifacts or real objects, wherein the step of training the generator network further comprises the step of training the generator network using the classification from the discriminator network.

[0111] 5. The method of aspect 2, wherein the step of transforming the first panoramic image further includes a step of applying a generator network to the first panoramic image to insert one or more patterns indicative of weather conditions into the first panoramic image, wherein the one or more patterns are not present in the first panoramic image.

[0112] 6. The method of aspect 5, further comprising: training a generator network to use the plurality of panoramic images and the plurality of images to identify one or more patterns indicative of weather conditions and to insert the one or more patterns into the plurality of panoramic images.

[0113] 7. The method of aspect 2, wherein the step of transforming the first panoramic image includes a step of applying a generator network to the first panoramic image to insert one or more light conditions indicating a first amount of sunlight into the first panoramic image, the first amount of sunlight being different from a second amount of sunlight shown in the first panoramic image.

[0114] 8. The method of aspect 7, further comprising: training a generator network to use the plurality of panoramic images and the plurality of images to identify one or more light conditions and to insert the one or more light conditions into the plurality of panoramic images.

[0115] 9. The method of any one of aspects 1 to 8, wherein the image is a non-panoramic image, and the step of transforming the first panoramic image includes the steps of extracting one or more physical objects from the non-panoramic image, identifying a projection type of the first panoramic image, mapping the one or more physical objects to a coordinate system of the projection type, and merging the mapped one or more physical objects into the first panoramic image to generate a second panoramic image.

[0116] 10. The method of any one of aspects 1 to 9, wherein the image illustrates a geographic area and the step of transforming the first panoramic image includes a step of comparing the image with the first panoramic image to identify one or more physical objects that are not in the first panoramic image.

[0117] 11. The method of any one of aspects 1 to 9, wherein the geographic area is a first geographic area and the image illustrates a second geographic area, and transforming the first panoramic image includes applying a machine learning model to the image trained to identify a plurality of predetermined objects, identifying that the image includes a predetermined object of the plurality of predetermined objects that is not present in the first panoramic image based on application of the machine learning model, and merging the predetermined object into the first panoramic image.

[0118] 12. The method of any one of aspects 1 to 11, wherein the image is captured at a second time that is after a first time at which the first panoramic image is captured, and the step of transforming the first panoramic image includes the step of transforming the first panoramic image into a second panoramic image that shows the geographic area at the second time.

[0119] 13. A method for creating a panoramic image, the method comprising: acquiring, by one or more processors, a first panoramic image; acquiring, by one or more processors, a non-panoramic image showing one or more features; and applying, by the one or more processors, the non-panoramic image to the first panoramic image to show the one or more features in a second panoramic image.

[0120] 14. The method of aspect 13, wherein the step of applying the non-panoramic image to the first panoramic image includes the steps of extracting one or more features from the non-panoramic image, identifying a projection type of the first panoramic image, mapping the one or more features to a coordinate system of the projection type, and merging the one or more mapped features into the first panoramic image to generate a second panoramic image.

[0121] 15. The method of aspect 14, wherein the one or more features include a physical object, and merging the one or more mapped features includes inserting the mapped physical object into the first panoramic image.

[0122] 16. The method of aspect 14 or 15, wherein the one or more features include an expression of an environmental state, and the step of merging the one or more mapped features includes a step of overlaying the mapped expression onto the first panoramic image.

[0123] 17. The method of any one of aspects 13 to 16, wherein the step of applying the non-panoramic image to the first panoramic image includes applying the non-panoramic image to the first panoramic image using a generator network of a generative adversarial network (GAN) including a generator network and a discriminator network to extract one or more features from the non-panoramic image, identify a projection type of the first panoramic image, map the one or more features to a coordinate system of the projection type, and merge the one or more mapped features into the first panoramic image to create the second panoramic image.

[0124] 18. The method of aspect 17, further comprising: applying a discriminator network to the second panoramic image to classify the second panoramic image as real or artifact; and, if the discriminator network classifies the second panoramic image as artifact, applying a non-panoramic image to the first panoramic image using a generator network to generate a third panoramic image.

[0125] 19. The method of aspect 17 or 18, further comprising: training a generator network to apply the non-panoramic images to the panoramic images using the plurality of panoramic images and the plurality of non-panoramic images; and training a discriminator network to classify the plurality of generated panoramic images as artifacts or real objects using the plurality of panoramic images and the plurality of generated panoramic images generated by the generator network, wherein training the generator network further comprises training the generator network using the classifications from the discriminator network.

[0126] 20. The method of any one of aspects 13 to 19, wherein the first panoramic image indicates a geographic location, the non-panoramic image indicates a geographic location, and applying the non-panoramic image to the first panoramic image includes comparing the first panoramic image and the non-panoramic image to identify one or more features that are not present in the first panoramic image.

[0127] 21. A computing device for creating a panoramic image, the computing device comprising one or more processors configured to implement a method according to any one of aspects 1 to 20.

[0128] 22. A computer-readable medium carrying instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any one of aspects 1 to 20.

[0129] Additional considerations The following additional considerations apply to the foregoing discussion: Throughout this specification, multiple instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are shown and described as separate operations, one or more of the individual operations may be performed simultaneously, and it is not required that the operations be performed in the order shown. Structures and functions presented as separate components in example configurations may be implemented as combined structures or components. Similarly, structures and functions presented as single components may be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter of this disclosure.

[0130] Additionally, some embodiments have been described herein as including logic or some components, modules, or mechanisms. A module may constitute either a software module (e.g., code stored on a machine-readable medium) or a hardware module. A hardware module is a tangible unit capable of performing some operations and may be configured or arranged in a certain manner. In an exemplary embodiment, one or more computer systems (e.g., standalone, client or server computer systems) or one or more hardware modules of a computer system (e.g., a processor or group of processors) may be configured by software (e.g., an application or application portion) as a hardware module that operates to perform some operations described herein.

[0131] In various embodiments, a hardware module may be implemented mechanically or electronically. For example, a hardware module may comprise dedicated circuitry or logic that is permanently configured (e.g., as a dedicated processor, such as a field programmable gate array (FPGA) or application specific integrated circuit (ASIC)) to perform some operations. A hardware module may also comprise programmable logic or circuitry (e.g., as contained within a general-purpose processor or other programmable processor) that is temporarily configured by software to perform some operations. It will be appreciated that the decision to implement a hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured (e.g., configured by software) circuitry is driven by cost and time considerations.

[0132] The term hardware should therefore be understood to encompass tangible entities and thus entities that are physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform some of the operations described herein. Considering embodiments in which hardware modules are temporarily configured (e.g., programmed), the hardware modules may not each be configured or instantiated at any one time. For example, if the hardware modules include a general-purpose processor configured using software, the general-purpose processor may be configured as each of the different hardware modules at different times. The software may thus configure the processor, for example, to configure a particular hardware module at one time and to configure a different hardware module at a different time.

[0133] Hardware and software modules may provide information to and receive information from hardware and / or software modules. Thus, the described hardware modules may be considered to be communicatively coupled. When multiple such hardware or software modules are present simultaneously, communication may be achieved through signal transmission (e.g., via appropriate circuits or buses) connecting the hardware or software modules. In embodiments in which multiple hardware or software modules are configured or instantiated at different times, communication between such hardware or software modules may be achieved, for example, through storage and retrieval of information in memory structures to which the multiple hardware or software modules have access. For example, one hardware or software module may perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. Additional hardware or software modules may then later access the memory device to retrieve and process the stored output. Hardware and software modules may also initiate communication with input or output devices and may operate on resources (e.g., collections of information).

[0134] Various operations of the example methods described herein may be performed at least in part by one or more processors, temporarily or permanently configured (e.g., by software) to perform the associated operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions. Modules referenced herein may, in some example embodiments, include processor-implemented modules.

[0135] Similarly, the methods or routines described herein may be at least partially processor-implemented. For example, at least some of the operations of the methods may be performed by one or more processors or processor-implemented hardware modules. Execution of some of the operations may be distributed among one or more processors that are not only resident within a single machine, but are also spread across several machines. In some exemplary embodiments, one or more processors may be located within a single location (e.g., in a home environment, an office environment, or as a server farm), while in other embodiments, the processors may be distributed across several locations.

[0136] The one or more processors may also operate to support execution of associated operations within a "cloud computing" environment or as SaaS. For example, as indicated above, at least some of the operations may be performed by a group of computers (as one example of a machine that includes a processor), and these operations are accessed over a network (e.g., the Internet) or via one or more suitable interfaces (e.g., APIs).

[0137] Execution of some of the operations may be distributed among one or more processors that are not only resident within a single machine, but are spread across several machines. In some exemplary embodiments, one or more processors or processor-implemented modules may be located within a single geographic location (e.g., in a home environment, an office environment, or as a server farm). In other exemplary embodiments, one or more processors or processor-implemented modules may be distributed across several geographic locations.

[0138] Some portions of this specification are presented as algorithms or symbolic representations of operations on data stored as bits or binary digital signals in a machine memory (e.g., computer memory). These algorithms or symbolic representations are examples of techniques used by those skilled in the data processing arts to convey the substance of their work to others skilled in the art. An "algorithm" or "routine" as used herein is a self-consistent sequence of operations or similar processes that produce a desired result. In this context, algorithms, routines, and operations require physical manipulations of physical quantities. Typically, although not necessarily, such quantities may take the form of electrical, magnetic, or optical signals capable of being stored, accessed, transferred, combined, compared, or otherwise manipulated by a machine. It is sometimes convenient, primarily for reasons of common usage, to refer to such signals using words such as "data," "contents," "bits," "values," "elements," "symbols," "characteristics," "conditions," "numbers," "numeric values," and the like. However, these words are merely convenient labels to associate with the appropriate physical quantities.

[0139] Unless otherwise specified, discussions herein using words such as "processing," "computing," "calculating," "determining," "presenting," "displaying," and the like refer to machine (e.g., computer) actions or processes that manipulate or transform data represented as physical (e.g., electronic, magnetic, or optical) quantities in one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information.

[0140] As used herein, reference to "one embodiment" or "an embodiment" means that a particular element, feature, structure, or characteristic described with respect to that embodiment is included in at least one embodiment. The appearances of "in one embodiment" in various places in the specification do not necessarily refer to the same embodiment.

[0141] Some embodiments may be described using the terms "coupled" and "connected," along with derivatives thereof. For example, some embodiments may be described using the term "coupled" to indicate that two or more elements are in direct physical or electrical contact. However, the term "coupled" can also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other. The embodiments are not limited in this context.

[0142] As used herein, "comprises," "comprising," "includes," "including," "has," "having," or any other variation thereof, is intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to those elements and may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Furthermore, unless expressly stated to the contrary, "or" refers to an inclusive disjunction rather than an exclusive logic. For example, a condition A or B is satisfied by any one of the following: A is true (i.e., present) and B is false (i.e., not present), A is false (i.e., not present) and B is true (i.e., present), and A and B are both true (i.e., present).

[0143] In addition, the use of "a" or "an" is employed to describe elements and components of the embodiments of this specification. This is merely for convenience and to give a general sense of the description. This specification should be read to include one or at least one, and the singular also includes the plural, unless it is clear that it is meant otherwise.

[0144] After reading this disclosure, those skilled in the art will appreciate still further alternative structures and functional designs for generating panoramic images through the principles disclosed in this disclosure. Thus, while specific embodiments and applications have been shown and described, it should be understood that the disclosed embodiments are not limited to the precise structures and components disclosed herein. Various modifications, changes and alterations that will be apparent to those skilled in the art may be made in the arrangement, operation and details of the methods and apparatus disclosed herein without departing from the spirit and scope as defined in the appended claims. [Explanation of symbols]

[0145] 100 Panoramic Image Generation System 102 Client computing device, client device 104 Processor (CPU) 106 Memory 108 Panoramic Image Application, Smart Image Application 110 Operating System (OS) 112 Global Positioning System (GPS) module 114 Positioning module, network interface 116 User Interface 118 Input / Output (I / O) Interface 120 Network 130 Panoramic image generation server, server 132 processors 134 Memory 136 Panoramic Image Generation Module 140 Database, Panorama Image Database 142 Database, User Generated Content (UGC) Database 144 Database, Map Database 202 panoramic images, images 204 Signs 206 images 208 signs 210 Conversion 212 Panorama Images 302 Panorama Images 306 images 310 Conversion 312 Panorama Images 400 User Interface 402 Display 404 User Interface Elements 406 User Interface Elements 408 Icon 410 User Interface Elements 412 Notification 416 Notification 420 User Interface 500 User Interface 502 Display 504 User Interface Elements 506 User Interface Elements 508 User Interface Elements 510 User Interface Elements 512 Tools 514 Tools 516 notifications 520 User Interface 606a A first set of panoramic images, a set of panoramic images, a panoramic image 606b A second set of panoramic images, a set of panoramic images, a panoramic image 606n nth set of panoramic images, set of panoramic images, panoramic image 608a First set of images, set of images 608b Second set of images, set of images 608n nth set of images, set of images 610 Generative Machine Learning Engine, Engine 612 Generator 614a~614n Set of panoramic images, panoramic images 614a First set of panoramic images 616 Discriminator 622 request 626 Source Panorama Images 628 Source UGC Images 634 Panorama Statue 636 Panorama Statue 642 UGC database 700 methods 800 ways

Claims

1. 1. A method for producing a panoramic image, comprising: acquiring, by one or more processors, a first panoramic image showing a geographic area; acquiring, by the one or more processors, an image showing one or more physical objects not in the first panoramic image, the image showing at least a portion of the geographic area; comparing, by the one or more processors, the first panoramic image to the image to identify the one or more physical objects that are absent from the first panoramic image; transforming, by the one or more processors, the first panoramic image into a second panoramic image showing the one or more physical objects and including at least a portion of the first panoramic image based on the comparison; wherein the second panoramic image includes a pattern or visual manifestation of an environmental state or a pattern or visual manifestation of a transient state that is not present in the first panoramic image.

2. The step of transforming the first panoramic image further comprises: extracting the one or more physical objects from the image; Merging the one or more physical objects into the first panoramic image to generate the second panoramic image. applying a generator network of a generative adversarial network (GAN) including a generator network and a discriminator network to the first panoramic image and the image, 2. The method of claim 1, comprising:

3. The step of transforming the first panoramic image further comprises: applying the discriminator network to the second panoramic image to classify the second panoramic image as real or art; if the discriminator network classifies the second panoramic image as a product, applying the generator network to the first panoramic image and the image to generate a third panoramic image showing the one or more physical objects; 3. The method of claim 2, comprising:

4. training the generator network to use a plurality of training panoramic images and a plurality of training images to extract physical objects from the plurality of training images and to merge the physical objects into the plurality of training panoramic images; training the discriminator network to classify the plurality of generated panoramic images as artifacts or real objects using the plurality of training panoramic images and the plurality of generated panoramic images generated by the generator network; and wherein the training of the generator network further comprises training the generator network using classifications from the discriminator network.

5. The step of transforming the first panoramic image further comprises: applying the generator network to the first panoramic image to insert one or more patterns indicative of weather conditions into the first panoramic image, the one or more patterns being absent from the first panoramic image; 3. The method of claim 2, further comprising:

6. training the generator network to identify the one or more patterns indicative of the weather condition using a plurality of training panoramic images and a plurality of training images and to insert the one or more patterns into the plurality of training panoramic images.

6. The method of claim 5, further comprising:

7. The step of transforming the first panoramic image further comprises: applying the generator network to the first panoramic image to insert one or more light conditions indicative of a first amount of sunlight into the first panoramic image, the first amount of sunlight being different from a second amount of sunlight shown in the first panoramic image; 3. The method of claim 2, comprising:

8. training the generator network to identify the one or more light conditions using a plurality of training panoramic images and a plurality of training images and to insert the one or more light conditions into the plurality of training panoramic images; 8. The method of claim 7, further comprising:

9. the image is a non-panoramic image, and the step of transforming the first panoramic image comprises: extracting the one or more physical objects from the non-panoramic image; identifying a projection type of the first panoramic image; mapping said one or more physical objects into said projection-type coordinate system; merging the mapped one or more physical objects into the first panoramic image to generate the second panoramic image; 9. The method of any one of claims 1 to 8, comprising:

10. The step of transforming the first panoramic image further comprises: applying a machine learning model to the image, the machine learning model being trained to identify a plurality of predefined objects; identifying, based on the application of the machine learning model, that the image includes a predetermined object of the plurality of predetermined objects that is not present in the first panoramic image; merging the predetermined object into the first panoramic image; 10. The method of any one of claims 1 to 9, comprising:

11. The image is captured at a second time after a first time at which the first panoramic image was captured, and the step of transforming the first panoramic image includes: Transforming the first panoramic image into a second panoramic image showing the geographic area at the second time.

11. The method of any one of claims 1 to 10, comprising:

12. 1. A method for producing a panoramic image, comprising: acquiring, by one or more processors, a first panoramic image indicative of a geographic location; acquiring, by the one or more processors, a non-panoramic image showing one or more features not present in the first panoramic image, the non-panoramic image showing at least a portion of the geographic location; comparing, by the one or more processors, the first panoramic image to the non-panoramic image to identify the one or more features that are absent in the first panoramic image; applying, by the one or more processors, based on the comparison, the non-panoramic image to the first panoramic image to generate a second panoramic image exhibiting the one or more features and including at least a portion of the first panoramic image. wherein the second panoramic image includes a pattern or visual manifestation of an environmental state or a pattern or visual manifestation of a transient state that is not present in the first panoramic image.

13. said applying said non-panoramic image to said first panoramic image further comprising: extracting the one or more features from the non-panoramic image; identifying a projection type of the first panoramic image; mapping the one or more features into the projection-type coordinate system; merging the one or more mapped features into the first panoramic image to generate the second panoramic image; 13. The method of claim 12, comprising:

14. the one or more features include a physical object, and the step of merging the one or more mapped features further comprises: Inserting the mapped physical object into the first panoramic image.

14. The method of claim 13, comprising:

15. merging the one or more mapped features, overlaying the mapped expression onto the first panoramic image.

15. The method of claim 13 or 14, comprising:

16. said applying said non-panoramic image to said first panoramic image further comprising: extracting the one or more features from the non-panoramic image; Identifying a projection type of the first panoramic image; Mapping the one or more features into the projection-type coordinate system; Merging the one or more mapped features into the first panoramic image to generate the second panoramic image. applying the non-panoramic image to the first panoramic image using a generator network of a generative adversarial network (GAN) including a generator network and a discriminator network to 16. The method of any one of claims 12 to 15, comprising:

17. applying the discriminator network to the second panoramic image to classify the second panoramic image as real or art; if the discriminator network classifies the second panoramic image as a product, applying the non-panoramic image to the first panoramic image using the generator network to generate a third panoramic image; 17. The method of claim 16, further comprising:

18. training the generator network to apply non-panoramic images to panoramic images using a plurality of training panoramic images and a plurality of training non-panoramic images; training the discriminator network to classify the plurality of generated panoramic images as artifacts or real objects using the plurality of training panoramic images and the plurality of generated panoramic images generated by the generator network; and wherein the training of the generator network further comprises training the generator network using classifications from the discriminator network.

18. The method of claim 16 or 17.

19. 19. A computing device for creating a panoramic image, comprising one or more processors configured to implement the method of any one of claims 1 to 18.

Images