Three-dimensional mapping using different visual data sets

Abstract

A three-dimensional (3D) mapping system may be configured to generate a 3D map of a real-world environment using annotations of a larger set of image data, where ground images may be programmed with accurate tags using remote-sensed top image data. A 3D mapping system may implement photogrammetry to create a point cloud. Each pixel in the point cloud may be classified based on the consistency of each frame. The point clouds may be co-registered to a remote sensing reference data set to provide precise spatial coordinates for each pixel. Different patches of the point cloud may be spliced together to provide a complete 3D map of a given area, such as an urban area of a city.

Description

[0001] Priority declaration

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 335,552 filed on April 27, 2022 and U.S. Patent Application Serial No. 17 / 970,810 filed on October 21, 2022, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to special-purpose machines that manage data processing and improvements to such variants, and to techniques by which such special-purpose machines are improved over other special-purpose machines for image processing using different vision data sets. Background Art

[0004] Generating three-dimensional (3D) maps of real-world environments is computationally difficult. Implementing robotics and computer vision systems to map real-world environments involves significant expenditure on computer vision and robotics equipment as well as computationally intensive processing to generate accurate results. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] To easily identify the discussion of any particular element or act, the most significant digit(s) in a reference number refers to the figure ("FIG") number in which the element or act is first introduced.

[0006] Figure 1 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network according to an example implementation.

[0007] Figure 2 is a diagram showing a method for performing a Figure 1 A block diagram of the messaging system for further details.

[0008] Figure 3 is a schematic diagram illustrating data that may be stored in a database of a messaging server system according to certain example embodiments.

[0009] Figure 4 is a schematic diagram illustrating the structure of a message generated by a messaging client application for communications according to some embodiments.

[0010] Figure 5 is a schematic diagram illustrating an example access restriction process according to some example embodiments, according to which access to content (e.g., a multimedia payload of an ephemeral message and associated data) or a collection of content (e.g., an ephemeral message story) can be time-restricted (e.g., made ephemeral).

[0011] Figure 6 An example flow chart for generating a map from different image sources is shown, according to some example implementations.

[0012] Figure 7 An apparatus for generating different sets of image data from orthogonal viewing angles is shown according to some example embodiments.

[0013] Figure 8 A 3D map generated from different data is shown according to some example embodiments.

[0014] Fig. 9 is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium (eg, a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to some example embodiments. DETAILED DESCRIPTION

[0015] The following description includes the system, method, technology, instruction sequence and computer program product of the illustrative embodiment of the present disclosure. In the following description, for the purpose of illustration, many specific details are set forth, so as to provide an understanding of the various embodiments of the subject of the present invention. However, it is obvious to those skilled in the art that the embodiment of the subject of the present invention can be implemented without these specific details. Usually, known instruction examples, protocols, structures and technologies are not necessarily shown in detail.

[0016] One challenge of computer vision is to generate a sufficient amount of training data for the massive growth of various use cases. Well-labeled ground training data usually requires spending a lot of money on equipment that captures data (e.g., cars with LIDAR (light detection and ranging) and imaging systems that drive around the city to collect data about the city). In addition, these methods require a lot of computing resources to generate accurate results (e.g., generating an accurate point cloud or 3D map of the city from a large amount of LIDAR and image data). It is even more difficult to generate 3D maps using remote sensing data. Examples of remote sensing data include image or video data generated by drones and remote sensing data purchased from companies that own aerial systems (e.g., aircraft, satellites according to some example embodiments). In order to address the aforementioned problems, a 3D mapping system can be configured to generate a 3D map of a real-world environment using annotations of a larger image data set (e.g., an end-user provided video provided by one or more end users of a website). The image data set includes ground images that can be programmatically labeled with accurate labels (e.g., neural network image segmentation), wherein remotely sensed overhead image data (e.g., aerial LIDAR) and location data (e.g., GPS data) are used to improve and enhance label accuracy. In some example embodiments, a 3D mapping system implements photogrammetry (e.g., Alice Vision, COLMAP) to create a point cloud from an image (e.g., a video clip of a city). Each pixel in the point cloud can be classified based on the consistency of each frame of a given video. The point cloud can be co-registered to a remote sensing reference dataset (e.g., data provided by an aerial device) to provide precise spatial coordinates for each pixel. Different patches of the point cloud can be stitched together to provide a complete 3D map of a given area (such as a downtown area of ​​a city).

[0017] Figure 1 A block diagram of an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network 106 is shown. The messaging system 100 includes a plurality of client devices 102, each of which hosts a plurality of applications including a messaging client application 104. Each messaging client application 104 is communicatively coupled to other instances of the messaging client application 104 and a messaging server system 108 via a network 106 (e.g., the Internet).

[0018] Thus, each messaging client application 104 is able to communicate and exchange data with another messaging client application 104 and with a messaging server system 108 via the network 106. The data exchanged between messaging client applications 104 and between messaging client applications 104 and messaging server system 108 includes functions (e.g., commands to call functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0019] The messaging server system 108 provides server-side functionality to certain messaging client applications 104 via the network 106. Although certain functions of the messaging system 100 are described herein as being performed by the messaging client applications 104 or by the messaging server system 108, it should be understood that the location of certain functions is a design choice as to whether they are within the messaging client applications 104 or within the messaging server system 108. For example, it may be technically preferable to initially deploy certain technologies and functions within the messaging server system 108 and later migrate the technologies and functions to the messaging client applications 104 where the client devices 102 have sufficient processing power.

[0020] The messaging server system 108 supports various services and operations provided to the messaging client applications 104. Such operations include sending data to the messaging client applications 104, receiving data from the messaging client applications 104, and processing data generated by the messaging client applications 104. As examples, the data may include message content, client device information, geographic location information, media annotations and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 100 is invoked and controlled by functions available through the user interface of the messaging client applications 104.

[0021] Turning now specifically to the messaging server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to an application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 in which data associated with messages processed by the application server 112 is stored.

[0022] The API server 110 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the API server 110 provides a collection of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client application 104 in order to activate the functionality of the application server 112. The API server 110 exposes various functions supported by the application server 112, including account registration; login functionality; sending messages from a particular messaging client application 104 to another messaging client application 104 via the application server 112; sending media files (e.g., images or videos) from the messaging client application 104 to the messaging server application 114 for possible access by another messaging client application 104; setting of a collection of media data (e.g., stories); retrieval of such a collection; retrieval of a friend list of a user of the client device 102; retrieval of messages and content; adding and removing friends to or from a social graph; the location of friends within a social graph; and opening application events (e.g., involving the messaging client application 104).

[0023] The application server 112 hosts multiple applications and subsystems, including a messaging server application 114, an image processing system 116, a social networking system 122, and a mapping system 123. The messaging server application 114 implements many message processing techniques and functions, particularly relating to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client application 104. As will be described in more detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or libraries). The messaging server application 114 then makes these collections available to the messaging client application 104. Given the hardware requirements for such processing, the messaging server application 114 may also perform other processor- and memory-intensive data processing on the server side.

[0024] The application server 112 also includes an image processing system 116 dedicated to performing various image processing operations, typically on images or videos received within the payload of messages at the messaging server application 114 .

[0025] The social networking system 122 supports various social networking functions and services and makes these functions and services available to the messaging server application 114. To this end, the social networking system 122 maintains and accesses an entity graph within the database 120 (e.g., Figure 3304 in the entity graph 304). Examples of functionality and services supported by the social networking system 122 include identification of other users of the messaging system 100 with whom a particular user has a relationship or with whom a particular user "follows," and identification of other entities and particular users of interest. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 in which data associated with messages processed by the messaging server application 114 is stored. The mapping system 123 is configured to generate 3D maps from different image sources, as discussed in further detail below.

[0026] Figure 2 1 is a block diagram showing additional details about the messaging system 100 according to an example implementation. Specifically, the messaging system 100 is shown to include a messaging client application 104 and an application server 112, which in turn contains a number of subsystems, namely, a transient timer system 202, a collection management system 204, and an annotation system 206.

[0027] The transient timer system 202 is responsible for implementing temporary access to content permitted by the messaging client application 104 and the messaging server application 114. To this end, the transient timer system 202 includes a plurality of timers that selectively display and enable access to messages and associated content via the messaging client application 104 based on durations and display parameters associated with a message or a collection of messages (e.g., a story). Additional details regarding the operation of the transient timer system 202 are provided below.

[0028] The collection management system 204 is responsible for managing collections of media (e.g., collections of text, images, video, and audio data). In some examples, collections of content (e.g., messages, including images, videos, text, and audio) may be organized into "event libraries" or "event stories." Such collections may be made available for a specified period of time (such as the duration of the event to which the content relates). For example, content related to a concert may be made available as a "story" for the duration of the concert. The collection management system 204 may also be responsible for publishing an icon that provides notification of the existence of a particular collection to the user interface of the messaging client application 104.

[0029] The collection management system 204 also includes a curation interface 208 that allows a collection manager to manage and curate a particular collection of content. For example, the curation interface 208 enables an event organizer to curate a collection of content related to a particular event (e.g., to remove inappropriate content or redundant messages). In addition, the collection management system 204 uses machine vision (or image recognition technology) and content rules to automatically curate content collections. In some embodiments, compensation may be paid to users to include user-generated content in a collection. In such a case, the curation interface 208 operates to automatically pay such users for use of their content.

[0030] The annotation system 206 provides various functions that enable users to annotate or otherwise modify or edit media content associated with a message. For example, the annotation system 206 provides functions related to generating and publishing media overlays for messages processed by the messaging system 100. The annotation system 206 is operable to supply media overlays (e.g., geographic filters or filters) to the messaging client application 104 based on the geographic location of the client device 102. In another example, the annotation system 206 is operable to supply media overlays to the messaging client application 104 based on other information (such as social network information of the user of the client device 102). The media overlay can include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos) at the client device 102. For example, the media overlay includes text that can be overlaid on a photo generated by the client device 102. In another example, the media overlay includes a location identifier (e.g., Venice Beach), the name of a live event, or a business name (e.g., Beach Cafe). In another example, the annotation system 206 uses the geographic location of the client device 102 to identify a media overlay that includes the name of a business at the geographic location of the client device 102. The media overlay may include other tags associated with the business. The media overlay may be stored in the database 120 and accessed through the database server 118.

[0031] In one example implementation, the annotation system 206 provides a user-based publishing platform that enables a user to select a geographic location on a map and upload content associated with the selected geographic location. The user can also specify the context in which particular content should be provided to other users. The annotation system 206 generates a media overlay including the uploaded content and associates the uploaded content with the selected geographic location.

[0032] In another example implementation, the annotation system 206 provides a merchant-based publishing platform that enables merchants to select specific media coverage associated with a geographic location via a bidding process. For example, the annotation system 206 associates the media coverage of the merchant with the highest bid with the corresponding geographic location within a predefined amount of time.

[0033] Figure 3 3 is a schematic diagram illustrating data 300 that may be stored in a database 120 of a messaging server system 108 according to certain example embodiments. Although the contents of the database 120 are illustrated as including a plurality of tables, it should be understood that the data 300 may be stored in other types of data structures (e.g., as an object-oriented database). The database 120 includes message data stored in a message table 314. An entity table 302 stores entity data, which includes an entity graph 304. The entities whose records are maintained in the entity table 302 may include individuals, corporate entities, organizations, objects, places, events, and the like. Regardless of the type, any entity that stores data about the messaging server system 108 may be an identified entity. Each entity is provided with a unique identifier and an entity type identifier (not shown).

[0034] The entity graph 304 also stores information about relationships and associations between or among entities. For example, such relationships may be social relationships, professional relationships (e.g., working in a common company or organization) based on interests or activities.

[0035] The database 120 also stores annotation data in the form of an example of a filter in the annotation table 312. The filter whose data is stored in the annotation table 312 is associated with a video (whose data is stored in the video table 310) and / or an image (whose data is stored in the image table 308) and is applied to the video (whose data is stored in the video table 310) and / or the image (whose data is stored in the image table 308). In one example, the filter is displayed as an overlay on the image or video during presentation to the recipient user. The filter can be of various types, including a user-selected filter from a library of filters presented to the sending user by the messaging client application 104 when the sending user is composing a message. Other types of filters include geographic location filters (also referred to as geographic filters), which can be presented to the sending user based on geographic location. For example, based on geographic location information determined by a global positioning system (GPS) unit of the client device 102, the messaging client application 104 can present a neighborhood or specific location-specific geographic location filter within the user interface. Another type of filter is a data filter, which can be selectively presented to the sending user by the messaging client application 104 based on other input or information collected during the message creation process by the client device 102. Examples of data filters include the current temperature at a particular location, the current speed at which the sending user is traveling, the battery life of the client device 102, or the current time.

[0036] Other annotation data that may be stored in the image table 308 is so-called "shot" data. A "shot" may be real-time special effects and sounds that may be added to an image or video.

[0037] As mentioned above, the video table 310 stores video data, in one embodiment, associated with messages whose records are maintained in the message table 314. Similarly, the image table 308 stores image data associated with messages whose message data is stored in the message table 314. The entity table 302 may associate various annotations from the annotation table 312 with the various images and videos stored in the image table 308 and the video table 310.

[0038] The story table 306 stores data about a collection of messages and associated images, videos, or audio data that are compiled into a collection (e.g., a story or library). The creation of a particular collection can be initiated by a particular user (e.g., each user whose record is maintained in the entity table 302). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by the user. To this end, the user interface of the messaging client application 104 may include an icon that the user can select to enable the sending user to add specific content to his or her personal story.

[0039] A collection may also constitute a "live story," which is a collection of content from multiple users that is created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" may constitute a curated stream of user-submitted content from different locations and events. Users whose client devices 102 have location services enabled and who are at a common location event at a particular time may be presented with an option, such as via a user interface of the messaging client application 104, to contribute content to a particular live story. The live story may be identified to the user by the messaging client application 104 based on his or her location. The end result is a "live story" told from the perspective of the community.

[0040] Another type of content collection is called a "location story," which enables users whose client devices 102 are located in a particular geographic location (e.g., on a college or university campus) to contribute to a particular collection. In some implementations, contributions to location stories may require a second degree of authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student on a university campus).

[0041] Figure 4 is a schematic diagram illustrating the structure of a message 400 generated by a messaging client application 104 for transmission to another messaging client application 104 or a messaging server application 114 according to some embodiments. The content of a particular message 400 is used to populate a message table 314 stored in a database 120 accessible to a messaging server application 114. Similarly, the content of the message 400 is stored in memory as "in transit" or "in flight" data of a client device 102 or application server 112. The message 400 is shown to include the following components:

[0042] Message identifier 402 : a unique identifier that identifies the message 400 .

[0043] Message text payload 404 : Text to be generated by a user via the user interface of the client device 102 and included in the message 400 .

[0044] Message image payload 406: captured by the camera component of the client device 102 or

[0045] Image data retrieved from the memory of the client device 102 and included in the message 400. • Message Video Payload 408: Video data captured by the camera component or retrieved from the memory component of the client device 102 and included in the message 400.

[0046] Message audio payload 410 : audio data captured by a microphone or retrieved from a memory component of the client device 102 and included in the message 400 .

[0047] Message annotation 412: Annotation data representing an annotation (eg, a filter, sticker, or other enhancement) to be applied to the message image payload 406, the message video payload 408, or the message audio payload 410 of the message 400.

[0048] Message duration parameter 414: indicates the content of message 400 in seconds (e.g.

[0049] Parameter values ​​for the amount of time that a message image payload 406, a message video payload 408, and a message audio payload 410) will be presented to a user via the messaging client application 104 or made accessible to the user.

[0050] Message geolocation parameters 416: Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message 400. Multiple message geolocation parameter 416 values ​​may be included in the payload, with each of these parameter values ​​being associated with a respective content item included in the content (e.g., a specific image in the message image payload 406 or a specific video in the message video payload 408).

[0051] Message story identifier 418: A value that identifies one or more content collections (e.g., "stories") associated with a particular content item in message image payload 406 of message 400. For example, multiple images within message image payload 406 may each be associated with multiple content collections using an identifier value.

[0052] Message tags 420: One or more tags, each of the one or more tags indicating a subject of the content included in the message payload. For example, where a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value may be included within the message tags 420 indicating the relevant animal. The tag value may be manually generated based on user input or may be automatically generated using, for example, image recognition.

[0053] • Message sender identifier 422: An identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​that indicates the user of the client device 102 on which the message 400 was generated and from which the message 400 was sent.

[0054] • Message recipient identifier 424: An identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​that indicates the user of the client device 102 to which the message 400 is addressed.

[0055] The content (e.g., value) of the various components of message 400 may be pointers to locations in a table in which the content data values ​​are stored. For example, the image value in message image payload 406 may be a pointer to a location (or address) in image table 308. Similarly, the value in message video payload 408 may point to data stored in video table 310, the value stored in message annotation 412 may point to data stored in annotation table 312, the value stored in message story identifier 418 may point to data stored in story table 306, and the values ​​stored in message sender identifier 422 and message recipient identifier 424 may point to user records stored in entity table 302.

[0056] Figure 5 is a schematic diagram illustrating an access restriction process 500 according to some example embodiments, according to which access to content (e.g., a multimedia payload of an ephemeral message 502 and associated data) or a collection of content (e.g., an ephemeral message story 504) can be time-restricted (e.g., made ephemeral).

[0057] The ephemeral message 502 is shown to be associated with a message duration parameter 506, the value of which determines the amount of time that the messaging client application 104 will display the ephemeral message 502 to the receiving user of the ephemeral message 502. In one embodiment, where the messaging client application 104 is an application client, the receiving user may view the ephemeral message 502 for up to 10 seconds, depending on the amount of time specified by the sending user using the message duration parameter 506.

[0058] The message duration parameter 506 and the message recipient identifier 424 are shown as inputs to a message timer 512, which is responsible for determining the amount of time that the transient message 502 is shown to a particular receiving user identified by the message recipient identifier 424. In particular, the transient message 502 is shown to the associated receiving user only for the period of time determined by the value of the message duration parameter 506. The message timer 512 is shown as providing an output to a more general transient timer system 202, which is responsible for the overall timing of displaying content (e.g., the transient message 502) to the receiving user.

[0059] Figure 5502 is included in a transient message story 504 (e.g., a personal story or an event story). The transient message story 504 has an associated story duration parameter 508, the value of which determines the duration for which the transient message story 504 is presented and accessible to users of the messaging system 100. For example, the story duration parameter 508 may be the duration of a concert, wherein the transient message story 504 is a collection of content belonging to the concert. Alternatively, when performing the setup and creation of the transient message story 504, the user (the owning user or the curator user) may specify the value of the story duration parameter 508.

[0060] Additionally, each ephemeral message 502 within the ephemeral message story 504 has an associated story engagement parameter 510, the value of which determines the duration for which the ephemeral message 502 will be accessible within the context of the ephemeral message story 504. Thus, a particular ephemeral message 502 may “expire” and become inaccessible within the context of the ephemeral message story 504 before the ephemeral message story 504 itself expires according to the story duration parameter 508.

[0061] The transient timer system 202 may also be operable to remove a particular ephemeral message 502 from the ephemeral message story 504 based on a determination that an associated story engagement parameter 510 has been exceeded. For example, when the sending user has established a story engagement parameter 510 of 24 hours from posting, the transient timer system 202 will remove the associated ephemeral message 502 from the ephemeral message story 504 after the designated 24 hours. The transient timer system 202 is also operable to remove ephemeral message stories 504 when the story engagement parameter 510 for each and every ephemeral message 502 within the ephemeral message story 504 has expired or when the ephemeral message story 504 itself has expired according to the story duration parameter 508.

[0062] In response to the transient timer system 202 determining that the transient message story 504 has expired (e.g., is no longer accessible), the transient timer system 202 communicates with the messaging system 100 (e.g., specifically, the messaging client application 104) so ​​that the indicia (e.g., icon) associated with the relevant transient message story 504 is no longer displayed within the user interface of the messaging client application 104.

[0063] The following is an example implementation of the mapping system 123 according to some example embodiments. First, the mapping system 123 performs semantic segmentation classifier training (e.g., image segmentation neural network) on frames from a video (such as frames from a video social media post or a video generated by a client device). For example, the mapping system 123 generates a programmatic label for each frame using a machine learning classifier that is trained to generate image segmentation labels for a ground-based image (e.g., image data generated from a ground-based camera device (such as a client device)). Second, the mapping system 123 places the semantically labeled image data into geographic space by converting the image frames into a point cloud using a photogrammetric computer vision solution such as AliceVision. One problem with conversion to a point cloud is the inaccuracy of geographic information contained in ground-based sources (e.g., side-view video, GPS data). In some example embodiments, to address the lack of accuracy, the mapping system 123 uses an external spatial reference dataset (e.g., aerial data) to assist in the geo-correction of the point cloud. In some example embodiments, the external reference data is provided in different forms such as a visual dataset generated from the air. That is, to address the problem of insufficient accuracy, the mapping system 123 implements an external spatial reference dataset to assist in the geo-correction of the point cloud generated on the ground. The external reference can take various forms, such as aerial LiDAR, point clouds derived from aerial or satellite-based tilt imaging techniques, high-resolution synthetic aperture radar (SAR), or other remote sensing sources. In some example embodiments, the mapping system 123 pre-processes the point cloud by dividing the video frames into segments (e.g., five-second segments) and constructing a 3D point cloud in a photogrammetric pipeline (e.g., implementing a photometric imaging scheme (SfM)). According to some example embodiments, the example photogrammetric pipeline includes (1) camera device startup, (2) followed by image feature extraction, (3) followed by image matching, (4) followed by future matching, and (5) followed by performing motion recovery structure (SfM). SfM involves estimating the 3D structure of a scene from a collection of two-dimensional images. The SfM data may be generated in different ways based on different factors, such as the number and type of cameras used, whether the images are ordered, and whether the images are taken from different cameras (eg, cameras of different user devices).

[0064] In some example embodiments, to co-register the point cloud to an external spatial reference, the mapping system 123 maximizes the number of possible point matches between the external reference data and the ground-based point cloud via densification. In some example embodiments, additional densification is performed because the two data sets are different data sets collected from different perspectives, such as orthogonal viewpoints (e.g., side view and top-down view). In some example embodiments, image registration (e.g., co-registration) between multiple images is an image processing technique for aligning multiple scenes into a single integrated image.

[0065] In some example embodiments, to address these difficulties, the mapping system 123 increases the number of potential points to match by identifying (e.g., interpolating) aerial data sources and ground data sources. In some example embodiments, for aerial data sources, the mapping system 123 densifies point data corresponding to facades (e.g., sides) of buildings to improve the data provided by the aerial, because due to the top-down perspective of the data collection device, vertical surfaces (e.g., walls, sides of buildings) typically accommodate few generated points in a given collection area. In some example embodiments, the mapping system 123 then semantically segments the LiDAR or 3D point cloud to determine building structure and ground structure. In some example embodiments, the mapping system 123 implements a machine learning neural network that is trained on the semantically segmented point cloud to perform densification (e.g., neural network-based point interpolation to densify and generate interpolated points).

[0066] In some example embodiments, the mapping system 123 then densifies the ground photogrammetrically derived point cloud using a ground densification pipeline that includes: (1) depth map mapping, (2) followed by depth map filtering, (3) followed by gridding, and (4) followed by grid filtering. The results of the pipeline generate an improved set of ground point cloud candidates that can be more easily co-registered to a reference external spatial dataset (e.g., an enhanced reference LiDAR point cloud from one or more aerial devices).

[0067] In some example embodiments, the co-registration of the two data sets includes first utilizing GPS data to derive telemetry for approximating the location of the point cloud, then performing an odometry-based alignment of the SfM point cloud, and then performing a pose graph optimization of the SfM point cloud to the external spatial reference data. In this manner, the mapping system 123 locks the SfM-derived point cloud to a robust spatial reference. In some example embodiments, the mapping system 123 then stitches each of the SfM point clouds together with one another to create a geographic location (such as Figure 8A seamless panoramic scroll of mixed 3D point clouds of a city (an urban area of ​​the city shown), where each ground point cloud is colored or shaded differently (e.g., one client device generates an image of the building from the right, another client generates an image of the building from the left, and so on).

[0068] One advantage of the segmentation and co-registration process is that each individual point cloud used to generate the 3D map is relatively small and therefore does not require a lot of computation to derive. In this way, when different point clouds are stitched together to generate a 3D map, the smaller SfM models ensure that errors do not propagate far (e.g., into neighboring point clouds that may spread errors). In this way, local errors in each ground point cloud are never correlated with its neighboring SfM models (e.g., ground point clouds derived from other client devices). In addition, in some example embodiments, image fragments (e.g., image segments of an image or video frame) from an image frame of a client device used to generate the ground point cloud are subsequently stitched (e.g., projected, applied as a surface texture) to the ground point cloud. Figure 8 3D maps, so that the 3D maps have a more realistic photographic appearance. In this way, the mapping system generates a high-accuracy point cloud from a commercial camera (e.g., a client device camera) via enhancements from an external spatial reference dataset.

[0069] Figure 6 An example flow chart of a method 600 for generating a 3D map using a mapping system 123 via registration of different sets of images generated from orthogonal perspectives (e.g., ground-based point clouds, aerial-based point clouds) is shown according to some example embodiments. At operation 605, the mapping system 123 identifies a ground point cloud data set. For example, multiple client user devices generate video data of different portions of a geographic location (e.g., an urban area of ​​a city), a point cloud is generated from the video data of each client device, and the multiple point clouds are identified by the mapping system 123 for further processing.

[0070] At operation 610, the mapping system 123 identifies the remote sensing data. At operation 615, the mapping system 123 enhances the ground point cloud data (eg, densifies the point cloud, neural network-based densification to add additional points between sparse points).

[0071] At operation 620, the mapping system 123 enhances a collection of remotely sensed data, such as aerial device data of a geographic location acquired from a top-down perspective. In some example embodiments, the mapping system 123 enhances the remotely sensed data via interpolation (e.g., densification) to densify elevations of vertical surfaces (e.g., exterior walls of a building) captured in the remotely sensed data.

[0072] At operation 625, the mapping system 123 generates a 3D map of the physical environment. In some example embodiments, the mapping system 123 generates the 3D map by co-registering the enhanced ground point cloud datasets to the enhanced remote sensing data. The co-registration of each enhanced ground point cloud dataset stitches the patches to each other and to the remote sensing data to create an accurate 3D map.

[0073] Figure 7 Examples of different data sets (such as different image data from orthogonal perspectives) are shown according to some example embodiments. In the example top-down perspective 700, the remote sensing data includes data generated from a top-down perspective (e.g., images, videos, ranging data, point clouds). Different remote devices can provide remote sensing data, such as aircraft or satellites that physically move over a geographic location and generate remote sensing data using LIDAR or an imaging device (e.g., a complementary metal oxide semiconductor (CMOS) camera). In the example side perspective 750, local data includes data generated by a ground device that generates data from a side perspective (e.g., images, videos, ranging data, point clouds). Different devices can provide local data, such as user devices (e.g., smart phones, cameras, car-based imaging systems, ranging systems such as lidar).

[0074] Figure 8 An example 3D map 800 of a geographic area (e.g., downtown Boulder, Colorado) generated by the mapping system 123 is shown in accordance with some example implementations. Figure 8 In the illustrated example of , the 3D map 800 is shaded with different patterns to indicate areas of the 3D map 800 corresponding to different point clouds generated from different ground devices (e.g., different end-user devices). The different point clouds depict buildings, streets, and other physical features of a geographic area (such as downtown Boulder, Colorado). For example, according to some example embodiments, a first user device (not depicted) generates video data while a user of the device is stationary or walking around a geographic area, and the video data is then used to create a first SfM-based point cloud region 805 via a photometric pipeline, where processing is implemented as described above (e.g., using enhancement and co-registration of remote sensing data).

[0075] In addition, a second user device (not depicted) generates video data while a second user of the second user device is stationary or walking around the geographic area, and the second video data set is then used to create a second SfM-based point cloud region 810 (e.g., which is processed and enhanced via remote sensing data as described above). In addition, a third user device (not depicted) generates a third video data set while a third user of the third user device is stationary or walking around the geographic area, and the third video data set is then used to create a third SfM-based point cloud region 815 (e.g., which is processed and enhanced via remote sensing data as described above).

[0076] In addition, a fourth user device (not depicted) generates a fourth video dataset while a fourth user of the fourth user device is stationary or walking around the geographic area, and the fourth video dataset is then used to create a fourth SfM-based point cloud region 820 (e.g., which is processed and enhanced via remote sensing data as described above). The resulting point clouds can then be stitched together to create a map 800 (e.g., a complete 3D panoramic map of the geographic area). In some example embodiments, the video dataset is a clip of a social media post (e.g., a short message), while in other example embodiments, the video dataset includes video data created from an off-the-shelf consumer imaging solution (such as a video recorder, a digital single-lens reflex camera (DSLR), or a mirrorless camera).

[0077] As described above, one benefit of independently derived point cloud regions 805, 810, 815, and 820 is that errors are localized in each point cloud region and do not spread to adjacent regions. This may be beneficial in situations where, for example, one of the point clouds is inaccurate (e.g., due to poor quality video or aerial data), while still being able to create a highly useful 3D map 800.

[0078] In some example embodiments, rather than displaying different patterns of different point clouds, 3D map 800 applies image fragments (eg, image segments) of video data as image textures to 3D map 800 so that 3D map 800 appears more photo-realistic.

[0079] Example 1. A method comprising: identifying ground source image data generated using multiple client devices; identifying aerial-based image data generated from an orthogonal perspective relative to the ground source image data; generating enhanced ground source image data by correlating points of the ground source image data and the aerial-based image data; generating a three-dimensional map from the enhanced ground source image data, the 3D map comprising enhanced stitched portions of ground source image data sets from different client devices among the multiple client devices.

[0080] Example 2. A method according to Example 1, wherein the ground source image data includes multiple point clouds.

[0081] Example 3. A method according to any of Examples 1 or 2, wherein the multiple point clouds are generated by applying an imaging scheme to video sequences generated by multiple client devices.

[0082] Example 4. A method according to any one of Examples 1 to 3, wherein the imaging scheme is a photometric imaging scheme that generates point cloud data from image data.

[0083] Example 5. A method according to any one of Examples 1 to 4, wherein the aerial-based image data is generated by an aircraft.

[0084] Example 6. A method according to any one of Examples 1 to 9, wherein the aerial-based image data includes aerial lidar data that images the ground from a top-down perspective.

[0085] Example 7. The method of any one of Examples 1 to 9 further includes enhancing the ground source image data by performing densification using interpolation to add image details.

[0086] Example 8. A method according to any one of Examples 1 to 9, wherein the machine learning scheme is trained to perform densification on ground source image data.

[0087] Example 9. The method of any one of Examples 1 to 9 further includes enhancing the ground source image data by performing densification using interpolation to add image details.

[0088] Example 10. A method according to any one of Examples 1 to 9, wherein correlating the points includes applying a point co-registration scheme to correlate the ground source image data and the aerial-based image data.

[0089] Example 11. A system comprising: one or more processors of a machine; and at least one memory storing instructions, which, when executed by the one or more processors, cause the machine to perform any method according to Example 10.

[0090] Example 12. A machine storage medium comprising instructions that, when executed by a machine, cause the machine to perform any of the methods of Examples 1 to 10.

[0091] Fig. 9 is a block diagram illustrating components of a machine 900 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methods discussed herein, according to some example embodiments. Specifically, Fig. 9A graphical representation of a machine 900 in the example form of a computer system is shown, within which instructions 916 (e.g., software, programs, applications, applet, app or other executable code) for causing the machine 900 to perform any one or more of the methods discussed herein can be executed. Similarly, instructions 916 can be used to implement the modules or components described herein. Instructions 916 convert a general unprogrammed machine 900 into a specific machine 900 that is programmed to perform the functions described and shown in the described manner. In alternative embodiments, the machine 900 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 900 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 900 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing, sequentially or otherwise, instructions 916 that specify actions to be taken by the machine 900. In addition, while only a single machine 900 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 916 to perform any one or more of the methodologies discussed herein.

[0092] The machine 900 may include a processor 910, a memory / storage device 930, and an input / output (I / O) component 950, which may be configured to communicate with each other, such as via a bus 902. The memory / storage device 930 may include a main memory 932, a static memory 934, and a storage unit 936, which may be accessed by the processor 910, such as via the bus 902. The storage unit 936 and the memory 932 store instructions 916 that embody any one or more of the methods or functions described herein. The instructions 916 may also reside, in whole or in part, within the memory 932, within the storage unit 936 (e.g., on a machine-readable medium 938), within at least one of the processors 910 (e.g., within a processor cache memory accessible to the processor 912 or 914), or within any suitable combination thereof during execution thereof by the machine 900. Therefore, the memory 932, the storage unit 936, and the memory of the processor 910 are examples of machine-readable media.

[0093] The I / O components 950 may include a variety of components that receive input, provide output, generate output, transmit information, exchange information, capture measurements, etc. The specific I / O components 950 included in a particular machine 900 will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It will be appreciated that the I / O components 950 may include Fig. 9 950 are grouped according to functions, and the grouping is by no means limiting. In various example embodiments, the I / O components 950 may include output components 952 and input components 954. The output components 952 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), auditory components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 954 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), pointing-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide the location and / or force of touch or touch gestures), audio input components (e.g., microphones), etc.

[0094] In other example embodiments, the I / O component 950 may include a biometric component 956, a motion component 958, an environmental component 960, or a positioning component 962, as well as various other components. For example, the biometric component 956 may include a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 958 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 960 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an auditory sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas sensor that detects concentrations of hazardous gases for safety reasons or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 962 may include a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure from which the altitude can be derived), an orientation sensor component (e.g., a magnetometer), etc.

[0095] A variety of technologies may be used to achieve communication. The I / O component 950 may include a communication component 964 that is operable to couple the machine 900 to a network 980 or a device 970 via a coupling 982 and a coupling 972, respectively. For example, the communication component 964 may include a network interface component or other suitable device to interface with the network 980. In other examples, the communication component 964 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Components (e.g., low power ), Device 970 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0096] In addition, the communication component 964 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 964 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following items: a one-dimensional barcode such as a universal product code (UPC) barcode; a multi-dimensional barcode such as a Quick Response (QR) code, an Aztec code, a data matrix, a data glyph, a MaxiCode, a PDF418, a super code, a UCC RSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone for an audio signal of an identification tag). In addition, various information can be obtained via the communication component 964, such as location via Internet Protocol (IP) geolocation, location via Signal triangulation to obtain location, location obtained via detection of NFC beacon signals that can indicate a specific location, etc.

[0097] A "carrier wave signal" in this context refers to any intangible medium capable of storing, encoding, or carrying instructions 916 for execution by the machine 900, and includes a digital or analog communications signal or other intangible medium to facilitate communication of such instructions 916. The instructions 916 may be transmitted or received over the network 980 using a transmission medium via a network interface device and using any of a number of well-known transmission protocols.

[0098] A "client device" in this context refers to any machine 900 that interfaces with the network 980 to obtain resources from one or more server systems or other client devices 102. The client device 102 may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a PDA, a smart phone, a tablet computer, an ultrabook, a netbook, a multiprocessor system, a microprocessor-based or programmable consumer electronics system, a game console, a STB, or any other communication device that a user can use to access the network 980.

[0099] A "communication network" in this context refers to one or more parts of a network 980, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, 980 may include a wireless network or a cellular network, and the coupling 982 may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate for GSM evolution (EDGE) technology, including 3G third generation partnership project (3GPP), fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other standards defined by various standard setting organizations, other remote protocols, or other data transmission technologies.

[0100] A "transient message" in this context refers to a message 400 that can be accessed for a limited duration of time. A transient message 502 can be text, an image, a video, etc. The access time for a transient message 502 can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message 400 is transient.

[0101] "Machine-readable medium" in this context refers to a component, device, or other tangible medium capable of temporarily or permanently storing instructions 916 and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database or associated cache memory and server) capable of storing instructions 916. The term "machine-readable medium" may also be taken to include any medium or combination of multiple media capable of storing instructions 916 (e.g., code) executed by the machine 900, such that the instructions 916, when executed by one or more processors 910 of the machine 900, cause the machine 900 to perform any one or more of the methods described herein. Thus, "machine-readable medium" refers to a single storage device or device as well as a "cloud-based" storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" does not include the signal itself.

[0102] "Component" in this context refers to a logical or device, physical entity having boundaries defined by function or subroutine calls, branch points, APIs, or other techniques that provide partitioning or modularization of specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be encapsulated functional hardware units designed for use with other components, and are part of a program that typically performs specific functions of related functions. Components can constitute software components (e.g., code included on a machine-readable medium) or hardware components.

[0103] "Hardware components" are tangible units that can perform certain operations and can be configured or arranged in a physical manner. In various example embodiments, one or more computer systems (e.g., independent computer systems, client computer systems, or server computer systems) or one or more hardware components of a computer system (e.g., processor 912 or a group of processors 910) can be configured to operate to perform certain operations described herein by software (e.g., application or application part). Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit or logic that is permanently configured to perform certain operations. A hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). A hardware component may also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of the machine 900) that is uniquely customized to perform the configured function, rather than a general-purpose processor 910.

[0104] It should be understood that cost and time considerations may determine whether a hardware component is mechanically implemented in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, i.e., an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or perform certain operations described herein.

[0105] Considering embodiments where hardware components are temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each of the hardware components at any one time in a timely manner. For example, where the hardware components include a general-purpose processor 912 that is configured by software to be a special-purpose processor, the general-purpose processor 912 can be configured as respectively different special-purpose processors (e.g., including different hardware components) at different times. The software configures a specific processor 912 or processor 910 accordingly to, for example, constitute a specific hardware component at one time and constitute different hardware components at different times.

[0106] Hardware components can provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components can be considered to be coupled in communication. In the case of multiple hardware components simultaneously, communication can be realized by signal transmission between or among two or more hardware components in the hardware components (for example, by appropriate circuits and buses). In an embodiment in which multiple hardware components are configured or instantiated at different times, communication between or among such hardware components can be realized, for example, by storing information in a memory structure accessed by multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled in communication with it. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (for example, a collection of information).

[0107] The various operations of the example methods described herein may be performed at least in part by one or more processors 910 that are temporarily configured or permanently configured to perform related operations (e.g., by software). Whether temporarily or permanently configured, such a processor 910 may constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors 910. Similarly, the methods described herein may be implemented at least in part by a processor, wherein a specific processor 912 or processor 910 is an example of hardware. For example, at least some of the operations of the method may be performed by one or more processors 910 or a processor-implemented component. In addition, one or more processors 910 may also be operated to support the execution of related operations in a "cloud computing" environment or to operate as "software as a service" (SaaS). For example, at least some of the operations in the operation may be performed by a group of computers (as an example of a machine 900 including a processor 910), wherein these operations can be accessed via a network 980 (e.g., the Internet) and via one or more appropriate interfaces (e.g., API). The execution of certain operations may be distributed among the processors 910, which may reside not only within a single machine 900, but may also be deployed on multiple machines 900. In some example embodiments, the processor 910 or a processor-implemented component may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, the processor 910 or a processor-implemented component may be distributed across multiple geographic locations.

[0108] A "processor" in this context refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor 912) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces corresponding output signals that are applied to operate the machine 900. The processor may be, for example, a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), or any combination thereof. The processor 910 may also be a multi-core processor 910 having two or more independent processors 912, 99 (sometimes referred to as "cores") that can execute instructions 916 simultaneously.

[0109] A "timestamp" in this context refers to a sequence of characters or coded information that identifies when some event occurred, for example, thereby giving a date and time of day, sometimes accurate to a fraction of a second.

Claims

1. A method comprising: identifying ground source image data generated using a plurality of client devices; identifying aerial-based image data generated from an orthogonal perspective relative to the ground-based source image data; generating enhanced ground-source image data by point correlating the ground-source image data and the air-based image data; A three-dimensional map is generated from the augmented ground source image data, the three-dimensional map comprising augmented stitched portions of ground source image data sets from different ones of the plurality of client devices.

2. The method according to claim 1, wherein: The ground source image data includes a plurality of point clouds.

3. The method according to claim 1, wherein: The plurality of point clouds are generated by applying an imaging scheme to video sequences generated by the plurality of client devices.

4. The method according to claim 4, wherein: The imaging scheme is a photometric imaging scheme that generates point cloud data from image data.

5. The method according to claim 1, wherein: The air-based image data is generated by an aircraft.

6. The method according to claim 5, wherein: The aerial-based image data includes aerial lidar data that images the ground from a top-down perspective.

7. The method according to claim 1, further comprising: The ground source image data is enhanced by performing densification using interpolation to add image detail.

8. The method according to claim 7, wherein: A machine learning scheme is trained to perform densification on the ground source image data.

9. The method according to claim 1, further comprising: The ground source image data is enhanced by performing densification using interpolation to add image detail.

10. The method according to claim 1, wherein: The correlating the points includes applying a point co-registration scheme to correlate the ground-source image data and the air-based image data.

11. A system comprising: one or more processors of a machine; as well as at least one memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising: identifying ground source image data generated using a plurality of client devices; identifying aerial-based image data generated from an orthogonal perspective relative to the ground-based source image data; generating enhanced ground-source image data by point correlating the ground-source image data and the air-based image data; A three-dimensional map is generated from the augmented ground source image data, the three-dimensional map comprising augmented stitched portions of ground source image data sets from different ones of the plurality of client devices.

12. The system according to claim 11, wherein: The ground source image data includes a plurality of point clouds.

13. The system according to claim 11, wherein: The plurality of point clouds are generated by applying an imaging scheme to video sequences generated by the plurality of client devices.

14. The system according to claim 13, wherein: The imaging scheme is a photometric imaging scheme that generates point cloud data from image data.

15. The system according to claim 11, wherein: The air-based image data is generated by an aircraft.

16. The system of claim 15, wherein: The aerial-based image data includes aerial lidar data that images the ground from a top-down perspective.

17. The system of claim 11, further comprising: The ground source image data is enhanced by performing densification using interpolation to add image detail.

18. The system of claim 17, wherein: A machine learning scheme is trained to perform densification on the ground source image data.

19. The system of claim 11, further comprising: The ground source image data is enhanced by performing densification using interpolation to add image detail.

20. A machine storage medium comprising instructions that, when executed by a machine, cause the machine to perform operations comprising: identifying ground source image data generated using a plurality of client devices; identifying aerial-based image data generated from an orthogonal perspective relative to the ground-based source image data; generating enhanced ground-source image data by point correlating the ground-source image data and the air-based image data; A three-dimensional map is generated from the augmented ground source image data, the three-dimensional map comprising augmented stitched portions of ground source image data sets from different ones of the plurality of client devices.

Images