System and method for generating annotations of structured static objects in aerial imagery using geometric transfer learning and probabilistic localization

Abstract

In some embodiments, aerial images of a geographic area are captured by an autonomous vehicle. In some embodiments, the locations of structures within a subset of the aerial images are manually annotated, and the manually annotated geographic locations are determined based on the pose information of the camera. In some embodiments, the manually annotated aerial images are used to train a machine learning model. The machine learning model is used to automatically generate annotations for other images of the geographic area, and the geographic locations determined from the manual annotations are used to determine the probability that the automatic annotations are accurate. Automatic annotations determined to be accurate can be used to retrain the machine learning model to improve its precision and recall.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on U.S. application No. 16 / 669,309 filed on October 30, 2019, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates generally to computer vision and, in particular, but not exclusively, to detecting objects in aerial imagery. Background Art

[0004] Structured objects are objects with a specific structure (geometric or semantic). Examples of structured objects in outdoor environments include, but are not limited to, utility poles, power lines, swimming pools, and traffic lights. Detecting such objects in aerial images is an important step in various applications such as mapping, image analysis, image stitching, and route planning. Summary of the invention

[0005] In some embodiments, a non-transitory computer-readable storage medium is provided. The storage medium has instructions stored thereon, the instructions, in response to being executed by one or more processors of a computing system, causing the computing system to perform actions, the actions comprising: determining a geographic location of one or more structures in a geographic area; obtaining an aerial image of the geographic area; and annotating the aerial image based at least on the determined geographic location of the one or more structures in the geographic area.

[0006] In some embodiments, a computer-implemented method for annotating aerial images is provided. An annotation of one or more structures within an aerial image in a set of aerial images of a geographic area is determined by a computing device. The computing device determines a geographic location associated with the annotation. The computing device updates a probabilistic locator model based on the determined geographic location. The computing device uses the probabilistic locator model to determine an annotation for a new aerial image of the geographic area. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Non-limiting and non-exhaustive embodiments of the present invention are described with reference to the following drawings, wherein like reference numerals refer to like parts in the various views unless otherwise specified. Not all instances of an element must be labeled so as not to clutter the drawings where appropriate. The drawings are not necessarily drawn to scale, with emphasis instead placed on illustrating the principles described.

[0008] Figure 1 is a schematic diagram illustrating a non-limiting example embodiment of an autonomous vehicle capturing aerial imagery in accordance with various aspects of the present disclosure.

[0009] Figure 2 is a block diagram illustrating components of a non-limiting example embodiment of an autonomous vehicle 200 in accordance with various aspects of the present disclosure.

[0010] Figure 3 is a block diagram illustrating a non-limiting example embodiment of an image processing system 300 in accordance with various aspects of the present disclosure.

[0011] Figure 4-Figure 6 is a flow chart illustrating a non-limiting example embodiment of a method of detecting structures in aerial imagery according to various aspects of the present disclosure.

[0012] Figure 7 Non-limiting example embodiments of aerial images and annotated aerial images in accordance with various aspects of the present disclosure are shown.

[0013] Figure 8 is a side view schematic diagram of using pose information and a terrain model to determine a geographic location of a structure according to various aspects of the present disclosure.

[0014] Fig. 9 is a schematic diagram illustrating an aerial image 900 that has been overlaid with a probabilistic locator grid 902 that divides the aerial image 900 into a plurality of segments in accordance with various aspects of the present disclosure.

[0015] Fig.10 It is shown Fig. 9 Schematic diagram of the probabilistic locator model without the details of the aerial imagery for clarity.

[0016] Fig.11 is a side view of a non-limiting example embodiment of a three-dimensional probabilistic locator model 1100 in accordance with various aspects of the present disclosure.

[0017] Fig.12 A non-limiting example embodiment of a set of aerial images of a single geographic area is shown in accordance with various aspects of the present disclosure.

[0018] Fig.13 is a block diagram illustrating a non-limiting example embodiment of a computing device suitable for use with embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] State-of-the-art algorithms for detecting objects in images typically involve machine learning (ML) models trained using supervised learning. Supervised learning typically uses a large number of manually annotated examples to learn. For the object detection task, tens of thousands of images that have been manually annotated (i.e., labeled by humans) are typically employed. The task of performing such annotation is extremely labor-intensive and therefore has considerable disadvantages:

[0020] 1. Manual annotation is very time-consuming, often taking weeks to generate a dataset that can be used for supervised learning of object detection. This can result in ML models that are slow to adapt to more recent data.

[0021] 2. Manual annotation of tens of thousands of images can be expensive.

[0022] 3. Large-scale manual annotation is prone to errors, resulting in both false positive and false negative annotations.

[0023] 4. Although some images may be suitable for use as training data, they may include sensitive information and therefore may not be allowed to be shared with (or by) annotators.

[0024] These shortcomings often result in low-quality or low-quantity datasets, both of which negatively impact the performance of ML models for object detection.

[0025] In embodiments of the present disclosure, these problems (and more) are addressed to generate high-quality training data to improve the precision and recall of static object detection in aerial imagery datasets. The generation of high-quality training data allows for the training of high-quality ML models, thereby improving the ability to automatically detect structured objects in aerial imagery. In general, embodiments of the present disclosure derive annotated geographic locations of structured objects in a first set of aerial images of a given geographic area for training an ML model, and these derived geographic locations can then be used to transfer knowledge gained from the first set of aerial images to subsequent sets of aerial images of the given geographic area.

[0026] Figure 1 is a schematic diagram of a non-limiting example embodiment of an autonomous vehicle for capturing aerial images according to various aspects of the present disclosure. As shown, the autonomous vehicle 200 flies in a region having a field of view of a geographic area. The camera 102 of the autonomous vehicle 200 is pointed at the geographic area and captures aerial images of the geographic area. In some embodiments, the camera 102 can be mounted to the autonomous vehicle 200 using a pivot mechanism 104. In operation, the pivot mechanism 104 can adjust the angle of the camera 102 based on input from an inertial measurement unit (IMU) of the autonomous vehicle 200 (e.g., an accelerometer-based IMU that determines the absolute orientation of the autonomous vehicle 200 relative to the direction of gravity). In response to input from the IMU, the pivot mechanism 104 can orient the camera 102 perpendicular to the ground to provide a common reference angle for aerial images. In other embodiments, the pivot mechanism 104 can orient the camera 102 at other angles.

[0027] A combination of the angle of the pivot mechanism 104, the absolute orientation of the autonomous vehicle 200, the position of the autonomous vehicle 200 obtained from a global navigation satellite system (GNSS) sensor, and / or other pose information for the camera 102 can be used to determine the geographic location of pixels in the aerial image. For example, in some embodiments, a set of aerial images including GNSS or other positioning information of the camera 102 for each aerial image can be used to generate a terrain model and / or can be used to obtain accurate pose information for each aerial image using techniques including, but not limited to, structure from motion (SFM) techniques.

[0028] As discussed herein, these geolocations can be used to transfer knowledge from a small manually annotated set of aerial images to a larger set of aerial images of the same geographic area. In some embodiments, precise pose information and terrain models of the aerial images can be used to find a rough geolocation for each annotation, and information from multiple annotations can be aggregated together in a probabilistic locator to remove false positives and increase confidence in true positives. This allows for the automatic generation of larger training sets of aerial images directly from the geolocations associated with high confidence annotations, thereby improving the quality of machine learning models trained using the data.

[0029] Figure 2 is a block diagram illustrating components of a non-limiting example embodiment of an autonomous vehicle 200 according to various aspects of the present disclosure. The autonomous vehicle 200 is configured to collect aerial images together with pose information for each collected aerial image. As described below, the autonomous vehicle 200 is also configured to transmit the collected aerial images and pose information to the image processing system 300. In some embodiments, the autonomous vehicle 200 is an aircraft, including but not limited to a rotorcraft. In other embodiments, any other type of autonomous vehicle 200 capable of collecting aerial images may be used, including but not limited to spacecraft and balloons. In some embodiments, the aerial images may be collected by a human-piloted vehicle rather than the autonomous vehicle 200.

[0030] As shown, autonomous vehicle 200 includes a communication interface 202 , one or more vehicle state sensor devices 204 , a power source 206 , one or more processors 208 , one or more propulsion devices 210 , and a computer-readable medium 212 .

[0031] In some embodiments, the communication interface 202 includes hardware and software that enables any suitable communication technology for communicating with the image processing system 300. In some embodiments, the communication interface 202 includes multiple communication interfaces, each of which is used for appropriate circumstances. For example, the communication interface 202 may include a long-range wireless interface, such as a 4G or LTE interface, or any other type of long-range wireless interface (e.g., 2G, 3G, 5G or WiMAX) for communicating with the image processing system 300 when traversing the route. The communication interface 202 may also include a medium-range wireless interface such as a Wi-Fi interface to be used when the autonomous vehicle 200 is near a starting location, an end point, or an area where Wi-Fi coverage is available. The communication interface 202 may also include a short-range wireless interface such as a Bluetooth interface to be used when the autonomous vehicle 200 is in a maintenance position or stationary and waiting for a route to be assigned. The communication interface 202 may also include a wired interface such as an Ethernet interface or a USB interface, which may also be used when the autonomous vehicle 200 is in a maintenance position or stationary and waiting for a route to be assigned.

[0032] In some embodiments, the vehicle state sensor device 204 is configured to detect the states of various components of the autonomous vehicle 200 and transmit signals representing these states to other components of the autonomous vehicle 200. Some non-limiting examples of the vehicle state sensor device 204 include a battery state sensor and a propulsion device health sensor. Some other non-limiting examples of the vehicle state sensor device 204 include sensors that allow the position of the camera 102 to be determined, including but not limited to sensors that report the position of the pivot mechanism 104, IMU sensors that report the orientation of the autonomous vehicle 200 relative to the gravity vector, and GNSS sensors that report the geographic location of the autonomous vehicle 200.

[0033] In some embodiments, the power source 206 may be any suitable device or system for storing and / or generating electricity. Some non-limiting examples of the power source 206 include one or more batteries, one or more solar panels, fuel tanks, and combinations thereof. In some embodiments, the propulsion device 210 may include any suitable device for causing the autonomous vehicle 200 to travel along a path. For an aircraft, the propulsion device 210 may include devices such as, but not limited to, one or more motors, one or more propellers, and one or more flight control surfaces. For a spacecraft, the propulsion device 210 may include devices such as, but not limited to, maneuvering thrusters. In some embodiments, the processor 208 may include any type of computer processor capable of receiving signals from other components of the autonomous vehicle 200 and executing instructions stored on the computer-readable medium 212. In some embodiments, the computer-readable medium 212 may include one or more devices capable of storing information for access by the processor 208. In some embodiments, the computer-readable medium 212 may include one or more of a hard drive, a flash drive, an EEPROM, and combinations thereof. In some embodiments, one or more of the processor 208 and the computer-readable medium 212 may be included in an ASIC, an FPGA, or other custom computing device.

[0034] As shown, the computer-readable medium 212 has stored thereon an image data store 214, an image collection engine 216, and a pose determination engine 218. In some embodiments, the image collection engine 216 is configured to cause the camera 102 to capture an aerial image and store the aerial image in the image data store 214. In some embodiments, the pose determination engine 218 is configured to collect information from the vehicle state sensor device 204 that can be used to determine the pose of the camera 102, and store the information together with the aerial image in the image data store 214. The information used to determine the pose of the camera 102 may include, but is not limited to, GNSS information, altimeter information, and a set of aerial images that can be used to determine the pose via visual SFM techniques. In some embodiments, the pose determination engine 218 may be configured to determine the pose itself (e.g., the position of the camera 102 in space, and the orientation of the camera 102), and the pose may be stored in the image data store 214.

[0035] "Engine" refers to a program that can be used in a programming language (such as C, C++, COBOL, JAVA TM , PHP, Perl, HTML, CSS, JavaScript, VBScript, ASPX, Microsoft.NET TM, Go, etc.) or hardware-embodied logic. The engine can be compiled into an executable program or written in an interpreted programming language. The software engine may be callable from other engines or themselves. Generally, the engine described herein refers to a logical module that can be merged with other engines or divided into sub-engines. The engine can be stored in any type of computer-readable medium or computer storage device, and can be stored and executed by one or more general-purpose computers, thereby creating a special-purpose computer configured to provide the engine or its functions.

[0036] "Data storage" refers to any suitable device configured to store data for access by a computing device. One example of a data storage is a highly reliable, high-speed relational database management system (DBMS) that executes on one or more computing devices and can be accessed via a high-speed network. Another example of a data storage is a key-value store. However, any other suitable storage technology and / or device that can quickly and reliably provide stored data in response to queries can be used, and the computing device can be accessible locally rather than via a network, or can be provided as a cloud-based service. Data storage can also include data stored in an organized manner on a computer-readable storage medium (such as a hard drive, flash memory, RAM, ROM, or any other type of computer-readable storage medium). Those of ordinary skill in the art will recognize that the separate data stores described herein may be combined into a single data store, and / or the single data store described herein may be separated into multiple data stores without departing from the scope of the present disclosure.

[0037] Figure 3 is a block diagram illustrating a non-limiting example embodiment of an image processing system 300 according to various aspects of the present disclosure. In general, the image processing system 300 is configured to receive aerial images and process them to detect structures. Starting with a relatively small set of manually annotated images, the image processing system 300 can build a large training data set by determining a probabilistic representation of the geographic location of the annotated structures and then annotating additional aerial images with the high probability geographic location. A machine learning model trained using such a technique can then provide highly accurate model-generated annotations of other geographic areas.

[0038] The image processing system 300 is shown as including a single computing device 302. In some embodiments, the functionality of the image processing system 300 can be provided by a single computing device 302. In other embodiments, the functionality of the image processing system 300 can be provided by more than one computing device 302, which are connected via a network and work together to provide the components shown. In some embodiments, the computing device 302 can be a server computing device, including but not limited to a rack-mounted computing device or a computing device of a cloud computing service. In some embodiments, the computing device 302 can be another form factor, including but not limited to a desktop computing device, a laptop computing device, or a mobile computing device. In some embodiments, at least some of the processing described as being performed by the image processing system 300 can be performed by the autonomous vehicle 200 instead.

[0039] As shown, computing device 302 includes communication interface 306, one or more processors 308, and computer-readable medium 304. In some embodiments, the communication interface 306 referred to by "communication interface" can implement any suitable communication technology, including but not limited to wired communication technology (including but not limited to Ethernet, USB, and FireWire), wireless communication technology (including but not limited to Wi-Fi, WiMAX, 2G, 3G, 4G, 5G, LTE, and Bluetooth), and combinations thereof. "Computer-readable medium" refers to a removable or non-removable device that implements any technology capable of storing information in a volatile or non-volatile manner to be read by a processor of a computing device, including but not limited to: hard disk drives; flash memory; solid-state drives; random access memory (RAM); read-only memory (ROM); CD-ROM, DVD, or other disk storage; magnetic tape cassettes; magnetic tape; and magnetic disk storage.

[0040] As shown, a manual annotation engine 310 , an image collection engine 312 , an automatic annotation engine 314 , a model training engine 316 , a geographic location determination engine 318 , and a model data storage 320 are stored on the computer-readable medium 304 .

[0041] In some embodiments, the manual annotation engine 310 is configured to collect manual annotations of aerial images from human operators and store the manual annotations with the aerial images. In some embodiments, the image collection engine 312 is configured to receive aerial images from the autonomous vehicle 200 (or from more than one autonomous vehicle 200) and provide the aerial images to other components for further processing. In some embodiments, the automatic annotation engine 314 is configured to automatically annotate aerial images using a machine learning model stored in the model data store 320. In some embodiments, the model training engine 316 is configured to train a machine learning model and retrain an existing machine learning model using the training data stored in the model data store 320. In some embodiments, the geographic location determination engine 318 is configured to determine the geographic location of annotations associated with the aerial image. In some embodiments, the model data store 320 is configured to store aerial images, annotations, pose information, and machine learning models for use by components of the image processing system 300.

[0042] Further description of the actions performed by each of these elements is provided below.

[0043] Figure 4-Figure 6 4 is a flow chart illustrating a non-limiting example embodiment of a method 400 for detecting structures in aerial imagery according to various aspects of the present disclosure. The method 400 can be used to detect structures of interest, including but not limited to utility poles, light poles, or other static structures that a machine learning model is trained to detect, as described below.

[0044] The method 400 proceeds from a start block to block 402, where the camera 102 of the autonomous vehicle 200 captures a set of aerial images of a geographic area, and stores the aerial images and pose information for each aerial image in the image data store 214. In some embodiments, the autonomous vehicle 200 traverses a route that passes within visual range of the geographic area to be imaged, and the image collection engine 216 causes the camera 102 to periodically capture aerial images. In some embodiments, the image collection engine 216 can collect aerial images covering the entire desired geographic area while the autonomous vehicle 200 is in motion. In some embodiments, the image collection engine 216 can use the pivot mechanism 104 to aim the camera 102 at the desired geographic area.

[0045] In some embodiments, the image collection engine 216 may collect pose information from the vehicle state sensor device 204 to accompany each aerial image, and may store the aerial image and the pose information in the image data store 214. In some embodiments, the pose determination engine 218 may be used to collect pose information from the vehicle state sensor device 204. In some embodiments, the pose information may be collected and stored as raw information (e.g., the angle of the pivot mechanism 104, the orientation of the autonomous vehicle 200, the position of the autonomous vehicle 200), while in some embodiments, the pose information may be collected and then transformed by the pose determination engine 218 to reflect the actual pose of the camera 102 (e.g., the position and orientation of the camera in three-dimensional space). In some embodiments, the pose information may include absolute position information, such as GNSS information, including but not limited to latitude, longitude, and altitude.

[0046] At block 404, the autonomous vehicle 200 transmits the aerial image set and the pose information to the image processing system 300. In some embodiments, if the communication interface 202 includes long-range wireless communication technology, the autonomous vehicle 200 may transmit the data to the image processing system 300 while traversing the route and capturing the aerial images. In some embodiments, if the communication interface 202 includes only short-range wireless communication technology or wired communication technology, the transmission of the data may occur after the autonomous vehicle 200 returns to the base.

[0047] At block 406, the image collection engine 216 of the image processing system 300 stores the aerial image set and the pose information in the model data store 320. In some embodiments, the aerial image set and the pose information may be stored in a data store (such as a training data store) separate from the data store used to store the machine learning model. However, for the sake of brevity, a single data store is shown and described herein.

[0048] The method 400 then proceeds to the for loop defined between the for loop start box 408 and the for loop end box 418, where each aerial image in the aerial image set is processed. In some embodiments, less than all of the aerial images in the aerial image set collected by the image collection engine 216 may be processed in the for loop. Due to the manual work in the for loop, the number of aerial images processed in the for loop may be small. One of the benefits of the method 400 is that even processing a small number of aerial images allows the benefits of such processing to be automatically extended to a larger set of aerial images. The description of the for loop assumes that each aerial image includes at least one structure to be annotated. If there is no structure to be annotated in a given aerial image, the aerial image can be skipped.

[0049] Method 400 proceeds from for loop start block 408 to block 410, where the manual annotation engine 310 of the image processing system 300 creates manual annotations of structures of interest in the aerial image. In some embodiments, the aerial image may depict multiple structures (such as buildings, trees, bridges, roads, etc.), but the manual annotations may be limited to the types of structures of interest (such as utility poles, etc.). In some embodiments, the manual annotation engine 310 causes an annotation user interface to be presented to the user, either on a display device of the image processing system 300, on a display device of a client computing device on a network (such as by generating a web interface or providing an application programming interface (API) accessed by an application executed on the client computing device), or using any other suitable technology. The annotation user interface presents the aerial image to the user and allows the user to add annotations to the structures that the user sees in the aerial image. In some embodiments, each annotation may indicate the corner of the associated structure. In some embodiments, particularly for structures that are tall and thin and therefore may have an obscure position depending on the viewing angle, the annotation may indicate the base of the structure.

[0050] Figure 7 A non-limiting example embodiment of an aerial image and a manually annotated aerial image according to various aspects of the present disclosure is shown. Aerial image 702 shows several houses, a street, and two utility poles. The utility poles are objects of interest to be annotated within the aerial image. The tops of the utility poles can be seen, as can the power lines running between the utility poles, and the shadows cast by the utility poles. After processing as discussed in block 410, two manual annotations are shown in the annotated aerial image 704: a first annotation 706 and a second annotation 708. The first annotation 706 and the second annotation 708 each surround the location of the base of the utility pole. A user can add annotations by clicking on the corners of the annotation box, by dragging the annotation box to the appropriate location in the aerial image 702, or using any other suitable technique.

[0051] return Figure 4 At block 412, the manual annotation engine 310 stores the manual annotations in association with the aerial image in a training data set in the model data store 320. Again, the training data set may be stored in a data store separate from the model data store 320, but for simplicity, a single data store is shown and described herein. In some embodiments, the association between the manual annotations and the aerial image may be created by storing a record that includes a reference to the aerial image and each annotation.

[0052] At block 414, the geographic location determination engine 318 of the image processing system 300 determines the geographic location of the structure of interest in the aerial image based on the manual annotation and the pose information. In some embodiments, the geographic location may include the latitude and longitude values ​​of the structure. In some embodiments, the geographic location may also include an altitude value.

[0053] As described above, the pose information includes information indicating the pose of the camera 102 or that can be used to derive the pose of the camera 102. The pose information may include, but is not limited to, the position of the camera 102 in space (including, but not limited to, latitude, longitude, and altitude), and the orientation of the camera 102 in space (including, but not limited to, the pitch of the camera 102 relative to a horizontal or vertical plane, the yaw of the camera 102 relative to a reference direction (such as true north), and the roll of the camera 102 relative to a horizontal plane). The pose information may also include the zoom level of the camera 102. In some embodiments, the pose information may include only the position of the camera 102 in space.

[0054] In some embodiments, pose information may be included in multiple aerial images of a geographic area, such that by acquiring multiple overlapping aerial images including the geographic area and associated GNSS position data of camera 102 associated with each overlapping aerial image, a precise pose of each image may be estimated using techniques such as visual structure from motion (VisualSFM). In some embodiments, such a precise pose will include a position estimate with sub-meter error and an orientation estimate with less than 1 degree error.

[0055] In some embodiments, the precise pose derived from the visual SFM technique can be used with overlapping aerial images of the geographic area to generate a 3D model of the terrain of the geographic area. Any suitable technique can be used to generate the terrain model, including but not limited to patch-based multi-view stereo techniques or planar scanning stereo techniques. In some embodiments, another source of terrain model information, such as a GIS system, can be used. Once the pose of camera 102 and the terrain model are determined, ray tracing / ray casting or similar techniques can be used to determine the geographic locations annotated within the terrain model.

[0056] Figure 8 8 is a schematic side view of determining a geographic location of a structure using pose information and a terrain model according to various aspects of the present disclosure. As shown, a terrain model 808 representing the outline of a geographic area has been determined. Figure 8 The illustration in is a two-dimensional cross-section, but it will be understood that in practice the terrain model 808 may be three-dimensional.

[0057] As shown, to determine the location of structure 802 within terrain model 808, ray 810 is cast from the precise location of camera 102 in autonomous vehicle 200 along a vector extending from the camera optical center through the center pixel of the annotated location of structure 812 associated with the image of structure 806. A terrain model intersection point 814 is then found based on where ray 810 intersects terrain model 808. The geographic location associated with terrain model intersection point 814 (which is part of terrain model 808) can then be used as the geographic location of structure 802.

[0058] return Figure 4 At block 416, the geographic location determination engine 318 stores the geographic location of the structure of interest in the model data store 320 and updates the probabilistic locator model stored in the model data store 320. In some embodiments, the probabilistic locator model is configured to represent an aggregate probability that a given geographic location includes a structure. In some embodiments, the probabilistic locator model divides the geographic area into segments (such as squares or grids for a two-dimensional geographic area, or voxels for a three-dimensional geographic area).

[0059] Fig. 9 An aerial image 900 is shown having superimposed thereon a probabilistic locator grid 902 that divides the aerial image 900 into a plurality of segments. The segments include a first grid square 904 including a first structure and a second grid square 906 including a second structure. Fig.10 is a schematic diagram showing a probabilistic locator model 1000 without the details of an aerial image for clarity. The numbers within the grid squares of the probabilistic locator model 1000 indicate the probability that the structure actually exists at the geographic location associated with the grid square, based on the annotation of the aerial image. Grid squares with a probability of zero are left blank for clarity.

[0060] As shown, the first grid square 904 and the second grid square 906 have a high probability of structure existence due to the fact that the locations of the annotations of the first structure and the second structure will appear in those grid squares. Since the first structure and the second structure are close to the edges of the grid square, some adjacent squares, such as the third grid square 1002, the fourth grid square 1004, and the fifth grid square 1006 adjacent to the first grid square 904, and such as the sixth grid square 1008, the seventh grid square 1010, and the eighth grid square 1012 adjacent to the second grid square 906 exhibit a non-zero probability of structure existence. This may be due to inaccuracies in the determination of the geographic location of the annotations, inaccuracies in the pose estimation, or due to any other reasons. However, it will be noted that by updating the probabilistic locator model 1000 with annotations from multiple other aerial images, the probabilities associated with the first grid square 904 and the second grid square 906 can be increased more than the grid squares associated with inaccurate data. Each new annotation can be used to update the probabilities included in the probabilistic locator model 1000 using any suitable technique (including but not limited to Bayesian inference).

[0061] Just for clarity, Fig.10 The probabilistic locator model 1000 shown in FIG. 1 is two-dimensional. In some embodiments, a three-dimensional probabilistic locator model may be used. Fig.11 1 is a side view of a non-limiting example embodiment of a three-dimensional probabilistic locator model 1100 according to various aspects of the present disclosure. As shown, the three-dimensional probabilistic locator grid 1102 divides the geographic area into a set of voxels. Instead of finding the voxels where the ray 810 intersects the terrain model 808, Fig.10 Grid square shown. In the illustration, this is the first voxel 1104. Thus, when the annotation associated with image location 812 is processed, Bayesian inference (or another suitable technique) can be used to update the probability associated with the first voxel 1104 of the probabilistic locator model 1100.

[0062] return Figure 4 , the method 400 then proceeds to the for loop end block 418. If there are still more aerial images to be processed, the method 400 returns to the for loop start block 408 to process the next aerial image. Otherwise, the method 400 proceeds to the continuation terminal ("Terminal A").

[0063] Method 400 starts from endpoint A( Figure 5) proceeds to box 502, where the model training engine 316 of the image processing system 300 trains a bootstrap machine learning model using the training data set constructed within the for loop 408-418. The machine learning model is referred to as a "bootstrap" machine learning model because it is not expected to have particularly high performance, but is intended to "bootstrap" the rest of the process by enabling it to begin creating model generation annotations (which can be refined using geographic information). Any suitable type of machine learning model for detecting objects in an image can be used, including, but not limited to, convolutional neural networks. In addition, any suitable technique for training a bootstrap machine learning model using the training data set can be used, including, but not limited to, gradient descent. At box 504, the model training engine 316 stores the bootstrap machine learning model in the model data store 320.

[0064] At block 506, the image collection engine 312 receives a first new set of aerial images of a geographic area and pose information associated with the new aerial images. In some embodiments, the geographic area depicted in the first new set of aerial images is the same as the geographic area depicted in the aerial images processed between the for loop start block 408 and the for loop end block 418. In some embodiments, the geographic area depicted in the first new set of aerial images overlaps with the geographic area depicted in the aerial images processed between the for loop start block 408 and the for loop end block 418, in which case further processing may be limited to the overlapping portions. In some embodiments, the new aerial images are of the same geographic area, but were taken from a different pose, at a different time of day, during a different season, during different weather conditions, or otherwise appear different than previously processed aerial images of the geographic area.

[0065] Fig.12 A non-limiting example embodiment of an aerial image set of a single geographic area according to various aspects of the present disclosure is shown. As shown, the aerial image set includes a first aerial image 1202 and a second aerial image 1204. The first aerial image 1202 and the second aerial image 1204 show the same street, the same set of houses, and the same structures to be annotated. However, the two aerial images are taken from different poses, such that the second aerial image 1204 is rotated approximately 180 degrees from the first aerial image 1202, and the second aerial image 1204 is taken from a slightly different angle (see the top view of the car in the lower right portion of the first aerial image 1202, while the side window of the same car can be seen in the upper left portion of the second aerial image 1204). Fig.12 Annotations of structures are shown, together with how aerial images of the same geographic area taken from different poses can increase the size of the training dataset, since the structure will look different in each different pose (but the geographic location of the structure will remain the same).

[0066] return Figure 5 , the method 400 proceeds to a for loop defined between a for loop start block 508 and a for loop end block 516, where each new aerial image in the first set of new aerial images is processed to add the model-generated annotations. The method 400 proceeds from the for loop start block 508 to block 510, where the automatic annotation engine 314 generates model-generated annotations for the new aerial images using the guided machine learning model. Typically, the new aerial images are provided as input to the guided machine learning model, and the locations of the model-generated annotations within the new aerial images are provided as output by the guided machine learning model.

[0067] At block 512 , the geolocation determination engine 318 determines the geolocation of the structure in the new aerial image based on the model-generated annotations and the pose information. In some embodiments, the determination of the geolocation is similar to the determination made in block 416 .

[0068] At block 514, the automatic annotation engine 314 updates the probabilistic locator model based on the geographic location of the structure in the new aerial image. Fig. 9 and Fig.10 As shown, the probabilistic locator model can be updated using any suitable technique, including using Bayesian inference to update existing probability values ​​of segments of the probabilistic locator model associated with the geographic location of each structure in the new aerial image as determined based on the model-generated annotations. In some embodiments, the confidence scores generated by the guided machine learning model for the model-generated annotations can be used to update the probabilities stored in the probabilistic locator model.

[0069] The method 400 then proceeds to the for loop end block 516. If there are still additional new aerial images to be processed, the method 400 returns to the for loop start block 508. Otherwise, the method 400 proceeds to the continue terminal ("Terminal B").

[0070] Method 400 starts from endpoint B ( Figure 6 ) proceeds to block 602, where the image collection engine 312 receives a second set of new aerial images of the geographic area and pose information associated with the new aerial images. The method 400 then proceeds to a for loop defined between a for loop start block 604 and a for loop end block 612, where each aerial image in the second set of new aerial images is annotated using the probabilistic locator model. By doing so, the learning provided by the model-generated annotations and the probabilistic locator model can be geometrically transferred to the second set of new aerial images.

[0071] The method 400 proceeds from the for loop start block 604 to block 606, where the geolocation determination engine 318 determines a location within the new aerial image that is associated with a high probability location indicated by the probabilistic locator model. For example, a probability threshold may be established that indicates when the probability stored in the probabilistic locator model is high enough to indicate the possible presence of a structure. For example, referring to Fig.10 , a probability threshold of 70% will correctly result in the first grid square 904 and the second grid square 906 being selected as highly likely geographic locations.

[0072] At block 608, the automatic annotation engine 314 generates a geo-based annotation for the determined location within the new aerial image. In some embodiments, this process may be similar to the process described above for determining the geo-location of the annotation, but in reverse. Fig.11 , the first voxel 1104 has been previously determined to be a highly probable voxel by the probabilistic locator model 1100. A ray 810 is drawn extending from the camera 102 of the autonomous vehicle 200 through the first voxel 1104. The image location 812 for the geo-based annotation can then be determined by determining the point at which the ray 810 intersects the image 804. It will be noted that the accuracy and precision of the geo-based annotation may be affected by the granularity of the probabilistic locator grid 1102, as well as the accuracy of the pose information, the length of the ray 810, and the distance between the terrain model 808 and the image surface 804.

[0073] return Figure 6 , at block 610, the automatic annotation engine 314 adds the new aerial image and the geo-based annotation to the updated training data set in the model data store 320. The method 400 then proceeds to a for loop end block 612. If there are still additional new aerial images to be processed in the second set of new aerial images, the method 400 returns to the for loop start block 604 to process the next new aerial image. Otherwise, the method 400 proceeds to block 614.

[0074] At block 614, the model training engine 316 retrains the guided machine learning model using the updated training data set to create a retrained machine learning model. Using the geo-based annotations as the basis for the updated training data set allows the knowledge used to build the guided machine learning model to be transferred to the retrained machine learning model via the geo-information, and this transfer allows the retrained machine learning model to have a much higher precision and recall due to the ability to use a larger training set.

[0075] At box 616, the model training engine 316 stores the retrained machine learning model in the model data store 320. At this point, the retrained machine learning model can be used for any suitable purpose. For example, at box 618, the image collection engine 312 receives aerial images of the new geographic area. At box 620, the automatic annotation engine 314 generates model-generated annotations for structures in the new geographic area using the retrained machine learning model. The model-generated annotations generated by the retrained machine learning model have higher precision and recall than the guided machine learning model, so the results for the new geographic area will be of higher quality. At box 622, a route is planned for the unmanned vehicle to avoid geographic locations associated with structures in the new geographic area. Because the model-generated annotations generated by the retrained machine learning model have higher quality, they are reliable enough for navigation, even in the case where the retrained machine learning model is not trained on images of the new geographic area.

[0076] The method 400 then proceeds to an end block and terminates.

[0077] Fig.13 1 is a block diagram illustrating aspects of an exemplary computing device 1300 suitable for use as a computing device in accordance with the present disclosure. Although a number of different types of computing devices are discussed above, exemplary computing device 1300 depicts various elements that are common to many different types of computing devices. Fig.13 1300 is described with reference to a computing device implemented as a device on a network, but the following description is applicable to servers, personal computers, mobile phones, smart phones, tablet computers, embedded computing devices, and other devices that can be used to implement portions of the embodiments of the present disclosure. In addition, those of ordinary skill in the art and others will recognize that the computing device 1300 can be any one of any number of currently available or yet to be developed devices.

[0078] In its most basic configuration, the computing device 1300 includes at least one processor 1302 and system memory 1304 connected via a communication bus 1306. Depending on the exact configuration and type of device, the system memory 1304 may be a volatile or non-volatile memory such as a read-only memory ("ROM"), random access memory ("RAM"), EEPROM, flash memory, or similar memory technology. Those of ordinary skill in the art and others will recognize that the system memory 1304 typically stores data and / or program modules that are immediately accessible to and / or currently being operated on by the processor 1302. In this regard, the processor 1302 may serve as the computational center of the computing device 1300 by supporting the execution of instructions.

[0079] like Fig.13As further shown, computing device 1300 may include a network interface 1310 that includes one or more components for communicating with other devices over a network. Embodiments of the present disclosure may access basic services that utilize network interface 1310 to perform communications using common network protocols. Network interface 1310 may also include a wireless network interface that is configured to communicate via one or more wireless communication protocols (e.g., Wi-Fi, 2G, 3G, LTE, WiMAX, Bluetooth, Bluetooth Low Energy, etc.). As will be appreciated by those of ordinary skill in the art, Fig.13 The network interface 1310 shown in FIG. 13 may represent one or more wireless interfaces or physical communication interfaces described and illustrated above with respect to specific components of the computing device 1300 .

[0080] exist Fig.13 In the exemplary embodiment shown, computing device 1300 also includes storage media 1308. However, the service may be accessed using a computing device that does not include means for persisting data to a local storage medium. Fig.13 The storage medium 1308 depicted in FIG. 1 is represented by a dashed line to indicate that the storage medium 1308 is optional. In any case, the storage medium 1308 can be volatile or non-volatile, removable or non-removable, implemented using any technology capable of storing information, such as but not limited to a hard drive, solid-state drive, CD ROM, DVD or other disk storage, magnetic cassette, tape, magnetic disk storage, etc.

[0081] As mentioned above, the term "computer-readable media" includes volatile and nonvolatile, and removable and non-removable media implemented in any method or technology capable of storing information such as computer-readable instructions, data structures, program modules or other data. In this regard, Fig.13 The system memory 1304 and storage media 1308 depicted in FIG. 1 are merely examples of computer-readable media.

[0082] Suitable implementations of computing devices including processor 1302, system memory 1304, communication bus 1306, storage media 1308, and network interface 1310 are known and commercially available. For ease of illustration and because this is not important to understanding the claimed subject matter, Fig.13Some typical components of many computing devices are not shown. In this regard, computing device 1300 may include input devices such as a keyboard, a keypad, a mouse, a microphone, a touch input device, a touch screen, a tablet computer, etc. Such input devices may be coupled to computing device 1300 via a wired or wireless connection (including RF, infrared, serial, parallel, Bluetooth, Bluetooth low energy, USB, or other suitable connection protocols using wireless or physical connections). Similarly, computing device 1300 may also include output devices such as a display, a speaker, a printer, etc. Since these devices are well known in the art, they are not further illustrated or described herein.

[0083] Although illustrative embodiments have been illustrated and described, it will be appreciated that various changes may be made therein without departing from the spirit and scope of the present disclosure. For example, although the above discussion primarily relates to aerial images, in some embodiments, other types of images associated with pose information and locations of annotated objects may be used.

[0084] In the foregoing description, numerous specific details are set forth to provide a thorough understanding of the embodiments. However, those skilled in the relevant art will recognize that the techniques described herein can be practiced without one or more of the specific details, or using other methods, components, materials, etc. In other cases, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects.

[0085] References throughout this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the phrases "in one embodiment" or "in an embodiment" appearing in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0086] The order in which some or all of the process blocks appear in each process should not be considered limiting. Rather, one of ordinary skill in the art having the benefit of this disclosure will appreciate that some of the process blocks may be performed in various orders not shown, or even in parallel.

[0087] The above description of the illustrated embodiments of the present invention (including what is described in the Abstract) is not intended to be exhaustive or to limit the present invention to the precise forms disclosed. Although specific embodiments and examples of the present invention are described herein for illustrative purposes, various modifications may be made within the scope of the present invention as will be appreciated by those skilled in the relevant art.

[0088] These modifications can be made to the present invention in light of the above detailed description. The terms used in the appended claims should not be interpreted as limiting the present invention to the specific embodiments disclosed in the specification. Instead, the scope of the present invention will be determined entirely by the appended claims, which will be interpreted in accordance with established principles of claim interpretation.

Claims

1. A non-transitory computer-readable storage medium having instructions stored thereon, the instructions, in response to being executed by one or more processors of a computing system, causing the computing system to perform actions, the actions comprising: determining the geographic location of one or more structures within a geographic area; obtain aerial imagery of geographic areas; as well as annotating the aerial image based at least on the determined geographic location of the one or more structures in the geographic area; wherein determining the geographic location of the one or more structures in the geographic area comprises determining annotations of the one or more structures within an aerial image in a set of aerial images of the geographic area; and Wherein, determining annotations of the one or more structures within an aerial image in a set of aerial images of the geographic area comprises: receiving a first set of aerial images of a geographic area; creating a manual annotation of one or more structures within the first set of aerial images; Training guided machine learning models based on the first aerial image set and manual annotations; receiving a second set of aerial images of the geographic area; and Creating model-generated annotations of one or more structures in a second set of aerial images using the guided machine learning model.

2. The computer-readable storage medium of claim 1, wherein: Annotating the aerial image based at least on the determined geographic location of the one or more structures in the geographic area includes, for each of the one or more structures: Use a probabilistic locator model to determine the most likely geographic location of the structure; Determining the pose of the camera that captured the aerial image; as well as The annotated image locations are determined by determining the intersection between the image plane of the aerial image and a ray extending from the camera's pose through the terrain model at the most probable geographical location.

3. The computer-readable storage medium of claim 1, wherein: The actions also include: Adding annotated aerial images to the training dataset; and Train the machine learning model using the training dataset.

4. The computer-readable storage medium of claim 1, wherein: Determining the geographic location of the one or more structures in the geographic area includes: determining a geographic location associated with the annotation; and The probabilistic locator model is updated based on the determined geographic location.

5. The computer-readable storage medium of claim 4, wherein: Updating the probabilistic locator model based on the determined geographic location includes using Bayesian inference to update one or more segments of the probabilistic locator model based on the determined geographic location.

6. The computer-readable storage medium of claim 4, wherein: Determining the geographic location associated with the annotation includes, for each annotation and associated aerial image: Determining the pose of the camera that captured the aerial image; as well as The geographic location is determined based on the annotated image location and the camera pose.

7. The computer-readable storage medium of claim 6, wherein: Determining the geographic location based on the annotated image position and the camera pose includes: Determine the terrain model of the geographic area; and Determine the intersection between the terrain model and a ray extending from the camera to the annotation.

8. The computer-readable storage medium of claim 1, wherein: The actions also include planning a route for the unmanned vehicle to avoid the determined geographic locations of the one or more structures.

9. A computer-implemented method for annotating an aerial image, the method comprising: determining annotations of one or more structures within an aerial image in a collection of aerial images of a geographic area; determining a geographic location associated with the annotation; updating the probabilistic locator model based on the determined geographic location; and Annotation of new aerial imagery using a probabilistic locator model to determine geographic areas; Wherein, determining the geographic location associated with the annotation comprises, for each annotation and the associated aerial image: determining a pose of a camera that captured the associated aerial image; and determining the geographic location based on the annotated image location and the camera pose; and Wherein, determining the geographic location based on the annotated image position and the camera posture comprises: Determine the terrain model of the geographic area; and Determines the intersection between the terrain model and a ray extending from the camera to the annotation.

10. The computer-implemented method of claim 9, wherein: Updating the probabilistic locator model based on the determined geographic location includes using Bayesian inference to update one or more segments of the probabilistic locator model based on the determined geographic location.

11. The computer-implemented method of claim 9, wherein: Determining annotations of the one or more structures within an aerial image in a set of aerial images of a geographic area includes: receiving a first set of aerial images of a geographic area; creating a manual annotation of one or more structures within the first set of aerial images; Training guided machine learning models based on the first aerial image set and manual annotations; receiving a second set of aerial images of the geographic area; and Creating model-generated annotations of one or more structures in a second set of aerial images using the guided machine learning model.

12. The computer-implemented method of claim 11, wherein: Creating the model-generated annotations of the one or more structures within the second set of aerial images includes determining confidence scores for the annotations using a probabilistic locator model.

13. The computer-implemented method of claim 9, wherein: Determining a terrain model for the geographic area includes using structure from motion (SFM) techniques to determine a terrain model based on a plurality of aerial images of the geographic area.

14. The computer-implemented method of claim 9, wherein: Determining, using the probabilistic locator model, annotation of new aerial images of the geographic area includes, for each new aerial image and for each of the one or more structures: Use a probabilistic locator model to determine the most likely geographic location of the structure; Determine the pose of the camera that captured the new aerial image; as well as The annotated image locations are determined by determining the intersection between the image plane of the new aerial image and a ray extending from the camera's pose through the terrain model at the most likely geographical location.

15. The computer-implemented method of claim 14, further comprising: Add new aerial images to the updated training dataset; as well as Retrain the machine learning model using the updated training dataset.

16. The computer-implemented method of claim 15, further comprising: obtain aerial imagery of new geographic areas; as well as Annotate aerial imagery of new geographic areas using retrained machine learning models.

17. The computer-implemented method of claim 9 further comprises planning a route for the unmanned vehicle to avoid the determined geographic location associated with the annotation.

18. A non-transitory computer-readable storage medium having stored thereon instructions that, in response to being executed by one or more processors of a computing system, cause the computing system to perform actions, the actions comprising: determining the geographic location of one or more structures within a geographic area; obtain aerial imagery of geographic areas; as well as annotating the aerial image based at least on the determined geographic location of the one or more structures in the geographic area; as well as wherein determining the geographic location of the one or more structures in the geographic area comprises determining annotations of the one or more structures within an aerial image in a set of aerial images of the geographic area and determining the geographic location associated with the annotations; Wherein, determining the geographic location associated with the annotation comprises, for each annotation and the associated aerial image: determining the pose of a camera that captured the aerial image; and determining the geographic location based on the annotated image location and the camera pose; and Wherein, determining the geographic location based on the annotated image position and the camera posture comprises: Determine the terrain model of the geographic area; and Determines the intersection between the terrain model and a ray extending from the camera to the annotation.

19. The computer-readable storage medium of claim 18, wherein: Determining the geographic locations of the one or more structures in the geographic area further includes updating a probabilistic locator model based on the determined geographic locations associated with the annotations.

20. The computer-readable storage medium of claim 18, wherein: Determining annotations of the one or more structures within an aerial image in a set of aerial images of a geographic area includes: receiving a first set of aerial images of a geographic area; creating a manual annotation of one or more structures within the first set of aerial images; Training guided machine learning models based on the first aerial image set and manual annotations; receiving a second set of aerial images of the geographic area; and Creating model-generated annotations of one or more structures in a second set of aerial images using the guided machine learning model.

21. A non-transitory computer-readable storage medium having stored thereon instructions that, in response to being executed by one or more processors of a computing system, cause the computing system to perform actions, the actions comprising: determining annotations of one or more structures within an aerial image in a collection of aerial images of a geographic area; determining a geographic location associated with the annotation; updating the probabilistic locator model based on the determined geographic location; and Annotation of new aerial imagery using a probabilistic locator model to determine geographic areas; Wherein, determining the geographic location associated with the annotation comprises, for each annotation and the associated aerial image: determining a pose of a camera that captured the associated aerial image; and determining the geographic location based on the annotated image location and the camera pose; and Wherein, determining the geographic location based on the annotated image position and the camera posture comprises: Determine the terrain model of the geographic area; and Determines the intersection between the terrain model and a ray extending from the camera to the annotation.

Images