Camera pose refinement with ground-to-satellite image registration

Abstract

The present disclosure provides "camera pose refinement with ground-to-satellite image registration". A computer includes a processor and a memory including instructions executable by the processor to estimate a relative rotation between a ground view image and an aerial view image with a rotation estimator. And projecting a ground feature map and a confidence map corresponding to the ground view angle image to an aerial feature map corresponding to the aerial view angle image according to the relative rotation to create a projection overlook view angle feature map. A spatial correlation is used to determine a translation difference between the projected top view feature map and the aerial feature map. A high definition estimated three degree of freedom pose of the ground view camera is determined based on the relative rotation and the translation difference.

Description

Technical Field

[0001] The present disclosure relates to satellite imagery guided geolocation to enhance the determination of the high-resolution pose of a vehicle. Background Art

[0002] The computer can be used to operate a system, including a vehicle, a robot, a drone, and / or an object tracking system. Data, including images, can be acquired by sensors and processed using a computer to determine the position of the system relative to objects in the environment surrounding the system. The computer can use the position to determine a trajectory for moving the system in the environment. The computer can then determine control data to be transmitted to system components to control the system components to move the components according to the determined trajectory. Summary of the invention

[0003] Systems (including vehicles, robots, drones, etc.) can operate by acquiring sensor data about the environment surrounding the system and processing the sensor data to determine a path on which to operate the system or part of the system. Sensor data can be processed to determine the location of objects in the environment. Objects can include roads, buildings, conveyors, vehicles, pedestrians, manufactured parts, etc. Sensor data can be processed to determine the attitude of the system, where the "attitude" specifies the position and orientation of an object (such as a system and / or its components). The system attitude can be determined based on a full six-degree-of-freedom (DoF) attitude, which includes x, y, and z position coordinates, and roll, pitch, and yaw rotation coordinates relative to the x, y, and z axes, respectively. The six-DoF attitude can be determined relative to a global coordinate system such as a Cartesian coordinate system, where points can be specified based on latitude, longitude, and altitude or some other x, y, and z axes.

[0004] A vehicle is used herein as a non-limiting example of a system. A simpler three-DoF pose can be used to position the vehicle relative to the environment surrounding the vehicle, which assumes that the vehicle is supported on a planar surface such as a road that fixes the z, pitch, and roll coordinates of the vehicle to match the road. The vehicle pose can be described by x and y position coordinates and yaw rotation coordinates to provide a three-DoF pose that defines the position and orientation of the vehicle relative to the supporting surface.

[0005] Vehicle sensors can provide data that can be used to determine the vehicle's attitude and, in turn, can be used to locate the vehicle relative to an aerial image that includes position data in global coordinates. For example, vehicle sensors can provide data to determine position and / or attitude based on a satellite-based global positioning system (GPS) and / or an accelerometer-based inertial measurement unit (IMU). For example, position data included in an aerial image can be used to determine the position of any pixel address location in the aerial image in global coordinates. Aerial images can be obtained by satellite, aircraft, drone, or other aerial platform. Satellite data will be used herein as a non-limiting example of aerial image data. For example, it can be obtained by downloading GOOGLE TM Maps, etc. to obtain satellite images.

[0006] The use of global coordinate data included in or with satellite images to determine the attitude of an object (such as a vehicle) relative to satellite image data can generally provide attitude data within + / -3 meter position and + / -3 degree orientation resolution. Operating a vehicle can rely on attitude data, which includes a position resolution of one meter or less and an orientation resolution of one degree or less. For example, + / -3 meter position data may not be enough to determine the position of a vehicle relative to a lane on a road. The technology for geolocation guided by satellite images as discussed herein can determine the vehicle attitude within a specified resolution, which is generally within a position resolution of one meter or less and an orientation resolution of one degree or less, for example, a resolution sufficient to operate a vehicle on a road. The vehicle attitude data determined within a specified resolution (that is, exceeding one or more specified resolution thresholds, for example, a position resolution of one meter or less and an orientation resolution of one degree or less in an exemplary implementation) is referred to as high-definition attitude data in this article.

[0007] The technology described herein uses satellite image guided geolocation to enhance the determination of the high definition pose of the vehicle. Satellite image guided geolocation uses images acquired by sensors included in the vehicle to determine the high definition pose relative to the satellite image without the need for a predetermined high definition (HD) map. The vehicle sensor image and the satellite image are input to a rotation estimator and two separate neural networks that extract features from the image as well as a confidence map. In some examples, the two separate neural networks can be the same neural network. 3D feature points from the vehicle image are matched with 3D feature points from the satellite image to determine the high definition pose of the vehicle relative to the satellite image. The high definition pose of the vehicle can be used to operate the vehicle by determining the vehicle path based on the high definition path.

[0008] A system is disclosed herein, the system comprising a computer, the computer comprising a processor and a memory. The memory comprises instructions executable by the processor to estimate a relative rotation between a ground view image and an aerial view image using a rotation estimator. A ground feature map and a confidence map corresponding to the ground view image are projected to an aerial feature map corresponding to the aerial view image according to the relative rotation to create a projected top-view feature map. A translation difference between the projected top-view feature map and the aerial feature map is determined using spatial correlation. A high-definition estimated three-degree-of-freedom pose of a ground view camera is determined based on the relative rotation and the translation difference.

[0009] The system may include instructions for determining the ground feature map and the confidence map from the ground view image using a first neural network and instructions for determining the aerial feature map from the aerial view image using a second neural network. The system may be supervised using a combination of self-supervised learning and weak supervision.

[0010] The rotation estimator may include instructions for extracting ground features from the ground view image using a third neural network and extracting aerial features from the aerial view image using a fourth neural network. The rotation estimator may include instructions for projecting the ground features onto the aerial features to create a top-down view projection. The rotation estimator may include instructions for estimating a relative rotation between the ground features and the top-down view projection using a neural pose optimizer.

[0011] The estimated three-degree-of-freedom pose of the ground-view camera may be determined based on an initial estimate of the three-degree-of-freedom pose of the ground-view camera. The first neural network and the second neural network may have a U-net architecture. The aerial view image may be a satellite image.

[0012] The high-resolution estimated three-degree-of-freedom pose of the ground-view camera is output and used to operate a vehicle. The high-resolution estimated three-degree-of-freedom pose of the ground-view camera and the aerial-view image are used to determine a vehicle path on which to operate the vehicle.

[0013] A method is disclosed herein, the method comprising estimating a relative rotation between a ground view image and an aerial view image using a rotation estimator. The method may include projecting a ground feature map and a confidence map corresponding to the ground view image to an aerial feature map corresponding to the aerial view image according to the relative rotation to create a projected top view feature map. A translation difference between the projected top view feature map and the aerial feature map is determined using spatial correlation. The method may also include determining a high-definition estimated three-degree-of-freedom pose of a ground view camera based on the relative rotation and the translation difference.

[0014] The method may also include determining the ground feature map and the confidence map from the ground view image using a first neural network, and determining the aerial feature map from the aerial view image using a second neural network. The first neural network and the second neural network may be supervised using a combination of self-supervised learning and weak supervision.

[0015] The method may also include randomly rotating and translating the aerial training image, and training the rotation estimator based on the aerial training image and the randomly rotated and translated aerial training image. The method may also include extracting a triangular region of the randomly rotated and translated aerial training image. Estimating the relative rotation may include extracting ground features from the ground view image using a third neural network and extracting aerial features from the aerial view image using a fourth neural network.

[0016] Estimating the relative rotation may include projecting the ground feature onto the aerial feature to create a bird's-eye view projection. Estimating the relative rotation may include estimating the relative rotation between the ground feature and the bird's-eye view projection using a neural pose optimizer. The estimated three-degree-of-freedom pose of the ground view camera may be determined based on an initial estimate of the three-degree-of-freedom pose of the ground view camera.

[0017] The method may also include outputting the high-definition estimated three-degree-of-freedom pose of the ground-view camera to operate a vehicle. The method may also include determining a vehicle path on which to operate the vehicle based on the high-definition estimated three-degree-of-freedom pose of the ground-view camera and the aerial view image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a block diagram of an example vehicle sensing system.

[0019] Figure 2 is a diagram of an example satellite image that includes a vehicle.

[0020] Figure 3is a diagram of another example satellite image that includes a vehicle.

[0021] Figure 4A Is used for training Figure 4B A diagram of an example system of a rotation estimator system is shown.

[0022] Figure 4B is a diagram of an example rotation estimator for determining an estimated relative rotation between a ground image and an aerial image.

[0023] Figure 5 is a diagram of an example pose refinement system for determining a high-resolution three-degree-of-freedom vehicle position in global coordinates.

[0024] Figure 6 is a flow chart of an example process for determining a high-resolution three-degree-of-freedom vehicle position in global coordinates.

[0025] Figure 7 is a flow chart of an example process for determining an estimated relative rotation between a ground image and an aerial image.

[0026] Figure 8 is a flow chart of an example process for operating a vehicle based on a high-resolution vehicle position in global coordinates. DETAILED DESCRIPTION

[0027] Figure 1 is a diagram of a sensing system 100. The sensing system 100 includes a vehicle 110 that can be operated by a user and / or under the control of a computing device 115, which can include one or more vehicle electronic control units (ECUs) or computers, such as are known, which ECUs or computers may include additional hardware, software, and / or programming, as described herein. The computing device 115 can receive data about the operation of the vehicle 110 from sensors 116. The computing device 115 can operate the vehicle 110 or components thereof in lieu of or in combination with the control of a human user. The system 100 can also include a server computer 120 that can communicate with the vehicle 110 via a network 130.

[0028] The computing device 115 includes a processor and memory such as are known. In addition, the memory includes one or more forms of computer-readable media and stores instructions that can be executed by the processor to perform various operations (including operations as disclosed herein). For example, the computing device 115 may include programming to operate one or more of vehicle braking, propulsion (e.g., controlling the acceleration of the vehicle 110 by controlling one or more of an internal combustion engine, an electric motor, a hybrid engine, etc.), steering, climate control, interior lights and / or exterior lights, etc., and determine whether and when the computing device 115 (rather than a human operator) controls such operations.

[0029] The computing device 115 may include more than one computing device, for example, a controller, ECU, etc. included in the vehicle 110 for monitoring and / or controlling various vehicle subsystems (e.g., propulsion subsystem 112, braking subsystem 113, steering subsystem 114, etc.), or may be communicatively coupled to the more than one computing device, for example, via a vehicle communication bus as further described below. The computing device 115 is typically arranged to communicate on a vehicle communication network (e.g., including a bus in the vehicle 110, such as a controller area network (CAN), etc.); the vehicle network may additionally or alternatively include, for example, known wired or wireless communication mechanisms, such as Ethernet or other communication protocols.

[0030] The computing device 115 may transmit messages to and / or receive messages from various devices in the vehicle (e.g., controllers, actuators, sensors (including sensor 116), etc.) via the vehicle network. Alternatively or additionally, where the computing device 115 actually includes multiple devices, the vehicle communication network may be used for communication between devices represented as computing device 115 in this disclosure. In addition, as mentioned below, various controllers or sensing elements (such as sensor 116) may provide data to the computing device 115 via the vehicle communication network.

[0031] Additionally, computing device 115 may be configured to communicate with remote server computer 120 (e.g., a cloud server) via network 130 through vehicle-to-infrastructure (V2X) interface 111, as described below, which includes hardware, firmware, and software that permit computing device 115 to communicate with remote server computer 120 (e.g., a cloud server) via a vehicle-to-infrastructure (V2X) interface 111, such as a wireless Internet connection. The V2X interface 111 may thus include a computer configured to utilize various wired and / or wireless networking technologies (e.g., cellular, The computing device 115 may be configured to communicate with other vehicles 110 through a V2X (vehicle to the outside world) interface 111 using, for example, a vehicle-to-vehicle (V2V) network formed between nearby vehicles 110 on the basis of a mobile ad hoc network or through an infrastructure-based network (e.g., based on cellular communication (C-V2X) wireless communication cellular, dedicated short-range communication (DSRC) and / or similar communications). The computing device 115 also includes a non-volatile memory such as known. The computing device 115 can record data by storing the data in the non-volatile memory for later retrieval and transmission to the server computer 120 or the user mobile device 160 via the vehicle communication network and the vehicle to infrastructure (V2X) interface 111.

[0032] As already mentioned, the instructions stored in the memory and executable by the processor of the computing device 115 generally include programming for operating one or more vehicle 110 components (e.g., braking, steering, propulsion, etc.). Using data received in the computing device 115 (e.g., sensor data from sensors 116, server computer 120, etc.), the computing device 115 can make various determinations and / or control various vehicle 110 components and / or operations. For example, the computing device 115 can include programming to regulate or control vehicle 110 operating behavior (e.g., the physical manifestation of vehicle 110 operation), such as speed, acceleration, deceleration, steering, etc., as well as strategic behavior (e.g., generally controlling operating behavior in a manner intended to achieve efficient traversal of a route), such as the distance between vehicles and / or the amount of time between vehicles, lane changes, minimum gaps between vehicles, left turn crossing path minimums, arrival times at specific locations, and the minimum time from arrival at an intersection to crossing the intersection (without a signal light).

[0033] Each of the subsystems 112, 113, 114 may include a corresponding processor and memory and one or more actuators. The subsystems 112, 113, 114 may be programmed and connected to a vehicle 110 communication bus, such as a controller area network (CAN) bus or a local interconnect network (LIN) bus, to receive instructions from a computing device 115 and control the actuators based on the instructions.

[0034] Sensors 116 may include a variety of devices such as are known to provide data via a vehicle communication bus. For example, a radar fixed to a front bumper (not shown) of vehicle 110 may provide a distance from vehicle 110 to the next vehicle in front of vehicle 110, or a global positioning system (GPS) sensor disposed in vehicle 110 may provide geographic coordinates of vehicle 110. The distances provided by radar and / or other sensors 116 and / or the geographic coordinates provided by a GPS sensor may be used by computing device 115 to operate vehicle 110.

[0035] The vehicle 110 is typically a land-based vehicle 110 with three or more wheels, such as a passenger car, a light truck, etc. The vehicle 110 includes one or more sensors 116, a V2X interface 111, a computing device 115, and one or more subsystems 112, 113, 114. The sensor 116 can collect data related to the vehicle 110 and the operating environment of the vehicle 110. By way of example and not limitation, the sensor 116 can include, for example, an altimeter, a camera, a lidar, a radar, an ultrasonic sensor, an infrared sensor, a pressure sensor, an accelerometer, a gyroscope, a temperature sensor, a pressure sensor, a Hall sensor, an optical sensor, a voltage sensor, a current sensor, a mechanical sensor (such as a switch), etc. The sensor 116 can be used to sense the operating environment of the vehicle 110, for example, the sensor 116 can detect phenomena such as weather conditions (rainfall, external ambient temperature, etc.), road slope, road position (e.g., using road edges, lane markings, etc.) or the position of a target object (such as a neighboring vehicle 110). The sensors 116 may also be used to collect data, including dynamic vehicle 110 data related to the operation of the vehicle 110 , such as speed, yaw rate, steering angle, engine speed, brake pressure, oil pressure, power levels applied to subsystems 112 , 113 , 114 in the vehicle 110 , connectivity between components, and accurate and timely performance of components of the vehicle 110 .

[0036] The server computer 120 generally has features in common with the V2X interface 111 and computing device 115 of the vehicle 110, such as a computer processor and memory and a configuration for communicating via the network 130, and therefore these features will not be described further. The server computer 120 can be used to develop and train software that can be downloaded or otherwise installed to the computing device 115 in the vehicle 110.

[0037] Figure 2is a diagram of a satellite image 200. The satellite image 200 may be a map downloaded to a computing device 115 in a vehicle 110 via a network 130, for example, from a source such as GOOGLE maps. The satellite image 200 includes a road 202 indicated by a straight shape, a building 204, and foliage 206 indicated by an irregular shape. The version of the satellite image 200 used herein is a version of a photographic portrait including objects such as the road 202, the building 204, and the foliage 206. A vehicle, such as the vehicle 110, is included in the satellite image 200. The vehicle 110 includes a sensor 116, including a camera. The satellite image 200 includes four fields of view 208, 210, 212, 214 (e.g., spatial areas within which the corresponding cameras can capture images) for four cameras respectively included in the front, right, rear, and left sides of the vehicle 110.

[0038] Figure 3 is a diagram of satellite image 200 including estimated three DoF pose 302 of vehicle 110. For example, initial estimated three DoF pose 302 of vehicle 110 relative to satellite image 200 may be based on vehicle sensor data including a GPS sensor included in vehicle 110. Due to the limited resolution of the GPS sensor and the limited resolution of satellite image 200, estimated three DoF pose 302 of vehicle 110 generally does not represent a sufficiently accurate pose of vehicle 110. Due to the limited resolution of the GPS sensor and satellite image 200, estimated three DoF pose 302 is generally not used to operate vehicle 110.

[0039] One approach to the problem of obtaining high-definition data for operating the vehicle 110 may be to generate an HD map of all areas over which the vehicle 110 operates. High-definition maps typically require a significant amount of cartographic work and a significant amount of computer resources to generate and store the HD maps, and typically require a significant amount of network bandwidth to download the HD maps to the vehicle 110, not to mention the significant amount of computer memory typically required to store the maps in a computing device 115 included in the vehicle. The satellite image-guided pose refinement technique described herein uses 3D feature points determined based on video images acquired by a camera included in the vehicle 110 to determine a high-definition three-DoF pose of the vehicle 110 based on satellite imagery without requiring the significant amount of computer processing, networking, and / or memory resources typically required to generate, transmit, and store HD maps.

[0040] The images corresponding to the fields of view 208, 210, 212, 214 may be acquired by a camera included in the vehicle 110. For example, the images may be red, green, and blue (RGB) color images acquired at a standard video resolution of approximately 2K x 1K pixels. The satellite image guided geolocation techniques described herein may generally determine the three DoF pose of the vehicle 110 to within one meter of x,y position and one degree of yaw orientation, which is more accurate than the 3+ meter position and 3+ degree orientation accuracy of the estimated three DoF pose 302. The satellite image guided geolocation techniques determine the three DoF pose of the vehicle 110 by determining the positions of the fields of view 208, 210, 212, 214 relative to the satellite image 200 by extracting 3D feature points from the images and matching them with the 3D feature points extracted from the satellite image 200.

[0041] Given a rough position and orientation of a ground camera, the disclosed system and method refines this pose through ground-to-satellite image registration. Compared to previous techniques that assume a large training dataset with highly accurate pose labels of ground images, the disclosed pose refinement technique includes a self-supervised learning strategy that does not require such labels.

[0042] The disclosed system and method for camera pose refinement using ground-to-satellite image registration includes two stages. The first stage uses a rotation estimator 400 ( Figure 4B ) to predict the rotational alignment between the ground view image and the aerial view image (e.g., satellite image). In the second stage, the translation estimator 500 ( Figure 5 ) estimates the translation between the ground view image and the aerial view image based on the estimated rotation from the rotation estimator 400. The rotation estimator 400 and the translation estimator 500 may be implemented using software instructions operating on a vehicle computer, such as a computing device 115 included in the vehicle 110. The rotation estimator 400 and the translation estimator 500 may be trained on a server computer 120 and downloaded or otherwise installed to the computing device 115 in the vehicle 110.

[0043] like Figure 4AAs shown, to train the rotation estimator 400, a satellite image-based reference image and query image pair (402, 404) with accurate relative poses can be generated. This is achieved by rotating and translating 406 the reference satellite image 402 using a randomly generated pose R*, t*. A mask 408 can be applied to the transformed (i.e., rotated and translated) satellite image to extract a triangular region corresponding to the FoV of the ground camera. This technique provides a synthetic ground view image 404 derived from the reference satellite image 402. Therefore, the relative pose between the reference satellite image 402 and the synthetic ground view image 404 is known. To train the rotation estimator 400, the synthetic ground view image 404 is used as a query image, and the original satellite image 402 is used as a reference image.

[0044] A dual-branch feature extractor is used to extract deep features from the reference image and the query image. A reference feature extractor 410 can determine a reference satellite feature map, and a query feature extractor 412 can determine a query feature map. Based on an initial pose (typically a coarse pose to be refined), the query feature map is rotated and translated 414. The difference between the query feature map and the reference satellite image feature map is used as input to a neural pose optimizer 416, which outputs an updated relative pose (R, t) between the reference map and the query map. The neural pose optimizer 416 is implemented from a coarse to fine feature level to provide a global search as well as fine-tuning around local minima. The network can be supervised by:

[0045]

[0046] Where R l and t l represents the network prediction at feature level l, R* and t* indicate the corresponding ground truth, the 1-DoF in-plane rotation R corresponds to the yaw angle of the ground camera, and the 2-DoF translation t corresponds to the position of the ground camera along the latitude and longitude directions, and |·| denotes the L1 norm.

[0047] refer to Figure 4BOnce the rotation estimator 400 has been trained, the query ground image 405 can be input into the rotation estimator 400 during inference. The projected geometry of the satellite image and the ground image captured by the pinhole camera is similar. They both map straight lines in the real world to straight lines in the image. Therefore, their feature extractors can be shared. In other words, the feature extractors 410 and 412 can share weights. The rotation and translation block 414 from the training phase can be replaced with a top-down view projection module 418 to handle the viewpoint difference between the satellite view 403 and the ground view 405 during evaluation. The top-down view projection module 418 utilizes the ground plane homography to project the ground view observation to the top-down view, and it has no learnable parameters. The output from the neural pose optimizer 416 is taken as the estimated relative pose between the ground view 405 and the satellite image 403.

[0048] The neural network output may be sensitive to the rotation of the input, which means that even small differences in rotation may be amplified in the network output. Therefore, the posture optimizer 416 built using a neural network shows good performance in rotation estimation. In contrast, the translation difference in the input signal may be absorbed by the high-level deep features inside the neural optimizer 416 due to the aggregation layer, making the translation difference in the input image unknown. This may lead to less than optimal translation estimation performance. However, the equivariant property of the convolutional neural network to translation means that the relative translation between the two input signals can be recovered by spatial correlation. Therefore, in the above-mentioned second stage, the translation is estimated using spatial correlation based on the estimated rotation from the rotation estimator 400. The translation estimate from the rotation estimator 400 is ignored in favor of the translation estimate based on spatial correlation.

[0049] Figure 5 is a diagram of a satellite image guided pose refinement system 500 as described herein. The satellite image guided pose refinement system 500 may be executed on a computing device 115 included in a vehicle 110. The pose refinement system 500 may input an image 405 from a vehicle sensor and a satellite image 403 from a computing device 115 memory or downloaded from the Internet, and output a high definition estimated three DoF pose of the vehicle 110. The satellite image 403 may be selected based on the estimated three DoF pose 302 of the vehicle 110 and retrieved from a memory or downloaded to the computing device 115 from a network 130 (e.g., including via the Internet).

[0050] like Figure 5As shown, the pose refinement system 500 estimates translation using a dual-branch convolutional network applied to a satellite image 403 and a ground view image 405. When the pinhole camera captures the ground image, the two branches 506, 508 can share weights. However, when the ground image is a panorama, they do not share weights. This is because the spherical projection of the panorama maps straight lines in the real world to curves in the image. Therefore, the feature patterns used for learning satellite images and ground panoramas are not shareable. Each branch can be a U-Net architecture and extract a multi-level representation of the original image.

[0051] For the ground branch feature extractor 506, not only the feature representation is extracted ×C , and also extract the confidence map at feature level l The confidence map 512 indicates whether the feature at the corresponding spatial pixel location is credible. For example, dynamic objects (e.g., cars) in an image may be detrimental to localization performance, while road structures are features of interest. The higher the confidence, the more reliable the corresponding feature. It should be noted that no explicit supervision is applied to the confidence map 512. Instead, it is encoded in the cross-view similarity matching process and statistically learned from the similarity matching training objective.

[0052] The satellite branch feature extractor 508 only extracts the feature representation of the satellite branches at feature level l There is no confidence map because the inventors have empirically determined that learning confidence maps for satellite images will hurt performance. The hypothesized reasons are twofold: (1) dynamic objects are rare in satellite images and they occupy relatively small areas in satellite images compared to ground-view images; therefore, their impact on localization performance is low; and (2) neither the camera pose nor the confidence map is explicitly supervised, so learning confidence maps for satellite images increases the difficulty of network training.

[0053] The feature extractors 410, 412, 506, 508 may be convolutional neural networks (CNNs) that include a convolutional layer followed by a fully connected layer. The convolutional layer extracts latent variables indicating the locations of feature points by convolving an input image (such as image 405) with a series of convolutional kernels. The latent variables are input to a fully connected layer that determines feature points by combining the latent variables using linear and nonlinear functions. The convolutional kernels and the linear and nonlinear functions are programmed using weights determined by training the feature extractor.

[0054] The trained rotation estimator 400 estimates the relative rotation between the ground view image 405 and the satellite image 403. The ground view features and confidence map are then projected to the top view according to the estimated rotation R and zero translation t=0. Figure 4B, the top-down projection 514 utilizes the ground plane homography and has no learnable parameters. l g and C l g denote the projected ground features and confidence map at feature level l, respectively. The ground plane in this context is the plane that coincides with the road or other surface (such as a parking lot) supporting the vehicle.

[0055] Confidence-guided feature matching 516 can be used with spatial correlation to determine translation. Considering that the rotation of the projected top-down feature map 514 has been matched to the observed satellite feature map F s The only remaining parallax between them is the translation difference. The translation difference is the distance and direction between the initial pose and the predicted pose, which can be expressed as an x ​​and y displacement from the initial pose measured in meters, for example. To calculate this translation difference, spatial correlation is exploited. Specifically, the projected top-down view feature map 514 is used as a sliding window and its comparison with the reference satellite feature map F is calculated when aligned at different locations. s This generates a similarity score S between two view features at the same level l when aligned at each location (u,v) l (u, v). These similarity scores can be presented in a similarity map 518. Considering the ground view confidence map 512, the mathematical representation of the spatial correlation process is as follows:

[0056]

[0057] in The purpose is to highlight important features while suppressing unreliable features for localization. The pixel coordinates corresponding to the maximum similarity indicate the most likely ground camera position at the finest feature level. The difference from the initial position estimate (eg, estimated three DoF pose 302) is the translation. Using the translation and rotation estimates, the initial position 302 can be refined (eg, translated and rotated) to a high-resolution pose.

[0058] The network can be trained using a combination of self-supervised learning and weak supervision, depending on the relative accuracy of the available training data. In the "self-supervised learning" described in this paper, the pose errors of the ground images in the training data are the same as the pose errors of the ground images to be enhanced during deployment. For example, the coarse pose estimates for the training set and the test set come from the same noisy GPS and compass sensors.

[0059] Deep metric learning can be used for network self-supervision. For a query ground view image, based on the query ground view image and its matching and non-matching satellite images (respectively denoted as S lpos and S l neg ) to determine the similarity graph. l pos Maximize the maximum similarity in S l neg Minimize the maximum similarity in:

[0060]

[0061] where α controls the convergence speed and can be set to, for example, 10.

[0062] Weak supervision can be used in examples where relatively more accurate position labels are available in the training data for ground-view images than the accuracy of the pose to be refined during use. The statistical position error is represented as d meters in the training data. Additional training objectives can incorporate this signal as follows:

[0063]

[0064] Where (u*, v*) indicates the location labels provided by the training data and has an error of up to d meters, and γ represents the ground resolution of the similarity map in meters / pixel. This training objective forces the global maximum in the similarity map to be equal to the local maximum, where the local region is centered at the location label with a radius of d meters. The combined training objective of the weakly supervised version is L=L2+L3. The statistical location error d can be set to d=5 meters. The training objective can be a combined training loss for translation estimation and is expressed as:

[0065]

[0066] Where λ = 0 indicates that the network is trained in a fully self-supervised manner.

[0067] Figure 6 About Figures 1 to 5 Flowchart of process 600 for determining high-resolution estimated three-DoF pose based on satellite image guided geolocation is described. Process 600 may be implemented in a computing device 115 included in a vehicle 110. Process 600 includes multiple blocks that may be executed in the order shown. Alternatively or additionally, process 600 may include fewer blocks, or may include blocks that are executed in a different order.

[0068] The process 600 begins at block 602, where a computing device 115 in a vehicle 110 receives an image 405 from one or more cameras included in the vehicle 110. The one or more images 405 include image data regarding an environment surrounding the vehicle 110, and may include any portion of the environment surrounding the vehicle, including overlapping fields of view 208, 210, 212, 214, as long as the image 405 includes data regarding a ground plane, where the ground plane is a plane that coincides with a road or surface supporting the vehicle 110.

[0069] At block 604, computing device 115 receives an aerial view image (e.g., satellite image 403). For example, satellite image 403 may be obtained by downloading satellite image 403 from the Internet via network 130. Satellite image 403 may also be retrieved from a memory included in computing device 115. Satellite image 403 includes position data in global coordinates, which may be used to determine the position of any point in satellite image 403 in global coordinates. Satellite image 403 may be selected to include estimated three DoF pose 302. Estimated three DoF pose 302 may be determined by acquiring data from vehicle sensors 116 (e.g., GPS).

[0070] At block 606, the computing device 115 inputs the received ground-view image 405 into a trained first neural network, such as the feature extractor 506. The first neural network may be trained on the server computer 120 and downloaded or otherwise installed to the computing device 115 in the vehicle 110. The first neural network determines a ground feature map F corresponding to the acquired ground-view image 405. g and ground confidence map C g .

[0071] At block 608, the computing device 115 also inputs the received aerial perspective image (e.g., satellite image 403) into a trained second neural network, e.g., feature extractor 508. The second neural network may be trained on a server computer 120 and downloaded or otherwise installed to a computing device 115 in the vehicle 110. The second neural network determines an aerial feature map F corresponding to the acquired aerial perspective image 403. s .

[0072] At block 610, computing device 115 utilizes rotation estimator 400 to estimate the relative rotation between ground view image 405 and aerial view image 403. Figure 7 The rotation estimator process is further described.

[0073] At block 612, the computing device 115 creates a projected aerial view feature map 514. The computing device 115 projects the ground feature map F corresponding to the ground view image 405 according to the relative rotation. g and confidence map C g Projected onto the aerial feature map F corresponding to the aerial view image 403 s , to create a projected top-down perspective feature map 514.

[0074] At block 614, the computing device 115 uses spatial correlation to determine whether the projected top-down feature map 514 and the aerial feature map F s The translation difference between .

[0075] At block 616, computing device 115 determines a high-resolution estimated three-DoF pose of the ground-view camera based on the relative rotation and translation differences. Computing device 115 outputs the high-resolution estimated three-DoF pose for use in operating vehicle 110, as described below with respect to Figure 8 After block 616, process 600 ends.

[0076] Figure 7 About Figures 1 to 6 Described for determining Figure 6 Flowchart of a process 610 for rotation estimation as described in . The process 610 may be implemented in a computing device 115 included in a vehicle 110. The process 610 includes a plurality of blocks that may be executed in the order shown. Alternatively or additionally, the process 610 may include fewer blocks, or may include blocks that are executed in a different order.

[0077] Process 610 begins at block 702 , where computing device 115 in vehicle 110 receives ground perspective image 405 from a camera included in vehicle 110 .

[0078] At block 704, computing device 115 receives an aerial view image (eg, satellite image 403). For example, satellite image 403 may be obtained by downloading satellite image 403 from the Internet via network 130. Satellite image 403 may also be retrieved from a memory included in computing device 115.

[0079] At box 706, the computing device 115 inputs the received ground perspective image 405 into a trained third neural network (e.g., feature extractor 412) to extract ground features from the ground perspective image 405. The third neural network may be trained on the server computer 120 and downloaded or otherwise installed to the computing device 115 in the vehicle 110.

[0080] At box 708, the computing device 115 also inputs the received aerial perspective image (e.g., satellite image 403) into the trained fourth neural network (e.g., feature extractor 410) to extract aerial features from the aerial perspective image 403. The fourth neural network can be trained on the server computer 120 and downloaded or otherwise installed to the computing device 115 in the vehicle 110. In an example, the third neural network and the fourth neural network can be the same. In other words, the third neural network and the fourth neural network can share weights.

[0081] At block 710 , the computing device 115 projects the ground features onto the aerial features to create the bird's-eye view projection 418 .

[0082] At block 712, computing device 115 utilizes neural pose optimizer 416 to estimate the relative rotation between ground features and overhead perspective projection 418. Neural pose optimizer 416 may be trained on server computer 120 and downloaded or otherwise installed to computing device 115 in vehicle 110.

[0083] The rotation estimator 400 (e.g., feature extractors 410, 412, and neural pose optimizer 416) may be trained by randomly rotating and translating the aerial training image 402 and extracting triangular regions of the randomly rotated and translated aerial training image to derive the synthesized ground perspective image 404. The rotation estimator may be trained based on the aerial training image 402 and the randomly rotated and translated aerial training image (i.e., the synthesized ground perspective image 404).

[0084] Figure 8 About Figures 1 to 7 Flowchart of process 800 for operating vehicle 110 based on high-resolution estimated three-DoF attitude determined by satellite image-guided attitude refinement system 500 is described. Process 800 may be implemented by computing device 115 included in vehicle 110. Process 800 includes multiple blocks that may be performed in the order shown. Alternatively or additionally, process 800 may include fewer blocks, or may include blocks that are performed in a different order.

[0085] Process 800 begins at block 802, where a computing device 115 in a vehicle 110 acquires one or more ground-view images 405 from one or more cameras included in the vehicle 110 and acquires satellite images 403 by downloading via a network 130 or retrieving from a memory included in the computing device 115. An estimated three-DoF pose 302 of the vehicle 110 is determined based on the data acquired by the vehicle sensors 116.

[0086] At block 804, computing device 115 performs the following operations by utilizing FIG. 4A to FIG. 6The depicted satellite image guided pose refinement system 500 processes one or more ground view images 405 and satellite images 403 to enhance the estimated three DoF pose 302 into a high resolution estimated three DoF pose.

[0087] At box 806, the computing device 115 uses the high-definition estimated three-DoF pose to determine a vehicle path for the vehicle 110. The vehicle can operate on the road based on the vehicle path by determining commands that direct the vehicle's propulsion (e.g., powertrain), braking, and steering subsystems to operate the vehicle so as to travel along the path. The vehicle path is generally a polynomial function on which the vehicle 110 can operate. Sometimes referred to as a path polynomial, the polynomial function can specify the vehicle position (e.g., in terms of x, y, and z coordinates) and / or pose (e.g., roll, pitch, and yaw) over time. That is, the path polynomial can be a polynomial function of three or less degrees that describes the movement of the vehicle on the ground. The movement of the vehicle on the road is described by a multi-dimensional state vector, which includes the vehicle position, orientation, velocity, and acceleration. Specifically, the vehicle motion vector may include position in x, v, z, yaw, pitch, roll, yaw rate, pitch rate, roll rate, heading velocity, and heading acceleration, which may be determined, for example, by fitting a polynomial function to a continuous 2D position relative to the ground including the vehicle motion vector. In addition, for example, the path polynomial p(x) is a model that predicts the path as a line described by a polynomial equation. The path polynomial p(x) predicts a predetermined upcoming distance x (e.g., measured in meters) of the path by determining a lateral coordinate p:

[0088] p(x)=a0+a1x+a2x 2 +a3x 3 (6)

[0089] Where a0 is the offset, eg, the lateral distance between the path and the centerline of the vehicle 110 at the upcoming distance x, a1 is the heading angle of the path, a2 is the curvature of the path, and a3 is the rate of change of the curvature of the path.

[0090] The polynomial function may be used to guide the vehicle 110 from a current position indicated by the high-definition estimated three-DoF pose to another position in the environment surrounding the vehicle while maintaining minimum and maximum limits on lateral and longitudinal accelerations. The vehicle 110 may be operated along the vehicle path by transmitting commands to the subsystems 112, 113, 114 to control vehicle propulsion, steering, and braking. After block 806, the process 800 ends.

[0091] Computing devices such as those described herein typically each include commands that can be executed by one or more computing devices such as those identified above and used to implement the blocks or steps of the processes described above. For example, the process blocks described above can be embodied as computer executable commands.

[0092] The computer executable instructions may be compiled or interpreted by a computer program created using a variety of programming languages ​​and / or techniques, including but not limited to the following in single or combined form: Java TM , C, C++, Python, Julia, SCALA, Visual Basic, Java Script, Perl, HTML, etc. Typically, a processor (e.g., a microprocessor) receives commands, such as from a memory, a computer-readable medium, etc., and executes these commands, thereby performing one or more processes including one or more of the processes described herein. A variety of computer-readable media can be used to store such commands and other data in files and to transmit such commands and other data. Files in a computing device are typically collections of data stored on computer-readable media such as storage media, random access memory, etc.

[0093] Computer-readable media (also referred to as processor-readable media) include any non-transitory (i.e., tangible) media that participate in providing data (i.e., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media can take many forms, including, but not limited to, non-volatile media and volatile media. Instructions can be transmitted via one or more transmission media, including optical fibers, wires, wireless communications, including internals that make up a system bus coupled to a processor of a computer. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, FLASH-EEPROM, any other memory chip or cassette, or any other medium from which a computer can read.

[0094] Unless otherwise expressly indicated herein, all terms used in the claims are intended to be given their ordinary and customary meanings as understood by those skilled in the art. Specifically, unless a claim recites an express limitation to the contrary, the use of singular articles such as "a," "an," "the," and "said" should be interpreted as reciting one or more of the indicated elements.

[0095] The term "exemplary" is used herein in the sense of referring to an example; for example, reference to "exemplary widget" should be interpreted to refer merely to an example of a widget.

[0096] The adverb “approximately” modifying a value or result means that the shape, structure, measurement, value, determination, calculation, etc. may vary from the exactly described geometry, distance, measurement, value, determination, calculation, etc. due to imperfections in materials, machining, manufacturing, sensor measurement, calculation, processing time, communication time, etc.

[0097] In the accompanying drawings, the same candidate mark indicates the same element. In addition, some or all of these elements can be changed. With respect to the media, processes, systems, methods, etc. described herein, it should be understood that although the steps or frames of such processes, etc. have been described as occurring according to a sequence of a specific order, such processes can be practiced by the described steps performed in an order other than the order described herein. It should be understood that certain steps can be performed simultaneously, other steps can be added, or certain steps described herein can be omitted. In other words, the description of the process herein is provided for the purpose of illustrating certain embodiments and should never be interpreted as limiting the claimed invention. Any use of "based on" and "in response to" herein (including reference to the media, processes, systems, methods, etc. described herein) indicates a causal relationship, not just a temporal relationship.

[0098] According to the present invention, a system is provided, which has: a computer, the computer includes a processor and a memory, the memory includes instructions, and the instructions can be executed by the processor to: use a rotation estimator to estimate the relative rotation between a ground perspective image and an aerial perspective image; project a ground feature map and a confidence map corresponding to the ground perspective image to an aerial feature map corresponding to the aerial perspective image according to the relative rotation to create a projected overhead perspective feature map; use spatial correlation to determine the translation difference between the projected overhead perspective feature map and the aerial feature map; and determine a high-definition estimated three-degree-of-freedom posture of a ground perspective camera based on the relative rotation and the translation difference.

[0099] According to an embodiment, the instructions also include instructions for determining the ground feature map and the confidence map from the ground perspective image using a first neural network and instructions for determining the aerial feature map from the aerial perspective image using a second neural network.

[0100] According to an embodiment, the instructions further include instructions for supervising the system using a combination of self-supervised learning and weak supervision.

[0101] According to an embodiment, the rotation estimator includes instructions for extracting ground features from the ground perspective image using a third neural network and extracting aerial features from the aerial perspective image using a fourth neural network.

[0102] According to an embodiment, the rotation estimator further comprises instructions for projecting the ground features onto the aerial features to create a bird's-eye view projection.

[0103] According to an embodiment, the rotation estimator further comprises instructions for estimating the relative rotation between the ground feature and the top-down perspective projection using a neural pose optimizer.

[0104] According to an embodiment, the instructions for estimating the three-degree-of-freedom pose of the ground view camera include instructions for estimating the three-degree-of-freedom pose based on an initial estimate of the three-degree-of-freedom pose of the ground view camera.

[0105] According to an embodiment, the first neural network and the second neural network may have a U-net architecture.

[0106] According to an embodiment, the instructions further include instructions for outputting the high-resolution estimated three-degree-of-freedom pose of the ground-view camera to operate a vehicle.

[0107] According to an embodiment, the invention also features a vehicle computer configured to determine a vehicle path on which to operate the vehicle based on the high-resolution estimated three-degree-of-freedom pose of the ground-view camera and the aerial-view image.

[0108] According to the present invention, a method includes: using a rotation estimator to estimate the relative rotation between a ground perspective image and an aerial perspective image; projecting a ground feature map and a confidence map corresponding to the ground perspective image to an aerial feature map corresponding to the aerial perspective image according to the relative rotation to create a projected top-down perspective feature map; using spatial correlation to determine the translation difference between the projected top-down perspective feature map and the aerial feature map; and determining a high-definition estimated three-degree-of-freedom posture of a ground perspective camera based on the relative rotation and the translation difference.

[0109] In one aspect of the invention, the method includes determining the ground feature map and the confidence map from the ground perspective image using a first neural network, and determining the aerial feature map from the aerial perspective image using a second neural network.

[0110] In one aspect of the invention, the method includes supervising the first neural network and the second neural network using a combination of self-supervised learning and weak supervision.

[0111] In one aspect of the invention, the method comprises randomly rotating and translating an aerial training image, and training the rotation estimator based on the aerial training image and the randomly rotated and translated aerial training image.

[0112] In one aspect of the invention, the method includes extracting triangular regions of the randomly rotated and translated aerial training image.

[0113] In one aspect of the invention, estimating the relative rotation includes extracting ground features from the ground perspective image using a third neural network and extracting aerial features from the aerial perspective image using a fourth neural network.

[0114] In one aspect of the invention, estimating the relative rotation includes projecting the ground features onto the aerial features to create a bird's-eye view projection.

[0115] In one aspect of the invention, estimating the relative rotation includes estimating the relative rotation between the ground feature and the top-down perspective projection using a neural pose optimizer.

[0116] In one aspect of the invention, the estimated three-degree-of-freedom pose of the ground-view camera is determined based on an initial estimate of the three-degree-of-freedom pose of the ground-view camera.

[0117] In one aspect of the invention, the method includes outputting the high-resolution estimated three-degree-of-freedom pose of the ground-view camera to operate a vehicle.

Claims

1. A system, comprising: A computer comprising a processor and a memory, the memory comprising instructions executable by the processor to: Using a rotation estimator to estimate the relative rotation between the ground view image and the aerial view image; Projecting the ground feature map and the confidence map corresponding to the ground perspective image onto the aerial feature map corresponding to the aerial perspective image according to the relative rotation to create a projected top-down perspective feature map; Determining a translation difference between the projected top-down feature map and the aerial feature map using spatial correlation; and A high-resolution estimated three-degree-of-freedom pose of a ground-view camera is determined based on the relative rotation and the translation difference.

2. The system of claim 1 , wherein the instructions further comprise instructions for determining the ground feature map and the confidence map from the ground perspective image using a first neural network and instructions for determining the aerial feature map from the aerial perspective image using a second neural network.

3. The system of claim 1, wherein the instructions further comprise instructions for supervising the system using a combination of self-supervised learning and weak supervision.

4. The system of claim 1, wherein the rotation estimator includes instructions for extracting ground features from the ground-perspective image using a third neural network and extracting aerial features from the aerial-perspective image using a fourth neural network.

5. The system of claim 4, wherein the rotation estimator further comprises instructions for projecting the ground features onto the aerial features to create a bird's-eye view projection.

6. The system of claim 5, wherein the rotation estimator further comprises instructions for estimating a relative rotation between the ground feature and the top-down perspective projection using a neural pose optimizer.

7. The system of claim 1, wherein the instructions for estimating the three-degree-of-freedom pose of the ground-view camera include instructions for estimating the three-degree-of-freedom pose based on an initial estimate of the three-degree-of-freedom pose of the ground-view camera.

8. The system of any one of claims 1 to 7, wherein the instructions further comprise instructions for outputting the high-resolution estimated three-degree-of-freedom pose of the ground-view camera to operate a vehicle.

9. A method comprising: Using a rotation estimator to estimate the relative rotation between the ground view image and the aerial view image; Projecting the ground feature map and the confidence map corresponding to the ground perspective image onto the aerial feature map corresponding to the aerial perspective image according to the relative rotation to create a projected top-down perspective feature map; Determining a translation difference between the projected top-down feature map and the aerial feature map using spatial correlation; and A high-resolution estimated three-degree-of-freedom pose of a ground-view camera is determined based on the relative rotation and the translation difference.

10. The method of claim 9 further comprises determining the ground feature map and the confidence map from the ground perspective image using a first neural network, and determining the aerial feature map from the aerial perspective image using a second neural network.

11. The method of claim 10, further comprising supervising the first neural network and the second neural network using a combination of self-supervised learning and weak supervision.

12. The method of claim 9, further comprising randomly rotating and translating an aerial training image, and training the rotation estimator based on the aerial training image and the randomly rotated and translated aerial training image. 13 . The method of claim 12 , further comprising extracting a triangular region of the randomly rotated and translated aerial training image.

14. The method of any one of claims 9 to 13, wherein estimating the relative rotation comprises extracting ground features from the ground-view image using a third neural network and extracting aerial features from the aerial-view image using a fourth neural network.

15. The method of claim 14, wherein estimating the relative rotation comprises projecting the ground features onto the aerial features to create a bird's-eye view projection.

Images