A positioning method based on air-ground cross-view visual odometry

Abstract

The application discloses a positioning method based on ground-air cross visual angle visual odometer, and the positioning precision is judged by judging the mathematical features of the probability graph output by the ground-air visual angle positioning network, which is used as the condition for whether to fuse with the internal odometer, so that the inaccurate ground-air teaching positioning result can be removed, the positioning result with higher precision is fused with the internal odometer, and higher precision can be obtained; the ground panoramic image and the off-line acquired air visual angle image are matched and positioned, and the global positioning information can be provided under the condition that the GNSS signal is interfered; the ground-air cross visual angle positioning and the odometer fusion are used, the cumulative error caused by the long-term operation of the internal odometer is eliminated, and the positioning precision is improved.

Description

Technical Field

[0001] This invention belongs to the field of simultaneous positioning and mapping positioning technology, specifically relating to a positioning method based on visual odometry from a ground-to-air cross-view perspective. Background Technology

[0002] Road extraction aims to extract road networks from aerial images. Some methods detect roads using road features such as lines, edges, and textures. However, roads do not always possess general characteristics due to lighting, shadows, and occlusion. Human-described features are often insufficient to fully describe roads. With the rapid development of neural network technology, an increasing number of studies utilize neural networks for feature extraction.

[0003] Ground-to-air cross-view image metric localization treats the task of determining the position of a ground-view image within an aerial-view image as an image regression problem. Generally, the approximate location of a vehicle can be provided by other devices, such as an IMU or odometer; the vehicle only needs a single ground-view image to obtain its accurate location within a small geographical area. Therefore, determining the position of a ground-view image within an aerial-view image can be viewed as an image regression problem: the approximate location of the aerial-view image can be determined by other, less precise devices, while the panoramic image can be provided by the vehicle's panoramic camera. Then, a neural network is used to predict the location of the ground-view image from the aerial-view image.

[0004] Visual odometry (VOM) on ground-based unmanned platforms uses onboard cameras, along with other sensors such as IMUs and wheel speedometers, to observe the surrounding environment and determine the platform's position. While VOM is typically accurate within a small area, measurement errors accumulate as the platform moves, limiting its long-term usability. Furthermore, VOM is prone to failure in highly dynamic scenarios, making it impossible to continue estimating pose. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a positioning method based on visual odometry with ground-to-air cross-view, which can improve positioning accuracy.

[0006] A positioning method based on visual odometry from a ground-to-air cross-viewpoint includes:

[0007] Step S1: Obtain aerial view images, ground view images and their corresponding road network ground truth values, and establish an image training set;

[0008] Step S2: Provide a ground view image and an aerial view image at the location of the ground view image, as well as a road network image corresponding to the aerial view image, and input them into the ground-air intersection view positioning network. Establish a loss function and backpropagate to optimize the network parameters to achieve training.

[0009] The ground-air cross-perspective localization network includes a twin network, a cross-excitation module, a feature map fusion module, a probabilistic map estimation module, and a road extraction auxiliary training module.

[0010] A twin network is two networks with the same structure, divided into an aerial view twin network branch and a ground view twin network branch; both branches of the twin network use the ResNet34 network.

[0011] The aerial view image Siamese network branch takes an aerial view image as its input image; ResNet34 has 4 layers: the first layer has 3 residual blocks and outputs feature map a. 11 The second layer has 3 residual blocks, and the output feature map is a. 21 The third layer has 6 residual blocks, and the output feature map is a. 31 The fourth layer has 3 residual blocks, outputting an aerial feature map a. 41 Feature map a, output from layer 2 of the twin network branch for aerial view images. 21 The data is fed into the road extraction auxiliary training module.

[0012] The ground-view image Siamese network branch uses the same structure, with the input being a panoramic ground image; the output feature map of each layer is a. 12 a 22 a 32 a 42 ;

[0013] In the cross-excitation module, the aerial feature map a output via the twin network backbone 41 and ground feature map a 42 After passing through the feature activation layer, the air activation vector b1 and the ground activation vector b2 are obtained; then the air activation vector b1 is compared with the ground feature map a. 42 Multiply by channel, combining the ground excitation vector b2 with the aerial feature map a. 41 Multiplying by the channel yields the excitation-induced aerial feature map c1 and the excitation-induced ground feature map c2, respectively.

[0014] In the feature map fusion module, the excited aerial feature map c1 is sequentially passed through a 3×3 convolution, a normalization layer, and an activation function to obtain the convolutional aerial feature map d1; the excited ground feature map c2 is sequentially passed through a 3×3 convolution and an activation function to obtain the convolutional ground feature map d2; the convolutional aerial feature map d1 and the convolutional ground feature map d2 are expanded into a 256×1 matrix according to each channel; each vector is multiplied by matrix according to the corresponding channel to obtain the fused feature map e;

[0015] The fused feature map e is first passed through a 3×3 convolutional block with a stride of 2 to obtain the feature map f.

[0016] In the probabilistic map estimation module, feature map f is processed by a deconvolutional network to obtain probabilistic map g0. Specifically, the deconvolutional network consists of three deconvolutional blocks; each deconvolutional block is formed by connecting a 1×1 convolutional block, a deconvolutional block, and another 1×1 convolutional block; feature map f is processed by the feature map a output from the second layer residual block of the twin network branch of the aerial view image. 21 The concatenation is performed, and after passing through the first deconvolution module, feature map g2 is obtained; feature map g2 is then combined with feature map a output from the first layer residual block of the twin network branch of the aerial view image. 11 The concatenation is performed, and after passing through the second deconvolution module, the feature map g1 is obtained; the feature map g1 is then passed through the third deconvolution module to obtain the probability map g0.

[0017] The road extraction auxiliary training module consists of a deconvolution module, which uses the feature map f and the backbone network of the aerial image twin network to output a. 21 Channel-wise concatenation is performed to obtain feature map h3; three deconvolution blocks are used to upsample feature map h3 to feature map h1 with a size of 32×512×512; finally, a 3×3 convolutional layer and sigmoid activation function are used to obtain the road image mask image h0 from the aerial viewpoint.

[0018] Step S3: In a localization task where the initial pose is known, the unmanned platform performs localization using its internal odometry. During operation, it acquires a panoramic image of the estimated current position of the ground platform using a fisheye camera or a surround-view camera, thus obtaining a ground-view image. Based on the pose estimated by the odometry at the previous moment, it obtains an aerial-view image.

[0019] Step S4: Input the aerial view image and the ground view image into the trained ground-air cross-view localization network to obtain the estimated probability map g0.

[0020] Step S5: Calculate the mathematical characteristics of probability map g0, and determine whether to use ground-to-air cross-view correction of internal odometer based on its mathematical characteristics. If so, proceed to step 6.

[0021] Step S6: Perform joint optimization of the ground-to-air cross-view positioning and the pose map of the internal odometry to obtain the positioning result.

[0022] Preferably, the loss function is:

[0023] loss = L headmap + BCE ;

[0024] in:

[0025]

[0026] in, The value of each pixel in the probability map \(g_0\) representing the network output, \(x\) represents the value of each pixel in the ground truth map corresponding to the probability map \(g_0\), \(N\) represents the number of pixels in the probability map, and \(\alpha,\beta\) are parameters;

[0027]

[0028] where \(y\) represents the ground truth of each pixel in the road image mask image \(h_0\) respectively, represent the predicted value of each pixel in the whole image respectively, and \(N\) represents the number of pixels in the probability map.

[0029] Preferably, in step S5, let the matrix representation of the probability map \(g_0\) be \(H\), its height be \(h\), and its width be \(w\). For any point \((x,y)\in H\), there is \(0\leq H(x,y)\leq1\); let the sum of the pixel values of the probability map \(g_0\) be

[0030] The mean value of the probability map \(g_0\) in the \(x\) and \(y\) directions is:

[0031]

[0032]

[0033] The covariance matrix \(S\) of the probability map \(g_0\) is:

[0034]

[0035] Let \(P(x,y)\) be a normal distribution with mean and variance \(S\), and \(Q(x,y)\) be the distribution after normalizing \(H(x,y)\).

[0036] \(Q(x,y)=H(x,y) / M\), and the KL divergence is used to calculate the similarity between \(Q(x,y)\) and the normal distribution:

[0037]

[0038] Denote \(\det|S|\) as the determinant of \(S\). When \(kl < k1\) and \(\det|S| < k2\) and then, it is considered that the estimated position is accurate and the ground-air cross-positioning result is credible; define the coefficient for whether to perform ground-air cross-view positioning as:

[0039]

[0040] where when \(c(k)=1\), the ground-air cross-view is used to correct the internal odometer; when \(c(k)=2\), the ground-air cross-view is not used to correct the internal odometer; \(k1,k2,k3\) are set thresholds.

[0041] Ideally, k1 = 60000, k2 = 50000, and k3 = 0.5.

[0042] Preferably, in step S6, if the ground-to-air cross-location result is reliable, the pixel coordinates of the maximum value of probability map g0 are... The position of the ground-view camera in the aerial image; this is converted into the position of the panoramic camera in the global coordinate system after cross-positioning. Assuming the height of the panoramic camera and the height of the internal odometer are the same, both are... The pose estimation of the panoramic camera in the k-th frame is then expressed as:

[0043]

[0044] For the ground-to-air cross-view positioning results of the panoramic camera This represents the pose of the panoramic camera in the world coordinate system obtained by performing ground-to-air cross-matching on the current frame. For the next ground-to-air cross-matching, m frames apart from the current ground-to-air cross-matching, the value to be optimized is... Regarding the measurement results of the internal odometer, This represents the pose of frame k+i+1 relative to frame k+i. Since the panoramic camera and the internal visual odometry are rigidly connected, the optimized change in the panoramic camera's position and the change in the internal odometry's pose should be equal. Therefore, the optimization objective is:

[0045]

[0046] After each selected ground-to-air cross-view matching and localization, nonlinear optimization methods such as the Levenberg-Marquardt algorithm are used to minimize e, thus obtaining the optimal solution of the above equation and the optimized pose for each frame. That is, the optimized positioning result is obtained after ground-air cross-view matching;

[0047] Let [tq] denote pose, where t is translation and q is the quaternion corresponding to rotation. For the translation variable, the position in the world coordinate system obtained by the internal odometry is... The position of the next frame in the world coordinate system is and If we consider the variable to be optimized, then in the world coordinate system, the translation of the next frame relative to the current frame is:

[0048]

[0049] set up Let C be the rotation quaternion of the current frame relative to the world coordinate system. k In the coordinate system, the next frame C k+1 The translation is:

[0050]

[0051] Let the position change of the next frame relative to the current frame obtained by the internal odometer be... The positional residual for:

[0052]

[0053] Here t var The variance representing the internal odometer translation is taken as 0.1;

[0054] For the rotation variable, the rotation quaternion of the current frame in the world coordinate system is: The rotation quaternion in the world coordinate system for the next frame is: and For the variable to be optimized, in the current frame C k In the coordinate system, the next frame C k+1 The rotation is:

[0055]

[0056] Let the rotation change of the next frame relative to the current frame obtained by the internal odometer be... The residual of the rotation is for:

[0057]

[0058] Here q var The variance representing the rotation of the map optimization module;

[0059] For ground-to-air cross-view positioning, use This represents the position of the panoramic camera in the world coordinate system obtained by performing ground-to-air cross-matching on the current frame. This represents the location variable to be optimized. For the next ground-to-air cross-location, m frames after the current ground-to-air cross-location, its observed value is... The value to be optimized is Since the panoramic camera and the internal odometer are rigidly connected, the positional changes of the optimized panoramic camera and the internal odometer should be equal. Therefore:

[0060]

[0061] The ground-to-air cross-positioning error in the kth frame is:

[0062]

[0063] Here o vaqr The variance representing the positioning at the intersection of ground and air perspectives;

[0064] Frames within w frames of the current frame k are selected for optimization; the final error to be optimized is the sum of all errors, where [·;·;·] represents column-wise concatenation of vectors:

[0065]

[0066] After each selected ground-to-air cross-view matching and positioning, nonlinear optimization methods such as the Levenberg-Marquardt algorithm are used to minimize e, thus obtaining the optimal solution of the above equation and the optimized translation amount for each frame. and rotation amount This yields the optimized positioning result of the internal odometer after ground-to-air cross-view matching.

[0067] The present invention has the following beneficial effects:

[0068] This invention determines positioning accuracy by analyzing the mathematical characteristics of the probability map output by the ground-to-air perspective positioning network. This serves as a condition for fusion with the internal odometry system. Inaccurate ground-to-air positioning results can be discarded, while higher-accuracy results are fused with the internal odometry to achieve higher precision. This invention uses ground panoramic images and offline acquired aerial perspective images for matching positioning, providing global positioning information even under conditions of GNSS signal interference.

[0069] This invention uses ground-air cross-view positioning and odometer fusion to eliminate the cumulative error caused by long-term operation of the internal odometer and improve positioning accuracy. Attached Figure Description

[0070] Figure 1 This is a flowchart of the visual odometry positioning method based on the ground-air cross-view of the present invention;

[0071] Figure 2 This is a network structure diagram of the ground-air cross-view image localization method of the present invention;

[0072] Figure 3 This is a factor map of the fusion of ground-to-air cross-view images and visual odometry in this invention. Detailed Implementation

[0073] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0074] This invention provides a positioning method based on visual odometry from a ground-to-air cross-view perspective, comprising the following steps:

[0075] Step S1: Obtain aerial view images, ground view images, and their corresponding road network ground truth values ​​to establish the image training set for this method;

[0076] Step S2: Provide a ground view image and an aerial view image at the location of the ground view image, as well as a road network image corresponding to the aerial view image, and input them together into the ground-air cross-view localization network. Establish a loss function and backpropagate to optimize the network parameters to achieve training. The ground-air cross-view localization network includes a Siamese network, a cross-activation module, a feature map fusion module, a probability map estimation module, and a road extraction auxiliary training module.

[0077] The inputs to the ground-to-air cross-view localization network are given by [variable name]. A twin network refers to two networks with identical structures, divided into an aerial view twin network branch and a ground view twin network branch. Both branches of the twin network use ResNet34.

[0078] The aerial view image Siamese network branch takes an aerial view image as input, with a size of 3×512×512. The ResNet34 has four layers: the first layer has three residual blocks, and the output feature map is a. 11 The size is 256×128×128; the second layer has 3 residual blocks, and the output feature map is a. 21 The size is 128×64×64; the third layer has 6 residual blocks, and the output feature map is a. 31 The size is 256×32×32; the 4th layer has 3 residual blocks, and the output is an aerial feature map a. 41 The size is 512×16×16. Feature map a, output from layer 2 of the Siamese network branch of the aerial view image. 21 The data is fed into the road extraction auxiliary training module.

[0079] The ground-view image twin network branch uses the same structure, with the input being a ground panoramic image of size 3×1024×512. The output of each layer is called a. 12 a 22 a 32 a 42 Output ground feature map a 42 It has a size of 512×32×16 channels.

[0080] Following the Siamese network is the cross-excitation module. The aerial feature map a output by the Siamese network backbone... 41 and ground feature map a 42 After passing through the feature activation layer, the aerial activation vector b1 and the ground activation vector b2 are obtained. The feature activation layer consists of a 1×1 convolution, a ReLU activation function, a batch normalization layer, a fully connected layer, and a sigmoid activation function connected sequentially. Then, the aerial activation vector b1 is combined with the ground feature map a. 42 Multiply by channel, combining the ground excitation vector b2 with the aerial feature map a. 41Multiplying by the channels yields the excitation-induced aerial feature map c1 and the excitation-induced ground feature map c2, with sizes of 32×16 and 16×16 respectively, both containing 512 channels. See the following formula:

[0081] c1 = 41 ·Excite(b2)

[0082] c2 = 42 ·Excite(b1)

[0083] Excite(x) = × Conv(x) +

[0084] Where Excite(·) is the feature activation layer, W,b are the parameters of the fully connected layer, Conv(·) is a convolutional block composed of convolution, normalization layer and activation function, · is the corresponding element-wise multiplication by channel, and × is the matrix multiplication.

[0085] In the feature map fusion module, the activated aerial feature map c1 is subjected to a 3×3 convolution, a normalization layer, and an activation function. The horizontal stride of the convolution is 2, and the vertical stride is 1, resulting in the convolved aerial feature map d1 with a size of 256×16×16. The activated ground feature map c2 is subjected to a 3×3 convolution and an activation function, with both the horizontal and vertical strides being 1, resulting in the convolved ground feature map d2 with a size of 256×16×16. The convolved aerial feature map d1 and the convolved ground feature map d2 are then expanded into a 256×1 matrix for each channel, resulting in a total of 256 channels. Each vector is then multiplied by its corresponding channel to obtain the fused feature map e, with a size of 256×256 and 256 channels. The above process can be represented by the following formula:

[0086] d1 = onv(c1)

[0087] d2 = onv(c2)

[0088]

[0089] Here, Conv represents a convolutional block composed of convolutional layers, normalization layers, and activation functions, and × represents multiplication by the corresponding channel matrix.

[0090] The fused feature map e is first processed by a 3×3 convolutional block with a stride of 2 to obtain feature map f, which has a size of 256×64×64.

[0091] f = Conv()

[0092] In the probabilistic map estimation module, feature map f is processed by a deconvolutional network to obtain probabilistic map g0, with a size of 1×512×512. Specifically, the deconvolutional network consists of three deconvolutional blocks. Each deconvolutional block is formed by connecting a 1×1 convolutional block, a deconvolutional block, and another 1×1 convolutional block. Feature map f is compared with feature map a output from the second layer residual block of the twin network branch of the aerial view image. 21 The concatenation is performed, and after passing through the first deconvolution module, feature map g2 is obtained; feature map g2 is then combined with feature map a output from the first layer residual block of the twin network branch of the aerial view image. 11 The concatenation is followed by a second deconvolution module to obtain feature map g1; feature map g1 is then followed by a third deconvolution module to obtain probability map g0. The above process is represented as follows:

[0093] g2 = Deconv(Concat(f,a) 21 ))

[0094] g1 = Deconv(Concat(f,a) 11 ))

[0095] g0 = Deconv(g1)

[0096] Deconv is a deconvolution block, and Concat is an image concatenation by channel.

[0097] The road extraction auxiliary training module consists of deconvolution modules, which receive feature maps (a) from the second layer output of the aerial view image from the Siamese network. 21 The output is an aerial view image, a road view image, and a mask image h0. To avoid the network learning only road features, the road extraction auxiliary training module uses feature map f and the backbone network output a of the aerial image twin network. 21 Channel-wise concatenation is performed to obtain feature map h3. This invention uses three deconvolutional blocks to upsample feature map h3 to a feature map h1 of size 32×512×512. Finally, this invention uses a 3×3 convolutional layer and a sigmoid activation function to obtain road semantics h0, with an output shape of 1×512×512 and a data range of 0 to 1.

[0098] h3 = Concat(f, a 21 ))

[0099] h1=Deconv(Deconv(Deconv(h3)))

[0100] h0 = sigmoid(Conv(h1))

[0101] The formula for calculating the probabilistic graphical loss function g0 is:

[0102]

[0103] in, represents the value of each pixel in the probability map g0 output by the network, x represents the value of each pixel in the ground truth map corresponding to the probability map g0, N represents the number of pixels in the probability map, and α and β are parameters that can be 0.5.

[0104] To train the network to detect roads, the loss function used in this invention is the binary cross-entropy (BCE) loss:

[0105]

[0106] y represents the ground truth value of each pixel in the road image mask image h0. These represent the predicted value of each pixel in the entire image, and N represents the number of pixels in the probability map.

[0107] The final loss function is as follows:

[0108] loss = L headmap +L BCE .

[0109] Step S3: In a localization task where the initial pose is known, the unmanned platform performs localization using its internal odometry. During operation, it obtains a panoramic image of the current position estimated by the ground platform using a fisheye camera or a surround-view camera. Based on the pose estimated by the odometry at the previous moment, it obtains an aerial view image.

[0110] In step S3, the initial pose of the unmanned platform is known. The platform observes the environment and performs positioning using its internal odometry. This internal odometry uses cameras, inertial measurement units, and wheel speedometers as sensors. This positioning method is relatively accurate for short operating distances, but due to errors in the initial pose and sensor measurements, the error increases with the operating distance. During operation, the unmanned platform can obtain a panoramic image of its current position using a fisheye or surround-view camera, and align the center of the panoramic image with north using the yaw angle estimated by the platform's internal odometry. Before operation, the unmanned platform stores aerial view images of the operating area. During operation, it can obtain an aerial view image of its current approximate position based on the previous frame of the odometry. The specific method is as follows:

[0111] Let O be the ground-based panoramic camera coordinate system, C be the internal odometry coordinate system, and W be the world coordinate system. Assume the panoramic camera's coordinate system and the world coordinate system are aligned, and the panoramic camera's height is the same as the stereo camera's. The pose of the panoramic camera relative to the internal odometry in the k-th frame is: Before operation, the initial pose of the unmanned ground platform is known. During operation, its pose in the global coordinate system in the k-th frame is determined by the internal odometry. At this moment, the pose of the panoramic camera in the world coordinate system is:

[0112]

[0113] If the ground-based unmanned platform has already stored aerial view images with geographic coordinate information before operation, then the global coordinate pose is determined by the visual odometry based on the previous frame of the ground-air cross-view. Obtain the pixel coordinates of the panoramic camera's aerial view image. Centered on this point, capture aerial image patches. The pose is then determined based on internal odometry. Get the yaw angle of the current frame. k Rotate the current frame's panoramic image according to the yaw angle to align the panoramic image northward.

[0114] Step S4: Input the aerial view image and the ground view image into the trained ground-air cross-view positioning network to obtain the estimated probability map; In step S4, after the unmanned platform has traveled a certain distance, it inputs the aerial view image of the current approximate position and the currently acquired panoramic image into the trained ground-air cross-view positioning network to obtain the probability map g0 of the ground position in the aerial view image of the first frame.

[0115] Step S5: Calculate the mathematical characteristics of the probability map and determine whether to use the ground-to-air cross-view correction for the internal odometer based on its mathematical characteristics.

[0116] In step S5, based on the probability map g0 of the ground position in the aerial view image, the mathematical characteristics of the probability map are calculated to determine whether the pose obtained by the internal sensor can be corrected by ground-air cross-view positioning.

[0117] Let H be the matrix representation of the probabilistic graph g0, with height h and width w. For any point (x, y) ∈ H, 0 ≤ H(x, y) ≤ 1. Let the sum of the pixel values ​​of the probabilistic graph g0 be...

[0118] The mean of the probability graph g0 in both the x and y directions for:

[0119]

[0120]

[0121] The covariance matrix S of the probability graph g0 is:

[0122]

[0123] Let P(x,y) be the mean. The normal distribution with variance S, and Q(x, y) is the distribution after normalizing H(x, y), where Q(x, y) = H(x, y) / M. The KL divergence is used to calculate the similarity between Q(x, y) and the normal distribution.

[0124]

[0125] Denote det|S| as the determinant of S. When kl < k1 and det|S| < k2 and (where k1, k2, k3 are thresholds, and k1 = 60000, k2 = 50000, k3 = 0.5 can be taken), it is considered that the fitting effect of the distribution is good, the estimated position is accurate, and the ground-air cross-positioning result is credible. Define the coefficient for whether to perform ground-air cross-view positioning as:

[0126]

[0127] Step S6: Jointly optimize the ground-air cross-view positioning and the pose graph of the internal odometer to obtain the positioned result.

[0128] In step S6, if the ground-air cross-positioning result is credible, take the pixel coordinates of the maximum value of the probability map g0 as the position of the ground view in the aerial image. Convert it to the position of the panoramic camera after cross-positioning in the global coordinate system as Assume that the height of the panoramic camera is the same as that of the internal odometer, both are Then the pose estimation of the panoramic camera in the k-th frame is expressed as:

[0129]

[0130] For the ground-air cross-view positioning result of the panoramic camera, represents the pose of the panoramic camera in the world coordinate system obtained by ground-air cross-matching in the current frame. For the next ground-air cross-positioning, which is m frames apart from the current ground-air cross-positioning, the value to be optimized is For the measurement result of the internal odometer, represents the pose of the (k + i + 1)-th frame relative to the (k + i)-th frame. Since the panoramic camera and the internal visual odometer are rigidly connected, the position change of the panoramic camera and the pose change of the internal odometer after optimization should be equal. Then the optimization objective is:

[0131]

[0132] After each filtered ground-air cross-view matching positioning, use nonlinear optimization methods such as the Levenberg-Marquardt algorithm to make e minimum, obtain the optimal solution of the above formula, and get the pose of each optimized frame That is, obtain the optimized positioning result after ground-air cross-view matching.

[0133] Specifically:

[0134] Let [tq] denote pose, where t is translation and q is the quaternion corresponding to rotation. For the translation variable, the position in the world coordinate system obtained by the internal odometry is... The position of the next frame in the world coordinate system is and If we consider the variable to be optimized, then in the world coordinate system, the translation of the next frame relative to the current frame is:

[0135]

[0136] set up Let C be the rotation quaternion of the current frame relative to the world coordinate system. k In the coordinate system, the next frame C k+1 The translation is:

[0137]

[0138] Let the position change of the next frame relative to the current frame obtained by the internal odometer be... The positional residual for:

[0139]

[0140] Here t var The variance representing the internal odometer translation is taken as 0.1.

[0141] For the rotation variable, the rotation quaternion of the current frame in the world coordinate system is: The rotation quaternion in the world coordinate system for the next frame is: and For the variable to be optimized, in the current frame C k In the coordinate system, the next frame C k+1 The rotation is:

[0142]

[0143] Let the rotation change of the next frame relative to the current frame obtained by the internal odometer be... The residual of the rotation is for:

[0144]

[0145] Here q var The variance representing the rotation of the map optimization module is set to 0.03.

[0146] For ground-to-air cross-view positioning, use This represents the position of the panoramic camera in the world coordinate system obtained by performing ground-to-air cross-matching on the current frame. This represents the location variable to be optimized. For the next ground-to-air cross-location, m frames after the current ground-to-air cross-location, its observed value is... The value to be optimized is Since the panoramic camera and the internal odometer are rigidly connected, the positional changes of the optimized panoramic camera and the internal odometer should be equal. Therefore:

[0147]

[0148] The ground-to-air cross-positioning error in the kth frame is:

[0149]

[0150] Here o var The variance of the ground-to-air cross-view positioning is taken as 0.01.

[0151] Optimize frames within w frames of the current frame k, where w can be 1000. The final error to be optimized is the sum of all errors, where [·;·;·] represents column-wise concatenation of vectors.

[0152]

[0153] After each selected ground-to-air cross-view matching and positioning, nonlinear optimization methods such as the Levenberg-Marquardt algorithm are used to minimize e, thus obtaining the optimal solution of the above equation and the optimized translation amount for each frame. and rotation amount This yields the optimized positioning result of the internal odometer after ground-to-air cross-view matching.

[0154] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A positioning method based on air-to-ground cross-view visual odometry, characterized in that, The method comprises the following steps: Step S1, obtaining an aerial perspective image, a ground perspective image and a corresponding road network true value, and establishing an image training set; Step S2, providing the ground perspective image and the aerial perspective image at the position of the ground perspective image, and a road network image corresponding to the aerial perspective image, inputting them into a ground-aerial cross-perspective positioning network, establishing a loss function, and optimizing the network parameters through back propagation to realize training; The ground-aerial cross-perspective positioning network comprises a twin network, a cross excitation module, a feature map fusion module, a probability map estimation module and a road extraction auxiliary training module; The twin network refers to two networks with the same structure, which are divided into an aerial perspective image twin network branch and a ground perspective image twin network branch; the two branches of the twin network both use a ResNet34 network; The aerial perspective image twin network branch inputs an image as an aerial perspective image; the ResNet34 has 4 layers: the first layer has 3 residual blocks, and outputs a feature map ; the second layer has 3 residual blocks, and outputs a feature map ; the third layer has 6 residual blocks, and outputs a feature map ; the fourth layer has 3 residual blocks, and outputs an aerial feature map ; the feature map output by the second layer of the aerial perspective image twin network branch is sent to the road extraction auxiliary training module; The ground perspective image twin network branch adopts the same structure, and the input is a ground panorama image; the output feature map of each layer is 、 、 、 ; In the cross-excitation module, the aerial feature map output via the twin network backbone and ground feature map After passing through the feature activation layer, the air activation vector is obtained. and ground excitation vector Then, the air excitation vector Ground feature map Multiply by channel to obtain the ground excitation vector aerial feature map Multiplying by the channels yields the corresponding excitation-generated aerial feature maps. and excitation-induced ground feature map ; In the feature map fusion module, the excited aerial feature map is... Passing through in sequence Convolution, normalization layer, activation function, to obtain the aerial feature map after convolution. ; the excitation ground feature map Passing through in sequence Convolution and activation functions are used to obtain the ground feature map after convolution. ; the feature map in the air after convolution Feature map of the ground after convolution Each channel is expanded into The matrix is ​​obtained by performing matrix multiplication on each vector according to its corresponding channel, resulting in the fused feature map. ; For the fused feature map First, go through the size The feature map is obtained by a convolutional block with a stride of 2. In the probabilistic graphical estimation module, the feature map After passing through a deconvolutional network, a probability graph is obtained. Specifically, the deconvolutional network consists of three deconvolutional blocks; each deconvolutional block is composed of... Convolutional blocks, deconvolution, Convolutional blocks are connected together; feature map Feature map output from the second layer residual block of the twin network branch with aerial view image The concatenation is performed, and after passing through the first deconvolution module, the feature map is obtained. Feature map Feature map output from the first layer residual block of the twin network branch for aerial view image The concatenation is followed by a second deconvolution module to obtain the feature map. Feature map The probability map is obtained after the third deconvolution module. ; The road extraction auxiliary training module is composed of a deconvolution module, uses a feature map and a branch network output of the aerial image twin network Performs channel splicing to obtain a feature map ; uses three deconvolution blocks to upsample the feature map to a feature map with a size of 32x512x512 ; finally, a 3x3 convolution layer and a sigmoid activation function are used to obtain a road image mask image of the aerial perspective image perspective ; Step S3, in a positioning task with an initial pose known, the unmanned platform is positioned through an internal odometer; in the process of running, a panoramic image of the current position estimated by the ground platform is obtained through a fisheye camera or a look-around camera, and a ground perspective image is obtained; according to the previous time pose estimated by the internal odometer, an aerial perspective image is obtained: Step S4, input the aerial perspective image and the ground perspective image into the trained ground-aerial cross-perspective positioning network to obtain an estimated probability map ; Step S5, calculating a probability map mathematical features, according to which it is judged whether to use the ground-to-air cross-view angle correction internal odometry, and if so, step 6 is executed; Step S6, jointly optimizing the ground-aerial cross-perspective positioning and the pose graph of the internal odometer to obtain a positioning result.

2. The positioning method based on air-to-ground cross-view visual odometry according to claim 1, wherein, The loss function is: ； Wherein: wherein, a probability map representing the network output the value of each pixel, a probability map the value of each pixel in the corresponding ground truth map, N represents the number of probability map pixels, is a parameter; wherein, respectively represent the true value of each pixel in the road image mask image respectively represent the predicted value of each pixel in the entire image, and N represents the number of pixels of the probability image.​ 3. The positioning method based on air-to-ground cross-view visual odometry of claim 1, wherein, In step S5, let the matrix representation of the probability map be denoted as with height and width , and let any point in the matrix have value ; let the probability map have a sum of pixel values ; Probability map In both x and y directions Is: Probability map The covariance matrix S of the probability map is: set up The mean A normal distribution with variance S. for Normalized distribution / M, calculated using KL divergence. Similarity to the normal distribution: Record The determinant is S, when And And The ground-to-air cross-location result is considered reliable; define whether to perform ground-to-air cross-angle positioning coefficient: ； wherein, = 1, use ground-air intersection angle correction for internal odometry; = 0, do not use ground-air intersection angle correction for internal odometry; is a set threshold value.

4. The positioning method based on air-to-ground intersection visual odometry of claim 3, wherein, Take .

5. The positioning method based on air-to-ground cross-view visual odometry according to claim 3, wherein, In step S6, if the ground-to-air cross location result is reliable, the probability map Maximum pixel coordinate As the position of the ground view in the aerial image; convert it to the position of the cross-located panoramic camera in the global coordinate system , assuming that the height of the panoramic camera and the height of the internal odometer are the same, both are , then the pose estimation of the panoramic camera at the first frame is represented as: For the ground-to-space cross-view positioning result of the panoramic camera, The pose of the panoramic camera in the world coordinate system obtained by ground-to-space cross matching on behalf of the current frame, for the next ground-to-space cross positioning, which is m frames away from the current ground-to-space cross positioning, the value to be optimized is ; For the measurement of the internal odometry, representing the pose of the frame relative to the frame; since the panoramic camera and the internal visual odometry are rigidly connected, the change in position of the panoramic camera and the change in pose of the internal odometry after optimization should be equal, then the optimization objective is: The specific implementation process is as follows: with denotes the pose, wherein is the translation, is the rotation corresponding quaternion, for the translation variable, the position in the world coordinate system obtained by the internal odometry is , the position in the world coordinate system of the next frame is , and are variables to be optimized, then the translation of the next frame relative to the current frame in the world coordinate system is: Let be the rotation quaternion of the current frame with respect to the world coordinate system, and let be the translation of the next frame in the current frame coordinate system. Let the change in position of the next frame relative to the current frame obtained by the internal odometry be Then the residual in position is: Here The variance representing the internal odometry translation is taken as 0.1; For the rotational variables, the rotation quaternion of the current frame in the world coordinate system is , and the rotation quaternion of the next frame in the world coordinate system is , and are the variables to be optimized, and the rotation of the next frame in the current frame coordinate system is: Let the rotation change of the next frame relative to the current frame obtained by the internal odometer be Then the residual of the rotation is is: Here a variance representative of a rotation of the map optimization module; For air-to-ground cross-view positioning, use the position of the panoramic camera in the world coordinate system obtained by air-to-ground cross matching of the current frame, represent the position variable to be optimized; for the next air-to-ground cross positioning, it is separated from the current air-to-ground cross positioning by m frames, and the observation value is , and the value to be optimized is ; since the panoramic camera and the internal odometer are rigidly connected, the position change of the panoramic camera after optimization and the position change of the internal odometer should be equal, that is: The kth frame ground-aerial cross positioning error is: Here a variance representative of a cross-eye perspective positioning; The selection of the frame within the current frame k to optimize is done by w minimizing the error between the reconstructed frame and the original frame. The last error to optimize is the sum of all errors, where The representative vectors are concatenated by columns: After each selected ground-to-air cross-view matching and positioning, the Levenberg-Marquardt nonlinear optimization algorithm is used to achieve... To find the optimal solution to the above equation, we obtain the optimized translation amount for each frame. and rotation amount This yields the optimized positioning result of the internal odometer after ground-to-air cross-view matching.

Images