Orthographic image generation method, apparatus, and electronic device

Abstract

The application provides an orthographic image generation method, device and electronic equipment, the method comprising: performing feature extraction on a current image, obtaining key points in the current image, and determining road marker points matched with the key points in the current image; determining whether the current image is a key frame image based on the road marker points, if yes, optimizing local map information corresponding to the key frame image based on GPS error and re-projection error, and determining orthographic property of the key frame image based on the optimized local map; if the orthographic property is greater than a threshold, projecting the key frame image onto a ground plane, cutting the projected key frame image into multiple image tiles, and fusing each image tile and an orthographic value tile of each image tile to generate an orthographic image of each image tile. The application can generate a high-quality orthographic image in real time to accurately determine the position of a measurement object from the orthographic image.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus and electronic device for generating orthophotos. Background Technology

[0002] Due to the limited field of view and relatively low flight altitude of airborne drones, the real-time generation of orthophotos from drone aerial images is widely used.

[0003] Currently, orthorectified images are mostly generated using Structure from Motion (SfM) methods based on multi-view solid geometry. However, this method involves a large amount of computation, cannot be completed in real time, and has low efficiency. Alternatively, simultaneous localization and mapping (SLAM) methods are used to generate orthorectified images, but these methods lead to camera pose drift and decreased fusion quality, failing to maintain good orthorectification and resulting in low positional accuracy of measured objects in the generated orthorectified images. Summary of the Invention

[0004] This invention provides a method, apparatus, and electronic device for generating orthophotos, which addresses the deficiency of low positional accuracy of measured objects in orthophotos generated in the prior art.

[0005] This invention provides a method for generating orthophotos, comprising:

[0006] Feature extraction is performed on the current image to obtain key points in the current image, and landmark points matching the key points are determined in the current image;

[0007] Based on the landmarks, determine whether the current image is a keyframe image. If so, optimize the local map information corresponding to the keyframe image based on GPS error and reprojection error, and determine the orthorectification of the keyframe image based on the optimized local map. The local map information includes all keyframe images, all landmarks, and similarity transformation from the visual coordinate system to the geographic coordinate system on the local map.

[0008] If the orthorectivity is greater than the threshold, the keyframe image is projected onto the ground plane, and the projected keyframe image is divided into multiple image tiles. The image tiles and the orthorectified tiles of each image tile are then fused to generate an orthorectified image of each image tile.

[0009] According to an orthophoto generation method provided by the present invention, the step of determining the landmark points matching the key points in the current image includes:

[0010] Based on camera parameters and motion model, determine the current camera initial pose corresponding to the current image;

[0011] Based on the current initial pose of the camera, determine the initial landmark point in the current image that matches the key point;

[0012] The initial pose of the current camera is optimized based on the matched key points and initial landmarks. After optimization, the initial landmarks with a matching degree less than a preset matching degree with the key points are removed from the initial landmarks to obtain the landmarks.

[0013] According to the orthophoto generation method provided by the present invention, determining the initial pose of the current camera corresponding to the current image based on camera parameters and a motion model includes:

[0014] Based on the similarity transformation of the current image from the visual coordinate system to the geographic coordinate system, and the GPS information of the current image, the camera translation vector in the camera parameters is determined.

[0015] Based on the rigid body transformation representation of the motion model, the translation vectors of the current image and the previous image, and the rotation matrix of the previous image, the camera rotation matrix in the camera parameters is determined.

[0016] The initial pose of the current camera is determined based on the camera translation vector and the camera rotation matrix.

[0017] According to the orthophoto image generation method provided by the present invention, the step of determining whether the current image is a keyframe image based on the landmark points includes:

[0018] If the number of landmarks is greater than a preset number or the geographical deviation between the landmarks in the current image and the landmarks in the previous image is greater than a preset value, the current image is determined to be the keyframe image.

[0019] According to the orthophoto generation method provided by the present invention, the local map information corresponding to the keyframe image is optimized based on the following formula:

[0020]

[0021] e i,j =p i,j -π(exp(ξ i )P j )

[0022] Where N represents all keyframe images, O i ρ represents all valid keypoints in keyframe image i. H This represents the Huber function. Represents the information matrix, w g It is a parameter used to balance reprojection error and GPS error. This indicates extracting the translation component from the transformation, e i,j p represents the reprojection error. i,j P represents the keypoint corresponding to the j-th landmark in the i-th keyframe image. j Let represent the position of the j-th landmark in world coordinates, exp represent the transformation from Lie algebraic form to matrix form, and π represent the transformation from camera coordinate system to image coordinate system.

[0023] According to the orthorectified image generation method provided by the present invention, the step of fusing each image tile and the orthorectified value tile of each image tile to generate an orthorectified image of each image tile includes:

[0024] Based on each image tile, generate the corresponding Laplacian pyramid;

[0025] Generate the corresponding Gaussian pyramid based on the orthophoto tiles;

[0026] The Laplace pyramid and the Gaussian pyramid are fused layer by layer from high to low to generate orthophotos of each image tile.

[0027] According to the orthorectified image generation method provided by the present invention, the step of fusing the Laplacian pyramid and the Gaussian pyramid layer by layer from high to low to generate orthorectified images of each image tile includes:

[0028] In the case where there is an overlapping area between the image tile and the orthorectified tile, the orthorectified value of each pixel in each layer of the Laplacian pyramid and the Gaussian pyramid is the orthorectified value corresponding to the tile with the maximum orthorectified value, and the pixel value of each pixel is the image pixel corresponding to the tile with the maximum orthorectified value.

[0029] The orthorectified image is recovered from the Laplacian pyramid based on the orthorectified value of each pixel and the pixel value.

[0030] According to an orthorectified image generation method provided by the present invention, the orthorectified value tiles are used to characterize the orthorectified value of the projected keyframe image, and the orthorectified value of the projected keyframe image is determined based on the following formula:

[0031]

[0032] Among them, w d The orthorectified value of the projected keyframe image is represented by Alt, where Alt represents the altitude at the time of camera capture, θ represents the angle between the camera's observation direction and the ground normal, r represents the resolution of the orthorectified image, and D represents the resolution of the orthorectified image. mThis represents half the diagonal pixel length of the keyframe image, and D represents the pixel distance from the pixel to the center of the image.

[0033] The present invention also provides an orthophoto image generation apparatus, comprising:

[0034] The matching unit is used to extract features from the current image, obtain key points in the current image, and determine landmark points in the current image that match the key points;

[0035] An optimization unit is used to determine whether the current image is a keyframe image based on the landmark points. If so, it optimizes the local map information corresponding to the keyframe image based on GPS error and reprojection error, and determines the orthorectification of the keyframe image based on the optimized local map. The local map information includes all keyframe images, all landmark points, and similarity transformation from the visual coordinate system to the geographic coordinate system on the local map.

[0036] The generation unit is configured to project the keyframe image onto the ground plane if the orthorectivity is greater than a threshold, divide the projected keyframe image into multiple image tiles, and fuse each image tile and the orthorectified tile of each image tile to generate an orthorectified image of each image tile.

[0037] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the orthophoto generation method as described above.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the orthophoto generation method as described above.

[0039] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the orthophoto generation method as described above.

[0040] The orthophoto generation method, apparatus, and electronic device provided by this invention optimize the local map information corresponding to the keyframe image based on GPS error and reprojection error, thereby obtaining an optimized keyframe image with high robustness and accuracy. By fusing each image tile and the orthophoto value tile of each image tile, the pixel information in the image is taken into account, and a high-quality orthophoto image can be generated in real time to accurately determine the position of the measurement object from the orthophoto image. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is one of the flowcharts illustrating the orthophoto generation method provided by the present invention;

[0043] Figure 2 This is the second flowchart of the orthophoto generation method provided by the present invention;

[0044] Figure 3 This is a schematic diagram of the orthophoto image generation device provided by the present invention;

[0045] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0047] Currently, orthorectified images are typically generated using SfM-based methods. These methods involve: feature extraction and matching, image registration, sparse point cloud generation, dense point cloud and mesh generation, texture mapping, and orthorectified image generation. However, these steps involve a large amount of computation and therefore cannot be completed in real time.

[0048] In addition, some methods combine visual SLAM with image stitching techniques for real-time orthorectified image generation. However, while increasing speed, this leads to camera pose drift and a decrease in fusion quality. To overcome the poor performance of pure visual methods in areas with weak texture and darkness, attempts have been made to fuse GNSS (Global Navigation Satellite System) information and visual information to estimate camera pose. However, the combination of GNSS and visual information is not tight or effective enough to be applicable to various practical application scenarios. Furthermore, existing methods do not well preserve the orthorectification of orthorectified images, which further affects the accuracy of object position measurement and the usability of orthorectified images.

[0049] In response, the present invention provides a method for generating orthophoto images. Figure 1 This is one of the flowcharts illustrating the orthophoto generation method provided by the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0050] Step 110: Extract features from the current image, obtain key points in the current image, and determine the landmark points that match the key points in the current image.

[0051] Specifically, the current image can be understood as the original image taken by the drone, meaning that an orthophoto needs to be generated based on this image to obtain the position of the object being measured. When extracting features from the current image, ORB (oriented FAST and rotated BRIEF) can be used to extract features and obtain key points in the current image, and these key points can be identified.

[0052] Since key points are used to characterize key information in the current image, and landmarks are usually key information in the original image taken by the drone, this embodiment of the invention determines landmarks that match the key points in the current image. If there are enough landmarks in the current image, it indicates that the current image carries more key information, that is, the probability that the current image is a keyframe image is greater.

[0053] Understandably, after acquiring the keypoints in the current image, initialization can be performed using the initialization methods in ORB-SLAM (a versatile and accurate monocular SLAM system). Specifically, when initializing the similarity transformation from the visual coordinate system to the geographic coordinate system, the following formula can be used:

[0054]

[0055] Among them, T wg This represents a similarity transformation from the visual coordinate system to the geographic coordinate system, t. i G represents the position of the camera in the world coordinate system in the i-th frame. i This represents the GPS information corresponding to the camera in the i-th frame, specifically represented using UTM (Universal Transverse Mercator) coordinates. This represents the similarity transformation from the initialized visual coordinate system to the geographic coordinate system.

[0056] Step 120: Based on the landmarks, determine whether the current image is a keyframe image. If so, optimize the local map information corresponding to the keyframe image based on GPS error and reprojection error, and determine the orthorectification of the keyframe image based on the optimized local map. The local map information includes all keyframe images on the local map, all landmarks, and the similarity transformation from the visual coordinate system to the geographic coordinate system.

[0057] Specifically, GPS error is used to characterize the overall geographic information of the current image, and reprojection error is used to characterize the visual information of landmarks and key points in the current image. In this embodiment of the invention, the two are combined to optimize the local map information corresponding to the key frame image, so that the optimized local map integrates geographic information and location information, thereby enabling accurate determination of the orthorectification of the key frame image based on the optimized local map.

[0058] Optionally, when optimizing the keyframe image, the pose, landmark position, and similarity transformation from the world coordinate system to the geographic coordinate system on the corresponding local map can be updated simultaneously; the orthorectification of the keyframe image itself can be calculated by calculating the angle between the camera observation direction and the ground plane direction.

[0059] Step 130: If the orthorectivity is greater than the threshold, the keyframe image is projected onto the ground plane, and the projected keyframe image is divided into multiple image tiles. The image tiles and the orthorectified tiles of each image tile are then fused to generate the orthorectified image of each image tile.

[0060] Specifically, when the orthorectivity is greater than a threshold, it indicates that the corresponding keyframe image has good orthorectivity. At this time, the keyframe image can be projected onto the ground plane, and the projected keyframe image can be divided into multiple image tiles, such as each image tile being 256×256 in size. In addition, when expanding the local map, blank areas may be generated, which will change the mosaic parameters. In this case, blank tiles can be added to the blank areas.

[0061] After obtaining multiple image tiles, effective region tiles and orthorectified value tiles can be generated for each image tile. The effective region tiles indicate whether a pixel is a pixel in the original image, and the orthorectified value tiles represent the orthorectified value of the projected keyframe image. If tiles exist at the same location after different keyframe images are projected, the image tiles and their orthorectified value tiles are fused to obtain the orthorectified image of each image tile.

[0062] The orthophoto generation method provided in this invention optimizes the local map information corresponding to the keyframe image based on GPS error and reprojection error, thereby obtaining an optimized keyframe image with high robustness and accuracy. It also fuses each image tile and the orthophoto value tile of each image tile, taking into account the pixel information in the image, and thus can generate a high-quality orthophoto image in real time to accurately determine the position of the measurement object from the orthophoto image.

[0063] Based on the above embodiments, determining landmark points that match key points in the current image includes:

[0064] Based on camera parameters and motion model, determine the initial pose of the current camera corresponding to the current image;

[0065] Based on the current initial pose of the camera, determine the initial landmark points in the current image that match the key points;

[0066] The initial pose of the current camera is optimized based on the matched key points and initial landmarks. After optimization, the initial landmarks with a matching degree less than the preset matching degree are removed from the initial landmarks to obtain the landmarks.

[0067] Specifically, camera parameters can include camera translation vectors, camera rotation matrices, etc. The motion model is used to characterize the transformation relationship between the current image and the previous image. Based on the camera parameters and the motion model, the initial pose of the current camera can be determined.

[0068] After determining the initial pose of the current camera, landmark points matching the key points in the current image can be found by projection. For example, in this embodiment of the invention, enough landmark points matching the key points can be obtained based on the motion model of GPS or bag-of-words vectors. If the number of matches is insufficient, the key point correspondence between the current image and the previous image can be found, and the search can be accelerated by fixing the same node in one layer of the bag-of-words tree. After finding a landmark point matching the key point, since there is a one-to-one correspondence between key points and landmark points in the key frame image, the matching relationship between the key points and landmark points in the current image is obtained.

[0069] It is understandable that after determining the initial landmark points that match the key points in the current image, the current camera's initial pose can be optimized using PNP (perspective-n-point) based on the matched key points and initial landmark points, and bad matching relationships can be eliminated after optimization (such as eliminating key points and initial landmark points with a matching degree less than the preset matching degree).

[0070] Based on any of the above embodiments, determining the initial pose of the current camera corresponding to the current image, based on camera parameters and a motion model, includes:

[0071] Based on the similarity transformation of the current image from the visual coordinate system to the geographic coordinate system, and the GPS information of the current image, determine the camera translation vector in the camera parameters;

[0072] Based on the rigid body transformation representation of the motion model, the translation vectors between the current image and the previous image, and the rotation matrix of the previous image, the camera rotation matrix in the camera parameters is determined.

[0073] Based on the camera translation vector and the camera rotation matrix, the corresponding initial pose of the current camera is determined.

[0074] Specifically, the current initial pose of the camera is determined based on the following formula:

[0075] T c =(R c |t c )

[0076] Among them, R c Let t represent the camera rotation matrix. c T represents the camera translation vector. c This indicates the current initial pose of the camera.

[0077] The GPS-based motion model estimates the initial translation of the current image using GPS and the initial rotation of the current image using the relative rotation of the previous two images.

[0078] t c =T wg g c ,R c =R ( R l

[0079] Among them, R l Let g represent the rotation matrix of the previous image. c Indicates the GPS location of the current image.

[0080] Furthermore, the motion model, which represents the transformation relationship between two consecutive frames, can be represented by rigid body transformation:

[0081] T v =(R v |t v )

[0082] R v Indicates the rotational part, t v T represents the translation part. v Represents the rigid body transformation of the motion model.

[0083] Based on any of the above embodiments, determining whether the current image is a keyframe image based on landmark points includes:

[0084] If the number of landmarks exceeds the preset number or the geographical deviation between the landmarks in the current image and the landmarks in the previous image exceeds the preset value, the current image is determined to be a keyframe image.

[0085] Specifically, if the number of landmarks is greater than the preset number or the geographical deviation between the landmarks in the current image and the landmarks in the previous image is greater than the preset value, it indicates that the current image contains more key information. Therefore, the current image is used as a keyframe image so that the keyframe image can be optimized subsequently based on GPS error and reprojection error.

[0086] Understandably, after determining that the current image is a keyframe image, a match can be searched in the local map corresponding to the current image, and landmarks can be generated through triangulation. Then, redundant or failed-to-track landmarks can be filtered out.

[0087] Based on any of the above embodiments, the local map corresponding to the keyframe image is optimized using the following formula:

[0088]

[0089] e i,j =p i,j -π(exp(ξ i )P j )

[0090] Where N represents all keyframe images, O i ρ represents all valid keypoints in keyframe image i. H This represents the Huber function. Represents the information matrix, w g It is a parameter used to balance reprojection error and GPS error. This indicates extracting the translation component from the transformation, ||·||2 represents the L2 norm of the vector, and e i,j p represents the reprojection error between the i-th keyframe image and the j-th landmark. i,j P represents the keypoint corresponding to the j-th landmark in the i-th keyframe image. j Let ξ represent the position of the j-th landmark in world coordinates, exp represent the transformation from Lie algebraic form to matrix form, and π represent the transformation from camera coordinates to image coordinates. Here, the rigid body transformation representation of the camera pose is converted into the Lie algebraic form ξ, and the similarity transformation representation from world coordinates to geographic coordinates is converted into the Lie algebraic form ζ. gw .

[0091] Furthermore, the tolerance for GPS errors is controlled by a function f:

[0092]

[0093] Where, d thr The threshold for GPS error is defined as follows: when the GPS error d is less than or equal to d... thr At that time, the GPS error remained unchanged.

[0094] w g and d thr This is a hyperparameter that can be manually adjusted. g The value can be 100, d thr It can take the value 0.5.

[0095] Based on any of the above embodiments, the image tiles and their orthorectified tiles are fused to generate an orthorectified image of each image tile, including:

[0096] Based on each image tile, generate the corresponding Laplacian pyramid;

[0097] Generate the corresponding Gaussian pyramid based on the orthophoto tiles;

[0098] The Laplace Pyramid and Gauss Pyramid are merged layer by layer from high to low to generate orthophotos of each image tile.

[0099] The Laplacian pyramid and the Gaussian pyramid are fused layer by layer from high to low to generate orthophotos of each image tile, including:

[0100] In the case where there is an overlapping area between image tiles and orthorectified tiles, the orthorectified value of each pixel in each layer of the Laplacian pyramid and Gaussian pyramid is the orthorectified value corresponding to the tile with the maximum orthorectified value, and the pixel value of each pixel is the image pixel corresponding to the tile with the maximum orthorectified value.

[0101] The orthorectified image is recovered from the Laplacian pyramid based on the orthorectified value of each pixel and the pixel value.

[0102] Specifically, for image tiles, a k-layer Laplacian pyramid can be generated; for orthophoto tiles, a k-layer Gaussian pyramid can be generated, where k is an adjustable parameter, such as k=4.

[0103] After generating the corresponding pyramid, the tiles are merged from the top to the bottom. If a pixel in a certain layer of the pyramid has a larger orthophoto value than the original orthophoto value, the new orthophoto value is used to replace the original orthophoto value, and the RGB values ​​in the corresponding image tile are modified. For the merging of effective area tiles, it is only necessary to consider whether the effective area of ​​the existing mosaic needs to be expanded.

[0104] Based on any of the above embodiments, orthorectified tiles are used to characterize the orthorectified value of the projected keyframe image, and the orthorectified value of the projected keyframe image is determined based on the following formula:

[0105]

[0106] Among them, w d The orthorectified value of the keyframe image after projection is given, where Alt represents the altitude at the time of camera capture, θ represents the angle between the camera's observation direction and the ground normal, r represents the resolution of the orthorectified image, and D... m It represents half the diagonal pixel length of the keyframe image, and D represents the pixel distance from the pixel to the center of the image.

[0107] Based on any of the above embodiments, the present invention also provides a method for generating orthophoto images, such as... Figure 2 As shown, the method includes:

[0108] First, the GPS-based motion model of the georeferenced tracking module determines whether there are enough landmarks to match key points in the current image. If not, matching is performed based on bag-of-words vectors. If so, camera pose optimization is performed, and the current image is determined to be a keyframe image based on the number of landmarks or the geographical deviation between the landmarks in the current image and the landmarks in the previous image.

[0109] If the current image is a keyframe image, the keyframe image is input into the georeferenced map module. After generating and filtering landmarks on the keyframe image, the keyframe image is optimized based on GPS error and reprojection error, and the orthorectification of the keyframe image is calculated.

[0110] If the orthorectification is greater than the threshold, the keyframe image is input to the orthorectification-preserving stitching module, the keyframe image is projected onto the ground plane, and the current image is segmented into multiple image tiles. The orthorectification value is calculated pixel by pixel to determine the orthorectified tile. Then, the image tiles and the orthorectified tile are fused to obtain the orthorectified image.

[0111] Therefore, this embodiment of the invention provides a reliable initial value to the system by combining the motion model of GPS, which accelerates the convergence of subsequent nonlinear optimization and reduces the possibility of system relocation. Then, a robust optimization objective with joint GPS constraints and visual constraints is designed on the GPS-based local map, and the tolerance for GPS errors is considered. Finally, the camera shooting angle and pixel position are specifically considered to improve the quality of the real-time generated orthophoto.

[0112] The orthophoto image generation apparatus provided by the present invention will be described below. The orthophoto image generation apparatus described below can be referred to in correspondence with the orthophoto image generation method described above.

[0113] Based on any of the above embodiments, the present invention also provides an orthophoto image generation apparatus, such as... Figure 3 As shown, the device includes:

[0114] The matching unit 310 is used to extract features from the current image, obtain key points in the current image, and determine landmark points in the current image that match the key points.

[0115] The optimization unit 320 is used to determine whether the current image is a keyframe image based on the landmark points. If so, it optimizes the local map information corresponding to the keyframe image based on GPS error and reprojection error, and determines the orthorectification of the keyframe image based on the optimized local map. The local map information includes all keyframe images, all landmark points, and similarity transformation from the visual coordinate system to the geographic coordinate system on the local map.

[0116] The generation unit 330 is configured to project the keyframe image onto the ground plane if the orthorectivity is greater than a threshold, divide the projected keyframe image into multiple image tiles, and fuse each image tile and the orthorectified value tile of each image tile to generate an orthorectified image of each image tile.

[0117] Based on any of the above embodiments, the matching unit 310 includes:

[0118] The rigid body transformation determination unit is used to determine the initial pose of the current camera corresponding to the current image based on camera parameters and motion model.

[0119] A matching subunit is used to determine an initial landmark point in the current image that matches the key point based on the current initial pose of the camera.

[0120] The filtering unit is used to optimize the current camera's initial pose based on the matched key points and initial landmark points, and after optimization, remove the initial landmark points whose matching degree with the key points is less than a preset matching degree from the initial landmark points to obtain the landmark points.

[0121] Based on any of the above embodiments, the rigid body transformation determining unit includes:

[0122] The translation vector determination unit is used to determine the camera translation vector in the camera parameters based on the similarity transformation of the current image from the visual coordinate system to the geographic coordinate system and the GPS information of the current image.

[0123] The rotation matrix determination unit is used to determine the camera rotation matrix in the camera parameters based on the rigid body transformation representation of the motion model, the translation vector between the current image and the previous image, and the rotation matrix of the previous image.

[0124] The calculation unit is used to determine the current initial pose of the camera based on the camera translation vector and the camera rotation matrix.

[0125] Based on any of the above embodiments, the optimization unit 320 is used for:

[0126] If the number of landmarks is greater than a preset number or the geographical deviation between the landmarks in the current image and the landmarks in the previous image is greater than a preset value, the current image is determined to be the keyframe image.

[0127] Based on any of the above embodiments, the local map information corresponding to the keyframe image is optimized based on the following formula:

[0128]

[0129] e i,j =p i,j -π(exp(ξ i )P j )

[0130] Where N represents all keyframe images, O i ρ represents all valid keypoints in keyframe image i. H This represents the Huber function. Represents the information matrix, w g It is a parameter used to balance reprojection error and GPS error. This indicates extracting the translation component from the transformation, e i,j p represents the reprojection error. i,j P represents the keypoint corresponding to the j-th landmark in the i-th keyframe image. j Let represent the position of the j-th landmark in world coordinates, exp represent the transformation from Lie algebraic form to matrix form, and π represent the transformation from camera coordinate system to image coordinate system.

[0131] Based on any of the above embodiments, the generation unit 330 includes:

[0132] The first pyramid generation unit is used to generate the corresponding Laplacian pyramid based on each image tile;

[0133] The second pyramid generation unit is used to generate the corresponding Gaussian pyramid based on the orthophoto tiles;

[0134] The fusion unit is used to fuse the Laplacian pyramid and the Gaussian pyramid layer by layer from high to low to generate orthophotos of each image tile.

[0135] Based on any of the above embodiments, the fusion unit includes:

[0136] A pixel determination unit is used to determine the orthorectified value of each pixel in each layer of the Laplacian pyramid and the Gaussian pyramid as the orthorectified value corresponding to the tile with the maximum orthorectified value, and the pixel value of each pixel as the image pixel corresponding to the tile with the maximum orthorectified value, when there is an overlapping area between the image tile and the orthorectified value tile.

[0137] An image restoration unit is used to recover the orthophoto image from the Laplacian pyramid based on the orthophoto value of each pixel and the pixel value.

[0138] Based on any of the above embodiments, the orthorectified tile is used to characterize the orthorectified value of the projected keyframe image, and the orthorectified value of the projected keyframe image is determined based on the following formula:

[0139]

[0140] Among them, w d The orthorectified value of the projected keyframe image is represented by Alt, where Alt represents the altitude at the time of camera capture, θ represents the angle between the camera's observation direction and the ground normal, r represents the resolution of the orthorectified image, and D represents the resolution of the orthorectified image. m This represents half the diagonal pixel length of the keyframe image, and D represents the pixel distance from the pixel to the center of the image.

[0141] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include: a processor 410, a memory 420, a communication interface 430, and a communication bus 440, wherein the processor 410, the memory 420, and the communication interface 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 420 to execute an orthorectified image generation method. The method includes: extracting features from the current image to obtain key points in the current image, and determining landmarks in the current image that match the key points; determining whether the current image is a keyframe image based on the landmarks; if so, optimizing the local map information corresponding to the keyframe image based on GPS error and reprojection error, and determining the orthorectification of the keyframe image based on the optimized local map; the local map information includes all keyframe images, all landmarks, and similarity transformation from the visual coordinate system to the geographic coordinate system on the local map; if the orthorectification is greater than a threshold, projecting the keyframe image onto the ground plane, dividing the projected keyframe image into multiple image tiles, and fusing each image tile and the orthorectified value tile of each image tile to generate an orthorectified image of each image tile.

[0142] Furthermore, the logical instructions in the aforementioned memory 420 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0143] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, when the program instructions are executed by the computer, the computer is able to execute the orthorectified image generation method provided by the above methods, the method including: extracting features from the current image, obtaining key points in the current image, and determining landmarks in the current image that match the key points; based on the landmarks, determining whether the current image is a keyframe image, if so, optimizing the local map information corresponding to the keyframe image based on GPS error and reprojection error, and determining the orthorectification of the keyframe image based on the optimized local map; the local map information including all keyframe images, all landmarks, and similarity transformation from visual coordinate system to geographic coordinate system on the local map; if the orthorectification is greater than a threshold, projecting the keyframe image onto the ground plane, dividing the projected keyframe image into multiple image tiles, and fusing each image tile and the orthorectified value tile of each image tile to generate an orthorectified image of each image tile.

[0144] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described orthorectified image generation methods. The method includes: extracting features from a current image to obtain key points in the current image, and determining landmarks in the current image that match the key points; determining, based on the landmarks, whether the current image is a keyframe image; if so, optimizing the local map information corresponding to the keyframe image based on GPS error and reprojection error, and determining the orthorectification of the keyframe image based on the optimized local map; the local map information includes all keyframe images, all landmarks, and a similarity transformation from the visual coordinate system to the geographic coordinate system on the local map; if the orthorectification is greater than a threshold, projecting the keyframe image onto a ground plane, dividing the projected keyframe image into multiple image tiles, and fusing each image tile and its orthorectified value tile to generate an orthorectified image of each image tile.

[0145] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0146] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating orthophotos, characterized in that, include: Feature extraction is performed on the current image to obtain key points in the current image, and landmark points matching the key points are determined in the current image; Based on the landmarks, determine whether the current image is a keyframe image. If so, optimize the local map information corresponding to the keyframe image based on GPS error and reprojection error, and determine the orthorectification of the keyframe image based on the optimized local map. The local map information includes all keyframe images, all landmarks, and similarity transformation from the visual coordinate system to the geographic coordinate system on the local map. If the orthorectivity is greater than the threshold, the keyframe image is projected onto the ground plane, and the projected keyframe image is divided into multiple image tiles. The image tiles and the orthorectified tiles of each image tile are then fused to generate the orthorectified image of each image tile. The local map information corresponding to the keyframe image is optimized based on the following formula: ； ； in, Represents all keyframe images. Represents keyframe images All the key points that are effective in the middle, This represents the Huber function. Represents the information matrix. It is a parameter used to balance reprojection error and GPS error. This indicates extracting the translation component from the transformation. This indicates the reprojection error. Indicates the first The first keyframe image The key points corresponding to each road sign point Indicates the first The location of each landmark in world coordinates This represents the transformation from Lie algebraic form to matrix form. This represents the transformation from the camera coordinate system to the image coordinate system; function Controlling tolerance for GPS errors: ； in, Indicates GPS error. The threshold representing GPS error.

2. The orthophoto generation method according to claim 1, characterized in that, The step of determining the landmark point that matches the key point in the current image includes: Based on camera parameters and motion model, determine the current camera initial pose corresponding to the current image; Based on the current initial pose of the camera, determine the initial landmark point in the current image that matches the key point; The initial pose of the current camera is optimized based on the matched key points and initial landmarks. After optimization, the initial landmarks with a matching degree less than a preset matching degree with the key points are removed from the initial landmarks to obtain the landmarks.

3. The orthophoto generation method according to claim 2, characterized in that, Determining the initial pose of the current camera corresponding to the current image based on camera parameters and motion model includes: Based on the similarity transformation of the current image from the visual coordinate system to the geographic coordinate system, and the GPS information of the current image, the camera translation vector in the camera parameters is determined. Based on the rigid body transformation representation of the motion model, the translation vectors of the current image and the previous image, and the rotation matrix of the previous image, the camera rotation matrix in the camera parameters is determined. The initial pose of the current camera is determined based on the camera translation vector and the camera rotation matrix.

4. The orthophoto generation method according to any one of claims 1 to 3, characterized in that, Determining whether the current image is a keyframe image based on the landmark points includes: If the number of landmarks is greater than a preset number or the geographical deviation between the landmarks in the current image and the landmarks in the previous image is greater than a preset value, the current image is determined to be the keyframe image.

5. The orthophoto generation method according to any one of claims 1 to 3, characterized in that, The process of fusing each image tile and its orthorectified value tile to generate an orthorectified image of each image tile includes: Based on each image tile, generate the corresponding Laplacian pyramid; Generate the corresponding Gaussian pyramid based on the orthophoto tiles; The Laplace pyramid and the Gaussian pyramid are fused layer by layer from high to low to generate orthophotos of each image tile.

6. The orthophoto generation method according to claim 5, characterized in that, The process of fusing the Laplacian pyramid and the Gaussian pyramid layer by layer from high to low to generate orthophotos of each image tile includes: In the case where there is an overlapping area between the image tile and the orthorectified tile, the orthorectified value of each pixel in each layer of the Laplacian pyramid and the Gaussian pyramid is the orthorectified value corresponding to the tile with the maximum orthorectified value, and the pixel value of each pixel is the image pixel corresponding to the tile with the maximum orthorectified value. The orthorectified image is recovered from the Laplacian pyramid based on the orthorectified value of each pixel and the pixel value.

7. The orthophoto generation method according to any one of claims 1 to 3, characterized in that, The orthorectified tiles are used to characterize the orthorectified value of the projected keyframe image, and the orthorectified value of the projected keyframe image is determined based on the following formula: ； in, This represents the orthorectified value of the projected keyframe image. This indicates the altitude at which the camera took the picture. This indicates the angle between the camera's observation direction and the ground normal. This indicates the resolution of the orthophoto. This represents half the diagonal pixel length of the keyframe image. This represents the pixel distance from the center of the image.

8. An orthophoto image generation apparatus, characterized in that, include: The matching unit is used to extract features from the current image, obtain key points in the current image, and determine landmark points in the current image that match the key points; An optimization unit is used to determine whether the current image is a keyframe image based on the landmark points. If so, it optimizes the local map information corresponding to the keyframe image based on GPS error and reprojection error, and determines the orthorectification of the keyframe image based on the optimized local map. The local map information includes all keyframe images, all landmark points, and similarity transformation from the visual coordinate system to the geographic coordinate system on the local map. The generation unit is configured to project the keyframe image onto the ground plane if the orthorectivity is greater than a threshold, divide the projected keyframe image into multiple image tiles, and fuse each image tile and the orthorectified value tile of each image tile to generate an orthorectified image of each image tile. The local map information corresponding to the keyframe image is optimized based on the following formula: ； ； in, Represents all keyframe images. Represents keyframe images All the key points that are effective in the middle, This represents the Huber function. Represents the information matrix. It is a parameter used to balance reprojection error and GPS error. This indicates extracting the translation component from the transformation. This indicates the reprojection error. Indicates the first The first keyframe image The key points corresponding to each road sign point Indicates the first The location of each landmark in world coordinates This represents the transformation from Lie algebraic form to matrix form. This represents the transformation from the camera coordinate system to the image coordinate system; function Controlling tolerance for GPS errors: ； in, Indicates GPS error. The threshold representing GPS error.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the orthophoto generation method as described in any one of claims 1 to 7.

Images