Air-ground image matching method and device, equipment and storage medium

Abstract

The invention discloses an air-ground image matching method, device and equipment and a storage medium, and the method comprises the steps: carrying out the sparse reconstruction of an aerial image, obtaining a sparse point cloud model of an aerial image pose and a scene, generating the camera visual angle information of a new camera according to the sparse point cloud model, and enabling the camera visual angle information to serve as a middle visual angle of the scene, the visual angle difference between the space images and the ground images can be reduced, three-dimensional reconstruction is carried out through camera visual angle information, the rendered image of the middle visual angle after rendering is obtained, feature points corresponding to the rendered image are further removed after feature point matching is carried out on the ground image, the aerial image and the rendered image, and the image quality is improved. At the moment, a matching relationship between the ground image and the aerial image can be obtained, and an air-ground joint scene graph is generated according to the matching relationship to complete matching of the air-ground images. According to the embodiment of the invention, the homonymous connection points between the air-ground images can be automatically generated, and automatic air-ground joint model construction is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to an aerial-ground image matching method, device, equipment, and storage medium. Background Technique

[0002] The aerial-ground image matching technology is a new type of image processing technology that integrates aerial photography and ground photography methods. This technology can be used to generate high-precision and high-resolution 3D models by matching and fusing images obtained by high-altitude photography equipment carried by drones or airplanes with images captured by ground cameras. This process involves multiple key steps, including data acquisition, feature extraction, feature matching, and 3D reconstruction. Among them, 3D reconstruction is a key technology for building a real-scene 3D base. Existing technologies usually use traditional oblique photography methods to construct large-scale 3D models. However, due to the lack of perspective, only using aerial images cannot obtain information on the lower facades of buildings and occluded areas, resulting in problems such as structural missing and texture streaking at the bottom of the constructed model.

[0003] To solve the problem of perspective loss, existing technologies usually use the method of handheld shooting on the ground or shooting with a vehicle-mounted camera to supplement the information of the missing area. However, due to the large perspective difference between aerial and ground images, the description information of image feature points is quite different, and it is difficult to directly match them through feature similarity. Therefore, the method of manually adding connection points is usually adopted to construct homologous matching points for aerial-ground images. Although the method of manually adding connection points can obtain a 3D model with complete structure and clear texture, it is time-consuming and laborious, and the efficiency is not high. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide an aerial-ground image matching method, device, equipment, and storage medium, which can automatically generate homologous connection points between aerial-ground images and realize the automatic construction of an aerial-ground joint model.

[0005] To achieve the above purpose, the embodiments of the present invention provide an aerial-ground image matching method, including:

[0006] Obtain an aerial image and a ground image;

[0007] Perform sparse reconstruction on the aerial image to obtain the pose of the aerial image and the sparse point cloud model of the scene;

[0008] Generate the camera position information of the target camera according to the sparse point cloud model, and generate the camera view information of the target camera according to the camera position information;

[0009] Perform three-dimensional reconstruction based on the neural radiance field using the aerial image and the pose of the aerial image, and render a rendered image under the camera view information through the neural radiance field;

[0010] Perform feature point matching on the ground image, the aerial image, and the rendered image to obtain matching trajectories;

[0011] Remove the feature points corresponding to the rendered image in the matching trajectories to obtain updated matching trajectories, so as to generate a joint scene graph of the ground image and the aerial image according to the updated matching trajectories.

[0012] As an improvement to the above solution, the sparse reconstruction of the aerial image to obtain the pose of the aerial image and the sparse point cloud model of the scene includes:

[0013] Extract feature points from the aerial image;

[0014] Based on the similarity between the image position information of the aerial image and the extracted feature point descriptors, perform feature point matching between aerial images to obtain matching relationships;

[0015] Generate an initial scene graph according to the matching relationships;

[0016] According to the initial scene graph, perform pose positioning on the aerial image to obtain the pose of the aerial image and generate a sparse point cloud model of the scene.

[0017] As an improvement to the above solution, the generation of the camera position information of the target camera according to the sparse point cloud model includes:

[0018] Obtain the coordinate ranges of the scene and the position where the original camera is located according to the sparse point cloud model;

[0019] Perform grid division on the coordinate ranges and calculate the initial sampling point coordinates corresponding to each grid;

[0020] Use the pre-constructed signed distance field to judge the positions of each initial sampling point coordinate to screen out at least one target sampling point coordinate located on the outer surface of the scene;

[0021] Use at least one target sampling point coordinate located on the outer surface of the scene as the camera position information of the target camera.

[0022] As an improvement to the above solution, the generation of the camera view information of the target camera according to the camera position information includes:

[0023] For all initial sampling points, determine the horizontal direction angle of the target camera according to a preset horizontal interval;

[0024] Based on each horizontal direction angle, determine the vertical direction angle of the target camera according to a preset vertical interval;

[0025] Generate the camera view information of the target camera according to the horizontal direction angle, the vertical direction angle and the target sampling point coordinates.

[0026] As an improvement of the above solution, before inputting the camera view information and the aerial image pose into the three-dimensional reconstruction model based on the neural radiance field, the method further includes:

[0027] Train a three-dimensional reconstruction model based on the neural radiance field; wherein, the input of the three-dimensional reconstruction model is a scene image and its corresponding pose, and after the neural radiance field training is completed, use the three-dimensional reconstruction model to output a rendered image in a new view.

[0028] As an improvement of the above solution, the feature point matching of the ground image, the aerial image and the rendered image to obtain a matching trajectory includes:

[0029] Number all the feature points of the ground image, the aerial image and the rendered image;

[0030] When there are mutually matching feature points in the ground image, the aerial image and the rendered image, obtain the corresponding feature point numbers and construct a matching trajectory.

[0031] As an improvement of the above solution, after generating the joint scene graph of the ground image and the aerial image according to the updated matching trajectory, the method further includes:

[0032] Perform sparse reconstruction on the joint scene graph to obtain a sparse point cloud model of the air-ground joint scene.

[0033] To achieve the above object, an air-ground image matching device is further provided in an embodiment of the present invention, including:

[0034] An image data acquisition module for acquiring an aerial image and a ground image;

[0035] A sparse reconstruction module for performing sparse reconstruction on the aerial image to obtain the pose of the aerial image and a sparse point cloud model of the scene;

[0036] A camera position information generation module for generating the camera position information of the target camera according to the sparse point cloud model;

[0037] A camera view information generation module for generating the camera view information of the target camera according to the camera position information;

[0038] A rendering image generation module, configured to perform three-dimensional reconstruction based on a neural radiance field by using the aerial image and the pose of the aerial image, and render a rendering image under the camera view information through the neural radiance field;

[0039] A matching trajectory generation module, configured to perform feature point matching on the ground image, the aerial image, and the rendering image to obtain matching trajectories;

[0040] A matching trajectory update module, configured to remove the feature points corresponding to the rendering image in the matching trajectories to obtain updated matching trajectories;

[0041] A joint scene graph generation module, configured to generate a joint scene graph of the ground image and the aerial image according to the updated matching trajectories.

[0042] To achieve the above object, an embodiment of the present invention further provides a ground-air image matching device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the ground-air image matching method described in any of the above embodiments is implemented.

[0043] To achieve the above object, an embodiment of the present invention further provides a computer-readable storage medium. The computer-readable storage medium includes a stored computer program. When the computer program runs, it controls the device where the computer-readable storage medium is located to execute the ground-air image matching method described in any of the above embodiments.

[0044] Compared with the prior art, the present invention proposes a ground-air image matching method, device, equipment, and storage medium. Starting from the perspective of reducing the view angle difference between ground-air images, using the ability of a neural radiance field (Neural Radiance Fields, NRF) to generate images of new views of a scene, and rendering intermediate-layer scene images through the ability of scene grid division and the inner and outer surface judgment of a signed distance field (Signed Distance Field, SDF), an automated ground-air image joint modeling is realized. By generating intermediate view images of the scene, the present invention reduces the view angle difference between ground-air images, constructs a matching relationship between ground-air images, and at the same time, taking the intermediate view images as a bridge, realizes the joint matching of ground-air images at the feature point level, so as to achieve the joint global optimization of ground-air images and improve the accuracy of the model. In addition, the present invention has relatively low requirements for input images and has no requirements for the coordinate systems of ground-air images, as long as the two are taken in the same area, and has relatively wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flowchart of a ground-air image matching method provided by an embodiment of the present invention;

[0046] Figure 2 It is a structural block diagram of an air - ground image matching device provided by an embodiment of the present invention;

[0047] Figure 3 It is a structural block diagram of an air - ground image matching device provided by an embodiment of the present invention. Specific embodiments

[0048] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0049] See Figure 1 , Figure 1 It is a flowchart of an air - ground image matching method provided by an embodiment of the present invention. The air - ground image matching method includes:

[0050] S1. Obtain aerial images and ground images;

[0051] S2. Perform sparse reconstruction on the aerial images to obtain the pose of the aerial images and the sparse point cloud model of the scene;

[0052] S3. Generate the camera position information of the target camera according to the sparse point cloud model, and generate the camera view information of the target camera according to the camera position information;

[0053] S4. Perform three - dimensional reconstruction based on the neural radiance field using the aerial images and the pose of the aerial images, and render the rendered images under the camera view information through the neural radiance field;

[0054] S5. Perform feature point matching on the ground images, the aerial images, and the rendered images to obtain matching trajectories;

[0055] S6. Remove the feature points corresponding to the rendered images in the matching trajectories to obtain updated matching trajectories, so as to generate a joint scene graph of the ground images and the aerial images according to the updated matching trajectories.

[0056] Exemplarily, the aerial images are a general term for all images taken from an aerial perspective, and the imaging tools include but are not limited to drones, satellites, etc.; the ground images are a general term for all images taken from a ground perspective, and the imaging tools include but are not limited to cameras, mobile phones, etc. After sparse reconstruction of the aerial images, the pose of the aerial images and the sparse point cloud model of the scene can be obtained. Then, based on the sparse point cloud model, the camera view information of the target camera (new camera) can be generated. The camera view information can be used as an intermediate view of the scene, which can reduce the view difference between the aerial and ground images. In the present invention, first, the aerial images and their poses are used for three-dimensional reconstruction based on the neural radiance field (a trained neural radiance field is obtained here, which encodes the three-dimensional information of the entire scene). Then, according to the camera view information, the rendered images at this view can be rendered through the neural radiance field. Then, feature point matching is performed on the ground images, aerial images, and rendered images. Using the rendered images as an intermediate bridge, the feature point matching between the ground images and the aerial images is completed. After the matching is completed, the feature points corresponding to the rendered images are further removed. At this time, the matching relationship between the ground images and the aerial images can be obtained. According to this matching relationship, an aerial-ground joint scene graph is generated to complete the matching of the aerial and ground images. The embodiments of the present invention can automatically generate homologous connection points between the aerial and ground images and realize the automatic construction of the aerial-ground joint model.

[0057] Specifically, in step S2, the sparse reconstruction of the aerial images to obtain the pose of the aerial images and the sparse point cloud model of the scene includes:

[0058] S21. Extract feature points from the aerial images;

[0059] S22. Based on the similarity between the image position information of the aerial images and the extracted feature point descriptors, perform feature point matching between the aerial images to obtain a matching relationship;

[0060] S23. Generate an initial scene graph according to the matching relationship;

[0061] S24. According to the initial scene graph, perform pose determination and positioning on the aerial images to obtain the pose of the aerial images and generate a sparse point cloud model of the scene.

[0062] Exemplarily, the present invention needs to perform sparse reconstruction on aerial images to obtain the pose of each aerial image and simultaneously obtain a sparse point cloud model of the scene. First, feature points are extracted from all aerial images. The feature points here can be traditional handcrafted features, such as SIFT (Scale-Invariant Feature Transform) feature points, or deep learning features, such as Super point, etc. After the feature point extraction is completed, based on the similarity of the image position information and the feature point descriptors, feature point matching between images can be performed. It should be noted that the image position information here refers to the GPS position of the aerial image, which can be obtained during the image acquisition stage and is default available for aerial photogrammetry. Since many feature points can be extracted from each aerial image, the feature point matching relationships between different images (air-ground, between aerial images, between ground images) are not known at the beginning. Therefore, feature matching is required. Feature point matching is a necessary step in the sparse reconstruction process. Sparse reconstruction is also called Structure From Motion (SFM). Through the sparse reconstruction process, the matching relationships of all aerial images in the scene will be recorded. There will be an "edge" between the images with matching relationships, and thus an initial scene graph can be formed. According to the initial scene graph, through the SFM technology, the aerial images can be positioned and oriented to obtain the poses of all aerial images, and at the same time, a sparse point cloud model of the scene is generated.

[0063] Specifically, in step S3, the generating the camera position information of the target camera according to the sparse point cloud model includes:

[0064] S31. Obtain the coordinate ranges of the scene and the position where the original camera is located according to the sparse point cloud model;

[0065] S32. Divide the coordinate ranges into grids and calculate the initial sampling point coordinates corresponding to each grid;

[0066] S33. Use the pre-constructed signed distance field to judge the positions of each initial sampling point coordinate to screen out at least one target sampling point coordinate located on the outer surface of the scene;

[0067] S34. Use at least one target sampling point coordinate located on the outer surface of the scene as the camera position information of the target camera.

[0068] Exemplarily, in steps S31 to S32, according to the sparse point cloud model and the positions of the scene and the original camera, the coordinate ranges of the entire scene and the camera position can be obtained. This coordinate range includes x mun 、x max 、y min, y max , z min , z max Six coordinate values. Based on the coordinate range, a spatial grid is divided. Since the scenes are all in the true geographic coordinate system, the unit is usually meters. When dividing the grid, the size (interval) of the grid in the three coordinate dimensions can be determined according to actual needs, which are g x , g y , g z . After the grid interval is determined, the coordinates t n = (x i , y j , z l ) of all initial sampling points (there is one sampling point in a grid) in space can be calculated:

[0069]

[0070] Where:

[0071]

[0072] For the same scene, the larger the grid interval, the fewer the initial sampling points, and vice versa. The coordinates of these initial sampling points are the candidate positions of the target camera.

[0073] Exemplarily, in steps S33 - S34, the present invention needs to pre - construct a Signed Distance Field (SDF). The Signed Distance Field (SDF) is a data structure that can be used to store the contour information of a shape. Using the SDF, a three - dimensional scene can be implicitly represented. Its input is the three - dimensional point coordinates in space, and the output is the closest distance from this point to the target object. This distance can be positive (indicating outside the contour) or negative (indicating inside the contour). In addition, the SDF can be expressed using a neural network. Its input is the sparse point cloud of the target scene, and then relying on the powerful fitting ability of the neural network, a continuous SDF of the target scene can be implicitly constructed. Based on this SDF, the distance from any position in the scene to the nearest surface can be calculated, and whether this position is inside or outside the scene surface can be determined by the sign of the distance.

[0074] Since the candidate positions of the target camera obtained in the above step S32 may be outside the scene or inside the scene, and the present invention only needs the images outside the scene, it is necessary to select the cameras (i.e., target sampling points) located outside the scene. Based on the initial sampling point coordinates t n and the constructed SDF, it can be determined whether the generated camera is inside the scene (value is 0) or outside the scene (value is 1). The camera position information of the target camera is the coordinates of at least one target sampling point on the outer surface of the scene. At this time, all t n containing the value of 1 are collected into a set Among them, the determination process of the target sampling points satisfies the following formula:

[0075]

[0076] Specifically, for each target sampling point, a new interpolation view needs to be generated, and then new images are rendered under the constructed neural radiance field based on these views. The camera position information of all target cameras is obtained in steps S31 to S34, and the orientation of each target camera also needs to be generated. There are many schemes for generating new views. An embodiment of the present invention provides a method for generating camera views according to horizontal intervals and vertical intervals. At this time, in step S3, the generation of the camera view information of the target camera according to the camera position information includes:

[0077] S35. For all initial sampling points, determine the horizontal direction angle of the target camera according to a preset horizontal interval;

[0078] S36. On the basis of each horizontal direction angle, determine the vertical direction angle of the target camera according to a preset vertical interval;

[0079] S37. Generate the camera view information of the target camera according to the horizontal direction angle, the vertical direction angle, and the target sampling point coordinates.

[0080] Exemplarily, in step S35, at each initial sampling point, first taking the z-axis as the rotation axis, with the positive y-axis direction as 0°, and the clockwise direction as the positive direction of the angle, a total of 360° (radian 2π). In this way, the horizontal direction orientation of the camera can be determined according to the given number of horizontal intervals g h , and the horizontal direction angle γ i (radian) of each orientation satisfies the following formula:

[0081] γ i =i*Δr h , i = 0, 1,..., g h -1 (4);

[0082] Among them, Δr h is the horizontal radian coefficient, which satisfies the following formula:

[0083] Δr h =2π / g h (5).

[0084] Exemplarily, in step S36, at each horizontal direction angle position, starting from 0° with the positive z-axis direction and ending at 180° (radian π) with the negative z-axis direction, according to the given number of vertical intervals g v , the vertical direction orientation of the camera can be determined, and the vertical direction angle θ of each orientationj (in radians) satisfies the following formula:

[0085]

[0086] where, Δr v is the vertical radian coefficient and satisfies the following formula:

[0087] Δr v = π / g v (7).

[0088] It should be noted that since the vertical angle θ j being 0° means shooting vertically upward, that is, shooting at the sky, which is meaningless, so the value of j in formula (6) starts from 1.

[0089] Exemplarily, in step S37, according to the horizontal angle γ i and the vertical angle θ j , the orientation d m of each viewing angle can be calculated, which satisfies the following formula:

[0090] d m = (sin(θ j )sin(γ i ), sin(θ j )cos(γ i ), cos(θ j ), m = 1, …, g h *(g v - 1) (8);

[0091] Then, based on the target sampling point coordinates t n obtained in steps S31 to S34, the poses T of all cameras can be obtained, which satisfies the following formula:

[0092] T = [d m |t n (9);

[0093] where,

[0094] Finally, collect the poses T of all cameras together to form a new view set Taking the new view set as the camera view information of the target camera.

[0095] Specifically, before executing step S4, the method further includes: training a three-dimensional reconstruction model based on a neural radiance field; wherein, the input of the three-dimensional reconstruction model is the scene image and its corresponding pose, and after the neural radiance field training is completed, use the three-dimensional reconstruction model to output the rendered image in the new view.

[0096] It should be noted that in this step, the result obtained from the process of training the 3D reconstruction model based on the neural radiance field is a trained implicit 3D model (neural radiance field), while the rendered image is the result output by inputting the above new view camera pose into the trained neural radiance field. Therefore, the input of the 3D reconstruction model is the scene image and its corresponding pose. After the neural radiance field training is completed, according to the new view camera pose, the 3D reconstruction model can be used to output the rendered image from the new view.

[0097] Exemplarily, the embodiments of the present invention need to pre-execute the process of implicitly constructing the scene based on the neural radiance field to construct the neural radiance field (NRF). The neural radiance field is a deep learning-based method for generating and understanding the continuous representation of 3D scenes. This technology is usually used in the fields of computer vision and graphics, especially in applications such as 3D reconstruction, virtual reality, and augmented reality. The NRF model can learn the 3D structure of the scene from images observed from multiple angles and generate high-quality 3D views. The core idea of NRF is to use a neural network to model the radiance field of the scene, that is, how the color and brightness of each point in the scene change with the change of the viewing angle. Through training, the neural network can capture the geometric and lighting characteristics of the scene, thereby achieving the ability to generate new views from any perspective. In the training stage, its input is the image of the scene and its corresponding pose. After training is completed, according to the given spatial point coordinates and viewing direction, the pixel color value at the corresponding position can be rendered. The rendering process can be expressed by the following formula:

[0098] F(x,d)→(c,σ) (10);

[0099] Where x is the 3D point coordinate in the scene, d is the viewing direction, which is the input of NRF; F(·) represents a deep learning neural network, c represents color, and σ represents the volume grid density, which is the output of NRF.

[0100] It should be noted that here x generally refers to the coordinate of the pixel in the image in 3D space and has no connection with the feature points. This is the process of new view rendering. After the NRF training is completed, given a new view (the parameters and pose of the image), the ray direction (i.e., d) at each pixel in the image and the 3D space coordinate of the pixel (i.e., x) can be calculated through the camera projection process, and then the image from this new view can be rendered through NRF. Rendering the image from the new view is the main goal of NRF, and the viewing direction is related to the pose.

[0101] Specifically, for a rendered image, let the camera ray be r(t) = o + td, where o is the camera position (which can be used as the starting point), d is the ray direction vector, and t is the distance from a point on the ray to the starting point o. It is a scalar parameter that controls the position of the point on the ray. When t = 0, the ray is at the starting point o; when t increases, the ray extends forward along the direction vector d; when t decreases, the ray extends in the opposite direction of the direction vector d.

[0102] Then the color value of the current ray can be obtained through the following formula:

[0103]

[0104] where t 1 and t 2 are the starting and ending points of the ray, σ(r(t)) is the volume grid density at the ray r(t), c(r(t), d) is the volume grid color at the ray r(t), T(t) is the cumulative transparency, representing the probability that the ray is not blocked from the starting point t 1 to t. The calculation process of T(t) satisfies the following formula:

[0105]

[0106] where r(u) is equivalent to r(t) mentioned in formula (11). Since the integrand variables are the same, here it is just to distinguish from formula (11), so r(t) is replaced by r(u) in formula (12). Similarly, du corresponds to the integral symbol dt in formula (11).

[0107] By tracing the camera rays corresponding to each pixel on the image to be rendered, the above integral can be calculated. However, in the actual calculation process, the above integral is usually discretized, that is, the ray is sampled (such as equally spaced division, etc.), divided into N points, and the above integral is approximated by summation to obtain the integral result satisfying the following formula:

[0108]

[0109] where c i is the color of the sampling point i on the ray (obtained by calculating through the neural network MLP), α i and T i are the alpha composite transparency and cumulative transparency at the sampling point i respectively, and are obtained by the following formula:

[0110] α i = 1 - exp(-σ i δ i ) (14);

[0111]

[0112] Among them, σ i is the volume grid density at sampling point i, and δ i is the position of sampling point i, σ j is the volume grid density at sampling point j, and δ j is the position of sampling point j.

[0113] Through the above calculation process, the images from any perspective in the scene can be rendered. Since the density σ and color c in the above process are both fully expressed by the neural network MLP, this method is called implicit scene representation.

[0114] Exemplarily, in step S4, using the camera view information of the target camera obtained in step S3 (i.e., the new view set ), corresponding image rendering is performed in the pre-constructed NRF to obtain the rendered image set

[0115] Specifically, in step S5, the feature point matching of the ground image, the aerial image, and the rendered image to obtain the matching trajectory includes: numbering all the feature points of the ground image, the aerial image, and the rendered image; when there are mutually matching feature points in the ground image, the aerial image, and the rendered image, obtaining the corresponding feature point numbers and constructing the matching trajectory.

[0116] Exemplarily, image feature point extraction is performed on the ground image and the rendered image set . The feature point extraction algorithm here is the same as that in step S21. Then, the extraction results are combined with the aerial image and its feature points in step S21 for feature matching. Independent numbers (ids) can be assigned to each image, denoted as I i , i = 1, 2,..., n img , where n img is the sum of the numbers of all ground images, aerial images, and rendered images. At the same time, the feature points on each image are numbered in sequence, denoted as represents the j-th feature point on the i-th image, where represents the number of feature points on the i-th image. Then, if the j a -th feature point on the i b -th image and the j c -th feature point on the i d -th image have a matching relationship, it can be denoted as Furthermore, if the j e -th feature point on the i f -th image and the j c -th feature point on the i dThere is also a matching relationship between the feature points, which indicates that and There is a matching relationship between them, and they can be put together as a matching trajectory, denoted as Collect all different matching trajectories track and denote it as In the above manner, all feature points with matching relationships among the ground image, the aerial image, and the rendered image can be found.

[0117] Specifically, in step S6, the feature points corresponding to the rendered image in the matching trajectory are removed to obtain an updated matching trajectory, so as to generate a joint scene graph of the ground image and the aerial image according to the updated matching trajectory.

[0118] Exemplarily, although NRF can be used to render the scene image from a new perspective, due to the limitations of the algorithm, the quality of these images cannot reach the quality level of real shooting. For example, there will be problems such as floating objects, pixel blurring, and missing in the rendered image. If the rendered image and the aerial-ground image are optimized and modeled together, the final model will have its quality degraded due to the interference of the rendered image. Therefore, in the embodiments of the present invention, the rendered image is regarded as an intermediate bridge for the matching of the aerial-ground image, and the rendered image needs to be removed first before global optimization. By traversing all the matching trajectory sets According to the id value I of the rendered image i , all the feature points of the corresponding rendered image in the matching trajectory are removed. During the removal process, if the number of feature points corresponding to a certain matching trajectory is less than 2, then the matching trajectory is directly deleted (because a valid matching trajectory should contain at least 2 corresponding feature points). Finally, a set of feature point matching trajectories of the rendered image removed can be obtained, and a joint scene graph of the aerial-ground image is generated using this set of matching trajectories.

[0119] Specifically, after step S6 is executed, the method further includes: performing sparse reconstruction on the joint scene graph to obtain a sparse point cloud model of the aerial-ground joint scene.

[0120] Exemplarily, similar to step S2, based on the joint scene graph of the aerial-ground image, through the SFM technology, the aerial-ground image can be positioned and oriented to obtain the poses of all the aerial-ground images, and at the same time, a sparse point cloud model of the aerial-ground joint scene is generated. This sparse point cloud model can be stored for subsequent data processing of the aerial-ground image. The present invention does not limit the subsequent processing process.

[0121] Compared with the prior art, the aerial-ground image matching method disclosed in the present invention has the following beneficial effects:

[0122] 1) From the perspective of reducing the perspective differences between aerial and ground images, leveraging the ability of Neural Radiance Fields (NRF) to generate new perspective images of a scene, and through scene grid division and the ability to determine the inner and outer surfaces of the Signed Distance Field (SDF), the intermediate layer scene images are rendered to achieve automated joint modeling of aerial and ground images. By generating the intermediate perspective images of the scene, the present invention reduces the perspective differences between aerial and ground images, constructs the matching relationship between aerial and ground images, and solves the problem of difficult matching caused by excessive perspective differences between aerial and ground images.

[0123] 2) Using the intermediate perspective images as a bridge, joint matching of aerial and ground images at the feature point level is achieved, enabling joint global optimization of aerial and ground images and improving the accuracy of the model.

[0124] 3) The present invention has relatively low requirements for the input images and has no requirements for the coordinate systems of aerial and ground images, as long as both are taken in the same area, thus having a wide range of applicability. In the embodiments of the present invention, it is only required that the given aerial image and ground image are taken for the same area. After the input images are given, the aerial image is first used for positioning and pose determination and the required neural implicit field is generated, and then the ground image and the rendered image are used together with the aerial image for feature matching and joint aerial and ground scene restoration, without the requirement of aligning the coordinate systems of aerial and ground images and data preprocessing requirements, and the algorithm has better applicability.

[0125] See Figure 2 , Figure 2 FIG.

[0126] An image data acquisition module 11, configured to acquire an aerial image and a ground image;

[0127] A sparse reconstruction module 12, configured to perform sparse reconstruction on the aerial image to obtain the pose of the aerial image and the sparse point cloud model of the scene;

[0128] A camera position information generation module 13, configured to generate the camera position information of the target camera according to the sparse point cloud model;

[0129] A camera viewing angle information generation module 14, configured to generate the camera viewing angle information of the target camera according to the camera position information;

[0130] A rendered image generation module 15, configured to perform three-dimensional reconstruction based on the neural radiance field using the aerial image and the pose of the aerial image, and render the rendered image under the camera viewing angle information through the neural radiance field;

[0131] A matching trajectory generation module 16 is configured to perform feature point matching on the ground image, the aerial image, and the rendered image to obtain matching trajectories;

[0132] A matching trajectory update module 17 is configured to remove the feature points corresponding to the rendered image from the matching trajectories to obtain updated matching trajectories;

[0133] A joint scene graph generation module 18 is configured to generate a joint scene graph of the ground image and the aerial image according to the updated matching trajectories.

[0134] Specifically, the sparse reconstruction module 12 is configured to: extract feature points from the aerial image; perform feature point matching between aerial images based on the similarity of the image position information of the aerial images and the extracted feature point descriptors to obtain matching relationships; generate an initial scene graph according to the matching relationships; perform pose positioning on the aerial images according to the initial scene graph to obtain the poses of the aerial images and generate a sparse point cloud model of the scene.

[0135] Specifically, the camera position information generation module 13 is specifically configured to: obtain the coordinate ranges of the scene and the position where the original camera is located according to the sparse point cloud model; divide the coordinate ranges into grids and calculate the initial sampling point coordinates corresponding to each grid; use a pre-constructed signed distance field to perform position judgment on each initial sampling point coordinate to filter out at least one target sampling point coordinate located on the outer surface of the scene; use at least one target sampling point coordinate located on the outer surface of the scene as the camera position information of the target camera.

[0136] Specifically, the camera viewing angle information generation module 14 is specifically configured to: for all initial sampling points, determine the horizontal direction angle of the target camera according to a preset horizontal interval; on the basis of each horizontal direction angle, determine the vertical direction angle of the target camera according to a preset vertical interval; generate the camera viewing angle information of the target camera according to the horizontal direction angle, the vertical direction angle, and the target sampling point coordinates.

[0137] Specifically, the aerial and ground image matching device 100 further includes:

[0138] A three-dimensional reconstruction model training module is configured to train a three-dimensional reconstruction model based on a neural radiance field; wherein, the input of the three-dimensional reconstruction model is a scene image and its corresponding pose, and after the neural radiance field training is completed, use the three-dimensional reconstruction model to output a rendered image in a new view.

[0139] Specifically, the matching trajectory generation module 16 is specifically configured to: number all the feature points of the ground image, the aerial image, and the rendered image; when there are mutually matching feature points in the ground image, the aerial image, and the rendered image, obtain the corresponding feature point numbers and construct a matching trajectory.

[0140] Specifically, the sparse reconstruction module 12 is further configured to perform sparse reconstruction on the joint scene graph to obtain a sparse point cloud model of the air-ground joint scene.

[0141] It should be noted that the working processes of the various modules in the air-ground image matching device 100 according to the embodiments of the present invention may refer to the working processes of the air-ground image matching method described in the above embodiments, and will not be elaborated here.

[0142] See Figure 3 , Figure 3 is a structural block diagram of an air-ground image matching device 200 provided by an embodiment of the present invention. The air-ground image matching device 200 includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, the steps in the above embodiments of the various air-ground image matching methods are implemented.

[0143] Exemplarily, the computer program may be divided into one or more modules / units. The one or more modules / units are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the air-ground image matching device 200.

[0144] The air-ground image matching device 200 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art can understand that the schematic diagram is only an example of the air-ground image matching device 200, and does not constitute a limitation on the air-ground image matching device 200. It may include more or fewer components than shown, or combine certain components, or different components. For example, the air-ground image matching device 200 may further include an input / output device, a network access device, a bus, etc.

[0145] The processor 21 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The processor 21 is the control center of the air-ground image matching device 200, and connects various parts of the entire air-ground image matching device 200 through various interfaces and lines.

[0146] The memory 22 can be used to store the computer programs and / or modules. The processor 21 realizes various functions of the air-ground image matching device 200 by running or executing the computer programs and / or modules stored in the memory 22, and by calling the data stored in the memory 22. The memory 22 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the mobile phone (such as audio data, phone book, etc.). In addition, the memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0147] Among them, if the modules / units integrated in the vacant land image matching device 200 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-described embodiment methods of the present invention, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 21, the steps of the above various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0148] The above is the preferred implementation manner of the present invention. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. A space-ground image matching method, characterized in that: include: Acquire aerial and ground imagery; Performing sparse reconstruction on the aerial image to obtain a sparse point cloud model of the aerial image pose and scene; Generating camera position information of a target camera according to the sparse point cloud model, and generating camera viewing angle information of the target camera according to the camera position information; Performing three-dimensional reconstruction based on a neural radiation field using the aerial image and the aerial image pose, and rendering a rendered image under the camera viewing angle information through the neural radiation field; Performing feature point matching on the ground image, the aerial image and the rendered image to obtain a matching trajectory; The feature points corresponding to the rendered image in the matching trajectory are eliminated to obtain an updated matching trajectory, so as to generate a joint scene graph of the ground image and the aerial image according to the updated matching trajectory.

2. The air-ground image matching method according to claim 1, characterized in that: The sparse reconstruction of the aerial image to obtain a sparse point cloud model of the aerial image pose and scene includes: Extracting feature points from the aerial image; Based on the similarity between the image position information of the aerial images and the extracted feature point descriptors, matching the feature points between the aerial images is performed to obtain a matching relationship; Generate an initial scene graph based on the matching relationship; According to the initial scene graph, the aerial image is positioned and oriented to obtain the aerial image pose, and a sparse point cloud model of the scene is generated.

3. The air-ground image matching method according to claim 1, characterized in that: The step of generating the camera position information of the target camera according to the sparse point cloud model comprises: Acquire the coordinate range of the scene and the original camera position according to the sparse point cloud model; Divide the coordinate range into grids, and calculate the coordinates of the initial sampling points corresponding to each grid; Using a pre-constructed signed distance field to perform position determination on each initial sampling point coordinate, so as to screen out at least one target sampling point coordinate located on the outer surface of the scene; The coordinates of at least one target sampling point located on the outer surface of the scene are used as the camera position information of the target camera.

4. The air-ground image matching method according to claim 3, characterized in that: The step of generating the camera viewing angle information of the target camera according to the camera position information includes: For all initial sampling points, the horizontal direction angle of the target camera is determined according to the preset horizontal interval; Based on each horizontal direction angle, determining the vertical direction angle of the target camera according to a preset vertical interval; The camera viewing angle information of the target camera is generated according to the horizontal direction angle, the vertical direction angle and the target sampling point coordinates.

5. The air-ground image matching method according to claim 1, characterized in that: Before performing three-dimensional reconstruction based on the neural radiation field using the aerial image and the aerial image posture, the method further includes: Train a three-dimensional reconstruction model based on the neural radiation field; wherein the input of the three-dimensional reconstruction model is a scene image and its corresponding posture. After the neural radiation field training is completed, the three-dimensional reconstruction model is used to output a rendered image under a new perspective.

6. The air-ground image matching method according to claim 1, characterized in that: The performing feature point matching on the ground image, the aerial image and the rendered image to obtain a matching trajectory includes: Numbering all feature points of the ground image, the aerial image and the rendered image; When there are mutually matching feature points in the ground image, the aerial image, and the rendered image, the corresponding feature point numbers are obtained and a matching trajectory is constructed.

7. The air-ground image matching method according to claim 1, characterized in that: After generating a joint scene graph of the ground image and the aerial image according to the updated matching trajectory, the method further includes: The joint scene graph is sparsely reconstructed to obtain a sparse point cloud model of the air-ground joint scene.

8. An open-air image matching device, characterized in that: include: Image data acquisition module, used to acquire aerial images and ground images; A sparse reconstruction module, used for performing sparse reconstruction on the aerial image to obtain a sparse point cloud model of the aerial image pose and scene; A camera position information generating module, used to generate camera position information of a target camera according to the sparse point cloud model; A camera viewing angle information generating module, used to generate the camera viewing angle information of the target camera according to the camera position information; A rendering image generation module, used to perform three-dimensional reconstruction based on the neural radiation field using the aerial image and the aerial image posture, and render a rendering image under the camera viewing angle information through the neural radiation field; A matching trajectory generation module, used for matching feature points of the ground image, the aerial image and the rendered image to obtain a matching trajectory; A matching trajectory updating module, used to remove feature points corresponding to the rendered image in the matching trajectory to obtain an updated matching trajectory; The joint scene graph generation module is used to generate a joint scene graph of the ground image and the aerial image according to the updated matching trajectory.

9. An air-ground image matching device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the air-ground image matching method as described in any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the air-ground image matching method according to any one of claims 1 to 8.

Images