Method, device, equipment, medium and product for generating air-ground bridging image

Abstract

The application discloses a kind of air-ground bridging image generation method, device, equipment, medium and product, the method includes obtaining the aerial image and the ground image of target area respective space three results;According to sparse point cloud data in space three results, aerial image and ground image are aligned, obtain target aerial image and target ground image;Target aerial image and target ground image are projected, and the overlapping region of target aerial image and target ground image is determined;According to the position parameter of bridging image, the position parameter of bridging image is calculated according to bridging image;According to the position parameter of bridging image and the 3D Gaussian rendering model after training, bridging image is generated.Air, ground image can be aligned in space by plane element, avoid precision instability problem, reduce labor cost, introduce bridging image to connect air, ground image of cross view domain, solve the problem that air, ground image view difference is big and leads to unable matching, to improve the precision of three-dimensional reconstruction model.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction technology, and in particular to a method, apparatus, equipment, medium, and product for generating air-ground bridging images. Background Technology

[0002] Because 3D models can accurately reproduce the three-dimensional information of the real world, they are widely used in urban planning, emergency rescue, and low-altitude economy. However, current 3D scene models have limitations in accuracy and cannot meet high-precision requirements. To construct high-precision 3D models, it is necessary to jointly reconstruct 3D models from ground and aerial photographs.

[0003] The methods for refined 3D reconstruction based on aerial and ground imagery include: The first type of method manually adds connection matching points to the aerial triangulation results to solve the problem of difficult matching of features between aerial and ground images across the field of view, and then performs 3D reconstruction. However, adding connection matching points manually increases labor costs and the accuracy of the added matching points is unstable, affecting the image reconstruction results. The second type of method uses UAVs for close-range photogrammetry, which significantly increases the original images in the aerial and ground imagery to avoid the problem of direct matching between aerial and ground images across the field of view, and then performs 3D reconstruction. However, by significantly increasing the original UAV imagery data, additional acquisition work is required, which can easily lead to unnecessary waste of labor costs. In addition, it is difficult to achieve automatic acquisition of UAV images in scenarios with weak satellite signals. Summary of the Invention

[0004] This invention provides a method, apparatus, device, medium, and product for generating air-ground bridging images. Without adding extra acquisition work, it introduces intermediate bridging images to connect air and ground images across the field of view, supporting subsequent refined 3D reconstruction. This avoids the need for manual addition of matching connection points and solves the problem of mismatch due to large differences in viewing angles.

[0005] To achieve the above objectives, embodiments of the present invention provide a method for generating air-to-ground bridging images, comprising:

[0006] Aerial triangulation results are obtained for aerial and ground images of the target area; wherein, the aerial triangulation results include the image intrinsic parameters, coordinate system transformation parameters, and sparse point cloud data of the aerial and ground images;

[0007] Based on the sparse point cloud data, the aerial imagery and the ground imagery are spatially aligned to obtain the target aerial imagery and the target ground imagery.

[0008] The aerial and ground images of the target are projected to determine the overlapping area between them; the position parameters of the bridging image are calculated based on the overlapping area.

[0009] Based on the position parameters of the bridging image and the trained 3D Gaussian rendering model, a bridging image of the target aerial image and the target ground image is generated.

[0010] As an improvement to the above scheme, the step of spatially aligning the aerial imagery and ground imagery based on the sparse point cloud data to obtain target aerial imagery and target ground imagery includes:

[0011] Based on the sparse point cloud data, construct planar primitive pairs corresponding to the aerial and ground images;

[0012] Based on the planar primitive pairs, the transformation parameters of the aerial image and the ground image are calculated to spatially align the aerial image and the ground image, thereby obtaining the target aerial image and the target ground image.

[0013] As an improvement to the above scheme, the step of constructing planar primitive pairs corresponding to the aerial image and the ground image based on the sparse point cloud data includes:

[0014] Planar features are extracted from the sparse point cloud data to obtain the corresponding aerial plane set and ground plane set;

[0015] Based on the multi-scale geometric feature relationships of the planar primitives in the aerial planar set and the ground planar set, planar primitive pairs corresponding to the aerial image and the ground image are constructed.

[0016] As an improvement to the above scheme, the projection of the target aerial image and the target ground image to determine the overlapping area of ​​the target aerial image and the target ground image includes:

[0017] A ray casting algorithm is used to project the target aerial image and the target ground image, and to determine the image index of each pixel in the target aerial image and the target ground image;

[0018] The area where the image indexes of both the target aerial image and the target ground image exist simultaneously is defined as the overlapping area of ​​the target aerial image and the target ground image.

[0019] As an improvement to the above scheme, the step of calculating the position parameters of the bridging image based on the overlapping area includes:

[0020] Set the basic parameters of the bridging image; wherein the basic parameters include image focal length, image resolution, pixel size, horizontal viewing angle and vertical viewing angle;

[0021] If the current layer bridging image is the first layer bridging image, calculate the position parameters of the current layer bridging image based on the overlapping area and the basic parameters;

[0022] If the resolution of the current layer bridging image does not meet the preset resolution requirement, the position parameters of the next layer bridging image are calculated based on the position parameters of the current layer bridging image, the overlapping area, and the basic parameters, until the resolution of the next layer bridging image meets the preset resolution requirement.

[0023] As an improvement to the above scheme, if the current layer bridging image is the first layer bridging image, calculating the position parameters of the current layer bridging image based on the overlapping area and the basic parameters includes:

[0024] If the current layer bridging image is the first layer bridging image, calculate the target distance between the current layer bridging image and the overlapping area based on the size of the overlapping area, the horizontal viewing angle, and the vertical viewing angle.

[0025] The position parameters of the current layer bridging image are calculated based on the target distance, the centroid position of the bounding box of the overlapping region, and the plane normal vector.

[0026] To achieve the above objectives, embodiments of the present invention provide an apparatus for generating air-to-ground bridging images, comprising:

[0027] The aerial triangulation result acquisition module is used to acquire aerial triangulation results corresponding to aerial and ground images of the target area; wherein, the aerial triangulation results include the image intrinsic parameters, coordinate system transformation parameters and sparse point cloud data of the aerial and ground images;

[0028] The target image alignment module is used to spatially align the aerial image and the ground image based on the sparse point cloud data to obtain the target aerial image and the target ground image.

[0029] The position parameter calculation module is used to project the target aerial image and the target ground image, determine the overlapping area of ​​the target aerial image and the target ground image, and calculate the position parameters of the bridging image based on the overlapping area.

[0030] The bridging image generation module is used to generate a bridging image of the target aerial image and the target ground image based on the position parameters of the bridging image and the trained 3D Gaussian rendering model.

[0031] To achieve the above objectives, embodiments of the present invention provide an apparatus for generating air-to-ground bridging images, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the above-described method for generating air-to-ground bridging images.

[0032] To achieve the above objectives, embodiments of the present invention also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the above-described method for generating air-to-ground bridging images.

[0033] To achieve the above objectives, embodiments of the present invention also provide a computer program product, which is stored in a storage medium and executed by at least one processor to implement the steps of the above-described method for generating air-to-ground bridging images.

[0034] Compared with existing technologies, the present invention discloses a method, apparatus, device, medium, and product for generating air-to-ground bridging imagery. This involves acquiring aerial triangulation results corresponding to aerial and ground images of a target area. The aerial triangulation results include image intrinsic parameters, coordinate system transformation parameters, and sparse point cloud data of the aerial and ground images. Based on the sparse point cloud data, the aerial and ground images are spatially aligned to obtain target aerial and target ground images. The target aerial and target ground images are then projected to determine their overlapping region. Based on the overlapping region, the position parameters of the bridging image are calculated. Finally, based on the position parameters of the bridging image and a trained 3D Gaussian rendering model, a bridging image of the target aerial and target ground images is generated. It can achieve spatial alignment of aerial and ground images through planar primitive pairs, avoiding the instability of accuracy caused by manually adding connection matching points, reducing labor costs. At the same time, it introduces intermediate bridging images to connect aerial and ground images across the field of view, solving the problem of mismatch caused by large differences in the perspective of aerial and ground images across the field of view. It does not require a significant increase in the original image data of UAVs, reducing the waste of labor costs and avoiding the problem of limited weak signal acquisition, thereby improving the accuracy of the 3D reconstruction model. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating a method for generating air-to-ground bridging images according to an embodiment of the present invention.

[0036] Figure 2 This is a schematic diagram of the structure of an air-to-ground bridging image generation device provided in an embodiment of the present invention;

[0037] Figure 3 This is a structural block diagram of an air-to-ground bridging image generation device provided in an embodiment of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] It should be noted that the terms "comprising" and "specific" in this invention, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0040] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for generating air-to-ground bridging imagery according to an embodiment of the present invention. The method for generating air-to-ground bridging imagery includes:

[0041] S1, acquire the aerial triangulation results corresponding to the aerial and ground images of the target area; wherein, the aerial triangulation results include the image intrinsic parameters, coordinate system transformation parameters and sparse point cloud data of the aerial and ground images;

[0042] S2, Based on the sparse point cloud data, spatially align the aerial image and the ground image to obtain the target aerial image and the target ground image;

[0043] S3, Project the target aerial image and the target ground image to determine the overlapping area of ​​the target aerial image and the target ground image; Calculate the position parameters of the bridging image based on the overlapping area;

[0044] S4. Based on the position parameters of the bridging image and the trained 3D Gaussian rendering model, generate a bridging image of the target aerial image and the target ground image.

[0045] For example, the method for generating air-to-ground bridging images described in this embodiment of the invention is implemented by an air-to-ground bridging image generation server, which is capable of interacting with the target user. The air-to-ground bridging image generation service acquires aerial and ground images of the target area using photographic equipment (e.g., drones, mobile surveying vehicles). Aerial triangulation (hereinafter referred to as aerial triangulation) is performed on both aerial and ground images to obtain their respective aerial triangulation results (e.g., image intrinsic parameters, coordinate system transformation parameters, and sparse point cloud data). Planar features are extracted from the aerial and ground sparse point cloud data to obtain aerial and ground plane primitive sets, which are then spatially aligned to obtain the target aerial and target ground images. A ray casting algorithm is used to project the target aerial and target ground images, determining their overlapping area. Based on this overlapping area, the position parameters of the bridging image are calculated. These position parameters are then input into a 3D Gaussian rendering model trained based on the aerial triangulation results of the aerial images to generate the bridging image. This invention achieves spatial alignment of aerial and ground images through planar primitive pairs, avoiding the instability in accuracy caused by manually adding connection matching points and reducing labor costs. At the same time, it introduces intermediate bridging images to connect aerial and ground images across the field of view, solving the problem of mismatch caused by large differences in the perspectives of aerial and ground images across the field of view. This eliminates the need to significantly increase the amount of original UAV image data, reducing labor costs and avoiding the problem of limited weak signal acquisition, thereby improving the accuracy of the 3D reconstruction model.

[0046] Specifically, step S1 includes:

[0047] S11, acquire aerial and ground images of the target area;

[0048] S12, perform aerial triangulation on the aerial images and ground images to obtain the aerial triangulation results corresponding to the aerial images and ground images.

[0049] For example, when using a drone to acquire aerial imagery of a target area, during the acquisition process, parameters such as the drone's flight path, altitude, shooting angle, and shooting interval should be rationally planned based on the target area's extent, terrain complexity, and reconstruction accuracy requirements to ensure that the acquired imagery comprehensively and clearly covers the target area. After using 3D reconstruction software to perform image feature extraction, feature matching, and bundle adjustment on the aerial imagery, the aerial triangulation result of the aerial imagery is obtained, including the image intrinsic parameter K of the aerial imagery. a Coordinate system transformation parameters RT a Sparse point cloud data PC aerial(Composed of a series of three-dimensional coordinate points, which are feature points with spatial location information determined in the image through aerial triangulation processing);

[0050] The expression for the sparse point cloud data is as follows:

[0051]

[0052] The expression for the image intrinsic parameters is:

[0053]

[0054] The expression for the coordinate system transformation parameters is:

[0055]

[0056] In the formula, This represents the nth three-dimensional coordinate point in the sparse point cloud data of the aerial image. (x n ,y n ,z n K represents the coordinates of the nth point in three-dimensional space. a It is a 3×3 matrix that reflects the camera's internal parameters. This indicates the camera focal length in the x and y directions of the aerial image. This represents the pixel coordinates of the camera foothold in the aerial image along the x and y directions in the pixel coordinate system; R a A 3×3 rotation matrix is ​​used to describe the rotation information of aerial imagery, T a It is a 3×1 translation vector used to describe the translation information of aerial imagery, and the superscript and subscript 'a' are used to indicate the identification of aerial imagery.

[0057] Ground images of the target area are acquired using a mobile surveying vehicle. After feature extraction, feature matching, and bundle adjustment using 3D reconstruction software, the aerial triangulation results of the ground images are obtained, including the image intrinsic parameter K. g Coordinate system transformation parameters RT g Sparse point cloud data PC ground (composed of the three-dimensional coordinates of a series of ground feature points);

[0058] The expression for the sparse point cloud data is as follows:

[0059]

[0060] The expression for the image intrinsic parameters is:

[0061]

[0062] The expression for the coordinate system transformation parameters is:

[0063]

[0064] In the formula, This represents the m-th three-dimensional coordinate point in the sparse point cloud data of the ground image. K represents the coordinates of the m-th point in three-dimensional space. g It is a 3×3 matrix that reflects the camera's internal parameters. This indicates the camera focal length in the x and y directions of the ground image. The pixel coordinates of the camera footpoint in the ground image along the x and y directions in the pixel coordinate system are represented by R; g A 3×3 rotation matrix is ​​used to describe the rotation information of the ground image, T g It is a 3×1 translation vector used to describe the translation information of the ground image, and the superscript and subscript g are used to indicate the identification of the aerial image.

[0065] Specifically, step S2 includes:

[0066] S21, Based on the sparse point cloud data, construct planar primitive pairs corresponding to the aerial image and the ground image;

[0067] S22, based on the planar primitive pair, calculate the transformation parameters of the aerial image and the ground image to spatially align the aerial image and the ground image to obtain the target aerial image and the target ground image.

[0068] For example, by performing planar feature calculations on the sparse point cloud data, an aerial planar set and a ground planar set are obtained. The multi-scale geometric feature relationships of the planar primitives in the aerial and ground planar sets are calculated, several identical planar primitive pairs are found, the transformation matrix of the planar primitive pairs is calculated, and their matching degree is evaluated. Finally, the transformation parameter with the strongest support is selected to achieve point cloud registration, obtaining the optimal transformation parameter. The two platform images are then aligned in a coordinate system to obtain the target aerial image and the target ground image. Taking four planar primitive pairs as an example, for each corresponding planar primitive pair, the candidate coordinate system transformation parameter RT for aligning the ground point cloud data to the aerial point cloud data is calculated using the least squares method. i Evaluate the candidate transformation parameters RT for each coordinate system. i Matching degree: First, check whether the centroids of the four planes in the ground point cloud are projected onto the matching planes in the aerial point cloud. Then, calculate q for all planar point clouds. iThe matching score after transformation (the smaller the value, the better the transformation parameters, i.e., the more accurate the matching between the transformed point and the nearest point) is selected. The transformation matrix with the strongest matching score is selected as the optimal transformation parameter RT of the coordinate system. Based on the optimal transformation parameter of the coordinate system, the aerial image and the ground image are aligned in one coordinate system to obtain the target aerial image and the target ground image. The matching score is calculated as follows:

[0069]

[0070] In the formula, z is the total number of points in the planar point cloud, and c i It is a planar point cloud q i The nearest point is obtained through nearest neighbor search after the transformation.

[0071] More specifically, step S21 includes:

[0072] S211, perform planar feature extraction on the sparse point cloud data to obtain the corresponding aviation planar set and ground planar set;

[0073] S212, based on the multi-scale geometric feature relationships of the planar primitives in the aerial planar set and the ground planar set, construct the planar primitive pairs corresponding to the aerial image and the ground image.

[0074] For example, planar feature calculations are performed on the sparse point cloud data to obtain an aerial planar set and a ground planar set. For instance, for aerial sparse point cloud data PC... aerial any point in With point Construct points with radius R centered on the point. Calculate the spherical neighborhood of the point. covariance matrix M a , for M a Eigenvalue decomposition yields three eigenvalues. Obtain the standard deviation along the directions of the three eigenvectors corresponding to the three eigenvalues, and calculate based on the standard deviation. The line features are used to retain points whose line features are greater than a certain threshold (e.g., 0.7) as plane points. Spatial distance clustering is then used to process these plane points to obtain the aviation plane set. The covariance matrix M is... a The expression is:

[0075]

[0076] The i-th feature value Standard deviation of the corresponding eigenvector direction The expression is:

[0077]

[0078] The line feature planarity a The expression is:

[0079]

[0080] The aviation ground plane set PL a The expression is:

[0081]

[0082] In the formula, N is a point The total number of points in the spherical neighborhood, q j Let q be the j-th point within the spherical neighborhood. j =(x j ,y j ,z j ), (x j ,y j ,z j ) represents the coordinates of the j-th point in three-dimensional space, and T is the transpose identifier; Let be the o-th plane in the set of aviation planes.

[0083] PC for sparse point cloud data on the ground ground any point in With point Construct points with radius R centered on the point. Calculate the spherical neighborhood of the point. covariance matrix M g , for M g Eigenvalue decomposition yields three eigenvalues. Obtain the standard deviation along the directions of the three eigenvectors corresponding to the three eigenvalues, and calculate based on the standard deviation. The line features are used to select points whose line features are greater than a certain threshold (e.g., 0.7) as planar points. Spatial distance clustering is then used to process these planar points to obtain a set of ground planes. The covariance matrix M is... g The expression is:

[0084]

[0085] The i-th feature value Standard deviation of the corresponding eigenvector direction The expression is:

[0086]

[0087] The line feature planarity g The expression is:

[0088]

[0089] The ground plane set PL g The expression is:

[0090]

[0091] In the formula, M is a point The total number of points in the spherical neighborhood, q k Let q be the k-th point within the spherical neighborhood. k =(x k ,y k ,z k ), (x k ,y k ,z k ) represents the coordinates of the k-th point in three-dimensional space, and T is the transpose identifier; Let l be the l-th plane in the set of ground planes.

[0092] For each plane in the aerial plane set and the ground plane set, the random sample consensus algorithm is used to fit the plane and obtain the plane normal vector. Assume that the plane normal vector is n. p = (a, b, c), by determining the plane normal vector n p With ground normal vector n g = (0,0,1), determine the normal vector n of the plane. p Is the corresponding plane an elevation? For example, if the plane normal vector n p With ground normal vector n g If the dot product between them is 0, then the normal vector n of the plane is... p The corresponding planes are elevations. Several elevations are randomly selected from the aerial planar set (taking four elevations as an example) to form the planar primitives of the aerial image. Each planar primitive must satisfy the condition that three planes are not parallel and the fourth plane is not coplanar. Based on the aerial planar primitives, planar primitives with the same multi-scale geometric characteristics are searched in the ground planar set. These found plane combinations are the planar primitives corresponding to the ground image. Multi-scale geometric characteristics include the angle between the normal vectors of the planes, the orthogonal projection distance between the centroids of the planes, and the proportional relationship of the projections between the planes. These relationships are stored in a lookup table for subsequent use. Planar primitive pairs corresponding to the aerial and ground images are constructed using these relationships.

[0093] Specifically, step S3, which involves projecting the target aerial image and the target ground image to determine the overlapping area of ​​the target aerial image and the target ground image, includes:

[0094] S31, using a ray casting algorithm, project the target aerial image and the target ground image, and determine the image index of each pixel in the target aerial image and the target ground image;

[0095] S32, the area where the image index of both the target aerial image and the target ground image exists simultaneously is taken as the overlapping area of ​​the target aerial image and the target ground image.

[0096] For example, based on the ray casting algorithm, the first triangular patch V passed through by each pixel coordinate f(x,y) in each target aerial image I is calculated. i The image index is recorded, and the image index calculation process is as follows:

[0097] First, determine the ray parameter equation r(t) for each pixel:

[0098] r(t) = center i +td,

[0099] In the formula, center i The optical center of the camera can be obtained using the optimal coordinate transformation parameter RT, and d is the ray direction, which can be obtained using the coordinates f(x,y) of each pixel and the camera intrinsic parameter matrix K, such as:

[0100] center i =-R T T,

[0101]

[0102] V i ={p1,p2,p3},

[0103] In the formula, {p1,p2,p3} are triangular facets V i The coordinates of the three vertices represent three points in three-dimensional space (with x, y, z coordinate components), and the triangular facet V... i normal vector n i It can be calculated using the following formula:

[0104] n i = (p2-p1)×(p3-p1),

[0105] Substituting the ray parametric equations into the plane equations of the triangular facets:

[0106] n i ·(center i +td-p1)=0,

[0107]

[0108] The sign of t determines the direction of intersection between the ray and the triangular facet plane. If t is positive, the ray intersects the triangular facet plane. The centroid coordinate method is used to determine whether the intersection point of the camera and the plane is inside the triangular facet, such as:

[0109]

[0110] If 0 ≤ α, β ≤ 1, and α + β ≤ 1, determine if the ray intersects with the triangular facet. Calculate the intersection between the camera and all triangular facets, then calculate the spatial distance to obtain the triangular facet closest to the camera, and record the image index of the closest triangular facet as the image index of the corresponding pixel; the image index is the identifier of the triangular facet in the entire image data, used to distinguish the triangular facets corresponding to different pixels. Using a ray casting algorithm, calculate the first triangular facet passed through by each pixel coordinate in each target ground image, and record the image index. The image index calculation process for the target ground image is similar to that for the target aerial image, and will not be repeated here. Detect the image index recorded for each triangular facet, and the area where the image indices of both the target aerial image and the target ground image coexist is taken as the overlapping area of ​​the target aerial image and the target ground image.

[0111] Specifically, step S3, calculating the position parameters of the bridging image based on the overlapping area, includes:

[0112] S33, Set the basic parameters of the bridged image; wherein, the basic parameters include image focal length, image resolution, pixel size, horizontal viewing angle and vertical viewing angle;

[0113] S34, if the current layer bridging image is the first layer bridging image, calculate the position parameters of the current layer bridging image based on the overlapping area and the basic parameters;

[0114] S35, if the resolution of the current layer bridging image does not meet the preset resolution requirement, then the position parameters of the next layer bridging image are calculated based on the position parameters of the current layer bridging image, the overlapping area, and the basic parameters, until the resolution of the next layer bridging image meets the preset resolution requirement.

[0115] More specifically, step S34 includes:

[0116] S341, if the current layer bridging image is the first layer bridging image, calculate the target distance between the current layer bridging image and the overlapping area based on the size of the overlapping area, the horizontal viewing angle, and the vertical viewing angle;

[0117] S342, calculate the position parameters of the current layer bridging image based on the target distance, the centroid position of the bounding box of the overlapping region, and the plane normal vector.

[0118] For example, set the basic parameters of the bridged image, such as image focal length f, image resolution (W, H), and pixel size (p). x ,p y ), horizontal angle θ x Vertical angle θ y Calculate the centroid position p of the bounding box based on the overlapping region. c ={x c ,y c ,z c} and size W p ×H p The bounding box refers to the smallest cuboid that tightly encloses the overlapping region, and the centroid of the bounding box refers to the geometric center of the smallest cuboid that tightly encloses the overlapping region. The positional parameters of the bridging image are calculated in a multi-level manner, and the specific calculation process is as follows:

[0119] First, calculate the image location parameters of the first layer of fully covered overlapping area (first layer bridging image). To ensure that the image can completely cover the overlapping area, the distance D1 between the location of the first layer bridging image and the target in the overlapping area can be calculated using the following formula:

[0120]

[0121] In the formula, max(W) p H p ) is the larger of the height and width of the overlapping region's bounding box, max(θ) x ,θ y () is the larger value of the image viewpoint. The plane normal vector n of the overlapping region. over = (n1, n2, n3), the location parameter C1 of the first layer bridging image is calculated as follows:

[0122]

[0123] In the formula, ||n over || is the plane normal vector n over The model.

[0124] If the resolution of the first-layer bridging image does not meet the preset resolution requirement (the preset resolution can be set as needed), the position parameters of the second-layer bridging image need to be calculated. The second-layer bridging image needs to plan more shooting points within the coverage area of ​​the first-layer bridging image, while ensuring an overlap rate r between the images (the overlap rate r can be set as needed). To increase the resolution, the distance D2 from the camera to the plane can be reduced (representing the distance from the position of the second-layer bridging image to the target in the overlapping area). The coverage area of ​​the second-layer bridging image is calculated as follows:

[0125] D2 = D1 / 2,

[0126]

[0127] Calculate the number of rows N required for the second-layer bridging image. row Number of columns N col :

[0128]

[0129] Location parameters of the second-layer bridging image (x) i ,y j z2) is:

[0130]

[0131] z2 = p c +D2·n3,

[0132] In the formula, W c2 H represents the coverage width of the second-layer bridging image. c2 The height of the coverage area of ​​the second-layer bridging image, i = 0, 1, 2, ..., N col -1, j = 0, 1, 2, ..., N row -1.

[0133] If the resolution of the second-layer bridging image has not yet reached the preset resolution requirement, the position parameters of the next-layer bridging image are calculated based on the position parameters of the second-layer bridging image, the overlapping area, and the basic parameters, until the resolution of the next-layer bridging image reaches the preset resolution requirement. It is worth noting that the calculation process for the position parameters of the next-layer bridging image is similar to the calculation process for the position parameters of the second-layer bridging image described above, and will not be repeated here.

[0134] Furthermore, before basing the method on the position parameters of the bridging image and the trained 3D Gaussian rendering model, the method further includes:

[0135] S401, using the aerial triangulation results of the aerial image, the 3D Gaussian rendering model is trained to obtain the trained 3D Gaussian rendering model.

[0136] Specifically, step S401 includes:

[0137] Using the aerial triangulation results of the aerial images, training data for a 3D Gaussian rendering model is constructed.

[0138] Based on the training data, the 3D Gaussian rendering model is trained using an optimization algorithm to obtain the trained 3D Gaussian rendering model.

[0139] For example, training data for a 3D Gaussian rendering model is constructed using image intrinsic parameters, coordinate system transformation parameters, and sparse point cloud data from aerial triangulation results of aerial imagery. The model is trained by optimizing the algorithm until it can generate images that conform to the characteristics of aerial imagery based on the input position parameters (when the 3D Gaussian rendering model converges), thus obtaining the trained 3D Gaussian rendering model.

[0140] This invention discloses a method for generating air-to-ground bridging imagery, which involves acquiring aerial triangulation results corresponding to aerial and ground images of a target area. The aerial triangulation results include image intrinsic parameters, coordinate system transformation parameters, and sparse point cloud data of the aerial and ground images. Based on the sparse point cloud data, the aerial and ground images are spatially aligned to obtain target aerial and target ground images. The target aerial and target ground images are then projected to determine their overlapping region. Based on the overlapping region, the position parameters of the bridging imagery are calculated. Finally, based on the position parameters of the bridging imagery and a trained 3D Gaussian rendering model, a bridging imagery of the target aerial and target ground images is generated. It can achieve spatial alignment of aerial and ground images through planar primitives, avoiding the instability of accuracy caused by manually adding connection matching points, reducing labor costs. At the same time, it introduces intermediate bridging images to connect aerial and ground images across the field of view, solving the problem of mismatch caused by large differences in the perspective of aerial and ground images across the field of view. It does not require a significant increase in the original image data of UAVs, reducing the waste of labor costs and avoiding the problem of limited weak signal acquisition, thereby improving the accuracy of the 3D reconstruction model.

[0141] See Figure 2 , Figure 2 This is a schematic diagram of the structure of an air-to-ground bridging image generation device 10 provided in an embodiment of the present invention. The air-to-ground bridging image generation device 10 includes:

[0142] The aerial triangulation result acquisition module 11 is used to acquire the aerial triangulation results corresponding to the aerial images and ground images of the target area; wherein, the aerial triangulation results include the image intrinsic parameters, coordinate system transformation parameters and sparse point cloud data of the aerial images and ground images;

[0143] The target image alignment module 12 is used to spatially align the aerial image and the ground image based on the sparse point cloud data to obtain the target aerial image and the target ground image.

[0144] The position parameter calculation module 13 is used to project the target aerial image and the target ground image, determine the overlapping area of ​​the target aerial image and the target ground image, and calculate the position parameters of the bridging image based on the overlapping area.

[0145] The bridging image generation module 14 is used to generate a bridging image of the target aerial image and the target ground image based on the position parameters of the bridging image and the trained 3D Gaussian rendering model.

[0146] Furthermore, the air-to-ground bridging image generation device 10 also includes:

[0147] The rendering model training module is used to train the 3D Gaussian rendering model using the aerial triangulation results of the aerial imagery, and obtain the trained 3D Gaussian rendering model.

[0148] The air-to-ground bridging image generation apparatus 10 provided in this embodiment of the invention can realize all the processes of the air-to-ground bridging image generation method of the above embodiments. The functions and technical effects of each module in the apparatus are the same as those of the air-to-ground bridging image generation method of the above embodiments, and will not be repeated here.

[0149] See Figure 3 , Figure 3 This is a schematic diagram of the structure of an air-to-ground bridging image generation device 20 provided in an embodiment of the present invention. The air-to-ground bridging image generation device 20 of this embodiment includes: a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the above-described air-to-ground bridging image generation method embodiment. Alternatively, when the processor 21 executes the computer program, it implements the functions of each module in the above-described air-to-ground bridging image generation apparatus embodiment.

[0150] For example, the computer program may be divided into one or more modules, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the air-to-ground bridging image generation device 20.

[0151] The air-to-ground bridging image generation device 20 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The air-to-ground bridging image generation device 20 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the air-to-ground bridging image generation device 20 and does not constitute a limitation on the device. It may include more or fewer components than illustrated, or combine certain components, or use different components. For example, the air-to-ground bridging image generation device 20 may also include input / output devices, network access devices, buses, etc.

[0152] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The processor 21 is the control center of the air-to-ground bridging image generation device 20, connecting all parts of the air-to-ground bridging image generation device 20 via various interfaces and lines.

[0153] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the air-to-ground bridging image generation device 20 by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0154] The module integrated into the air-to-ground bridging image generation device 20, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable medium may be appropriately added to or subtracted from the content as required by the legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium may not include electrical carrier signals and telecommunication signals.

[0155] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0156] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the method for generating air-to-ground bridging images as described in the above embodiments.

[0157] Furthermore, embodiments of the present invention also provide a computer program product, which is stored in a storage medium and executed by at least one processor to implement the steps of the method for generating air-to-ground bridging images described in the above embodiments.

[0158] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for generating air-to-ground bridging imagery, characterized in that, include: Aerial triangulation results are obtained for aerial and ground images of the target area; wherein, the aerial triangulation results include the image intrinsic parameters, coordinate system transformation parameters, and sparse point cloud data of the aerial and ground images; Based on the sparse point cloud data, construct planar primitive pairs corresponding to the aerial image and the ground image; based on the planar primitive pairs, calculate the transformation parameters of the aerial image and the ground image to spatially align the aerial image and the ground image to obtain the target aerial image and the target ground image. The aerial and ground images of the target are projected to determine the overlapping area between them; the position parameters of the bridging image are calculated based on the overlapping area. Based on the position parameters of the bridging image and the trained 3D Gaussian rendering model, a bridging image of the target aerial image and the target ground image is generated. The step of calculating the position parameters of the bridging image based on the overlapping region includes: Set the basic parameters of the bridging image; wherein the basic parameters include image focal length, image resolution, pixel size, horizontal viewing angle and vertical viewing angle; If the current layer bridging image is the first layer bridging image, calculate the position parameters of the current layer bridging image based on the overlapping area and the basic parameters; If the resolution of the current layer bridging image does not meet the preset resolution requirement, the position parameters of the next layer bridging image are calculated based on the position parameters of the current layer bridging image, the overlapping area, and the basic parameters, until the resolution of the next layer bridging image meets the preset resolution requirement.

2. The method for generating air-to-ground bridging images as described in claim 1, characterized in that, The step of constructing planar primitive pairs corresponding to the aerial and ground images based on the sparse point cloud data includes: Planar features are extracted from the sparse point cloud data to obtain the corresponding aerial plane set and ground plane set; Based on the multi-scale geometric feature relationships of the planar primitives in the aerial planar set and the ground planar set, planar primitive pairs corresponding to the aerial image and the ground image are constructed.

3. The method for generating air-to-ground bridging images as described in claim 1, characterized in that, The projection of the target aerial image and the target ground image to determine the overlapping area of ​​the target aerial image and the target ground image includes: A ray casting algorithm is used to project the target aerial image and the target ground image, and to determine the image index of each pixel in the target aerial image and the target ground image; The area where the image indexes of both the target aerial image and the target ground image exist simultaneously is defined as the overlapping area of ​​the target aerial image and the target ground image.

4. The method for generating air-to-ground bridging images as described in claim 1, characterized in that, If the current layer bridging image is the first layer bridging image, the position parameters of the current layer bridging image are calculated based on the overlapping area and the basic parameters, including: If the current layer bridging image is the first layer bridging image, calculate the target distance between the current layer bridging image and the overlapping area based on the size of the overlapping area, the horizontal viewing angle, and the vertical viewing angle. The position parameters of the current layer bridging image are calculated based on the target distance, the centroid position of the bounding box of the overlapping region, and the plane normal vector.

5. An apparatus for generating air-to-ground bridging images, characterized in that, include: The aerial triangulation result acquisition module is used to acquire aerial triangulation results corresponding to aerial and ground images of the target area; wherein, the aerial triangulation results include the image intrinsic parameters, coordinate system transformation parameters and sparse point cloud data of the aerial and ground images; The target image alignment module is used to construct planar primitive pairs corresponding to the aerial image and the ground image based on the sparse point cloud data; and to calculate the transformation parameters of the aerial image and the ground image based on the planar primitive pairs, so as to spatially align the aerial image and the ground image to obtain the target aerial image and the target ground image. The position parameter calculation module is used to project the target aerial image and the target ground image, determine the overlapping area of ​​the target aerial image and the target ground image, and calculate the position parameters of the bridging image based on the overlapping area. The bridging image generation module is used to generate a bridging image of the target aerial image and the target ground image based on the position parameters of the bridging image and the trained 3D Gaussian rendering model. The step of calculating the position parameters of the bridging image based on the overlapping region includes: Set the basic parameters of the bridging image; wherein the basic parameters include image focal length, image resolution, pixel size, horizontal viewing angle and vertical viewing angle; If the current layer bridging image is the first layer bridging image, calculate the position parameters of the current layer bridging image based on the overlapping area and the basic parameters; If the resolution of the current layer bridging image does not meet the preset resolution requirement, the position parameters of the next layer bridging image are calculated based on the position parameters of the current layer bridging image, the overlapping area, and the basic parameters, until the resolution of the next layer bridging image meets the preset resolution requirement.

6. A device for generating air-to-ground bridging images, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the method for generating air-to-ground bridging imagery as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the method for generating air-to-ground bridging images as described in any one of claims 1-4.

8. A computer program product, characterized in that, The computer program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the method for generating air-to-ground bridging imagery as described in any one of claims 1-4.

Images