Three-dimensional modeling method based on fusion of space-based image and Internet image

Abstract

The embodiment of the invention provides a three-dimensional modeling method based on fusion of a space-based image and an internet image. The three-dimensional modeling method comprises the following steps: performing connection point matching on each satellite image in a multi-angle satellite image set to obtain a matched multi-angle satellite image set; performing adjustment processing on each matched satellite image in the matched multi-angle satellite image set to obtain an adjusted multi-angle satellite image; dense point cloud extraction is carried out on the multi-angle satellite image after adjustment to obtain dense point cloud of the satellite image; performing camera pose processing on each open source Internet image in the open source Internet image set through a multi-view matching algorithm to generate Internet image dense point cloud; based on the dense point cloud of the satellite image and the dense point cloud of the internet image, generating a fused point cloud; and based on the fused point cloud, generating a fused image three-dimensional model with textures. According to the method, the satellite image and the internet image are combined, so that the three-dimensional modeling fineness is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to a three-dimensional modeling method based on the fusion of space-based imagery and Internet images. Background Technology

[0002] With the increasing demand for high-precision 3D scene modeling in fields such as geographic information mapping, disaster emergency monitoring, and overseas target analysis, image-based 3D reconstruction technology has become a core means of acquiring scene spatial information. Currently, mainstream image-based 3D reconstruction technologies mainly rely on two types of data sources: space-based Earth observation satellite imagery and open-source imagery from the Internet / network reconnaissance. However, both types of data sources have significant limitations in their individual applications, making it difficult to meet the refined modeling requirements of "large-area coverage + high-precision details + rich textures." Using multi-view satellite images acquired through space-based Earth observation for 3D reconstruction has advantages such as large single acquisition area, periodic observation, and no geographical limitations, making it suitable for large-area and overseas target 3D modeling. Its disadvantages include low local resolution, limited observation angles, poor modeling effect, and weak texture information. Using images acquired through network reconnaissance or open-source Internet sources for 3D reconstruction has advantages such as strong real-time performance, high local accuracy, and rich textures. However, it also faces difficulties due to various challenges such as cluttered, disordered, parameterless, and lack of positioning information in the images. Summary of the Invention

[0003] To address the aforementioned issues, this paper proposes a method for 3D modeling based on the fusion of space-based imagery and internet images. Through an improved registration method, dense point clouds from both imagery sources are combined to form a 3D surface model, which is then seamlessly textured.

[0004] One aspect of the present invention provides a method for 3D modeling based on the fusion of space-based imagery and Internet images, comprising:

[0005] Step 101: Acquire a multi-angle satellite image set;

[0006] Step 102: Perform tie point matching on each satellite image in the multi-angle satellite image set to obtain the matched satellite image set;

[0007] Step 103: Perform adjustment processing on each matched satellite image in the matched satellite image set to obtain adjusted multi-angle satellite images;

[0008] Step 104: Extract dense point clouds from the adjusted multi-angle satellite images to obtain dense point clouds of satellite images;

[0009] Step 105: Obtain an open-source internet image set;

[0010] Step 106: Using a multi-viewpoint matching algorithm, perform camera pose processing on each open-source internet image in the open-source internet image set to obtain geometric reference information;

[0011] Step 107: Generate a dense point cloud of the Internet image based on the geometric reference information;

[0012] Step 108: Generate a fused point cloud based on the dense point cloud of the satellite imagery and the dense point cloud of the internet imagery;

[0013] Step 109: Based on the fused point cloud, generate a textured fused image 3D model.

[0014] Optionally, the step of matching the connection points of each satellite image in the multi-angle satellite image set to obtain a matched multi-angle satellite image set includes: generating initial affine transformation relationship information based on each satellite image in the satellite image set, a pre-constructed image pyramid, and satellite image parameters; generating a preliminary set of corresponding image points based on the initial affine transformation relationship information, each satellite image, and the top-level data of the image pyramid; filtering the preliminary set of corresponding image points to obtain a filtered set of corresponding image points and updated affine transformation relationship information; generating a full-layer set of corresponding image points based on the updated affine transformation relationship information, the lower-layer data of the image pyramid, and the lower-layer images of each satellite image; and generating the matched satellite image set based on the full-layer set of corresponding image points and the image-space coordinate system information of each satellite image.

[0015] Optionally, the adjustment process for each matched satellite image in the matched satellite image set to obtain an adjusted satellite image set includes: generating initial values ​​of affine transformation parameters for each satellite image and initial ground 3D coordinates for each matched connection point based on the matching tie points and basic geometric information of each matched satellite image; generating a nonlinear error equation set based on the initial values ​​of the affine transformation parameters for each satellite image, the initial ground 3D coordinates for each matched connection point, and rational function model parameters, wherein the linear error equation set includes model control point error equations and model tie point error equations; linearizing the nonlinear error equation set to obtain a linearized error equation set; and performing iterative adjustment based on the linearized error equation set and a preset convergence threshold to obtain an adjusted satellite image set, wherein the adjusted satellite image set includes the precise geometric parameters of each satellite image and the precise ground 3D coordinates of each tie point.

[0016] Optionally, the step of extracting dense point clouds from the adjusted multi-angle satellite images to obtain dense point clouds of satellite images includes: selecting one adjusted satellite image from the adjusted multi-angle satellite images as a reference image, and using the remaining satellite images in the satellite image set as a search image set; selecting a target image point on the reference image; obtaining the initial elevation value of the object space corresponding to the target image point through pre-matching; constructing a phase point spatial beam of the target image point through the target image point camera station; generating an initial point in the object space based on the plane of the phase point spatial beam and the initial elevation value; and generating an initial point in the object space based on the initial elevation value. Determine the elevation error to obtain the elevation range; based on the elevation range and a fixed elevation step size, generate a preset number of discrete elevation values ​​to obtain a discrete elevation value set; based on the discrete elevation value set and the phase point spatial beam, generate image points corresponding to each search image in the search image set to obtain an image point set; generate a matching measure between the reference image and each image point in the image point set to obtain a matching measure result; based on the matching measure result and a preset matching measure threshold range, generate effective object space points for each image point to obtain an effective object space point set; construct a dense point cloud based on the effective object space point set to obtain the satellite image dense point cloud.

[0017] Optionally, the step of processing the camera pose of each open-source internet image in the open-source internet image set using a multi-viewpoint matching algorithm to obtain geometric reference information includes: performing ordered preprocessing on each open-source internet image in the open-source internet image set to obtain an ordered internet image set and inter-image matching feature points; generating an adjusted camera pose and a sparse point cloud of a 3D scene based on the ordered internet image set and inter-image matching feature points, and using the adjusted camera pose as the geometric reference information.

[0018] Optionally, generating a dense point cloud of internet images based on the geometric reference information includes: generating the viewpoint overlap of each ordered internet image in the ordered internet image set based on the geometric reference information to obtain a set of matching image pairs; performing stereo matching on each matching image pair in the set of matching image pairs based on a depth map fusion algorithm, and generating the object-space depth of each pixel on each image in each matching image pair to obtain a local depth map; and combining the local depth map and the sparse point cloud of the three-dimensional scene to obtain the dense point cloud of the internet images.

[0019] Optionally, generating a fused point cloud based on the dense point cloud of the satellite imagery and the dense point cloud of the internet imagery includes: preprocessing the dense point cloud of the satellite imagery and the dense point cloud of the internet imagery to obtain preprocessed dense point cloud of the satellite imagery and preprocessed dense point cloud of the internet imagery; extracting key points from the preprocessed dense point cloud of the satellite imagery and the preprocessed dense point cloud of the internet imagery using the Harris 3D feature extraction algorithm to obtain a point set of the satellite imagery and a point set of the internet imagery, wherein both the point set of the satellite imagery and the point set of the internet imagery include a key point set and a corresponding FPFH descriptor set; coarsely registering the point set of the satellite imagery and the point set of the internet imagery based on the SAC-IA algorithm to obtain a coarsely registered heterogeneous point cloud pair; performing the following steps on the coarsely registered heterogeneous point cloud pair: selecting a preset number of points from the coarsely registered heterogeneous point cloudry through random sampling to obtain a reference point set; for each reference point in the reference point set, determining its relationship with the... In the preprocessed dense point cloud of the internet image, the points with the closest Euclidean distance are used to obtain a set of corresponding point pairs. The corresponding point sets are then filtered to obtain a valid set of corresponding point pairs. Using the centroid calculation formula, the centroids of the reference point set and the corresponding point set are generated respectively, resulting in the reference centroid and the centroid of the corresponding point. Based on minimizing the objective function, a rotation matrix and a translation matrix are generated according to the valid set of corresponding point pairs, the reference centroid, the centroid of the corresponding point, and the preprocessed dense point cloud of the satellite image. Based on the transformation formula, a transformed point is generated using the rotation matrix, the translation matrix, and a single reference point in the reference point set. The current error is generated based on the transformed point and the single reference point. The above steps are repeated. In response to the difference between the current error and the previous error being less than a preset threshold, a registered heterogeneous point cloud is generated, whereby the registered heterogeneous point cloud includes the dense points of the satellite image and the aligned internet point cloud. The registered heterogeneous point clouds are merged to obtain a preliminary combined point cloud. Redundancy is removed from the preliminary combined point cloud to obtain the fused point cloud.

[0020] Optionally, generating a textured fused image 3D model based on the fused point cloud includes: preprocessing the fused point cloud to obtain a preprocessed fused point cloud; performing triangulation on the preprocessed fused point cloud to obtain an initial triangulation surface and surface information, wherein the surface information includes a forward front and a priority queue of candidate triangles; performing the following steps on the initial triangulation surface: determining triangles that meet preset criteria from the priority queue of candidate triangles to obtain candidate triangles; adding the candidate triangles to the initial triangulation surface to obtain an updated surface; extracting the non-coincident edges of the candidate triangles to obtain an updated forward front; and generating new candidate triangles based on the updated forward front to obtain... An updated priority queue of candidate triangles is established. In response to the updated priority queue not containing any triangles meeting preset criteria, a complete 3D surface model is obtained. Based on the surface triangular mesh and image calibration information, a visibility image set for each triangle facet and a surface triangular mesh matching the surface are acquired. Based on the visibility image set for each triangle facet and the surface triangular mesh matching the surface, a texture patch and an optimal reference image for each triangle facet are generated. Based on the texture patch and the optimal reference image for each triangle facet, an initial texture image is generated. Gaps in the initial texture image are removed to obtain a seamless texture image. The seamless texture image is mapped onto the complete 3D surface model to obtain the textured fused image 3D model.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided. Computer program instructions are stored on the computer-readable storage medium, wherein, when executed by a processor, the computer program instructions cause the processor to perform the three-dimensional modeling method according to the first aspect of the present invention.

[0022] The inventive points and technical effects of this invention:

[0023] The 3D modeling method and related equipment based on the fusion of space-based imagery and Internet images provided by the embodiments of the present invention can utilize space-based imagery to provide a large-scale basic framework, and combine Internet images to supplement local high-precision details, making up for the limitations of a single data source in terms of viewpoint, resolution, occlusion, and texture details, and achieving high-precision, multi-detail refined 3D modeling.

[0024] Further aspects and scope of adaptability become apparent from the description provided herein. It should be understood that various aspects of this application may be implemented individually or in combination with one or more other aspects. It should also be understood that the descriptions and specific embodiments herein are for illustrative purposes and are not intended to limit the scope of this application. A three-dimensional surface model is used to obtain the textured fused image three-dimensional model. Attached Figure Description

[0025] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0026] Figure 1 A flowchart of a three-dimensional modeling method for fusing space-based imagery and Internet images, according to an embodiment of the present invention, is shown. Detailed Implementation

[0027] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0028] like Figure 1 As shown, a 3D modeling method based on the fusion of space-based imagery and internet images in an embodiment of the present invention includes:

[0029] Step 101: Acquire a multi-angle satellite image set. The multi-angle satellite image set can be obtained by acquiring multiple images of the same target area from different observation angles (azimuth angle, zenith angle). The acquisition method mainly depends on the satellite's orbital design, attitude control capabilities, and imaging system configuration (e.g., single-satellite "multi-pass" observation (acquiring multiple perspectives in the time dimension) and multi-satellite "cooperative observation" (acquiring multiple perspectives in the spatial dimension)).

[0030] Step 102: Perform tie point matching on each satellite image in the multi-angle satellite image set to obtain the matched multi-angle satellite image set.

[0031] In some optional embodiments, the step of matching the connection points of each satellite image in the multi-angle satellite image set to obtain the matched multi-angle satellite image set may include:

[0032] The first step is to generate initial affine transformation relationship information based on the various satellite images in the multi-angle satellite image set, the pre-constructed image pyramid, and the satellite image parameters. The satellite image parameters can be the RPC (Rational Polynomial Coefficients) parameters of the satellite images (used for "RPC guidance," providing an initial geometric reference). In practice, the steps for generating the initial affine transformation relationship information can be as follows: First, guided by the RPC parameters, perform SURF (Accelerated Robust Feature) extraction and initial matching on the multi-angle satellite images, quickly calculating through feature similarity; then, establish the initial affine transformation relationship between the reference image and the search image (describing the geometric mapping rules between the two images), i.e., the initial affine transformation relationship information.

[0033] The second step involves generating a preliminary set of corresponding image points based on the initial affine transformation relationship information, the various satellite images, and the top-level data of the image pyramid. In practice, feature points can first be extracted from the reference image (constructing a "feature point library" to be matched), where the reference image can be one image in the satellite image set; then, using the initial affine transformation relationship information as geometric constraints, the window (local pixel region) containing the feature points of the reference image is mapped to the search image, where the search image can be another image that is matched with the reference image (it needs to be aligned with the reference image through geometric transformation); then, the corresponding window of the search image is sampled and corrected to the image-space coordinate system of the reference image (unifying the coordinate reference and reducing geometric deviation); finally, correlation coefficient matching is performed on the corrected window (sifting similar windows through pixel grayscale correlation to determine potential corresponding points) to obtain the preliminary set of corresponding image points.

[0034] The third step involves filtering the initial set of corresponding image points to obtain a filtered set of corresponding image points and updated affine transformation relationship information. In practice, a polynomial iterative method can be used to remove gross errors from the initial set of corresponding image points (identifying and deleting incorrect matching points caused by noise, occlusion, etc.) to obtain filtered valid corresponding image points. Then, using the filtered valid corresponding image points, the affine transformation relationship between the reference image and the search image is recalculated, the geometric mapping model is updated, and the updated affine transformation relationship information is obtained.

[0035] The fourth step involves generating a set of corresponding image points for each layer based on the updated affine transformation relationship information, the lower-level data of the image pyramid, and the lower-level images of each satellite image. In practice, the updated affine transformation relationship information can be used as the new geometric constraint to repeat the window transformation, sampling correction, and correlation coefficient matching process for each lower-level image of the image pyramid, achieving hierarchical progressive matching (the top layer constrains the lower layers, ensuring consistency in matching across layers).

[0036] The fifth step involves generating the matched multi-angle satellite image set based on the set of corresponding image points across all layers and the image-space coordinate system information of each satellite image. In practice, the matching results of corresponding image points at each layer of the image pyramid can first be uniformly converted to the image-space coordinate system of the search image (achieving coordinate unification for all matching points); finally, least-squares matching is performed on the converted corresponding image points to further optimize the matching accuracy and eliminate residual bias by minimizing coordinate errors, thus obtaining the matched multi-angle satellite image set.

[0037] Step 103: Perform adjustment processing on each matched satellite image in the matched multi-angle satellite image set to obtain the adjusted multi-angle satellite image.

[0038] In some optional embodiments, the adjustment processing of each matched satellite image in the matched multi-angle satellite image set to obtain the adjusted multi-angle satellite image set includes:

[0039] The first step involves generating initial values ​​for the affine transformation parameters of each satellite image and the initial 3D ground coordinates of each matching tie point based on the matching tie points and the underlying geometric information of the satellite images. The underlying geometric information of the satellite images can be the initial affine transformation prototype obtained from the initial matching, image RPC parameters, etc., which can be used to assist in initial value estimation. In practice, based on the image-side coordinate relationship of the matching tie points and combined with the initial geometric reference of the images (such as RPC parameters), the initial values ​​for the affine transformation parameters of each satellite image (describing the preliminary geometric mapping between the image-side and object-side) are determined. Then, using the inverse calculation of the affine transformation parameters or the RPC model, combined with the image-side coordinates of the tie points, the initial 3D ground coordinates of each tie point (initial estimation results of the object-side X, Y, and Z coordinates) can be preliminarily determined.

[0040] The second step involves generating a set of nonlinear error equations based on the initial values ​​of the affine transformation parameters of each satellite image, the initial 3D ground coordinates of each matching tie point, and the rational function model parameters. This set of linear error equations includes error equations for model control points and model tie points. The rational function model parameters can be RFM (Reasonable Function Model) parameters of the satellite image (such as RPC coefficients). In practice, firstly, the error equations for the control points can be listed using the rational function model parameters as a mathematical basis, starting from the image-side coordinates of the control points. Then, the error equations for the tie points can be listed from the initial 3D ground coordinates of each matching tie point (since the 3D ground coordinates of the tie points are unknown, both the image geometric parameters and the object-side coordinates of the tie points need to be optimized simultaneously; therefore, the equation form differs from that of the control points and must include unknown object-side coordinates). Finally, the error equations of all images, all control points, and all tie points are integrated to form the system of equations for adjustment calculations, i.e., the set of nonlinear error equations.

[0041] The third step is to linearize the nonlinear error equations to obtain a linearized set of error equations. In practice, the initial values ​​of the affine transformation parameters and the initial three-dimensional ground coordinates of each matching connection point can be used as the initial points for iteration. Mathematical methods such as Taylor expansion can be employed to transform the nonlinear error equations into a linearized set of error equations (converting the nonlinear problem into a linear problem solvable through matrix operations). This ensures that the parameter correction can be calculated using the least squares principle to obtain the linearized set of error equations.

[0042] The fourth step involves iterative adjustment based on the linearized error equations and a preset convergence threshold to obtain an adjusted multi-angle satellite image set. This set includes the precise geometric parameters of each satellite image and the precise ground 3D coordinates of each connection point. The preset convergence threshold can be a fixed value and is not specifically limited. In practice, firstly, the linearized error equations can be solved using the least squares principle to obtain corrections for the image affine transformation parameters and the ground 3D coordinates of the connection points. Then, the corrections are used to update the previous parameter values ​​(initial value + correction = new parameter value). Next, the updated parameter values ​​can be used as the new "initial point" to re-form the error equations, re-linearize, and re-calculate the corrections, entering the next iteration. Finally, the iteration continues until the parameter correction is less than the convergence threshold and the overall error meets the accuracy requirements, at which point the adjustment is considered converged and the iteration stops.

[0043] Step 104: Extract dense point clouds from the adjusted multi-angle satellite images to obtain dense point clouds of satellite images.

[0044] In some optional embodiments, the step of extracting dense point clouds from the adjusted multi-angle satellite imagery to obtain dense point clouds of satellite imagery includes:

[0045] The first step involves selecting one adjusted satellite image from the multi-angle satellite imagery as the reference image, and using the remaining satellite images in the multi-angle satellite imagery set as the search image set. Both the reference image and the search image set include their respective camera stations.

[0046] The second step is to select a target image point on the reference image. In practice, the target image point is selected randomly without specific limitations.

[0047] The third step involves obtaining the initial elevation value of the object space corresponding to the target image point through pre-matching. In practice, the initial elevation value of the object space corresponding to the target image point can be obtained through pre-matching or pyramid matching, and this initial elevation value carries a fixed error Z. Δ .

[0048] The fourth step is to construct the phase point spatial beam of the target image point using the target image point camera station. In practice, the beam C0P0 can be constructed starting from image point P0 of the reference image I0.

[0049] Fifth, based on the plane of the phase point spatial beam and the initial elevation value, generate the initial point of the object space. In practice, the beam C0P0 can be made to intersect the plane of the initial elevation Z0 to obtain the initial point P0(X0,Y0,Z0) of the object space. Finally,

[0050] Step 6: Determine the elevation error based on the initial elevation value to obtain the elevation range. In practice, this can be done based on the fixed error Z. Δ The elevation range is determined to be (Z0-Z1). Δ ,Z0+Z Δ ).

[0051] Step 7: Based on the elevation range and fixed elevation step size, generate a preset number of discrete elevation values ​​to obtain a discrete elevation value set. The fixed elevation step size is a fixed value and is not specifically limited. In practice, within the elevation range (Z0-Z1),... Δ ,Z0+Z Δ Within the specified range, multiple elevation values ​​are selected using the fixed elevation step size, forming the discrete elevation value set. Finally, the matching measure of these image points (reflecting the similarity between image points) is calculated to establish the correspondence between "image points and elevation changes".

[0052] Step 8: Based on the discrete elevation value set and the phase point spatial beam, generate image points corresponding to each search image in the search image set to obtain an image point set. In practice, for each elevation value in the discrete elevation value set, the object-side point on the C0PO beam can be projected onto the image-side of each search image to obtain the corresponding image points.

[0053] The ninth step is to generate the matching measure of the reference image and each image point in the image point set, and obtain the matching measure result. In practice, the matching measure of each image point can be obtained by generating the matching measure result to establish the correspondence between "image point-elevation change".

[0054] Step 10: Based on the matching metric results and the preset matching metric threshold range, generate valid object points for each image point to obtain a set of valid object points. In practice, the elevation value can be determined to be correct by judging whether the matching metric meets the threshold range. If it does, the corresponding object point is determined to be a valid point in the dense point cloud. Then, repeat steps 2 to 10 on the reference image I0 to gradually accumulate all object points that meet the conditions.

[0055] Step 11: Construct a dense point cloud based on the set of valid object points to obtain the dense point cloud of the satellite image. In practice, the construction of the dense point cloud of the satellite image can be completed by summarizing all valid object points.

[0056] Step 105: Obtain an open-source internet image dataset. This open-source internet image dataset can be an image / video dataset that is publicly shared online and can be freely used for compliant scenarios such as research and development (e.g., 3D reconstruction). It can be obtained directly by downloading from the internet.

[0057] Step 106: Using a multi-viewpoint matching algorithm, perform camera pose processing on each open-source internet image in the open-source internet image set to obtain geometric reference information.

[0058] In some optional embodiments, the step of performing camera pose processing on each open-source internet image in the open-source internet image set using a multi-viewpoint matching algorithm to obtain geometric reference information includes:

[0059] The first step involves performing ordered preprocessing on each open-source internet image in the aforementioned open-source internet image set to obtain an ordered internet image set and image-matching feature points. This preprocessing may include feature point extraction and description for each unordered image (e.g., using algorithms such as SIFT and ORB to extract salient features like corners and edges and generate feature descriptors), rapid feature point retrieval and matching based on the feature descriptors (finding "corresponding feature points" between different images through similarity comparison), and determining the image capture order based on the matching results of corresponding feature points (establishing temporal / spatial relationships between images), establishing connections between images, and initially determining the basic spatial relationships of all images (e.g., "which image is adjacent to which image from the same viewpoint"). The image-matching feature points can be pairs of corresponding feature points between different images.

[0060] The second step involves generating an adjusted camera pose and a sparse point cloud of the 3D scene based on the ordered internet image set and the matching feature points between the images, using the adjusted camera pose as the geometric reference information. In practice, the adjusted camera pose and the sparse point cloud of the 3D scene can be generated using SFM (structure from motion). The process can be as follows: First, recover the camera pose. This can be done by back-calculating the camera position (camera station coordinates) and attitude (angle parameters) of each ordered image based on the spatial correspondence of the matching feature points (i.e., "recovering the camera motion trajectory"); then, solve for the object-side coordinates. Based on the camera pose and the image-side coordinates of the matching feature points, calculate the object-side 3D coordinates of the feature points in the "global coordinate system" (forming a preliminary point cloud skeleton); finally, optimize using bundle adjustment. A global adjustment can be performed on the camera pose and the object-side coordinates of the feature points (i.e., adjustment processing of the extracted feature points in the text), minimizing the overall geometric error to optimize the accuracy of the camera pose and the feature point coordinates, thus obtaining the adjusted camera pose and the sparse point cloud of the 3D scene.

[0061] Step 107: Generate a dense point cloud of the Internet image based on the geometric reference information.

[0062] In some alternative embodiments, generating a dense point cloud of an internet image based on the geometric reference information includes:

[0063] The first step is to generate the viewpoint overlap of each ordered internet image in the ordered internet image set based on the geometric reference information, thereby obtaining a set of matching image pairs. In practice, the viewpoint overlap of different images in the ordered image set can be determined based on the geometric reference information, and the optimal matching image pairs with the largest overlap area can be selected to obtain the set of matching image pairs.

[0064] The second step involves performing stereo matching on each matching pair in the matching image pair set based on a depth map fusion algorithm. This generates the object-space depth of each pixel in each image within the matching image pair, resulting in a local depth map set. In practice, the MVS method based on depth map fusion can be used to perform stereo matching on each image and its optimal matching pair, calculating the "object-space depth" (i.e., the Z-value in three-dimensional coordinates) corresponding to each pixel in the image, generating multiple local depth maps, and thus obtaining the local depth map set.

[0065] The third step involves combining the local depth map and the sparse point cloud of the 3D scene to obtain the dense point cloud of the internet image. In practice, the local depth map can be combined with the sparse point cloud to supplement areas not covered by the sparse point cloud (such as texture details, smooth surfaces, etc.), and finally all depth information can be integrated to form a point cloud with uniform density and complete details, thus obtaining the dense point cloud of the internet image.

[0066] Step 108: Generate a fused point cloud based on dense point clouds from satellite imagery and dense point clouds from internet images.

[0067] In some optional embodiments, generating the fused point cloud based on the satellite imagery dense point cloud and the internet imagery dense point cloud includes:

[0068] The first step involves preprocessing the dense point cloud from the satellite imagery and the dense point cloud from the internet imagery to obtain preprocessed dense point clouds from the satellite imagery and the internet imagery. This preprocessing may include removing obvious noise points and standardizing the data format.

[0069] The second step involves using the Harris 3D feature extraction algorithm to extract key points from both the preprocessed satellite image dense point cloud and the preprocessed internet image dense point cloud, resulting in point sets for the satellite image point cloud and internet images. Both the satellite image point cloud set and the internet image point set include a key point set and a corresponding FPFH (Fast Point Feature Histogram) descriptor set. In practice, the Harris 3D feature extraction algorithm can be used to select geometrically significant key points (such as edge points, corner points, and abrupt surface changes) from the point cloud. Then, for each extracted key point, its FPFH descriptor is calculated. The FPFH generates a rotation-invariant feature vector by statistically analyzing the point cloud normal vectors and distances within the key point's neighborhood.

[0070] The third step involves coarsely registering the point cloud set of the satellite imagery and the point set of the Internet imagery based on the SAC-IA algorithm to obtain coarsely registered heterogeneous point cloud pairs. In practice, firstly, feature matching can be used. By comparing the FPFH descriptors of key points in two types of point clouds (calculating descriptor similarity), key point pairs with similar features can be found (i.e., a key point in the satellite point cloud and a key point in the Internet point cloud share the same object space). Then, coarse registration calculation is performed. The SAC-IA (Sample Consistency Initial Alignment) algorithm can be used to remove outliers from the matched key point pairs (removing incorrect pairs caused by misjudgment of feature similarity) and calculate the initial transformation matrix (including translation, rotation, and scale parameters) based on the inliers (correct pairs). Finally, preliminary alignment is performed. The initial transformation matrix can be applied to the Internet image point cloud to initially align it to the vicinity of the coordinate system of the satellite image point cloud, significantly reducing the position, pose, and scale differences between the two types of point clouds (solving the problem of "initial position sensitivity" in subsequent fine registration algorithms). Finally, the coarsely registered heterogeneous point cloud pairs and the initial transformation matrix are obtained.

[0071] The fourth step involves performing the following steps on the coarsely registered heterogeneous point cloud pairs:

[0072] The first sub-step involves selecting a preset number of points from the coarsely registered heterogeneous point cloud through random sampling to obtain a reference point set. The preset number can be adjusted according to actual conditions and is not specifically limited.

[0073] The second sub-step involves determining, for each reference point in the reference point set, the point with the closest Euclidean distance to the dense point cloud of the preprocessed Internet image, thus obtaining the corresponding point pair set.

[0074] The third sub-step involves filtering the corresponding point set to obtain a set of valid corresponding point pairs. In practice, this can be achieved by deleting point pairs with a distance greater than a threshold, retaining the valid corresponding point pairs. The threshold is a fixed value and is not specifically limited.

[0075] The fourth sub-step involves generating the centroids of the reference point set and the corresponding point set using the centroid calculation formula, thus obtaining the reference centroid and the corresponding point centroid. The centroid calculation formula can be as follows:

[0076]

[0077] Where, ψ p Let ψ be the centroid of the reference points in the set of reference points. q Let p be the centroid of the corresponding point in the set of corresponding points, N be the number of valid points, and p be the centroid of the corresponding point in the set of corresponding points. i Let q be the i-th reference point. i Let i be the i-th corresponding point.

[0078] The fifth sub-step involves generating a rotation matrix and a translation matrix based on the minimized objective function, using the effective set of corresponding point pairs, the reference centroid, the corresponding point centroids, and the preprocessed dense point cloud of the satellite imagery. The formula for the minimized objective function E can be as follows:

[0079]

[0080] Among them, R k Let T be the rotation matrix for the k-th iteration. k Let n be the translation matrix for the k-th iteration, where k is the iteration number and n is the number of iterations. qi The normal value is the normal value of the corresponding point in the dense point cloud of the satellite image.

[0081] The sixth sub-step involves generating the transformed point based on the transformation formula, using the rotation matrix, the translation matrix, and a single reference point from the reference point set. The transformation formula can be as follows:

[0082]

[0083] Among them, the The point is the transformed point.

[0084] The seventh sub-step involves generating the current error based on the transformed point and the single reference point. In practice, the current error can be generated using the following formula:

[0085]

[0086] Where, d k Let be the error of the k-th iteration.

[0087] The eighth sub-step involves repeating the steps described above.

[0088] Fifth, in response to the difference between the current error and the previous error being less than a preset threshold, a registered heterogeneous point cloud is generated. This registered heterogeneous point cloud includes the dense points from the satellite imagery and the aligned internet point cloud.

[0089] The sixth step is to merge the registered heterogeneous point clouds to obtain a preliminary combined point cloud. In practice, two types of point clouds in the same coordinate system can be merged to form the preliminary combined point cloud containing global information from satellite point clouds and detailed information from internet point clouds.

[0090] Step 7: Redundancy removal is performed on the preliminary combined point cloud to obtain the fused point cloud. In practice, redundancy removal can be achieved by generating Euclidean distances between points and eliminating duplicate points that are too close (such as overlapping points in satellite point clouds and Internet point clouds at the same object location).

[0091] Step 109: Generate a textured fused image 3D model based on the fused point cloud.

[0092] In some alternative embodiments, generating a textured fused image 3D model based on the fused point cloud includes:

[0093] The first step is to preprocess the fused point cloud to obtain a preprocessed fused point cloud. This preprocessing may include using point cloud filtering algorithms (such as Gaussian filtering or statistical filtering) to remove noise points (such as isolated points or measurement error points) from the point cloud, and reducing the number of points through downsampling (such as voxel mesh downsampling) to reduce the computational complexity of subsequent triangulation while preserving core geometric features.

[0094] The second step involves performing triangulation on the preprocessed fused point cloud to obtain an initial triangulated surface and surface information. The surface information includes a forward front and a priority queue of candidate triangles. The triangulation operation can be Delaunay triangulation. In practice, Delaunay triangulation can be performed on the preprocessed fused point cloud first, defining the triangle radius R as the radius of the smallest sphere passing through the three vertices of the triangle and not containing other sampling points. Then, the initial triangulated surface is determined to consist of triangles conforming to the above radius definition. The boundary edges of these triangles together constitute the forward front (i.e., the current edge boundary of the surface, along which the surface will be subsequently expanded). Finally, a greedy algorithm is used: starting with the "Delaunay triangle with the smallest radius," a priority queue of candidate triangles is established, retaining only valid triangles that can be generated on the current forward front (i.e., triangles satisfying the radius constraint).

[0095] The third step involves performing the following steps on the initial triangulated surface:

[0096] The first sub-step involves determining triangles that meet preset criteria from the candidate triangle priority queue, thus obtaining candidate triangles. The preset criteria may include having the smallest radius.

[0097] The second sub-step involves adding the candidate triangles to the initial triangulation surface to obtain the updated surface.

[0098] The third sub-step involves extracting the non-coincident edges of the candidate triangles to obtain the updated advancing front.

[0099] The fourth sub-step involves generating new candidate triangles based on the updated forward front, resulting in an updated candidate triangle priority queue. In practice, a triangle can be formed by searching for nearby point cloud points along a certain edge of the new edge of the updated forward front, thus obtaining the new candidate triangles. These new candidate triangles are then added to the candidate triangle priority queue, resulting in the updated candidate triangle priority queue.

[0100] The fourth step is to obtain a complete three-dimensional surface model in response to the updated candidate triangle priority queue not having any triangles that meet the preset criteria.

[0101] The fifth step involves obtaining a visibility image set for each triangular facet and a surface triangular grid matching the surface, based on the surface triangular grid and image calibration information. The surface triangular grid can be generated during SFM reconstruction (corresponding one-to-one with the triangular faces of the complete 3D surface model), and the image calibration information can be image calibration information retained during SFM reconstruction (such as camera pose and intrinsic parameters). In practice, the triangular faces of the complete 3D surface model can first be associated with the triangular grid reconstructed by SFM to ensure geometric consistency; then, for each triangular facet, a visibility image set is generated based on the image calibration information, excluding images that are occluded or have an excessively off-center view.

[0102] Step 6: Based on the visibility image set of each triangular facet and the surface triangular grid matching the surface, generate a texture patch and the optimal reference image for each triangular facet. In practice, the optimal reference image can be selected first, that is, for each triangular facet, the image with the smallest angle between the center normal of the triangular facet and the camera's line of sight is selected from its visibility image set as the optimal reference image for that triangular facet; then, triangular facet clustering can be performed to generate texture patches, that is, based on the visibility image set of the triangular facet, the optimal reference image, and the neighborhood topology (adjacent triangular faces are grouped into one class if their optimal reference images are the same or their textures are continuous), the triangular faces are clustered into several groups, and a reference image texture patch is generated for each group (that is, the texture region of the corresponding triangular facet group is cropped from the optimal reference image), finally obtaining the texture patch and the optimal reference image for each triangular facet.

[0103] Step 7: Based on the texture patches and the optimal reference image for each triangle, generate an initial texture image. In practice, the texture patches can be sorted first, that is, all generated texture patches are sorted according to their size; then, a texture image framework is generated, that is, the overall size and patch layout of the texture image are planned according to the size of the sorted patches, and a blank texture image is generated; finally, the patch pixels can be weighted, that is, information such as the vertex coordinates of the triangle (to determine the pixel position), the image viewpoint (to correct view distortion), and the model reconstruction error (regions with smaller errors have higher weights) is fused to generate the pixel value of each texture patch, which is then filled into the corresponding position in the blank texture image, and finally the initial texture image is obtained.

[0104] Step 8: Remove the gaps in the initial texture image to obtain a seamless texture image. In practice, a multi-scale space can be established first, that is, the initial texture image is decomposed into multiple resolutions (such as Gaussian pyramid decomposition) to establish a multi-scale space of texture images (texture images at different resolution levels); then cross-scale fusion is performed, that is, in the multi-scale space, the pixels at the patch gaps are weighted and fused (low-resolution levels eliminate overall gaps, while high-resolution levels retain detail clarity) to ensure a natural texture transition at the gaps; finally, the seamless texture image is reconstructed, that is, the fused multi-scale texture information is reconstructed into a single texture image, and the seamless texture image is finally obtained.

[0105] Step nine involves mapping the seamless texture image onto the complete 3D surface model to obtain the textured fused image 3D model. In practice, this can be achieved by establishing a multi-scale space for the texture image T, thus ensuring seamless texture mapping while maintaining sharpness. The formula is as follows:

[0106]

[0107] Where (x,y) are pixel coordinates, σ are scale coordinates, and G(x,y,σ) is a Gaussian function. For an input texture image T(x,y), the output multi-scale image L(x,y,σ) is generated by interpolation to form the transition region of the texture image.

[0108] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0109] In embodiments of this disclosure, a computer-readable storage medium is also provided, on which computer program instructions are stored. When executed, for example, by a processor, these computer program instructions can implement the steps of the 3D modeling method based on the fusion of space-based imagery and internet images described in any of the above embodiments. In some possible implementations, various aspects of this application can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the 3D modeling method based on the fusion of space-based imagery and internet images according to various exemplary embodiments of this application.

[0110] The program product for implementing the above-described method according to embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of this application is not limited thereto. In this document, a readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0111] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0112] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0113] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0114] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0115] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for 3D modeling based on the fusion of space-based imagery and internet images, comprising: Acquire a multi-angle satellite image set; Connectivity points are matched among the satellite images in the multi-angle satellite image set to obtain the matched multi-angle satellite image set. Adjustment processing is performed on each matched satellite image in the matched multi-angle satellite image set to obtain the adjusted multi-angle satellite image. Dense point cloud extraction is performed on the adjusted multi-angle satellite images to obtain dense point cloud of satellite images; Obtain open-source internet image sets; Using a multi-viewpoint matching algorithm, camera pose processing is performed on each open-source internet image in the open-source internet image set to obtain geometric reference information; Based on the geometric reference information, generate a dense point cloud of an internet image; Based on the dense point cloud of the satellite imagery and the dense point cloud of the Internet imagery, a fused point cloud is generated; Based on the fused point cloud, a textured fused image 3D model is generated.

2. The method for 3D modeling based on the fusion of space-based imagery and internet images according to claim 1, wherein, The step of matching the connection points of each satellite image in the multi-angle satellite image set to obtain the matched multi-angle satellite image set includes: Based on the various satellite images in the multi-angle satellite image set, the pre-constructed image pyramid, and the satellite image parameters, initial affine transformation relationship information is generated; Based on the initial affine transformation relationship information, the various satellite images, and the top-level data of the image pyramid, a preliminary set of image points with the same name is generated. The initial set of corresponding image points is filtered to obtain the filtered set of corresponding image points and the updated affine transformation relationship information. Based on the updated affine transformation relationship information, the lower layer data of the image pyramid, and the lower layer images of each satellite image, a set of corresponding image points for the entire layer is generated. Based on the set of identical image points across the entire layer and the image-space coordinate system information of each satellite image, the matched multi-angle satellite image set is generated.

3. The method for 3D modeling based on the fusion of space-based imagery and internet images according to claim 1, wherein, The adjustment process is performed on each matched satellite image in the matched multi-angle satellite image set to obtain the adjusted multi-angle satellite image set, including: Based on the matching connection points of each matched satellite image and the basic geometric information of the satellite image, the initial values ​​of the affine transformation parameters of each satellite image and the initial three-dimensional ground coordinates of each matching connection point are generated. Based on the initial values ​​of the affine transformation parameters of each satellite image and the initial three-dimensional ground coordinates and rational function model parameters of each matching connection point, a set of nonlinear error equations is generated, wherein the set of linear error equations includes the model control point error equation and the model connection point error equation. The nonlinear error equations are linearized to obtain the linearized error equations. Iterative adjustment is performed based on the linearized error equation set and the preset convergence threshold to obtain an adjusted multi-angle satellite image set, wherein the adjusted multi-angle satellite image set includes the precise geometric parameters of each satellite image and the precise ground three-dimensional coordinates of each connection point.

4. The method for 3D modeling based on the fusion of space-based imagery and internet images according to claim 1, wherein, The step of extracting dense point clouds from the adjusted multi-angle satellite images to obtain dense point clouds of satellite images includes: Select one adjusted satellite image from the adjusted multi-angle satellite images as the reference image, and use the remaining satellite images in the multi-angle satellite image set as the search image set; Select a target phase point on the reference image; By pre-matching, the initial elevation value of the object space corresponding to the target phase point is obtained; The phase point spatial beam of the target phase point is constructed by the target phase point camera station; Based on the plane of the phase point spatial beam and the initial elevation value, generate the initial point of the object space; The elevation error is determined based on the initial elevation value, and the elevation range is obtained. Based on the elevation range and fixed elevation step size, a preset number of discrete elevation values ​​are generated to obtain a set of discrete elevation values. Based on the discrete elevation value set and the phase point spatial beam, image points corresponding to each search image in the search image set are generated to obtain the image point set; Generate the matching measure of the reference image and each image point in the image point set, and obtain the matching measure result; Based on the matching measurement results and the preset matching measurement threshold range, the effective object points of each image point are generated to obtain the effective object point set; A dense point cloud is constructed based on the effective object point set to obtain the dense point cloud of the satellite image.

5. The method for 3D modeling based on the fusion of space-based imagery and internet images according to claim 1, wherein, The step involves processing the camera pose of each open-source internet image in the open-source internet image set using a multi-viewpoint matching algorithm to obtain geometric reference information, including: The open-source internet images in the open-source internet image set are preprocessed in an ordered manner to obtain an ordered internet image set and matching feature points between images; Based on the ordered Internet image set and the matching feature points between images, an adjusted camera pose and a sparse point cloud of a 3D scene are generated, and the adjusted camera pose is used as the geometric reference information.

6. The method for 3D modeling based on the fusion of space-based imagery and internet images according to claim 5, wherein, The step of generating a dense point cloud of an internet image based on the geometric reference information includes: Based on the geometric reference information, the viewpoint overlap of each ordered Internet image in the ordered Internet image set is generated to obtain a set of matching image pairs; Based on the depth map fusion algorithm, stereo matching is performed on each matching image pair in the matching image pair set, and the object-space depth of each pixel on each image in each matching image pair is generated to obtain a local depth map set. By combining the local depth atlas and the sparse point cloud of the 3D scene, the dense point cloud of the Internet image is obtained.

7. The method for 3D modeling based on the fusion of space-based imagery and internet images according to claim 1, wherein, The process of generating a fused point cloud based on the dense point cloud of the satellite imagery and the dense point cloud of the internet imagery includes: The dense point cloud of the satellite image and the dense point cloud of the Internet image are preprocessed to obtain the preprocessed dense point cloud of the satellite image and the preprocessed dense point cloud of the Internet image. Using the Harris3D feature extraction algorithm, key points are extracted from the preprocessed satellite image dense point cloud and the preprocessed Internet image dense point cloud, respectively, to obtain the point set of the satellite image point cloud and the point set of the Internet image. The point set of the satellite image point cloud and the point set of the Internet image both include a key point set and a corresponding FPFH descriptor set. Based on the SAC-IA algorithm, coarse registration is performed on the point cloud set of the satellite image and the point set of the Internet image to obtain coarsely registered heterogeneous point cloud pairs; The following steps are performed on the coarsely registered heterogeneous point cloud pairs: A reference point set is obtained by randomly sampling a predetermined number of points from the coarsely registered heterogeneous point cloud. For each reference point in the reference point set, determine the point with the closest Euclidean distance to the dense point cloud of the preprocessed Internet image to obtain the corresponding point pair set; The corresponding point set is filtered to obtain a valid set of corresponding point pairs; Using the centroid calculation formula, the reference point set and the corresponding point set centroids are generated respectively, thus obtaining the reference centroid and the corresponding point centroid. Based on minimizing the objective function, a rotation matrix and a translation matrix are generated according to the effective set of corresponding point pairs, the reference centroid, the corresponding point centroid, and the preprocessed dense point cloud of satellite imagery. Based on the transformation formula, the transformed point is generated using the rotation matrix, the translation matrix, and a single reference point in the reference point set; The current error is generated based on the transformed point and the single reference point; Repeat the above steps; In response to the fact that the difference between the current error and the previous error is less than a preset threshold, a registered heterogeneous point cloud is generated, wherein the registered heterogeneous point cloud includes the dense points of the satellite image and the aligned Internet point cloud; The registered heterogeneous point clouds are merged to obtain a preliminary combined point cloud; Redundancy is removed from the initial combined point cloud to obtain the fused point cloud.

8. The method for 3D modeling based on the fusion of space-based imagery and internet images according to claim 1, wherein, The step of generating a textured fused image 3D model based on the fused point cloud includes: The fused point cloud is preprocessed to obtain a preprocessed fused point cloud; Triangulation is performed on the preprocessed fused point cloud to obtain an initial triangulation surface and surface information, wherein the surface information includes a forward front and a priority queue of candidate triangles. Perform the following steps on the initial triangulated surface: Triangles that meet the preset criteria are determined from the candidate triangle priority queue to obtain candidate triangles; The candidate triangles are added to the initial triangulation surface to obtain the updated surface; Extract the non-coincident edges of the candidate triangles to obtain the updated advancing front. Based on the updated forward front, new candidate triangles are generated, resulting in an updated candidate triangle priority queue. If the updated candidate triangle priority queue does not contain any triangles that meet the preset criteria, a complete three-dimensional surface model is obtained. Based on surface triangular grids and image calibration information, the visibility image set of each triangular face and the surface triangular grid matching the surface are obtained; Based on the visibility image set of each triangular facet and the surface triangular grid that matches the surface, a texture patch and an optimal reference image for each triangular facet are generated. Based on the texture patch and the optimal reference image for each triangle, an initial texture image is generated; gaps in the initial texture image are removed to obtain a seamless texture image. The seamless texture image is mapped onto the complete 3D surface model to obtain the textured fused image 3D model.

9. A computer-readable storage medium having computer program instructions stored thereon, wherein, When the computer program instructions are executed by the processor, the processor performs the method for three-dimensional modeling based on the fusion of space-based imagery and Internet images according to any one of claims 1 to 8.

Images