Three-dimensional model and DEM fusion method and device based on multi-source image matching

Abstract

The invention relates to a three-dimensional model and DEM fusion method and device based on multi-source image matching, and the method can effectively solve the problems of displacement, floating or subsidence during the data superposition of a three-dimensional model and a three-dimensional earth platform, and achieves the fusion of a three-dimensional model and a DEM through the matching of the pixel points of a satellite image and an aerial image in a three-dimensional geographic engine. A digital elevation model is utilized to calculate accurate three-dimensional geodetic coordinates and elevation values, a high-precision image control point file is provided for three-dimensional modeling software, the image control point file enables the generated three-dimensional model to have high absolute geographic coordinate precision, and therefore it is guaranteed that the three-dimensional model is accurately fused with existing geographic data in a three-dimensional geographic engine. According to the method, the geographic space accuracy of the three-dimensional model can be remarkably improved while the high detail expressive force of the aerial image is kept, and the method is particularly suitable for an application scene where a local fine model and a large-range geographic background need to be precisely integrated.

Description

Technical Field

[0001] This application relates to the field of 3D model construction technology, and in particular to a method and apparatus for fusing 3D models and DEMs based on multi-source image matching. Background Technology

[0002] Unmanned aerial vehicle (UAV) oblique photogrammetry, as a core method for rapidly constructing realistic 3D models, has wide application value in fields such as urban planning and environmental monitoring. However, limited by the accuracy of the GPS systems carried by UAVs, civilian-grade GPS typically exhibits positioning deviations of several meters in open areas. Furthermore, the position and attitude systems of these UAVs accumulate errors due to drift of the inertial measurement unit and external interference. These factors result in 3D models that can only maintain the accuracy of their internal relative geometry, severely lacking absolute geographic coordinate accuracy. When such models need to be accurately overlaid with digital elevation models and high-resolution satellite imagery in a 3D geographic information system platform, significant horizontal displacement occurs, while vertically the model appears to float above or sink below the surface. The elevation difference problem is particularly prominent. Because altitude measurement is more sensitive to system errors, the model's positioning inaccuracy in the real geographic environment is significant, making it difficult to meet the stringent requirements for accurate geographic data registration in scenarios such as disaster assessment and infrastructure management. Summary of the Invention

[0003] The main objective of this disclosure is to propose a method and apparatus for fusing a 3D model and a DEM based on multi-source image matching. It utilizes high-precision satellite imagery as a geographic reference benchmark, automatically matches UAV images with satellite images using computer vision technology, and leverages the 3D geospatial capabilities of a 3D geographic engine to map 2D matching points to 3D geospatial coordinates, automatically generating control point files for controlling 3D modeling. The 3D modeling model then calls the 3D model generated from the control point files, and finally imports the data into the 3D geographic engine to achieve the fusion of the 3D model, digital elevation model, and satellite imagery.

[0004] A first aspect of this application provides a method for fusing a 3D model with a DEM based on multi-source image matching, the method comprising: In response to the fusion command, the digital elevation model and satellite image of the target area are loaded into the 3D geographic engine, and aerial images of the target area are acquired. Match the same pixels in the satellite image and the aerial image; In the 3D geographic engine, the 3D geodetic coordinates and elevation values ​​based on the digital elevation model of the same pixel are calculated according to the position of the same pixel in the satellite image, and a ground control point file is generated according to the 3D geodetic coordinates and elevation values. Based on the control point file, a 3D model of the target area is generated in the corresponding 3D modeling software; The 3D model is loaded into the 3D geographic engine to achieve the fusion of the 3D model, the digital elevation model, and the satellite imagery.

[0005] The 3D model and DEM fusion method based on multi-source image matching provided in this embodiment has at least the following beneficial effects: This embodiment introduces satellite imagery and digital elevation models (DEMs) as high-precision geographic references, providing reliable absolute geographic coordinate correction for 3D models generated from aerial images. Compared to the relative accuracy issues commonly found in existing UAV models, this method effectively solves the displacement, floating, or sinking phenomena that occur when 3D models are overlaid with 3D earth platform data. This method matches the pixels of satellite and aerial images in the 3D geographic engine and uses DEMs to calculate precise 3D geodetic coordinates and elevation values. This method provides high-precision control point files for 3D modeling software, enabling the generated 3D model to have high absolute geographic coordinate accuracy, thereby ensuring its accurate integration with existing geographic data in the 3D geographic engine. This method can significantly improve the geospatial accuracy of 3D models while maintaining the high detail of aerial images, and is particularly suitable for application scenarios that require precise integration of local fine models with large-scale geographic backgrounds.

[0006] In a second aspect, this application provides a device for fusing a 3D model with a DEM based on multi-source image matching, the device comprising: The instruction response module is used to respond to fusion instructions, load the digital elevation model and satellite image of the target area in the 3D geographic engine, and acquire aerial images of the target area; A pixel matching module is used to match identical pixels in the satellite image and the aerial image. The coordinate and elevation value calculation module is used to calculate the three-dimensional geodetic coordinates and elevation values ​​of the same pixel points in the satellite image based on the location points of the same pixel points in the three-dimensional geographic engine, and generate a ground control point file based on the three-dimensional geodetic coordinates and elevation values. The 3D model generation module is used to generate a 3D model of the target area in the corresponding 3D modeling software based on the control point file. A 3D model fusion module is used to load the 3D model into the 3D geographic engine to achieve the fusion of the 3D model, the digital elevation model, and the satellite imagery. A third aspect of this application provides an electronic device including at least one controller and a memory for communicatively connecting to the controller; the memory stores instructions executable by the at least one controller, the instructions being executed by the at least one controller to cause the at least one controller to perform a 3D model and DEM fusion method based on multi-source image matching as described above.

[0007] In a fourth aspect, this application provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a method for fusing a 3D model and DEM based on multi-source image matching as described above.

[0008] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart of a method for fusing a 3D model and a DEM based on multi-source image matching, provided in an embodiment of this application. Figure 2 This is a quality report image related to control points (CCPs) provided in this application embodiment, which uses CCP files to perform aerial triangulation on drone aerial photographs. Figure 3 This is an example of the horizontal and vertical deviation phenomenon when a 3D reconstruction model without using control points is loaded into a Cesium scene, as provided in this application embodiment. Figure 4 This is a precise fusion result image provided in this application embodiment, which loads the 3D reconstruction result model obtained using the method provided in the embodiment into a Cesium scene; Figure 5 This is a structural diagram of a 3D model and DEM fusion device based on multi-source image matching provided in an embodiment of this application; Figure 6 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0012] In the description of this application, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0013] In the description of this application, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed or function in a specific orientation, and therefore should not be construed as a limitation of this application.

[0014] like Figure 1 As shown in one embodiment of this application, a method for fusing a 3D model and a DEM based on multi-source image matching is provided. The method includes: Step S110: In response to the fusion command, load the digital elevation model and satellite image of the target area into the 3D geographic engine, and acquire aerial images of the target area; Step S120: Match the same pixels in the satellite image and the aerial image. Step S130: In the 3D geographic engine, the 3D geodetic coordinates and elevation values ​​based on the digital elevation model of the same pixel are calculated according to the position points of the same pixel in the satellite image, and the image control point file is generated according to the 3D geodetic coordinates and elevation values. Step S140: Generate a 3D model of the target area in the corresponding 3D modeling software based on the control point file; Step S150: Load the 3D model into the 3D geographic engine to achieve the fusion of the 3D model, digital elevation model and satellite imagery.

[0015] For ease of understanding, the following explains some key terms in this embodiment: A 3D geographic engine is a software platform for loading, displaying, analyzing, and managing geospatial data. It provides visualization capabilities of a 3D Earth and supports the integration and rendering of various geographic data formats, such as the Cesium engine.

[0016] A Digital Elevation Model (DEM) is a model that represents ground elevation information in discrete digital form. The model is typically stored in the form of a grid or triangular network (TIN) and is used to describe the undulating terrain. It is one of the fundamental data sources for geospatial analysis and 3D scene construction.

[0017] Satellite images are images of the Earth's surface obtained from space using sensors carried by artificial Earth satellites.

[0018] Aerial imagery refers to images obtained by taking pictures of ground targets from the air using an aircraft. Aerial imagery typically has high resolution and rich detail, providing detailed information about the terrain features of the target area, and is often used for realistic 3D modeling.

[0019] A control point file is a file containing the coordinates of image points and their corresponding ground control points. Control point files play a crucial role in photogrammetry and 3D modeling, used for georegistration, correction, and model generation to ensure the model has accurate geographical locations.

[0020] 3D modeling software is a professional software used to construct 3D models based on image data, point cloud data, or geometric data. The software can perform aerial triangulation, texture mapping, model optimization, and other functions to generate realistic 3D scenes or object models.

[0021] In this embodiment, step S110 first responds to the fusion command, loads the digital elevation model and satellite image of the target area in the three-dimensional geographic engine, and acquires aerial images of the target area.

[0022] In practice, users can select a target area through the 3D geographic engine interface and manually specify the paths to the digital elevation model (DEM) and satellite image files to be loaded. Users can also upload pre-captured aerial images. Alternatively, the system can automatically retrieve and load the corresponding DEM and satellite images from a database based on a preset geographical area, prompting the user to provide the aerial images via a network interface or local storage.

[0023] In step S120, matching pixels are found between the pixels in the satellite image and the pixels in the aerial image.

[0024] One implementation of this embodiment involves an operator manually selecting and marking corresponding pixels with distinct features, such as corner points and center points, in both satellite and aerial images. Another implementation involves using image feature descriptor-based matching algorithms, such as Scale Invariant Feature Transform (SIFT) or Speed-Up Robust Feature Transform (SURF), to extract and match feature points in both images, thereby automatically identifying identical pixels present in both images.

[0025] In step S130, the three-dimensional geodetic coordinates and elevation values ​​based on the digital elevation model of the same pixel are calculated in the three-dimensional geographic engine according to the location of the same pixel in the satellite image, and a control point file is generated based on the three-dimensional geodetic coordinates and elevation values.

[0026] In this embodiment, the 3D geographic engine can calculate the corresponding 3D geodetic coordinates (longitude and latitude) and elevation values ​​based on the digital elevation model of the satellite image, based on the 2D pixel coordinates of the matched pixels on the satellite image, combined with the georeferenced information of the satellite image and the loaded digital elevation model, using an internal algorithm. These calculated coordinates and elevation values ​​are then organized into a structured data file, namely, a control point file.

[0027] In S140, a 3D model of the target area is generated in the corresponding 3D modeling software based on the control point file.

[0028] In one implementation, the generated control point files are imported into 3D modeling software, along with aerial images of the target area. The 3D modeling software uses these control points as ground control points to perform aerial triangulation and geometric correction on the aerial images, thereby constructing a georeferenced 3D model. Another implementation involves the 3D modeling software providing an interface to directly receive control point data and aerial image data from the 3D geospatial engine and automatically initiate the 3D model generation process, including point cloud generation, mesh construction, and texture mapping.

[0029] In step S150, a 3D model is loaded into the 3D geographic engine to achieve the fusion of the 3D model, digital elevation model and satellite imagery.

[0030] The generated 3D model can be exported to a common format supported by the 3D geospatial engine, such as 3D Tiles or glTF, and then imported into the 3D geospatial engine. The 3D geospatial engine spatially registers and renders the 3D model with the previously loaded digital elevation model and satellite imagery, thereby achieving seamless overlay and visualization of different geographic data sources; This embodiment introduces satellite imagery and digital elevation models (DEMs) as high-precision geographic references, providing reliable absolute geographic coordinate correction for 3D models generated from aerial images. Compared to the relative accuracy issues commonly found in existing UAV models, this method effectively solves the displacement, floating, or sinking phenomena that occur when 3D models are overlaid with 3D Earth platform data (such as DEMs and satellite imagery). This method matches the pixels of satellite and aerial images in a 3D geographic engine and uses DEMs to calculate precise 3D geodetic coordinates and elevation values. This method provides high-precision control point files for 3D modeling software, enabling the generated 3D model to possess high absolute geographic coordinate accuracy, thus ensuring accurate integration with existing geographic data in the 3D geographic engine. This method significantly improves the geospatial accuracy of 3D models while maintaining the high detail of aerial images, making it particularly suitable for applications requiring precise integration of detailed local models with large-scale geographic backgrounds.

[0031] In some embodiments of this application, step S120, which involves matching identical pixels between the satellite image and the aerial image, includes: Step S210: Extract the pixels from the satellite image based on the coverage area of ​​the aerial image; Step S220: According to the preset feature pixel matching algorithm, the pixels in the satellite image are matched with the corresponding pixels in the aerial image to find the same pixels.

[0032] In this process, pixels are extracted from satellite images based on the coverage area of ​​aerial images. Then, pixels in satellite images are matched with corresponding pixels in aerial images using a preset feature pixel matching algorithm to identify identical pixels. This step is used to establish a precise correspondence between the two images.

[0033] Pre-defined feature pixel matching algorithms are algorithms that can automatically identify points in an image with unique textures, shapes, or color patterns, and describe these points to find their counterparts in different images. For example, the Scale Invariant Feature Transform (SIFT) algorithm can be used, which can detect key points in an image and generate descriptors robust to scale, rotation, and illumination changes. These descriptors are then compared to match identical pixels. Another option is the Speed-Up Robust Feature Transform (SURF) algorithm, which provides faster computation while maintaining similar performance to SIFT. Alternatively, the Oriented Fast and Rotated BRIEF (ORB) algorithm can be used, which combines Fast corner detection and BRIEF descriptors, offering high computational efficiency.

[0034] This embodiment effectively avoids unnecessary calculations in irrelevant areas when matching pixels between satellite and aerial images, significantly reducing the computational complexity and time cost of the matching process. At the same time, due to the precise limitation of the matching range, background noise and interference are reduced, thereby greatly improving the accuracy and reliability of matching identical pixels. This provides a solid data foundation for the subsequent accurate calculation of three-dimensional geodetic coordinates and elevation values, as well as the high-quality generation of three-dimensional models.

[0035] In some embodiments of this application, step S130, which calculates the three-dimensional geodetic coordinates and elevation values ​​based on the digital elevation model of the same pixel in the three-dimensional geographic engine according to the location points of the same pixel in the satellite image, includes: Step S310: Extract the world coordinates of the location on the three-dimensional Earth surface in the three-dimensional geographic engine based on the position of the same pixel in the satellite image. Step S320: Determine the latitude and longitude and elevation values ​​of the same pixel points in radians based on world coordinates; Step S330: Convert the latitude and longitude in radians and their elevation values ​​into latitude and longitude in angles and their elevation values ​​for the same pixel points.

[0036] In a 3D geographic engine, extracting the world coordinates of a pixel on the 3D Earth's surface based on its position in a satellite image refers to mapping pixels on a 2D image to their precise location in 3D space. This serves as the basis for subsequent geographic coordinate calculations. This can be achieved through the application programming interface (API) or software development kit (SDK) provided by the 3D geographic engine. For example, by inputting image pixel coordinates, a specific function can be called to obtain the world coordinates of that pixel on the 3D Earth's surface. Alternatively, a pre-established image-to-3D space mapping relationship can be used to directly convert pixels in the satellite image to the world coordinate system within the 3D geographic engine.

[0037] Determining the latitude, longitude, and elevation values ​​of identical pixels in radians using world coordinates converts the engine's internal abstract 3D coordinates into standard geospatial information, facilitating geographic analysis and data exchange. For example, Cartesian world coordinates can be converted into latitude, longitude, and ellipsoidal elevation in radians; additionally, mathematical formulas can be used to calculate the corresponding geographic longitude (radians), latitude (radians), and elevation from Cartesian coordinates based on an Earth ellipsoid model.

[0038] Converting latitude and longitude in radians and their elevation values ​​to latitude and longitude in angles with the same number of pixels is to meet the common requirements of subsequent 3D modeling software for input geographic coordinate formats, thereby improving data compatibility and ease of use.

[0039] This embodiment achieves precise, step-by-step conversion from the engine's internal world coordinates to standard geographic coordinates when calculating the 3D geodetic coordinates and elevation values ​​of the same pixel in a 3D geographic engine. First, extracting the world coordinates ensures the accurate mapping of the pixel's position in 3D space. Second, determining the latitude and longitude in radians and their elevation values ​​guarantees the accuracy of geographic coordinate calculations and compatibility with the internal processing of the geographic information system. Finally, converting the radian system to degrees greatly improves the data interoperability and ease of use with various 3D modeling software when generating control point files. This standardized coordinate acquisition and conversion process effectively avoids misalignment problems when fusing 3D models and digital elevation models due to coordinate system mismatch or insufficient accuracy, significantly improving the accuracy and reliability of the fusion results and laying a solid foundation for subsequent high-quality 3D model generation and fusion.

[0040] In some embodiments of this application, the same pixel includes the four corner points and the center point within the coverage area; Step S140, which involves generating a 3D model of the target region in 3D modeling software based on the control point file, includes: The image control point (ARP) file is generated by selecting the latitude, longitude, and elevation values ​​corresponding to the four corner points and the center point. The same pixel includes the four corner points and the center point within the coverage area, which defines the specific selection strategy for generating the ARPP file. Within the coverage area of ​​the aerial image, selecting the four corner points and the center point as key ARPPs aims to provide a minimized and optimized set of ARPPs that covers the entire area while maintaining accuracy in the central region. This selection can be achieved, for example, through image processing algorithms. First, the coverage area of ​​the aerial image on the satellite image is determined, then the geometric center point of that coverage area is calculated, and the four vertices of the rectangular coverage area are identified as corner points. Alternatively, the user interface can allow operators to manually or semi-automatically select and mark these five key points on the satellite image after determining the coverage area of ​​the aerial image to ensure accuracy. The step of selecting the latitude, longitude, and elevation values ​​corresponding to the four corner points and the center point to generate the ARPP file is clearly defined. The method for generating control point files involves precisely filtering the geographic information corresponding to the five key points from the previously calculated latitude, longitude, and elevation values ​​of all identical pixels, and organizing them into a standard control point file format. For example, the system can maintain a list of all matching pixels and their geographic information. Once the four corner points and the center point are determined, the latitude, longitude, and elevation values ​​of these five points are extracted by querying the list and packaged according to the preset control point file format. Alternatively, after calculating the geographic information of all identical pixels, the data can be directly filtered, retaining only the pixel data located at the corner and center positions of the coverage area, and then these simplified data can be directly output as control point files.

[0041] In this embodiment, when generating control point files, instead of blindly using all matched pixels, the four corner points and the center point within the coverage area of ​​the aerial image are strategically selected as control points. This concise and representative control point selection method significantly reduces the amount of control point data, thereby effectively reducing the computational complexity and time cost of 3D modeling software when processing control points. At the same time, since these key points have good spatial distribution, the geometric accuracy of the entire target area can be better controlled, avoiding errors introduced by redundant or low-quality control points, thus improving the efficiency and accuracy of 3D model generation.

[0042] In some embodiments of this application, step S140 generates a 3D model of the target region in the corresponding 3D modeling software, including: Step S410: In the 3D modeling software corresponding to the control point file, load the control point file and the aerial image, and perform aerial triangulation calculation; Step S420: After the aerial triangulation calculation, generate an aerial triangulation quality report; Step S430: Determine the quality of the control points based on the aerial triangulation quality report; Step S440: Remove control points with quality below a preset threshold, and generate a 3D model of the target area based on the retained control points and aerial images.

[0043] The process of loading control point files and aerial images and performing aerial triangulation involves importing control point files containing real ground coordinates and multiple overlapping aerial image sequences using professional photogrammetry software or a custom algorithm, and then initiating the aerial triangulation process. Aerial triangulation recovers the exterior orientation elements (camera position and attitude) and the three-dimensional coordinates of ground points in each image by matching and calculating corresponding points in multiple images. Control points serve as known constraints in this process to improve the accuracy of the calculation.

[0044] Generating an aerial triangulation quality report refers to outputting a detailed report after the aerial triangulation solution is completed. This report contains various accuracy indicators and statistical information of the aerial triangulation solution results. This report is an important basis for evaluating the reliability of the solution and the contribution of control points. The report is usually presented in text, PDF or HTML formats, and the content may include control point residuals, tie point residuals, exterior orientation element accuracy, ground point coordinate accuracy, etc.

[0045] Determining the quality of control points (COPs) refers to evaluating the contribution of each COP to the overall solution accuracy and its own reliability based on the data in the aerial triangulation quality report. Low-quality COPs may be due to measurement errors, identification errors, or unclear ground features. The quality of COPs can be determined by analyzing the residuals of each COP in the quality report (e.g., reprojection errors in image space or coordinate errors in ground space). If the residuals are much larger than the average value or a preset threshold, the quality is considered low. In addition, the quality of COPs can be comprehensively judged by combining factors such as the connectivity of COPs in multiple images, the number of images involved in the solution, and the uniformity of their distribution throughout the survey area.

[0046] Removing control points with quality below a preset threshold means removing low-quality control points from the control point set after they have been identified. This aims to avoid these low-quality points from introducing errors, thereby improving the accuracy of subsequent 3D model generation. The removal operation can be performed by manually selecting and deleting these points in the interface of the 3D modeling software, or by automatically identifying and removing the corresponding records from the control point file through a script.

[0047] Generating a 3D model of the target area based on retained control points and aerial images refers to reconstructing a more accurate 3D model using an optimized set of control points and the original aerial images after removing low-quality control points. This ensures that the generated 3D model has higher geometric accuracy and texture detail. In 3D modeling software, the 3D model generation process can be restarted. At this time, the software will perform steps such as dense point cloud generation, mesh construction, and texture mapping based on the optimized (low-quality points removed) control point file and aerial images, ultimately outputting a high-precision 3D model.

[0048] This embodiment effectively addresses the problem of decreased 3D model accuracy caused by poor control point (COP) quality by introducing an evaluation and optimization mechanism. First, the COP files and aerial images are loaded, and aerial triangulation is performed. Aerial triangulation not only restores the exterior orientation elements of the images but also provides fundamental data for evaluating COP quality. Next, the system generates an aerial triangulation quality report, which details the solution residuals and accuracy indicators for each COP. By analyzing this report, the quality of each COP can be accurately determined based on a preset accuracy threshold. COPs with quality below the preset threshold are identified and removed from the COP set, thus preventing these low-quality points from introducing errors into subsequent 3D model generation. Finally, based only on the selected and retained high-quality COPs and the original aerial images, the 3D model of the target area is regenerated or continued. This iterative optimization process ensures that the generated 3D model has higher geometric accuracy and reliability, providing high-quality fundamental data for subsequent fusion with digital elevation models and satellite imagery in the 3D geographic engine, significantly improving the overall fusion accuracy and visual effect.

[0049] In some embodiments of this application, the 3D geographic engine is the Cesium engine; the 3D modeling software is iTwinCapture Modeler software. The aerial images are images taken by a drone.

[0050] In this embodiment, by specifying the 3D geographic engine as the Cesium engine and the 3D modeling software as iTwin Capture Modeler, the entire fusion process is ensured to be smooth and efficient. The Cesium engine, as a high-performance WebGIS 3D earth and map visualization library, can efficiently load and render large-scale digital elevation models, satellite images, and the final fused 3D model, providing a smooth interactive experience. Meanwhile, iTwin Capture Modeler, as a professional photogrammetric modeling software, has significant advantages in generating high-precision 3D models from aerial images and control point files. Its aerial triangulation and model generation algorithms are optimized to ensure a high degree of consistency between the generated 3D model and real-world geographic information. When the fusion command is responded to, the Cesium engine is responsible for loading the digital elevation model and satellite images of the target area and acquiring aerial images. Subsequently, in the Cesium engine, by matching the pixels of the satellite images and aerial images and calculating the 3D geodetic coordinates and elevation values ​​based on the digital elevation model for these identical pixels, control point files are generated. These control point files, along with the aerial images, are imported into iTwin Capture. In the Modeler software, iTwin Capture Modeler utilizes its professional aerial triangulation capabilities to accurately calculate the interior and exterior orientation elements of the aerial imagery and optimize it based on the quality of the control point files. This results in a high-precision 3D model of the target area. Finally, this 3D model generated by iTwin Capture Modeler is reloaded into the Cesium engine and seamlessly integrated with the previously loaded digital elevation model and satellite imagery. This specific software combination optimizes the entire data processing, model generation, and visualization fusion process, ensuring data consistency, processing efficiency, and the accuracy of the final display.

[0051] This embodiment specifically defines aerial images as images taken by drones, making the entire 3D model and DEM fusion method based on multi-source image matching more flexible and adaptable in the image data acquisition stage. Specifically, after responding to the fusion command, the 3D geographic engine loads the digital elevation model and satellite images while acquiring aerial images of the target area through drone aerial photography. These drone aerial images typically have extremely high resolution and the latest ground feature information, which can more accurately reflect the current topography and building details of the target area. Subsequently, when matching the same pixels in the satellite images and the drone aerial images, the accuracy and reliability of the matching can be improved due to the high precision of the drone images. Based on these precisely matched pixels, the 3D geographic engine calculates their 3D geodetic coordinates and elevation values ​​based on the digital elevation model, and generates control point files. These control points are more reliable due to the high precision of the drone images. Consequently, the 3D model generated in the corresponding 3D modeling software based on these high-quality control point files and drone aerial images will have significantly improved geometric accuracy and texture details. Ultimately, loading this high-precision 3D model into the 3D geographic engine enables a more accurate and seamless fusion between the 3D model, digital elevation model, and satellite imagery, thereby providing a more realistic and immersive 3D geographic information display.

[0052] This embodiment limits aerial images to images taken by drones, effectively solving the problems of high cost, low efficiency, and numerous limitations faced by traditional aerial photography methods when acquiring high-resolution, timely images of specific areas. Drone aerial photography technology can acquire detailed image data of target areas at a lower cost, faster speed, and higher flexibility. It is especially suitable for areas with complex terrain, inconvenient transportation, or areas that require frequent data updates. This not only improves the convenience and economy of aerial image acquisition, but also significantly improves the accuracy of subsequent pixel matching, the reliability of control point generation, and the geometric accuracy and texture detail of the final 3D model construction, since drone images usually have higher resolution and stronger timeliness. Therefore, the fusion method of this application can generate more realistic, accurate, and timely 3D geographic models, greatly enhancing the fusion effect and application value of 3D models, digital elevation models, and satellite images in 3D geographic engines.

[0053] like Figures 2 to 4 One embodiment of this application provides a method for fusing a 3D model with a DEM based on multi-source image matching. This embodiment aims to solve the positional deviation problem when fusing a UAV oblique photogrammetry model with a DEM in a real geographic coordinate system. The method includes: Step S910, Data loading and preparation; Launch the Cesium-based front-end application and load the DEM and satellite imagery of the target area.

[0054] To load DEM terrain, you can use Cesium.CesiumTerrainProvider.fromIonAssetId(1) to load global terrain provided by Cesium Ion, or you can use custom terrain. After obtaining DEM digital elevation model data in TIFF format with a specific resolution for the target area, you can use CesiumLab or terr2cesiumApp to convert it into tile data that Cesium can use, and then deploy it to the server to get a URL. Then you can use Cesium.CesiumTerrainProvider.fromUrl(url) to load the custom DEM terrain.

[0055] Satellite imagery can be loaded either by loading Tianditu satellite imagery tiles or by customizing and obtaining high-resolution raster TIFF format satellite imagery of the target area, publishing it as a WMTS layer service via GeoServer, and then loading it. Both methods utilize Cesium.WebMapTileServiceImageryProvider for loading.

[0056] Step S920: Multi-source image matching; Call the background matching service (such as using the SIFT algorithm in OpenCV), read satellite image slices to obtain satellite images and read UAV images, and perform feature extraction and matching.

[0057] Based on the geographical range of the UAV image, a rooftop corner or other feature point on the satellite image is automatically located and matched with the corresponding point on the UAV image to form a pair of identical points. This process generates a large number of matching point pairs for multiple UAV images and satellite imagery.

[0058] Step S930, Geospatial coordinate calculation; For each matching point obtained in step S920 (assuming it's point P_sat on the satellite image), the precise world coordinates (3D Cartesian space coordinates, Cartesian3 coordinates) of point P_sat on the 3D Earth surface are obtained using Cesium's viewer.camera.getPickRay ray picking method. Then, the coordinates are converted to radian latitude, longitude, and elevation (longitude, latitude, altitude) using Cesium's ellipsoid.cartesianToCartographic method. Finally, the coordinates are converted to angular latitude, longitude, and elevation using Cesium.Math.toDegrees. The elevation here is the surface elevation combined with the DEM terrain.

[0059] Step S940: Image control point file is automatically generated; From all the matched points, select 4 to 6 evenly distributed points (such as the four corner points and the center point of the covered area). Write the coordinate information of these points into a text file (.txt or .csv) according to the format (name, Longitude, Latitude, Height) required by the spatial reference system of the iTwin Capture Modeler software for the CGCS2000 coordinate system. An example of the generated file content is shown below: pt0,112.749338,28.155430,72.3483; pt1,112.745694,28.154774,82.9562; pt2,112.745134,28.156999,102.6322; pt3,112.747650,28.158090,96.5970; pt4,112.747935,28.157133,123.0797; Step S950: Precise 3D model reconstruction; Users import the UAV image and the control point file generated in step S940 into iTwin CaptureModeler. In the aerial triangulation settings, the control point file is specified, and the control points are automatically matched and associated with the feature points of the corresponding potential match UAV image in the iTwin Capture Modeler software interface.

[0060] After aerial triangulation is performed, an aerial triangulation quality report for the control points will be generated, such as... Figure 2 As shown, the figure displays the name, category, accuracy, number of calibrated images, reprojection error RMS, distance RMS from the light source, 3D error, horizontal error, vertical error, and indicators showing whether the reprojection error is too large for each control point. Control points with poor quality are removed (e.g., ...). Figure 2 (PT4 control points are used to avoid excessive reprojection errors). The reconstruction process then runs, and the software uses these control points to geographically constrain the entire model, producing a high-precision realistic 3D model.

[0061] Step S960: Precise integration and display; The OSGB format model produced in step S950 is converted to 3dtiles and loaded into the Cesium engine scene in S910. Since the model already possesses precise absolute coordinates during reconstruction, it will closely match the DEM terrain and satellite imagery base map, achieving true seamless integration, as shown in the image. Figure 4 As shown.

[0062] For clearer comparison, the 3D reconstruction result model without using image control points is also loaded into the Cesium scene in step S910, such as... Figure 3 and Figure 4 As shown.

[0063] Figure 3 There is a significant horizontal displacement deviation compared to satellite images. Figure 3 (as shown on the left) and the floating 3D model ( Figure 3 As shown on the right), however, loading using the result of step S950 does not exhibit the aforementioned significant deviation (as shown on the right). Figure 4 Therefore, when using Cesium.Cesium3DTileset.fromUrl to load the resulting 3Dtiles model without control points, it is necessary to set the model offset matrix, which requires manually trying out the latitude, longitude, and elevation offsets, which is quite complicated. However, using the resulting 3Dtiles model from step S950 does not require setting the offset matrix and is already accurately fused with the DEM.

[0064] This embodiment loads a Digital Elevation Model (DEM) terrain and satellite imagery into the Cesium engine. Using a feature matching algorithm, it automatically matches the UAV aerial images with the satellite images to obtain corresponding point pairs. Within the Cesium engine, based on the position of the matched points on the satellite imagery, it automatically calculates their high-precision latitude, longitude, and elevation coordinates. Based on the calculated coordinates, it automatically generates control point files conforming to the format of 3D modeling software such as iTwin Capture Modeler. These control point files are then imported into the modeling process for aerial triangulation, generating a 3D model with accurate geographic coordinates. Finally, the model can be seamlessly and accurately overlaid with the DEM within the Cesium engine. This embodiment automates the entire process from image matching to control point generation, replacing traditional manual point marking and significantly improving the efficiency and accuracy of 3D model geolocation.

[0065] like Figure 5 As shown in one embodiment of this application, a device for fusing a 3D model and a DEM based on multi-source image matching is provided. The device includes: The instruction response module 1001 is used to respond to fusion instructions, load the digital elevation model and satellite image of the target area in the 3D geographic engine, and acquire aerial images of the target area; The pixel matching module 1002 is used to match the same pixels in satellite images and aerial images; The coordinate and elevation calculation module 1003 is used to calculate the three-dimensional geodetic coordinates and elevation values ​​based on the digital elevation model of the same pixel in the three-dimensional geographic engine, based on the location of the same pixel in the satellite image, and generate a ground control point file based on the three-dimensional geodetic coordinates and elevation values. The 3D model generation module 1004 is used to generate a 3D model of the target area in the corresponding 3D modeling software based on the control point file. The 3D model fusion module 1005 is used to load 3D models into the 3D geographic engine to achieve the fusion of 3D models, digital elevation models and satellite imagery.

[0066] It should be noted that the 3D model and DEM fusion device based on multi-source image matching provided in this embodiment is based on the same inventive concept as the above-mentioned 3D model and DEM fusion device based on multi-source image matching. Therefore, the content of the 3D model and DEM fusion method based on multi-source image matching in the above embodiment is also applicable to the content of the 3D model and DEM fusion device based on multi-source image matching in this embodiment, and will not be repeated here.

[0067] like Figure 6 One embodiment of this application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method for fusing a 3D model with a DEM based on multi-source image matching. The electronic device includes: At least one battery; At least one memory; At least one processor; At least one program; The program is stored in memory, and the processor executes at least one program to implement the above-described method for fusing a 3D model and DEM based on multi-source image matching.

[0068] Electronic devices can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0069] The electronic devices according to embodiments of this application will now be described in detail.

[0070] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure. The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to implement a method for fusing a 3D model and DEM based on multi-source image matching according to an embodiment of this disclosure.

[0071] The input / output interface 1800 is used to implement information input and output. The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900); The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0072] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described method for fusing a 3D model and DEM based on multi-source image matching.

[0073] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0074] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0075] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0076] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0077] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0078] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0079] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0081] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0083] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0084] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

Claims

1. A method for fusing a three-dimensional model and a DEM based on multi-source image matching, characterized in that, The method comprises: in response to a fusion instruction, loading a digital elevation model and a satellite image of a target area in a three-dimensional geographic engine, and obtaining a aerial image of the target area; matching the same pixel points in the pixel points of the satellite image and the pixel points of the aerial image; in the three-dimensional geographic engine, according to the position of the same pixel points in the satellite image, calculating the three-dimensional geodetic coordinates and the elevation value based on the digital elevation model of the same pixel points, and generating a photo control point file according to the three-dimensional geodetic coordinates and the elevation value; generating a three-dimensional model of the target area in a corresponding three-dimensional modeling software according to the photo control point file; loading the three-dimensional model in the three-dimensional geographic engine to realize the fusion of the three-dimensional model, the digital elevation model and the satellite image. 2.The method of claim 1, wherein, The matching of the same pixel points in the pixel points of the satellite image and the pixel points of the aerial image comprises: extracting the pixel points in the satellite image according to the coverage range of the aerial image; matching the pixel points in the satellite image with the corresponding pixel points in the aerial image according to a preset feature pixel point matching algorithm to match the same pixel points. 3.The method of claim 2, wherein, The calculation of the three-dimensional geodetic coordinates and the elevation value based on the digital elevation model of the same pixel points in the three-dimensional geographic engine according to the position of the same pixel points in the satellite image comprises: extracting the world coordinates of the position of the same pixel points on the surface of the three-dimensional earth according to the position of the same pixel points in the satellite image in the three-dimensional geographic engine; determining the radian system longitude and latitude and the elevation value of the same pixel points according to the world coordinates; converting the radian system longitude and latitude and the elevation value into the angle system longitude and latitude and the elevation value of the same pixel points.

4. The method of claim 3, wherein, The same pixel points include four corner points and a center point in the coverage range; The generation of the three-dimensional model of the target area in the three-dimensional modeling software according to the photo control point file comprises: selecting the angle system longitude and latitude and the elevation value corresponding to the four corner points and the center point to generate a photo control point file.

5. The method of claim 1, wherein, The generation of the three-dimensional model of the target area in the corresponding three-dimensional modeling software comprises: loading the photo control point file and the aerial image in the three-dimensional modeling software corresponding to the photo control point file, and performing aerial triangulation; generating an aerial triangulation quality report after the aerial triangulation; determining the quality of the photo control points according to the aerial triangulation quality report; removing the photo control points with a quality lower than a preset threshold, and generating the three-dimensional model of the target area according to the remaining photo control points and the aerial image.

6. The method of claim 1, wherein, The three-dimensional geographic engine is a Cesium engine, and the three-dimensional modeling software is iTwin Capture Modeler software.

7. The method of claim 1, wherein, The aerial image is an image taken by a drone.

8. A device for fusing a three-dimensional model and a DEM based on multi-source image matching, characterized in that, The device comprises: an instruction response module for loading a digital elevation model and a satellite image of a target area in a three-dimensional geographic engine in response to a fusion instruction, and obtaining a aerial image of the target area; A pixel matching module is used to match identical pixels in the satellite image and the aerial image. The coordinate and elevation value calculation module is used to calculate the three-dimensional geodetic coordinates and elevation values ​​of the same pixel points in the satellite image based on the location points of the same pixel points in the three-dimensional geographic engine, and generate a ground control point file based on the three-dimensional geodetic coordinates and elevation values. The 3D model generation module is used to generate a 3D model of the target area in the corresponding 3D modeling software based on the control point file. A 3D model fusion module is used to load the 3D model into the 3D geographic engine to achieve the fusion of the 3D model, the digital elevation model, and the satellite image.

9. An electronic device, comprising: It includes at least one controller and a memory for communicatively connecting with the controller; the memory stores instructions that can be executed by the at least one controller, the instructions being executed by the at least one controller to cause the at least one controller to perform a 3D model and DEM fusion method based on multi-source image matching as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions for causing a computer to perform a method for fusing a 3D model and a DEM based on multi-source image matching as described in any one of claims 1 to 7.

Images