A map fusion method and device based on baseless visual guidance, equipment and medium

Abstract

The application provides a map fusion method and device based on bottomless visual guidance, equipment and medium, wherein each sub-map image is collected, and a visual control point image with the same mark is collected at the same physical position in the overlapping area of the sub-maps; the visual features of the control point image are extracted, and an association index of the mark and the features is established; the same name control point image is recalled from the index according to the mark, the relative poses between the sub-maps are obtained by matching the features; the initial poses of the coordinate systems of the sub-maps are obtained, the relative poses are converted to the same reference coordinate system, and the global poses are obtained; the pose transformation chain is constructed based on the global poses, the geometric consistency of different global poses of the same sub-map is checked, and the relative poses that do not pass the check are removed; finally, according to the checked global poses, the sub-map images are spliced into the same coordinate system to generate a fused map. By using the above method, the spatial accuracy and scene integrity of the fused map can be ensured, and the robustness of the map fusion process is significantly improved.

Description

A baseless visual guidance-based map fusion method, apparatus, device, and medium Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method, apparatus, device, and medium for map fusion based on baseless visual guidance. Background Technology

[0002] With the rapid development of technologies such as digital twins, smart cities, and digital preservation of cultural relics, the application demand for large-scale 3D reconstruction, such as city-level 3D modeling, multi-scale reconstruction of large industrial parks, and construction of integrated indoor and outdoor navigation maps, is becoming increasingly prominent. Map fusion, as a core and key link in large-scale 3D reconstruction, plays an important role in achieving accurate stitching of sub-maps collected from multiple sources and regions. Its technical implementation effect directly affects the spatial accuracy and scene integrity of the 3D reconstruction results, making it a research focus and key application in this field.

[0003] The current mainstream map fusion method is based on high-precision physical control points. This method requires first setting up high-precision physical control points in the scene to be reconstructed by using a total station. These control points are used as the core registration basis for sub-map fusion. At the same time, a dedicated base and high-precision positioning equipment are used to collect the spatial position data of the control points. Then, based on the accurate coordinate information of the control points, the pose registration and data fusion between different sub-maps are realized. This is currently a widely used technical solution in large-scene map fusion.

[0004] The aforementioned map fusion method based on high-precision physical control points has a significant drawback: high deployment and operational complexity. This method has stringent professional requirements for the deployment of high-precision physical control points, requiring professional technicians to operate specialized equipment such as total stations to complete the target acquisition and deployment of control points. Furthermore, in large-scale scene reconstruction, a large number of physical control points need to be deployed to ensure fusion accuracy, making the overall deployment process cumbersome, time-consuming, and labor-intensive. At the same time, the use of specialized bases and high-precision positioning equipment further increases the professional threshold for operation, making it difficult to adapt to the actual operational needs of rapid data acquisition and efficient fusion in large-scale scene map fusion. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a map fusion method, device, equipment and medium based on baseless visual guidance, which can ensure the spatial accuracy and scene integrity of the fused map and significantly improve the robustness of the map fusion process.

[0006] In a first aspect, embodiments of this application provide a map fusion method based on baseless visual guidance, the method comprising:

[0007] Collect submap images of different submaps, and collect images of visual control points with the same labels at the same physical location within the overlapping area of ​​each submap.

[0008] Extract the visual features of each visual control point image and establish an association index between the identifier and the visual features;

[0009] Based on the identifier of the visual control point image in the current sub-map, recall visual control point images of other sub-maps with the same identifier from the associated index;

[0010] The visual features of the current visual control point image are matched with the visual features of the recalled visual control point images of the same name to obtain the relative pose between the current sub-map and other sub-maps.

[0011] Obtain the initial pose of each sub-map in its respective coordinate system, and use the relative pose to transform the initial pose of each sub-map to the same reference coordinate system to obtain the global pose of each sub-map.

[0012] Based on the global pose, a pose transformation chain is constructed between sub-maps. Geometric consistency is checked on multiple global poses of the same sub-map obtained through different pose transformation chains, and the relative poses that fail the check are removed.

[0013] Based on the verified global pose, the sub-map images of each sub-map are stitched together to the same coordinate system to generate a fused map.

[0014] Optionally, the step of extracting visual features from the images of each visual control point and establishing an association index between the identifier and the visual features includes:

[0015] A dedicated image library is constructed for each visual control point corresponding to the aforementioned identifier, wherein the dedicated image library contains multiple images of visual control points acquired at the same physical location from different angles or at different times;

[0016] Extract the local visual feature vector of each visual control point image in the dedicated image library, and associate and store the local visual feature vector with the identifier;

[0017] A multimodal retrieval structure is established using the identifier as the primary index and the approximate retrieval structure of the local visual feature vector as the secondary index, thus obtaining the associated index.

[0018] Optionally, the step of recalling visual control point images of other sub-maps with the same identifier from the associated index based on the identifier of the visual control point image in the current sub-map includes:

[0019] Get the identifiers of the visual control point images contained in the current sub-map;

[0020] Using the identifier as a primary index, locate the dedicated image library corresponding to the identifier from the associated index;

[0021] Extract the visual feature vector of the current visual control point image;

[0022] The visual feature vector is used as a query vector and input into the exclusive image library located by the identifier in the associated index. The visual control point images that meet the preset threshold of similarity with the query vector are retrieved through the secondary index and used as the recalled visual control point images with the same name.

[0023] Optionally, the step of matching the visual features of the current visual control point image with the visual features of the recalled visual control point images of the same name to obtain the relative pose between the current sub-map and other sub-maps includes:

[0024] Feature point matching is performed between the current visual control point image and the recalled visual control point images with the same name.

[0025] The RANSAC algorithm is used to remove mismatched point pairs, and the geometric constraints of the remaining matching point pairs are verified by estimating the fundamental matrix or homography matrix.

[0026] Based on the verified matching point pairs, calculate the relative rotation matrix and translation vector between the current visual control point image and the recalled image, which serves as the relative pose between the current submap and other submaps.

[0027] Optionally, obtaining the initial pose of each sub-map in its respective coordinate system, and using the relative pose to transform the initial pose of each sub-map to the same reference coordinate system to obtain the global pose of each sub-map, includes:

[0028] The initial pose of each sub-map in its own reconstructed coordinate system is obtained through a simultaneous localization and mapping algorithm.

[0029] Select the coordinate system of one of the sub-maps as the global reference coordinate system;

[0030] For other sub-maps, based on the relative poses between the sub-map and the reference sub-map connected through adjacent sub-maps, the relative poses are multiplied sequentially to obtain the transformation matrix of the sub-map from its own coordinate system to the global reference coordinate system.

[0031] Based on the transformation matrix, the initial pose of each sub-map is transformed to the global reference coordinate system to obtain the global pose of each sub-map.

[0032] Optionally, the step of constructing a pose transformation chain between sub-maps based on the global pose, performing geometric consistency verification on multiple global poses of the same sub-map obtained through different pose transformation chains, and removing the relative poses that fail the verification, includes:

[0033] Based on the overlap relationship between sub-maps and the relative poses, at least one pose transformation chain is constructed from the base sub-map to each sub-map.

[0034] When a submap has more than two different pose transformation chains, calculate the global pose transmitted to the submap through each pose transformation chain.

[0035] Calculate the difference in rotation angle and translation distance between different global poses as geometric residuals;

[0036] The geometric residual is compared with a preset threshold.

[0037] If the geometric residual is less than the threshold, the corresponding pose transformation chain is deemed to have passed the verification.

[0038] If the geometric residual is greater than or equal to the threshold, the corresponding pose transformation chain is determined to have failed the verification. The relative poses constituting the pose transformation chain are removed, and the recall and matching steps are re-executed for the overlapping areas between the sub-maps involved in the pose transformation chain.

[0039] Optionally, the step of stitching the sub-map images of each sub-map to the same coordinate system based on the verified global pose to generate a fused map includes:

[0040] Based on the global pose verified by geometric consistency, the sub-map images of each sub-map are unified to the global coordinate system through rigid transformation;

[0041] The transformed sub-map images are stitched together to generate a merged map.

[0042] Secondly, embodiments of this application provide a map fusion device based on baseless visual guidance, the device comprising:

[0043] The visual control point image acquisition module is used to acquire sub-map images of different sub-maps and to acquire visual control point images with the same labels at the same physical location in the overlapping area of ​​each sub-map.

[0044] The association index building module is used to extract the visual features of each visual control point image and build an association index between the identifier and the visual features;

[0045] The visual control point image retrieval module is used to retrieve visual control point images of other sub-maps with the same identifier from the associated index based on the identifier of the visual control point images in the current sub-map.

[0046] The relative pose matching module is used to match the visual features of the current visual control point image with the visual features of the recalled visual control point images of the same name to obtain the relative pose between the current sub-map and other sub-maps.

[0047] The global pose determination module is used to obtain the initial pose of each sub-map in its respective coordinate system, and to transform the initial pose of each sub-map to the same reference coordinate system using the relative pose to obtain the global pose of each sub-map.

[0048] The consistency verification module is used to construct a pose transformation chain between sub-maps based on the global pose, perform geometric consistency verification on multiple global poses of the same sub-map obtained through different pose transformation chains, and remove the relative poses that fail the verification.

[0049] The fused map generation module is used to stitch the sub-map images of each sub-map to the same coordinate system based on the verified global pose, thereby generating a fused map.

[0050] Optionally, the step of extracting visual features from the images of each visual control point and establishing an association index between the identifier and the visual features includes:

[0051] A dedicated image library is constructed for each visual control point corresponding to the aforementioned identifier, wherein the dedicated image library contains multiple images of visual control points acquired at the same physical location from different angles or at different times;

[0052] Extract the local visual feature vector of each visual control point image in the dedicated image library, and associate and store the local visual feature vector with the identifier;

[0053] A multimodal retrieval structure is established using the identifier as the primary index and the approximate retrieval structure of the local visual feature vector as the secondary index, thus obtaining the associated index.

[0054] Optionally, the step of recalling visual control point images of other sub-maps with the same identifier from the associated index based on the identifier of the visual control point image in the current sub-map includes:

[0055] Get the identifiers of the visual control point images contained in the current sub-map;

[0056] Using the identifier as a primary index, locate the dedicated image library corresponding to the identifier from the associated index;

[0057] Extract the visual feature vector of the current visual control point image;

[0058] The visual feature vector is used as a query vector and input into the exclusive image library located by the identifier in the associated index. The visual control point images that meet the preset threshold of similarity with the query vector are retrieved through the secondary index and used as the recalled visual control point images with the same name.

[0059] Optionally, the step of matching the visual features of the current visual control point image with the visual features of the recalled visual control point images of the same name to obtain the relative pose between the current sub-map and other sub-maps includes:

[0060] Feature point matching is performed between the current visual control point image and the recalled visual control point images with the same name.

[0061] The RANSAC algorithm is used to remove mismatched point pairs, and the geometric constraints of the remaining matching point pairs are verified by estimating the fundamental matrix or homography matrix.

[0062] Based on the verified matching point pairs, calculate the relative rotation matrix and translation vector between the current visual control point image and the recalled image, which serves as the relative pose between the current submap and other submaps.

[0063] Optionally, obtaining the initial pose of each sub-map in its respective coordinate system, and using the relative pose to transform the initial pose of each sub-map to the same reference coordinate system to obtain the global pose of each sub-map, includes:

[0064] The initial pose of each sub-map in its own reconstructed coordinate system is obtained through a simultaneous localization and mapping algorithm.

[0065] Select the coordinate system of one of the sub-maps as the global reference coordinate system;

[0066] For other sub-maps, based on the relative poses between the sub-map and the reference sub-map connected through adjacent sub-maps, the relative poses are multiplied sequentially to obtain the transformation matrix of the sub-map from its own coordinate system to the global reference coordinate system.

[0067] Based on the transformation matrix, the initial pose of each sub-map is transformed to the global reference coordinate system to obtain the global pose of each sub-map.

[0068] Optionally, the step of constructing a pose transformation chain between sub-maps based on the global pose, performing geometric consistency verification on multiple global poses of the same sub-map obtained through different pose transformation chains, and removing the relative poses that fail the verification, includes:

[0069] Based on the overlap relationship between sub-maps and the relative poses, at least one pose transformation chain is constructed from the base sub-map to each sub-map.

[0070] When a submap has more than two different pose transformation chains, calculate the global pose transmitted to the submap through each pose transformation chain.

[0071] Calculate the difference in rotation angle and translation distance between different global poses as geometric residuals;

[0072] The geometric residual is compared with a preset threshold.

[0073] If the geometric residual is less than the threshold, the corresponding pose transformation chain is deemed to have passed the verification.

[0074] If the geometric residual is greater than or equal to the threshold, the corresponding pose transformation chain is determined to have failed the verification. The relative poses constituting the pose transformation chain are removed, and the recall and matching steps are re-executed for the overlapping areas between the sub-maps involved in the pose transformation chain.

[0075] Optionally, the step of stitching the sub-map images of each sub-map to the same coordinate system based on the verified global pose to generate a fused map includes:

[0076] Based on the global pose verified by geometric consistency, the sub-map images of each sub-map are unified to the global coordinate system through rigid transformation;

[0077] The transformed sub-map images are stitched together to generate a merged map.

[0078] Thirdly, embodiments of this application provide a computer device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the map fusion method based on baseless visual guidance described in any of the optional embodiments of the first aspect are executed.

[0079] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the baseless visual guidance-based map fusion method described in any of the optional embodiments of the first aspect.

[0080] The technical solution provided in this application includes, but is not limited to, the following beneficial effects:

[0081] The process involves acquiring submap images from different submaps, and then acquiring images of visual control points (VNCs) with identical identifiers at the same physical location within the overlapping areas of each submap. This step provides raw data with visual anchor points for the entire map fusion process, serving as the foundation for all subsequent fusion operations. Planning the acquisition operations within the overlapping areas of the submaps provides a spatial basis for matching and fusion between submaps. Setting identical unique identifiers for VNCs at the same physical location provides a clear basis for the algorithm to identify control points with the same name and achieve accurate retrieval and matching across submaps. Simultaneously, the acquired submap images and VNC images provide complete raw data support for subsequent visual feature extraction and matching processes.

[0082] Visual features of each visual control point image are extracted, and an association index between the identifier and the visual features is established. Extracting the visual features of the visual control point images yields core feature bases that can be used for subsequent matching, providing quantifiable comparison standards for image matching across sub-maps. Establishing the association index between identifiers and visual features achieves precise binding between unique control point identifiers and visual features, constructing the basic system for feature retrieval. This lays the data foundation for subsequent name-based fast indexing and visual feature-based precise matching, avoiding aimless global traversal during subsequent retrieval and improving the overall efficiency of subsequent retrieval and matching.

[0083] Based on the identifiers of visual control point images in the current sub-map, visual control point images of other sub-maps with the same identifiers are retrieved from the associated index. Relying on the uniqueness of the control point identifiers for retrieval, the system can quickly locate visual control point images of other sub-maps with corresponding physical locations, significantly reducing the search space for visual feature matching. This fundamentally avoids random cross-regional retrieval, effectively reducing the amount of unnecessary computation in subsequent feature matching and improving the efficiency and accuracy of image retrieval. Simultaneously, accurate image retrieval provides a prerequisite for reducing the feature mismatch rate, ensuring that feature matching is only performed within the image range of control points with the same name.

[0084] The visual features of the current visual control point image are matched with the visual features of the recalled visual control point images of the same name to obtain the relative pose between the current submap and other submaps. Performing visual feature matching within the range of images of the same name effectively reduces the image mismatch rate and improves the accuracy of feature matching results. The relative pose between submaps calculated based on the effective matching results accurately describes the coordinate system transformation relationship between different submaps. This provides core quantitative parameters for unifying the initial poses of each submap to the same reference coordinate system, which is crucial for achieving submap coordinate system unification. The accuracy of its calculation directly determines the reliability of subsequent global pose determination.

[0085] The initial poses of each sub-map in its own coordinate system are obtained. Using the relative poses, the initial poses of each sub-map are transformed to the same reference coordinate system, resulting in the global pose of each sub-map. The initial poses of each sub-map in its own coordinate system provide a stable local spatial reference frame, ensuring the pose consistency of each sub-map. By using relative poses to transform the initial poses of all sub-maps to the same reference coordinate system, the coordinate system differences between the sub-maps are successfully eliminated, achieving global uniformity of sub-map poses. This gives the previously independent sub-maps a unified spatial reference standard, laying a spatial foundation for subsequent global geometric consistency verification and map stitching fusion.

[0086] Based on the global pose, a pose transformation chain is constructed between sub-maps. Geometric consistency verification is performed on multiple global poses of the same sub-map obtained through different pose transformation chains, and the relative poses that fail the verification are removed. The pose transformation chain constructed based on the global pose enables pose transfer from the base sub-map to all sub-maps, establishing a global geometric verification framework. By performing geometric consistency verification on the global poses of different paths within the same sub-map, incorrectly matched relative poses caused by factors such as local visual ambiguity can be accurately identified. Consequently, the relative poses that fail the verification are systematically removed, avoiding the impact of incorrect matching results on the subsequent fusion process. This effectively improves the geometric consistency of the global poses of all sub-maps and ensures the overall accuracy of map fusion.

[0087] Based on the verified global pose, the sub-map images of each sub-map are stitched together to the same coordinate system to generate a fused map. Using the globally verified pose as the stitching basis, each sub-map can achieve precise and rigid stitching within a unified global coordinate system, ensuring the spatial geometric consistency of the fused map and realistically restoring the physical spatial relationships of the scene to be reconstructed. Simultaneously, stitching all sub-map images together to the same coordinate system successfully achieves integrated fusion of multiple sub-maps, giving the fused map both integrity and accuracy, meeting the core accuracy and scene restoration requirements of large-scene 3D reconstruction for map fusion.

[0088] The seven interconnected and progressively advancing steps form a complete map fusion technology workflow. From raw data acquisition with visual anchor points, to visual feature extraction and index building, then to identifier-based image retrieval, feature matching, and relative pose calculation, ultimately achieving global consistency of sub-map poses, eliminating erroneous matches through geometric consistency verification, and finally completing precise stitching and fusion, each step provides reliable technical support for subsequent stages. Through the synergistic effect of each step, the entire process effectively reduces the image mismatch rate, improves the retrieval efficiency, matching accuracy, and pose consistency of map fusion, successfully achieving accurate and efficient fusion of multiple sub-maps in the same coordinate system, ensuring the spatial accuracy and scene integrity of the fused map, and significantly enhancing the robustness of the map fusion process.

[0089] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0090] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0091] Figure 1 shows a flowchart of a baseless visual guidance-based map fusion method provided in Embodiment 1 of this application;

[0092] Figure 2 shows a flowchart of a method for establishing an association index provided in Embodiment 1 of this application;

[0093] Figure 3 shows a flowchart of a visual control point image retrieval method provided in Embodiment 1 of this application;

[0094] Figure 4 shows a flowchart of a relative pose determination method provided in Embodiment 1 of this application;

[0095] Figure 5 shows a flowchart of a global pose determination method provided in Embodiment 1 of this application;

[0096] Figure 6 shows a flowchart of a consistency verification method provided in Embodiment 1 of this application;

[0097] Figure 7 shows a flowchart of a fused map generation method provided in Embodiment 1 of this application;

[0098] Figure 8 shows a schematic diagram of a baseless visual guidance-based map fusion device provided in Embodiment 2 of this application;

[0099] Figure 9 shows a schematic diagram of the structure of a computer device provided in Embodiment 3 of this application. Detailed Implementation

[0100] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0101] Example 1

[0102] To facilitate understanding of this application, the following detailed description of Embodiment 1 of this application, in conjunction with the flowchart of the baseless visual guidance-based map fusion method shown in Figure 1, will be provided.

[0103] Referring to Figure 1, Figure 1 shows a flowchart of a map fusion method based on baseless visual guidance provided in Embodiment 1 of this application, wherein the method includes steps S101 to S107:

[0104] S101: Collect sub-map images of different sub-maps, and collect images of visual control points with the same labels at the same physical location within the overlapping area of ​​each sub-map.

[0105] Specifically, this step is the foundation of algorithm fusion. Its core is to provide raw data with "visual anchor points" for subsequent operations. The data acquisition process relies on mobile acquisition devices such as PortalCam (mobile acquisition device) and is divided into three core stages: scene planning, visual control point acquisition, and data storage.

[0106] During the scene planning phase, the large scene to be reconstructed needs to be logically divided into several independently collectible sub-map areas, and it is essential to ensure that there are overlapping areas between adjacent sub-maps to provide a foundation for subsequent map fusion.

[0107] Visual control point acquisition needs to be divided into three categories based on the different fusion scenarios: ground-to-ground fusion, air-to-ground fusion, and high-definition re-capture. Each category of control point has clear acquisition specifications: Ground-to-ground fusion control points must be planned in the overlapping area of ​​any two sub-maps to be merged, with a straight-line distance between points greater than 10 meters, located in well-lit, textured open spaces, and control points at the same physical location in different sub-maps must be given the exact same name as unique identifiers; Air-to-ground fusion control points must be planned at the drone takeoff point, with adjacent points spaced approximately 100 meters apart, and drone takeoff data must correspond one-to-one with control point data; High-definition re-capture control points must be located in the area to be re-captured and facing the target to be re-captured, and the scanning starting point of the re-capture device should be located above the control point.

[0108] For all types of control point acquisition operations, the equipment should be placed stably at the control point location. Select the corresponding type of control point and set the name through equipment applications such as LCC Scan (map fusion acquisition application). It is also recommended to take multiple angle shots around the control point to obtain 360° image coverage. The number of images should be about six or seven, which can be adjusted according to the actual situation of the acquisition equipment. When acquiring control points with air-ground fusion, it is also necessary to ensure that the orientation of the equipment is consistent with the orientation of the drone lens.

[0109] During the data storage phase, all collected data, including images of each sub-map, images of different types of control points, and related metadata, must be stored uniformly according to preset rules. Image data of control points with the same name will be stored together to provide a data index for the subsequent visual recall phase.

[0110] S102: Extract the visual features of each visual control point image and establish an association index between the identifier and the visual features.

[0111] Specifically, visual feature extraction is the "memory" step of the algorithm. A visual feature database that can be used for fast retrieval and matching needs to be built. Feature extraction is divided into global image feature extraction and local feature extraction of control point images. The extracted feature vectors must all have rotation and scale invariance to provide a reliable basis for subsequent matching.

[0112] Feature extraction can be achieved using two technical approaches. One is based on deep learning CNN (Convolutional Neural Network) models, using existing networks such as VGG (Visual Geometry Group) and pre-trained ResNet (Residual Network) without any improvements to the network structure. The other is to use traditional feature algorithms such as SIFT (Scale-Invariant Feature Transform) and ORB (Oriented Fast and Rotated BRIEF). Both methods can effectively extract global image feature vectors and local visual feature vectors of control point images.

[0113] The association index is built based on a multimodal index structure. First, a dedicated image library is built for each visual control point identified by a unique name. Then, the control point name is associated with the local visual feature vector of the corresponding image library and stored. Finally, a two-layer index is built to achieve accurate binding between the identifier and visual features, providing a foundation for subsequent visual retrieval.

[0114] S103: Based on the identifier of the visual control point image in the current sub-map, recall visual control point images of other sub-maps with the same identifier from the associated index.

[0115] Specifically, the recall process adopts a hierarchical visual recall mechanism of "coarse recall based on name index + fine recall based on visual feature index", which is the key to reducing the false matching rate of images and improving the matching efficiency. It is divided into two steps: initial recall based on name and precise recall based on visual features.

[0116] In the initial recall phase based on name, when fusing target sub-maps, the algorithm first reads the list of all control point names contained in the sub-map. Using this as the retrieval basis, it quickly locates the dedicated image library of control points with the same name in the associated index, directly skipping the image data of irrelevant control points and significantly narrowing the search scope.

[0117] In the precise recall stage based on visual features, the algorithm first calculates the visual feature vector of the control point image in the current sub-map, then queries the global control point image library to recall all historical control point images under that name, and measures the similarity of feature vectors by calculating cosine distance, Euclidean distance and other methods.

[0118] The algorithm sets a similarity threshold based on a fixed empirical value. This threshold is derived from testing a large amount of data and is not tied to a specific scene. Then, based on the threshold, it selects the Top-K most similar images from historical images as a candidate matching set to complete the accurate recall operation.

[0119] S104: Match the visual features of the current visual control point image with the visual features of the recalled visual control point images of the same name to obtain the relative pose between the current sub-map and other sub-maps.

[0120] Specifically, feature matching is not just a simple feature point comparison, but a complete process that includes robust matching and geometric verification. It also has a dual mismatch rate reduction mechanism to ensure the accuracy of the matching results and provide a reliable feature point pair basis for relative pose calculation.

[0121] In the robust matching and geometric verification stage, for each pair of current control point images and historical control point images in the candidate matching set, the RANSAC (Random Sample Consensus) algorithm is used to perform coarse matching of feature point pairs. Then, the geometric constraint verification of the coarsely matched feature point pairs is performed by estimating the fundamental matrix or homography matrix, eliminating erroneous matching pairs and retaining valid matching results.

[0122] In the relative pose calculation stage, for valid matching point pairs that have passed geometric verification, the relative pose transformation relationship between the two images is calculated. Specifically, it is represented by the rotation matrix R and the translation vector t. This transformation relationship is the relative pose between the current sub-map and other sub-maps.

[0123] The mismatch rate reduction mechanism includes two aspects: first, control point constraints, which strictly limit the matching operation to the image library of control points with the same name, thus fundamentally avoiding random mismatches across regions; second, multi-control point fusion, which requires at least 3 control points to provide effective relative poses at the same time. If there are more than 3 control points, fusion can also be achieved. The more control points there are, the better the fusion stability. At the same time, global optimization is performed through Bundle Adjustment to further eliminate abnormal matching results that may exist for individual control points.

[0124] S105: Obtain the initial pose of each sub-map in its respective coordinate system, and use the relative pose to transform the initial pose of each sub-map to the same reference coordinate system to obtain the global pose of each sub-map.

[0125] Specifically, this step is the core of realizing the sub-map coordinate system one, which is divided into four stages: initial pose acquisition, reference coordinate system selection, relative pose transfer, and global pose generation. Through coordinate system transformation, the originally independent sub-maps have a unified spatial reference standard, laying the foundation for subsequent map stitching and fusion.

[0126] During the initial pose acquisition phase, each independently acquired sub-map obtains its initial pose information in its own coordinate system through SLAM (Simultaneous Localization and Mapping) algorithm or other methods during the reconstruction process. This information includes rotation and translation parameters, which can provide a relatively stable local reference frame for the sub-map.

[0127] During the reference coordinate system selection phase, the algorithm selects the coordinate system of one of the sub-maps as the global reference coordinate system. Usually, the coordinate system of the first acquired sub-map is selected first, and this serves as a unified reference to ensure the consistency of pose transformation of all subsequent sub-maps.

[0128] In the relative pose transfer stage, the relative poses between adjacent sub-maps obtained in step S104 are represented in the form of transformation matrix T. For a sub-map that is not directly adjacent to the reference sub-map, the transformation matrices T are multiplied in sequence according to the relative poses between it and the reference sub-map through the concatenation of adjacent sub-maps to obtain the overall transformation matrix of the sub-map from its own coordinate system to the global reference coordinate system.

[0129] In the global pose generation stage, based on the calculated global transformation matrix, the initial poses of each sub-map in its own coordinate system are transformed to the global reference coordinate system one by one. After the transformation is completed, the unified global pose of each sub-map can be obtained.

[0130] S106: Construct a pose transformation chain between sub-maps based on the global pose, perform geometric consistency verification on multiple global poses of the same sub-map obtained through different pose transformation chains, and remove the relative poses that fail the verification.

[0131] Specifically, this step is a key step in the algorithm to eliminate incorrect matches and improve the reliability of the fusion results. It relies on the construction of the pose transformation chain and closed-loop detection to realize global geometric consistency verification. For results that fail the verification, error elimination and rematching operations will be performed to ensure that all retained global poses have spatial geometric consistency.

[0132] In the pose transformation chain construction stage, for a fusion task involving multiple sub-maps, the pose transformation chain from the reference sub-map to all other sub-maps is constructed using the relative pose transformation matrix T between adjacent sub-maps. Before construction, the consistency of a single map is ensured, and the cumulative error of a single map is eliminated through loop closure correction, assuming that the cumulative error elimination of a single map has been completed.

[0133] In the geometric consistency verification stage, firstly, sub-maps with "closed loops" are identified through loop detection, that is, sub-maps that have a direct or indirect overlap with the base sub-map or the merged map group. Then, the two global poses of the sub-map are calculated through two different paths (direct matching path and pose chain propagation path). Subsequently, the geometric residual between the two global poses is calculated, mainly including the difference in rotation angle and the difference in translation distance. The geometric residual is compared with a preset threshold. If the residual is less than the threshold, the matching relationship is determined to be geometrically consistent and is retained; if the residual exceeds the threshold, it is determined to be an incorrect match.

[0134] In the error removal and rematching stage, for sub-maps that are determined to be mismatched, the algorithm will automatically remove all of their current matching relationships and backtrack to the visual recall and matching steps to repeat the operation. During the process, the similarity threshold can be adjusted or the candidate matching set can be expanded to find more reliable matching relationships. If a geometrically consistent result cannot be obtained after multiple rematches, the algorithm will prompt the user to check the collected data or control point settings of the sub-map.

[0135] S107: Based on the verified global pose, stitch the sub-map images of each sub-map to the same coordinate system to generate a fused map.

[0136] Specifically, this step is the final output stage of the algorithm and needs to be carried out under the premise that the global pose of all sub-maps is geometrically consistent. It not only completes the physical stitching of sub-map data, but also outputs the corresponding result data. At the same time, the implementation of this step relies on the core innovation of the algorithm's "baseless visual guidance", which does not require a special base or high-precision positioning equipment, and completes the fusion only through visual information.

[0137] During the map stitching stage, the algorithm integrates the point cloud, grid, or image data of all sub-maps into a unified global coordinate system based on their global poses that have passed geometric consistency verification, forming the final fused map. Rigid stitching ensures that the spatial structure of the sub-map data does not change, restoring the real physical scene.

[0138] In the output phase, in addition to the core fusion result of the global fused point cloud / model, the algorithm also outputs two types of supporting data: first, a global pose list for each sub-map, which clearly shows the pose information of each sub-map in the global coordinate system; and second, a control point matching report, which contains detailed information on all successful and incorrect control point matching, providing a basis for subsequent result verification and data optimization.

[0139] In an optional implementation, referring to Figure 2, which shows a flowchart of an association index establishment method provided in Embodiment 1 of this application, wherein the step of extracting visual features of each visual control point image and establishing an association index between the identifier and the visual features includes steps S201~S203:

[0140] S201: Construct a dedicated image library for each visual control point corresponding to the aforementioned identifier, wherein the dedicated image library contains multiple visual control point images acquired at the same physical location from different angles or at different times.

[0141] Specifically, the dedicated image database is the basic carrier for visual feature retrieval. It is constructed using the land cover name of the control point as the unique identifier. Each identifier corresponds to a dedicated image database, and they do not overlap, ensuring the accuracy of retrieval.

[0142] The image library contains all relevant images acquired during the acquisition phase of the control point, including 360° coverage images taken from multiple angles around the control point, as well as images of the same control point at different times. The three types of control points—ground-to-ground fusion, air-to-ground fusion, and high-definition re-enhancing—are each constructed with their own dedicated image libraries according to this rule to ensure the retrieval needs of various fusion scenarios.

[0143] S202: Extract the local visual feature vector of each visual control point image in the dedicated image library, and associate and store the local visual feature vector with the identifier.

[0144] Specifically, the extraction of local visual feature vectors should follow the same technical standards as the extraction of global image features. Pre-trained CNN deep learning models such as ResNet and VGG, or traditional feature algorithms such as SIFT and ORB can be used to ensure that the extracted feature vectors have rotation and scale invariance and can be effectively matched with the image features of the same control points of other sub-maps.

[0145] The associated storage must be executed according to preset rules, binding the extracted local visual feature vectors with the corresponding unique identifiers of control points. At the same time, all collected data, including control point images, local visual feature vectors and related metadata, are stored according to unified rules, providing clear and standardized data support for subsequent index building and visual retrieval.

[0146] S203: Using the identifier as the primary index and the approximate retrieval structure of the local visual feature vector as the secondary index, a multimodal retrieval structure is established to obtain the associated index.

[0147] Specifically, the multimodal retrieval structure is a two-layer index system, with the primary and secondary indexes each performing their respective functions, forming a retrieval path from coarse to fine, which greatly improves the retrieval efficiency of visual features and provides a foundation for quickly completing visual recall.

[0148] The primary index is built based on the unique identifier of the control point name. Its core function is to enable rapid location of the dedicated image library for a specific control point, avoiding traversing all image libraries during retrieval and fundamentally narrowing the search scope.

[0149] The secondary index is built within a dedicated image library for each control point. It is based on an approximate retrieval structure of local visual feature vectors and can use structures such as KD-tree (k-dimension tree) and inverted index. Its core function is to support efficient similarity retrieval of visual feature vectors within the library, and can quickly filter out candidate images with similar features from the images in the library.

[0150] In an optional implementation, referring to Figure 3, Figure 3 shows a flowchart of a visual control point image retrieval method provided in Embodiment 1 of this application, wherein the step of retrieving visual control point images of other sub-maps with the same identifier from the association index based on the identifier of the visual control point images in the current sub-map includes steps S301~S304:

[0151] S301: Obtain the identifiers of the visual control point images contained in the current sub-map.

[0152] Specifically, this identifier is a unified name set for various types of control points when acquiring visual control points using devices such as LCC Scan. It is the sole basis for the algorithm to identify control points at the same physical location. The method of acquisition is to directly extract this name identifier from the control point image metadata of the current sub-map without any additional calculation or retrieval operations.

[0153] S302: Using the identifier as a primary index, locate the dedicated image library corresponding to the identifier from the associated index.

[0154] Specifically, relying on a first-level index built based on control point names, the algorithm can directly match the same name identifier in the associated index, quickly locate the corresponding control point's dedicated image library, and skip the image library data of other irrelevant control points. This step is a coarse recall based on names, which can significantly reduce the computational load of subsequent visual feature retrieval and improve the overall recall efficiency.

[0155] S303: Extract the visual feature vector of the current visual control point image.

[0156] Specifically, when extracting the visual feature vector, the same algorithm used when extracting local visual feature vectors as when building the dedicated image library should be adopted. Pre-trained CNN models such as ResNet and VGG, or traditional feature algorithms such as SIFT and ORB can be selected to ensure that the extracted feature vector has rotation and scale invariance and can effectively measure similarity with the feature vectors in the dedicated image library.

[0157] The extracted feature vectors will serve as query vectors for subsequent similarity retrieval. The accuracy of their extraction directly affects the results of subsequent precise recall. Therefore, it is necessary to strictly follow a unified feature extraction standard to ensure matching with feature vectors in the database.

[0158] S304: The visual feature vector is used as a query vector and input into the exclusive image library located by the identifier in the associated index. The visual control point images that meet the preset threshold in similarity with the query vector are retrieved through the secondary index and used as the recalled visual control point images with the same name.

[0159] Specifically, this step is a precise recall based on visual features, which is completed by relying on the secondary index in the dedicated image database. During the retrieval, the query vector is used as the basis, and the similarity of all local visual feature vectors in the database is measured through approximate retrieval structures such as KD trees and inverted indexes.

[0160] The similarity measurement method is to calculate the cosine distance or Euclidean distance between the query vector and the feature vector in the database. The smaller the distance, the higher the feature similarity. The algorithm will filter images based on a preset similarity threshold. This threshold is a fixed empirical value, which is derived through testing a large amount of data and is not bound to a specific scenario.

[0161] The algorithm will select images from the database that meet the preset threshold of similarity to form a candidate matching set of the Top-K most similar historical images. This candidate set is the final recalled images of visual control points with the same name, providing a data foundation for subsequent feature matching.

[0162] In an optional implementation, referring to Figure 4, which shows a flowchart of a relative pose determination method provided in Embodiment 1 of this application, wherein the step of matching the visual features of the current visual control point image with the visual features of the recalled visual control point images of the same name to obtain the relative pose between the current sub-map and other sub-maps includes steps S401~S403:

[0163] S401: Perform feature point matching between the current visual control point image and the recalled visual control point images with the same name.

[0164] Specifically, this step is the coarse feature point matching stage. The core is to perform a preliminary feature point comparison between the query vector extracted from the current control point image and the local visual feature vector of the recalled control point images with the same name. By calculating the cosine distance, Euclidean distance and other similarity measures between the feature vectors, preliminary feature point pairs are selected.

[0165] The coarse matching stage does not perform geometric constraint verification, but only performs direct comparison of feature vectors. Therefore, it will include some erroneous matching point pairs caused by local visual ambiguity. Its purpose is to provide a basic set of feature point pair candidates for subsequent accurate matching, thereby reducing the computational cost of subsequent geometric verification.

[0166] S402: The RANSAC algorithm is used to remove mismatched point pairs, and the geometric constraints of the remaining matching point pairs are verified by estimating the fundamental matrix or homography matrix.

[0167] Specifically, this step is the core of achieving accurate feature point matching. It consists of two parts: removing mismatched point pairs and verifying geometric constraints. Through this dual screening, we ensure that the final retained feature point pairs are valid matching results, providing a reliable basis for relative pose calculation.

[0168] First, the RANSAC algorithm is used to filter the feature point pairs after coarse matching. This algorithm can effectively identify and eliminate erroneous matching point pairs caused by factors such as local visual ambiguity and image noise, and retain feature point pairs with preliminary geometric correlation.

[0169] Then, the remaining matching point pairs after the RANSAC algorithm are geometrically constrained by the estimation of the fundamental matrix or homography matrix. The spatial geometry principle is used to determine whether the feature point pairs conform to the real physical spatial position relationship, and the residual erroneous matching point pairs are further eliminated, so as to finally obtain accurate and effective feature point pairs.

[0170] S403: Based on the verified matching point pairs, calculate the relative rotation matrix and translation vector between the current visual control point image and the recalled image, which serves as the relative pose between the current sub-map and other sub-maps.

[0171] Specifically, the verified matching point pairs reflect the spatial correspondence between the current control point image and the recalled image. Based on this correspondence, the relative pose transformation relationship between the two images can be obtained through spatial geometric calculations. This relationship is specifically represented by the rotation matrix R and the translation vector t.

[0172] The rotation matrix R describes the relative rotation angle between two images in space, and the translation vector t describes the relative translation distance between the two images in space. The combination of the two can fully represent the relative pose between the current submap and other submaps. This relative pose is stored in the form of a transformation matrix T, which provides core parameters for subsequent coordinate system transformations.

[0173] If multiple control points with the same name provide valid matching point pairs, global optimization will be performed based on all relative pose results through Bundle Adjustment to eliminate abnormal matching results of individual control points and improve the accuracy and stability of relative pose.

[0174] In an optional implementation, referring to Figure 5, which shows a flowchart of a global pose determination method provided in Embodiment 1 of this application, the step of obtaining the initial pose of each sub-map in its respective coordinate system and transforming the initial pose of each sub-map to the same reference coordinate system using the relative pose to obtain the global pose of each sub-map includes steps S501~S504:

[0175] S501: Obtain the initial pose of each sub-map in its own reconstructed coordinate system through a synchronous localization and mapping algorithm.

[0176] Specifically, the Simultaneous Localization and Mapping (SLAM) algorithm is a commonly used method for obtaining pose information during the independent reconstruction of sub-maps. This algorithm can construct a reconstruction coordinate system for each sub-map and obtain the initial pose information in that coordinate system without the need for external positioning devices.

[0177] The initial pose information includes rotation and translation parameters, which can completely describe the spatial position of each acquired data in the sub-map in its own local coordinate system, providing a relatively stable local reference system for the sub-map. This pose information is also stored in association with the sub-map's imagery, point cloud, and other data, providing a basis for subsequent coordinate system transformations.

[0178] Besides the SLAM algorithm, the initial pose of the submap can also be obtained through other methods. The algorithm does not have any special restrictions on the method of obtaining the initial pose, as long as the pose information contains complete rotation and translation parameters.

[0179] S502: Select the coordinate system of one of the sub-maps as the global reference coordinate system.

[0180] Specifically, the selection of a global reference coordinate system is a prerequisite for achieving a unified coordinate system for all sub-maps. The algorithm usually prioritizes the coordinate system of the first acquired sub-map as the global reference coordinate system. Using this as a standard, subsequent pose transformations can form a unified reference, ensuring the spatial consistency of the fused map.

[0181] There are no mandatory fixed rules for selecting the global reference coordinate system. It can be adjusted according to the actual fusion scenario and the operating habits of the data acquisition team. Just make sure that all sub-maps use this coordinate system as the reference for pose transformation.

[0182] S503: For other sub-maps, based on the relative poses between the sub-map and the reference sub-map connected through adjacent sub-maps, the relative poses are multiplied sequentially to obtain the transformation matrix of the sub-map from its own coordinate system to the global reference coordinate system.

[0183] Specifically, the relative pose between adjacent submaps is represented by a transformation matrix T, which contains transformation parameters from one submap coordinate system to another adjacent submap coordinate system. For a submap that is directly adjacent to the reference submap, its transformation matrix is ​​the relative pose transformation matrix T between the two.

[0184] For submaps that are not directly adjacent to the reference submap, the transformation matrices T between the connected adjacent submaps need to be multiplied sequentially according to the pose transformation chain between the submap and the reference submap. By transferring the pose, the overall transformation matrix of the submap from its own coordinate system to the global reference coordinate system is obtained. This matrix provides the core calculation parameters for subsequent pose transformation.

[0185] S504: Based on the transformation matrix, the initial pose of each sub-map is transformed to the global reference coordinate system to obtain the global pose of each sub-map.

[0186] Specifically, the core of pose transformation is to perform spatial geometric operations on the initial pose of the sub-map in its own coordinate system and the calculated global transformation matrix. Through matrix transformation, the initial pose is mapped to the global reference coordinate system, thus completing the pose unification of a single sub-map.

[0187] After all sub-maps have completed pose transformation according to this rule, the global pose of each sub-map in the global reference coordinate system can be obtained. This global pose includes the rotation and translation parameters of the sub-maps in a unified coordinate system, which can clearly describe the spatial positional relationship of each sub-map, providing a foundation for subsequent map stitching and geometric consistency verification.

[0188] The global pose is stored in association with all the acquired data of the submap, and all subsequent operations are carried out based on the global pose to ensure that all submap data are fused in the same spatial reference frame.

[0189] In an optional implementation, referring to Figure 6, which shows a flowchart of a consistency verification method provided in Embodiment 1 of this application, the step of constructing a pose transformation chain between sub-maps based on the global pose, performing geometric consistency verification on multiple global poses of the same sub-map obtained through different pose transformation chains, and removing the relative poses that fail the verification includes steps S601 to S606:

[0190] S601: Based on the overlap relationship between sub-maps and the relative poses, construct at least one pose transformation chain from the reference sub-map to each sub-map.

[0191] Specifically, the overlapping relationship between submaps is the basis for constructing pose transformation chains. Only adjacent submaps with overlapping areas can form nodes of the pose transformation chain through the relative pose transformation matrix T. Submaps without overlapping areas cannot directly construct pose transformation relationships.

[0192] The pose transformation chain is constructed starting from the reference submap and extending sequentially to all other submaps based on the overlap relationship and relative pose of adjacent submaps. For submaps that overlap with multiple submaps, at least two different pose transformation chains will be constructed, providing conditions for subsequent loop closure detection and geometric consistency verification.

[0193] Specifically, before constructing the pose transformation chain, loop closure correction is performed on each independent sub-map to eliminate its own cumulative error and ensure the pose consistency of the single map. The algorithm assumes that the cumulative error of the single map has been eliminated, and the pose transformation chain is constructed based on this assumption.

[0194] S602: When a submap has two or more different pose transformation chains, calculate the global pose transmitted to the submap through each pose transformation chain.

[0195] Specifically, for a submap with multiple pose transformation chains, each transformation chain represents a pose transfer path from the base submap to the current submap. The algorithm will multiply the relative pose transformation matrices T of each adjacent submap in the order of the nodes of each transformation chain to obtain the overall transformation matrix corresponding to each path.

[0196] Based on the overall transformation matrix of each path, the initial pose of the sub-map in its own coordinate system is transformed to the global reference coordinate system, resulting in multiple global poses transmitted through different pose transformation chains. These global poses are the pose results of the sub-map in the same global reference coordinate system, providing data for subsequent geometric residual calculation.

[0197] The pose calculation process strictly follows the matrix transformation rules of spatial geometry to ensure the accuracy of each global pose result and avoid the impact of calculation errors on subsequent geometric consistency verification.

[0198] S603: Calculate the difference in rotation angle and translation distance between different global poses as geometric residuals.

[0199] Specifically, geometric residual is the core indicator for measuring the deviation between global poses obtained from different paths. Its calculation is only for different global poses of the same sub-map under the same global reference coordinate system. The matching effectiveness of the pose transformation chain can be judged by the magnitude of the residual.

[0200] Rotation angle difference is the angular deviation between rotation matrices of different global poses, reflecting the inconsistency of pose in the spatial rotation dimension; translation distance difference is the spatial distance deviation between translation vectors of different global poses, reflecting the inconsistency of pose in the spatial translation dimension.

[0201] The geometric residual is composed of the difference in rotation angle and the difference in translation distance. Both are quantitative indicators that can intuitively reflect the degree of deviation in the transmission results of different pose transformation chains, providing a clear quantitative basis for subsequent threshold judgment.

[0202] S604: Compare the geometric residual with a preset threshold.

[0203] Specifically, the preset threshold is a quantitative standard for the algorithm to determine geometric consistency. This threshold is derived by testing a large amount of map fusion data from different scenarios. It is a fixed empirical value and is not bound to a specific fusion scenario. It can be applied to various map fusion scenarios such as ground-to-ground fusion, air-to-ground fusion, and high-definition reshooting.

[0204] The preset thresholds set corresponding judgment criteria for the difference in rotation angle and the difference in translation distance. Both the difference in rotation angle and the difference in translation distance must be less than the threshold to be judged as geometrically consistent. If either indicator exceeds the threshold, it is judged as geometrically inconsistent.

[0205] The threshold setting balances fusion accuracy and algorithm efficiency. While ensuring the spatial consistency of fusion results, it avoids the rejection of too many valid matching results due to an overly strict threshold setting, thereby improving the robustness of the algorithm.

[0206] S605: If the geometric residual is less than the threshold, the corresponding pose transformation chain is determined to have passed the verification.

[0207] Specifically, if the geometric residual is less than the preset threshold, it indicates that the global pose obtained through this pose transformation chain is consistent in spatial geometry, meaning that the relative poses between all adjacent sub-maps in this transformation chain are valid matching results, with no erroneous matching.

[0208] For pose transformation chains that pass the verification, their corresponding global poses will be retained as the effective pose results of the sub-map in the global reference coordinate system, providing a basis for subsequent map stitching and fusion. Furthermore, the transformation chain will continue to participate in the pose transfer and geometric consistency verification of other sub-maps.

[0209] S606: If the geometric residual is greater than or equal to the threshold, the corresponding pose transformation chain is determined to have failed the verification, the relative pose constituting the pose transformation chain is removed, and the recall and matching steps are re-executed for the overlapping areas between the sub-maps involved in the pose transformation chain.

[0210] Specifically, if the geometric residual is greater than or equal to a preset threshold, it indicates that there is an incorrectly matched relative pose in the pose transformation chain, resulting in spatial geometric inconsistency in the transmitted global pose. Therefore, the transformation chain needs to be culled and rematched.

[0211] First, all relative poses that make up the failed pose transformation chain will be automatically removed. These relative poses will no longer participate in subsequent pose transfer and map fusion operations, thus preventing the further spread of erroneous results.

[0212] Specifically, the visual recall and matching steps will then be revisited for the overlapping areas between all sub-maps involved in the transformation chain. The recall and feature matching operations of the control point images will be carried out again. During the process, the similarity threshold can be adjusted or the candidate matching set can be expanded according to the actual situation to find more reliable relative poses.

[0213] If, after multiple rematching operations, a global pose with a geometric residual less than a preset threshold cannot be obtained, the algorithm will terminate the rematching and prompt the user to check the data collected in that sub-map or the control point settings, such as whether the control point names are consistent, whether the collected images are clear, and whether the distribution of control points meets the requirements.

[0214] In an optional implementation, referring to Figure 7, which shows a flowchart of a fused map generation method provided in Embodiment 1 of this application, wherein the step of stitching the sub-map images of each sub-map to the same coordinate system based on the verified global pose to generate a fused map includes steps S701~S702:

[0215] S701: Based on the global pose verified by geometric consistency, the sub-map images of each sub-map are unified to the global coordinate system through rigid transformation.

[0216] Specifically, rigid transformation refers to maintaining the shape, size, and spatial structure of the sub-map data during spatial transformation, only performing coordinate system transformation and spatial position translation and rotation, thereby ensuring that the fused map can accurately restore the physical spatial relationships of the scene to be reconstructed and avoid data distortion.

[0217] The transformation is based on the global pose of each sub-map after passing the geometric consistency check. The algorithm will rigidly transform all the acquired data of the sub-map, such as images, point clouds, and grids, according to the rotation and translation parameters contained in the global pose, and map them to a unified global coordinate system to complete the coordinate system unification of all sub-map data.

[0218] During the transformation process, the global reference coordinate system will be used as the sole standard to ensure that the spatial position of all sub-map data in the global coordinate system is consistent with the actual physical scene, and the overlapping areas of adjacent sub-maps can be accurately connected, providing a foundation for subsequent stitching and fusion.

[0219] S702: The transformed sub-map images are stitched together to generate a merged map.

[0220] Specifically, the objects to be stitched and fused include not only the image data of the sub-map, but also all the 3D reconstruction data such as point clouds and grids collected from the sub-map. The algorithm will seamlessly stitch all the sub-map data that have undergone rigid transformation in the global coordinate system to form a complete global fused map.

[0221] The stitching and fusion process relies on the core advantage of the algorithm's "baseless visual guidance," which eliminates the need for high-precision physical control points such as total station targets, as well as dedicated bases or high-precision positioning equipment. It can achieve accurate fusion of multiple sub-maps simply by collecting visual control point information and image data. At the same time, the introduction of control points significantly reduces the image mismatch rate and improves the robustness and accuracy of large-scene reconstruction.

[0222] In addition to generating the core fused map output—a global fused point cloud / model—the algorithm also outputs supporting result data, including a global pose list for each sub-map and a control point matching report. The control point matching report contains detailed information on all successful and incorrect control point matches, facilitating users to subsequently verify, optimize, and apply the fused results.

[0223] This fusion method has strong scene adaptability. By adjusting the acquisition rules of control points, such as distribution, interval, and associated data, it can be flexibly applied to various scenarios that require large-scale, multi-view, and high-precision 3D reconstruction, such as city-level 3D modeling, large industrial park scanning, digitization of cultural relics and historical sites, and construction of integrated indoor and outdoor navigation maps.

[0224] Example 2

[0225] Referring to Figure 8, which shows a schematic diagram of a baseless visual guidance-based map fusion device provided in Embodiment 2 of this application, the device includes:

[0226] The visual control point image acquisition module 801 is used to acquire sub-map images of different sub-maps and to acquire visual control point images with the same label at the same physical location in the overlapping area of ​​each sub-map.

[0227] The association index establishment module 802 is used to extract the visual features of each visual control point image and establish an association index between the identifier and the visual features;

[0228] The visual control point image retrieval module 803 is used to retrieve visual control point images of other sub-maps with the same identifier from the associated index based on the identifier of the visual control point images in the current sub-map.

[0229] The relative pose matching module 804 is used to match the visual features of the current visual control point image with the visual features of the recalled visual control point images of the same name to obtain the relative pose between the current sub-map and other sub-maps.

[0230] The global pose determination module 805 is used to obtain the initial pose of each sub-map in its respective coordinate system, and to use the relative pose to transform the initial pose of each sub-map to the same reference coordinate system to obtain the global pose of each sub-map.

[0231] The consistency verification module 806 is used to construct a pose transformation chain between sub-maps based on the global pose, perform geometric consistency verification on multiple global poses of the same sub-map obtained through different pose transformation chains, and remove the relative poses that fail the verification.

[0232] The fused map generation module 807 is used to stitch the sub-map images of each sub-map to the same coordinate system based on the verified global pose, thereby generating a fused map.

[0233] In an optional implementation, the step of extracting visual features from the images of each visual control point and establishing an association index between the identifier and the visual features includes:

[0234] A dedicated image library is constructed for each visual control point corresponding to the aforementioned identifier, wherein the dedicated image library contains multiple images of visual control points acquired at the same physical location from different angles or at different times;

[0235] Extract the local visual feature vector of each visual control point image in the dedicated image library, and associate and store the local visual feature vector with the identifier;

[0236] A multimodal retrieval structure is established using the identifier as the primary index and the approximate retrieval structure of the local visual feature vector as the secondary index, thus obtaining the associated index.

[0237] In an optional implementation, the step of recalling visual control point images of other sub-maps with the same identifier from the associated index based on the identifier of the visual control point image in the current sub-map includes:

[0238] Get the identifiers of the visual control point images contained in the current sub-map;

[0239] Using the identifier as a primary index, locate the dedicated image library corresponding to the identifier from the associated index;

[0240] Extract the visual feature vector of the current visual control point image;

[0241] The visual feature vector is used as a query vector and input into the exclusive image library located by the identifier in the associated index. The visual control point images that meet the preset threshold of similarity with the query vector are retrieved through the secondary index and used as the recalled visual control point images with the same name.

[0242] In an optional implementation, matching the visual features of the current visual control point image with the visual features of the recalled visual control point images of the same name to obtain the relative pose between the current sub-map and other sub-maps includes:

[0243] Feature point matching is performed between the current visual control point image and the recalled visual control point images with the same name.

[0244] The RANSAC algorithm is used to remove mismatched point pairs, and the geometric constraints of the remaining matching point pairs are verified by estimating the fundamental matrix or homography matrix.

[0245] Based on the verified matching point pairs, calculate the relative rotation matrix and translation vector between the current visual control point image and the recalled image, which serves as the relative pose between the current submap and other submaps.

[0246] In an optional implementation, obtaining the initial pose of each sub-map in its respective coordinate system, and using the relative pose to transform the initial pose of each sub-map to the same reference coordinate system to obtain the global pose of each sub-map, includes:

[0247] The initial pose of each sub-map in its own reconstructed coordinate system is obtained through a simultaneous localization and mapping algorithm.

[0248] Select the coordinate system of one of the sub-maps as the global reference coordinate system;

[0249] For other sub-maps, based on the relative poses between the sub-map and the reference sub-map connected through adjacent sub-maps, the relative poses are multiplied sequentially to obtain the transformation matrix of the sub-map from its own coordinate system to the global reference coordinate system.

[0250] Based on the transformation matrix, the initial pose of each sub-map is transformed to the global reference coordinate system to obtain the global pose of each sub-map.

[0251] In an optional implementation, the step of constructing a pose transformation chain between sub-maps based on the global pose, performing geometric consistency verification on multiple global poses of the same sub-map obtained through different pose transformation chains, and removing the relative poses that fail the verification, includes:

[0252] Based on the overlap relationship between sub-maps and the relative poses, at least one pose transformation chain is constructed from the base sub-map to each sub-map.

[0253] When a submap has more than two different pose transformation chains, calculate the global pose transmitted to the submap through each pose transformation chain.

[0254] Calculate the difference in rotation angle and translation distance between different global poses as geometric residuals;

[0255] The geometric residual is compared with a preset threshold.

[0256] If the geometric residual is less than the threshold, the corresponding pose transformation chain is deemed to have passed the verification.

[0257] If the geometric residual is greater than or equal to the threshold, the corresponding pose transformation chain is determined to have failed the verification. The relative poses constituting the pose transformation chain are removed, and the recall and matching steps are re-executed for the overlapping areas between the sub-maps involved in the pose transformation chain.

[0258] In an optional implementation, the step of stitching the sub-map images of each sub-map to the same coordinate system based on the verified global pose to generate a fused map includes:

[0259] Based on the global pose verified by geometric consistency, the sub-map images of each sub-map are unified to the global coordinate system through rigid transformation;

[0260] The transformed sub-map images are stitched together to generate a merged map.

[0261] Example 3

[0262] Based on the same application concept, referring to Figure 9, Figure 9 shows a structural schematic diagram of a computer device provided in Embodiment 3 of this application. As shown in Figure 9, the computer device 900 provided in Embodiment 3 of this application includes:

[0263] The computer device 900 includes a processor 901, a memory 902, and a bus 903. The memory 902 stores machine-readable instructions that can be executed by the processor 901. When the computer device 900 is running, the processor 901 communicates with the memory 902 via the bus 903. When the machine-readable instructions are executed by the processor 901, they perform the steps of the map fusion method based on baseless visual guidance as described in Embodiment 1.

[0264] Example 4

[0265] Based on the same concept, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the map fusion method based on the baseless visual guidance described in any of the above embodiments.

[0266] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0267] The computer program product for performing baseless visual-guided map fusion provided in this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0268] The baseless visual guidance-based map fusion device provided in this application embodiment can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this application embodiment are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0269] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0270] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0271] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0272] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0273] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0274] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A map fusion method based on baseless visual guidance, characterized in that, The method includes: acquiring sub-map images of different sub-maps, and acquiring visual control point images with the same identifier at the same physical location within the overlapping area of ​​each sub-map; extracting visual features of each visual control point image and establishing an association index between the identifier and the visual features; based on the identifier of the visual control point image in the current sub-map, recalling visual control point images of other sub-maps with the same identifier from the association index; matching the visual features of the current visual control point image with the visual features of the recalled visual control point images with the same name to obtain the relative pose between the current sub-map and other sub-maps; obtaining the initial pose of each sub-map in its respective coordinate system, and using the relative pose to transform the initial pose of each sub-map to the same reference coordinate system to obtain the global pose of each sub-map; constructing a pose transformation chain between sub-maps based on the global pose, performing geometric consistency verification on multiple global poses of the same sub-map obtained through different pose transformation chains, and removing the relative poses that fail the verification; and stitching the sub-map images of each sub-map to the same coordinate system based on the verified global poses to generate a fused map.

2. The method according to claim 1, characterized in that, The step of extracting visual features from the images of each visual control point and establishing an association index between the identifier and the visual features includes: constructing a dedicated image library for each visual control point corresponding to the identifier, wherein the dedicated image library contains multiple images of visual control points acquired at the same physical location at different angles or at different times; extracting the local visual feature vector of each visual control point image in the dedicated image library, and storing the local visual feature vector in association with the identifier; establishing a multimodal retrieval structure with the identifier as the primary index and the approximate retrieval structure of the local visual feature vector as the secondary index to obtain the association index.

3. The method according to claim 2, characterized in that, The step of recalling visual control point images of other sub-maps with the same identifier from the associated index based on the identifier of the visual control point image in the current sub-map includes: obtaining the identifier of the visual control point image contained in the current sub-map; using the identifier as a primary index, locating the dedicated image library corresponding to the identifier from the associated index; extracting the visual feature vector of the current visual control point image; using the visual feature vector as a query vector, inputting it into the dedicated image library located by the identifier in the associated index, and retrieving visual control point images whose similarity to the query vector meets a preset threshold through the secondary index, as the recalled visual control point images with the same name.

4. The method according to claim 1, characterized in that, The step of matching the visual features of the current visual control point image with the visual features of the recalled visual control point images of the same name to obtain the relative pose between the current sub-map and other sub-maps includes: performing feature point matching between the current visual control point image and the recalled visual control point images of the same name; using the RANSAC algorithm to remove mismatched point pairs, and performing geometric constraint verification on the remaining matching point pairs through the estimation of the fundamental matrix or homography matrix; and calculating the relative rotation matrix and translation vector between the current visual control point image and the recalled image based on the verified matching point pairs, as the relative pose between the current sub-map and other sub-maps.

5. The method according to claim 1, characterized in that, The process of obtaining the initial pose of each sub-map in its own coordinate system and transforming the initial pose of each sub-map to the same reference coordinate system using the relative pose to obtain the global pose of each sub-map includes: obtaining the initial pose of each sub-map in its own reconstructed coordinate system through a simultaneous localization and mapping algorithm; selecting the coordinate system of one sub-map as the global reference coordinate system; for other sub-maps, multiplying the relative poses sequentially based on the relative poses between the sub-map and the reference sub-map via adjacent sub-maps to obtain the transformation matrix of the sub-map from its own coordinate system to the global reference coordinate system; and transforming the initial pose of each sub-map to the global reference coordinate system based on the transformation matrix to obtain the global pose of each sub-map.

6. The method according to claim 1, characterized in that, The step of constructing pose transformation chains between sub-maps based on the global pose, performing geometric consistency checks on multiple global poses of the same sub-map obtained through different pose transformation chains, and removing the relative poses that fail the check, includes: constructing at least one pose transformation chain from a base sub-map to each sub-map based on the overlap relationship between sub-maps and the relative poses; when a sub-map has two or more different pose transformation chains, calculating the global poses transmitted to the sub-map through each pose transformation chain; calculating the rotation angle difference and translation distance difference between different global poses as geometric residuals; comparing the geometric residuals with a preset threshold; if the geometric residuals are less than the threshold, determining that the corresponding pose transformation chain has passed the check; if the geometric residuals are greater than or equal to the threshold, determining that the corresponding pose transformation chain has failed the check, removing the relative poses constituting the pose transformation chain, and re-executing the recall and matching steps for the overlapping areas between the sub-maps involved in the pose transformation chain.

7. The method according to claim 1, characterized in that, The step of stitching the sub-map images of each sub-map to the same coordinate system based on the verified global pose to generate a fused map includes: unifying the sub-map images of each sub-map to the global coordinate system through rigid transformation according to the verified global pose; and stitching and fusing the transformed sub-map images to generate a fused map.

8. A map fusion device based on baseless visual guidance, characterized in that, The device includes: a visual control point image acquisition module, used to acquire sub-map images of different sub-maps, and to acquire visual control point images with the same identifier at the same physical location within the overlapping area of ​​each sub-map; an association index establishment module, used to extract visual features of each visual control point image and establish an association index between the identifier and the visual features; a visual control point image retrieval module, used to retrieve visual control point images of other sub-maps with the same identifier from the association index based on the identifier of the visual control point image in the current sub-map; and a relative pose matching module, used to match the visual features of the current visual control point image with the visual features of the retrieved visual control point images with the same name. The system performs matching to obtain the relative poses between the current sub-map and other sub-maps; a global pose determination module is used to obtain the initial poses of each sub-map in its respective coordinate system, and use the relative poses to transform the initial poses of each sub-map to the same reference coordinate system to obtain the global poses of each sub-map; a consistency verification module is used to construct a pose transformation chain between sub-maps based on the global poses, perform geometric consistency verification on multiple global poses of the same sub-map obtained through different pose transformation chains, and remove the relative poses that fail the verification; a fused map generation module is used to stitch the sub-map images of each sub-map to the same coordinate system based on the verified global poses to generate a fused map.

9. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the baseless visual guidance-based map fusion method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the baseless visual guidance-based map fusion method as described in any one of claims 1 to 7.

Images