Real-time puzzle method based on monocular SLAM fused with GPS

Abstract

The invention discloses a real-time puzzle method based on monocular SLAM (Simultaneous Localization and Mapping) fused with GPS (Global Positioning System), which is characterized in that an unmanned aerial vehicle is utilized to load a camera to shoot an image sequence under the camera and a GPS module is utilized to receive latitude and longitude signals, and high-precision real-time tracking and map construction of an outdoor large scene are realized by optimizing a feature extraction and matching algorithm of monocular ORBSLA3. Generating a sparse point cloud map and a position attitude of the key frame; a GPS module signal is subjected to UTM coordinate conversion, and fusion of the GPS module signal and key frame information is realized through linear search of a difference value; the key frame position attitude is converted to global UTM geographic coordinates through fusion transformation; overlapping region fusion is carried out on key frames with close spatial positions by using a Laplacian Gaussian pyramid to obtain an accurate panorama; according to the method, large-scene image splicing can be realized, the overlapping region splicing effect is good, the splicing real-time performance is good, and the method is suitable for application occasion requirements in multiple fields of outdoor aerial photography, aerial survey and the like.

Description

Technical Field

[0001] The invention belongs to the technical field of video image stitching, and in particular relates to a real-time stitching method based on monocular SLAM fused with GPS, which is applied to outdoor flight data collection of unmanned aerial vehicles. Background Art

[0002] When aerial survey drones acquire images, it is difficult for a single image to completely include the area of ​​interest due to the limitations of flight altitude and the focal length of digital cameras. In order to compensate for the defects of a small field of view of a single image and overlapping areas between multiple images, images acquired from different positions can be smoothly and seamlessly stitched and fused to expand the field of view, which helps to intuitively understand the content of the aerial photography.

[0003] CN201310628020.9A discloses local image extraction FAST+BRIEF to construct descriptor vectors as training image features, and uses hierarchical clustering method to quantify the image features into words and construct an offline vocabulary tree; in the online stage, the observed image features are retrieved according to the local descriptor using the vocabulary tree obtained in the training stage to obtain spatially adjacent images for matching, and the overlapping areas of the above matching images are fused using the fast interpolation method.

[0004] The above process has the following disadvantages in the application field of aerial survey UAVs: the offline vocabulary tree needs to be trained in advance before UAV splicing, and the vocabulary tree has insufficient representation ability for unknown target areas in aerial surveys; the feature descriptor vector cannot effectively handle the problem of rapid rotation.

[0005] CN111141295A, the present invention discloses an automatic map recovery method based on monocular ORB-SLAM, which solves the problem of map tracking failure due to complex environment by improving the ORB-SLAM mapping process, and can quickly restore tracking mapping without returning to the original scene for repositioning through RTK information-assisted vision, and automatically integrates the map information before loss into the current mapping system, thereby improving the mapping efficiency and ensuring the scale consistency and integrity of visual map construction.

[0006] The above invention uses the ORB-SLAM system to solve the problem of map tracking failure due to complex environment. Through the form of RTK information-assisted vision, it can quickly restore tracking and mapping without returning to the original scene for repositioning. However, it cannot solve the purpose of real-time puzzle of environmental scenes.

[0007] CN1 09029417A discloses a UAV SLAM method based on hybrid visual odometer and multi-scale map, equipped with a top-down monocular camera, a front-view binocular camera and an onboard computer. The monocular camera is used for visual odometer based on direct method, and the binocular camera is used for visual odometer based on feature point method. The hybrid visual odometer fuses the outputs of the two visual odometers, constructs a local map for positioning, and obtains the real-time position and posture of the UAV.

[0008] The above invention mainly relies on binocular cameras to achieve real-time posture estimation and environmental perception of the UAV in a non-GPS environment, and cannot achieve large-scale stitching of video images collected by the UAV.

[0009] CN102201115B discloses a method for stitching a real-time panoramic image of drone aerial video. The method uses a video acquisition card to capture images transmitted back to a base station by a drone through a microwave channel in real time, selects key frames for the image sequence, and performs image enhancement on the key frames. In the image stitching process, a SURF feature detection method with good robustness is firstly used to detect features of image frames and match between frames. A frame-to-panoramic image transformation method is then used to reduce the image multiplication cumulative error. In combination with the GPS position information of the drone, image frames whose flight paths are not adjacent in time sequence but adjacent in space are determined, the transformation relationship from frame to panorama is optimized, the image overlapping area is determined, image fusion and panorama construction are realized, and a real-time effect of stitching while flying is achieved.

[0010] The above invention uses SURF as the matching feature between image frames. The algorithm requires a large amount of operator resources and time overhead. In addition, GPS information is not fully utilized during the real-time stitching of large outdoor scenes, and there is stitching stacking at the overlap of multiple frames of images.

[0011] Traditional puzzles need to wait until all images are collected before they can be performed. It is necessary to calculate the visual overlap area between multiple groups of images and calculate the relative position and posture between the images. This method does not accept new images during the stitching process and lacks real-time performance in practical applications. At the same time, as the stitching area is continuously generated, the stitching time is greatly increased. Therefore, the development of a panoramic stitching method for large scenes has great market prospects. Summary of the invention

[0012] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a real-time puzzle method based on monocular SLAM fused with GPS, which adopts SLAM to realize real-time positioning and map construction of real-time image sequences of unmanned aerial vehicles, and combines the RTK sensor data of GPS for fusion, so as to realize the splicing of global maps.

[0013] In order to solve the technical problem, the technical method of the present invention is: a real-time puzzle method based on monocular SLAM fused with GPS, comprising the following steps:

[0014] Step 1: Obtain images and GPS information, and obtain the images captured by the monocular camera and the GPS information provided by the GPS in real time;

[0015] Step 2: Convert GPS information into UTM information;

[0016] Step 3: Extract ORB features from the image and rasterize it: construct a multi-layer pyramid model for the image in step 1, extract ORB features from each layer of the pyramid layered image, and rasterize the ORB features of each layer of the pyramid layered image;

[0017] Step 4: Initialization of visual odometry;

[0018] Step 5: Real-time tracking of visual odometer: Send the image and rasterized ORB features of step 1 to the ORBSLAM3 system, output the keyframe queue, and generate the keyframe map pose;

[0019] Step 6: Fuse the UTM information and the key frame map pose in step 5 to calculate the precise global real geographic coordinates of the key frame;

[0020] Step 7: Based on the global real geographic coordinates, use the Laplace pyramid multi-band to achieve real-time mosaic of large scenes.

[0021] Preferably, step 1 is specifically as follows: the drone is equipped with a camera to shoot the scene of the flight process, and then the video and GPS information of the GPS module are transmitted back to the control base station in real time through a microwave channel, an image acquisition card is installed on the video receiver to acquire image frames, and then transmitted to the working computer to achieve the acquisition of image frames and GPS information.

[0022] Preferably, the step 2 is specifically:

[0023] Step 2-1: Get GPS information: Get the GPS information to be converted, which includes latitude, longitude, latitude, altitude, and relative height information;

[0024] Step 2-2: Determine the UTM projection area: Determine the UTM projection area according to the given longitude information; Step 2-3: Calculate the central meridian: Calculate the central meridian longitude according to the UTM projection area; for zone 1, the central meridian longitude is -177 degrees, and for zones 2 to 60, the calculation formula for the central meridian longitude is: CentralMeridian = -177 + (ZoneNumber-1) * 6;

[0025] Step 2-4: Use the conversion formula of UTM projection to convert the GPS coordinates of the GPS information into UTM coordinates; the conversion formula of UTM projection is as follows:

[0026] N=(a / (1-e 2 *sin(lat) 2 )) 0.5 *k1*(A+(1-T+C)*A 3 / 6+(5-18*T+T 2 +72*C-58*e_prime 2 )*A 5 / 120)+false_northing

[0027] E=k1*(lon-lon0)*cos(lat)+E0

[0028] Where:

[0029] N is the UTM vertical axis coordinate;

[0030] E is the UTM horizontal axis coordinate;

[0031] lat is latitude, degrees;

[0032] lon is longitude, degrees;

[0033] lon0 is the longitude of the central meridian of the UTM projection, degrees;

[0034] a is the semi-major axis of the reference ellipsoid;

[0035] e 2 is the square of the first eccentricity of the reference ellipsoid;

[0036] kl is the scale factor;

[0037] A, C, e_prime, and T are intermediate variables calculated based on latitude;

[0038] E0, false_northing is the offset parameter of the UTM projection, which is used to align the coordinate origin to the specified area;

[0039] Step 2-5: Output the converted UTM coordinates, which include transverse Mercator coordinates and longitudinal Mercator coordinates, as well as the UTM projection zone and central meridian longitude.

[0040] Preferably, the step 3 is specifically:

[0041] Step 3-1: Construct a multi-layer pyramid model for the input image in step 1;

[0042] Step 3-2: Extract fast FAST corner point detection from each layer of pyramid layer image of the multi-layer pyramid model;

[0043] Step 3-3: For each FAST corner point, extract the 128-dimensional BRIEF descriptor as the ORB feature;

[0044] Step 3-4: rasterize the feature descriptor and assign each ORB feature descriptor to the set grid number;

[0045] Step 3-5: Feature matching, ORB feature extraction uses Hamming distance to measure the similarity between two feature descriptors;

[0046] Step 3-6: Sort the ORB feature similarity within 20 pixels based on the pixel distance between the previous and next two frames, and select the feature point with the greatest similarity as the matching point pair;

[0047] Step 3-7: Perform RANSAC random fitting on the matching point pairs and historical key point clouds of the previous and next two frames to calculate the relative motion pose between the two frames;

[0048] Step 3-8: In the initialization phase, the 3D points obtained by triangulation are used to generate map points. After the initialization is completed, the pose calculated is used to project the key points of the current frame to update the sparse point cloud.

[0049] Preferably, step 4 is specifically as follows: initializing ORB-SLAM3: before processing the image, the ORB-SLAM3 system needs to be initialized, including loading camera parameters, configuring algorithm parameters, and preparing data structures for tracking and mapping; using the visual odometer to calculate the moving distance between frame sequences, if it exceeds a pre-calculated threshold, the current frame is set as a key frame for storage; in the initialization stage, the point cloud is updated between multiple key frames using the pose calculated by SLAM to form the three-dimensional point cloud information for tracking initialization.

[0050] Preferably, the step 5 is specifically:

[0051] Step 5-1: Image processing: Send the image and rasterized ORB features from step 1 to the ORBSLAM3 system, and ORB-SLAM3 will perform feature extraction, feature matching, pose estimation, and map update;

[0052] Step 5-2: Key frame extraction: ORB-SLAM3 selects key frames based on motion and perspective change criteria. Key frames are important frames for subsequent map construction and posture estimation.

[0053] Step 5-3: Sparse point cloud generation: ORB-SLAM3 uses feature matching and triangulation techniques between key frames to generate sparse point clouds, which are obtained by converting matching feature points between key frames into points in three-dimensional space;

[0054] Step 5-4: Position and attitude estimation: ORB-SLAM3 estimates the position and attitude of the camera by tracking the motion between key frames and using the optimization-based nonlinear least squares method;

[0055] Step 5-5: Real-time update: As the video is processed, ORB-SLAM3 will continue to update the sparse point cloud and position and pose estimates to generate keyframe map poses.

[0056] Preferably, the step 6 is specifically:

[0057] Step 6-1: Collect the UTM information obtained in step 2 and the key frame map pose calculated by ORB-SLAM3;

[0058] Step 6-2: Find the two closest key frames in the timestamp sequence of the key frames calculated by ORB-SLAM3 through linear search according to their timestamp information;

[0059] Step 6-3: The real-time video is used to obtain the sparse point cloud and position posture of the key frame sequence through the ORB-SLAM3 algorithm, and the key frame information is filtered;

[0060] Step 6-4: Construct a UTM quadtree using UTM information and keyframe map poses;

[0061] Step 6-5: Filter the M neighboring points of the UTM quadtree, establish a similarity transformation model from the visual coordinate system to the real geographic coordinate system of the visual map, and obtain the precise global real geographic coordinates of the key frame.

[0062] Preferably, the sparse point cloud and position posture of the key frame sequence obtained by the ORB-SLAM3 algorithm in step 6-3 are as follows:

[0063] The sparse point cloud data is agreed to be in is the coordinate position of the jth map point of the sparse point cloud in the current local visual coordinate system; The superscript c in the figure indicates the camera visual coordinate system, and lm is the abbreviation of the map point landmark.

[0064] Position and posture in is the rotation matrix and translation vector of the transformation from the camera coordinate system to the current local visual coordinate system of the i-th key frame; then the position coordinates in the local visual coordinate system are in Where T represents the transpose operation of the matrix; the position of the key frame in the global UTM geographic coordinate system after interpolation is

[0065] Preferably, the filtering and screening of the key frame information in step 6-3 is specifically as follows:

[0066] 6-3-1: Calculate UTM global geographic distance: Use the UTM coordinates of the current keyframe and the previous keyframe to calculate the UTM global geographic distance (dis_utm) between them;

[0067] 6-3-2: Calculate visual distance: For the sparse point cloud of the current key frame, based on coordinate transformation Calculate the depth information of each point; quickly sort the depth of all points and take the point cloud depth of the median index as the depth (vis_depth) estimate of the current keyframe;

[0068] 6-3-3: Calculate the translation vector distance: Use the key frame translation vector (vis_dis) provided by ORB-SLAM3 to represent the relative translation distance between the current key frame and the previous key frame;

[0069] 6-3-4: Determine threshold condition: Calculate the ratio of vis_dis to vis_depth (vis_dis_ratio = vis_dis / vis_depth) and compare it with a preset threshold; if vis_dis_ratio exceeds the threshold, it means that the local visual distance of the current frame is large, and it is considered to correspond to a distant scene;

[0070] 6-3-5: Save or discard keyframe: If both the UTM global geographic distance (dis_utm) and the local visual distance (vis_dis_ratio) exceed the set threshold, the information of the current keyframe is saved; otherwise, the keyframe is discarded;

[0071] The steps 6-5 are specifically as follows:

[0072] Establish a similarity transformation model from the visual coordinate system to the real geographic coordinate system of the visual map to obtain the scaling, translation and rotation relationship of the visual information: where s cg is the scale factor and satisfies s cg >0, R cg ,t cg The three-dimensional space rotation matrix and translation vector from the local visual coordinate system to the UTM global geographic coordinate system;

[0073] Calculate the ratio of dis_utm to vis_dis between the query neighbor key frame information cg, and then calculate the average value as the initial scale factor for each key frame; the optimization least squares objective function is used to solve the rotation matrix and translation vector of the similarity transformation, which is defined as the mean square error between the visual estimation value after the similarity transformation and the aligned UTM data set as follows:

[0074]

[0075] Will s cg Substituting the above formula and normalizing the translation vector, the optimization objective function is transformed into:

[0076] The two sets of three-dimensional point coordinate transformations are solved to obtain the precise global real geographic coordinates of the key frames.

[0077] Preferably, the step 7 specifically comprises: based on the global real geographic coordinates, using the Laplace pyramid multi-band to achieve real-time jigsaw puzzle of large scenes;

[0078] Step 7-1: Prepare input images; prepare two input images, called source image and target image, which should have the same size; in the first puzzle process, the UTM global coordinate system of the target image is used as the starting point of the puzzle, and the puzzle is performed pixel by pixel according to the required puzzle resolution accuracy;

[0079] Step 7-2: Construct Gaussian pyramid: In this process, the UTM coordinate system of the current image and the target global coordinate system need to be calculated for rough positioning, and then Gaussian pyramids are constructed for the source image and the target image respectively. The Gaussian pyramid is an image pyramid constructed through multiple downsampling and Gaussian blur operations. Each pyramid layer is 1 / 2 the size of the previous layer and is more blurred.

[0080] Step 7-3: Construct Laplacian Pyramid: Construct Laplacian Pyramid from Gaussian Pyramid. Laplacian Pyramid is obtained by the difference between each layer in Gaussian Pyramid and its previous layer.

[0081] Step 7-4: Generate fusion weights: Create a fusion weight image that has the same size as the target image; the central area of ​​the weight image is 1 and gradually decreases to 0 to create a gradient weight;

[0082] Step 7-5: Fusing Laplacian pyramids: Get each layer from the Laplacian pyramid of the source image and the Laplacian pyramid of the target image respectively, and then linearly fuse the two Laplacian pyramids according to the fusion weights. This fusion process is achieved by multiplying each pyramid layer with the corresponding weight image.

[0083] Step 7-6: Reconstruct the image: Reconstruct the final fused image from the fused Laplacian pyramid by adding each fused level to the previous level starting from the bottom layer; this process continuously grows the fused pixel points in combination with the global UTM coordinate system, accurately splices the fused image points, and expands the puzzle area;

[0084] Step 7-7: Fusion result: The final fused image is displayed to the user or saved as a file to generate a panoramic image that can meet the application scenario.

[0085] Compared with the prior art, the advantages of the present invention are:

[0086] (1) The present invention discloses a real-time mosaic method based on monocular SLAM fused with GPS, which uses a drone payload camera to shoot an image sequence directly below and a GPS module to receive longitude and latitude signals, and optimizes the feature extraction and matching algorithm of the monocular ORBSLAM3 to achieve high-precision real-time tracking and map construction of large outdoor scenes, and generate a sparse point cloud map and position posture of key frames; the GPS module signal is converted through UTM coordinates, and is fused with the key frame information through linear search difference; the key frame position posture is converted to the global UTM geographic coordinates through fusion transformation; the Laplace Gaussian pyramid is used to achieve overlapping area fusion of key frames with similar spatial positions to obtain an accurate panoramic view. The present invention can achieve large-scene image mosaic, good overlapping area mosaic effect, and good real-time mosaic, and is suitable for outdoor aerial photography, aerial survey and other application occasions.

[0087] (2) The present invention converts GPS information into the UTM global universal coordinate system, which is convenient for converting the earth's curved surface into a plane, facilitating image splicing and making the puzzle closer to the real scene; the present invention limits the position of key points on the image plane through raster distribution, thereby improving the accuracy of feature point matching;

[0088] (3) The present invention does not require pre-setting of image acquisition control points on the ground. The precise position parameters of the camera can be updated and calculated in real time through the SLAM algorithm. At the same time, combined with the point cloud map, it can provide accurate basic data for real-time puzzle. The puzzle method supports single-channel or multi-channel parallel processing, and integrates high-performance computing such as multi-threading and parallel processing. The cost of application implementation is low, and the puzzle can achieve efficient and fast splicing for real-time processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 , a flow chart of a real-time puzzle method based on monocular SLAM fused with GPS in the present invention;

[0090] Figure 2 , the outdoor background map in Embodiment 7 of the present invention;

[0091] Figure 3, the real-time puzzle result diagram in Example 7 of the present invention;

[0092] Figure 4 , the real-time effect diagram of ORB-SLAM fusion UTM for front-end tracking in Example 7 of the present invention;

[0093] Figure 5 , the global sparse point cloud map in Example 7 of the present invention. DETAILED DESCRIPTION

[0094] The specific implementation mode of the present invention is described below in conjunction with embodiments:

[0095] It should be noted that the contents shown in the drawings of this specification are only used to cooperate with the principles, features and content descriptions disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the conditions under which the present invention can be implemented. Without affecting the effects and purposes that can be achieved by the present invention, they should still fall within the scope of the technical contents disclosed in the present invention.

[0096] SLAM (Simultaneous Localization and Mapping) is a technology commonly used in the fields of unmanned systems, robots and computer vision. It can extract and process information from continuous images, calculate its own position and posture, build maps, and obtain high-precision positioning results. It provides an automated geographic information mapping method for UAV aerial surveys and is the key to the autonomous navigation and control of mobile robots in unknown environments.

[0097] A single visual sensor cannot meet the requirements of high-precision positioning and map construction in complex outdoor environments, so multi-sensor collaboration is a common solution. GPS-assisted visual SLAM

[0098] It is a common multi-sensor collaborative method that combines the location information provided by the global positioning system with the image information captured by the visual sensor to improve the accuracy of positioning and map construction.

[0099] Example 1

[0100] The present application discloses a real-time puzzle method based on monocular SLAM fused with GPS, comprising the following steps:

[0101] Step 1: Obtain images and GPS information, and obtain the images captured by the monocular camera and the GPS information provided by the GPS in real time;

[0102] Step 2: Convert GPS information into UTM information;

[0103] Step 3: Extract ORB features from the image and rasterize it: construct a multi-layer pyramid model for the image in step 1, extract ORB features from each layer of the pyramid layered image, and rasterize the ORB features of each layer of the pyramid layered image;

[0104] Step 4: Initialization of visual odometry;

[0105] Step 5: Real-time tracking of visual odometer: Send the image and rasterized ORB features of step 1 to the ORBSLAM3 system, output the keyframe queue, and generate the keyframe map pose;

[0106] Step 6: Fuse the UTM information and the key frame map pose in step 5 to calculate the precise global real geographic coordinates of the key frame;

[0107] Step 7: Based on the global real geographic coordinates, use the Laplace pyramid multi-band to achieve real-time mosaic of large scenes.

[0108] Example 2

[0109] Preferably, step 1 is specifically as follows: the drone is equipped with a camera to shoot the scene of the flight process, and then the video and GPS information of the GPS module are transmitted back to the control base station in real time through a microwave channel, an image acquisition card is installed on the video receiver to acquire image frames, and then transmitted to the working computer to achieve the acquisition of image frames and GPS information.

[0110] The drone payload mount camera and GPS navigation module are used to obtain image sequence data and RTK location information. GPS navigators or mobile devices usually provide longitude, latitude and altitude values. The longitude and latitude values ​​are of little reference value for the distance information of the actual puzzle. They need to be converted to the UTM global universal coordinate system to facilitate the calculation of the earth's curved surface into a flat surface for image stitching.

[0111] The conversion of GPS information into UTM information in step 2 is specifically as follows:

[0112] Step 2-1: Get GPS information: Get the GPS information to be converted, which includes latitude, longitude, latitude, altitude, and relative height information;

[0113] Step 2-2: Determine the UTM projection area: According to the given longitude information, determine the UTM projection area;

[0114] Step 2-3: Calculate the central meridian: Calculate the central meridian longitude according to the UTM projection area; for area 1, the central meridian longitude is -177 degrees, and for areas 2 to 60, the calculation formula for the central meridian longitude is: CentralMeridian = -177 + (Zone Number-1) * 6;

[0115] Step 2-4: Use the conversion formula of UTM projection to convert the GPS coordinates of the GPS information into UTM coordinates; the conversion formula of UTM projection is as follows:

[0116] N=(a / (1-e 2 *sin(lat) 2 )) 0.5 *k1*(A+(1-T+C)*A 3 / 6+(5-18*T+T 2 +72*C-58*e_prime 2 )*A 5 / 120)+false_northing

[0117] E=k1*(lon-lon0)*cos(lat)+E0

[0118] Where:

[0119] N is the UTM vertical axis coordinate;

[0120] E is the UTM horizontal axis coordinate;

[0121] lat is latitude, degrees;

[0122] lon is longitude, degrees;

[0123] lon0 is the longitude of the central meridian of the UTM projection, degrees;

[0124] a is the semi-major axis of the reference ellipsoid;

[0125] e 2 is the square of the first eccentricity of the reference ellipsoid;

[0126] k1 is the proportional factor;

[0127] A, C, e_prime, and T are intermediate variables calculated based on latitude;

[0128] E0, false_northing is the offset parameter of the UTM projection, which is used to align the coordinate origin to the specified area;

[0129] Step 2-5: Output the converted UTM coordinates, which include transverse Mercator coordinates and longitudinal Mercator coordinates, as well as the UTM projection zone and central meridian longitude.

[0130] Example 3

[0131] When the drone is flying, the moving distance of the ORB feature points between the previous and next frames will not jump over a long distance. Therefore, the position of the key points on the image plane can be restricted through raster distribution to improve the accuracy of feature point matching.

[0132] Furthermore, in order to more accurately estimate the position and posture of the keyframes and calculate the sparse point cloud coordinates, the ORB feature extraction and matching algorithm of the open source code is optimized and upgraded. The detailed steps of ORB feature extraction and matching are as follows:

[0133] Step 3-1: Construct a multi-layer pyramid model for the input image in step 1;

[0134] Step 3-2: Extract fast FAST corner point detection from each layer of pyramid layer image of the multi-layer pyramid model;

[0135] FAST corner detector is a fast corner detection algorithm that determines whether a pixel is a corner by comparing pixel values. In ORB feature extraction, the number of key points extracted can be controlled by adjusting the threshold of FAST detector.

[0136] Step 3-3: For each FAST corner point, extract the 128-dimensional BRIEF descriptor as the ORB feature;

[0137] The feature descriptor is a description of the image area around the key point, which is usually implemented using the BRIEF binary descriptor. The BRIEF descriptor is a fast binary descriptor that can effectively describe the image features around the key point.

[0138] Step 3-4: rasterize the feature descriptor and assign each ORB feature descriptor to the set grid number;

[0139] ORB feature points are generally multi-scale pyramids. The distance change on the scaled pyramid is smaller, so only the feature points on the original image can be matched to reduce the matching complexity and calculation time. Each feature point in the reference frame will have a corresponding grid number after gridding, and the feature points in the grids with the same number in the current frame and a certain range around it are obtained as matching candidate feature points.

[0140] Step 3-5: Feature matching, ORB feature extraction uses the Hamming distance to measure the similarity between two feature descriptors; in ORB-SLAM, feature matching usually adopts the Brute-Force Matching method, that is, for each key point, the Hamming distance between it and all candidate matching points is calculated, and the point with the closest distance is selected as the matching point;

[0141] Step 3-6: Sort the ORB feature similarity within 20 pixels based on the pixel distance between the previous and next two frames, and select the feature point with the greatest similarity as the matching point pair;

[0142] Step 3-7: Perform RANSAC random fitting on the matching point pairs and historical key point clouds of the previous and next two frames to calculate the relative motion pose between the two frames;

[0143] Step 3-8: In the initialization phase, the 3D points obtained by triangulation are used to generate map points. After the initialization is completed, the pose calculated is used to project the key points of the current frame to update the sparse point cloud.

[0144] In ORB-SLAM3, since the position change range of feature points between consecutive frames is relatively limited, the matching speed of key points can be improved by searching and matching in the grid area within a certain spatial range of the ORB feature points in the previous frame. This method can not only reduce the probability of incorrect matching points, but also improve the robustness of multiple modules of ORB-SLAM3, such as system initialization and tracking, and sparse point cloud map construction.

[0145] Example 4

[0146] ORBSLAM3 is a monocular vision SLAM method. The system is divided into three parallel processing threads, namely motion tracking estimation thread, backend optimization thread and tracking estimation thread. We input the rasterized features obtained in the above steps into the ORBSLAM3 system. Through the motion tracking estimation thread, we can generate a sparse point cloud map, calculate the position and posture of each frame image, and generate a new key frame.

[0147] Preferably, step 4 is specifically as follows: initializing ORB-SLAM3: before processing the image, the ORB-SLAM3 system needs to be initialized, including loading camera parameters, configuring algorithm parameters, and preparing data structures for tracking and mapping; using the visual odometer to calculate the moving distance between frame sequences, if it exceeds a pre-calculated threshold, the current frame is set as a key frame for storage; in the initialization stage, the point cloud is updated between multiple key frames using the pose calculated by SLAM to form the three-dimensional point cloud information for tracking initialization.

[0148] Preferably, the step 5 is specifically:

[0149] Step 5-1: Image processing: Send the image and rasterized ORB features from step 1 to the ORBSLAM3 system, and ORB-SLAM3 will perform feature extraction, feature matching, pose estimation, and map update;

[0150] Step 5-2: Key frame extraction: ORB-SLAM3 selects key frames based on motion and perspective change criteria. Key frames are important frames for subsequent map construction and posture estimation.

[0151] Step 5-3: Sparse point cloud generation: ORB-SLAM3 uses feature matching and triangulation techniques between key frames to generate sparse point clouds, which are obtained by converting matching feature points between key frames into points in three-dimensional space;

[0152] Step 5-4: Position and attitude estimation: ORB-SLAM3 estimates the position and attitude of the camera by tracking the motion between key frames and using the optimization-based nonlinear least squares method;

[0153] Step 5-5: Real-time update: As the video is processed, ORB-SLAM3 will continue to update the sparse point cloud and position and pose estimates to generate keyframe map poses.

[0154] Example 5

[0155] The UTM coordinates converted from GPS information are integrated with the sparse map and position posture data (key frame map pose) of the key frames calculated by ORB-SLAM3. A quadtree is constructed through the sparse point cloud map, the visual movement distance between two key frames, and the global geographic movement distance, and the information of M neighboring key frames is queried; a similarity transformation model of the visual map from the visual coordinate system to the real geographic coordinate system is constructed; and the pose information of the current key frame is converted to the UTM global geographic coordinates.

[0156] Preferably, the step 6 is specifically:

[0157] Step 6-1: Collect the UTM information obtained in step 2 and the key frame map pose calculated by ORB-SLAM3; ensure that each data point has corresponding timestamp information. For each UTM information data point, the two closest key frames can be found in the timestamp sequence of the key frames calculated by ORB-SLAM3 through linear search based on its timestamp information; once the two closest key frames are found, use the nearest neighbor linear interpolation method to interpolate between the two key frames based on the timestamp information.

[0158] Step 6-2: Find the two closest key frames in the timestamp sequence of the key frames calculated by ORB-SLAM3 through linear search according to their timestamp information;

[0159] Step 6-3: The real-time video is used to obtain the sparse point cloud and position posture of the key frame sequence through the ORB-SLAM3 algorithm, and the key frame information is filtered;

[0160] Step 6-4: Construct a UTM quadtree using UTM information and keyframe map poses;

[0161] In the UTM quadtree, each longitudinal band is divided into multiple tiles of equal size. These tiles can be square or rectangular. The tiles are connected by a quadtree. Each tile can be further divided into four sub-tiles to form a quadtree structure. Each tile is uniquely identified by its longitudinal band number and the position of the quadtree (upper left, upper right, lower left, lower right).

[0162] UTM quadtree can be used for spatial indexing and querying to improve the storage and retrieval efficiency of geospatial data. By organizing geospatial data into a quadtree structure, data in a specific area can be quickly located and queried, reducing unnecessary data processing and search range. The keyframe information saved in the previous link is inserted into the UTM quadtree, and the M keyframe information of the current keyframe's neighbors is queried.

[0163] Step 6-5: Filter the M neighboring points of the UTM quadtree, establish a similarity transformation model from the visual coordinate system to the real geographic coordinate system of the visual map, and obtain the precise global real geographic coordinates of the key frame.

[0164] Preferably, the sparse point cloud and position posture of the key frame sequence obtained by the ORB-SLAM3 algorithm in step 6-3 are as follows:

[0165] The sparse point cloud data is agreed to be in is the coordinate position of the jth map point of the sparse point cloud in the current local visual coordinate system; The superscript c in the figure indicates the camera visual coordinate system, and lm is the abbreviation of the map point landmark.

[0166] Position and posture in is the rotation matrix and translation vector of the transformation from the camera coordinate system to the current local visual coordinate system of the i-th key frame; then the position coordinates in the local visual coordinate system are in Where T represents the transpose operation of the matrix; the position of the key frame in the global UTM geographic coordinate system after interpolation is

[0167] Preferably, the filtering and screening of the key frame information in step 6-3 is specifically as follows:

[0168] 6-3-1: Calculate UTM global geographic distance: Use the UTM coordinates of the current keyframe and the previous keyframe to calculate the UTM global geographic distance (dis_um) between them;

[0169] 6-3-2: Calculate visual distance: For the sparse point cloud of the current key frame, based on coordinate transformation Calculate the depth information of each point; quickly sort the depth of all points and take the point cloud depth of the median index as the depth (vis_depth) estimate of the current keyframe;

[0170] 6-3-3: Calculate the translation vector distance: Use the key frame translation vector (vis_dis) provided by ORB-SLAM3 to represent the relative translation distance between the current key frame and the previous key frame;

[0171] 6-3-4: Determine threshold condition: Calculate the ratio of vis_dis to vis_depth (vis_dis_ratio = vis_dis / vis_depth) and compare it with a preset threshold; if vis_dis_ratio exceeds the threshold, it means that the local visual distance of the current frame is large, and it is considered to correspond to a distant scene;

[0172] 6-3-5: Save or discard keyframe: If both the UTM global geographic distance (dis_utm) and the local visual distance (vis_dis_ratio) exceed the set threshold, the information of the current keyframe is saved; otherwise, the keyframe is discarded;

[0173] The steps 6-5 are specifically as follows:

[0174] Establish a similarity transformation model from the visual coordinate system to the real geographic coordinate system of the visual map to obtain the scaling, translation and rotation relationship of the visual information: where s cg is the scale factor and satisfies s cg >0, R cg ,t cg The three-dimensional space rotation matrix and translation vector from the local visual coordinate system to the UTM global geographic coordinate system;

[0175] Calculate the ratio of dis_utm to vis_dis between the query neighbor key frame information cg , and then calculate the average value as the initial scale factor for each key frame; the optimization least squares objective function is used to solve the rotation matrix and translation vector of the similarity transformation, which is defined as the mean square error between the visual estimation value after the similarity transformation and the aligned UTM data set as follows:

[0176]

[0177] Will s cg Substituting into the above formula and normalizing the translation vector, the optimization objective function is transformed into:

[0178] The two sets of three-dimensional point coordinate transformations are solved to obtain the precise global real geographic coordinates of the key frames.

[0179] Example 6

[0180] In order to make the overlapping area transition smoothly and improve the image quality, the Laplacian Gaussian pyramid fusion method is used to gradually transition to avoid obvious boundaries. The specific steps of the method include: scaling the source image and the target image to the same size; constructing a multi-layer Gaussian pyramid, a Laplacian pyramid containing high-frequency information of the image and a fused weight pyramid image for the scaled image; each layer of the Laplacian pyramid of the source image and the target image is multiplied by the weight image; starting from the highest resolution layer of the fused Laplacian pyramid, the high-frequency information of the previous layer is fused layer by layer; using the fusion algorithm to obtain a better visual effect and generate a panoramic image that can meet the application occasion.

[0181] Preferably, the step 7 specifically comprises: based on the global real geographic coordinates, using the Laplace pyramid multi-band to achieve real-time jigsaw puzzle of large scenes;

[0182] Step 7-1: Prepare input images; prepare two input images, called source image and target image, which should have the same size; in the first puzzle process, the UTM global coordinate system of the target image is used as the starting point of the puzzle, and the puzzle is performed pixel by pixel according to the required puzzle resolution accuracy;

[0183] Step 7-2: Construct Gaussian pyramid: In this process, the UTM coordinate system of the current image and the target global coordinate system need to be calculated for rough positioning, and then Gaussian pyramids are constructed for the source image and the target image respectively. The Gaussian pyramid is an image pyramid constructed through multiple downsampling and Gaussian blur operations. Each pyramid layer is 1 / 2 the size of the previous layer and is more blurred.

[0184] Step 7-3: Construct Laplacian Pyramid: Construct Laplacian Pyramid from Gaussian Pyramid. Laplacian Pyramid is obtained by the difference between each layer in Gaussian Pyramid and its previous layer.

[0185] Step 7-4: Generate fusion weights: Create a fusion weight image that has the same size as the target image; the central area of ​​the weight image is 1 and gradually decreases to 0 to create a gradient weight;

[0186] Step 7-5: Fusing Laplacian pyramids: Get each layer from the Laplacian pyramid of the source image and the Laplacian pyramid of the target image respectively, and then linearly fuse the two Laplacian pyramids according to the fusion weights. This fusion process is achieved by multiplying each pyramid layer with the corresponding weight image.

[0187] Step 7-6: Reconstruct the image: Reconstruct the final fused image from the fused Laplacian pyramid by adding each fused level to the previous level starting from the bottom layer; this process continuously grows the fused pixel points in combination with the global UTM coordinate system, accurately splices the fused image points, and expands the puzzle area;

[0188] Step 7-7: Fusion result: The final fused image is displayed to the user or saved as a file to generate a panoramic image that can meet the application scenario.

[0189] The key to this process is to merge the detailed information of images at different scales to ensure that the fused image is seamless in the transition area. Laplacian pyramid fusion is often used in image editing and synthesis, such as blending two images together or creating smooth transition effects in panorama stitching. This technology allows high-quality image fusion.

[0190] Example 7

[0191] This embodiment discloses a method for realizing panoramic stitching of large outdoor scenes based on monocular SLAM fusion GPS signal. The camera of the drone load captures the scene image directly below and synchronously receives the RTK real-time signal of the GPS module. The algorithm of the present invention can realize seamless stitching of thousands or even tens of thousands of images of large outdoor scenes in real time. The present invention is a real-time mosaic method based on monocular SLAM fusion GPS, such as Figure 1 As shown, the steps include:

[0192] Step 1: Obtain images and GPS information, and obtain the images captured by the monocular camera and the GPS information provided by the GPS in real time;

[0193] Step 2: Convert GPS signals into UTM information

[0194] GPS receivers or mobile devices usually provide longitude and latitude values. This set of parameters is relative, has little reference value for real-time puzzles, and is not accurate enough for local coordinate representation errors. UTM is a globally used coordinate system that uses rectangular coordinates to convert points on the earth's curved surface into points on a plane, making it easier to calculate distances and azimuths. The UTM coordinate system uses the Transverse Mercator Projection. The relevant calculation formulas and algorithms have been widely studied and optimized, so the efficiency is high in the calculation and conversion process.

[0195] Step 2-1: Obtain GPS coordinate information such as latitude, longitude, altitude, etc. through the GPS module;

[0196] Step 2-2: Determine the UTM projection area based on the given longitude information;

[0197] Step 2-3: Calculate the central meridian longitude according to the UTM projection area;

[0198] For zone 1, the central meridian longitude is -177 degrees. For other zones (2 to 60), the central meridian longitude can be calculated by the following formula: Central Meridian = -177 + (Zone Number - 1) * 6;

[0199] Step 2-4: Convert GPS coordinates to UTM coordinates using the conversion formula of UTM projection;

[0200] Step 2-5: Output the converted UTM coordinates.

[0201] Step 3: Extract ORB features from the image and rasterize it;

[0202] The present invention adopts the open source ORBSLAM3 algorithm for real-time jigsaw puzzles of large scenes, and finds that there are splicing gaps and errors in the spliced ​​images. After algorithm analysis, the ORB feature points have key point matching in the initialization stage and there is a long-distance matching, and the corresponding matching point movement distance between the front and rear frame images of the drone during flight cannot be too far. Therefore, a rasterization processing key point distribution position is designed, which can further optimize the accuracy of key point matching and lay a solid foundation for real-time jigsaw puzzle construction. The main process is as follows:

[0203] Step 3-1: Perform multi-scale Gaussian pyramid on the input image to improve the robustness and speed of feature extraction;

[0204] Step 3-2, extract fast FAST corner point detection for each scale image of the Gaussian pyramid;

[0205] Step 3-3: For each key point, extract and calculate a 128-dimensional ORB feature descriptor;

[0206] Step 3-4, assign each ORB feature descriptor to a set grid number;

[0207] Step 3-5: Feature matching, ORB feature extraction uses Hamming distance to measure the similarity between two feature descriptors;

[0208] Step 3-6: Sort the ORB feature similarity within 20 pixels based on the pixel distance between the previous and next two frames, and select the feature point with the greatest similarity as the matching point pair;

[0209] Step 3-7: Perform RANSAC random fitting on the matching point pairs and historical key point clouds of the previous and next two frames to calculate the relative motion pose between the two frames;

[0210] Step 3-8: In the initialization phase, the 3D points obtained by triangulation are used to generate map points. After the initialization is completed, the pose calculated is used to project the key points of the current frame to update the sparse point cloud.

[0211] Step 3: The image and ORB features are sent to the ORBSLAM3 system, the key frame queue is output, and the sparse point cloud map and camera pose are generated;

[0212] ORBSLAM3 is a monocular vision SLAM method. The system is divided into three parallel processing threads, namely motion tracking estimation thread, backend optimization thread and mapping thread. The motion tracking estimation thread can generate sparse point cloud maps, calculate the position and posture of each frame image, and decide whether to generate a new key frame.

[0213] Step 4: Initialization of visual odometer;

[0214] Step 5: Real-time tracking of visual odometer: Send the image and rasterized ORB features of step 1 to the ORBSLAM3 system, output the keyframe queue, and generate the keyframe map pose;

[0215] Step 6: Fuse the UTM information and the key frame map pose in step 5 to calculate the precise global real geographic coordinates of the key frame;

[0216] Step 6-1: Collect UTM information and sparse map and position pose of key frames calculated by ORB-SLAM3 Step 6-1: Collect UTM information obtained in step 2 and key frame map pose calculated by ORB-SLAM3;

[0217] Step 6-2: Find the two closest key frames in the timestamp sequence of the key frames calculated by ORB-SLAM3 through linear search according to their timestamp information;

[0218] Step 6-3: The real-time video is used to obtain the sparse point cloud and position posture of the key frame sequence through the ORB-SLAM3 algorithm, and the key frame information is filtered;

[0219] Step 6-4: Construct a UTM quadtree using UTM information and keyframe map poses;

[0220] Step 6-5: Filter M neighboring points of the UTM quadtree, establish a similarity transformation model from the visual coordinate system to the real geographic coordinate system of the visual map, and obtain the precise global real geographic coordinates of the key frame;

[0221] Step 7: Based on the global real geographic coordinates, use the Laplace pyramid multi-band to achieve real-time mosaic of large scenes

[0222] Step 7-1: Prepare input images; prepare two input images, called source image and target image, which should have the same size; in the first puzzle process, the UTM global coordinate system of the target image is used as the starting point of the puzzle, and the puzzle is performed pixel by pixel according to the required puzzle resolution accuracy;

[0223] Step 7-2: Construct Gaussian pyramid: In this process, the UTM coordinate system of the current image and the target global coordinate system need to be calculated for rough positioning, and then Gaussian pyramids are constructed for the source image and the target image respectively. The Gaussian pyramid is an image pyramid constructed through multiple downsampling and Gaussian blur operations. Each pyramid layer is 1 / 2 the size of the previous layer and is more blurred.

[0224] Step 7-3: Construct Laplacian Pyramid: Construct Laplacian Pyramid from Gaussian Pyramid. Laplacian Pyramid is obtained by the difference between each layer in Gaussian Pyramid and its previous layer.

[0225] Step 7-4: Generate fusion weights: Create a fusion weight image that has the same size as the target image; the central area of ​​the weight image is 1 and gradually decreases to 0 to create a gradient weight;

[0226] Step 7-5: Fusing Laplacian pyramids: Get each layer from the Laplacian pyramid of the source image and the Laplacian pyramid of the target image respectively, and then linearly fuse the two Laplacian pyramids according to the fusion weights. This fusion process is achieved by multiplying each pyramid layer with the corresponding weight image.

[0227] Step 7-6: Reconstruct the image: Reconstruct the final fused image from the fused Laplacian pyramid by adding each fused level to the previous level starting from the bottom layer; this process continuously grows the fused pixel points in combination with the global UTM coordinate system, accurately splices the fused image points, and expands the puzzle area;

[0228] Step 7-7: Fusion result: The final fused image is displayed to the user or saved as a file to generate a panoramic image that can meet the application scenario.

[0229] The present invention uses the ORB-SLAM3 algorithm and integrates the RTK signal of the GPS module (GPS information is converted into UTM information) to achieve real-time jigsaw puzzles of large-scale complex outdoor scenes. It has the advantages of high positioning accuracy, good algorithm robustness, and jigsaw puzzle effects without overlapping gaps or loopholes.

[0230] The method of the present invention has been put into practical use. Figure 2 This is the real topography of the background of the data of one square kilometer measured by the drone in Chunhua. The rectangular area directly measured by the analysis tool of Westworld is marked in the figure. The single-channel splicing area is more than 1000m*1000m; Figure 3 The real-time jigsaw puzzle effect diagram of an example of the present invention is shown. The splicing mask effect after using the real-time jigsaw puzzle method of the present invention shows that the splicing of the highway, the factory building in the lower left corner, the white road, and the green wheat field in the layer is relatively complete; Figure 4 This is the real-time effect diagram of the entire system front-end tracking through ORB-SLAM fusion UTM, including the position and posture of the key frames generated during the stitching process (green in the drawing); the white large area point cloud is the historical point cloud during the stitching process, which is convenient for global repositioning; the red point cloud is the global key frame point cloud within the visible range of the current key frame; Figure 5 It is a sparse point cloud map of complex scenes such as Chunhua roads, valleys, buildings, farmlands, etc. It is a global sparse point cloud map generated after the stitching process is completed, which can be used to correct the puzzle effect.

[0231] The invention discloses a real-time mosaic method based on monocular SLAM fused with GPS. The method uses a drone payload camera to shoot an image sequence directly below and a GPS module to receive longitude and latitude signals. By optimizing the feature extraction and matching algorithm of the monocular ORBSLAM3, high-precision real-time tracking and map construction of large outdoor scenes are achieved, and a sparse point cloud map and position posture of key frames are generated; GPS module signals are converted through UTM coordinates, and fusion with key frame information is achieved through linear search difference; the key frame position posture is converted to global UTM geographic coordinates through fusion transformation; Laplace Gaussian pyramid is used to achieve overlapping area fusion of key frames with similar spatial positions, and an accurate panoramic image is obtained. The invention can achieve large-scene image mosaic, good overlapping area mosaic effect, and good real-time mosaic, and is suitable for outdoor aerial photography, aerial survey and other application occasions.

[0232] The present invention converts GPS information into the UTM global universal coordinate system, which is convenient for converting the earth's curved surface into a plane, facilitating image splicing, and making the puzzle closer to the real scene; the present invention limits the position of key points in the image plane through raster distribution, thereby improving the accuracy of feature point matching.

[0233] The present invention does not require pre-setting of image acquisition control points on the ground. The precise posture parameters of the camera can be updated and calculated in real time through the SLAM algorithm. At the same time, combined with the point cloud map, it can provide accurate basic data for real-time jigsaw puzzles. The jigsaw puzzle method supports single-channel or multi-channel parallel processing, and integrates high-performance computing such as multi-threading and parallel processing. The cost of application implementation is low, and the jigsaw puzzle can achieve efficient and fast splicing of real-time processing.

[0234] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary geologists in this field without departing from the purpose of the present invention.

[0235] Many other changes and modifications may be made without departing from the concept and scope of the present invention.It should be understood that the present invention is not limited to the specific embodiments, and the scope of the present invention is defined by the appended claims.

Claims

1. A real-time puzzle method based on monocular SLAM fused with GPS, characterized in that: The following steps are involved: Step 1: Obtain images and GPS information, and obtain the images captured by the monocular camera and the GPS information provided by the GPS in real time; Step 2: Convert GPS information into UTM information; Step 3: Extract ORB features from the image and rasterize it: construct a multi-layer pyramid model for the image in step 1, extract ORB features from each layer of the pyramid layered image, and rasterize the ORB features of each layer of the pyramid layered image; Step 4: Initialization of visual odometer; Step 5: Real-time tracking of visual odometer: Send the image and rasterized ORB features of step 1 to the ORBSLAM3 system, output the keyframe queue, and generate the keyframe map pose; Step 6: Fuse the UTM information and the key frame map pose in step 5 to calculate the precise global real geographic coordinates of the key frame; Step 7: Based on the global real geographic coordinates, use the Laplace pyramid multi-band to achieve real-time mosaic of large scenes.

2. A real-time puzzle method based on monocular SLAM fusion GPS according to claim 1, characterized in that: The step 1 is specifically as follows: the drone is equipped with a camera to shoot the scene of the flight process, and then the video and GPS information of the GPS module are transmitted back to the control base station in real time through the microwave channel. An image acquisition card is installed on the video receiver to acquire the image frame, and then transmit it to the working computer to obtain the image frame and GPS information.

3. A real-time puzzle method based on monocular SLAM fusion GPS according to claim 1, characterized in that: The step 2 is specifically as follows: Step 2-1: Get GPS information: Get the GPS information to be converted, which includes latitude, longitude, latitude, altitude, and relative height information; Step 2-2: Determine the UTM projection area: Determine the UTM projection area according to the given longitude information; Step 2-3: Calculate the central meridian: Calculate the central meridian longitude according to the UTM projection area; for zone 1, the central meridian longitude is -177 degrees, and for zone 2N60, the calculation formula of the central meridian longitude is: Central Meridian = -177 + (ZoneNumber-1) * 6; Step 2-4: Use the conversion formula of UTM projection to convert the GPS coordinates of GPS information into UTM coordinates; The conversion formula for the UTM projection is as follows: N=(a / (1-e 2 *sin(lat) 2 )) 0.5 *k1*(A+(1-T+C)*A 3 / 6+(5-18*T+T 2 +72*C-58*e_prime 2 )*A 5 / 120)+false_norrthingE=k1*(lon-lon)*cos(lat)+E0 Where: N is the UTM vertical axis coordinate; E is the UTM horizontal axis coordinate; lat is latitude, degrees; lon is longitude, degrees; lon0 is the longitude of the central meridian of the UTM projection, degrees; a is the semi-major axis of the reference ellipsoid; e 2 is the square of the first eccentricity of the reference ellipsoid; kl is the scale factor; A, C, e_prime, and T are intermediate variables calculated based on latitude; E0, false_northing is the offset parameter of the UTM projection, which is used to align the coordinate origin to the specified area; Step 2-5: Output the converted UTM coordinates, which include transverse Mercator coordinates and longitudinal Mercator coordinates, as well as the UTM projection zone and central meridian longitude.

4. A real-time puzzle method based on monocular SLAM fusion GPS according to claim 1, characterized in that: The step 3 is specifically as follows: Step 3-1: Construct a multi-layer pyramid model for the input image in step 1; Step 3-2: Extract fast FAST corner point detection for each layer of pyramid layered image of the multi-layer pyramid model; Step 3-3: For each FAST corner point, extract the 128-dimensional BRIEF descriptor as the ORB feature; Step 3-4: rasterize feature descriptors, assign each ORB feature descriptor to a set grid number; Step 3-5: feature matching, ORB feature extraction uses Hamming distance to measure the similarity between two feature descriptors; Step 3-6: Sort the ORB feature similarity within 20 pixels based on the pixel distance between the previous and next two frames, and select the feature point with the greatest similarity as the matching point pair; Step 3-7: Perform RANSAC random fitting on the matching point pairs and historical key point clouds of the previous and next two frames to calculate the relative motion pose between the two frames; Step 3-8: In the initialization phase, the 3D points obtained by triangulation are used to generate map points. After the initialization is completed, the pose calculated is used to project the key points of the current frame to update the sparse point cloud.

5. A real-time puzzle method based on monocular SLAM fusion GPS according to claim 1, characterized in that: The step 4 is specifically as follows: Initialize ORB-SLAM3: Before processing the image, the ORB-SLAM3 system needs to be initialized, including loading camera parameters, configuring algorithm parameters, and preparing data structures for tracking and mapping; using the visual odometer to calculate the moving distance between frame sequences. If it exceeds a pre-calculated threshold, the current frame is set as a key frame for storage; in the initialization stage, the point cloud is updated between multiple key frames using the pose calculated by SLAM to form the three-dimensional point cloud information for tracking initialization.

6. A real-time puzzle method based on monocular SLAM fusion GPS according to claim 1, characterized in that: The step 5 is specifically as follows: Step 5-1: Image processing: Send the image and rasterized ORB features from step 1 to the ORBSLAM3 system, and ORB-SLAM3 will perform feature extraction, feature matching, pose estimation, and map update; Step 5-2: Key frame extraction: ORB-SLAM3 selects key frames based on motion and perspective change criteria. Key frames are important frames for subsequent map construction and posture estimation. Step 5-3: Sparse point cloud generation: ORB-SLAM3 uses feature matching and triangulation techniques between key frames to generate sparse point clouds, which are obtained by converting matching feature points between key frames into points in three-dimensional space; Step 5-4: Position and attitude estimation: ORB-SLAM3 estimates the position and attitude of the camera by tracking the motion between key frames and using the optimization-based nonlinear least squares method; Step 5-5: Real-time update: As the video is processed, ORB-SLAM3 will continue to update the sparse point cloud and position and pose estimates to generate keyframe map poses.

7. A real-time puzzle method based on monocular SLAM fusion GPS according to claim 6, characterized in that: The step 6 is specifically as follows: Step 6-1: Collect the UTM information obtained in step 2 and the key frame map pose calculated by ORB-SLAM3; Step 6-2: Find the two closest key frames in the timestamp sequence of the key frames calculated by ORB-SLAM3 through linear search according to their timestamp information; Step 6-3: The real-time video is used to obtain the sparse point cloud and position posture of the key frame sequence through the ORB-SLAM3 algorithm, and the key frame information is filtered; Step 6-4: Construct a UTM quadtree using UTM information and keyframe map poses; Step 6-5: Filter the M neighboring points of the UTM quadtree, establish a similarity transformation model from the visual coordinate system to the real geographic coordinate system of the visual map, and obtain the precise global real geographic coordinates of the key frame.

8. A real-time puzzle method based on monocular SLAM fusion GPS according to claim 1, characterized in that: The sparse point cloud and position posture of the key frame sequence obtained by the ORB-SLAM3 algorithm in step 6-3 are as follows: The sparse point cloud data is agreed to be in is the coordinate position of the jth map point of the sparse point cloud in the current local visual coordinate system; The superscript c in the figure indicates the camera visual coordinate system, and 1m is the abbreviation of the map point 1andmark. Position and posture in is the rotation matrix and translation vector of the transformation from the camera coordinate system to the current local visual coordinate system of the i-th key frame; then the position coordinates in the local visual coordinate system are in Where T represents the transpose operation of the matrix; the position of the key frame in the global UTM geographic coordinate system after interpolation is 9. A real-time puzzle method based on monocular SLAM fusion GPS according to claim 8, characterized in that: The filtering and screening of the key frame information in step 6-3 is specifically as follows: 6-3-1: Calculate UTM global geographic distance: Use the UTM coordinates of the current keyframe and the previous keyframe to calculate the UTM global geographic distance (dis_utm) between them; 6-3-2: Calculate visual distance: For the sparse point cloud of the current key frame, based on coordinate transformation Calculate the depth information of each point; quickly sort the depth of all points and take the point cloud depth of the median index as the depth (vis_depth) estimate of the current keyframe; 6-3-3: Calculate the translation vector distance: Use the key frame translation vector (vis_dis) provided by ORB-SLAM3 to represent the relative translation distance between the current key frame and the previous key frame; 6-3-4: Determine threshold condition: Calculate the ratio of vis_dis to vis_depth (vis_dis_ratio = vis_dis / vis_depth) and compare it with a preset threshold; if vis_dis_ratio exceeds the threshold, it means that the local visual distance of the current frame is large, and it is considered to correspond to a distant scene; 6-3-5: Save or discard keyframe: If both the UTM global geographic distance (dis_utm) and the local visual distance (vis_dis_ratio) exceed the set threshold, the information of the current keyframe is saved; otherwise, the keyframe is discarded; The steps 6-5 are specifically as follows: Establish a similarity transformation model from the visual coordinate system to the real geographic coordinate system of the visual map to obtain the scaling, translation and rotation relationship of the visual information: where s cg is the scale factor and satisfies s cg >0, R cg ,t cg The three-dimensional space rotation matrix and translation vector from the local visual coordinate system to the UTM global geographic coordinate system; Calculate the ratio of dis_utm to vis_dis between the query neighbor key frame information cg , and then calculate the average value as the initial scale factor for each key frame; the rotation matrix and translation vector of the similarity transformation are solved by optimizing the least squares objective function, which is defined as the mean square error between the visual estimation value after the similarity transformation and the aligned UTM data as follows: Will s cg Substituting the above formula and normalizing the translation vector, the optimization objective function is transformed into: The two sets of three-dimensional point coordinate transformations are solved to obtain the precise global real geographic coordinates of the key frames.

10. The real-time puzzle method based on monocular SLAM and GPS fusion according to claim 1, characterized in that: The step 7 specifically includes: based on the global real geographic coordinates, using the Laplace pyramid multi-band to achieve real-time jigsaw puzzle of large scenes; Step 7-1: Prepare input images; prepare two input images, called source image and target image, which should have the same size; During the first puzzle process, the UTM global coordinate system of the default target image is used as the starting point of the puzzle, and the puzzle is performed point by point at the pixel level according to the required puzzle resolution accuracy; Step 7-2: Construct Gaussian pyramid: In this process, the UTM coordinate system of the current image and the target global coordinate system need to be calculated for rough positioning, and then Gaussian pyramids are constructed for the source image and the target image respectively. The Gaussian pyramid is an image pyramid constructed through multiple downsampling and Gaussian blur operations. Each pyramid layer is 1 / 2 the size of the previous layer and is more blurred. Step 7-3: Construct Laplacian Pyramid: Construct Laplacian Pyramid from Gaussian Pyramid. Laplacian Pyramid is obtained by the difference between each layer in Gaussian Pyramid and its previous layer. Step 7-4: Generate fusion weights: Create a fusion weight image that has the same size as the target image; the central area of ​​the weight image is 1 and gradually decreases to 0 to create a gradient weight; Step 7-5: Fusing Laplacian pyramids: Get each layer from the Laplacian pyramid of the source image and the Laplacian pyramid of the target image respectively, and then linearly fuse the two Laplacian pyramids according to the fusion weights. This fusion process is achieved by multiplying each pyramid layer with the corresponding weight image. Step 7-6: Reconstructing the image: Reconstruct the final fused image from the fused Laplacian pyramid, starting from the bottom layer, and adding each fused layer to the previous layer. This process continuously grows the fused pixel points in combination with the global UTM coordinate system, accurately splices the fused image points, and expands the puzzle area. Step 7-7: Fusion result: The final fused image is displayed to the user or saved as a file to generate a panoramic image that can meet the application scenario.

Images