Geographic mosaicking method and apparatus for video images, and computer device and storage medium

Abstract

The present application relates to a geographic mosaicking method and apparatus for video images, and a computer device and a storage medium. The method comprises: acquiring real-time live streaming images of a real-world scene that are collected by a plurality of unmanned aerial vehicles, and GNSS information of the plurality of unmanned aerial vehicles; using a visual SLAM algorithm to perform input frame tracking on the real-time live streaming images, and performing pose estimation on successfully tracked input frames by combining visual trajectories and the GNSS information, so as to generate georeferenced camera poses; using the camera poses and a surface reconstruction algorithm to perform densification processing on the input frames, so as to generate a depth map of the real-world scene, and mapping the depth map into dense three-dimensional point clouds, so as to construct a three-dimensional surface model of the real-world scene; and using the camera pose and the three-dimensional surface model to perform orthorectification on the input frames, and integrating the orthorectified input frames into a global mosaicked map, so as to generate a global image of the real-world scene. By means of the embodiments of the present application, a real-time picture of a real-world scene can be quickly acquired, thereby providing robust support for a rapid emergency response in a real-world scene.

Description

Video image geographic stitching method, apparatus, computer equipment and storage medium Technical Field

[0001] This application belongs to the field of deep learning technology, and specifically relates to a video image geographic stitching method, apparatus, computer equipment, and storage medium. Background Technology

[0002] In today's society, the rapid pace of urbanization has intensified population concentration, making safety issues increasingly prominent. Against this backdrop, frequent emergencies have created a strong demand for rapidly and accurately capturing panoramic views of incidents and providing real-time updates of critical information. Traditional fixed-view cameras, due to limitations such as fixed angles and limited coverage, are insufficient to meet the needs of on-site personnel for real-time understanding of scene developments. Therefore, how to quickly acquire a comprehensive image of a scene has become an urgent problem to be solved.

[0003] With the rapid development and widespread application of drone technology, innovative solutions have been provided for the problem of global scene imaging in emergency scenarios. Compared with traditional fixed-view cameras, drones, with their flexibility and versatility, can assist on-site personnel in grasping the dynamics of the scene more promptly and comprehensively in emergency situations. However, drone footage is often scattered, only covering local information, which is not conducive to personnel's comprehensive understanding of the overall scene. With the continuous advancement of visual SLAM technology, the combination of drone aerial images and visual SLAM technology can achieve efficient integration of discrete image data, making real-time stitching of drone aerial images a reality, thus providing on-site personnel with a more intuitive scene view. However, current drone image stitching mainly relies on image fusion technology, involving complex technical steps such as feature extraction and matching, which is often time-consuming and inefficient, making it difficult to process data in real time during drone flight, thus restricting the timeliness of emergency search and rescue operations. In particular, the entire processing flow from mission initiation to the final generation of a panoramic map can take several hours, which undoubtedly greatly increases the time cost of mission execution and makes it difficult to meet the application requirements of rapid response emergency missions.

[0004] Summary of the Invention

[0005] This application provides a video image geographic stitching method, apparatus, computer equipment, and storage medium, which aims to at least partially solve one of the aforementioned technical problems in the prior art.

[0006] To address the above problems, this application provides the following technical solution:

[0007] A video image geographic stitching method, comprising:

[0008] Acquire real-time live images of real-world scenes collected by multiple drones, as well as GNSS information from multiple drones;

[0009] The visual SLAM algorithm is used to track the input frames of the live image, and the pose of the successfully tracked input frames is estimated by combining visual trajectory and GNSS information to generate a camera pose with geographic reference.

[0010] The camera pose and surface reconstruction algorithm is used to densify the successfully tracked input frame, reconstruct the dense 3D point cloud of the input frame, generate a depth map of the real scene, and project the depth map onto the dense 3D point cloud through a digital surface model to construct a 3D surface model of the real scene.

[0011] The input frame is orthorectified using the camera pose and 3D surface model, and the orthorectified input frame is integrated into the global mosaic map to generate a global image of the real scene.

[0012] The technical solution adopted in this application embodiment also includes: acquiring real-time live images of the real scene collected by multiple drones and GNSS information of multiple drones, specifically:

[0013] The node uses real-time video communication technology to receive live images collected by multiple UAVs and processes the live images according to a set frame rate. At the same time, another node uses real-time communication to obtain GNSS information of multiple UAVs. The GNSS information includes the current location latitude and longitude, heading angle, pitch angle, roll angle and altitude.

[0014] The technical solution adopted in this application embodiment further includes: after acquiring the real-time live images of the real scene collected by multiple drones and the GNSS information of the multiple drones, it further includes:

[0015] The real-time live image is enhanced using a histogram equalization algorithm, and the GNSS information is transformed using projection transformation to obtain geographic coordinates.

[0016] The technical solution adopted in this application embodiment further includes: using a visual SLAM algorithm to track the input frames of the real-time live image, and combining visual trajectory and GNSS information to estimate the pose of the successfully tracked input frames to generate a camera pose with geographic reference, specifically:

[0017] A visual SLAM interface is created, through which a live-streaming image is input. Input frame tracking is performed on the input image. If tracking fails, the state is switched to "lost," and no visual pose is set. Successfully tracked input frames are divided into frame insertion and keyframe insertion. It is determined whether the current input frame contains important new information. Successfully tracked input frames are sent to a geographic reference. The geographic reference combines visual trajectory and GNSS information to identify and determine the transformation relationship from the local visual coordinate system to the global geographic coordinate system. The camera pose matrix M of the input frame is calculated using a visual SLAM algorithm, and a sparse point cloud is constructed, where M is a 4x4 transformation matrix from the camera coordinate system to the world coordinate system.

[0018] Where, r ij These are the components of the (3x3) rotation matrix R. These are the UTM coordinates of the global camera position. It refers to the relative height of the camera.

[0019] The technical solution adopted in this application embodiment further includes: using the camera pose and surface reconstruction algorithm to densify the successfully tracked input frame, reconstructing the dense 3D point cloud of the input frame, and generating a depth map of the real scene, specifically:

[0020] The applicability of the input frame is evaluated. For the camera pose obtained through visual trajectory, the dense 3D point cloud of the input frame is reconstructed through a dense interface or sparse cloud interpolation. Then, the sparse point cloud generated by the visual SLAM algorithm is reprojected onto the image plane, and depth values ​​are written at the pixel positions according to the interpolation method to generate a relatively sparse depth map. The depth map is then repaired to generate a complete and continuous depth map.

[0021] The technical solution adopted in this application embodiment further includes: the step of projecting the depth map onto a dense 3D point cloud using a digital surface model to construct a 3D surface model of the real scene: specifically:

[0022] The input frame is evaluated based on the pose estimation results and the densification processing results, and the input frame is divided into the following three types:

[0023] Category 1: GNSS information frames with failed pose estimation, fixed attitude, and no point cloud frames;

[0024] The second category includes camera pose frames, precise pose frames, and sparse point cloud frames when densification processing fails.

[0025] The third category consists of camera pose frames, accurate pose frames, and dense point cloud frames where both pose estimation and densification processing were successful.

[0026] A digital surface model is extracted based on the evaluation results. The digital surface model is constructed in the form of a 2.5D elevation map to represent the three-dimensional surface model of the real scene. First, the surface assumptions are determined. The first and second types of input frames with GNSS information but lacking or having only sparse surface information are classified as "planar" frames, and the third type of input frames with dense point clouds are classified as "convex" frames. Then, the depth map of the corresponding input frame is projected onto the dense three-dimensional point cloud based on the classification results using the digital surface model to construct the three-dimensional surface model of the real scene.

[0027] The technical solution adopted in this application embodiment further includes: performing orthorectification on the input frame using the camera pose and three-dimensional surface model, and integrating the orthorectified input frame into the global mosaic map to generate a global image of the real scene, specifically:

[0028] The input frame is orthorectified using a grid-based orthogonal projection method to generate a high-resolution orthorectified image;

[0029] The orthorectified orthophoto image is integrated into the global mosaic map, and a probabilistic method is used to mix spatial information and minimize variance for global image mosaicking. The global image mosaicking process includes: receiving the first frame of orthophoto image for initialization, using subsequently received orthophoto images as map update frames, and dividing the map update frames into regions without prior information and overlapping regions based on their association with the first frame of orthophoto image. The probabilistic method introduces three new data layers into the global mosaic map: elevation variance, elevation assumption, and observation count. When a new orthophoto image is received and it is in a non-overlapping region, a temporary floating average of the new elevation is calculated for each grid cell.

[0030] in n represents the current average elevation in the global mosaic map. jj For the number of observations, Update the elevation of the map; simultaneously estimate the floating sample variance. Defined as:

[0031] if If the variance is below the set variance threshold, the new floating sample variance and temporary floating mean are written to the grid map layer, and the observation count of the current grid cell is increased; if If the variance exceeds the set threshold, the new floating sample variance will be written into the elevation hypothesis layer, while the existing data will be retained.

[0032] Another technical solution adopted in this application embodiment is: a video image geographic stitching device, comprising:

[0033] Data acquisition module: used to acquire real-time live images of real-world scenes collected by multiple drones, as well as GNSS information from multiple drones;

[0034] Pose estimation module: used to track the input frames of the real-time live image using the visual SLAM algorithm, and to estimate the pose of the successfully tracked input frames by combining visual trajectory and GNSS information, and to generate camera pose with geographic reference.

[0035] Surface generation module: used to densify the successfully tracked input frame using the camera pose and surface reconstruction algorithm, reconstruct the dense 3D point cloud of the input frame, generate a depth map of the real scene, and project the depth map onto the dense 3D point cloud through a digital surface model to construct a 3D surface model of the real scene.

[0036] Image stitching module: Used to perform orthorectification on the input frame using the camera pose and 3D surface model, and integrate the orthorectified input frame into the global stitching ground map to generate a global image of the real scene.

[0037] Another technical solution adopted in this application embodiment is: a computer device, the computer device including a processor and a memory coupled to the processor, wherein,

[0038] The memory stores program instructions for implementing the video image geographic stitching method;

[0039] The processor is used to execute the program instructions stored in the memory to control the video image geographic stitching method.

[0040] Another technical solution adopted in this application embodiment is: a storage medium storing program instructions that can be run by a processor, the program instructions being used to execute the video image geographic stitching method.

[0041] Compared to existing technologies, the beneficial effects of the embodiments of this application are as follows: The video image geographic stitching method, apparatus, computer equipment, and storage medium of the embodiments of this application introduce real-time communication technology and UAV multi-video strategy to acquire real-time live images and GNSS information of real-world scenes in real time. The real-time live images can provide intuitive and realistic on-site conditions, and the GNSS information can ensure the accuracy and completeness of global image stitching. It also fully leverages the advantages of visual SLAM, and achieves efficient global image stitching of real-world scenes through multi-UAV collaborative efforts by using techniques such as attitude estimation, dense mapping, surface generation, orthorectification, and image stitching. This significantly improves the real-time performance of data acquisition, enabling rapid acquisition of real-time images in real-world scenes. It ensures that on-site personnel can quickly grasp global information, providing an unprecedented perspective for on-site personnel to comprehensively and deeply understand emergency scenarios, and providing strong support for rapid emergency response in real-world scenarios, thereby effectively protecting the lives and property of disaster victims. Attached Figure Description

[0042] Figure 1 is a flowchart of a video image geographic stitching method according to an embodiment of this application;

[0043] Figure 2 is a schematic diagram of the structure of the video image geographic stitching device according to an embodiment of this application;

[0044] Figure 3 is a schematic diagram of the computer device structure according to an embodiment of this application;

[0045] Figure 4 is a schematic diagram of the structure of the storage medium according to an embodiment of this application. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0047] The terms "first," "second," and "third" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indications also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or computer device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or computer devices.

[0048] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0049] Specifically, please refer to Figure 1, which is a flowchart of a video image geographic stitching method according to an embodiment of this application. The video image geographic stitching method according to an embodiment of this application includes the following steps:

[0050] S100: Employs real-time communication technology to acquire real-time live images of real-world scenes collected by multiple UAVs, as well as GNSS (Global Navigation Satellite System) information from multiple UAVs;

[0051] In this step, to improve the efficiency of acquiring global images of real-world scenarios in emergency situations, this embodiment of the application introduces real-time communication technology and a multi-video strategy for UAVs to acquire real-time live images and GNSS information of the real-world scene. The live images provide an intuitive and realistic view of the situation, while the GNSS information ensures the accuracy and completeness of the global image stitching. Specifically, firstly, using real-time video communication technologies such as WebRTC, a node control receives live images collected by multiple UAVs and processes them according to a set frame rate to ensure smooth reception. Simultaneously, another node uses real-time communication methods such as WebSocket to acquire GNSS information such as the current latitude and longitude, heading angle, pitch angle, roll angle, and altitude of the multiple UAVs, providing a basis for subsequent pose estimation. Furthermore, when implementing the multi-video strategy, an independent node is configured for each UAV to ensure efficient reception of live images and GNSS information from multiple UAVs.

[0052] S110: The histogram equalization algorithm is used to enhance the real-time live images, and the projection transformation is used to transform the geographic coordinates of the GNSS information.

[0053] In this step, the original live stream image may appear unclear due to a narrow grayscale distribution. For example, overexposed live stream images may have grayscale concentrated in bright areas, while underexposed live stream images may have grayscale concentrated in darker areas. To achieve the best global image stitching effect, this embodiment uses a histogram equalization algorithm to adjust the histogram of the live stream image, transforming the histogram of the original live stream image into a uniform distribution. This changes the grayscale levels of pixels, expands the dynamic range between pixel grayscale values, enhances the overall contrast of the live stream image, optimizes the visual effect of the scene, and provides more accurate and reliable image material for subsequent global image stitching, thus improving the quality and accuracy of global image stitching. For GNSS information from multiple UAVs, this embodiment uses projection transformation to perform geographic coordinate transformation to improve the stitching quality and accuracy of the global image, meeting the needs of subsequent attitude estimation.

[0054] S120: The visual SLAM algorithm is used to track the input frames of the real-time live image after image enhancement, and the pose of the successfully tracked input frames is estimated by combining the visual trajectory and the GNSS information after geographic coordinate transformation, so as to generate the camera pose with geographic reference.

[0055] In this step, during the pose estimation stage, a visual SLAM interface is first created. Real-time live images are input through this interface, and input frame tracking is performed on the input live images. If the visual SLAM framework fails to track due to featureless surfaces such as water or flat terrain, the state is switched to "missing," and no visual pose is set. Successfully tracked input frames are divided into frame insertion and keyframe insertion to determine if the current input frame contains important new information. Successfully tracked input frames are then sent to a geographic reference. The geographic reference combines visual trajectory and GNSS information to identify and determine the transformation relationship from the local visual coordinate system to the global geographic coordinate system. The camera pose matrix M of the current input frame is calculated using the visual SLAM algorithm, and a sparse point cloud is constructed to initially represent the scene structure, where M is a 4x4 transformation matrix from the camera coordinate system to the world coordinate system. The finally calculated camera pose is described as:

[0056] Where, r ij These are the components of the (3x3) rotation matrix R. These are the UTM (Universal Transverse Mercator coordinates, a Cartesian coordinate system) coordinates of the global camera position. It refers to the relative height of the camera.

[0057] To obtain robust camera pose calculation results, a reliable set of initial measurements is required. Therefore, the incoming live images need to be paused and queued until the estimation error is reduced to below a set threshold. After the scale problem is resolved and the visual trajectory is aligned with the GNSS information, pose estimation is then performed on all previously paused frames and newly tracked frames.

[0058] S130: The successfully tracked input frame is passed to the densification module. The densification module uses the camera pose and surface reconstruction algorithm of the input frame to reconstruct the dense 3D point cloud of the real scene and generate the depth map of the real scene.

[0059] In this step, the camera pose of the current input frame is calculated in the geographic coordinate system using visual trajectories or based on GNSS information. Visual trajectories can achieve high accuracy, but lack robustness in featureless regions. The camera pose calculated using GNSS information is fixed, but has high uncertainty. Therefore, this application implements a denser point cloud in the denser module using an input frame with visually estimated pose to generate a dense 3D point cloud of the real scene, thereby describing the surface details of the real scene in a more detailed manner.

[0060] Specifically, the densification module's process of reconstructing the dense 3D point cloud of the current input frame using visual pose and an integrated surface reconstruction algorithm includes: First, assessing the applicability of the current input frame. For camera poses obtained through visual trajectories, the dense 3D point cloud of the current input frame can be reconstructed via a densification interface or sparse cloud interpolation. Specifically, a set of frames is passed through the densification interface, an advanced reconstruction framework is integrated, and surface reconstruction is performed using multi-view geometry and other frames. Sparse cloud interpolation is only applicable to a single frame. This method only interpolates existing data and cannot generate new information; its accuracy and resolution depend on the quality of the sparse point cloud generated by the visual SLAM algorithm. Then, the sparse point cloud generated by the visual SLAM algorithm is reprojected onto the image plane, and depth values ​​are written at pixel locations according to the interpolation method to generate a relatively sparse depth map. The sparse depth map is then repaired by filling in the blank areas between sampling points to generate a more complete and continuous depth map.

[0061] S140: By projecting a depth map onto a dense 3D point cloud using a digital surface model, a 3D surface model of the real scene is constructed.

[0062] In this step, during the 3D surface model generation stage, the current input frame needs to be evaluated based on the pose estimation results and the densification processing results, and the input frame is divided into the following three types:

[0063] Category 1: GNSS information frames with failed pose estimation, fixed attitude, and no point cloud frames;

[0064] The second category includes camera pose frames, precise pose frames, and sparse point cloud frames when densification processing fails.

[0065] The third category consists of camera pose frames, accurate pose frames, and dense point cloud frames where both pose estimation and densification processing were successful.

[0066] After the evaluation is completed, a digital surface model is extracted based on the evaluation results. This model uses a 2.5D elevation map to accurately depict the observed real-world scene, thus constructing a three-dimensional surface model of the real-world scene. The specific method for constructing the digital surface model is as follows: First, surface assumptions are determined. Input frames of the first and second categories, which have GNSS information but lack or only have sparse surface information, are classified as "planar" frames, while input frames of the third category, which have dense point clouds, are classified as "convex" frames. Then, based on the classification results, the depth map of the corresponding input frame is projected onto the dense three-dimensional point cloud using the digital surface model to construct a three-dimensional surface model of the real-world scene.

[0067] S150: Uses camera pose and 3D surface model to orthorectify the input frame and generate a high-resolution orthorectified image;

[0068] In this step, the goal of orthorectification is to correct image visual distortions caused by viewpoint and surface structure using the constructed 3D surface model and camera pose, ensuring geometric accuracy during the global image stitching process. Specifically, this embodiment uses a grid-based orthogonal projection (i.e., back projection) method to orthorectify the input frame. When the mapped input frame is significantly larger than the image that multi-view reconstruction can process, the initial grid is adjusted to the maximum expected ground sampling distance (GSD) using the grid-based orthogonal projection method. By saving the elevation of the real scene as a 2.5D grid map, it is treated as a regular single-channel image with floating-point data rather than intensity values. Spatial resolution and texture resolution can be treated as independent parameters. Spatial resolution mainly depends on the densification process and the generation of the 3D surface model, while texture resolution is independent and can be set to any value limited only by the original image resolution.

[0069] S160: Integrate the orthorectified orthophoto image into the global stitching map for stitching to generate a global image of the real scene;

[0070] In this step, after completing the orthorectification of the input frames of all UAVs, the orthorectified images of all UAVs are integrated into the global stitching ground map and fused into a complete global image. On-site personnel can observe the stitching of the global image in real time to ensure the accuracy and real-time performance of the global image stitching.

[0071] Specifically, this application embodiment uses a probabilistic method to mix spatial information and minimize its variance to achieve optimal global image stitching. First, the global mapping is initialized by receiving the first orthorectified frame of the image. Subsequent orthorectified images are used as map update frames, and these frames are categorized into regions with no prior information and overlapping regions based on their association with the first orthorectified frame. The probabilistic method introduces three new data layers into the globally stitched map: elevation variance, elevation assumption, and observation count. When a new orthorectified image is received and it is in a non-overlapping region, a temporary floating average of the new elevation is calculated for each grid cell.

[0072] in n represents the current average elevation in the global mosaic map. ij For the number of observations, Update the elevation of the map. Simultaneously, estimate the floating sample variance. Defined as:

[0073] if If the variance is below the set variance threshold, the new floating sample variance and temporary floating mean are written to the grid map layer, and the observation count of the current grid cell is increased; if If the variance exceeds a set threshold, the new floating sample variance is written into the elevation hypothesis layer, while existing data is retained. These strategies minimize variance, thereby reducing noise in the elevation data.

[0074] Based on the above, the video image geographic stitching method of this application introduces real-time communication technology and UAV multi-video strategy to acquire real-time live images and GNSS information of the real scene. The real-time live images can provide an intuitive and realistic on-site situation, and the GNSS information can ensure the accuracy and completeness of global image stitching. It also fully leverages the advantages of visual SLAM. By using techniques such as attitude estimation, dense mapping, surface generation, orthorectification, and image stitching, it achieves efficient global image stitching of the real scene through multi-UAV collaboration, which greatly improves the real-time performance of data acquisition. It can quickly acquire real-time images of the real scene, ensuring that on-site personnel can quickly grasp global information. It provides an unprecedented perspective for on-site personnel to comprehensively and deeply understand the emergency scene, and provides strong support for rapid emergency response in the real scene, thereby effectively protecting the lives and property of disaster victims.

[0075] Please refer to Figure 2, which is a schematic diagram of the structure of a video image geographic stitching device according to an embodiment of this application. The video image geographic stitching device 40 according to an embodiment of this application includes:

[0076] Data acquisition module 41: used to acquire real-time live images of the real scene collected by multiple UAVs and GNSS information of multiple UAVs;

[0077] Pose estimation module 42: Used to track the input frame of the real-time live image using the visual SLAM algorithm, and combine visual trajectory and GNSS information to estimate the pose of the successfully tracked input frame to generate a camera pose with geographic reference.

[0078] Surface generation module 43: Used to densify the successfully tracked input frame using the camera pose and surface reconstruction algorithm, reconstruct the dense 3D point cloud of the input frame, generate a depth map of the real scene, and project the depth map onto the dense 3D point cloud through a digital surface model to construct a 3D surface model of the real scene.

[0079] Image stitching module 44: Used to perform orthorectification on the input frame using the camera pose and 3D surface model, and integrate the orthorectified input frame into the global stitching ground map to generate a global image of the real scene.

[0080] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0081] The apparatus provided in this application can be applied to the foregoing method embodiments. For details, please refer to the description of the above method embodiments, which will not be repeated here.

[0082] Please refer to Figure 3, which is a schematic diagram of the structure of a computer device according to an embodiment of this application. The computer device 50 includes:

[0083] Memory 51 storing executable program instructions;

[0084] Processor 52 connected to memory 51;

[0085] The processor 52 is used to call the executable program instructions stored in the memory 51 and perform the following steps: acquire real-time live images of the real scene collected by multiple UAVs and GNSS information of multiple UAVs; use a visual SLAM algorithm to track the input frames of the real-time live images, and combine visual trajectory and GNSS information to estimate the pose of the successfully tracked input frames to generate a camera pose with georeference; use the camera pose and surface reconstruction algorithm to densify the successfully tracked input frames, reconstruct the dense 3D point cloud of the input frames, generate a depth map of the real scene, and project the depth map onto the dense 3D point cloud through a digital surface model to construct a 3D surface model of the real scene; use the camera pose and 3D surface model to orthorectify the input frames, and integrate the orthorectified input frames into a global mosaic map to generate a global image of the real scene.

[0086] The processor 52 can also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0087] Please refer to Figure 4, which is a schematic diagram of the structure of the storage medium in an embodiment of this application. The storage medium in this embodiment stores program instructions 61 capable of implementing the following steps: acquiring real-time live images of a real scene collected by multiple UAVs and GNSS information of multiple UAVs; using a visual SLAM algorithm to track the input frames of the real-time live images, and combining visual trajectory and GNSS information to estimate the pose of the successfully tracked input frames, generating a camera pose with georeference; using the camera pose and surface reconstruction algorithm to densify the successfully tracked input frames, reconstructing a dense 3D point cloud of the input frames, generating a depth map of the real scene, and projecting the depth map onto the dense 3D point cloud using a digital surface model to construct a 3D surface model of the real scene; using the camera pose and 3D surface model to orthorectify the input frames, and integrating the orthorectified input frames into a global mosaic map to generate a global image of the real scene. The program instructions 61 can be stored in the aforementioned storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network computer device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage media include: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program instructions, or terminal computer devices such as computers, servers, mobile phones, and tablets. Servers can be standalone servers or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0088] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, apparatuses, or units, and may be electrical, mechanical, or other forms.

[0089] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for geographic stitching of video images, characterized in that, include: Acquire real-time live images of real-world scenes collected by multiple drones, as well as GNSS information from multiple drones; The visual SLAM algorithm is used to track the input frames of the live image, and the pose of the successfully tracked input frames is estimated by combining visual trajectory and GNSS information to generate a camera pose with geographic reference. The camera pose and surface reconstruction algorithm is used to densify the successfully tracked input frame, reconstruct the dense 3D point cloud of the input frame, generate a depth map of the real scene, and project the depth map onto the dense 3D point cloud through a digital surface model to construct a 3D surface model of the real scene. The input frame is orthorectified using the camera pose and 3D surface model, and the orthorectified input frame is integrated into the global mosaic map to generate a global image of the real scene.

2. The video image geographic stitching method according to claim 1, characterized in that, The acquisition of real-time live images of the real-world scene collected by multiple drones and the GNSS information of the multiple drones specifically includes: The node uses real-time video communication technology to receive live images collected by multiple UAVs and processes the live images according to a set frame rate. At the same time, another node uses real-time communication to obtain GNSS information of multiple UAVs. The GNSS information includes the current location latitude and longitude, heading angle, pitch angle, roll angle and altitude.

3. The video image geographic stitching method according to claim 2, characterized in that, After acquiring real-time live images of the real-world scene collected by multiple drones and GNSS information from the multiple drones, the process also includes: The real-time live image is enhanced using a histogram equalization algorithm, and the GNSS information is transformed using projection transformation to obtain geographic coordinates.

4. The video image geographic stitching method according to any one of claims 1 to 3, characterized in that, The process involves using a visual SLAM algorithm to track input frames of the live video feed, and combining visual trajectory and GNSS information to estimate the pose of successfully tracked input frames, generating a georeferenced camera pose. Specifically: Create a visual SLAM interface, pass in a real-time live image through the visual SLAM interface, and perform input frame tracking on the passed real-time live image. If the tracking fails, switch the state to lost and do not set the visual pose. Successfully tracked input frames are divided into frame insertion and keyframe insertion. It is determined whether the current input frame contains important new information. Successfully tracked input frames are then sent to a geographic reference system. This system combines visual trajectory and GNSS information to identify and determine the transformation relationship from the local visual coordinate system to the global geographic coordinate system. The camera pose matrix M of the input frame is calculated using a visual SLAM algorithm, and a sparse point cloud is constructed. Here, M is a 4x4 transformation matrix from the camera coordinate system to the world coordinate system. Where, r ij These are the components of the (3x3) rotation matrix R. These are the UTM coordinates of the global camera position. It refers to the relative height of the camera.

5. The video image geographic stitching method according to claim 4, characterized in that, The step involves using the camera pose and surface reconstruction algorithm to densify the successfully tracked input frame, reconstructing a dense 3D point cloud of the input frame, and generating a depth map of the real-world scene. Specifically: The applicability of the input frame is evaluated. For the camera pose obtained through visual trajectory, a dense 3D point cloud of the input frame is reconstructed through a denser interface or sparse cloud interpolation. Then, the sparse point cloud generated by the visual SLAM algorithm is reprojected onto the image plane, and depth is written at the pixel location according to the interpolation method. The values ​​are used to generate a relatively sparse depth map, and then the depth map is repaired to generate a complete and continuous depth map.

6. The video image geographic stitching method according to claim 5, characterized in that, The step of projecting the depth map onto a dense 3D point cloud using a digital surface model to construct a 3D surface model of the real-world scene specifically involves: The input frame is evaluated based on the pose estimation results and the densification processing results, and the input frame is divided into the following three types: Category 1: GNSS information frames with failed pose estimation, fixed attitude, and no point cloud frames; The second category includes camera pose frames, precise pose frames, and sparse point cloud frames when densification processing fails. The third category consists of camera pose frames, accurate pose frames, and dense point cloud frames where both pose estimation and densification processing were successful. A digital surface model is extracted based on the evaluation results. The digital surface model is constructed in the form of a 2.5D elevation map to represent the three-dimensional surface model of the real scene. First, the surface assumptions are determined. The first and second types of input frames with GNSS information but lacking or having only sparse surface information are classified as "planar" frames, and the third type of input frames with dense point clouds are classified as "convex" frames. Then, the depth map of the corresponding input frame is projected onto the dense three-dimensional point cloud based on the classification results using the digital surface model to construct the three-dimensional surface model of the real scene.

7. The video image geographic stitching method according to claim 6, characterized in that, The input frame is orthorectified using the camera pose and 3D surface model, and the orthorectified input frame is integrated into the global mosaic map to generate a global image of the real scene, specifically: The input frame is orthorectified using a grid-based orthogonal projection method to generate a high-resolution orthorectified image; The orthorectified orthophoto image is integrated into the global stitching ground map, and a probabilistic method is used to mix spatial information and minimize variance for global image stitching; The global image stitching process includes: receiving the first frame of orthorectified image for initialization; using subsequently received orthorectified images as map update frames; and dividing the map update frames into regions without prior information and overlapping regions based on their association with the first frame of orthorectified image. The probabilistic method introduces three new data layers into the globally stitched map: elevation variance, elevation assumption, and observation count. When a new orthorectified image is received and it is in a non-overlapping region, a temporary floating average of the new elevation is calculated for each grid cell. in n represents the current average elevation in the global mosaic map. ij For the number of observations, Update the elevation of the map; simultaneously estimate the floating sample variance. Defined as: if If the variance is below the set variance threshold, the new floating sample variance and temporary floating mean are written to the grid map layer, and the observation count of the current grid cell is increased; if If the variance exceeds the set threshold, the new floating sample variance will be written into the elevation hypothesis layer, while the existing data will be retained.

8. A video image geographic stitching device, characterized in that, include: Data acquisition module: used to acquire real-time live images of real-world scenes collected by multiple drones, as well as GNSS information from multiple drones; Pose estimation module: used to track the input frames of the real-time live image using the visual SLAM algorithm, and to estimate the pose of the successfully tracked input frames by combining visual trajectory and GNSS information, and to generate camera pose with geographic reference. Surface generation module: This module utilizes the camera pose and surface reconstruction algorithm to densify the successfully tracked input frame, reconstructing a dense 3D point cloud of the input frame to generate a depth map of the real-world scene. The depth map is then projected onto a dense 3D point cloud using a digital surface model to construct a 3D surface model of the real scene. Image stitching module: Used to perform orthorectification on the input frame using the camera pose and 3D surface model, and integrate the orthorectified input frame into the global stitching ground map to generate a global image of the real scene.

9. A computer device, characterized in that, The computer device includes a processor and a memory coupled to the processor, wherein, The memory stores program instructions for implementing the video image geographic stitching method according to any one of claims 1-7; The processor is used to execute the program instructions stored in the memory to control the video image geographic stitching method.

10. A storage medium, characterized in that, The device stores processor-executable program instructions for performing the video image geographic stitching method according to any one of claims 1 to 7.

Images