Real-time unmanned aerial vehicle image splicing method and system and storage medium

Abstract

The invention discloses a real-time unmanned aerial vehicle image splicing method and system and a storage medium, and belongs to the technical field of geographic information. An unmanned aerial vehicle collects image data according to a planned route; transmitting the acquired image data to a remote controller; performing real-time splicing on the image data through an image processing algorithm built in a remote controller to obtain a spliced image; and displaying the spliced image on a display screen of the remote controller so as to achieve the purpose of carrying out real-time unmanned aerial vehicle image local splicing in a limited resource environment through the unmanned aerial vehicle and the corresponding unmanned aerial vehicle remote controller.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of geographic information, and in particular relates to a real-time unmanned aerial vehicle image splicing method and system and a storage medium. BACKGROUND

[0002] Unmanned aerial vehicles are widely used in many fields such as surveying and mapping, emergency rescue, agricultural monitoring, and urban planning. It is often necessary for the unmanned aerial vehicle to take real-time images of the target area to meet the needs of users for surveying and mapping, emergency rescue, agricultural monitoring, and urban planning.

[0003] Due to the flight height and camera focal length limitations of unmanned aerial vehicles, a single frame of image often cannot cover the entire target area. In order to obtain large-area, high-resolution panoramic images, multiple images taken by the unmanned aerial vehicle need to be spliced to provide more comprehensive and accurate basic data to meet the needs of various fields for large-scale, high-precision images.

[0004] In traditional technology, after the unmanned aerial vehicle completes the image collection of the area, the original pictures need to be transmitted to the backend server for processing. The processing procedure is as follows: S1: Data collection: the unmanned aerial vehicle takes a large number of pictures according to the preset flight route; S2: Data return: the original pictures are transmitted back to the server through wireless transmission methods such as 4G / 5G, wireless wifi, or wired network; S3: Backend processing: the server uses professional software such as Pix4D, Agisoft Metashape, etc. to perform image splicing and three-dimensional reconstruction, and generates a spliced map.

[0005] Based on the above existing technology, the following defects exist: (1) Transmission bandwidth bottleneck: high-resolution image data is large, and a single image can reach dozens of MB, with significant delay in long-distance transmission; (2) Lack of real-time performance: it can take several hours from shooting to generating a map, which cannot meet the needs of emergency rescue, dynamic monitoring, and other scenarios; (3) Over-reliance on cloud computing power: the backend server needs to be configured with a high-performance GPU / CPU cluster, which is costly.

[0006] (4) Requires communication network support: in some environments with limited resources, the communication network is not good enough to complete the data return.

[0007] Based on the above, there is an urgent need to develop a map generation technology that can complete the local splicing of unmanned aerial vehicle images in real time to meet more extensive application scenarios. SUMMARY

[0008] In view of this, in order to solve the above problems existing in the prior art, the purpose of the present application is to provide a real-time unmanned aerial vehicle image splicing method, system and storage medium to achieve the purpose of being able to perform real-time local splicing of unmanned aerial vehicle images in a limited resource environment through an unmanned aerial vehicle and a corresponding unmanned aerial vehicle remote controller.

[0009] The technical scheme adopted by the present application is as follows: a real-time unmanned aerial vehicle image splicing method, the method comprising: S1: the unmanned aerial vehicle collects image data according to a planned route; S2: each frame of collected image data is transmitted to the remote controller in real time; S3: the image data is spliced in real time through an image processing algorithm built in the remote controller to obtain a spliced image; S4: the spliced image spliced in real time is displayed and dynamically updated on the display screen of the remote controller.

[0010] Further, the image processing algorithm comprises: A1: extracting feature points of a first frame; A2: extracting feature points of a second frame and matching the feature points of the first frame to obtain matching points; A3: solving a relative pose relationship of the second frame relative to the first frame; A4: calculating a rotation matrix M of camera coordinates to geographic coordinates and synchronously constructing a triangular net, wherein the triangular net is constructed based on an initial point cloud; A5: constructing a geographic coordinate triangular net W according to the triangular net and the rotation matrix M; A6: generating a preliminary projection image through GPU texture mapping projection.

[0011] Further, the image processing algorithm further comprises: B1: extracting feature points of an Nth frame, and synchronously executing B2 and B3; B2: matching with an (N-1)th frame, if matching, then entering B4; if not matching, then searching for a matching frame within a tolerance radius, if there is no matching frame, then ending, if there is a matching frame, then entering B4; wherein searching for a matching frame within a tolerance radius can reduce the search range, reduce the calculation complexity of frame matching, replace full-quantity traversal with spatial constraint, and reduce the calculation times of feature matching, compared with traditional frame matching which traverses all historical frames, the calculation power requirement is exponentially reduced; B3: matching with the initial point cloud, solving a relative pose relationship of the Nth frame relative to the (N-1)th frame, calculating a rotation matrix M' of current frame coordinates to geographic coordinates, and synchronously executing B4; B4: updating the triangular net based on a new point cloud; B5: constructing a geographic coordinate triangular net W' according to the updated triangular net and the rotation matrix M', and searching for a triangular best matching frame within a tolerance radius. B6: generating a projection image by GPU texture mapping projection; Wherein, N≥3, B1-B6 are executed in a loop to update the projection image in real time by 1 frame per superposition.

[0012] Further, the method for calculating the rotation matrix M of the camera coordinates to the geographic coordinates is: A4011: calculating the camera center coordinates based on the camera coordinate system; A4012: reading the geographic coordinates of the camera center based on the geographic coordinate system; A4013: calculating the rotation matrix M of the camera coordinates to the geographic coordinates.

[0013] Further, the method for constructing the triangular mesh is: A4021: triangulating based on the relative pose relationship of the second frame relative to the first frame; A4022: calculating the camera coordinates; A4023: performing coarse screening on the triangulated three-dimensional point cloud to filter out invalid data in advance, avoiding redundant calculation and greatly reducing the data size for subsequent processing; A4024: performing local bundle adjustment optimization, which sacrifices global optimization to pursue local accuracy and greatly reduces the calculation complexity; A4025: performing fine screening on the three-dimensional point cloud and obtaining an initial point cloud; A4026: constructing a triangular mesh based on the initial point cloud.

[0014] Further, the method for calculating the rotation matrix M' of the current frame coordinates to the geographic coordinates is: B301: calculating the camera center coordinates of the current frame based on the camera coordinate system; B302: reading the geographic coordinates of the camera center based on the geographic coordinate system; B303: calculating the rotation matrix M' of the camera coordinates of the current frame to the geographic coordinates.

[0015] Further, the method for updating the triangular mesh is: B401: triangulating based on the relative pose relationship of the Nth frame relative to the (N-1)th frame; B402: calculating the camera coordinates; B403: performing coarse screening on the triangulated three-dimensional point cloud; B404: performing local bundle adjustment optimization, which sacrifices global optimization to pursue local accuracy and greatly reduces the calculation complexity; B405: performing fine screening on the three-dimensional point cloud and obtaining a new point cloud; B406: performing local bundle adjustment optimization based on the new point cloud and updating the triangular mesh.

[0016] Further, the GPU texture mapping includes a planar projection and an orthographic projection, the GPU texture mapping converts serial pixel operations into parallel processing, and the GPU is assisted by the inherent graphic acceleration capability; on the one hand, the algorithmic power bottleneck of the CPU serial processing can be avoided; on the other hand, the hardware driver is highly optimized, and complex algorithm development is not required.

[0017] The application further provides a real-time unmanned aerial vehicle image splicing system, which comprises an unmanned aerial vehicle and a remote controller in communication connection with the unmanned aerial vehicle, a memory in communication connection with a master control chip of the remote controller, and an application program built in the memory and called by the master control chip to execute the real-time unmanned aerial vehicle image splicing method.

[0018] The application further provides a storage medium, wherein the storage medium stores program instructions, and a computer reads the program instructions to execute the real-time unmanned aerial vehicle image splicing method.

[0019] The application has the following beneficial effects: The real-time image splicing demand can be met in a limited resource environment, the entire image splicing process does not need to transmit data to a background server or other ground processing equipment, and the built-in processor of the remote controller completes the real-time image splicing; After the user controls the unmanned aerial vehicle to take off and take a picture, the user can directly generate a spliced image of the picture taken by the unmanned aerial vehicle on the remote controller and view the spliced image in real time, which reduces the dependence on external resources and improves the user experience.

[0020] The invalid data is reduced in advance through coarse screening of the point cloud at the data level, the calculation amount is reduced from the source, the matching range is reduced through the tolerance radius search at the search level, the feature calculation times are reduced, the CPU bottleneck is avoided through GPU texture mapping at the hardware level, the variable scale is compressed through local bundle adjustment at the optimization level, the global high complexity operation is avoided, and the calculation complexity is maximally reduced under the premise of meeting the unmanned aerial vehicle image splicing accuracy demand. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a logic flow chart of the real-time unmanned aerial vehicle image splicing method provided by the application. DETAILED DESCRIPTION

[0022] Embodiments of the present application are described below in detail, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar modules or modules with the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are for the purpose of explanation only, and are not to be understood as limiting the present application. On the contrary, embodiments of the present application include all changes, modifications and equivalents falling within the spirit and scope of the appended claims.

[0023] Embodiment 1 In special scenarios, due to limited network communication resources, a large amount of data cannot be transmitted, thus, there is a data interaction obstacle with a device having high computing performance, or due to limited computing resources and data processing hardware resources, a device with high computing performance cannot be carried, thus, image stitching cannot be achieved.

[0024] As shown in Figure 1 In the present embodiment, a real-time unmanned aerial vehicle image stitching method is specifically disclosed, which can realize real-time unmanned aerial vehicle image stitching in a limited resource environment through an unmanned aerial vehicle and a corresponding remote controller. The image stitching method comprises: S1: The unmanned aerial vehicle collects image data according to a planned route, and the collection of image data is realized by a camera on the unmanned aerial vehicle body.

[0025] S2: Each frame of collected image data is transmitted to the remote controller in real time, and the transmission of image data is based on an existing transmission channel between the remote controller and the unmanned aerial vehicle, generally using 2.4GHz wireless communication and 5.8GHz wireless communication. The present embodiment is more suitable for using 5.8GHz wireless communication between the unmanned aerial vehicle and the remote controller, which has fast transmission speed and good anti-interference performance.

[0026] S3: The image data is stitched in real time by an image processing algorithm built-in the remote controller to obtain a stitched image; the image processing algorithm comprises: A1: Extracting feature points of the first frame, wherein the feature points are points in the image that have unique characteristics, are easy to identify and match, such as corner points, edge points, etc. A specific feature extraction algorithm, such as SIFT, SURF, ORB, etc., is used to extract representative points on the first frame image as feature points.

[0027] A2: Similarly, the feature extraction algorithm is used to extract feature points from the second frame image, and the feature points are matched with the feature points of the first frame to obtain matching points; in the present embodiment, a vector describing the features around the feature points is used, and a method such as Euclidean distance is used to find a point pair with high similarity between the feature points of the two frames as matching points.

[0028] A3: solve the relative pose relationship of the second frame relative to the first frame, the relative pose relationship refers to the rotation and translation relative relationship between two frames; in this embodiment, the relative pose is recovered from the essential matrix, the essential matrix contains the rotation and translation information between two frames of cameras, and the rotation matrix M and the translation vector t of the camera are obtained by decomposing the essential matrix, so as to recover the relative pose relationship between two frames of cameras.

[0029] A4: calculate the rotation matrix M of the camera coordinate to the geographic coordinate and simultaneously perform the construction of the triangular net, specifically: On the one hand, the method for calculating the rotation matrix M of the camera coordinate to the geographic coordinate is: A4011: calculate the camera center coordinate based on the camera coordinate system, the camera center can be understood as the camera optical center, and then the position coordinate of the camera center in the camera coordinate system is obtained; A4012: read the geographic coordinate of the camera center based on the geographic coordinate system, the coordinate of the camera optical center in the geographic coordinate system can be directly obtained through an external device, such as a GPS positioning module, prior geographic information, etc.; A4013: describe how to rotate the x / y / z axes of the camera coordinate to align with the geographic coordinate system through the rotation matrix M, if the vector in the camera coordinate system is , the vector in the geographic coordinate system is , then: (1) The rotation matrix M of the camera coordinate to the geographic coordinate can be calculated based on the above formula (1).

[0030] On the other hand, the triangular net construction is to connect the points in the initial point cloud into a triangular mesh to represent the surface shape of the three-dimensional space, such as using the Delaunay triangulation algorithm, etc. In this embodiment, the method for constructing the triangular net is as follows: A4021: triangulate based on the relative pose relationship of the second frame relative to the first frame, wherein the triangulation is to calculate the three-dimensional point cloud corresponding to the matching points according to the pixel coordinates of the matching points in two frames of images, the intrinsic parameters of the camera and the relative pose relationship, by using the projection relationship of the two frames; A4022: calculate the camera coordinate to obtain the coordinates of the three-dimensional point cloud in the current camera coordinate system (understood as: local coordinate); A4023: coarsely screen the triangulated three-dimensional point cloud, the core purpose of coarsely screening the point cloud is to filter invalid noise points, and the camera coordinate is a key basis for judging whether the three-dimensional point cloud is valid; the coarsely screening method is as follows: A40231: the camera coordinate provides the spatial range basis for screening and determines the "reasonable three-dimensional point space range" based on the camera coordinate; A40232: Eliminating "outliers" caused by matching errors. The accuracy of triangulation depends on the matching quality of image feature points. If there are incorrect matches (such as mismatching points from different scenes as corresponding points), "abnormal 3D points" that deviate from the real scene will be generated. The spatial location of these outliers usually does not match the "reasonable shooting range" corresponding to the camera coordinates (such as being too far or too close to the camera). By setting a spatial threshold based on the camera coordinates, these outliers can be quickly and coarsely filtered out.

[0031] A4024: Perform local bundle adjustment optimization. Local bundle adjustment is based on the idea of ​​bundle adjustment and optimizes the position of the 3D point cloud and the pose of the camera within a local range. A4025: Fine-screening of 3D point clouds to obtain initial point clouds. Based on coarse screening and local bundle adjustment, further screening is performed to select more accurate and reliable 3D point clouds as initial point clouds. A4026: Constructing a triangular mesh based on an initial point cloud, connecting the points in the initial point cloud into a triangular mesh to represent the surface shape in three-dimensional space, and constructing the triangular mesh using the Delaunay triangulation algorithm.

[0032] A5: Construct a geographic coordinate triangulation W based on the triangulation and rotation matrix; transform the triangulation to the geographic coordinate triangulation W based on the rotation matrix M, which is to transform the camera coordinate system-based triangulation to the geographic coordinate system, so that it has a real geographic spatial location.

[0033] A6: A preliminary projected image is generated through GPU texture mapping projection. GPU texture mapping utilizes the parallel computing power of the graphics processing unit (GPU) to map the texture information of the image, such as color and texture details, onto a triangular mesh, giving the 3D model a realistic appearance. The GPU texture mapping includes planar projection and orthographic projection. Planar projection projects the 3D triangular mesh onto a plane; orthographic projection is a special type of perpendicular projection that can obtain an effect similar to an orthographic image, facilitating subsequent image stitching and display.

[0034] The above section describes the image stitching method based on the first and second frames. Building upon this, the image stitching method for the subsequent third, fourth, ... N-1 and Nth frames is performed using the following method: B1: Extract feature points from the Nth frame. Use a feature extraction algorithm to extract feature points from each subsequent frame image to prepare for matching with existing 3D point clouds or previous frames. Simultaneously execute B2 and B3. B2: Match with frame N-1. The matching method is to match the feature points of frame N with the projection features of the 3D point cloud in frame N-1 by determining whether there are enough correspondences to perform the matching. If a match is found, proceed to B4; If no match is found, a matching frame is searched within the tolerance radius. If no matching frame is found, the current workflow cannot continue and the process ends. If a matching frame is found, it means that there is a part that can be spliced ​​between the Nth frame and its previous frame, and the process proceeds to B4. The tolerance radius is a set range within which possible matching frames are searched. This is to expand the search range and try to find frames that overlap with subsequent frames when a match fails. Searching for matching frames within the tolerance radius compresses the search range from "all frames" to "local small-range frames" and makes the matching index lightweight, avoiding complex feature calculations.

[0035] B3: Match with the initial point cloud, solve the relative pose relationship between the Nth frame and the (N-1)th frame, and calculate the rotation and translation parameters of the camera in the Nth frame relative to the camera in the (N-1)th frame. That is: calculate the rotation matrix M' from the current frame coordinates to the geographic coordinates and execute B4 simultaneously; the method used to calculate the rotation matrix M' from the current frame coordinates to the geographic coordinates is as follows: B301: Calculate the camera center coordinates of the current frame based on the camera coordinate system, that is: calculate the position coordinates of the camera optical center in the camera coordinate system of the current frame; B302: Based on the geographic coordinate system, the geographic coordinates of the camera center can be read. Similarly, through external devices, such as GPS positioning modules and prior geographic information, the coordinates of the camera optical center in the geographic coordinate system can be directly obtained. B303: Calculate the rotation matrix M' from the camera coordinates to the geographic coordinates of the current frame. The rotation matrix M' describes how the x / y / z axes of the camera coordinates are rotated to align with the geographic coordinate system. It is obtained by calculating the above formula (1).

[0036] B4: Update the triangular network based on the new point cloud; wherein, the method used to update the triangular network is as follows: B401: Triangulation is performed based on the relative pose relationship between the Nth frame and the (N-1th)th frame. Triangulation is performed by using the projection relationship between the two frames and calculating the 3D point cloud corresponding to the matching point based on the pixel coordinates of the matching point in the two frames, the camera's intrinsic parameters, and the relative pose relationship. B402: Calculate camera coordinates, which is the same as "A4022" above, and will not be repeated here; B403: Coarse screening of triangulated 3D point clouds, the principle is the same as "A4023" above, and will not be repeated here; B404: Performs local beam adjustment optimization, which is the same as the principle of "A4024" above, and will not be described again; B405: Refines the 3D point cloud and obtains a new point cloud. The principle is the same as "A4025" above, so it will not be described again. B406: Local beam adjustment optimization is performed based on the new point cloud. The local beam adjustment optimizes the 3D point cloud after the addition of the new point cloud, reduces the reprojection error, and improves the accuracy of the 3D point cloud and camera pose. Based on this, the triangular mesh is readjusted and updated according to the newly added point cloud so that it can more accurately represent the surface of three-dimensional space.

[0037] B5: Construct a geographic coordinate triangulation W' based on the updated triangulation and rotation matrix M' to transform the updated triangulation to the actual geographic coordinate system, ensuring the correctness of its geographic spatial location, and searching for the best matching frame of the triangle within the tolerance radius; Searching for the best-matching frame for the triangle within the tolerance radius is to find the next frame to be processed, ensuring that the subsequent frame has sufficient overlap with the currently processed point cloud or frame to guarantee the continuity of image stitching. This narrows the search range and reduces the computational complexity of frame matching. The method for searching for the best-matching frame for the triangle is as follows: B501: Setting the tolerance radius Typically, the centroid or vertex of the target triangle is used as the center of a circle, and the radius value is set based on the drone's flight altitude, camera field of view (FOV), and frame overlap rate (for example, when the flight altitude is 100 meters and the FOV is 60°, the radius is set to 10-20 meters) to ensure that frames within the radius can cover the triangle. B502: Filtering candidate frames Retrieve the camera geographic coordinates of all stored image frames, and filter out all frames whose camera geographic coordinates fall within the tolerance radius as candidate frames; B503: Calculate the matching degree between the candidate frame and the target triangle, where the matching degree indexes are: feature point overlap rate, texture similarity and projection error; B504: Select the best matching frame All candidate frames are weighted and scored according to matching metrics (e.g., feature point overlap rate of 40%, texture similarity of 40%, projection error of 20%). The frame with the highest score is selected as the best matching frame. The best matching frame contains the scene texture that best matches the geographic coordinate triangulation W'.

[0038] B6: Based on the best matching frame and the geographic coordinate triangulation W', a projected image is generated by GPU texture mapping, which includes planar projection and orthographic projection; wherein, N≥3, and B1-B6 are executed in a loop to update the projected image in real time by superimposing 1 frame at a time.

[0039] S4: Display and dynamically update the stitched image in real time on the remote control's screen.

[0040] The above explanation indicates that the first frame, the second frame, ..., the (N-1)th frame and the Nth frame refer to the first frame image, the second frame image, ..., the (N-1)th frame image and the Nth frame image, respectively.

[0041] Example 2 This embodiment also provides a real-time UAV image stitching system, which includes: a UAV, a remote controller that establishes a communication connection with the UAV, the main control chip of the remote controller being communicatively connected to a memory, the memory containing an application program, and the main control chip executing the real-time UAV image stitching method as described in Embodiment 1 above by calling the application program.

[0042] When the system is running, the drone and the remote controller are connected via a wireless local area network. It does not need to connect to external 4G, 5G or other external networks. Data collection and processing are all completed locally, without the need for a third device, such as a computer or server, which greatly expands the scope of its application scenarios.

[0043] Example 3 This embodiment also provides a storage medium storing program instructions. After reading the program instructions, the computer executes the real-time UAV image stitching method as described in Embodiment 1 above.

[0044] It should be noted that any process or method description in the flowchart or otherwise described herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including in substantially the same manner or in the reverse order of the functions involved, as should be understood by those skilled in the art to which the embodiments of this application pertain.

[0045] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0046] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0047] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0048] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0049] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A real-time drone image stitching method, characterized in that, The method comprises: S1: the unmanned aerial vehicle collects image data according to a planned route; S2: each frame of collected image data is transmitted to a remote controller in real time; S3: the image data is spliced in real time by an image processing algorithm built in the remote controller to obtain a spliced image; S4: the spliced image spliced in real time is displayed on a display screen of the remote controller and is dynamically updated.

2. The real-time drone image stitching method of claim 1, wherein, The image processing algorithm comprises: A1: extracting feature points of a first frame; A2: extracting feature points of a second frame and matching the feature points of the second frame with the feature points of the first frame to obtain matching points; A3: solving a relative pose relationship of the second frame relative to the first frame; A4: calculating a rotation matrix M of camera coordinates to geographic coordinates and synchronously constructing a triangular network, wherein the triangular network is constructed based on an initial point cloud; A5: constructing a geographic coordinate triangular network W according to the triangular network and the rotation matrix M; A6: generating a preliminary projection image through GPU texture mapping projection.

3. The real-time drone image stitching method of claim 2, wherein, The image processing algorithm further comprises: B1: extracting feature points of an Nth frame, and synchronously executing B2 and B3; B2: matching with an (N-1)th frame, if matching, entering B4; if not matching, searching for a matching frame within a tolerance radius, if there is no matching frame, ending, if there is a matching frame, entering B4; B3: matching with the initial point cloud, solving a relative pose relationship of the Nth frame relative to the (N-1)th frame, calculating a rotation matrix M' of current frame coordinates to geographic coordinates, and synchronously executing B4; B4: updating the triangular network based on a new point cloud; B5: constructing a geographic coordinate triangular network W' according to the updated triangular network and the rotation matrix M', and searching for a triangular best matching frame within the tolerance radius; B6: generating a projection image through GPU texture mapping projection; wherein N is greater than or equal to 3, B1-B6 are executed in a loop to update the projection image in real time by adding 1 frame each time.

4. The real-time drone image stitching method of claim 2, wherein, A method for calculating the rotation matrix M of camera coordinates to geographic coordinates comprises: A4011: calculating a camera center coordinate based on a camera coordinate system; A4012: reading geographic coordinates of the camera center based on a geographic coordinate system; A4013: calculating the rotation matrix M of camera coordinates to geographic coordinates.

5. The real-time drone image stitching method of claim 2, wherein, A method for constructing the triangular network comprises: A4021: triangulating based on the relative pose relationship of the second frame relative to the first frame; A4022: calculating camera coordinates; A4023: coarsely screening the triangulated three-dimensional point cloud; A4024: performing local bundle adjustment optimization; A4025: finely screening the three-dimensional point cloud and obtaining an initial point cloud; A4026: constructing the triangular network based on the initial point cloud.

6. The real-time drone image stitching method of claim 3, wherein, A method for calculating the rotation matrix M' of current frame coordinates to geographic coordinates comprises: B301: calculating a camera center coordinate of the current frame based on a camera coordinate system; B302: reading geographic coordinates of the camera center based on a geographic coordinate system; B303: calculating the rotation matrix M' of camera coordinates to geographic coordinates of the current frame.

7. The real-time drone image stitching method of claim 3, wherein, A method for updating the triangular network comprises: B401: triangulating based on the relative pose relationship of the Nth frame relative to the (N-1)th frame; B402: calculating camera coordinates; B403: coarsely screening the triangulated three-dimensional point cloud; B404: performing local bundle adjustment optimization; B405: fine screening of the three-dimensional point cloud and obtaining a new point point cloud; B406: local bundle adjustment optimization based on the new point point cloud and updating the triangular net.

8. The real-time drone image stitching method of claim 2 or 3, wherein, The GPU texture mapping includes a planar projection and an orthographic projection.

9. A real-time drone image stitching system, the system comprising: The application relates to a remote controller for establishing a communication connection with a UAV, wherein a main control chip of the remote controller is connected with a memory, the memory is built-in with an application program, and the main control chip executes the real-time UAV image splicing method as claimed in any one of claims 1-7 by calling the application program. 10.A storage medium, wherein a program instruction is stored in the storage medium, and a computer reads the program instruction to execute the real-time UAV image splicing method as claimed in any one of claims 1-7.

Images