Image quick stitching method and device capable of real-time display

Abstract

This application discloses a real-time image stitching method and apparatus. The method includes: receiving images uploaded to the cloud by a camera device in real time; extracting feature points from the images and matching feature points between the current frame and the previous frame; calculating the ratio of the homography matrix score to the sum of the homography matrix score and the fundamental matrix score to determine whether to use the homography matrix or the fundamental matrix to recover the image's pose parameters and map points, and initializing the map; estimating the pose of the current frame and optimizing the pose of the current frame using candidate frames; converting the coordinates of the map points to two-dimensional coordinates and performing plane fitting; converting the coordinates of the image corner points in the camera coordinate system to the object coordinate system, calculating perspective transformation parameters and performing geometric transformation on the image; and performing Gaussian pyramid fusion on the generated tiles. This method solves the problems of low efficiency in scene reconstruction using motion reconstruction algorithms and the inability to process real-time photogrammetric data.

Description

Technical Field

[0001] This application relates to the field of real-time photogrammetry technology, and in particular to a method and apparatus for real-time image stitching. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) can be described as follows: A robot starts moving from an unknown location in an unknown environment, performs self-localization based on its location and map during the movement, and builds an incremental map based on its self-localization to achieve autonomous localization and navigation of the robot.

[0003] Traditional methods for generating orthophoto maps and elevation models from UAV imagery mostly use Structure from Motion (SFM) algorithms, which reconstruct the scene by restoring the camera's pose at the moment of capture. With the development of SLAM technology, it is now possible to combine GPS information from when the image was captured to simultaneously reconstruct the image's pose relative to the world coordinate system and the scale information of the scene model, thereby achieving the reconstruction of the real scene.

[0004] However, the method of reconstructing scenes using motion reconstruction algorithms suffers from low efficiency and cannot perform real-time processing of real-time photogrammetry data. Summary of the Invention

[0005] In this application embodiment, by providing a real-time image stitching method and apparatus, the problem of low efficiency in scene restoration using motion reconstruction structure algorithm is solved, and the problem of not being able to process real-time photogrammetric data is also solved.

[0006] In a first aspect, embodiments of this application provide a real-time image stitching method, comprising: receiving images uploaded to the cloud by a camera device in real time; extracting feature points from the images and performing feature point matching between the current frame and the previous frame of the images; calculating the matrix scores of the homography matrix and the fundamental matrix respectively, and determining whether to use the homography matrix or the fundamental matrix to recover the pose parameters and map points of the images by calculating the ratio of the homography matrix score to the sum of the homography matrix score and the fundamental matrix score, and initializing the map; wherein the pose parameters include rotation matrix and translation vector; estimating the pose of the current frame image and optimizing the pose of the current frame image using candidate frame images; converting the coordinates of the map points to two-dimensional coordinates and performing plane fitting on the converted map points; converting the coordinates of the image corner points in the camera coordinate system to the object coordinate system, calculating perspective transformation parameters and performing geometric transformation on the images; performing tile processing on the images and performing Gaussian pyramid fusion on the generated tiles.

[0007] In conjunction with the first aspect, in one possible implementation, the step of determining whether to use the homography matrix or the base matrix to recover the pose parameters and map points of the image by calculating the ratio of the homography matrix score to the sum of the homography matrix score and the base matrix score includes: determining whether the calculated ratio is greater than a first threshold; if the determination result is yes, using the homography matrix to recover the pose parameters and map points of the image; if the determination result is no, using the base matrix to recover the pose parameters and map points of the image.

[0008] In conjunction with the first aspect, in one possible implementation, estimating the pose of the current frame image includes: fitting the localization system coordinates of the current frame image and the previous frame image, and converting both the image pose and map points to the object coordinate system; finding the corresponding set between map points and feature points between the current frame image and the previous frame image to estimate the pose of the current frame image.

[0009] In conjunction with the first aspect, in one possible implementation, optimizing the pose of the current frame image using candidate frame images includes: obtaining multiple keyframes closely related to the current frame image based on the matching relationship between the current frame image and the previous frame image, such that the multiple keyframes correspond one-to-one with the multiple map points; performing local bundling adjustment optimization on the multiple keyframes and their corresponding multiple map points; determining whether the distance between the current frame image and the multiple keyframes is less than a second threshold since the last loop closure detection; if the determination result is yes, finding candidate frame images with a similarity not lower than a third threshold; if the determination result is no, performing loop closure detection.

[0010] In one possible implementation, the transformation of the coordinates of the image corner point in the camera coordinate system to the object coordinate system includes: calculating the coordinates of the image corner point in the camera coordinate system, and transforming the image corner point from the camera coordinate system to the pixel coordinate system and then to the image space coordinate system; rotating and scaling the image corner point in the camera coordinate system and transforming the coordinates of the image corner point to the object coordinate system.

[0011] In one possible implementation, the calculation of perspective transformation parameters and the geometric transformation of the image include: using the pixel coordinates of the corner points of the image as source coordinates and the object coordinates as target coordinates.

[0012] In one possible implementation, the tile-making process of the image includes: obtaining the projection coordinates of the image; converting the projection coordinates of the image into Web Mercator coordinates or WGS84 coordinates; cutting from the top layer tile and generating the lower layer tiles sequentially.

[0013] Secondly, embodiments of this application provide a multi-stage dense reconstruction device based on uncertainty. The device includes: a receiving module for receiving images uploaded to the cloud by a camera device in real time; a feature point matching module for extracting feature points from the images and performing feature point matching between the current frame and the previous frame of the images; and a calculation module for calculating the matrix scores of the homography matrix and the fundamental matrix respectively, and determining whether to use the homography matrix or the fundamental matrix to recover the pose parameters and map points of the images by calculating the ratio of the homography matrix score to the sum of the homography matrix score and the fundamental matrix score. The map is initialized; the pose parameters include a rotation matrix and a translation vector; an optimization module is used to estimate the pose of the current frame image and optimize the pose of the current frame image using candidate frame images; a plane fitting module is used to convert the coordinates of map points into two-dimensional coordinates and perform plane fitting on the map points after coordinate conversion; a geometric transformation module is used to transform the coordinates of image corner points in the camera coordinate system to the object coordinate system, calculate perspective transformation parameters and perform geometric transformation on the image; a fusion module is used to perform tile processing on the image and perform Gaussian pyramid fusion on the generated tiles.

[0014] In conjunction with the second aspect, in one possible implementation, the step of determining whether to use the homography matrix or the base matrix to recover the pose parameters and map points of the image by calculating the ratio of the homography matrix score to the sum of the homography matrix score and the base matrix score includes: determining whether the calculated ratio is greater than a first threshold; if the determination result is yes, using the homography matrix to recover the pose parameters and map points of the image; if the determination result is no, using the base matrix to recover the pose parameters and map points of the image.

[0015] In conjunction with the second aspect, in one possible implementation, estimating the pose of the current frame image includes: fitting the localization system coordinates of the current frame image and the previous frame image, and converting both the image pose and map points to the object coordinate system; finding the corresponding set between map points and feature points between the current frame image and the previous frame image to estimate the pose of the current frame image.

[0016] In conjunction with the second aspect, in one possible implementation, optimizing the pose of the current frame image using candidate frame images includes: obtaining multiple keyframes closely related to the current frame image based on the matching relationship between the current frame image and the previous frame image, such that the multiple keyframes correspond one-to-one with the multiple map points; performing local bundling adjustment optimization on the multiple keyframes and their corresponding multiple map points; determining whether the distance between the current frame image and the multiple keyframes is less than a second threshold since the last loop closure detection; if the determination result is yes, finding candidate frame images with a similarity not lower than a third threshold; if the determination result is no, performing loop closure detection.

[0017] In conjunction with the second aspect, in one possible implementation, the transformation of the coordinates of the image corner point in the camera coordinate system to the object coordinate system includes: calculating the coordinates of the image corner point in the camera coordinate system, and transforming the image corner point from the camera coordinate system to the pixel coordinate system and then to the image space coordinate system; rotating and scaling the image corner point in the camera coordinate system and transforming the coordinates of the image corner point to the object coordinate system.

[0018] In conjunction with the second aspect, in one possible implementation, the calculation of perspective transformation parameters and the geometric transformation of the image include: using the pixel coordinates of the image corner points as source coordinates and the object coordinates as target coordinates.

[0019] In conjunction with the second aspect, in one possible implementation, the tile-making process of the image includes: obtaining the projection coordinates of the image; converting the projection coordinates of the image into Web Mercator coordinates or WGS84 coordinates; cutting from the top layer tile and generating the lower layer tiles sequentially.

[0020] Thirdly, embodiments of this application provide a real-time image stitching server, including a memory and a processor; the memory is used to store computer-executable instructions; the processor is used to execute the computer-executable instructions to implement the method described in the first aspect or any possible implementation of the first aspect.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing executable instructions, which, when executed by a computer, enable the implementation of the method described in accordance with the first aspect or any possible implementation thereof.

[0022] One or more technical solutions provided in the embodiments of this application have at least the following technical effects:

[0023] This application provides a real-time image stitching method. In implementation, the method receives images uploaded to the cloud by a camera device in real time. It matches feature points between the current and previous frames of the image, initializes the map by restoring the image's pose parameters and map points, optimizes the pose of the current frame, performs planar fitting on the map points, calculates perspective transformation parameters through coordinate system transformation, performs geometric transformation on the image, and finally performs tile-based processing on the generated tiles. This method solves the problems of low efficiency in scene restoration using motion reconstruction algorithms and the inability to process real-time photogrammetric data. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments of the present invention or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This application provides a flowchart of a real-time image stitching method for embodiments;

[0026] Figure 2 The flowchart illustrates the specific steps for restoring the pose parameters and map points of the image by calculating the ratio and using a homography matrix or a fundamental matrix, as provided in the embodiments of this application.

[0027] Figure 3 A flowchart illustrating the specific steps for estimating the pose of the current frame image provided in this application embodiment;

[0028] Figure 4 A flowchart illustrating the specific steps involved in optimizing the pose of the current frame image using candidate frame images, as provided in this embodiment of the application.

[0029] Figure 5 A flowchart illustrating the specific steps for transforming the coordinates of image corner points from the camera coordinate system to the object coordinate system, as provided in this application embodiment.

[0030] Figure 6 This is a flowchart illustrating the specific steps involved in tile-based image processing as provided in an embodiment of this application.

[0031] Figure 7 This application provides a first display result of overlaying a completed quick-paste image with a geographic map in an embodiment of the present application.

[0032] Figure 8 This application provides a second display result of overlaying a completed quick-paste image with a geographic map in an embodiment of the present application.

[0033] Figure 9 A schematic diagram of a real-time image stitching device provided in an embodiment of this application;

[0034] Figure 10 This is a schematic diagram of a real-time image stitching server provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0036] This application provides a method for real-time image stitching, such as... Figure 1 As shown, the method includes steps S101 to S107.

[0037] S101: Receives images uploaded to the cloud by the camera device in real time.

[0038] Before establishing the algorithm module, it is necessary to acquire the images transmitted by the drone in real time. The drone's onboard camera is equipped with a communication module that can upload the acquired images to the storage server. By establishing a link between the drone's communication module and the storage server, the images transmitted by the drone can be acquired in real time.

[0039] The specific steps are as follows: After the UAV's onboard camera collects images, it uploads the collected images through a 4G or 5G communication module. Specifically, the onboard camera uses a five-view tilt image, which can transmit the five-view image back and store it to the storage server. However, in this embodiment, it is necessary to display the texture of the test area in the form of orthophoto projection, so only the downward view image is needed.

[0040] After the images are stored on the storage server, task labels are generated for each image, and image identifiers are set. Cloud storage is used for image storage management, and the storage path of the downward-looking camera images is pushed to the relevant message queue according to the task identifier. Then, an algorithm module is established. However, since the algorithm module does not communicate directly with the camera, it needs to be linked to the designated message queue through the task identifier. Once the image is successfully returned and stored, the algorithm module can obtain the image link through the message queue.

[0041] S102: Extract feature points from the image and perform feature point matching between the current frame and the previous frame.

[0042] Specifically, in order to meet the requirements of real-time processing, full utilization of computing resources, and improvement of the scale-invariant, rotation-invariant, and light-variance-insensitive properties of feature points, scale-invariant feature transform (SIFT) is used to extract feature points from the image. Of course, other methods can also be selected to extract feature points from the image, and this application is not limited to the above methods.

[0043] Furthermore, when performing feature point matching between the current frame and the previous frame of the image, a successful match is considered to have occurred if the number of valid point pairs is greater than a threshold m; otherwise, the match is considered to have failed. Through multiple experiments, the threshold m has been determined to be 10, but other values ​​are also possible, and this application is not limited to these values.

[0044] S103: Calculate the matrix scores of the homography matrix and the fundamental matrix respectively, and determine the pose parameters and map points of the image to be restored by using the homography matrix or the fundamental matrix by calculating the ratio of the homography matrix to the sum of the homography matrix and the fundamental matrix. Then initialize the map. The pose parameters include the rotation matrix and the translation vector.

[0045] Figure 2 In the implementation step S103 provided in the embodiment of this application, the pose parameters and map points of the image are determined by using the homography matrix or the base matrix to recover the image by calculating the ratio of the homography matrix score to the sum of the homography matrix score and the base matrix score, including steps S201 to S203.

[0046] S201: Determine whether the calculated ratio is greater than the first threshold.

[0047] If the judgment result is yes, proceed to step S202: use homography matrix to recover the pose parameters and map points of the image.

[0048] If the judgment result is negative, proceed to step S203: use the basic matrix to recover the pose parameters and map points of the image.

[0049] Specifically, the Random Sample Consensus (RANSAC) algorithm can be used to calculate the homography matrix, homography matrix score, fundamental matrix, and fundamental matrix score between the current frame and the reference frame based on the matching point pair information. Let the homography matrix be H matrix and the homography matrix score be SH, and let the fundamental matrix be F matrix and the fundamental matrix score be SF. The ratio RH = SH / (SH + SF) is calculated. If RH is greater than a first threshold RM, the image rotation matrix and translation vector are recovered using homography moments; if RH is less than the first threshold RM, the image rotation matrix and translation vector are recovered using the fundamental matrix. The first threshold RM can be 0.4, but other values ​​are also possible; this application is not limited to these values.

[0050] It should be noted that the purpose of map initialization is to calculate the relative pose between two frames, thereby triangulating a set of initial map points.

[0051] S104: Estimate the pose of the current frame image and optimize the pose of the current frame image using candidate frame images.

[0052] Figure 3 The specific steps for estimating the pose of the current frame image in step S104 provided in the embodiments of this application include steps S301 to S302.

[0053] S301: Fit the localization system coordinates of the current frame image and the previous frame image, and transform the pose of the image and map points to the object coordinate system.

[0054] S302: Find the corresponding set of map points and feature points between the current frame image and the previous frame image, and then estimate the pose of the current frame image.

[0055] Specifically, in order to make the initial map have a true physical scale, it is also necessary to fit the positioning system coordinates of the current frame and the previous frame. After fitting, the affected poses and map points are transformed to the object coordinate system to obtain the corresponding three-dimensional spatial coordinates.

[0056] Feature matching is performed between the current frame image and the previous frame image. After map initialization, the pose of the current frame image can be obtained by solving the black-and-white transformation algorithm (PNP) between the corresponding sets of 3D map points and 2D feature points. The black-and-white transformation algorithm is a method for solving the object motion localization problem from 3D points to 2D points. After determining the set p of feature points corresponding to the pose of the 3D map points and the current frame image, estimation is performed by minimizing the weighted squared residual function. Wherein, error e i The measurement value p defined as a map pointi Compared with the predicted value p i The difference between ' And weight ω i This can reduce the impact of large residuals.

[0057] Furthermore, Among them, f x ,f y Let x0 and y0 be the focal length of the camera model, and x0 and y0 be the coordinates of the principal image point. c ,Y c Z c Coordinates of the photography center.

[0058] Figure 4 The specific steps for optimizing the pose of the current frame image using candidate frame images in step S103 provided in the embodiments of this application include steps S401 to S405.

[0059] S401: Based on the matching relationship between the current frame image and the previous frame image, obtain multiple key frames that are closely related to the current frame image, so that multiple key frames correspond one-to-one with multiple map points.

[0060] Specifically, it is also necessary to calculate the matching relationship between multiple keyframes and the current frame. Since 10 relevant keyframes are just right for optimizing the computational load and accuracy of the current keyframe, 10 keyframes closely related to the current frame image can be obtained. Of course, other values ​​can also be used, and this application is not limited to the above values.

[0061] S402: Perform local bundling adjustments and optimizations on multiple keyframes and their corresponding map points.

[0062] Specifically, the Perspective-n-Point (PNP) algorithm can be used to locally optimize the pose of the current frame image. During local optimization, local keyframes and their corresponding map points are used as input. The pose of the local keyframes and map points are optimized by minimizing the weighted residuals of the keypoints. Where μ is the pose of the keyframe, P is the map point coordinates, and K... L For local keyframes, P L w represents the map point corresponding to the keyframe. ij To determine whether the i-th map point is visible in the j-th camera, the visibility w is... ij The value is 1, and w is invisible. ij The value is 0, e ij The image point error of a map point in the i-th frame and the j-th frame is given.

[0063] S403: Determine whether the distance between the current frame image and multiple keyframes is less than the second threshold since the last loop closure detection.

[0064] If the judgment result is yes, proceed to step S404: find candidate frame images with similarity not lower than the third threshold.

[0065] If the judgment result is negative, proceed to step S405: perform closed-loop detection.

[0066] Loop closure detection refers to the ability of a mobile robot to determine that a map has already been built when it reaches a previously mapped location, and then update and correct the existing map. The second threshold can be 50 meters, but other values ​​are also possible; this application is not limited to these values. It should be noted that when determining the distance from the last loop closure detection, it is assumed that at least one loop closure detection has already been performed. If the distance between the current frame image and multiple keyframes is less than 50 meters, loop closure detection is not performed. Instead, candidate frame images with a similarity of at least a third threshold are identified. This third threshold can be 0.6, but other values ​​are also possible; this application is not limited to these values. Candidate frame images are those that are not connected to the current frame image. After identifying candidate frame images with a similarity of at least the third threshold, the pose of the current frame image is optimized.

[0067] S105: Convert the coordinates of map points to two-dimensional coordinates and perform plane fitting on the converted map points.

[0068] Since the coordinates of the obtained map points are relative to the first image, a coordinate transformation is needed to determine the coordinates of the map points in the physical world.

[0069] Specifically, when the number of keyframe images on the map reaches a certain number, the coordinates of all map points can be transformed: P t =local2ECEF.inverse()*P t After the map points are replaced, a plane fitting is performed to find the object plane. The plane parameters are represented by se(3), and local2ECEF is the rotation and translation represented by se(3). The transformation of local2ECEF is the world coordinate value of the exposure position of the first frame image in the map.

[0070] S106: Transform the coordinates of the image corner points in the camera coordinate system to the object coordinate system, calculate the perspective transformation parameters, and perform geometric transformation on the image.

[0071] Figure 5 The specific steps for transforming the coordinates of the image corner points in the camera coordinate system to the object coordinate system in step S106 provided in the embodiments of this application include steps S501 to S502.

[0072] S501: Calculate the coordinates of the image corner points in the camera coordinate system, and transform the image corner points from the camera coordinate system to the pixel coordinate system and then to the image space coordinate system.

[0073] Because the final stitched image is stitched in the object plane, it is necessary to calculate the pose of the keyframe images to be stitched relative to the object plane. To perform spatial coordinate transformation of image points, a coordinate system describing the position of image points in image space needs to be established, i.e., the image space coordinate system. The image space coordinate system is a commonly used transition coordinate system used to represent the position of image points in image space. Specifically, coordinate transformation is for perspective transformation of the image. However, perspective transformation is the process of transforming the image from the image space coordinate system to the object space coordinate system. Therefore, it is necessary to know the coordinates of the image's corner points in the object space coordinate system, and then calculate the parameters between these coordinates and the image space coordinate system coordinates. Finally, the image is transformed based on these parameters.

[0074] S502: Rotate and scale the image corner points in the camera coordinate system and transform the image corner point coordinates to the object coordinate system.

[0075] Specifically, since the coordinates of the image corner points after rotation and scaling are relative to the current camera coordinate system, it is also necessary to transform the image corner point coordinates to the object coordinate system.

[0076] It should be noted that the image space coordinate system is used to describe the position of image points, while the object space coordinate system is used to describe the position of ground points. The transformation relationship between three-dimensional geographic coordinates and two-dimensional satellite image coordinates is generally referred to as the transformation from object space to image space.

[0077] Specifically, calculating perspective transformation parameters and performing geometric transformation on the image includes: using the pixel coordinates of the image corner points as source coordinates and the object coordinates as target coordinates, and performing geometric transformation on the image according to the perspective transformation parameters.

[0078] It should be noted that perspective is a type of geometric change. When taking an image, the actual scene on the ground and what is shown in the photograph are related by perspective. Therefore, correcting an image through perspective means correcting the image to the object plane.

[0079] Specifically, the corrected image needs to have projection information to fit with the geographic base map. This method uses the CGCS2000 Gaussian 3° zone projection, also known as the Gauss-Kruger coordinate system. The calculation method is: EPSG=((Lon-25*3) / 3)+1+4513, where Lon is the longitude of the center of the survey area.

[0080] Furthermore, the coordinates are calculated using the following method:

[0081] The top-left X-coordinate is: X = plane.getTranslation.x() + Xmin + x.

[0082] The top-left Y-coordinate is: Y = plane.getTranslation.y() + Ymax - y.

[0083] Where plane is the object plane represented by se(3), Xmin and Ymax are the coordinates of the upper left corner of the survey area, and x and y are the offsets of the current frame corrected image relative to the first frame image.

[0084] S107: Perform tile conversion on the image and then perform Gaussian pyramid fusion on the generated tiles.

[0085] A tile is a square raster image of a map within a certain area, cut into several rows and columns according to a certain size and format, and according to the zoom level or scale. The resulting square raster image is figuratively called a tile.

[0086] After the preceding processing, the images have been aligned and fitted to the plane. To address the issue that the large data volume of a single corrected image is unsuitable for loading into a geographic information system, the images need to be tiled. A set of tile datasets is generated by cutting the images according to certain rules, and the images are visualized using the tiles as the transmission objects.

[0087] Figure 6 The specific steps for performing tiled processing on the image in step S107 provided in the embodiments of this application include steps S601 to S603.

[0088] S601: Obtain the projection coordinates of the image.

[0089] S602: Convert the projected coordinates of the image to Web Mercator coordinates or WGS84 coordinates.

[0090] Since the input image is in Gauss-Kruger coordinate system, but the final result of cutting the tiles requires Web Mercator coordinates or WGS84 coordinates, a time coordinate transformation is required for the input image.

[0091] S603: Start cutting from the top layer of tiles and generate the next layer of tiles in sequence.

[0092] Specifically, the top layer of tiles refers to the lowest, highest-ranking tile, while the lower layers of tiles, starting from the top layer, are all formed by reducing the size of the upper layers.

[0093] The specific method for Gaussian pyramid fusion of the generated tiles in step S107 is as follows: Due to the large overlap between two adjacent frames of the UAV, the cut tiles also overlap. When new corrected images generate tiles, image fusion of the overlapping tile texture content is required. Gaussian pyramid fusion can be performed using the Laplacian pyramid method; of course, other methods can also be chosen, and this application is not limited to the above methods.

[0094] Specifically, the Laplacian pyramid minimizes image exposure differences and geometric misalignments, and through a fusion operation, yields tile images with continuous brightness and clear details. First, a k-order Gaussian pyramid is calculated, then... I-1 Level pyramid minus G l The pyramids can then be used to obtain the corresponding L-level Laplace pyramid: L l =G l -G l+1 , l < k, where 2n-1 is the size of the Laplace kernel, the sum of Wdx and Wdy is 1, and the highest level of the Laplace pyramid is LK and the highest level of the Gaussian pyramid is G. k Same as above, the default level k is 5.

[0095] Figure 7 This application provides a first display result of overlaying a quick-paste image with a geographic map, as shown in the embodiments of this application. Figure 8 The second display result is a completed quick-paste image overlaid with a geographic map, as provided in the embodiments of this application.

[0096] This application also provides an image stitching device 900 capable of real-time display, such as... Figure 9 As shown, the device includes: a receiving module 901, a feature point matching module 902, a calculation module 903, an optimization module 904, a plane fitting module 905, a geometric transformation module 906, and a fusion module 907.

[0097] The receiving module 901 is used to receive images uploaded to the cloud by the camera device in real time.

[0098] The feature point matching module 902 is used to extract feature points of the image and perform feature point matching between the current frame image and the previous frame image of the image.

[0099] The calculation module 903 is used to calculate the matrix scores of the homography matrix and the fundamental matrix respectively, and to determine whether to use the homography matrix or the fundamental matrix to recover the pose parameters and map points of the image by calculating the ratio of the homography matrix score to the sum of the homography matrix score and the fundamental matrix score, and to initialize the map; wherein, the pose parameters include the rotation matrix and the translation vector.

[0100] The optimization module 904 is used to estimate the pose of the current frame image and optimize the pose of the current frame image using candidate frame images.

[0101] The plane fitting module 905 is used to convert the coordinates of map points into two-dimensional coordinates and perform plane fitting on the converted map points.

[0102] The geometric transformation module 906 is used to transform the coordinates of image corner points in the camera coordinate system to the object coordinate system, calculate perspective transformation parameters, and perform geometric transformations on the image.

[0103] The fusion module 907 is used to perform tile-based processing on the image and to perform Gaussian pyramid fusion on the generated tiles.

[0104] The apparatus or module described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. For ease of description, the above apparatus is described by dividing it into various modules according to their functions. In implementing this application, the functions of each module can be implemented in one or more software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0105] The methods, apparatus, or modules described in this application can be implemented in a computer-readable program code manner. The controller can be implemented in any suitable manner, such as a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of a memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code manner, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included within it for implementing various functions can also be considered as structures within the hardware component. Alternatively, the device used to implement various functions can be viewed as either a software module that implements the method or a structure within a hardware component.

[0106] Some modules in the apparatus described in this application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0107] This application also provides an image stitching server capable of real-time display, such as... Figure 10 As shown, it includes a memory 1001 and a processor 1002; the memory 1001 is used to store computer-executable instructions; the processor 1002 is used to execute the computer-executable instructions to implement the real-time display image fast stitching method provided in the embodiments of this application.

[0108] This application also provides a computer-readable storage medium storing executable instructions. When a computer executes the executable instructions, it can implement the real-time display image fast stitching method provided in this application.

[0109] The aforementioned storage media include, but are not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), Cache, Hard Disk Drive (HDD), or Memory Card. The memory can be used to store computer program instructions.

[0110] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, or it can be embodied in the process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0111] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0112] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of this application.

Claims

1. A method for real-time image stitching, characterized in that, include: Receive images uploaded to the cloud in real time by the camera device; Extract feature points from the image and perform feature point matching between the current frame and the previous frame of the image; The matrix scores of the homography matrix and the fundamental matrix are calculated separately. The image pose parameters and map points are recovered using either the homography matrix or the fundamental matrix by calculating the ratio of the homography matrix score to the sum of the homography matrix score and the fundamental matrix score. The map is then initialized. The pose parameters include rotation matrix and translation vector. Estimate the pose of the current frame image and optimize the pose of the current frame image using candidate frame images; Convert the coordinates of map points to two-dimensional coordinates, and then perform plane fitting on the converted map points; Transform the coordinates of the image corner points in the camera coordinate system to the object coordinate system, calculate the perspective transformation parameters, and perform geometric transformation on the image; The image is tiled, and the generated tiles are then fused using a Gaussian pyramid.

2. The method according to claim 1, characterized in that, The step of determining whether to use the homography matrix or the fundamental matrix to recover the pose parameters and map points of the image by calculating the ratio of the homography matrix score to the sum of the homography matrix score and the fundamental matrix score includes: Determine whether the calculated ratio is greater than a first threshold; if the determination result is yes, use the homography matrix to recover the pose parameters and map points of the image; if the determination result is no, use the base matrix to recover the pose parameters and map points of the image.

3. The method according to claim 1, characterized in that, The estimation of the pose of the current frame image includes: Fit the localization system coordinates of the current frame image and the previous frame image, and transform the pose of the image and map points to the object coordinate system; Find the corresponding set of map points and feature points between the current frame image and the previous frame image, and then estimate the pose of the current frame image.

4. The method according to claim 1, characterized in that, The optimization of the pose of the current frame image using candidate frame images includes: Based on the matching relationship between the current frame image and the previous frame image, multiple key frames closely related to the current frame image are obtained, so that the multiple key frames and multiple map points have a one-to-one correspondence. Perform local bundling adjustments and optimizations on multiple keyframes and their corresponding map points; Determine whether the distance between the current frame image and multiple key frames is less than the second threshold since the last loop closure detection; if the determination result is yes, find candidate frame images with similarity not lower than the third threshold; if the determination result is no, perform loop closure detection.

5. The method according to claim 1, characterized in that, The transformation of the coordinates of the image corner points from the camera coordinate system to the object coordinate system includes: Calculate the coordinates of the image corner point in the camera coordinate system, and transform the image corner point from the camera coordinate system to the pixel coordinate system and then to the image space coordinate system; The image corner points are rotated and scaled in the camera coordinate system, and the coordinates of the image corner points are transformed to the object coordinate system.

6. The method according to claim 1, characterized in that, The calculation of perspective transformation parameters and the geometric transformation of the image include: The pixel coordinates of the image corner points are used as source coordinates, and the object coordinates are used as target coordinates.

7. The method according to claim 1, characterized in that, The step of tile-based processing of the image includes: Obtain the projection coordinates of the image; Convert the projected coordinates of the image to Web Mercator coordinates or WGS84 coordinates; Start cutting from the top layer of tiles and then proceed to create the next layer of tiles.

8. A real-time image stitching device, characterized in that, include: The receiving module is used to receive images uploaded to the cloud by the camera device in real time; The feature point matching module is used to extract feature points of the image and perform feature point matching between the current frame image and the previous frame image of the image. The calculation module is used to calculate the matrix scores of the homography matrix and the fundamental matrix respectively, and to determine whether to use the homography matrix or the fundamental matrix to recover the pose parameters and map points of the image by calculating the ratio of the homography matrix score to the sum of the homography matrix score and the fundamental matrix score, and to initialize the map; wherein, the pose parameters include rotation matrix and translation vector; The optimization module is used to estimate the pose of the current frame image and optimize the pose of the current frame image using candidate frame images. The plane fitting module is used to convert the coordinates of map points into two-dimensional coordinates and perform plane fitting on the converted map points. The geometric transformation module is used to transform the coordinates of image corner points in the camera coordinate system to the object coordinate system, calculate perspective transformation parameters, and perform geometric transformation on the image. The fusion module is used to perform tile-based processing on the image and to perform Gaussian pyramid fusion on the generated tiles.

9. A real-time image stitching server, characterized in that, Including memory and processor; The memory is used to store computer-executable instructions; The processor is configured to execute the computer-executable instructions to implement the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores executable instructions, which, when executed by a computer, enable the implementation of the method as described in any one of claims 1-7.

Images