UAV panoramic image stitching method based on UAV mother vehicle

Abstract

The present invention provides a method for stitching panoramic images of drones based on a drone mother vehicle, comprising: the drone mother vehicle acquires images taken by a drone; cylindrically projects the acquired images to acquire cylindrical projection images; performs corner point detection on the cylindrical projection images to acquire corner points; performs feature extraction and feature matching on the cylindrical projection images based on the acquired corner points to acquire feature matching point pairs of the stitched image; and stitches and fuses the cylindrical projection images based on the extracted feature matching points to form a panoramic image. In the technical solution provided by the present invention, the drone mother vehicle can provide real-time imaging when stitching panoramic images, and by improving the stitching algorithm, the stitching effect of the panoramic image can be improved without increasing the computational complexity.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) panoramic image stitching, and in particular to a human-machine panoramic image stitching method based on an unmanned aerial vehicle (UAV) mother vehicle. Background Art

[0002] With recent technological advancements, drone technology has matured, with advancements in inertial navigation control, high-precision attitude sensors, and small servo motors, achieving significant improvements in automation and intelligence. However, collecting evidence requires images from multiple perspectives to provide a more comprehensive and accurate picture. These images must be stitched together to present compelling evidence to users. However, in this new era, panoramic stitching algorithms are complex and computationally intensive, making real-time processing difficult. Summary of the Invention

[0003] Based on the defects in the prior art, the present invention provides a method for stitching panoramic images of drones based on a drone mother vehicle, comprising: the drone mother vehicle acquires images taken by the drone;

[0004] Perform cylindrical projection on the acquired image to obtain a cylindrical projection image; perform corner point detection on the cylindrical projection image to obtain corner points; perform feature extraction and feature matching on the cylindrical projection image based on the obtained corner points to obtain feature matching point pairs of the spliced ​​image;

[0005] The cylindrical projection images are stitched and fused according to the extracted feature matching points to form a panoramic image.

[0006] Wherein, binarizing the cylindrical projection image to obtain a binarized image;

[0007] Perform contour detection on the binary image according to the adaptive threshold and obtain the detected contour coordinates;

[0008] According to the contour coordinates, the image of the area surrounded by the contour coordinates is intercepted in the cylindrical projection image to form a new cylindrical projection image;

[0009] The best matching point pair is obtained from the feature matching point pairs, and the image is stitched using the best matching point pair.

[0010] A method for stitching panoramic images of drones based on a drone mother vehicle, further preferably performing end-to-end correction on the stitched panoramic images; and cropping the corrected panoramic images to obtain a final panoramic image.

[0011] A method for stitching panoramic images of a drone based on a drone mother vehicle, further preferably, preprocessing the image before performing cylindrical projection;

[0012] The preprocessing includes at least: obtaining the camera pose and drone pose corresponding to each image, constructing an orthographic projection transformation matrix; and converting the image into an orthographic projection direction through the orthographic projection transformation matrix.

[0013] A method for stitching panoramic images of drones based on a drone mother vehicle, further preferably, corner point detection at least includes:

[0014] Obtain the coordinates and eigenvalues ​​of each corner point to form the first corner point matrix;

[0015] Set the filtering conditions, filter the first corner point matrix, delete the corner points that do not meet the conditions, and form the second corner point matrix;

[0016] Set a first window of fixed size, traverse the second corner point matrix with the first window, and if the sum of the eigenvalues ​​of the corner points corresponding to all coordinates in the window area is not 0, obtain the coordinate value corresponding to the corner point with the largest eigenvalue in the window area;

[0017] With the corner point of the maximum eigenvalue as the center, the area enclosed by the first window is formed, the coordinates feature_coordinate corresponding to the enclosed area are obtained, and the pixel values ​​of the corresponding area in the cylindrical projection image are intercepted by the corresponding coordinates and recorded as feature_value.

[0018] A method for stitching panoramic images of a drone based on a drone mother vehicle, further preferably, obtaining the first corner point matrix at least includes:

[0019] Get the intensity factors Dx, Dy, and Dxy in the x-direction, y-direction, and xy-direction;

[0020] Perform preset rule filtering on the obtained intensity factors Dx, Dy, and Dxy;

[0021] The preset rule filtering includes: calculating the sum of Dx or Dy or Dxy of all points in the preset window area, and if the sum of the values ​​exceeds 255, setting Dx or Dy or Dxy of the center point of the window area to 255; traversing the entire image in sequence to obtain the filtered intensity factors Dx', Dy', Dxy';

[0022] The eigenvalues ​​of the candidate corner points are calculated using the intensity factor. When the eigenvalues ​​of the candidate corner points are greater than a preset threshold, the coordinates and eigenvalues ​​of the corner points are recorded to form a first corner point matrix.

[0023] A drone panoramic image stitching method based on a drone mother vehicle, further preferably, the filtering conditions for the first corner point matrix include:

[0024] Compare the corner point eigenvalues ​​in the first corner point matrix with the preset threshold, obtain all corner points greater than the preset threshold, and then set the corner point eigenvalues ​​less than the preset threshold to 0; set the corner point eigenvalues ​​in the preset edge area to 0 to form a second corner point matrix.

[0025] A method for stitching drone panoramic images based on a drone mother vehicle, further preferably, obtaining feature matching point pairs includes:

[0026] Extract features from cylindrical projection images through corner points;

[0027] For the adjacent i-th image and i+1-th image, take the yi coordinate of the corner point feature value of the i-th image as the comparison benchmark, traverse the square area surrounded by the yi coordinate and the preset length C4 of the corner point feature value in the i+1-th image, and obtain all corner points that meet the conditions and record them as candidate feature points;

[0028] Calculate the similarity distances between the feature points in the i-th image and the corresponding candidate feature points to form a first distance matrix;

[0029] Obtain the first two elements dmin and dmin1 with the smallest similarity distance in the first matrix, compare dmin with dmin1, and if the second preset condition is met, the feature point of the i-th image and the candidate feature point of the i+1-th image corresponding to the corresponding element of the first distance matrix are mutually matched feature points.

[0030] A drone panoramic image stitching method based on a drone mother vehicle, further preferably, for the stitching of the i-th image and the i+1-th adjacent image, traversing the i-th image along the y-axis with the yj value of the feature coordinate feature_coordinate;

[0031] If the value of the y coordinate of the corresponding feature point in the first feature coordinate matrix in the i+1 image is in the range of [yj-range,yj+range], then the distance between the feature_value value corresponding to the feature point coordinate feature_coordinate in the i-th image and the corresponding feature_value values ​​in the coordinate feature_coordinate of all feature points in the i+1-th image is calculated to form a first distance matrix.

[0032] A method for stitching drone panoramic images based on a drone mother vehicle, further preferably, obtaining mutually matching feature points includes:

[0033] Sort the elements in the first distance matrix from large to small, and obtain the minimum value dmin and the second minimum value dmin1 according to the sorting result;

[0034] If dmin / dmin1<=C1, the coordinates of the feature points in the i-th image and the corresponding coordinates of the feature points in the i+1-th image are stored to form elements in the first pairing matrix;

[0035] All feature points are traversed in sequence, and the coordinates of all matching feature points in the i-th image and the i+1-th image are obtained to form a final first pairing matrix. The first pairing matrix stores the coordinates corresponding to the matching feature points in the i-th image and the i+1-th image.

[0036] A method for stitching panoramic images of a drone based on a drone mother vehicle, further preferably, the feature matching points of the stitched image are concentrated on calculating the best matching point pair in the y-axis direction, and obtaining the best matching point pair includes:

[0037] Get the first pairing matrix. Each element in the first pairing matrix contains matching feature points. Calculate the shifted_value value. Then, take the first matching point from all elements of the first pairing matrix and subtract the shifted_value from each of them to obtain n diff_values. The set of n different diff_values ​​is recorded as difference.

[0038] Traverse all elements of the first pairing matrix in turn, find the mutually matching feature points that make inpoint obtain the maximum value, and define them as the best matching feature points. The shift_value corresponding to the best matching feature point is the best_shift_value.

[0039] A drone panoramic image stitching method based on a drone mother vehicle, further preferably, shifted_value is obtained including:

[0040] Take the first element in the first pairing matrix. The first element includes the coordinates of a pair of matching points, which are recorded as the first matching point and the second matching point. Subtract the coordinates of the first matching point from the coordinates of the second matching point in the matching point pair, and the obtained value is recorded as shift_value.

[0041] Then, all elements are taken from the first pairing matrix, and all second matching points are taken from all elements and subtracted from each other by shift_value. The set of obtained results is recorded as shifted_value.

[0042] A method for stitching panoramic images of a drone based on a drone mother vehicle, further preferably, the mutually matching feature points corresponding to the maximum value of inpoint include at least:

[0043] Set inpoint = 0, best_shift = 0. Take an element diff_value from difference, calculate the sum of squares of all values in diff_value. If the sum of squares is less than the preset first threshold, increment inpoint by 1. After traversing all diff_values in sequence, record inpoint. If the inpoint value is greater than the preset second threshold, assign the inpoint value to the preset second threshold, and assign the corresponding shift_value to best_shift_value;

[0044] Traverse all elements of the first pairing matrix in sequence to obtain the final best_shift_value;

[0045] Among them, best_shift_value is the coordinate difference obtained by subtracting the first matching point from the second matching point in the best matching point pair between the i-th image and the (i + 1)-th image.

[0046] A method for stitching panoramic images of unmanned aerial vehicles based on an unmanned aerial vehicle mother vehicle. Further preferably, during panoramic image stitching, calibrate the coordinates of adjacent images through best_shift_value, adjust the coordinate points with common areas in adjacent images so that the common coordinate points are in the same reference coordinate system, and perform image fusion on adjacent images.

[0047] A method for stitching panoramic images of unmanned aerial vehicles based on an unmanned aerial vehicle mother vehicle. Further preferably, the stitching and fusion of cylindrical projection images include:

[0048] Preset a window of a fixed size, traverse along the y-axis direction of the image, and perform stitching and fusion on adjacent images according to the value of the abscissa x;

[0049] When W / 2 – C2 <= x <= W / 2 + C2, the pixel value of the corresponding coordinate is calculated by the following formula:

[0050] Proportion = (x – seam + C2) / C2 ,

[0051] , ,

[0050] , ,

[0054] ,

[0049] ,

[0053] , , 2 ,

[0052] , ,

[0051] I(x, y) = (1 – Proportion)*img2[x, y] + Proportion*img1[x, y]

[0052] When x < W / 2 – C2, the pixel value of the corresponding coordinate I(x, y) = img1[x, y]

[0053] When x > W / 2 + C2, the pixel value of the corresponding coordinate I(x, y) = img2[x, y]

[0054] Where W is the width of the img1 image, C2 is the preset threshold, img1 and img2 are the images to be stitched; (x, y) is the horizontal coordinate of the images to be stitched.

[0055] A method for stitching panoramic images of drones based on a drone mother vehicle, further preferably, when the drone mother vehicle establishes a communication connection with the drone;

[0056] When the drone mother vehicle starts the panoramic photography command and sends it to the drone, the drone calls the panoramic photography program to take panoramic photos and sends the images for stitching to the drone mother vehicle frame by frame in the stitching order;

[0057] After the drone mother vehicle completes the panoramic image stitching, it marks the geographic location of the stitched panoramic image based on the current drone GPS information and posture information; and sends the panoramic image to the on-board Ethernet display for display.

[0058] Beneficial effects:

[0059] 1. In the technical solution provided by this invention, drones capture images and transmit them in real time to a drone carrier. The carrier stitches the received images into a panoramic image in real time, which is then transmitted to the vehicle's Ethernet display for display. Leveraging the carrier's powerful operational capabilities, this panoramic image stitching is accomplished in real time, facilitating drones in target search and tracking missions.

[0060] 2. In the technical solution provided by the present invention, when stitching panoramic images, by improving the stitching algorithm, the stitching effect of the panoramic image can be improved without increasing the computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The following drawings are only used to schematically illustrate and explain the present invention and are not intended to limit the scope of the present invention.

[0062] Figure 1 Schematic diagram of the architecture for stitching panoramic images of a drone mother vehicle and a drone according to an embodiment of the present invention.

[0063] Figure 2 This is a flow chart of a method for implementing panoramic image stitching on a drone mother vehicle in one embodiment of the present invention.

[0064] Figure 3 4 is a flow chart of a method for adaptively cropping a panoramic image in panoramic image stitching according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] To provide a clearer understanding of the technical features, objectives, and effects of this document, specific embodiments of the present invention are now described with reference to the accompanying drawings. Like reference numerals in the various figures represent like parts. To simplify the drawings, the various figures schematically illustrate parts relevant to the present invention and do not represent the actual structure of the product. Furthermore, to simplify the drawings and facilitate understanding, in some figures, only one of the components with the same structure or function is schematically depicted or labeled.

[0066] Regarding control systems, functional modules, and application programs (APPs), it is well known to those skilled in the art that they can take any appropriate form, either hardware or software, either multiple functional modules set discretely or multiple functional units integrated into one hardware. In the simplest form, the control system can be a controller, such as a combinational logic controller, a microprogram controller, etc., as long as the operations described in this application can be implemented. Of course, the control system can also be integrated into a physical device as different modules, which does not deviate from the basic principles and protection scope of the present invention.

[0067] The term "connection" in the present invention may include direct connection, indirect connection, communication connection, and electrical connection, unless otherwise specified.

[0068] The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that when used in the specification, the terms "include" and / or "comprise" refer to the presence of stated features, values, steps, operations, elements and / or components, but do not exclude the presence or additional addition of one or more other features, values, steps, operations, elements, components and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the listed related items.

[0069] It should be understood that the term "vehicle" or "vehicular" or other similar terms used herein generally include motor vehicles, such as passenger cars including sport utility vehicles (SUVs), buses, trucks, various commercial vehicles, watercraft including various boats, ships, aircraft, etc., and include hybrid vehicles, electric vehicles, plug-in hybrid electric vehicles, hydrogen-powered vehicles, and other alternative fuel vehicles (e.g., fuels derived from energy sources other than petroleum). As referred to herein, a hybrid vehicle is a vehicle having two or more power sources, such as both gasoline-powered and electric-powered vehicles.

[0070] Furthermore, the controller of the present disclosure may be embodied as a non-transitory computer-readable medium on a computer-readable medium containing executable program instructions executed by a processor, controller, or the like. Examples of computer-readable media include, but are not limited to, ROM, RAM, compact disc (CD)-ROMs, magnetic tapes, floppy disks, flash drives, smart cards, and optical data storage devices. The computer-readable recording medium may also be distributed among computer systems coupled via a network, such that the computer-readable medium is stored and executed in a distributed manner, for example, via a telematics server or a controller area network (CAN).

[0071] The present invention provides a UAV panoramic image stitching system based on a UAV mother vehicle, see Figure 1 As shown in Figure 3, it specifically includes at least a drone and a drone mother vehicle. The drone mother vehicle is equipped with a TSN gateway, and the drone and the drone mother vehicle communicate through the TSN gateway;

[0072] When the UAV mother vehicle establishes a communication connection with the UAV;

[0073] When the drone mother vehicle starts the panoramic photography command and sends it to the drone, the drone calls the panoramic photography program to take panoramic photos and sends the images for stitching to the drone mother vehicle frame by frame in the stitching order;

[0074] After receiving the image, the UAV mother vehicle calls the panoramic stitching algorithm to stitch the received images to form a panoramic image;

[0075] Mark the geographic location of the panoramic image based on the current drone GPS information and posture information;

[0076] The panoramic image is sent to the in-vehicle Ethernet display for display.

[0077] By setting the characteristics of the tracking target, the target is searched in the panoramic image.

[0078] The present invention also provides a method for stitching panoramic images of drones based on drone mother vehicles, such as Figure 2 As shown, it specifically includes one or all of the following steps:

[0079] Acquire images captured by drones;

[0080] Performing cylindrical projection on the acquired image to obtain a cylindrical projection image;

[0081] Perform corner detection on the cylindrical projection image to obtain corner points;

[0082] Perform feature extraction and feature matching on the cylindrical projection image based on the acquired corner points;

[0083] The cylindrical projection images are stitched together based on the extracted feature matching to form a panoramic image;

[0084] Perform end-to-end rectification on panoramic images;

[0085] The aligned panoramic image is cropped to obtain the final panoramic image.

[0086] Since drones fly in the sky, they are restricted by the complex aerial environment and are affected by wind and light, which can cause large differences in the captured images. Therefore, the images need to be pre-processed before stitching. The pre-processing step is before cylindrical projection;

[0087] The preprocessing includes at least: obtaining the camera pose and drone pose corresponding to each image, and constructing the orthographic projection transformation matrix;

[0088] Convert the image into the orthographic projection direction through the orthographic projection conversion matrix;

[0089] In this embodiment, the orthographic projection direction is defined as the camera shooting angle in the drone vertically downward;

[0090] Specifically, due to the differences in camera performance and the different depths of field, feature point extraction will be affected, resulting in reduced registration accuracy. To solve this technical problem, this embodiment performs cylindrical projection on the image. The specific steps of cylindrical projection include:

[0091] Get the image and the focal length corresponding to the image;

[0092] Construct a cylindrical projection relationship formula, and convert the image into a cylindrical projection image through the cylindrical projection relationship formula;

[0093] The precise position of each pixel is calculated through bilinear interpolation to obtain the projected image.

[0094] The projection relationship is:

[0095]

[0096]

[0097] Where u, v are the pixel coordinates of the projected image, x, y are the pixel coordinates of the image before projection, f is the focal length of the camera, width and height are the width and height of the image before projection respectively;

[0098] Specifically, based on the characteristics of drone imaging, the applicant discovered that after cylindrical projection, many outlier points are generated at the edges. These outlier points can affect subsequent corner point detection. Therefore, the cylindrical projection image is binarized using an adaptive threshold, and contour detection is performed to obtain contour coordinates. Based on these contour coordinates, a new cylindrical projection image is generated, eliminating the outlier points.

[0099] Specifically include:

[0100] Binarizing the cylindrical projection image to obtain a binary image;

[0101] Perform contour detection on the binary image according to the adaptive threshold and obtain the detected contour coordinates;

[0102] According to the contour coordinates, the area image surrounded by the contour coordinates is intercepted in the cylindrical projection image to form a new cylindrical projection image.

[0103] In existing technologies, image stitching primarily uses the SIFT and SUFT algorithms for feature extraction. While these algorithms can achieve good results, they are computationally intensive and cannot capture real-time panoramic images. To address this technical issue, this embodiment employs the following steps without compromising stitching quality.

[0104] One of the specific steps is:

[0105] If the image is a color image, convert it into a grayscale image;

[0106] Use Sobel or Canny gradient operator to calculate the gradient of each point in the grayscale image in the x and y directions, and obtain Ix and Iy of each point in the x and y directions;

[0107] Get the intensity factors in the x-direction, y-direction, and xy-direction, defined as Dx, Dy, Dxy;

[0108] Where Dx = Ix 2 , Dy=Iy 2 ,Dxy=Ix*Iy;

[0109] Filter the obtained intensity factors Dx, Dy, Dxy;

[0110] In the prior art, filtering by using mean filtering, median filtering and Gaussian filtering has a denoising effect, but noise still exists. In order to completely remove the noise and reduce interference, this embodiment improves the traditional filtering as follows:

[0111] In a preset window area, such as a 5*5 area, calculate the sum of all pixel values ​​in the window area. If the sum of the pixel values ​​exceeds 255, set the center pixel to 255. In this way, traverse the entire image to obtain the filtered intensity factors Dx', Dy', and Dxy'.

[0112] For each point in the image, construct Get R = det(M) - k*trace(M) 2 , when R is greater than the preset threshold, the point is saved as a corner point, and R is the feature value of the corner point.

[0113] Get the coordinates and eigenvalues ​​of each corner point to form the first corner point matrix.

[0114] Specifically, many corner points are obtained through corner point detection, but not every corner point is a true corner point, and there are still errors in the positions of corner points. To improve accuracy and obtain true corner points, this embodiment provides a solution, which specifically includes:

[0115] Compare the corner point eigenvalues ​​in the first corner point matrix with a preset threshold, obtain all corner points whose eigenvalues ​​are greater than the preset threshold, and then set the corner point eigenvalues ​​corresponding to the corner points whose eigenvalues ​​are less than the preset threshold to 0; set the corner point eigenvalues ​​within the preset edge area to 0 to form a second corner point matrix;

[0116] Set a first window of fixed size, traverse the second corner point matrix with the first window, and if the sum of the corner point eigenvalues ​​corresponding to all coordinates in the window area is not 0, obtain the coordinate value of the maximum corner point eigenvalue in the window area;

[0117] The area enclosed by the first window is formed with the corner point with the maximum corner point eigenvalue as the center, and the coordinates feature_coordinate corresponding to the enclosed area are obtained. The pixel values ​​are obtained at the corresponding coordinates on the cylindrical projection image, which are recorded as feature_value;

[0118] All feature_coordinate and feature_value are combined into the corresponding first feature coordinate matrix and first eigenvalue matrix.

[0119] Specifically, when obtaining the first eigencoordinate matrix and the first eigenvalue matrix, it is necessary to pair the feature points in adjacent images. For example, if the feature coordinates in the first image are (a1, b1), (a2, b2), (a3, b3), and the feature coordinates in the second image are (c1, d1), (c2, d2), (c3, d3), (c4, d4), it is necessary to determine whether the coordinates in the first image match the feature coordinates in the second image.

[0120] Specifically, the pairing in this embodiment adopts the following scheme:

[0121] In order to reduce the amount of calculation, in this embodiment, only the y direction is considered, and the similarity of all feature values ​​of the corresponding regions corresponding to the feature coordinates of adjacent images is calculated;

[0122] For the splicing of the i-th image and the i+1-th adjacent image, traverse the i-th image along the y-axis with the yj value of the feature coordinate feature_coordinate, and the corresponding value in the i+1 image is within the range of [yj-range,yj+range];

[0123] If the y-coordinate value of the corresponding feature point in the first feature coordinate matrix in the i+1 image is within the range [yj-range,yj+range], then calculate the distance between the feature_value value corresponding to the feature point coordinate feature_coordinate in the i-th image and the corresponding feature_value values ​​in the coordinate feature_coordinate of all feature points in the i+1-th image to form a first distance matrix. ;

[0124] Sort the elements in the first distance matrix from large to small, and obtain the minimum value dmin and the second minimum value dmin1 according to the sorting result. If dmin / dmin1<=C1, the value range of C1 is 0.4~0.7;

[0125] When dmin / dmin1<=C1, the coordinates of the feature points in the i-th image and the corresponding coordinates of the feature points in the i+1-th image are stored to form elements in the first pairing matrix;

[0126] Traverse all feature points in sequence, obtain all coordinates in the i-th image that match the i+1-th image, and form the final first pairing matrix;

[0127] The first pairing matrix stores the coordinates of the feature points that match each other in the i-th image and the i+1-th image;

[0128] Specifically, this embodiment uses corner point pairing for initial feature points. Unlike the SIFT feature point extraction method used in the prior art, there are multiple corner points. In theory, all paired points can be perfectly matched, but in practice, noise and computational errors can cause individual matches to vary. Conventional methods use the point with the minimum mean error as the optimal match. However, this approach requires multiple iterations and is computationally intensive. To address this issue, this embodiment employs the following method:

[0129] In obtaining the first pairing matrix, take out the first element in the first pairing matrix. The first element includes a pair of matching points, which are recorded as the first matching point and the second matching point. Subtract the coordinates of the second matching point from the coordinates of the first matching point in the matching point pair, and the obtained value is recorded as shift_value;

[0130] Then, take all elements from the first pairing matrix, take the coordinates of all second matching points from all elements and subtract shift_value from them respectively, and record the obtained result set as shifted_value; then take all first matching points from all elements and subtract shifted_value from them respectively, and obtain n diff_values ​​respectively. The set of n different diff_values ​​is recorded as difference;

[0131] Assume inpoint = 0, best_shift = 0, and take an element from difference, where each element includes a diff_value, and diff_value includes multiple values. Calculate the sum of the squares of all the values ​​in diff_value. If the sum of the squares is less than the threshold, then increase inpoint by 1. After traversing all diff-values ​​in sequence, record inpoint. If the inpoint value is greater than the preset second threshold, then assign the inpoint value to the preset second threshold, and assign the corresponding shift-value value to best_shift_value.

[0132] Traverse all elements of the first pairing matrix in turn, repeat the above calculation, and obtain the final best_shift_value.

[0133] Wherein, best_shift_value is the coordinate difference between the second matching point and the first matching point in the best matching point pair between the i-th image and the (i+1)-th image.

[0134] Specifically, when the best matching point pair is obtained, the best_shift_value is obtained by the best matching point to stitch the images;

[0135] Specifically, there is a best_shift_value between adjacent images. If there are 6 spliced ​​images, there are 5 corresponding best_shift_values.

[0136] During stitching, the coordinates of adjacent images are calibrated by best_shift -- value. By unifying the coordinate system, the coordinate points with common areas in adjacent images are adjusted to be the same, and then image fusion is performed in adjacent images.

[0137] When performing image fusion, weighted mean fusion is usually adopted in the prior art, that is, fusion is performed according to the distance ratio of pixel points from the dividing line. This is a relatively simple fusion method, and there are relatively serious color differences in the fused image. In order to solve the color difference and without increasing the computational complexity, this embodiment provides an improved method, which specifically includes:

[0138] Preset a fixed size, traverse the image along the y-axis direction, and splice and fuse adjacent images according to the value of the abscissa x;

[0139] When W / 2–C2 <= x <= W / 2+C2, the pixel value of the corresponding coordinate is calculated by the following formula:

[0140] Proportion=(x–seam+C2) / C2 2

[0141] I(x,y)=(1–Proportion)*img2[x,y]+Proportion*img1[x,y]

[0142] When x < W / 2–C2, the pixel value I(x,y) of the corresponding coordinate = img1[x,y]

[0143] When x > W / 2+C2, the pixel value I(x,y) of the corresponding coordinate = img2[x,y]

[0144] Where, W is the width of the img1 image, C2 is a preset threshold, img1 and img2 are the images to be spliced; (x,y) is the abscissa of the image to be spliced.

[0145] Specifically, adjacent images are spliced and fused in sequence to form a panoramic image;

[0146] Since there are multiple synthetic images, the synthesis of multiple images will cause the existence of cumulative errors. In this embodiment, only the similarity in the y direction is considered during synthesis. Therefore, multiple images in the y direction will cause too large cumulative errors and artifacts. In order to eliminate the artifacts, this embodiment performs artifact correction on the spliced panoramic image, which specifically includes:

[0147] Obtain the best_shift-value of adjacent images in all spliced images, and then accumulate all the best_shift-values in the x direction and y direction

[0148] to obtain sum_x and sum_y;

[0149] If sum_x*sum_y<=0, then take 0 as the starting point and the absolute value of sum_y as the end point, and obtain multi-directional adjustment values ​​Zq equal to the panoramic image width Wq at equal intervals, that is, obtain Wq multi-directional adjustment values ​​Zq;

[0150] Zqi=i*[(abs(sum_y)–0) / Wq]

[0151] Then, the pixels in the X direction are gradually scrolled upward by a displacement of Zqi in the Y direction in units of pixels with a step of 1, and the panoramic image is traversed in sequence to obtain a panoramic image after the artifacts are corrected.

[0152] Specifically, due to the use of multiple images for stitching, in the case of cylindrical projection and edge adjustment strategy in this embodiment, irregular black edges will be generated at the upper and lower edges of the image along the y-axis direction. In order to remove the black edges and keep the integrity of the stitched image as much as possible, this embodiment provides an adaptive irregular black edge removal method, such as Figure 3 As shown, specifically including:

[0153] Convert the acquired panoramic image into a grayscale image; binarize the grayscale image, and preset the comparison value threshold as C3;

[0154] When the pixel value of the grayscale image is greater than C3, the pixel value of the grayscale image is set to 255. When the pixel value of the grayscale image is less than C3, the pixel value of the grayscale image is set to 0 to obtain a binary image.

[0155] Specifically, C3 can be set to a number between 2 and 10;

[0156] Set the adaptive comparison benchmark threshold TC = the width of the panoramic image / L, where L is 50 to 200

[0157] Traverse the binary image from top to bottom along the y-axis. When the number of coordinate points with pixel values ​​equal to 0 in all (x, yi) of the binary image under the same yi coordinate is less than TC, get the upper boundary of the image top_border = yi, where the value range of x is [0, the width of the panoramic image].

[0158] Traverse the binary image from bottom to top along the y-axis. When the number of coordinate points with pixel values ​​equal to 0 in all (x, yi) of the binary image under the same yi coordinate is less than TC, get the lower boundary of the image bottom_border = yi, where the value range of x is [0, the width of the panoramic image].

[0159] The final panoramic image is obtained by intercepting the corresponding image on the panoramic image through the obtained upper boundary and lower boundary pair.

[0160] The above description is merely a preferred embodiment of the present invention, and the present invention is not limited to the above embodiment. It will be clear to those skilled in the art that the form of the embodiment is not limited thereto, and the adjustable manner is also not limited thereto. It will be understood that other improvements and variations directly derived or imagined by those skilled in the art without departing from the basic concept of the present invention should be considered to be included in the scope of protection of the present invention.

Claims

1. A method for stitching panoramic images of drones based on drone mother vehicles, characterized in that: include: The drone mother vehicle acquires images taken by the drone; Perform cylindrical projection on the acquired image to obtain a cylindrical projection image; perform corner point detection on the cylindrical projection image to obtain corner points; perform feature extraction and feature matching on the cylindrical projection image based on the obtained corner points to obtain feature matching point pairs of the spliced ​​image; The cylindrical projection images are stitched and fused according to the extracted feature matching points to form a panoramic image; Wherein, the cylindrical projection image is binarized to obtain a binarized image; Perform contour detection on the binary image according to the adaptive threshold and obtain the detected contour coordinates; According to the contour coordinates, the image of the area surrounded by the contour coordinates is intercepted in the cylindrical projection image to form a new cylindrical projection image; Obtain the best matching point pair from the feature matching point pairs, and perform image stitching through the best matching point pair; The feature matching points of the stitched image are concentrated on calculating the best matching point pair in the y-axis direction. The acquisition of the best matching point pair includes: Get the first pairing matrix. Each element in the first pairing matrix contains matching feature points. Calculate the shifted_value value. Then, take the first matching point from all elements of the first pairing matrix and subtract the shifted_value from each of them to obtain n diff_values. The set of n different diff_values ​​is recorded as difference. Traverse all elements of the first pairing matrix in turn, find the mutually matching feature points that make inpoint achieve the maximum value, define it as the best matching feature point, and the shift_value corresponding to the best matching feature point is the best_shift_value; The acquisition of shifted_value includes: Take the first element in the first pairing matrix. The first element includes the coordinates of a pair of matching points, which are recorded as the first matching point and the second matching point. Subtract the coordinates of the first matching point from the coordinates of the second matching point in the matching point pair, and the obtained value is recorded as shift_value. Then, all elements are taken from the first pairing matrix, and all second matching points are taken from all elements and subtracted from each other by shift_value. The set of obtained results is recorded as shifted_value.

2. The method for stitching panoramic images of a drone based on a drone mother vehicle according to claim 1, characterized in that: The stitched panoramic image is end-to-end rectified; and the rectified panoramic image is cropped to obtain the final panoramic image.

3. The method for stitching panoramic images of a drone based on a drone mother vehicle according to claim 1, characterized in that: Before cylindrical projection, the image is preprocessed; The preprocessing includes at least: obtaining the camera pose and drone pose corresponding to each image, constructing an orthographic projection transformation matrix; and converting the image into an orthographic projection direction through the orthographic projection transformation matrix.

4. The method for stitching panoramic images of a drone based on a drone mother vehicle according to claim 1, characterized in that: Corner detection includes at least: Obtain the coordinates and eigenvalues ​​of each corner point to form the first corner point matrix; Set the filtering conditions, filter the first corner point matrix, delete the corner points that do not meet the conditions, and form the second corner point matrix; Set a first window of fixed size, traverse the second corner point matrix with the first window, and if the sum of the eigenvalues ​​of the corner points corresponding to all coordinates in the window area is not 0, obtain the coordinate value corresponding to the corner point with the largest eigenvalue in the window area; With the corner point of the maximum eigenvalue as the center, the area enclosed by the first window is formed, the coordinates feature_coordinate corresponding to the enclosed area are obtained, and the pixel values ​​of the corresponding area in the cylindrical projection image are intercepted by the corresponding coordinates and recorded as feature_value.

5. The method for stitching panoramic images of drones based on a drone mother vehicle as claimed in claim 4, characterized in that: The first corner point matrix acquisition at least includes: Get the intensity factors Dx, Dy, and Dxy in the x-direction, y-direction, and xy-direction; Perform preset rule filtering on the obtained intensity factors Dx, Dy, and Dxy; The preset rule filtering includes: calculating the sum of Dx or Dy or Dxy of all points in the preset window area, and if the sum of the values ​​exceeds 255, setting Dx or Dy or Dxy of the center point of the window area to 255; traversing the entire image in sequence to obtain the filtered intensity factors Dx', Dy', Dxy'; The eigenvalue of the candidate corner point is calculated by the intensity factor. When the eigenvalue of the candidate corner point is greater than a preset threshold, the coordinates and eigenvalue of the corner point are recorded to form a first corner point matrix.

6. The method for stitching panoramic images of a drone based on a drone mother vehicle as claimed in claim 4, characterized in that: The filtering conditions for the first corner matrix include: Compare the corner point eigenvalues ​​in the first corner point matrix with the preset threshold, obtain all corner points greater than the preset threshold, and then set the corner point eigenvalues ​​less than the preset threshold to 0; set the corner point eigenvalues ​​in the preset edge area to 0 to form a second corner point matrix.

7. The method for stitching panoramic images of a drone based on a drone mother vehicle according to claim 1, characterized in that: The acquisition of feature matching point pairs includes: Extract features from cylindrical projection images through corner points; For the adjacent i-th image and i+1-th image, take the yi coordinate of the corner point feature value of the i-th image as the comparison benchmark, traverse the square area surrounded by the yi coordinate and the preset length C4 of the corner point feature value in the i+1-th image, and obtain all corner points that meet the conditions and record them as candidate feature points; Calculate the similarity distances between the feature points in the i-th image and the corresponding candidate feature points to form a first distance matrix; Obtain the first two elements dmin and dmin1 with the smallest similarity distance in the first matrix, compare dmin with dmin1, and if the second preset condition is met, the feature point of the i-th image and the candidate feature point of the i+1-th image corresponding to the corresponding element of the first distance matrix are mutually matched feature points.

8. The method for stitching panoramic images of a drone based on a drone mother vehicle as claimed in claim 7, characterized in that: For the splicing of the i-th image and the i+1-th adjacent image, traverse the i-th image along the y-axis with the yj value of the feature coordinate feature_coordinate; If the value of the y coordinate of the corresponding feature point in the first feature coordinate matrix in the i+1 image is in the range of [yj-range,yj+range], then the distance between the feature_value value corresponding to the feature point coordinate feature_coordinate in the i-th image and the corresponding feature_value values ​​in the coordinate feature_coordinate of all feature points in the i+1-th image is calculated to form a first distance matrix.

9. The method for stitching panoramic images of a drone based on a drone mother vehicle as claimed in claim 7, characterized in that: The acquisition of mutually matching feature points includes: Sort the elements in the first distance matrix from large to small, and obtain the minimum value dmin and the second minimum value dmin1 according to the sorting result; If dmin / dmin1 <= C1, store the feature point coordinates in the i-th image and the corresponding feature point coordinates in the (i + 1)-th image to form the elements in the first pairing matrix. Traverse all feature points in sequence, respectively obtain all the coordinates that match each other in the i-th image and the (i + 1)-th image, form the final first pairing matrix, and the first pairing matrix stores the coordinates corresponding to the feature points that match each other in the i-th image and the (i + 1)-th image.

10. The method for stitching panoramic images of a drone based on a drone mother vehicle according to claim 1, characterized in that: The mutually matching feature points corresponding to making inpoint obtain the maximum value at least include: Set inpoint = 0, best_shift = 0. Take an element diff_value from difference, calculate the sum of squares of all the values included in diff_value. If the sum of squares is less than the preset first threshold, then inpoint is incremented by 1. After traversing all diff_value in sequence, record inpoint. If the inpoint value is greater than the preset second threshold, then assign the inpoint value to the preset second threshold, and assign the value of the corresponding shift_value to best_shift_value. Traverse all elements of the first pairing matrix in sequence to obtain the final best_shift_value. Among them, best_shift_value is the coordinate difference obtained by subtracting the first matching point from the second matching point in the best matching point pair between the i-th image and the (i + 1)-th image.

11. The method for stitching panoramic images of a drone based on a drone mother vehicle according to claim 1, characterized in that: When stitching panoramic images, calibrate the coordinates of adjacent images through best_shift_value, adjust the coordinate points with common areas in adjacent images so that the common coordinate points are in the same reference coordinate system, and perform image fusion on adjacent images.

12. The method for stitching panoramic images of a drone based on a drone mother vehicle according to claim 1, characterized in that: The stitching and fusion of cylindrical projection images include: Preset a window of a fixed size, traverse along the y-axis direction of the image, and perform stitching and fusion on adjacent images according to the value of the abscissa x. When W / 2 – C2 <= x <= W / 2 + C2, the pixel value of the corresponding coordinate is calculated by the following formula: Proportion=(x–seam+C2) / C2 2 I(x,y) = (1 – Proportion)*img2[x,y] + Proportion*img1[x,y] When x < W / 2 – C2, the pixel value I(x,y) of the corresponding coordinate = img1[x,y] When x > W / 2 + C2, the pixel value I(x,y) of the corresponding coordinate = img2[x,y] Among them, W is the width of the img1 image, C2 is the preset threshold, and img1 and img2 are the images to be stitched.

13. The method for stitching panoramic images of drones based on a drone mother vehicle according to claim 1, characterized in that: When the drone mother vehicle establishes a communication connection with the drone; When the drone mother vehicle starts the command for panoramic photographing and sends it to the drone, the drone calls the panoramic photographing program to take a panoramic photograph, and sends the images for stitching to the drone mother vehicle frame by frame according to the stitching order. After the drone mother vehicle finishes stitching the panoramic image, mark the geographical location of the stitched panoramic image according to the current drone GPS information and pose information; send the panoramic image to the in-vehicle Ethernet display screen for display.

Images