An aerial image automatic splicing method and device

Abstract

This invention provides an automatic aerial image stitching method and apparatus, comprising the following steps: calculating the transformation correction matrix required to convert a first frame of aerial image into a standard image at a specified height and angle; calculating the edge position of the standard image at the specified height and angle placed in the background image in the opposite direction of the aircraft's flight; calculating the coordinates of the center and four corners of the next frame of aerial image at the specified height and angle based on the transformation correction matrix; calculating the overlap rate between the next frame and the previous frame, and calculating matching feature points of the overlapping portion; aligning the images based on whether the matching feature points are sufficient, and superimposing the transformed images onto the stitched image using different methods. This invention has no strict requirements on image features and overlap rate, has strong adaptability, and can achieve real-time stitching of newly added images over a wide range.

Description

Technical Field

[0001] This invention belongs to the field of image stitching processing technology, specifically relating to an automatic aerial image stitching method and apparatus. Background Technology

[0002] Using drones equipped with aerial cameras to acquire images of target areas can quickly obtain ground image information over a large area, and enable rapid searching and inspection of the target area through real-time image transmission. However, the information contained in a single image is limited, and drone aerial photography quickly generates a large number of images. To facilitate observation of the target area by ground personnel, multiple images need to be stitched together for display.

[0003] Existing aerial image stitching methods are mostly designed for UAV mapping, often employing long-term, meticulous post-processing and stitching. These methods result in high image overlap rates. Feature matching and other methods can be used for image registration and fusion, while others use ground calibration points for image calibration and reference point correction and stitching. These methods place high demands on operation time and the mission environment. For situations where operation time, air-to-ground data transmission, or environmental conditions result in low image overlap rates, indistinct features, or the lack of ground calibration objects, there is currently a lack of effective image stitching methods. Some methods use the UAV's attitude angle to transform images from rectangles to trapezoids based on tilt angles and geometric relationships before stacking, but this transformation has significant deviations. Other methods use UAV speed to stack images line by line, resulting in severe horizontal distortion and significant deformation of objects and their positional relationships. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and apparatus for automatic aerial image stitching. The solution of this invention can solve the problems existing in the prior art.

[0005] The technical solution of this invention:

[0006] According to the first aspect, an automatic aerial image stitching method is provided, comprising the following steps:

[0007] Step 1: Obtain the installation location of the image acquisition unit, the internal parameters of the image acquisition unit, and a map of the target area containing altitude and latitude / longitude information;

[0008] Step 2: During the flight of the aircraft, obtain the first frame of aerial image of the target area and the position and attitude information of the aircraft when the aerial image is obtained;

[0009] Based on the obtained image acquisition unit information and the aircraft's position and attitude information, calculate the transformation correction matrix required to convert the aerial image into a standard image at a specified altitude and angle.

[0010] Step 3: Calculate the standard image of the aerial image at a specified height and angle based on the first frame of the aerial image and the transformation correction matrix;

[0011] Set a blank base map, and place the standard image of the first aerial image on the edge of the base map in the opposite direction of the aircraft's flight.

[0012] Step 4: During the flight of the aircraft, obtain the next frame of aerial image of the target area and the position and attitude information of the aircraft when the aerial image is obtained;

[0013] Based on the obtained image acquisition unit information and the aircraft's position and attitude information, calculate the transformation correction matrix required to convert the aerial image into a standard image at a specified altitude and angle.

[0014] The coordinates of the center and four corners of the next frame of aerial image are calculated using a transformation correction matrix to determine the coordinates of the aerial image at a specified height and angle.

[0015] The overlap rate between the next frame and the previous frame is calculated based on the coordinates, and the matching feature points of the overlapping part are calculated based on the set feature detection threshold.

[0016] The system determines whether there are enough matching feature points for image alignment. If there are at least six matching feature points, the two images are registered according to these feature points. If there are fewer than six matching feature points, the feature detection threshold is lowered, and the matching feature points for the overlapping parts are recalculated. If at least six new matching feature points are calculated, the two images are registered according to these new matching feature points. During registration, the transformation matrix required for registering the new image is calculated. After calculating the transformation correction matrix and the transformation matrix, the new image is registered and superimposed onto the stitched image. If, after lowering the feature detection threshold, there are still fewer than six matching feature points, the transformation matrix required for superimposing the new image onto the stitched image is calculated based on the positions of the four corner vertices of the image. The image obtained after transformation using the transformation correction matrix and the transformation matrix is ​​then superimposed onto the stitched image.

[0017] Step five, repeat step four until the task is completed.

[0018] Furthermore, the internal parameters of the image acquisition unit include the ratio of the focal length of the image acquisition unit to the horizontal dimension of a unit pixel of the imager, fx; the ratio of the focal length of the image acquisition unit to the vertical dimension of a unit pixel of the imager, fy; the horizontal offset of the imager center relative to the optical axis, cx; the vertical offset of the imager center relative to the optical axis, cy; and the focal length of the image acquisition unit, f.

[0019] Furthermore, the aircraft's position and attitude information includes its longitude, latitude, altitude, speed, azimuth, pitch, and roll.

[0020] Furthermore, the method for calculating the transformation correction matrix is ​​as follows:

[0021] S2.1, Determine the coordinates of the optical center of the image acquisition unit, the center of the imager, and the coordinates of the four corner vertices in the spacecraft's central coordinate system PXYZ when stationary;

[0022] S2.2, Based on the coordinates obtained in S2.1 and the attitude angle of the aircraft at the imaging moment, determine the coordinates of the optical center of the image acquisition unit, the center of the imager, and the coordinates of the four corner vertices at the imaging moment in the PXYZ coordinate system of the aircraft center.

[0023] S2.3, the origin of the UAV's central coordinate system PXYZ is translated from point P to the ground position directly below the UAV to obtain the coordinate system OXYZ. Based on the coordinates obtained in S2.2, the position information of the aircraft at the imaging time, and the map information, the coordinates of the optical center of the image acquisition unit, the center of the imager, and the coordinates of the four corner vertices at the imaging time are determined in the coordinate system OXYZ.

[0024] S2.4, Based on the coordinates obtained in S2.3, calculate the coordinates of the actual positions of the images formed on the ground at the time of imaging, including the optical center of the image acquisition unit, the center of the imager, and the four corner vertices.

[0025] S2.5, Based on the coordinates obtained in S2.4, calculate the coordinates of the four corner vertices of a frame image in the standard image acquisition unit coordinate system in the coordinate system OXYZ;

[0026] S2.6, Based on the coordinates obtained in S2.5, calculate the pixel coordinates of the points on the ground that are actually located at the positions of the four corner vertices of an image frame under the imaging standard.

[0027] S2.7 Calculate the transformation correction matrix of the image based on the pixel coordinate relationship between the four corners of the original image and the corrected image.

[0028] Furthermore, the specified altitude is the sum of the most suitable relative altitude of the aircraft above the ground during flight and the average altitude of the shooting area.

[0029] Furthermore, the specified angle refers to the imaging angle when the aircraft's nose is facing due north and taking pictures vertically downwards.

[0030] According to the second aspect, an automatic aerial image stitching device is provided, comprising an aircraft, an image acquisition unit, an image transmission unit, and an image processing unit. The image acquisition unit is installed on the aircraft and transmits the acquired image of a specified area, along with the aircraft's position and attitude information at the time of image acquisition, to the image processing unit via the image transmission unit. The image processing unit presets map information of the specified area and information from the image acquisition unit. After receiving the image, it calculates the first frame image using a transformation correction matrix and places it on the edge opposite to the flight direction of the blank background map. For the next frame image, it calculates the overlap rate with the previous frame image by calculating the image of the four corner vertices using a transformation correction matrix and calculates the feature points of the overlapping part. It determines whether the number of matching feature points is sufficient for matching and aligning the two images. If there are at least 6 matching feature points, the two images are registered according to the feature points. If there are fewer than 6 matching feature points, the feature detection threshold is lowered, and the matching feature points of the overlapping part are recalculated. If there are at least 6 newly calculated matching feature points, the two images are registered according to the newly calculated matching feature points. During registration, the transformation matrix required for registering the new image is calculated. After calculating the transformation correction matrix and the transformation matrix, the new image is registered and superimposed onto the stitched image. If, after lowering the feature detection threshold, there are still fewer than 6 matching feature points, the transformation matrix required for superimposing the new image onto the stitched image is calculated based on the positions of the four corner vertices of the image. The image obtained after transformation using the transformation correction matrix and the transformation matrix is ​​then superimposed onto the stitched image.

[0031] Furthermore, the image acquisition unit is an aerial camera.

[0032] Furthermore, the image processing unit includes a transformation matrix calculation module, a feature point calculation module, an image transformation module, and an image stitching module. The transformation matrix calculation module, based on the parameters of the image acquisition unit and the aircraft's position and attitude information at the time of image acquisition, obtains the image transformation correction matrix and transmits it to the image transformation module. The image transformation module calculates the pixel coordinates of the points formed by the four corner vertices of the image at their actual ground positions under specified altitude and angle imaging conditions, and sends this information to the feature point calculation module. The feature point calculation module calculates the overlap rate between the new image and the previous frame image based on the pixel coordinates of the four corner vertices of the new image and the previous frame image, and calculates matching feature points in the overlapping portion of the two frames based on a specified feature detection threshold, determining the number of matching feature points. If the number of feature points is greater than 6, the data is sent to the image conversion module. If not, the feature detection threshold is lowered, the matching feature points of the overlapping parts of the two frames are recalculated, and then sent to the image conversion module. After receiving the feature points, the image conversion module determines whether the number of feature points is greater than 6. If so, the two images are registered according to the matching feature points, the transformation matrix required for registering the new image is calculated, and the image obtained after calculating the transformation correction matrix and transformation matrix of the aerial image is sent to the image stitching module. If not, the transformation matrix required for superimposing the new image onto the stitched image is calculated based on the positions of the four corner vertices of the image, and the image obtained after calculating the transformation correction matrix and transformation matrix is ​​sent to the image stitching module. The image stitching module then stitches the obtained images.

[0033] Furthermore, the aerial image automatic stitching device also includes a display unit, which displays the stitching result of the image processing unit.

[0034] According to a third aspect, an aircraft uses the aerial image automatic stitching method of this application to stitch images.

[0035] According to the fourth aspect, an aircraft that uses the aerial image automatic stitching device of this application to perform image stitching.

[0036] The beneficial effects of this invention compared to the prior art are as follows:

[0037] (1) This invention reduces unnecessary feature point calculations and unnecessary image conversion calculations by only converting the positions of the four corner vertices of the image in the early stage of image stitching, calculating the feature points of the overlapping parts and determining the stitching method, and then converting the overall image after determining the stitching method. This improves the calculation speed and achieves the effect of real-time image stitching.

[0038] (2) The present invention determines the stitching method by calculating the number of feature points. When there are insufficient feature points, stitching is performed by the projection positions of the four corner vertices of the image. It has no strict requirements on image features, overlap rate, etc., and has strong adaptability. It can quickly stitch new images on a large scale.

[0039] (3) The present invention obtains the ground height of the imaging area by pre-setting data such as camera parameters and installation parameters and providing a map. The camera is used to capture the position and attitude information of the aircraft during imaging, so that even without external support conditions such as reference points, it still has relatively accurate image-related data, which can improve image processing capabilities and accuracy.

[0040] (4) The image correction and positioning method provided by the present invention calculates the image position coordinates and correction transformation matrix in real time through coordinate system transformation, which can quickly correct the image to an orthophoto image obtained at a specified height, and at the same time obtain the geographic coordinates corresponding to the image shooting area, thereby improving the image correction quality and stitching accuracy. Attached Figure Description

[0041] The accompanying drawings, which form part of this specification, are provided to further illustrate embodiments of the invention and, together with the textual description, explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0042] Figure 1 The diagram illustrates the steps of an automatic aerial image stitching method according to an embodiment of the present invention.

[0043] Figure 2 A schematic diagram of an automatic aerial image stitching device according to an embodiment of the present invention is shown;

[0044] Figure 3 A schematic diagram of a coordinate transformation correction method provided according to an embodiment of the present invention is shown;

[0045] Figure 4 A schematic diagram illustrating the transformation between the imaging plane and actual space according to an embodiment of the present invention is shown;

[0046] Figure 5 A schematic diagram of a splicing method provided according to an embodiment of the present invention is shown. Detailed Implementation

[0047] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0049] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0050] like Figure 1 As shown, according to an embodiment of the present invention, a method for automatically stitching aerial images is provided according to a first aspect, comprising the following steps:

[0051] Step 1: Obtain the installation location of the image acquisition unit, the internal parameters of the image acquisition unit, and a map of the target area containing altitude and latitude / longitude information;

[0052] In one embodiment, the intrinsic parameters of the image acquisition unit include the ratio of the focal length of the image acquisition unit to the horizontal dimension of a unit pixel of the imager, fx; the ratio of the focal length of the image acquisition unit to the vertical dimension of a unit pixel of the imager, fy; the horizontal offset of the imager center relative to the optical axis, cx; the vertical offset of the imager center relative to the optical axis, cy; and the focal length of the image acquisition unit, f.

[0053] In one specific embodiment, the image acquisition unit is an aerial camera. According to the camera parameters, the camera focal length f and the horizontal dimension 2u and vertical dimension 2v of the camera imager are known. Let the center of the imager be Os, the upper left corner be LT, the upper right corner be RT, the lower left corner be LB, and the lower right corner be RB. Then the horizontal distance between the four corners of the imager and the center is u, and the vertical distance is v. When the camera images, assuming the camera's optical axis is vertically downward and the camera is facing due north, with the camera's optical center at the moment of imaging as the origin, the positive x-axis is horizontally east, the positive y-axis is horizontally north, and the positive z-axis is vertically upward, a camera center coordinate system OcXYZ is established. Then the coordinates of the camera's optical center, the imager center, and the four corner vertices are Oc(0,0,0), Os(cx,cy,-f), LT(cx-u,cy+v,-f), RT(cx+u,cy+v,-f), LB(cx-u,cy-v,-f), RB(cx+u,cy-v,-f), which can be written in vector form as follows:

[0054]

[0055] Step 2: During the flight of the aircraft, obtain the first frame of aerial image of the target area and the position and attitude information of the aircraft when the aerial image is obtained;

[0056] In one further embodiment, the aircraft's position and attitude information includes the aircraft's longitude, latitude, altitude, speed, azimuth, pitch, and roll.

[0057] The conversion correction matrix is ​​calculated based on the acquired image acquisition unit information and the aircraft's position and attitude information.

[0058] In a further embodiment, the transformation correction matrix is ​​calculated as follows:

[0059] S2.1, determine the coordinates of the optical center of the image acquisition unit, the center of the imager, and the coordinates of the four corner vertices in the aircraft's central coordinate system PXYZ when stationary; the aircraft's central coordinate system PXYZ in this application is defined as follows: taking the origin of the aircraft's body coordinate system as the aircraft center P, with the aircraft center as the coordinate origin, the right side of the fuselage as the positive x-axis, the nose direction as the positive y-axis, and the back direction as the positive z-axis. Based on the camera's installation position and camera parameters, the coordinates of the camera's optical center relative to the UAV's center are (xc, yc, zc). Then, by translating the origin of the camera's central coordinate system to the UAV's center, the UAV's central coordinate system PXYZ is obtained, and the coordinates of the camera's optical center, the imager center, and the four corner vertices are:

[0060]

[0061] S2.2, Based on the coordinates obtained in S2.1 and the attitude angle of the aircraft at the imaging moment, determine the coordinates of the optical center of the image acquisition unit, the center of the imager, and the coordinates of the four corner vertices at the imaging moment in the PXYZ coordinate system of the aircraft center.

[0062] In a specific embodiment, the aircraft is selected as a drone. The drone's angles are yaw, pitch, and roll. The yaw angle is the angle between the projection of the drone's forward / backward axis onto the horizontal plane and the Earth's axis in the due north direction (positive is north by east). The pitch angle is the angle between the drone's forward / backward axis and the horizontal plane (positive is nose-up). The roll angle is the angle between the drone's vertical central axis and the vertical plane containing the forward / backward axis. Therefore, for the drone's central coordinate system PXYZ, rotating the drone around the z-axis by an angle yaw, the coordinates of a point on the camera are equivalent to left-multiplying by a matrix. in

[0063]

[0064] Next, rotate the UAV's central coordinate system PXYZ around the z-axis by -yaw to form the coordinate system PX'Y'Z'. The coordinates of the points on the camera are then left-multiplied by the matrix.

[0065] Next, rotate the drone around the x-axis by a pitch angle; the coordinates of the point on the camera are equivalent to left-multiplying by a matrix. in

[0066]

[0067] Next, rotate the coordinate system PX'Y'Z' around the z-axis by -yaw to form the coordinate system PX"Y"Z". The coordinates of the points on the camera are equivalent to multiplying them by the matrix on the left.

[0068] Next, rotate the drone around the y-axis by an angle roll. The coordinates of the point on the camera are equivalent to left-multiplying the matrix. in

[0069]

[0070] Then the coordinate system is transformed from PX"Y"Z" back to PX'Y'Z' and then back to PXYZ. The coordinates of the points on the camera are equivalent to left-multiplying by a matrix.

[0071] Finally, the coordinates of the camera optical center, imager center, and four corner vertices in the PXYZ coordinate system are obtained at the moment of imaging:

[0072]

[0073] S2.3, the origin of the UAV's central coordinate system PXYZ is translated from point P to the ground position below the UAV to obtain the coordinate system OXYZ. Based on the coordinates obtained in S2.2, the position information of the aircraft at the imaging time, and the map information, the coordinates of the optical center of the image acquisition unit, the center of the imager, and the coordinates of the four corner vertices at the imaging time are determined in the coordinate system OXYZ.

[0074] In one specific embodiment, the origin of the UAV's central coordinate system PXYZ is translated from point P to the ground position directly below the UAV, resulting in the coordinate system OXYZ, as follows: Figure 3 As shown. The altitude (hp) and latitude / longitude of the UAV at the time of imaging can be obtained from the image auxiliary data. The ground altitude (hg) can be obtained from the elevation data using latitude and longitude. Therefore, the center coordinates of the UAV are (0,0,hp-hg). The coordinates of the camera optical center, imager center, and four corner vertices in the OXYZ coordinate system can be obtained by adding hp-hg to their respective z-axis coordinates.

[0075]

[0076] S2.4, Based on the coordinates obtained in S2.3, calculate the coordinates of the actual positions of the images formed on the ground at the time of imaging, including the optical center of the image acquisition unit, the center of the imager, and the four corner vertices.

[0077] The following explanation uses the top left corner (LT) of the imager as an example. Figure 3 As shown, the image formed at point LT is located at the actual ground position LT. 地 coordinates It can be represented in vector form:

[0078]

[0079] in Let Oc be the coordinates. The coordinates of point LT have already been obtained in the previous step. Therefore, it is easy to obtain the coordinates of the actual positions of the images formed at the center and four corner vertices of the imager on the ground [O]. s地 LT 地 RT 地 LB 地 RB 地 ].

[0080] S2.5, Based on the coordinates obtained in S2.4, calculate the coordinates of the four corner vertices of a frame image in the standard image acquisition unit coordinate system in the coordinate system OXYZ;

[0081] In one embodiment, the image acquisition unit is a camera. Let a point Q on the ground be imaged onto point Q' on the camera imager. The coordinates of Q' in the image coordinate system (in pixels) are (u, v), and its coordinates in the image plane coordinate system (in length units) are (x, y). Let the physical dimensions of each pixel in the x-axis and y-axis directions be dx and dy, respectively. Let the distances between the origin of the image coordinate system and the origin of the image plane coordinate system be Δu and Δv in the x-axis and y-axis directions, respectively. Then we have...

[0082]

[0083] With the camera's optical center as the origin, the rightward direction of the imaging platform as the positive x-axis, the upward direction of the imaging platform as the positive y-axis, and for ease of analysis, the vertically downward direction as the positive z-axis, a camera coordinate system is formed, as follows: Figure 4 As shown, the coordinates of point Q in the camera coordinate system are (Xc, Yc, Zc).

[0084]

[0085] achievable

[0086]

[0087] Here, A is the intrinsic parameter matrix that can be obtained through camera calibration.

[0088] Since the coordinates of the image formed at the center and four corner vertices of the imager in the OXYZ coordinate system have already been calculated through coordinate transformation, [O s地 LT 地 RT 地 LB 地 RB 地 ],like Figure 3 As shown. Assume the camera's optical center is located at the center of the imager, and the image formed is directly above the actual location on the ground, at a distance z from the sea level. stdAt a certain height, with the camera's optical axis pointing vertically downwards, the image plane facing due north above and due east to the right, using this configuration as the imaging standard for calibration, the coordinates of the camera's optical center in the OXYZ coordinate system are: Using this camera as the center, establish the camera coordinate system from the previous step as the standard camera coordinate system. Then, the coordinates of point Q (Xq, Yq, Zq) in the OXYZ coordinate system are:

[0089]

[0090] Therefore, according to We can obtain the pixel coordinates [uv] of point Q under this imaging standard. T .

[0091] S2.6 Based on the coordinates obtained in S2.5, calculate the pixel coordinates of the points on the ground at the actual positions of the images formed by the four corner vertices of a frame under the imaging standard, that is, the pixel coordinates of the four corner vertices of the corrected image. This is a well-known technique in the field.

[0092] S2.7 Calculate the transformation correction matrix of the image based on the pixel coordinate relationship between the four corners of the original image and the corrected image. This is a well-known technique in the field.

[0093] Step 3: Calculate the standard image of the aerial image at a specified height and angle based on the first frame of the aerial image and the transformation correction matrix;

[0094] In one specific embodiment, by transforming the correction matrix, the original image can be corrected to a standard image taken at a specified altitude and direction through transmission transformation, thus completing the image correction. Furthermore, based on the actual position coordinates of the images formed at the four corner vertices of the imager on the ground and the aircraft position information at the time of imaging, the latitude and longitude coordinates of the actual imaging area can be easily calculated.

[0095] Set a blank base map, place the standard image of the first aerial image on the edge of the base map in the opposite direction of the aircraft's flight. When stitching later, only the newly added image area needs to be modified. If it exceeds the range of the base map, the base map will be expanded or a new round of stitching will be started depending on the size of the base map.

[0096] Step 4: During the flight of the aircraft, obtain the next frame of aerial image of the target area and the position and attitude information of the aircraft when the aerial image is obtained;

[0097] The transformation correction matrix is ​​calculated based on the acquired image acquisition unit information and the aircraft's position and attitude information; see step three for details.

[0098] The coordinates of the center and four corners of the next frame of aerial image are calculated using a transformation correction matrix to determine the coordinates of the aerial image at a specified height and angle.

[0099] In one embodiment, the specified altitude is the sum of the most suitable relative altitude of the aircraft above the ground during flight and the average elevation of the shooting area; the specified angle refers to the imaging angle when the aircraft's nose is facing due north and shooting vertically downwards. By converting the raw images obtained from aerial photography into images at the specified altitude and angle, the image correction quality and stitching accuracy are improved.

[0100] Calculate the overlap rate between the next frame and the previous frame based on the coordinates, and calculate the matching feature points of the overlapping part.

[0101] In a further specific embodiment, in order to ensure the accuracy of the calculation, the overlap rate can be appropriately increased during the calculation. For example, if the calculated overlap rate is 17%, the overlap rate can be taken as 20%.

[0102] In one embodiment, the number of matching feature points is inversely correlated with the feature detection threshold. The larger the feature detection threshold, the smaller the number of matching feature points. Adjusting the feature detection threshold can achieve the purpose of adjusting the number of matching feature points.

[0103] To determine whether there are enough matched feature points for image alignment, in one embodiment, the criterion for sufficient matched feature points can be set to whether there are at least 6. If the number is insufficient, the feature detection threshold is lowered and the matched feature points are recalculated. The feature detection threshold is related to the detection algorithm. For example, using the SURF detection algorithm, the initial threshold can be set to 2000. If the number of detected matched feature points is less than 6, the threshold is lowered to 800 and the matched feature points are recalculated to determine whether more than 6 matched feature points can be found.

[0104] If there are enough matching feature points, or if there are enough matching feature points after lowering the detection threshold, then the two images are registered according to the matching feature points. The transformation matrix required for registering the new image is calculated. After calculating the transformation correction matrix and the transformation matrix, the new image is registered and superimposed on the stitched image. If there are not enough, then the transformation matrix required for superimposing the new image on the stitched image is calculated based on the positions of the four corner vertices of the image. The image obtained after calculating the transformation correction matrix and the transformation matrix is ​​superimposed on the stitched image.

[0105] In one further embodiment, during image stitching, the portion outside the overlapping area can be directly superimposed onto the stitched image. By setting a transition zone in the overlapping area and performing a weighted average on the pixels in the transition zone, the pixel values ​​at positions further forward in the image are closer to the previous image, and the pixel values ​​at positions further backward in the image are closer to the subsequent image, thereby avoiding obvious stitching seams; in another embodiment, such as Figure 5 As shown, the optimal stitching line algorithm can be used to search for a path that separates the front and back images in the overlapping area, minimizing the pixel difference on both sides of the path, thus optimizing the stitching seam while preserving the details of the front and back images.

[0106] Step five, repeat step four until the task is completed.

[0107] In a further embodiment, the specified altitude is the sum of the most suitable relative altitude of the aircraft above the ground during flight and the average altitude of the ground in the shooting area.

[0108] In a further embodiment, the specified angle refers to the imaging angle when the aircraft's nose is facing due north and taking pictures vertically downwards.

[0109] According to the second aspect of the embodiment, such as Figure 2 As shown, an automatic aerial image stitching device is provided, comprising an aircraft, an image acquisition unit, an image transmission unit, and an image processing unit. The image acquisition unit is installed on the aircraft and transmits the acquired image of a specified area, along with the aircraft's position and attitude information at the time of image acquisition, to the image processing unit via the image transmission unit. The image processing unit presets map information of the specified area and information from the image acquisition unit. After receiving the image, it calculates the first frame image using a transformation correction matrix and places it on the edge opposite to the flight direction of the blank background map. For the next frame image, it calculates the overlap rate with the previous frame image by calculating the image of the four corner vertices using a transformation correction matrix and calculates the feature points of the overlapping part. It determines whether the number of matching feature points is sufficient for matching and aligning the two images. If there are at least 6 matching feature points, the two images are registered according to the feature points. If there are fewer than 6 matching feature points, the feature detection threshold is lowered, and the matching feature points of the overlapping part are recalculated. If there are at least 6 newly calculated matching feature points, the two images are registered according to the newly calculated matching feature points. During registration, the transformation matrix required for registering the new image is calculated. After calculating the transformation correction matrix and the transformation matrix, the new image is registered and superimposed onto the stitched image. If, after lowering the feature detection threshold, there are still fewer than 6 matching feature points, the transformation matrix required for superimposing the new image onto the stitched image is calculated based on the positions of the four corner vertices of the image. The image obtained after transformation using the transformation correction matrix and the transformation matrix is ​​then superimposed onto the stitched image.

[0110] In another embodiment, the image acquisition unit is an aerial camera.

[0111] In a further embodiment, the image processing unit includes a transformation matrix calculation module, a feature point calculation module, an image transformation module, and an image stitching module. The transformation matrix calculation module obtains the transformation correction matrix of the image based on the parameters of the image acquisition unit, the aircraft position information and attitude information at the time of image acquisition, and transmits it to the image transformation module. The image transformation module calculates the pixel coordinates of the points at the actual ground positions formed by the four corner vertices of the image under the imaging conditions of a specified height and a specified angle, and sends them to the feature point calculation module. The feature point calculation module calculates the overlap rate between the new image and the previous frame image based on the pixel coordinates of the four corner vertices of the new image and the previous frame image, and calculates the matching feature points of the overlapping parts of the two frames images according to a specified feature detection threshold, and determines the number of matching feature points. If the number of feature points is greater than 6, the data is sent to the image conversion module. If not, the feature detection threshold is lowered, the matching feature points of the overlapping parts of the two frames are recalculated, and then sent to the image conversion module. Upon receiving the feature points, the image conversion module determines if the number of feature points is greater than 6. If so, it registers the two images according to the matching feature points, calculates the transformation matrix required for registering the new image, and sends the aerial image obtained after calculating the transformation correction matrix and transformation matrix to the image stitching module. If not, it calculates the transformation matrix required for superimposing the new image onto the stitched image based on the positions of the four corner vertices, and sends the image obtained after calculating the transformation correction matrix and transformation matrix to the image stitching module. The image stitching module then stitches the obtained images.

[0112] In a further embodiment, the aircraft, serving as a flight platform, provides the maneuverability for taking photos over a wide area. It is preferably a fixed-wing aircraft with high flight speed or long endurance, capable of acquiring its own position and attitude data. Specifically, in one embodiment, the aircraft may be a drone.

[0113] In a further embodiment, the aerial image automatic stitching device also includes a display unit that displays the stitching results from the image processing unit for easy image viewing.

[0114] In one specific embodiment, the display unit includes two interfaces: a map display and an image display. The map display window can use an electronic elevation map to display the drone's location and flight status in real time, and can obtain elevation values ​​for specified latitude and longitude. The image display window can display the currently stitched image in real time in a large image format, and provides a list of real-time received images and a list of stitched images in the form of a thumbnail list. Clicking on a thumbnail in the list can open the corresponding image for viewing in a large image format. The displayed large image provides zoom, drag and other operations for easy viewing.

[0115] To gain a better understanding of the automatic aerial image stitching method and apparatus provided by the present invention, a detailed description is provided below with reference to specific examples.

[0116] In a third aspect embodiment, an aircraft is provided that uses the aerial image automatic stitching method of this application to stitch images.

[0117] In a fourth aspect embodiment, an aircraft is provided that uses the aerial image automatic stitching device of this application to perform image stitching.

[0118] This invention stitches one 1920*1080 image every approximately 3-10 seconds, which basically meets the needs of real-time stitching. The stitched images show no significant deformation or distortion during stable flight. In contrast, some stitching methods require overall correction of all images after flight completion, which cannot be done in real time. Some stitching software in the surveying field requires a significant amount of time for stitching after image acquisition. Furthermore, some real-time stitching methods, in pursuit of speed, have simpler image processing, resulting in severe image distortion. The stitching method provided by this invention achieves good stitching quality while maintaining high speed, and has minimal requirements for shooting conditions.

[0119] In summary, the present invention offers at least the following advantages compared to existing technologies:

[0120] (1) This invention reduces unnecessary feature point calculation and registration calculation by only converting the positions of the four corner vertices of the image in the early stage of image stitching, calculating the feature points of the overlapping part and determining the stitching method, and then converting the whole image after determining the stitching method. This improves the calculation speed and achieves the effect of real-time image reference.

[0121] (2) The present invention determines the stitching method by calculating the number of feature points. When there are insufficient feature points, stitching is performed by the projection positions of the four corner vertices of the image. It has no strict requirements on image features, overlap rate, etc., and has strong adaptability. It can quickly stitch new images on a large scale.

[0122] (3) The present invention obtains the ground height of the imaging area by pre-setting data such as camera parameters and installation parameters and providing a map. The camera is used to capture the position and attitude information of the aircraft during imaging, so that even without external support conditions such as reference points, it still has relatively accurate image-related data, which can improve image processing capabilities and accuracy.

[0123] (4) The image correction and positioning method provided by the present invention calculates the image position coordinates and correction transformation matrix in real time through coordinate system transformation, which can quickly correct the image to an orthophoto image obtained at a specified height, and at the same time obtain the geographic coordinates corresponding to the image shooting area, thereby improving the image correction quality and stitching accuracy.

[0124] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An automatic aerial image stitching method, characterized by, Includes the following steps: Step 1: Obtain the installation location of the image acquisition unit, the intrinsic parameters of the image acquisition unit, and a map of the target area containing altitude and latitude / longitude information; Step 2: During the flight of the aircraft, obtain the first frame of aerial image of the target area and the position and attitude information of the aircraft when the aerial image is obtained; Based on the acquired image acquisition unit information and the aircraft's position and attitude information, the aerial images are converted into the required transformation correction matrix to transform the aerial images into standard images at a specified altitude and angle; Step 3: Calculate the standard image of the aerial image at a specified height and angle based on the first frame of the aerial image and the transformation correction matrix; Set a blank base map, and place the standard image of the first aerial image on the edge of the base map in the opposite direction of the aircraft's flight. Step 4: During the flight of the aircraft, obtain the next frame of aerial image of the target area and the position and attitude information of the aircraft when the aerial image is obtained; Based on the acquired image acquisition unit information and the aircraft's position and attitude information, the aerial images are converted into the required transformation correction matrix to transform the aerial images into standard images at a specified altitude and angle; The coordinates of the center and four corners of the next frame of aerial image are calculated using a transformation correction matrix to determine the coordinates of the aerial image at a specified height and angle. The overlap rate between the next frame and the previous frame is calculated based on the coordinates, and the matching feature points of the overlapping part are calculated based on the set feature detection threshold. Determine if there are enough matching feature points for image alignment. If there are at least 6 matching feature points, register the two images according to the feature points. If there are fewer than 6 matching feature points, lower the feature detection threshold and recalculate the matching feature points of the overlapping part. If there are at least 6 newly calculated matching feature points, register the two images according to the newly calculated matching feature points. During registration, calculate the transformation matrix required for registering the new image. After calculating the transformation correction matrix and transformation matrix, register the new image and overlay it onto the stitched image. If the number of matching feature points is still less than 6 after lowering the feature detection threshold, the transformation matrix required to overlay the newly added image onto the stitched image is calculated based on the positions of the four corner vertices of the image. The image obtained after transformation by the transformation correction matrix and the transformation matrix is ​​then overlaid onto the stitched image. Step five, repeat step four until the task is completed.

2. The method according to claim 1, wherein, The internal parameters of the image acquisition unit include the ratio of the focal length of the image acquisition unit to the horizontal dimension of a unit pixel of the imager (fx), the ratio of the focal length of the image acquisition unit to the vertical dimension of a unit pixel of the imager (fy), the horizontal offset of the imager center relative to the optical axis (cx), the vertical offset of the imager center relative to the optical axis (cy), and the focal length of the image acquisition unit (f).

3. The method for automatic aerial image stitching according to claim 2, characterized in that, The aircraft's position and attitude information includes its longitude, latitude, altitude, speed, azimuth, pitch, and roll.

4. The method for automatic aerial image stitching according to claim 2 or 3, characterized in that, The method for calculating the transformation correction matrix is ​​as follows: S2.1, Determine the coordinates of the optical center of the image acquisition unit, the center of the imager, and the coordinates of the four corner vertices in the spacecraft's central coordinate system PXYZ when stationary; S2.2, Based on the coordinates obtained in S2.1 and the attitude angle of the aircraft at the imaging moment, determine the coordinates of the optical center of the image acquisition unit, the center of the imager, and the coordinates of the four corner vertices at the imaging moment in the PXYZ coordinate system of the aircraft center. S2.3, the origin of the UAV's central coordinate system PXYZ is translated from point P to the ground position directly below the UAV to obtain the coordinate system OXYZ. Based on the coordinates obtained in S2.2, the position information of the aircraft at the imaging time, and the map information, the coordinates of the optical center of the image acquisition unit, the center of the imager, and the coordinates of the four corner vertices at the imaging time are determined in the coordinate system OXYZ. S2.4, Based on the coordinates obtained in S2.3, calculate the coordinates of the actual positions on the ground of the image acquisition unit optical center, the imager center, and the images formed on the four corner vertices at the imaging moment; S2.5, Based on the coordinates obtained in S2.4, calculate the coordinates of the four corner vertices of a frame image in the standard image acquisition unit coordinate system in the coordinate system OXYZ; S2.6, Based on the coordinates obtained in S2.5, calculate the pixel coordinates of the points on the ground where the images formed by the four corner vertices of a frame are actually located, under the imaging standard. S2.7 Calculate the transformation correction matrix of the image based on the pixel coordinate relationship between the four corners of the original image and the corrected image.

5. The method for automatic aerial image stitching according to claim 1, characterized in that, The specified altitude is the sum of the most suitable relative altitude of the aircraft above the ground during flight and the average altitude of the shooting area; the specified angle is the imaging angle when the aircraft's nose is facing due north and shooting vertically downwards.

6. An automatic aerial image stitching device using the automatic aerial image stitching method as described in any one of claims 1-5, characterized in that, The system includes an aircraft, an image acquisition unit, an image transmission unit, and an image processing unit. The image acquisition unit is mounted on the aircraft and transmits the acquired image of a specified area, along with the aircraft's position and attitude information at the time of image acquisition, to the image processing unit via the image transmission unit. The image processing unit presets map information for the specified area and information from the image acquisition unit. Upon receiving the image, it calculates a transformation correction matrix for the first frame image and places it on the edge opposite to the flight direction on a blank background map. For the next frame image, it calculates the overlap rate with the previous frame image by calculating the image of the four corner vertices using a transformation correction matrix, and calculates the feature points of the overlapping portion. Based on the number of matching feature points, it determines whether there are enough to match the two images. If there are at least 6 matching feature points, the two images are registered according to these matching feature points. If there are fewer than 6 matching feature points, the feature detection threshold is lowered, and the matching feature points of the overlapping parts are recalculated. If there are at least 6 newly calculated matching feature points, the two images are registered according to these newly calculated matching feature points. During registration, the transformation matrix required for registering the new image is calculated. After the aerial image is processed by the transformation correction matrix and the transformation matrix, the new image is registered and superimposed onto the stitched image. If there are still fewer than 6 matching feature points after lowering the feature detection threshold, the transformation matrix required for superimposing the new image onto the stitched image is calculated based on the positions of the four corner vertices of the image. The image obtained after transformation by the transformation correction matrix and the transformation matrix is ​​superimposed onto the stitched image.

7. The automatic aerial image stitching device according to claim 6, characterized in that, The image processing unit includes a transformation matrix calculation module, a feature point calculation module, an image transformation module, and an image stitching module. The transformation matrix calculation module obtains the transformation correction matrix of the image based on the parameters of the image acquisition unit, the aircraft position information and attitude information at the time of image acquisition, and transmits it to the image conversion module. The image conversion module calculates the pixel coordinates of the points on the ground formed by the four corner vertices of the image under specified height and angle imaging conditions, and sends them to the feature point calculation module. The feature point calculation module calculates the overlap rate between the new image and the previous frame image based on the pixel coordinates of the four corner vertices of the new image and the previous frame image, and calculates the matching feature points of the overlapping part of the two frames image according to the specified feature detection threshold. It determines whether the number of matching feature points is greater than 6. If so, it sends the data to the image conversion module. If not, it lowers the feature detection threshold, recalculates the matching feature points of the overlapping part of the two frames image, and sends the data to the image conversion module. After receiving feature points, the image conversion module determines whether the number of feature points is greater than 6. If so, it registers the two images according to the matching feature points, calculates the transformation matrix required for registering the new image, and sends the image obtained by calculating the transformation correction matrix and transformation matrix of the aerial image to the image stitching module. If not, it calculates the transformation matrix required for superimposing the new image onto the stitched image based on the positions of the four corner vertices of the image, and sends the image obtained by calculating the transformation correction matrix and transformation matrix to the image stitching module. The image stitching module then stitches the obtained images together.

8. The automatic aerial image stitching device according to claim 6, characterized in that, The aerial image automatic stitching device also includes a display unit, which displays the stitching results of the image processing unit.

9. An aircraft, characterized in that, The aerial image automatic stitching method according to any one of claims 1-5 is used for image stitching.

10. An aircraft, characterized in that, Image stitching is performed using the automatic aerial image stitching device according to any one of claims 6-8.

Images