A Visible Light and SAR Large View Angle Difference Image Registration Method and System for UAV Platforms

Abstract

The present invention discloses a visible light and SAR large-view-angle difference image registration method and system for an unmanned aerial vehicle platform. The method includes: constructing a set of matching point pairs according to the visible light image and the SAR image; obtaining a homography transformation matrix according to the set of matching point pairs; multiplying the homography transformation matrix by the SAR image to obtain a coarsely registered SAR image; obtaining respective corner points of the visible light image and the coarsely registered SAR image by combining a feature response model and the FAST feature point detection algorithm; constructing descriptors for all corner points in the image, and obtaining a feature consistency description vector for each corner point according to the descriptors; performing similarity measurement and corner point matching on the feature consistency description vectors of the visible light image and the coarsely registered SAR image according to a bidirectional matching cost function; removing incorrect matching points, obtaining a high-confidence matching point pair that satisfies geometric consistency constraints, and performing image registration according to the matching point pair to obtain a finely registered SAR image.

Description

Technical Field

[0001] The present invention relates to the technical field of image registration for unmanned aerial vehicle (UAV) platforms, and in particular, to a method and system for registering visible light and SAR images with large viewing angle differences for UAV platforms. Background Art

[0002] UAVs have been widely used in military fields and civilian fields such as search and rescue of field personnel and logistics aerial photography. With the diversification of UAV missions, the imaging of a single UAV sensor cannot meet the requirements of some special tasks. For example, tasks such as target detection and tracking in field operations in rainy and snowy weather and complex interference scenarios such as sand and dust and thick smoke.

[0003] UAVs usually carry SAR (Synthetic Aperture Radar) and visible light sensors. SAR uses an active imaging method, is almost unaffected by weather conditions, has strong penetrability, and can still clearly image in complex scenarios such as thick smoke and sand and dust. The imaging field of view of UAV-borne SAR is large, but it is greatly affected by coherent multiplicative noise and has poor interpretability, making it difficult to complete image interpretation. The imaging of visible light sensors reflects the radiation attributes of ground objects, has rich color details, clear contour textures, and is easy to understand, but it is easily affected by adverse weather such as rain and snow and cannot clearly image in thick smoke and fog scenarios. Fusing visible light and SAR images can provide complementary information of the same scene, give full play to the advantages of different modality images, and is conducive to better completing subsequent advanced vision tasks. Image registration is a key step in image fusion of different modalities, and its purpose is to obtain spatially aligned visible light and SAR images. In the prior art, the registration of visible light and SAR images usually focuses on the field of satellite remote sensing, and there are certain differences in the image registration ideas between the two modalities and UAV platforms. The registration of visible light and SAR images obtained by existing UAV platforms mainly relies on manual operation, and feature points are manually selected to complete image registration. However, due to the differences in imaging principles, the viewing angle differences between the visible light and SAR images obtained by UAVs are large, and there are significant differences in image scale and radiation characteristics. Directly using methods based on gray level and learning cannot complete registration. The method of manual registration is inefficient, consumes a large amount of resources, and can only be carried out offline, and cannot achieve end-to-end deployment and online image registration. That is, in the prior art, it is difficult to fuse unregistered visible light and SAR images obtained by UAVs, and the advantages between the two different modalities cannot be fully utilized.

[0004] Therefore, there is an urgent need for a new technical solution to solve the technical problem of how to quickly and accurately register visible light images and SAR images with large viewing angle differences in UAV platforms. Summary of the Invention

[0005] The present invention provides a method and system for registration of visible light and SAR images with large viewing angle differences for UAV platforms, which are used to solve the technical problem of how to achieve rapid and accurate registration of visible light images and SAR images with large viewing angle differences on UAV platforms.

[0006] To achieve the above objectives, the present invention provides a method for registration of visible light and SAR images with large viewing angle differences for UAV platforms, comprising:

[0007] Visible light images and SAR images of the target area are acquired through a drone; the three equal-division points on opposite sides of the visible light image are connected to form intersection points to obtain four first feature points; the geographic coordinates of the first feature points are solved; and feature points are selected from the SAR image based on the geographic coordinates to obtain four second feature points.

[0008] A matching point pair set is constructed according to the first feature point and the second feature point; a homography transformation matrix is obtained according to the matching point pair set; and the homography transformation matrix is multiplied by the SAR image to obtain a coarsely registered SAR image.

[0009] The corner points of each image are obtained by combining the feature response model and the FAST feature point detection algorithm with the visible light image and the coarsely registered SAR image. Circular descriptors and statistical histograms are constructed for all corner points in the image, and the feature consistency description vector of each corner point is obtained based on the circular descriptors and statistical histograms.

[0010] Based on the bidirectional matching cost function, similarity measurement and corner point matching are performed on the feature consistency description vectors of the visible light image and the coarsely registered SAR image, and the incorrect matching points are eliminated to obtain high-confidence matching point pairs that meet the geometric consistency constraints. Image registration is performed based on the matching point pairs to obtain the finely registered SAR image.

[0011] Preferably, solving the geographic coordinates of the first feature point includes:

[0012] The relationship between the coordinates of the object point P in the northeastern sky geographic coordinate system and the coordinates of the image point includes:

[0013] ;

[0014] in, Represents the camera intrinsic parameter matrix; Indicates that the image point coordinates of the object point P Coordinates converted to the camera coordinate system ; Indicates the coordinates of the camera coordinate system of the object point P Coordinates converted to the drone coordinate system The transformation matrix of Indicates the coordinates of the drone coordinate system of the object point P Coordinates converted to the Northeast Geographical Coordinate System transformation matrix; represents the geographical coordinate position of the UAV, i.e., the longitude and latitude of the origin of the Northeast Geographical Coordinate System.

[0015] Obtain the image point coordinates of the first feature point according to the visible light image, and convert the image point coordinates of the first feature point according to the relationship between the coordinates in the Northeast Geographical Coordinate System and the image point coordinates, then the geographical coordinates of the first feature point can be obtained.

[0016] Preferably, it further includes:

[0017] Camera internal parameter matrix including:

[0018] ;

[0019] wherein, represents the camera focal length; and represent the pixel size of the imaging system; represents the principal point coordinates of the image plane.

[0020] Transformation matrix is calculated through the attitude angles of the camera relative to the UAV, including:

[0021] ;

[0022] wherein, , and [[ID=ZJ=48]]respectively represent the roll angle, pitch angle and yaw angle of the camera, i.e., represent the rotation around the , and axes of the camera coordinate system.

[0023] Transformation matrix is calculated through the UAV attitude angles, including:

[0024] ;

[0025] wherein, , and respectively represent the roll angle, pitch angle and yaw angle of the UAV, i.e., represent the rotation around the , and axes of the UAV coordinate system.

[0026] Preferably, the corner points of each image obtained according to the visible light image and the coarsely registered SAR image in combination with the feature response model and the FAST feature point detection algorithm include:

[0027] Perform a first processing on the visible light image and the coarsely registered SAR image respectively to obtain the corner points of their respective images.

[0028] The first processing includes:

[0029] Perform downsampling on the original image for a preset number of times and at preset multiple scales, and apply smoothing and blurring operations with different preset standard deviations to the image at each scale through a Gaussian filter to obtain the preprocessed image at each scale; obtain a comprehensive feature map based on the preprocessed image at each scale in combination with a feature response model; perform key point extraction based on the comprehensive feature map in combination with the FAST feature point detection algorithm to obtain corner points; the number of corner points is detected by the FAST feature point detection algorithm.

[0030] Preferably, the Gaussian filter includes:

[0031] ;

[0032] where represents the natural base; represents the standard deviation; represents the Gaussian filter.

[0033] Preferably, the feature response model is based on the principle of phase consistency, used to enhance the saliency of the feature map, and accumulate the feature responses in preset directions at all scales, thereby generating a comprehensive feature map containing all scale and direction information.

[0034] Preferably, performing key point extraction based on the comprehensive feature map in combination with the FAST feature point detection algorithm to obtain corner points includes:

[0035] Define the neighborhood as a circular area containing Q pixels, and perform corner point detection by analyzing the gray value comparison within the neighborhood:

[0036] ;

[0037] where represents the gray value of the pixel point ; represents the gray value of the neighborhood pixel point of the pixel point ; represents a preset hyperparameter threshold; if within the neighborhood of the pixel point there are consecutive pixels that satisfy , then it can be determined that the pixel point is a corner point.

[0038] Preferably, circular descriptors and statistical histograms are constructed for all corner points in the image, and the feature consistency description vectors of each corner point are obtained from the circular descriptors and statistical histograms, including:

[0039] Taking the corner point as the center, a circular region with a radius of R is constructed.

[0040] The feature values of each pixel point in the circular region are calculated in a preset number and angle directions, and the direction corresponding to the maximum feature value of each pixel point is marked as the main direction of the pixel point; an XY-axis coordinate system is constructed in the circular region with the corner point as the origin to obtain a direction index map; the direction index map includes the main direction of each pixel point.

[0041] The main directions of each pixel point in the direction index map are statistically counted to obtain a statistical histogram.

[0042] The direction with the highest frequency of occurrence in the statistical histogram is set as the main direction of the corner point.

[0043] With the goal of rotating the main direction of the corner point to the positive X-axis direction, the circular region is rotated.

[0044] Direction correction is performed on all pixel points in the circular region through a four-quadrant voting strategy.

[0045] The circular region is evenly divided into partitions along the radial and tangential directions, and the main direction statistical histograms of all pixel points in each partition are statistically counted to obtain the feature vectors of each partition.

[0046] Starting from the positive X-axis, the feature vectors of each partition are sequentially connected in a clockwise direction from the outer circle to the inner circle to obtain the feature consistency descriptor of the corner point, and the feature consistency descriptor is normalized by the L2 norm to obtain the feature consistency description vector of the corner point.

[0047] All corner points in the image are processed to obtain the feature consistency description vectors of each corner point.

[0048] Preferably, performing direction correction on all pixel points in the circular region through a four-quadrant voting strategy includes:

[0049] If there are pixel point directions in the third or fourth quadrant of the XY-axis coordinate system after the circular region is rotated, the pixel point directions in the third or fourth quadrant of the XY-axis coordinate system are added with a π value.

[0050] The present invention also provides a visible light and SAR large-view-angle difference image registration system for an unmanned aerial vehicle platform, which is used for the method of the present invention. The system includes an unmanned aerial vehicle, SAR, a visible light sensor, and an edge computing platform.

[0051] The SAR, visible light sensor, and edge computing platform are all installed on the unmanned aerial vehicle; both the SAR and the visible light sensor are connected to the edge computing platform; the SAR is used to acquire SAR images and send the SAR images to the edge computing platform; the visible light sensor is used to acquire visible light images and send the visible light images to the edge computing platform.

[0052] The edge computing platform is used to connect the trisection points on the opposite sides of the visible light image to form intersection points, obtaining 4 first feature points; solve the geographical coordinates of the first feature points; select feature points from the SAR image according to the geographical coordinates, obtaining 4 second feature points.

[0053] The edge computing platform is also used to construct a set of matching point pairs based on the first feature points and the second feature points; obtain a homography transformation matrix according to the set of matching point pairs; multiply the homography transformation matrix with the SAR image to obtain a roughly registered SAR image.

[0054] The edge computing platform is also used to obtain the corner points of their respective images by combining the visible light image and the roughly registered SAR image with a feature response model and the FAST feature point detection algorithm; construct a circular descriptor and a statistical histogram for all the corner points in the image, and obtain a feature consistency description vector for each corner point according to the circular descriptor and the statistical histogram.

[0055] The edge computing platform is also used to perform similarity measurement and corner point matching on the feature consistency description vectors of the visible light image and the roughly registered SAR image according to a bidirectional matching cost function, eliminate the mis-matched points, obtain a set of high-confidence matching point pairs that satisfy the geometric consistency constraint, and perform image registration according to the matching point pairs to obtain a precisely registered SAR image.

[0056] The present invention has the following beneficial effects:

[0057] The visible light and SAR large-view-angle difference image registration method for an unmanned aerial vehicle (UAV) platform according to the present invention realizes end-to-end image registration of the UAV platform through a two-stage method of coarse registration first and then fine registration. Coarse registration solves the geographical location information of key feature points in the visible light image, constructs matching point pairs to calculate the transformation matrix, and obtains the coarsely registered SAR image. Then, the best matching point pairs are obtained through multi-scale feature point selection, feature consistency description vectors, and an adaptive matching strategy, thereby obtaining the finely registered SAR image. Through the registration of the two stages, the registration result has high precision. The method of the present invention does not need to rely on the way of manually extracting features for image registration, can run online, and further realizes the end-to-end deployment of the UAV platform. The registration speed is significantly improved, the registration accuracy is improved, and it can effectively solve the problem of unable to perform online registration caused by the view angle and modality differences between the visible light and SAR images obtained by the UAV platform, and is suitable for subsequent image fusion using the different modality advantages of visible light and SAR images. At the same time, the method of the present invention has high operating efficiency, breaks through the practical application barriers of efficient online image registration of visible light and SAR images on the UAV platform, and provides a solution for the practical registration of visible light and SAR images with large view angle differences. The method of the present invention can realize fast and accurate registration of visible light images and SAR images with large view angle differences in the UAV platform, and supports the efficient operation of visible light and SAR image registration on the UAV platform.

[0058] The visible light and SAR large-view-angle difference image registration system for an unmanned aerial vehicle (UAV) platform according to the present invention is used for the method of the present invention and has the same beneficial effects as the method of the present invention.

[0059] In addition to the purposes, features, and advantages described above, the present invention has other purposes, features, and advantages. The following will refer to the accompanying drawings and make a further detailed description of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0061] Figure 1 is a schematic flowchart of the method of the preferred embodiment of the present invention.

[0062] Figure 2 is a schematic diagram for generating the feature consistency description vector of the preferred embodiment of the present invention.

[0063] Figure 3 is an input visible light reference image obtained by the airborne visible light sensor of the preferred embodiment of the present invention.

[0064] Figure 4It is the SAR image to be registered obtained by the airborne SAR in the preferred embodiment of the present invention.

[0065] Figure 5 It is a schematic diagram of the rough registration SAR result in the preferred embodiment of the present invention.

[0066] Figure 6 It is the registration result checkerboard diagram in the preferred embodiment of the present invention.

[0067] Figure 7 It is a schematic diagram of the correct matching point pair result in the preferred embodiment of the present invention. Detailed implementation manners

[0068] The following will describe the embodiments of the present invention in detail with reference to the drawings. However, the present invention can be implemented in many different ways defined and covered by the claims.

[0069] See Figure 1 , in the preferred embodiment of the present invention, a method for registering visible light and SAR large-view difference images for an unmanned aerial vehicle (UAV) platform is provided, including:

[0070] S1. Obtain a visible light image and an SAR image of a target area through a UAV; connect the corresponding trisection points on the edges of the visible light image to form intersection points, and obtain 4 first feature points; solve the geographical coordinates of the first feature points; select feature points from the SAR image according to the geographical coordinates to obtain 4 second feature points.

[0071] In S1, solving the geographical coordinates of the first feature points includes:

[0072] The relationship between the coordinates of the object point P in the northeast-up geographical coordinate system and the image point coordinates includes:

[0073] ;

[0074] Among them, represents the camera internal parameter matrix; represents converting the image point coordinates of the object point P to the coordinates in the camera coordinate system; represents the transformation matrix for converting the coordinates of the object point P in the camera coordinate system to the coordinates in the UAV coordinate system; represents the transformation matrix for converting the coordinates

[0075] ​​Obtain the image point coordinates of the first feature point from the visible light image, and convert the image point coordinates of the first feature point according to the relationship between the coordinates in the northeast celestial geographic coordinate system and the image point coordinates, then the geographic coordinates of the first feature point can be obtained.

[0076] In a preferred embodiment of the present invention, the camera internal parameter matrix K includes:

[0077] ;

[0078] Among them, represents the camera focal length; and represent the pixel size of the imaging system; represents the principal point coordinates of the image plane.

[0079] In a preferred embodiment of the present invention, the transformation matrix is obtained by calculating the attitude angle of the camera relative to the UAV, and includes:

[0080] ;

[0081] Among them, , and respectively represent the roll angle, pitch angle and yaw angle of the camera, that is, represent the rotation around the , and axes of the camera coordinate system.

[0082] In a preferred embodiment of the present invention, the transformation matrix is obtained by calculating the UAV attitude angle, and includes:

[0083] ;

[0084] Among them, , and respectively represent the roll angle, pitch angle and yaw angle of the UAV, that is, represent the rotation around the , and axes of the UAV coordinate system.

[0085] S2. Construct a set of matching point pairs according to the first feature point and the second feature point; obtain the homography transformation matrix according to the set of matching point pairs; multiply the homography transformation matrix by the SAR image to obtain the coarsely registered SAR image.

[0086] S3. Combine the visible light image and the coarsely registered SAR image, and use the feature response model and the FAST feature point detection algorithm to obtain the corner points of each image; construct circular descriptors and statistical histograms for all the corner points in the image, and obtain the feature consistency description vectors of each corner point according to the circular descriptors and the statistical histograms.

[0087] In S3, obtaining the corner points of each image by combining the visible light image and the coarsely registered SAR image, and using the feature response model and the FAST feature point detection algorithm includes:

[0088] Perform the first processing on the visible light image and the coarsely registered SAR image respectively to obtain the corner points of each image.

[0089] The first processing includes:

[0090] Downsample the original image for a preset number of times and at a preset multiple of scales, and apply smoothing and blurring operations with different preset standard deviations to the image at each scale through a Gaussian filter to obtain the preprocessed image at each scale; obtain the comprehensive feature map according to the preprocessed image at each scale and the feature response model; perform key point extraction according to the comprehensive feature map and the FAST feature point detection algorithm to obtain the corner points; the number of corner points is detected by the FAST feature point detection algorithm.

[0091] In a preferred embodiment of the present invention, the downsampling for a preset number of times and at a preset multiple of scales includes four times of downsampling with sizes of 1 / 2, 1 / 4, 1 / 8, and 1 / 16.

[0092] In a preferred embodiment of the present invention, the Gaussian filter includes:

[0093] ;

[0094] where represents the natural logarithm base; represents the standard deviation; represents the Gaussian filter.

[0095] In a preferred embodiment of the present invention, the feature response model is based on the principle of phase consistency, which is used to enhance the significance of the feature mapping and accumulate the feature responses in the preset directions at all scales, and then generate a comprehensive feature map containing all scale and direction information.

[0096] In a preferred embodiment of the present invention, performing key point extraction according to the comprehensive feature map and the FAST feature point detection algorithm to obtain the corner points includes:

[0097] Define the neighborhood as a circular area containing Q pixels, and perform corner point detection by analyzing the gray value comparison within the neighborhood:

[0098] ;

[0099] Among them, represents the grayscale value of the pixel point ; represents the grayscale value of the neighboring pixel points of the pixel point ; represents a preset hyperparameter threshold; if within the neighborhood of the pixel point there are consecutive pixels that satisfy , then the pixel point can be determined as a corner point. In a preferred embodiment of the present invention, the range of Q is preferably 50 to 200 pixels, and Q is a positive integer.

[0100] See Figure 2 , in S3, construct a circular descriptor and a statistical histogram for all corner points in the image, and obtain the feature consistency description vector of each corner point according to the circular descriptor and the statistical histogram, including:

[0101] With the corner point as the center, construct a circular region with a radius of R to describe rotational invariance; the unit of R is pixels. In a preferred embodiment of the present invention, the range of R is preferably 36 to 108, and R is a positive integer.

[0102] Calculate the feature values of each pixel point in the circular region in a preset number and angle directions, and mark the direction corresponding to the maximum feature value of each pixel point as the main direction of the pixel point; construct an XY-axis coordinate system in the circular region with the corner point as the origin to obtain a direction index map; the direction index map includes the main direction of each pixel point. In a preferred embodiment of the present invention, the preset number and angle directions include 6 angle directions of 0°, 30°, 60°, 90°, 120°, and 150°, that is Figure 2 in θ 1 to θ 6.

[0103] Statistically count the main direction of each pixel point in the direction index map to obtain a statistical histogram.

[0104] Set the direction with the highest frequency of occurrence in the statistical histogram as the main direction of the corner point to complete the main direction estimation.

[0105] With the goal of rotating the main direction of the corner point to the positive X-axis direction, rotate the circular region.

[0106] Perform pixel point direction correction on all pixel points in the circular region through a four-quadrant voting strategy, including:

[0107] If there are pixel point directions in the third or fourth quadrant of the XY-axis coordinate system after the circular region is rotated, then add a π value to the pixel point directions in the third or fourth quadrant of the XY-axis coordinate system.

[0108] The circular region is evenly divided into partitions along the radial and tangential directions, and the main-direction statistical histogram of all pixel points in each partition is calculated to obtain the feature vector of each partition; starting from the positive half-axis of the X-axis, the feature vectors of each partition are sequentially connected in a clockwise direction from the outer circle to the inner circle to obtain the feature consistency descriptor of the corner points.

[0109] The feature consistency descriptor of the corner points is normalized by the L2 norm to obtain the feature consistency description vector of the corner points.

[0110] All corner points in the image are processed to obtain the feature consistency description vectors of each corner point.

[0111] S4. According to the bidirectional matching cost function, the feature consistency description vectors of the visible light image and the coarsely registered SAR image are subjected to similarity measurement and corner point matching, the mis-matched points are eliminated, and the high-confidence matching point pairs that satisfy the geometric consistency constraint are obtained. Image registration is performed according to the matching point pairs to obtain the finely registered SAR image.

[0112] In the preferred embodiment of the present invention, the bidirectional matching cost function is based on an adaptive association strategy.

[0113] In the preferred embodiment of the present invention, when eliminating the mis-matched points, the random sample consensus algorithm is used for elimination.

[0114] The visible light and SAR large-viewpoint difference image registration method for an unmanned aerial vehicle (UAV) platform according to the present invention realizes end-to-end image registration of the UAV platform through a two-stage method of coarse registration first and then fine registration. Coarse registration is achieved by solving the geographical location information of the key feature points of the visible light image, constructing a matching point pair to calculate the transformation matrix, and obtaining the coarsely registered SAR image. Then, the best matching point pair is obtained through multi-scale feature point selection, feature consistency description vector, and adaptive matching strategy, so as to obtain the finely registered SAR image. Through the registration of the two stages, the registration result has high precision. The method of the present invention does not need to rely on the way of manually extracting features for image registration, can run online, and further realizes the end-to-end deployment of the UAV platform. The registration speed is significantly improved, the registration accuracy is improved, and the problem of inability to perform online registration caused by the viewpoint and modality differences between the visible light and SAR images obtained by the UAV platform can be effectively solved, which is suitable for subsequent image fusion using the different modality advantages of the visible light and SAR images; at the same time, the method of the present invention has high operation efficiency, breaks through the practical application barrier of efficient online image registration of visible light and SAR images on the UAV platform, and provides a solution for the practical application of registering visible light and SAR images with large viewpoint differences. The method of the present invention can realize fast and accurate registration of visible light images and SAR images with large viewpoint differences in the UAV platform, and supports the efficient operation of visible light and SAR image registration on the UAV platform.

[0115] In a preferred embodiment of the present invention, there is also provided a visible light and SAR large-angle difference image registration system for a drone platform, which is used for the method of the present invention. The system includes a drone, an SAR, a visible light sensor, and an edge computing platform;

[0116] The SAR, the visible light sensor, and the edge computing platform are all installed on the drone; both the SAR and the visible light sensor are connected to the edge computing platform; the SAR is used to acquire an SAR image and send the SAR image to the edge computing platform; the visible light sensor is used to acquire a visible light image and send the visible light image to the edge computing platform.

[0117] The edge computing platform is used to connect the corresponding trisection points on the edge of the visible light image to form intersection points, obtaining 4 first feature points; solving the geographical coordinates of the first feature points; and selecting feature points from the SAR image according to the geographical coordinates, obtaining 4 second feature points.

[0118] The edge computing platform is also used to construct a set of matching point pairs according to the first feature points and the second feature points; obtaining a homography transformation matrix according to the set of matching point pairs; and multiplying the homography transformation matrix by the SAR image to obtain a roughly registered SAR image.

[0119] The edge computing platform is also used to obtain the corner points of their respective images according to the visible light image and the roughly registered SAR image in combination with the feature response model and the FAST feature point detection algorithm; constructing a circular descriptor and a statistical histogram for all the corner points in the image, and obtaining a feature consistency description vector for each corner point according to the circular descriptor and the statistical histogram.

[0120] The edge computing platform is also used to perform similarity measurement and corner point matching on the feature consistency description vectors of the visible light image and the roughly registered SAR image according to the bidirectional matching cost function, eliminating the wrongly matched points, obtaining a set of high-confidence matching point pairs that meet the geometric consistency constraint, and performing image registration according to the matching point pairs to obtain a precisely registered SAR image.

[0121] The visible light and SAR large-angle difference image registration system for a drone platform of the present invention, which is used for the method of the present invention, has the same beneficial effects as the method of the present invention.

[0122] Verification part:

[0123] In a preferred embodiment of the present invention, experimental tests are carried out at a certain experimental site. The area of the test site is about 1,200,000 square meters, and the test scenario covers ground and water areas. The SAR band used by the drone is the KU band, with a frequency between 12.5 and 18 GHz, and the working system is frequency-modulated continuous wave. The visible light band is between 380 and 750 nm, and the obtained video frame rate is 30 Hz, and the resolution is 1920×1080 pixels.

[0124] In the experiment of the preferred embodiment of the present invention, Figure 3 is the input visible light reference image acquired by the airborne visible light sensor. Figure 4 is the input SAR image to be registered acquired by the airborne SAR. Region A represents the region where the SAR image and the visible light image roughly correspond. Figure 5 are the points selected from the visible light image and the SAR image with the same geographical coordinates. After matrix transformation, the coarsely registered SAR result is obtained. From Figures 3 to 5 it can be seen that the coarsely registered SAR image basically establishes a rough regional alignment with the visible light image in space.

[0125] In the experiment of the preferred embodiment of the present invention, Figure 6 is the checkerboard image of the final registration result through the coarse registration stage and the fine registration stage. Figure 7 are the correct matching point pair results of the SAR image and the visible light image after final registration. It can be seen that the method of the present invention can meet the actual application requirements, realize the precise registration and alignment of visible light and SAR images under the background of large viewing angle differences, overcome the problem that the SAR and visible light images acquired by the UAV platform cannot be registered online, resulting in the inability to perform image fusion, and can achieve efficient and accurate image registration functions through end-to-end deployment and online operation within the large field of view of the UAV, and has strong application value.

[0126] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A visible light and SAR large viewing angle difference image registration method for an unmanned aerial vehicle platform, characterized in that, Including: Obtaining visible light images and SAR images of a target area by means of an unmanned aerial vehicle; Correspondingly connecting the trisection points on the opposite sides of the visible light image to form intersection points, thereby obtaining 4 first feature points; Solving the geographical coordinates of the first feature points; selecting feature points from the SAR image according to the geographical coordinates to obtain 4 second feature points; Constructing a set of matching point pairs according to the first feature points and the second feature points; obtaining a homography transformation matrix according to the set of matching point pairs; multiplying the homography transformation matrix by the SAR image to obtain a roughly registered SAR image; Obtaining the corner points of each image according to the visible light image and the roughly registered SAR image by combining a feature response model and a FAST feature point detection algorithm; constructing a circular descriptor and a statistical histogram for all the corner points in the image, and obtaining a feature consistency description vector for each corner point according to the circular descriptor and the statistical histogram, including: Constructing a circular area with a radius of R with the corner point as the center; Calculating the feature values of each pixel point in the circular area in a preset number and at preset angles, and marking the direction corresponding to the maximum feature value of each pixel point as the main direction of the pixel point; constructing an XY axis coordinate system in the circular area with the corner point as the origin to obtain a direction index map; the direction index map includes the main direction of each pixel point; Counting the main directions of each pixel point in the direction index map to obtain a statistical histogram; Setting the direction with the highest occurrence frequency in the statistical histogram as the main direction of the corner point; Rotating the circular area with the aim of rotating the main direction of the corner point to the positive X-axis direction; Performing direction correction on all the pixel points in the circular area through a four-quadrant voting strategy; Uniformly dividing the circular area into partitions along the radial and tangential directions, and counting the main direction statistical histograms of all the pixel points in each partition to obtain the feature vectors of each partition; Starting from the positive X-axis, sequentially connecting the feature vectors of each partition in a clockwise direction from the outer circle to the inner circle to obtain a feature consistency descriptor of the corner point, and obtaining a feature consistency description vector of the corner point according to the feature consistency descriptor; Performing similarity measurement and corner point matching on the feature consistency description vectors of the visible light image and the roughly registered SAR image according to a bidirectional matching cost function, eliminating incorrect matching points, obtaining high-confidence matching point pairs that satisfy geometric consistency constraints, and performing image registration according to the matching point pairs to obtain a precisely registered SAR image.

2. The visible light and SAR large view angle difference image registration method for an unmanned aerial vehicle platform according to claim 1, wherein Solving the geographical coordinates of the first feature points includes: The relationship between the coordinates of the object point P in the northeast-up geographical coordinate system and the image point coordinates includes: ； Among them, represents the camera internal parameter matrix; represents converting the image point coordinates of the object point P into the coordinates in the camera coordinate system ; represents converting the coordinates of the object point P in the camera coordinate system into the coordinates in the UAV coordinate system transformation matrix; represents converting the coordinates of the object point P in the UAV coordinate system into the coordinates in the north-east-down geographic coordinate system transformation matrix; represents the geographic coordinate position of the UAV, that is, the longitude and latitude of the origin of the north-east-down coordinate system; Obtaining the image point coordinates of the first feature point according to the visible light image, and converting the image point coordinates of the first feature point according to the relationship between the coordinates of the northeast-up geographical coordinate system and the image point coordinates, thereby obtaining the geographical coordinates of the first feature point.

3. The visible light and SAR large view angle difference image registration method for an unmanned aerial vehicle platform according to claim 2, characterized in that Also including: The camera intrinsic matrix comprises: ； Among them, represents the camera focal length; and represent the pixel size of the imaging system; represents the principal point coordinates of the image plane; The transformation matrix is calculated based on the attitude angles of the camera relative to the drone, and includes: ； Among them, , and respectively represent the roll angle, pitch angle, and yaw angle of the camera, that is, the rotation around the , and axes of the camera coordinate system; The transformation matrix is obtained by calculating the attitude angles of the UAV and includes: ； Among them, , and respectively represent the roll angle, pitch angle and yaw angle of the UAV, that is, the rotation about the , and axes of the UAV coordinate system.

4. The visible light and SAR large-view-angle difference image registration method for an unmanned aerial vehicle platform according to claim 3, wherein Obtaining the corner points of each image according to the visible light image and the roughly registered SAR image by combining a feature response model and a FAST feature point detection algorithm includes: Perform a first processing on the visible light image and the coarsely registered SAR image respectively to obtain the corner points of their respective images; The first processing includes: Perform downsampling of the original image with a preset number of times and preset multi-scales, and apply smoothing and blurring operations with different preset standard deviations to the images at each scale through a Gaussian filter to obtain the preprocessed images at each scale; obtain a comprehensive feature map according to the preprocessed images at each scale in combination with a feature response model; perform key point extraction according to the comprehensive feature map in combination with the FAST feature point detection algorithm to obtain the corner points; the number of the corner points is detected by the FAST feature point detection algorithm.

5. The visible light and SAR large-viewpoint difference image registration method for an unmanned aerial vehicle platform according to claim 4, wherein The Gaussian filter includes: ； Among them, represents the natural base; represents the standard deviation; represents the Gaussian filter.

6. The visible light and SAR large-view-angle difference image registration method for an unmanned aerial vehicle platform according to claim 5, wherein The feature response model is based on the principle of phase consistency, which is used to enhance the significance of feature mapping and accumulate the feature responses in the preset directions at all scales, and then generate a comprehensive feature map containing all scale and direction information.

7. The visible light and SAR large-view-angle difference image registration method for an unmanned aerial vehicle platform according to claim 6, wherein Performing key point extraction according to the comprehensive feature map in combination with the FAST feature point detection algorithm to obtain the corner points includes: Define the neighborhood as a circular area containing Q pixels, and perform corner point detection by analyzing the gray value comparison within the neighborhood: ； Among them, represents the grayscale value of the pixel point ; represents the grayscale value of the neighboring pixel points of the pixel point ; represents a preset hyperparameter threshold; if there are consecutive pixels in the neighborhood of the pixel point satisfy , the pixel point can be determined as a corner point.

8. The visible light and SAR large-view-angle difference image registration method for an unmanned aerial vehicle platform according to claim 7, wherein The direction correction of all pixel points within the circular area by the four-quadrant voting strategy includes: ​ 9. A visible light and SAR large view angle difference image registration system for a drone platform, used for the method according to any one of claims 1 to 8, characterized in that, ​ ​ ​ ​ ​ ​ Calculate the eigenvalues of each pixel point in the circular region in the directions of a preset number and angles, and mark the direction corresponding to the maximum eigenvalue of each pixel point as the main direction of the pixel point; construct an XY-axis coordinate system in the circular region with the corner point as the origin to obtain a direction index map; the direction index map includes the main direction of each pixel point; Statistically analyze the main directions of each pixel point in the direction index map to obtain a statistical histogram; Set the direction with the highest frequency of occurrence in the statistical histogram as the main direction of the corner point; Rotate the circular region with the goal of rotating the main direction of the corner point to the positive X-axis direction; Perform direction correction on all pixel points in the circular region through a four-quadrant voting strategy; Uniformly divide the circular region into partitions along the radial and tangential directions, and statistically analyze the main direction statistical histograms of all pixel points in each partition to obtain the feature vectors of each partition; Starting from the positive X-axis, connect the feature vectors of each partition in a clockwise direction from the outer circle to the inner circle to obtain the feature consistency descriptor of the corner point, and obtain the feature consistency description vector of the corner point according to the feature consistency descriptor; The edge computing platform is further configured to perform similarity measurement and corner point matching on the feature consistency description vectors of the visible light image and the coarsely registered SAR image according to a bidirectional matching cost function, eliminate mis-matched points, obtain high-confidence matching point pairs that satisfy geometric consistency constraints, and perform image registration according to the matching point pairs to obtain a finely registered SAR image.

Images