Visible light and SAR large view angle difference image registration method and system for unmanned aerial vehicle platform

Abstract

The invention discloses an unmanned aerial vehicle platform-oriented visible light and SAR large-view-angle difference image registration method and system. The method comprises the steps of constructing a matching point pair set according to a visible light image and an SAR image; obtaining a homography transformation matrix according to the matching point pair set; multiplying the homography transformation matrix by the SAR image to obtain a coarse registration SAR image; obtaining respective angular points according to the visible light image and the coarse registration SAR image in combination with a feature response model and a FAST feature point detection algorithm; constructing descriptors for all angular points in the image, and obtaining feature consistency description vectors of the angular points according to the descriptors; performing similarity measurement and corner matching on the feature consistency description vectors of the visible light image and the coarse registration SAR image according to a bidirectional matching cost function; and removing wrong matching points to obtain high-confidence matching point pairs meeting geometric consistency constraints, and carrying out image registration according to the matching point pairs to obtain a precisely registered SAR image.

Description

Technical Field

[0001] The present invention relates to the technical field of image registration for unmanned aerial vehicle (UAV) platforms, and particularly to a method and system for registering visible light and SAR images with large viewing angle differences for UAV platforms. Background Art

[0002] UAVs have been widely used in military fields and civilian fields such as search and rescue of field personnel and logistics aerial photography. With the diversification of UAV missions, the imaging of a single UAV sensor cannot meet the requirements of some special tasks. For example, tasks such as target detection and tracking in field operations in rainy and snowy weather and complex interference scenarios such as sand and thick smoke.

[0003] UAVs usually carry SAR (Synthetic Aperture Radar) and visible light sensors. SAR uses an active imaging method, is almost unaffected by weather conditions, has strong penetrability, and can still clearly image in complex scenarios such as thick smoke and sand. The imaging field of view of UAV-borne SAR is large, but it is greatly affected by coherent multiplicative noise and has poor interpretability, making it difficult to complete image interpretation. The imaging of visible light sensors reflects the radiation properties of ground objects, has rich color details, clear contour textures, and is easy to understand, but it is easily affected by bad weather such as rain and snow and cannot clearly image in thick smoke and fog scenarios. Fusing visible light and SAR images can provide complementary information of the same scene, give full play to the advantages of different modality images, and is conducive to better completing subsequent advanced vision tasks. Image registration is a key step in image fusion of different modalities, and its purpose is to obtain spatially aligned visible light and SAR images. In the prior art, the registration of visible light and SAR images usually focuses on the field of satellite remote sensing, and there are certain differences in the image registration ideas between the two modalities and UAV platforms. The registration of visible light and SAR images obtained by existing UAV platforms mainly relies on manual operation, and manually selects feature points to complete image registration. However, due to the differences in imaging principles, the viewing angle differences between the visible light and SAR images obtained by UAVs are large, and there are significant differences in image scale and radiation characteristics. Directly using methods based on gray level and learning cannot complete the registration. The method of manual registration is inefficient, consumes a large amount of resources, and can only be carried out offline, and cannot achieve end-to-end deployment and online image registration. That is, in the prior art, it is difficult to fuse unregistered visible light and SAR images obtained by UAVs, and the advantages between the two different modalities cannot be fully utilized.

[0004] Therefore, there is an urgent need for a new technical solution to solve the technical problem of how to quickly and accurately register visible light images and SAR images with large viewing angle differences in UAV platforms. Summary of the Invention

[0005] The present invention provides a method and system for registering visible light and SAR large-angle difference images for an unmanned aerial vehicle (UAV) platform, aiming to solve the technical problem of how to quickly and accurately register visible light images and SAR images with large-angle differences in the UAV platform.

[0006] To achieve the above object, the present invention provides a method for registering visible light and SAR large-angle difference images for an unmanned aerial vehicle (UAV) platform, including: Obtaining a visible light image and an SAR image of a target area through an unmanned aerial vehicle; connecting the corresponding trisection points on the opposite sides of the visible light image to form intersection points, and obtaining 4 first feature points; solving the geographical coordinates of the first feature points; and selecting feature points from the SAR image according to the geographical coordinates to obtain 4 second feature points.

[0007] Constructing a set of matching point pairs according to the first feature points and the second feature points; obtaining a homography transformation matrix according to the set of matching point pairs; and multiplying the homography transformation matrix by the SAR image to obtain a roughly registered SAR image.

[0008] Obtaining the corner points of each image according to the visible light image and the roughly registered SAR image in combination with a feature response model and a FAST feature point detection algorithm; constructing a circular descriptor and a statistical histogram for all the corner points in the image, and obtaining a feature consistency description vector for each corner point according to the circular descriptor and the statistical histogram.

[0009] Performing similarity measurement and corner point matching on the feature consistency description vectors of the visible light image and the roughly registered SAR image according to a bidirectional matching cost function, removing the incorrect matching points, obtaining a set of high-confidence matching point pairs that satisfy the geometric consistency constraint, and performing image registration according to the matching point pairs to obtain a finely registered SAR image.

[0010] Preferably, solving the geographical coordinates of the first feature points includes: The relationship between the coordinates of the object point P in the northeast-up geographical coordinate system and the image point coordinates includes: ; wherein, represents the camera internal parameter matrix; represents converting the image point coordinates of the object point P to the coordinates in the camera coordinate system; represents the transformation matrix for converting the coordinates of the object point P in the camera coordinate system to the coordinates in the UAV coordinate system; Indicates the geographical coordinate system position of the UAV, i.e., the longitude and latitude of the origin of the northeast-up coordinate system.

[0011] Obtain the image point coordinates of the first feature point according to the visible light image, and convert the image point coordinates of the first feature point according to the relationship between the coordinates in the northeast-up geographical coordinate system and the image point coordinates, then the geographical coordinates of the first feature point can be obtained.

[0012] Preferably, it further includes: Camera internal parameter matrix Includes: ; Wherein, Indicates the camera focal length; And Indicates the pixel size of the imaging system; Indicates the principal point coordinates of the image plane.

[0013] Transformation matrix Calculated through the attitude angles of the camera relative to the UAV, including: ; Wherein, , And Respectively represent the roll angle, pitch angle and yaw angle of the camera, that is, represent the rotation around the , And Axes of the camera coordinate system.

[0014] Transformation matrix Calculated through the UAV attitude angles, including: ; Wherein, , And Respectively represent the roll angle, pitch angle and yaw angle of the UAV, that is, represent the rotation around the , And Axes of the UAV coordinate system.

[0015] Preferably, obtaining the corner points of their respective images according to the visible light image and the coarsely registered SAR image in combination with the feature response model and the FAST feature point detection algorithm includes: Perform the first processing on the visible light image and the coarsely registered SAR image respectively to obtain the corner points of their respective images.

[0016] The first processing includes: The original image is downsampled a preset number of times and at a preset number of scales, and a Gaussian filter is used to apply smoothing and blurring operations with different preset standard deviations to the images at each scale, obtaining preprocessed images at each scale; a comprehensive feature map is obtained based on the preprocessed images at each scale in combination with a feature response model; key points are extracted according to the comprehensive feature map in combination with the FAST feature point detection algorithm, obtaining corner points; the number of corner points is detected by the FAST feature point detection algorithm.

[0017] Preferably, the Gaussian filter includes: ; wherein, represents the natural base; represents the standard deviation; represents the Gaussian filter.

[0018] Preferably, the feature response model is based on the principle of phase consistency, and is used to enhance the saliency of the feature map and accumulate the feature responses in a preset direction at all scales, thereby generating a comprehensive feature map containing all scale and direction information.

[0019] Preferably, extracting key points according to the comprehensive feature map in combination with the FAST feature point detection algorithm to obtain corner points includes: Define the neighborhood as a circular region containing Q pixels, and perform corner point detection by analyzing the gray value comparison within the neighborhood: ; wherein, represents the gray value of pixel point ; represents the gray value of the neighborhood pixel point of pixel point ; represents a preset hyperparameter threshold; if within the neighborhood of pixel point there are consecutive pixels that satisfy , then pixel point can be determined to be a corner point.

[0020] Preferably, constructing a circular descriptor and a statistical histogram for all corner points within the image, and obtaining a feature consistency description vector for each corner point according to the circular descriptor and the statistical histogram includes: Construct a circular region with a radius of R centered at the corner point.

[0021] Calculate the feature values of each pixel point within the circular region in a preset number and angle of directions, and mark the direction corresponding to the maximum feature value of each pixel point as the main direction of the pixel point; construct an XY axis coordinate system within the circular region with the corner point as the origin, obtaining a direction index map; the direction index map includes the main direction of each pixel point.

[0022] Statistically calculate the main direction of each pixel in the orientation index diagram to obtain a statistical histogram.

[0023] Set the direction with the highest frequency of occurrence in the statistical histogram as the main direction of the corner point.

[0024] Aim to rotate the circular region so that the main direction of the corner point is rotated to the positive X-axis direction.

[0025] Perform direction correction on all pixels in the circular region through a four-quadrant voting strategy.

[0026] Evenly divide the circular region into partitions along the radial and tangential directions, and statistically calculate the main direction statistical histogram of all pixels in each partition to obtain the feature vector of each partition.

[0027] Starting from the positive X-axis, connect the feature vectors of each partition in a clockwise direction from the outer circle to the inner circle to obtain the feature consistency descriptor of the corner point. Normalize the feature consistency descriptor through the L2 norm to obtain the feature consistency descriptor vector of the corner point.

[0028] Process all corner points in the image to obtain the feature consistency descriptor vectors of each corner point.

[0029] Preferably, performing direction correction on all pixels in the circular region through a four-quadrant voting strategy includes: If there are pixel directions in the third or fourth quadrant of the XY-axis coordinate system after the circular region is rotated, add a π value to the pixel directions in the third or fourth quadrant of the XY-axis coordinate system.

[0030] The present invention also provides a visible light and SAR large-view angle difference image registration system for an unmanned aerial vehicle (UAV) platform, which is used for the method of the present invention. The system includes a UAV, an SAR, a visible light sensor, and an edge computing platform.

[0031] The SAR, visible light sensor, and edge computing platform are all installed on the UAV; the SAR and visible light sensor are both connected to the edge computing platform; the SAR is used to acquire an SAR image and send the SAR image to the edge computing platform; the visible light sensor is used to acquire a visible light image and send the visible light image to the edge computing platform.

[0032] The edge computing platform is used to connect the corresponding trisection points on the sides of the visible light image to form intersection points to obtain 4 first feature points; solve the geographical coordinates of the first feature points; select feature points from the SAR image according to the geographical coordinates to obtain 4 second feature points.

[0033] The edge computing platform is also used to construct a set of matching point pairs based on the first feature points and the second feature points; obtain a homography transformation matrix according to the set of matching point pairs; and multiply the homography transformation matrix by the SAR image to obtain a roughly registered SAR image.

[0034] The edge computing platform is also used to obtain the corner points of each image according to the visible light image and the roughly registered SAR image in combination with the feature response model and the FAST feature point detection algorithm; construct circular descriptors and statistical histograms for all the corner points in the image, and obtain the feature consistency description vectors of each corner point according to the circular descriptors and the statistical histograms.

[0035] The edge computing platform is also used to perform similarity measurement and corner point matching on the feature consistency description vectors of the visible light image and the roughly registered SAR image according to the bidirectional matching cost function, eliminate the wrong matching points, obtain high-confidence matching point pairs that meet the geometric consistency constraints, and perform image registration according to the matching point pairs to obtain a precisely registered SAR image.

[0036] The present invention has the following beneficial effects: The visible light and SAR large-view-angle difference image registration method for an unmanned aerial vehicle (UAV) platform according to the present invention realizes end-to-end image registration of the UAV platform in a two-stage manner of rough registration first and then precise registration. The rough registration solves the geographical location information of the key feature points of the visible light image, constructs a matching point pair to calculate the transformation matrix, and obtains a roughly registered SAR image. Then, the best matching point pair is obtained through multi-scale feature point selection, feature consistency description vector, and adaptive matching strategy, so as to obtain a precisely registered SAR image. Through the registration of the two stages, the registration result has high precision. The method of the present invention does not need to rely on the way of manually extracting features for image registration, can run online, and further realizes the deployment of the end-to-end UAV platform. The registration speed is significantly improved, the registration accuracy is improved, and it can effectively solve the problem of inability to perform online registration caused by the view angle and modality differences between the visible light and SAR images obtained by the UAV platform, and is suitable for subsequent image fusion by taking advantage of the different modalities of the visible light and SAR images. At the same time, the method of the present invention has high operation efficiency, breaks through the practical application barrier of efficient online image registration of visible light and SAR images on the UAV platform, and provides a solution for the practical application of visible light and SAR images with large view angle differences. The method of the present invention can realize fast and accurate registration of visible light images and SAR images with large view angle differences in the UAV platform, and supports the efficient operation of visible light and SAR image registration on the UAV platform.

[0037] The visible light and SAR large-view-angle difference image registration system for an unmanned aerial vehicle (UAV) platform according to the present invention is used for the method of the present invention and has the same beneficial effects as the method of the present invention.

[0038] In addition to the purposes, features and advantages described above, the present invention has other purposes, features and advantages. The following will further describe the present invention in detail with reference to the accompanying drawings. Description of the Drawings

[0039] The drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings: Figure 1 is a schematic flowchart of the method of the preferred embodiment of the present invention.

[0040] Figure 2 is a schematic diagram for generating a feature consistency description vector of the preferred embodiment of the present invention.

[0041] Figure 3 is an input visible light reference image obtained by an airborne visible light sensor of the preferred embodiment of the present invention.

[0042] Figure 4 is a SAR image to be registered obtained by an airborne SAR of the preferred embodiment of the present invention.

[0043] Figure 5 is a schematic diagram of the rough registration SAR result of the preferred embodiment of the present invention.

[0044] Figure 6 is a registration result checkerboard diagram of the preferred embodiment of the present invention.

[0045] Figure 7 is a schematic diagram of the correct matching point pair result of the preferred embodiment of the present invention. Detailed Embodiment

[0046] The following will describe the embodiments of the present invention in detail with reference to the accompanying drawings. However, the present invention can be implemented in many different ways defined and covered by the claims.

[0047] Refer to Figure 1 , in the preferred embodiment of the present invention, a method for registering visible light and SAR large-angle difference images for an unmanned aerial vehicle platform is provided, including: S1. Obtain a visible light image and a SAR image of a target area through an unmanned aerial vehicle; connect the corresponding trisection points on the opposite sides of the visible light image to form intersection points, and obtain 4 first feature points; solve the geographical coordinates of the first feature points; select feature points from the SAR image according to the geographical coordinates to obtain 4 second feature points.

[0048] In S1, solving the geographical coordinates of the first feature points includes: The relationship between the coordinates of the object point P in the northeast celestial geographical coordinate system and the image point coordinates includes: ; Among them, represents the camera internal parameter matrix; represents converting the image point coordinates of the object point P to the coordinates in the camera coordinate system ; represents converting the coordinates of the object point P in the camera coordinate system to the coordinates in the UAV coordinate system transformation matrix; represents converting the coordinates of the object point P in the UAV coordinate system to the coordinates in the northeast celestial geographic coordinate system transformation matrix; represents the geographic coordinate position of the UAV, that is, the longitude and latitude of the origin of the northeast celestial coordinate system.

[0049] Obtain the image point coordinates of the first feature point according to the visible light image, and convert the image point coordinates of the first feature point according to the relationship between the coordinates in the northeast celestial geographic coordinate system and the image point coordinates, then the geographic coordinates of the first feature point can be obtained.

[0050] In the preferred embodiment of the present invention, the camera internal parameter matrix K includes: ; Among them, represents the camera focal length; and represent the pixel size of the imaging system; represents the principal point coordinates of the image plane.

[0051] In the preferred embodiment of the present invention, the transformation matrix is calculated through the attitude angles of the camera relative to the UAV, and includes: ; Among them, , and respectively represent the roll angle, pitch angle and yaw angle of the camera, that is, represent the rotation around the , and axes of the camera coordinate system.

[0052] In the preferred embodiment of the present invention, the transformation matrix is calculated through the UAV attitude angles, and includes: ; Among them, , and respectively represent the roll angle, pitch angle and yaw angle of the UAV, that is, represent the rotation around the UAV coordinate system's , and rotation of the shaft.

[0053] S2. Construct a set of matching point pairs based on the first feature point and the second feature point; obtain a homography transformation matrix according to the set of matching point pairs; multiply the homography transformation matrix by the SAR image to obtain a roughly registered SAR image.

[0054] S3. Obtain the corner points of each image by combining the visible light image and the roughly registered SAR image with the feature response model and the FAST feature point detection algorithm; construct circular descriptors and statistical histograms for all the corner points in the image, and obtain the feature consistency description vectors of each corner point according to the circular descriptors and the statistical histograms.

[0055] In S3, obtaining the corner points of each image by combining the visible light image and the roughly registered SAR image with the feature response model and the FAST feature point detection algorithm includes: Perform a first process on the visible light image and the roughly registered SAR image respectively to obtain the corner points of each image.

[0056] The first process includes: Perform downsampling on the original image for a preset number of times and at a preset multi-scale, and apply smoothing and blurring operations with different preset standard deviations to the image at each scale through a Gaussian filter to obtain a preprocessed image at each scale; obtain a comprehensive feature map according to the preprocessed image at each scale in combination with the feature response model; perform key point extraction according to the comprehensive feature map in combination with the FAST feature point detection algorithm to obtain corner points; the number of corner points is detected by the FAST feature point detection algorithm.

[0057] In the preferred embodiment of the present invention, the downsampling for a preset number of times and at a preset multi-scale includes four times of downsampling with sizes of 1 / 2, 1 / 4, 1 / 8, and 1 / 16.

[0058] In the preferred embodiment of the present invention, the Gaussian filter includes: ; wherein, represents the natural base; represents the standard deviation; represents the Gaussian filter.

[0059] In the preferred embodiment of the present invention, the feature response model is based on the principle of phase consistency, and is used to enhance the saliency of the feature map and accumulate the feature responses in the preset directions at all scales, thereby generating a comprehensive feature map containing all scale and direction information.

[0060] In the preferred embodiment of the present invention, performing key point extraction according to the comprehensive feature map in combination with the FAST feature point detection algorithm to obtain corner points includes: Define the neighborhood as a circular area containing Q pixels, and perform corner detection by analyzing the gray value comparison within the neighborhood: ; Among them, represents the gray value of pixel point . represents the gray value of the neighborhood pixel point of pixel point . represents a preset hyperparameter threshold; if consecutive pixels within the neighborhood of pixel point satisfy , then pixel point

[0061] can be determined as a corner point. In the preferred embodiment of the present invention, the range of Q is preferably 50 to 200 pixels, and Q is a positive integer. Figure 2 Referring to , in S3, construct a circular descriptor and a statistical histogram for all corner points in the image, and obtain the feature consistency description vector of each corner point according to the circular descriptor and the statistical histogram, including:

[0062] Taking the corner point as the center, construct a circular area with a radius of R to describe the rotational invariance; the unit of R is pixels. In the preferred embodiment of the present invention, the range of R is preferably 36 to 108, and R is a positive integer. Figure 2 Calculate the feature values of each pixel point in the circular area in a preset number and angle directions, and mark the direction corresponding to the maximum feature value of each pixel point as the main direction of the pixel point; construct an XY axis coordinate system in the circular area with the corner point as the origin to obtain a direction index map; the direction index map includes the main direction of each pixel point. In the preferred embodiment of the present invention, the preset number and angle directions include 6 angle directions of 0°, 30°, 60°, 90°, 120°, and 150°, that is θ 1 ~ θ 6 .

[0063] Statistically analyze the main direction of each pixel point in the direction index map to obtain a statistical histogram.

[0064] Set the direction with the highest frequency of occurrence in the statistical histogram as the main direction of the corner point to complete the main direction estimation.

[0065] Taking the rotation of the main direction of the corner point to the positive X-axis direction as the target, rotate the circular area.

[0066] Perform pixel point direction correction on all pixel points in the circular area through a four-quadrant voting strategy, including: If there are pixel point directions in the third or fourth quadrant of the XY-axis coordinate system after the circular region is rotated, add π to the pixel point directions in the third or fourth quadrant of the XY-axis coordinate system.

[0067] Divide the circular region evenly into partitions along the radial and tangential directions, and statistically calculate the main direction statistical histogram of all pixel points in each partition to obtain the feature vector of each partition; starting from the positive half-axis of the X-axis, connect the feature vectors of each partition in a clockwise direction from the outer circle to the inner circle to obtain the feature consistency descriptor of the corner points.

[0068] Normalize the feature consistency descriptor of the corner points through the L2 norm to obtain the feature consistency description vector of the corner points.

[0069] Process all corner points in the image to obtain the feature consistency description vectors of each corner point.

[0070] S4. According to the bidirectional matching cost function, perform similarity measurement and corner point matching on the feature consistency description vectors of the visible light image and the coarsely registered SAR image, eliminate the incorrect matching points, obtain the high-confidence matching point pairs that meet the geometric consistency constraints, and perform image registration according to the matching point pairs to obtain the finely registered SAR image.

[0071] In the preferred embodiment of the present invention, the bidirectional matching cost function is based on an adaptive association strategy.

[0072] In the preferred embodiment of the present invention, when eliminating the incorrect matching points, eliminate them based on the random sample consensus algorithm.

[0073] The visible light and SAR large-view-angle difference image registration method for an unmanned aerial vehicle (UAV) platform according to the present invention realizes end-to-end image registration of the UAV platform through a two-stage method of first rough registration and then fine registration. In the rough registration, the geographical location information of the key feature points of the visible light image is solved, a matching point pair is constructed to calculate the transformation matrix, and a roughly registered SAR image is obtained. Then, the best matching point pair is obtained through multi-scale feature point selection, feature consistency description vectors, and an adaptive matching strategy, so as to obtain a finely registered SAR image. Through the registration of the two stages, the registration result has high precision. The method of the present invention does not need to rely on the way of manually extracting features for image registration, can run online, and further realizes end-to-end deployment of the UAV platform. The registration speed is significantly improved, the registration accuracy is improved, and the problem of inability to perform online registration caused by the view angle and modality differences between the visible light and SAR images obtained by the UAV platform can be effectively solved, which is suitable for subsequent image fusion by taking advantage of the different modalities of the visible light and SAR images. At the same time, the method of the present invention has high operating efficiency, breaks through the practical application barrier of high-efficiency online image registration of visible light and SAR images on the UAV platform, and provides a solution for the practical registration of visible light and SAR images with large view angle differences. The method of the present invention can realize fast and accurate registration of visible light images and SAR images with large view angle differences in the UAV platform, and supports efficient operation of visible light and SAR image registration on the UAV platform.

[0074] In a preferred embodiment of the present invention, a visible light and SAR large-view-angle difference image registration system for an unmanned aerial vehicle (UAV) platform is further provided for the method of the present invention. The system includes a UAV, an SAR, a visible light sensor, and an edge computing platform; The SAR, the visible light sensor, and the edge computing platform are all installed on the UAV; the SAR and the visible light sensor are both connected to the edge computing platform; the SAR is used to acquire an SAR image and send the SAR image to the edge computing platform; the visible light sensor is used to acquire a visible light image and send the visible light image to the edge computing platform.

[0075] The edge computing platform is used to connect the corresponding trisection points on the sides of the visible light image to form intersection points, and obtain 4 first feature points; solve the geographical coordinates of the first feature points; select feature points from the SAR image according to the geographical coordinates, and obtain 4 second feature points.

[0076] The edge computing platform is further used to construct a set of matching point pairs according to the first feature points and the second feature points; obtain a homography transformation matrix according to the set of matching point pairs; multiply the homography transformation matrix by the SAR image to obtain a roughly registered SAR image.

[0077] The edge computing platform is also used to obtain the corner points of each image according to the visible light image and the roughly registered SAR image by combining the feature response model and the FAST feature point detection algorithm; construct circular descriptors and statistical histograms for all the corner points in the image, and obtain the feature consistency description vectors of each corner point according to the circular descriptors and the statistical histograms.

[0078] The edge computing platform is also used to perform similarity measurement and corner point matching on the feature consistency description vectors of the visible light image and the roughly registered SAR image according to the bidirectional matching cost function, eliminate the mis-matched points, obtain high-confidence matching point pairs that satisfy the geometric consistency constraint, and perform image registration according to the matching point pairs to obtain the precisely registered SAR image.

[0079] The visible light and SAR large-view-angle difference image registration system for the UAV platform of the present invention is used for the method of the present invention and has the same beneficial effects as the method of the present invention.

[0080] Verification part: In the preferred embodiment of the present invention, experimental tests are carried out at a certain experimental site. The area of the test site is about 1,200,000 square meters, and the test scenario covers ground and water areas. The SAR band used by the UAV is the KU band, the frequency is between 12.5 and 18 GHz, and the working system is frequency-modulated continuous wave. The visible light band is between 380 and 750 nm, the obtained video frame rate is 30 Hz, and the resolution is 1920×1080 pixels.

[0081] In the experiment of the preferred embodiment of the present invention, Figure 3 is the input visible light reference image obtained by the airborne visible light sensor. Figure 4 is the input SAR image to be registered obtained by the airborne SAR, and A represents the area where the SAR image roughly corresponds to the visible light image. Figure 5 are the points where the visible light image and the SAR image have the same geographical coordinates for the selected feature points. After matrix transformation, the roughly registered SAR result is obtained. From Figures 3 to 5 It can be seen that the roughly registered SAR image basically establishes the rough regional alignment with the visible light image in space.

[0082] In the experiment of the preferred embodiment of the present invention, Figure 6 is the checkerboard image of the final registration result through the rough registration stage and the precise registration stage. Figure 7It is the correct matching point pair result of the SAR image and the visible light image after final registration. It can be seen that the method of the present invention can meet the actual application requirements, realize the accurate registration and alignment of visible light and SAR images under the background of large viewing angle differences, overcome the problem that the SAR and visible light images obtained by the UAV platform cannot be registered online, resulting in the inability to perform image fusion, and can achieve efficient and accurate image registration functions through end-to-end deployment and online operation within the large viewing field of the UAV, and has strong application value.

[0083] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A visible light and SAR large viewing angle difference image registration method for UAV platform, characterized in that: include: Obtain visible light images and SAR images of the target area through drones; Connect the three equally divided points on the opposite sides of the visible light image to form intersection points to obtain four first feature points; Solving the geographic coordinates of the first feature point; selecting feature points from the SAR image according to the geographic coordinates to obtain four second feature points; Constructing a set of matching point pairs according to the first feature point and the second feature point; obtaining a homography transformation matrix according to the set of matching point pairs; and multiplying the homography transformation matrix with the SAR image to obtain a coarsely registered SAR image; Obtaining corner points of the respective images according to the visible light image and the coarsely registered SAR image in combination with a feature response model and a FAST feature point detection algorithm; constructing circular descriptors and statistical histograms for all corner points in the image, and obtaining a feature consistency description vector of each corner point according to the circular descriptors and the statistical histograms; The feature consistency description vectors of the visible light image and the coarsely registered SAR image are similarly measured and corner matched according to the bidirectional matching cost function, and the wrong matching points are eliminated to obtain high-confidence matching point pairs that meet the geometric consistency constraints. Image registration is performed based on the matching point pairs to obtain a finely registered SAR image.

2. The visible light and SAR large viewing angle difference image registration method for UAV platform according to claim 1 is characterized in that: Solving the geographic coordinates of the first feature point includes: The relationship between the coordinates of the object point P in the northeastern sky geographic coordinate system and the image point coordinates includes: ； in, Represents the camera intrinsic parameter matrix; Indicates that the image point coordinates of object point P Coordinates converted to the camera coordinate system ; Indicates the coordinates of the camera coordinate system of the object point P Coordinates converted to the drone coordinate system The transformation matrix of Indicates the coordinates of the drone coordinate system of the object point P Coordinates converted to the Northeastern Sky geographic coordinate system The transformation matrix of Indicates the geographic coordinate system position of the drone, that is, the latitude and longitude of the origin of the northeast sky coordinate system; The image point coordinates of the first feature point are obtained according to the visible light image, and the image point coordinates of the first feature point are converted according to the relationship between the coordinates of the northeast sky geographic coordinate system and the image point coordinates, so as to obtain the geographic coordinates of the first feature point.

3. The visible light and SAR large viewing angle difference image registration method for UAV platform according to claim 2 is characterized in that: Also includes: The camera intrinsic parameter matrix include: ； in, Indicates the focal length of the camera; and Represents the pixel size of the imaging system; represents the coordinates of the principal point in the image plane; The transformation matrix Calculated by the camera's attitude angle relative to the drone, including: ； in, , and Respectively represent the roll angle, pitch angle and yaw angle of the camera, that is, the , and Rotation of the axis; The transformation matrix Calculated through the drone attitude angle, including: ； in, , and Respectively represent the roll angle, pitch angle and yaw angle of the drone, that is, the , and Rotation of the axis.

4. The visible light and SAR large viewing angle difference image registration method for UAV platform according to claim 3 is characterized in that: The corner points of the respective images are obtained according to the visible light image and the coarsely registered SAR image in combination with the feature response model and the FAST feature point detection algorithm, including: Performing a first process on the visible light image and the coarsely registered SAR image respectively to obtain corner points of the respective images; The first process comprises: The original image is downsampled a preset number of times and at a preset multiple scales, and smoothing and blurring operations with different preset standard deviations are applied to the image at each scale through a Gaussian filter to obtain a preprocessed image at each scale; a comprehensive feature map is obtained based on the preprocessed image at each scale in combination with a feature response model; key points are extracted based on the comprehensive feature map in combination with a FAST feature point detection algorithm to obtain the corner points; the number of the corner points is detected by the FAST feature point detection algorithm.

5. The visible light and SAR large viewing angle difference image registration method for UAV platform according to claim 4 is characterized in that: The Gaussian filter comprises: ； in, Represents the natural base; represents standard deviation; represents a Gaussian filter.

6. The visible light and SAR large viewing angle difference image registration method for UAV platform according to claim 5 is characterized in that: The feature response model is based on the phase consistency principle, is used to enhance the saliency of feature mapping, and accumulates feature responses of preset directions at all scales, thereby generating a comprehensive feature map containing all scale and direction information.

7. The visible light and SAR large viewing angle difference image registration method for UAV platform according to claim 6 is characterized in that: According to the comprehensive feature map combined with the FAST feature point detection algorithm, key points are extracted to obtain the corner points including: The neighborhood is defined as a circular area containing Q pixels, and corner detection is achieved by analyzing the grayscale value comparison within the neighborhood: ； in, Represents pixel Gray value of Represents pixel Neighborhood pixels Gray value of Represents the preset hyperparameter threshold; if the pixel Continuous in the neighborhood Pixels meet When For corner points.

8. The visible light and SAR large viewing angle difference image registration method for UAV platform according to claim 7 is characterized in that: The step of constructing a circular descriptor and a statistical histogram for all corner points in the image, and obtaining a feature consistency description vector of each corner point according to the circular descriptor and the statistical histogram comprises: Taking the corner point as the center, construct a circular area with a radius of R; Calculate the eigenvalues ​​of each pixel point in the circular area in the directions of the preset number and angle, and mark the direction corresponding to the maximum eigenvalue of each pixel point as the main direction of the pixel point; construct an XY axis coordinate system in the circular area with the corner point as the origin to obtain a direction index map; the direction index map includes the main direction of each pixel point; Counting the main direction of each pixel in the direction index map to obtain a statistical histogram; The direction with the highest frequency of occurrence in the statistical histogram is set as the main direction of the corner point; The circular area is rotated with the goal of rotating the main direction of the corner point to the positive semi-axis direction of the X-axis; Direction correction is performed on all pixels within the circular area using a four-quadrant voting strategy; The circular area is evenly divided into partitions along the radial and tangential directions, and main direction statistical histograms of all pixels in each partition are counted to obtain a feature vector of each partition; Starting from the positive half axis of the X-axis, the feature vectors of each partition are connected in a clockwise direction from the outer circle to the inner circle to obtain the feature consistency descriptor of the corner point, and the feature consistency descriptor is normalized by the L2 norm to obtain the feature consistency description vector of the corner point; All corner points in the image are processed to obtain the feature consistency description vector of each corner point.

9. The visible light and SAR large viewing angle difference image registration method for UAV platform according to claim 8 is characterized in that: The direction correction of all pixels in the circular area by using a four-quadrant voting strategy includes: If there is a pixel point direction located in the third quadrant or the fourth quadrant of the XY axis coordinate system after the circular area is rotated, the value of π is added to the pixel point direction located in the third quadrant or the fourth quadrant of the XY axis coordinate system.

10. A visible light and SAR large viewing angle difference image registration system for UAV platforms, used in the method according to any one of claims 1 to 9, characterized in that: The system includes a drone, a SAR, a visible light sensor, and an edge computing platform; The SAR, visible light sensor and edge computing platform are all installed on the drone; the SAR and visible light sensor are both connected to the edge computing platform; the SAR is used to obtain SAR images and send the SAR images to the edge computing platform; the visible light sensor is used to obtain visible light images and send the visible light images to the edge computing platform; The edge computing platform is used to connect the three equally divided points on the opposite sides of the visible light image to form intersections to obtain four first feature points; solve the geographic coordinates of the first feature points; select feature points from the SAR image according to the geographic coordinates to obtain four second feature points; The edge computing platform is further used to construct a set of matching point pairs according to the first feature point and the second feature point; obtain a homography transformation matrix according to the set of matching point pairs; and multiply the homography transformation matrix by the SAR image to obtain a coarsely registered SAR image; The edge computing platform is also used to obtain corner points of the respective images according to the visible light image and the coarsely registered SAR image in combination with a feature response model and a FAST feature point detection algorithm; construct circular descriptors and statistical histograms for all corner points in the image, and obtain a feature consistency description vector of each corner point according to the circular descriptors and the statistical histograms; The edge computing platform is also used to perform similarity measurement and corner point matching on the feature consistency description vectors of the visible light image and the coarsely registered SAR image according to a bidirectional matching cost function, eliminate erroneous matching points, obtain high-confidence matching point pairs that meet geometric consistency constraints, perform image registration based on the matching point pairs, and obtain a precisely registered SAR image.

Images