Unmanned aerial vehicle multi-modal image registration method and system based on shooting distance parameter, and storage medium

Abstract

The application discloses a kind of unmanned vehicle multimodal image registration methods, system and storage medium based on shooting distance parameter, it is related to image processing technical field, it discloses the unmanned vehicle multimodal image registration method based on shooting distance parameter, comprising: the reference modal image and target modal image that multiple camera modules collected to different imaging principle on same unmanned vehicle platform are acquired;The distance between unmanned vehicle holder and the reference plane where the photographed target is determined as shooting distance;Based on the mapping relationship of distance and transformation parameter generated in calibration stage, determine the geometric transformation parameter of target modal image under shooting distance, registration to reference modal image;Based on geometric transformation parameter, target modal image is resampled to the coordinate system of reference modal image, to register the reference modal image and the target modal image, the transformation parameter used in the registration stage of distance parameter is used to realize the calibration of image, improve registration efficiency and registration stability.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, system and storage medium for multimodal image registration of unmanned aerial vehicles based on shooting distance parameters. Background Technology

[0002] Unmanned aerial vehicles (UAVs) widely employ multimodal image combinations, such as visible light heated infrared or visible light plus multispectral imaging, in infrastructure inspection, disaster monitoring, precision agriculture, and emergency rescue. Although multiple sensors are rigidly mounted on the same gimbal or payload platform, their relative poses are theoretically fixed. However, in actual flight, factors such as imperfect optical axis alignment, focal length and distortion differences, assembly tolerances, and minor extrinsic parameter drift can lead to non-strict alignment of cross-modal images. Therefore, calibration processing of images acquired by UAVs is necessary.

[0003] Related calibration methods are usually based on feature extraction and algorithm matching. However, in cross-spectral or weak texture scenarios, that is, when the same physical target appears completely different under different spectra or lacks significant consistent texture, this calibration method is prone to failure and it is difficult to guarantee the accuracy of image registration.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a method, system, and storage medium for multimodal image registration of unmanned aerial vehicles based on shooting distance parameters, aiming to solve the technical problem of low cross-modal image registration accuracy.

[0006] To achieve the above objectives, this application proposes a UAV multimodal image registration method based on shooting distance parameters, the UAV multimodal image registration method based on shooting distance parameters comprising:

[0007] Acquire reference modal images and target modal images from multiple camera modules with different imaging principles on the same UAV platform;

[0008] The shooting distance is determined by the distance between the drone gimbal and the reference plane where the target is located;

[0009] Based on the distance-transformation parameter mapping relationship generated during the calibration phase, the geometric transformation parameters of the target modal image registered to the reference modal image at the shooting distance are determined.

[0010] The target modal image is resampled to the coordinate system of the reference modal image based on the geometric transformation parameters, so as to register the reference modal image and the target modal image.

[0011] In one embodiment, before the step of acquiring reference modal images and target modal images collected by multiple camera modules with different imaging principles on the same UAV platform, the UAV multimodal image registration method based on the shooting distance parameter further includes the following steps:

[0012] Acquire reference images and target images with the same modal features in the same target scene, captured by the drone at multiple preset shooting distances;

[0013] Construct a set of image pairs consisting of the target image and the reference image at the multiple preset shooting distances, and extract and match the multimodal common features of each image pair in the set to establish a set of correspondence relationships for calibration;

[0014] Based on the set of correspondences, the transformation parameters corresponding to the multiple preset shooting distances are calculated;

[0015] Based on the transformation parameters under the multiple preset shooting distances, an interpolation or regression model is constructed to obtain the mapping relationship between the distance and the transformation parameters.

[0016] In one embodiment, the step of calculating the transformation parameters corresponding to the plurality of preset shooting distances based on the correspondence set includes:

[0017] Based on the set of correspondences, the affine transformation parameters between the reference image and the target image are solved at the multiple preset shooting distances respectively;

[0018] The step of constructing an interpolation or regression model based on the transformation parameters under the multiple preset shooting distances to obtain the mapping relationship between the distance and the transformation parameters includes:

[0019] By interpolating or regressing the affine transformation parameters, a mapping relationship between the distance and the transformation parameters is obtained. The corresponding affine transformation parameters are retrieved based on the mapping relationship constructed by the affine transformation parameters and the shooting distance.

[0020] In one embodiment, the step of resampling the target modal image to the coordinate system of the reference modal image based on the geometric transformation parameters to register the reference modal image and the target modal image includes:

[0021] Based on the shooting distance, the affine transformation parameters corresponding to the mapping relationship are called to construct the affine transformation matrix corresponding to the affine transformation parameters;

[0022] Calculate the inverse of the affine transformation matrix;

[0023] Multiply the coordinates of all pixel values ​​in the reference modal image by the inverse of the affine transformation matrix to obtain the source coordinates of all pixel values ​​in the target modal image.

[0024] The pixel values ​​at the source coordinates are interpolated and sampled to obtain the registration result.

[0025] In one embodiment, before the step of interpolating and sampling the pixel values ​​at the source coordinates to obtain the registration result, the UAV multimodal image registration method based on the shooting distance parameter further includes:

[0026] When the scaling factor of the affine transformation parameter is less than one and downsampling occurs, the target modal image is low-pass pre-filtered to suppress aliasing of the target modal image;

[0027] When the source coordinates exceed the effective area of ​​the target modal image, the pixel source coordinates that exceed the effective area are filled based on a preset boundary processing strategy, and effective area indication information is generated to identify the available pixels in the registration result;

[0028] Perform the step of interpolating and sampling the pixel values ​​at the source coordinates to obtain the registration result.

[0029] In one embodiment, when the scene of the reference image satisfies a single plane or near-plane condition, the step of calculating the transformation parameters corresponding to the plurality of preset shooting distances based on the set of correspondences includes:

[0030] Based on the set of correspondences, the homography transformation parameters from the target image to the reference image are solved at the multiple preset shooting distances respectively;

[0031] The step of constructing an interpolation or regression model based on the transformation parameters under the multiple preset shooting distances to obtain the mapping relationship between the distance and the transformation parameters includes:

[0032] Interpolation or regression modeling is performed on the homography transformation parameters to obtain the mapping relationship between the distance and the transformation parameters. The corresponding homography transformation matrix is ​​retrieved based on the mapping relationship constructed based on the shooting distance.

[0033] In one embodiment, the step of resampling the target modal image to the coordinate system of the reference modal image based on the geometric transformation parameters to register the reference modal image and the target modal image includes:

[0034] Based on the shooting distance, retrieve the homography transformation matrix corresponding to the mapping relationship;

[0035] Calculate the inverse matrix of the homography transformation matrix based on the homography transformation matrix;

[0036] Multiply the coordinates of all pixel values ​​in the reference image by the inverse of the homography transformation matrix, and perform perspective normalization to obtain the source coordinates of all pixel values ​​in the target modal image.

[0037] The pixel values ​​at the source coordinates are interpolated and sampled to obtain the registration result.

[0038] In one embodiment, after the step of acquiring reference modal images and target modal images collected by multiple camera modules with different imaging principles on the same UAV platform, the UAV multimodal image registration method based on the shooting distance parameter further includes:

[0039] Radial and tangential distortion corrections are performed on the reference modal image and the target modal image.

[0040] In addition, to achieve the above objectives, this application also proposes an image registration system, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the UAV multimodal image registration method based on shooting distance parameters as described above.

[0041] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the UAV multimodal image registration method based on shooting distance parameters as described above.

[0042] One or more technical solutions proposed in this application have at least the following technical effects:

[0043] First, reference modal images and target modal images acquired by camera modules with different imaging principles on the same UAV platform are obtained. Then, the shooting distance between the UAV gimbal and the reference plane where the target is located is determined. Subsequently, based on the mapping relationship between distance and transformation parameters generated in the calibration stage, the geometric transformation parameters for registering the target modal image to the reference modal image at the current shooting distance are determined. Finally, based on these parameters, the target modal image is resampled to the coordinate system of the reference modal image to complete the registration. In this way, the registration deviation caused by the failure of common feature detection and matching is effectively avoided during the registration process. This enables the reference modal image and the target modal image to achieve more accurate image coordinate system alignment in shooting scenes without consistent textures, providing a reliable foundation for the subsequent collaborative application of multimodal images. At the same time, it eliminates the need to re-perform complex image feature recognition or other derivation calculations each time registration is performed, further improving the efficiency and stability of the registration process. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the first embodiment of the UAV multimodal image registration method based on shooting distance parameter provided in this application;

[0047] Figure 2 This is a flowchart illustrating the second embodiment of the UAV multimodal image registration method based on shooting distance parameter provided in this application;

[0048] Figure 3 A schematic diagram of multimodal image pairs acquired using a calibration board during the registration and calibration phase of this application;

[0049] Figure 4 A schematic diagram of multimodal image pairs acquired using feature targets during the registration and calibration phase of this application;

[0050] Figure 5 This is a flowchart illustrating the fourth embodiment of the UAV multimodal image registration method based on shooting distance parameter provided in this application;

[0051] Figure 6 This is a schematic diagram of the UAV gimbal equipped with a visible light module and a thermal infrared module used in this application;

[0052] Figure 7 Example images showing the images before and after registration;

[0053] Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the UAV multimodal image registration method based on shooting distance parameter in the embodiments of this application.

[0054] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0056] Although multiple sensors for drone image acquisition are rigidly mounted on the same gimbal or payload platform, the optical axes of multiple lenses are not physically aligned; the differences in lens focal lengths and distortions are superimposed; and the slight extrinsic parameter offset caused by modular assembly tolerances result in images of different modalities being in a state of non-strict alignment.

[0057] While relevant calibration methods typically rely on feature extraction and feature matching for registration, these methods are prone to failure in cross-spectral or weak-texture scenarios—where the same physical target appears completely different under different spectra or lacks significant texture—making it difficult to guarantee the accuracy of image registration. Therefore, the main solution of this application is to acquire reference modal images and target modal images collected by multiple camera modules with different imaging principles on the same UAV platform.

[0058] The shooting distance is determined by the distance between the drone gimbal and the reference plane where the target is located;

[0059] Based on the distance-transformation parameter mapping relationship generated during the calibration phase, the geometric transformation parameters of the target modal image registered to the reference modal image at the shooting distance are determined.

[0060] The target modal image is resampled to the coordinate system of the reference modal image based on the geometric transformation parameters, so as to register the reference modal image and the target modal image.

[0061] Specifically,

[0062] This application provides a solution in which, during the registration process, the shooting distance parameter is obtained, and based on the mapping relationship between distance and transformation parameters generated in the calibration stage, the geometric transformation parameters of the target modal image to be registered to the reference modal image at the current shooting distance are accurately determined. The image is then transformed based on the geometric transformation parameters, so that the reference modal image and the target modal image can achieve more accurate coordinate system alignment in the specific shooting scene, thereby improving the efficiency and stability of the registration process.

[0063] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or image registration system capable of performing the above functions. The following description uses an image registration system as an example to illustrate this embodiment and the subsequent embodiments.

[0064] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0065] This application provides a method for UAV multimodal image registration based on shooting distance parameters, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the UAV multimodal image registration method based on shooting distance parameter of this application.

[0066] In this embodiment, the UAV multimodal image registration method based on the shooting distance parameter includes steps S10~S40:

[0067] Step S10: Acquire reference modal images and target modal images collected by multiple camera modules with different imaging principles on the same UAV platform.

[0068] A camera module typically refers to a complete imaging unit mounted on a drone, including a lens, image sensor, and related processing circuitry. In drone applications, multiple camera modules are often rigidly mounted on the same drone gimbal or platform. Camera modules based on different imaging principles refer to devices that acquire images based on different physical properties, such as visible light modules and thermal infrared modules, or visible light camera modules and multispectral camera modules, etc. Figure 6 As shown. The reference modal image is the image used as a reference during the registration process, and the target modal image is the image that needs to be aligned with the reference modal image. In this embodiment, the reference modal image is a visible light image, while the target modal image is a thermal infrared image.

[0069] As an optional implementation for acquiring reference and target modal images, a visible light camera module and a thermal infrared camera module are fixedly mounted on the gimbal of the UAV platform. This ensures that their optical axes are parallel and that their fields of view have sufficient overlap, guaranteeing that the acquired images contain the same scene portion, thus providing a scene basis for subsequent registration. Then, when the UAV flies to the target waypoint, a synchronous acquisition command is sent via the UAV control system, or an image acquisition command is issued to the UAV when the system detects that the image acquisition trigger conditions are met, so that multiple camera modules simultaneously acquire images of the current scene. Finally, the visible light image acquired at the same time is used as the reference modal image, and the thermal infrared image is used as the target modal image. The image data is stored in the UAV's local storage module or sent to a server for subsequent calculation and processing.

[0070] Optionally, the drone can be controlled to fly along a preset route, and a timed data collection mechanism can be set for the drone, activating two types of camera modules simultaneously at regular intervals to take pictures.

[0071] For example, in an urban building inspection scenario, a visible light camera module captures images containing details such as the building's exterior walls and windows, while a thermal infrared camera module simultaneously captures images reflecting the temperature distribution of the building's exterior walls. The visible light images are stored as reference modal images to be registered, while the thermal infrared images are the target modal images.

[0072] Optionally, after obtaining modal images of different modes, radial and tangential distortion corrections can be performed on the reference modal image and the target modal image to improve the accuracy of subsequent image registration.

[0073] Step S20: Determine the shooting distance as the distance between the drone gimbal and the reference plane where the target is located.

[0074] The term "shooting" refers to the vertical distance from the UAV gimbal to a reference plane within the target scene, which is also the distance from the optical center of the visible light camera along the optical axis to the reference plane. The reference plane is typically the largest plane in the target scene, such as the ground or building facades. In this embodiment, the distance from the optical center of the visible light camera along the optical axis to the reference plane can be calculated using onboard laser ranging modules, ultrasonic ranging modules, binocular or structured light visual ranging, or other measurement modules or methods. Alternatively, it can be obtained based on a pre-set inspection shooting distance along the flight path. Furthermore, other ranging methods can be used for calculation, and this application does not limit the specific methods used.

[0075] As an optional method for determining the shooting distance, a laser rangefinder can be mounted on the drone gimbal. Its measurement direction is aligned with the optical axis of the camera module, ensuring that the measured distance corresponds to the camera module's shooting angle. Then, a laser beam is emitted from the laser rangefinder towards the area of ​​the target. The laser beam is reflected by a reference plane and received. Finally, the vertical distance from the drone gimbal to the reference plane is calculated based on the laser propagation speed and time difference, thus obtaining the shooting distance. This shooting distance is then associated with and stored in conjunction with the image data acquired in step S10. For example, in a ground road inspection scenario, the reference plane is the road surface. The laser beam emitted by the laser rangefinder is perpendicular to the road surface and reflected. The calculated distance from the drone gimbal to the road surface is 80 meters, which is the shooting distance.

[0076] It is understandable that step S20 can be executed after step S10 or simultaneously with step S10.

[0077] In another alternative implementation, the shooting distance can be calculated using visual SLAM (Simultaneous Localization and Mapping) technology. This involves extracting feature points from a reference modality image, identifying stable features such as corners and edges. Then, combining this with data from the UAV's inertial measurement unit (IMU), the visual SLAM algorithm calculates the 3D coordinates of these feature points to construct a local scene map. Finally, a reference plane is identified from the scene map, and the vertical distance from the optical center of the UAV's gimbal-mounted visible light camera to this plane is calculated as the shooting distance.

[0078] Step S30: Based on the distance-transformation parameter mapping relationship generated in the calibration stage, determine the geometric transformation parameters of the target modal image registered to the reference modal image at the shooting distance.

[0079] The mapping relationship between distance and transformation parameters refers to the geometric transformation parameters corresponding to different shooting distances, established through calibration experiments. These parameters include affine transformation matrices, perspective transformation matrices, and homography transformation matrices. Other transformation matrices are also possible. Geometric transformation parameters describe the coordinate system transformation rules or mathematical parameters from the target modal image to the reference modal image.

[0080] In this embodiment, the mapping relationship can be obtained and stored through experimental testing, and may include preset relational expressions, preset mapping tables, etc.

[0081] When the mapping relationship is a preset relationship between distance and transformation parameters, the result obtained by substituting the shooting distance into the preset relationship can be used as the geometric transformation parameter.

[0082] When the mapping relationship is a mapping table between distance and transformation parameters, the mapping table can be queried by shooting distance, and the parameters in the mapping table that match the shooting distance can be used as geometric transformation parameters.

[0083] Step S40: Based on the geometric transformation parameters, the target modal image is resampled to the coordinate system of the reference modal image to register the reference modal image and the target modal image.

[0084] In this embodiment, resampling refers to the process of mapping the pixels of the target modal image to the coordinate system of the reference modal image according to the geometric transformation parameters, and then calculating the new pixel values ​​using an interpolation algorithm. A mapping relationship between the pixel coordinates of the target modal image and the coordinates of the reference modal image can be established based on the geometric transformation parameters. For each pixel in the reference modal image, its corresponding coordinates in the target modal image are found through inverse transformation. Finally, based on bilinear interpolation, the pixel value of that point is calculated according to the values ​​of the four integer pixels surrounding the corresponding coordinates, thereby generating a target modal image aligned with the coordinate system of the reference modal image, completing the registration.

[0085] For example, based on the transformation parameters corresponding to a distance of 80 meters, the pixels of the thermal infrared target image are mapped to the coordinate system of the visible light reference image, and the temperature value of each mapped point is calculated by bilinear interpolation to obtain a thermal infrared registration image aligned with the outline of the visible light image.

[0086] This embodiment provides a UAV multimodal image registration method based on shooting distance parameters. By acquiring multimodal images and the shooting distance at which they were acquired, geometric transformation parameters are determined based on a calibrated distance-transformation parameter mapping relationship. Pixel coordinate alignment is achieved through resampling. This method considers the differences in geometric characteristics of camera modules with different imaging principles at different shooting distances, enabling high-precision alignment between the reference and target modal images in specific scenarios while maintaining registration efficiency. This provides high-quality foundational data for the joint analysis of multimodal images.

[0087] Based on the first embodiment of this application, in the second embodiment, the same or similar content as the first embodiment can be referred to the above description, and will not be repeated hereafter. On this basis, it is necessary to calculate the mapping relationship between distance and transformation parameters during the calibration stage to ensure that the reference modal image and the target modal image can be accurately calibrated. Therefore, please refer to... Figure 2 Before step S10, the UAV multimodal image registration method based on the shooting distance parameter further includes steps S01 to S04:

[0088] Step S01: Acquire reference images and target images with the same modal features in the same target scene, captured by the drone at multiple preset shooting distances.

[0089] In this embodiment, the preset shooting distance refers to multiple pre-set distance values ​​covering the actual operating range of the drone, such as 5 meters, 10 meters, 20 meters, 30 meters, 60 meters, etc., or other distances, which will not be elaborated here. The same modal features refer to stable features that camera modules with different imaging principles can capture in the same target scene, including the corner points of fixed buildings, the outlines of trees, and the corner points corresponding to the calibration board during the calibration phase. It should be noted that the above parameters are for illustrative purposes only and are not intended to limit this application.

[0090] In this embodiment, during the calibration phase, a calibration board can be placed in the shooting scene. Then, the drone is controlled to hover at multiple preset shooting distances. Subsequently, at each preset shooting distance, the visible light camera module and the thermal infrared camera module carried by the drone are simultaneously triggered to shoot the same target scene, i.e., to shoot the visible light image and the thermal infrared image of the calibration board.

[0091] For example, calibration scenarios using a calibration board include... Figure 3 As shown, Figure 3 (A) in the image represents the target image captured by the infrared camera module. Figure 3 (B) in the image is a reference image captured by the visible light camera module.

[0092] In addition, a scene containing targets with shared features from both modal images can be selected as the calibration scene, such as marking the ground or other fixed planes as reference planes. Figure 4 As shown, Figure 4 (A) in the image represents the target image captured by the infrared camera module. Figure 4 (B) in the image is a reference image acquired by the visible light camera module. The features of the reference plane must be sufficiently distinct to ensure the consistency of the shooting target and reduce interference from scene changes. Then, the drone is manually controlled to stay or hover at a preset distance. The deviation between the actual distance and the preset value is confirmed to be within ±0.05 meters by high-precision ranging or high-precision RTK positioning. Simultaneously, different camera modules are controlled to acquire images, thereby obtaining multiple sets of images at multiple preset shooting distances.

[0093] For example, select a calibration scene containing low emissivity patches or a controllable heat source to acquire visible light and thermal infrared images.

[0094] By acquiring multiple reference and target images at preset shooting distances, raw data is provided for subsequent feature extraction and parameter calculation.

[0095] Step S02: Construct a set of image pairs consisting of target images and reference images at multiple preset shooting distances, and extract and match the common features of each image pair in the set to establish a set of correspondences for calibration.

[0096] In this embodiment, the image pair set refers to the combination of reference images and target images grouped according to a preset distance, obtained in step S01. Multimodal shared features refer to features that exist simultaneously in both the reference and target images and can be recognized by the algorithm, such as the edges and corners of the calibration plate, temperature abrupt changes in low-emissivity patches or controllable heat sources, and texture structures. The correspondence set refers to the list of pixel coordinate correspondences for the same feature in the reference and target images.

[0097] Specifically, images can be grouped according to preset distances to construct image pair sets, such as: {(5m visible light, 5m thermal infrared), (10m visible light, 10m thermal infrared), ... (Nm visible light, Nm thermal infrared)}, thus ensuring that images at each distance are processed separately and avoiding cross-distance interference. Next, for each image pair, features are extracted using the SIFT (Scale-Invariant Feature Transform) algorithm, and key points are detected in both the reference and target images to calculate feature descriptors. Finally, the FLANN (Fast Library for Approximate Nearest Neighbors) matcher is used to match the feature descriptors of the two images, and the RANSAC algorithm is used to eliminate incorrect matches. The remaining correct matches form a correspondence set, thereby improving the reliability of the correspondence. Alternatively, a highly reliable correspondence set can be obtained through manual selection.

[0098] Step S03: Based on the set of correspondences, construct the geometric correspondence between the coordinates of the reference modal image and the coordinates of the target modal image, and calculate the transformation parameters corresponding to multiple preset shooting distances.

[0099] Step S04: Based on the transformation parameters under multiple preset shooting distances, construct an interpolation or regression model to obtain the mapping relationship between distance and transformation parameters.

[0100] Transformation parameters refer to the mathematical parameters that describe the geometric transformation from the target image to the reference image. These transformation parameters are categorized into two types: affine transformation parameters and homography transformation parameters. Affine transformation parameters include horizontal and vertical scaling factors, horizontal displacement, vertical displacement, rotational deviation, and shearing factors, among others.

[0101] As an optional implementation method for calculating transformation parameters and constructing mapping relationships, the transformation parameters are affine transformation parameters. Therefore, based on feature point pairs in the correspondence set, the affine transformation parameters between the reference image and the target image can be solved at multiple preset shooting distances.

[0102] Specifically, for each preset shooting distance, a subset of feature point pairs at that distance is extracted from the corresponding relationship set, such as 4 pairs {(x) at a distance of 10 meters. i y i ), (x' i y' i Then, the coordinates of each pair of feature points are substituted into the affine transformation equations:

[0103] .

[0104] .

[0105] Finally, the system of equations was solved using the least squares method to calculate the six parameters that minimize the transformation error of all feature points. a, b, c, d, e, f These parameters are then used as affine transformation parameters for a preset distance of 10 meters.

[0106] After obtaining the affine transformation parameters, these transformation parameters need to be interpolated or regressed. Interpolation or regression modeling refers to using the known preset distance and its affine transformation parameters to construct a continuous function of the distance and the parameter, that is, to construct a difference or regression model with the distance as the independent variable and the transformation parameter as the dependent variable.

[0107] Specifically, the preset shooting distance D is obtained. i (i=1,2,…,n) and their corresponding affine transformation parameters ( a (D i ), b (D i ), c (D i ), d (D i ), e (D i ), f (D i Afterwards, a continuous mapping relationship needs to be constructed between the shooting distance D and these transformation parameters. The shooting distance D is the only independent variable, while the scaling, translation, and rotation parameters in the affine transformation are all dependent variables. The modeling goal is to provide a functional expression for the transformation parameters for any distance D, rather than only for discrete distances D. i This is applicable. Considering the calibration rule of 1 / D between the object distance and the pinhole imaging, this implementation adopts the following physically meaningful functional form:

[0108] ,

[0109] ,

[0110] ,

[0111] .

[0112] Where, Δ z is the difference in front-to-back distance between the optical centers of the two cameras along the optical axis (positive for forward and negative for backward relative to the visible light camera in thermal infrared), used to compensate for the difference in imaging scale due to the non-coplanarity of the optical centers of the two modules; it can be ignored when the difference is small. g is a constant related to rotational deviation during the calibration phase. θ (D) If the variation with distance in the calibration data is negligible, a constant can be taken. θ0; If there are small but systematic changes, the above formula with g can be used for fitting. When shearing is negligible, θ≈arctan2 (d,a).

[0113] For example, substituting a set of feature points into the system of equations x i '= a x i + b y i + c and y i '= d x i + e y i + f In this process, the optimal approximate solution is obtained using the least squares method, yielding the parameters as follows: a =4.0665, b =0.0114, c =-44.0, d =-51.0, e =48.7, f =-7.23, and simultaneously substitute into a =4.0665, b =0.0114, thus obtaining the rotation angle. θ It is approximately -0.69°. Finally, the obtained parameters are substituted into the mapping relationship to obtain the actual function relationship value, so that the mapping relationship can retrieve the corresponding affine transformation parameters based on the shooting distance.

[0114] This embodiment provides a UAV multimodal image registration method based on shooting distance parameters. By solving the affine transformation parameters through the correspondence set, accurate parameters at multiple preset distances are obtained, so as to extend the discrete preset distance parameters into a continuous functional relationship. This allows the subsequent registration process to directly and quickly obtain the transformation parameters based on any shooting distance, thereby improving the image registration accuracy.

[0115] Based on the second embodiment of this application, in the third embodiment of this application, as another optional implementation of calculating transformation parameters and constructing mapping relationships, when the scene of the reference image satisfies a single plane or near plane, homography transformation parameters can be used to improve the alignment accuracy.

[0116] Understandably, a single-plane scenario refers to a scenario where the main area of ​​the target scene is composed of a single geometric plane, with the vast majority of objects or feature points in the scene located on this plane, and the height difference between different objects and the plane is negligible. A near-plane scenario, on the other hand, refers to a scenario where the vertical distance from each object or feature point in the scene to a reference plane is much smaller than the shooting distance from the drone to that reference plane; that is, the ratio of "height difference / shooting distance" is extremely small, making the influence of the scene's 3D structure on image registration negligible, approximating a planar scene. Therefore, in these two scenarios, the shooting of the scene by different modal camera modules can be approximated as a perspective projection process from a plane to the imaging plane. Homography transformation parameters can accurately describe this projection relationship, including rotation, scaling, translation, and perspective distortion. Therefore, using homography transformation parameters for image registration can achieve high accuracy. Conversely, if the scene has a significant 3D structure, the single-plane or near-plane assumptions do not hold, and homography transformation parameters are no longer applicable.

[0117] Therefore, it is necessary to solve the homography transformation parameters from the target image to the reference image at multiple preset shooting distances based on the correspondence set. Among these, the homography transformation matrix is ​​based on the physical model of camera imaging. H (D) Rotation with camera extrinsic parameters R Translation t and reference plane parameter normal vector n Effective planar distance related to shooting distance D d (D) Directly related. In the camera-normalized coordinate system, the homography transformation matrix satisfies:

[0118] ,

[0119] in R Let be the rotation matrix between the two modal cameras, t be the translation vector, and n be the unit normal to the reference plane. d (D)=D-Δ z D is the shooting distance, Δ z Let be the offset of the optical centers of the two cameras along the optical axis. After scale normalization, we get:

[0120] ,

[0121] Based on this, after acquiring reference and target modal images at multiple preset shooting distances, planar feature point pairs can be extracted to form a correspondence set, and for each preset distance D... k The homography transformation matrix at this distance is solved using the direct linear transformation method. H k ,and H k = H 0+1 / (D k -Δ zH1, then construct about H 0、 H 1. Δ z For an overdetermined system of equations, the constant parameters are solved using the least squares method. H 0、 H 1 and Δ z Thus, the homography transformation parameters are obtained.

[0122] After obtaining the homography transformation parameters, it is necessary to perform interpolation or regression modeling on the homography-transformed eucalyptus tree to obtain the mapping relationship between distance and transformation parameters. Specifically, the calibration parameters corresponding to each preset distance can be... H 0、 H 1 and Δ z Substitute into the calculation formula H (D)= H 0+(1 / (D-Δ z ))· H In step 1, an arbitrary shooting distance D is input, and the mapping of the homography transformation matrix can be directly calculated. This yields a mapping relationship with the preset distance and its discrete distances as independent variables and the homography transformation parameters as dependent variables. In this embodiment, this mapping relationship between distance and transformation parameters can retrieve the corresponding homography transformation matrix based on the shooting distance.

[0123] This embodiment provides a UAV multimodal image registration method based on shooting distance parameters. By solving the homography transformation parameters through the correspondence set, accurate parameters at multiple preset distances are obtained, so as to extend the discrete preset distance parameters into a continuous functional relationship. This allows the subsequent registration process to directly and quickly obtain transformation parameters based on any shooting distance, thereby improving the image registration accuracy.

[0124] Based on the second embodiment of this application, in the fourth embodiment of this application, when the transformation parameter is an affine transformation parameter, please refer to... Figure 5 Step S40 also includes steps S41 to S44:

[0125] Step S41: Based on the shooting distance, call the affine transformation parameters corresponding to the mapping relationship to construct the affine transformation matrix corresponding to the affine transformation parameters.

[0126] In this embodiment, the affine transformation parameters are obtained by substituting the shooting distance D into the mapping relationship. Then, an affine transformation matrix is ​​constructed through parameter transformation to perform pixel coordinate interpolation sampling. The affine transformation matrix is ​​as follows:

[0127] .

[0128] Understandably, this matrix describes the forward transformation relationship from the target modal image to the reference modal image.

[0129] Step S42: Calculate the inverse matrix of the affine transformation matrix.

[0130] In this embodiment, after obtaining the affine transformation matrix, it is necessary to convert the forward transformation from the target modal image to the reference modal image into the inverse transformation from the reference modal image to the target modal image, so as to deduce the original position of the reference image pixels in the target image. Therefore, it is necessary to invert the affine transformation matrix A to obtain A -1 .

[0131] Step S43: Multiply all pixel coordinates in the reference modal image by the inverse of the affine transformation matrix to obtain the source coordinates of all pixel coordinates in the target modal image.

[0132] In this embodiment, the center of each pixel in the reference modal image can be... q =(x v y v ,1) T Calculate its source coordinates in the target modality image: .

[0133] Where, x v and y v The coordinates of the pixel center in the reference modal image are two-dimensional planar coordinates.

[0134] Specifically, the homogeneous coordinates of each pixel can be obtained by traversing all pixels of the reference modality image, and then all homogeneous coordinates are multiplied on the left by the inverse matrix A. -1 (D) is used to obtain the homogeneous source coordinates of the pixel in the target modal image. For example, the reference image pixel (200, 300) is multiplied by A on the left. -1 After (D), the source coordinates of the target image (198.5, 297.3) are obtained, which means that the pixel value of the reference image corresponds to the pixel value at position (198.5, 297.3) in the target image.

[0135] Step S44: Perform interpolation sampling on the source coordinates to obtain the registration result.

[0136] In this embodiment, the obtained source coordinates are usually non-integer. Since this parameter is not on the pixel grid of the target image, directly rounding it will lead to registration blurring or misalignment. Therefore, bilinear interpolation (4-neighborhood) or bicubic interpolation (16-neighborhood) is required to calculate the pixel value. Taking the source coordinates as the center, four or sixteen integer pixels around them are taken, and the pixel value at that position is calculated by weighting according to distance. Finally, after mapping and interpolating all reference image pixels in the above manner, a registered image with the same size as the reference image is generated. At this time, the feature points of the two modal images are spatially aligned, thus obtaining the registration result after the two modal images are registered.

[0137] It should be noted that the above parameters are for illustrative purposes only and are not intended to limit this application.

[0138] Optionally, before step S44, when the scaling factor s When (D) < 1, it indicates that the target modality image will undergo downsampling during registration. Since downsampling causes aliasing of high-frequency information, a low-pass pre-filter needs to be applied to the target modality image before resampling to suppress aliasing, and the pre-filter strength varies with... s (D) Decrease and increase incrementally to ensure that the downsampled image retains sufficient detail and sharpness. Specifically, when the scaling factor of the affine transformation parameter is less than one and downsampling occurs, a low-pass pre-filter is applied to the target modal image to suppress aliasing. Simultaneously, during interpolation sampling, if the calculated pixel value exceeds the effective range of the target modal image, a boundary strategy is applied, and an effective field-of-view mask M is generated to indicate the effective pixels. The effective pixels in the resampled image are then used as the registration output. That is, when the source coordinates exceed the effective area of ​​the target modal image, the source coordinates exceeding the effective area are filled based on a preset boundary processing strategy, and effective area indication information is generated to identify the usable pixels in the registration result. Example images of the image pairs before and after registration are shown below. Figure 7 As shown.

[0139] This embodiment provides a UAV multimodal image registration method based on shooting distance parameter. It calls affine transformation parameters by using the actual shooting distance, and performs pixel coordinate mapping processing based on the inverse matrix of the affine transformation matrix constructed according to the affine transformation parameters, thereby achieving pixel-level accurate correspondence. Combined with interpolation processing, it solves the sampling problem of non-integer coordinates, and finally improves the registration accuracy of multimodal images.

[0140] Furthermore, based on the third embodiment of this application, in the fifth embodiment of this application, when the transformation parameter is a homography transformation parameter, step S40 further includes steps S45~S48:

[0141] Step S45: Based on the shooting distance, retrieve the homography transformation matrix corresponding to the mapping relationship.

[0142] Step S46: Calculate the inverse matrix of the homography transformation matrix based on the homography transformation matrix.

[0143] Step S47: Multiply all pixel coordinates in the reference image by the inverse of the homography transformation matrix and perform perspective normalization to obtain the source coordinates of all pixel values ​​in the target modal image.

[0144] Step S48: Perform interpolation sampling on the pixel values ​​at the source coordinates to obtain the registration result.

[0145] In this embodiment, the image registration process based on the homography transformation matrix is ​​the same as that of the affine transformation matrix. Both processes retrieve parameters / matrices based on the shooting distance, then calculate the inverse matrix, convert the coordinates of the reference image to the corresponding source coordinates in the target modal image, and finally perform coordinate interpolation sampling to complete the image pairing. The specific process is not detailed in this application.

[0146] This application provides an image registration system, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the UAV multimodal image registration method based on shooting distance parameters in the first embodiment described above.

[0147] The following is for reference. Figure 8 This document illustrates a structural schematic diagram suitable for implementing the image registration system of the embodiments of this application. The device carrier of the image registration system in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The image registration system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0148] like Figure 8As shown, the image registration system may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the image registration system. The processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the image registration system to communicate wirelessly or wiredly with other devices to exchange data. Although an image registration system with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0149] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0150] The image registration system provided in this application employs the UAV multimodal image registration method based on shooting distance parameters in the above embodiments, which can solve the technical problem of low image registration accuracy. Compared with the prior art, the beneficial effects of the image registration system provided in this application are the same as those of the UAV multimodal image registration method based on shooting distance parameters provided in the above embodiments, and other technical features of this image registration system are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0151] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0152] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0153] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the UAV multimodal image registration method based on shooting distance parameters in the above embodiments.

[0154] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM, or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0155] The aforementioned computer-readable storage medium may be included in the image registration system; or it may exist independently and not be assembled into the image registration system.

[0156] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the image registration system, enable the image registration system to: acquire reference modal images and target modal images collected by multiple camera modules with different imaging principles on the same UAV platform;

[0157] The shooting distance is determined by the distance between the drone gimbal and the reference plane where the target is located;

[0158] Based on the distance-transformation parameter mapping relationship generated during the calibration phase, the geometric transformation parameters of the target modal image registered to the reference modal image at the shooting distance are determined.

[0159] The target modal image is resampled to the coordinate system of the reference modal image based on the geometric transformation parameters, so as to register the reference modal image and the target modal image.

[0160] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0161] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0162] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0163] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described UAV multimodal image registration method based on shooting distance parameters, thereby solving the technical problem of low image registration accuracy. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the UAV multimodal image registration method based on shooting distance parameters provided in the above embodiments, and will not be repeated here.

[0164] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for multimodal image registration of unmanned aerial vehicles (UAVs) based on shooting distance parameters, characterized in that, The UAV multimodal image registration method based on shooting distance parameter includes: Obtain the drone at multiple preset shooting distances D k Below, reference images and target images with the same modal features acquired in the same target scene; Construct a set of image pairs consisting of the target image and the reference image at the multiple preset shooting distances, and extract and match common features under multimodal conditions for each image pair in the set to establish a set of correspondences for calibration, including: extracting planar feature point pairs of the reference image and the target image; Based on the set of correspondences, the transformation parameters corresponding to the multiple preset shooting distances are calculated, including: solving for each preset shooting distance D based on the planar feature point pairs. k Below, the homography transformation matrix from the target image to the reference image H k ; Based on the camera imaging geometry model, the homography transformation matrix is ​​constructed. H k With the preset shooting distance D k The relationship between them: H k = H 0+1 / (D k -D z )·H1; Based on the transformation parameters at the multiple preset shooting distances, an interpolation or regression model is constructed to obtain the mapping relationship between the distance and the transformation parameters, including based on the homography transformation matrix at the multiple preset shooting distances. H k Building about H 0、 H 1. Δ z The overdetermined system of equations is solved using the least squares method to obtain the calibration parameters corresponding to the preset shooting distance. H 0、 H 1 and Δ z This yields a homography mapping model based on parameterized distance; Acquire reference modal images and target modal images from multiple camera modules with different imaging principles on the same UAV platform; The shooting distance D is defined as the distance between the drone gimbal and the reference plane where the target is located. The target is the exterior of an urban building, and the reference plane is a single plane or near-plane area of ​​the exterior of the urban building. Based on the distance-transformation parameter mapping relationship generated during the calibration phase, the geometric transformation parameters of the target modal image registered to the reference modal image at the shooting distance are determined, and the geometric transformation parameters are homography transformation matrices; Based on the geometric transformation parameters, the target modal image is resampled to the coordinate system of the reference modal image to register the reference modal image and the target modal image, including... Based on the shooting distance D, according to: Calculate the homography transformation matrix H(D) at the shooting distance; Calculate the inverse matrix of the homography transformation matrix H(D); Multiply the coordinates of all pixel values ​​in the reference modal image by the inverse matrix on the left, and perform perspective normalization to obtain the source coordinates of all pixel values ​​in the target modal image; The pixel values ​​at the source coordinates are interpolated and sampled to obtain the registration result.

2. The UAV multimodal image registration method based on shooting distance parameter as described in claim 1, characterized in that, The step of calculating the transformation parameters corresponding to the multiple preset shooting distances based on the correspondence set includes: Based on the set of correspondences, the affine transformation parameters between the reference image and the target image are solved at the multiple preset shooting distances respectively; The step of constructing an interpolation or regression model based on the transformation parameters under the multiple preset shooting distances to obtain the mapping relationship between the distance and the transformation parameters includes: By interpolating or regressing the affine transformation parameters, a mapping relationship between the distance and the transformation parameters is obtained. The corresponding affine transformation parameters are retrieved based on the mapping relationship constructed by the affine transformation parameters and the shooting distance.

3. The UAV multimodal image registration method based on shooting distance parameter as described in claim 2, characterized in that, The step of resampling the target modality image to the coordinate system of the reference modality image based on the geometric transformation parameters, so as to register the reference modality image and the target modality image, includes: Based on the shooting distance, the affine transformation parameters corresponding to the mapping relationship are called to construct the affine transformation matrix corresponding to the affine transformation parameters; Calculate the inverse of the affine transformation matrix; Multiply the coordinates of all pixel values ​​in the reference modal image by the inverse of the affine transformation matrix to obtain the source coordinates of all pixel values ​​in the target modal image. The pixel values ​​at the source coordinates are interpolated and sampled to obtain the registration result.

4. The UAV multimodal image registration method based on shooting distance parameter as described in claim 3, characterized in that, Before the step of interpolating and sampling the pixel values ​​under the source coordinates to obtain the registration result, the UAV multimodal image registration method based on the shooting distance parameter further includes: When the scaling factor of the affine transformation parameter is less than one and downsampling occurs, the target modal image is low-pass pre-filtered to suppress aliasing of the target modal image; When the source coordinates exceed the effective area of ​​the target modal image, the pixel source coordinates that exceed the effective area are filled based on a preset boundary processing strategy, and effective area indication information is generated to identify the available pixels in the registration result; Perform the step of interpolating and sampling the pixel values ​​at the source coordinates to obtain the registration result.

5. The UAV multimodal image registration method based on shooting distance parameter as described in claim 1, characterized in that, Following the step of acquiring reference modal images and target modal images collected by multiple camera modules with different imaging principles on the same UAV platform, the UAV multimodal image registration method based on shooting distance parameters further includes: Radial and tangential distortion corrections are performed on the reference modal image and the target modal image.

6. An image registration system, characterized in that, The image registration system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the UAV multimodal image registration method based on shooting distance parameters as described in any one of claims 1 to 5.

7. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the UAV multimodal image registration method based on shooting distance parameters as described in any one of claims 1 to 5.

Images