Multimodal optical and radar image registration method, apparatus, device and storage medium

Abstract

The application provides a multi-modal optical and radar image registration method, device and equipment and a readable storage medium, relates to the technical field of data fusion, and comprises the following steps: acquiring an optical image and a radar amplitude image of a same observation area; preprocessing the optical image and the radar amplitude image to obtain a first preprocessed image and a second preprocessed image; acquiring a corresponding point set between the first preprocessed image and the second preprocessed image; estimating an affine transformation matrix from the first preprocessed image to the second preprocessed image based on the corresponding point set by using a random sample consensus algorithm and a least square method; and performing coordinate transformation and pixel resampling on the first preprocessed image by using the affine transformation matrix to obtain a registered optical image. The application is used for solving the technical problems of poor feature repeatability and high false matching rate in the prior art.

Description

Technical Field

[0001] This invention relates to the field of data fusion technology, and more specifically, to a method, apparatus, device, and storage medium for multimodal optical and radar image registration. Background Technology

[0002] Optical images and radar (SAR) images differ inherently in their geometry and radiometric properties due to their different imaging principles, making high-precision registration between them a persistent technical challenge. Current mainstream methods aim to automatically find matching feature points between two images by improving feature extraction algorithms or utilizing deep learning models. However, when dealing with images from different sources, especially in the presence of noise and distortion, these methods often suffer from poor feature repeatability and high mismatch rates, resulting in limited registration accuracy and insufficient algorithm stability, making it difficult to meet the high-reliability fusion requirements of practical applications. Summary of the Invention

[0003] The purpose of this invention is to provide a multimodal optical and radar image registration method, apparatus, device, and readable storage medium to improve the aforementioned problems. To achieve the above objective, the technical solution adopted by this invention is as follows:

[0004] In a first aspect, this application provides a multimodal optical and radar image registration method, including:

[0005] Acquire optical images and radar amplitude maps of the same observation area;

[0006] The optical image is subjected to grayscale and normalization processing to obtain a first preprocessed image; the radar amplitude image is subjected to logarithmic transformation and noise suppression processing to obtain a second preprocessed image.

[0007] Obtain a set of corresponding points between the first preprocessed image and the second preprocessed image, wherein the set of corresponding points contains at least three non-collinear point pairs;

[0008] Based on the corresponding point set, the affine transformation matrix from the first preprocessed image to the second preprocessed image is estimated using the random sampling consensus algorithm and the least squares method.

[0009] The first preprocessed image is subjected to coordinate transformation and pixel resampling using the affine transformation matrix to obtain the registered optical image.

[0010] Secondly, this application also provides a multimodal optical and radar image registration device, comprising:

[0011] The data acquisition module is used to acquire optical images and radar amplitude maps of the same observation area;

[0012] The preprocessing module is used to perform grayscale and normalization processing on the optical image to obtain a first preprocessed image; and to perform logarithmic transformation and noise suppression processing on the radar amplitude map to obtain a second preprocessed image.

[0013] The corresponding point acquisition module is used to acquire a set of corresponding points between the first preprocessed image and the second preprocessed image, wherein the set of corresponding points contains at least three non-collinear point pairs.

[0014] The transformation estimation module is used to estimate the affine transformation matrix from the first preprocessed image to the second preprocessed image based on the corresponding point set, using a random sampling consensus algorithm and the least squares method.

[0015] The registration module is used to perform coordinate transformation and pixel resampling on the first preprocessed image using the affine transformation matrix to obtain the registered optical image.

[0016] Thirdly, this application also provides a multimodal optical and radar image registration device, comprising:

[0017] Memory, used to store computer programs;

[0018] A processor is used to implement the steps of the multimodal optical and radar image registration method when executing the computer program.

[0019] Fourthly, this application also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described multimodal optical and radar image registration method.

[0020] The beneficial effects of this invention are as follows:

[0021] This invention obtains a set of corresponding points between a first preprocessed image and a second preprocessed image, and then performs feature matching on the two images. Simultaneously, based on this set of corresponding points, it estimates the affine transformation matrix using a random sampling consensus algorithm and the least squares method. The affine transformation matrix is ​​then used to perform coordinate transformation and pixel resampling on the first preprocessed image, effectively resisting error interference and achieving high-precision coordinate transformation. Compared to existing methods, this invention does not require large-scale training data, supports small sample corresponding points, and is suitable for embedded devices.

[0022] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing embodiments of the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the multimodal optical and radar image registration method described in this embodiment of the invention;

[0025] Figure 2 This is a schematic diagram of the multimodal optical and radar image registration device described in an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of the structure of the multimodal optical and radar image registration device described in an embodiment of the present invention.

[0027] Marked in the image:

[0028] 800. Multimodal optical and radar image registration equipment; 801. Processor; 802. Memory; 803. Multimedia components; 804. I / O interface; 805. Communication components. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0030] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] Example 1:

[0032] This embodiment provides a multimodal optical and radar image registration method. In practice, it is necessary to use two imaging sensors fixed at the same geographical location to conduct synchronous observation of the exact same surface area.

[0033] See Figure 1 The figure shows that this method includes:

[0034] S1. Acquire optical images and radar amplitude maps of the same observation area;

[0035] Specifically, remote sensing equipment is used to acquire optical images of the observation area. The optical image It contains three color channels: red (R), green (G), and blue (B), with a space size of [missing information]. ,in, Represent real numbers, For height, For width.

[0036] Specifically, a ground-based synthetic aperture radar (GB-SAR) system is used to perform microwave scanning imaging of the observation area. The GB-SAR system moves the synthetic aperture along a linear track with its antenna and receives backscattered signals from ground objects. After imaging processing, a single-channel radar amplitude map is obtained. The radar amplitude map The intensity of microwave signal scattering by ground features was recorded, and its spatial dimensions were: ,in, Represent real numbers, For height, The width is specified. The term "ground features" refers to all types of ground targets and characteristics within the observation area.

[0037] Preferably, the optical image and the radar amplitude map need to meet the requirements of spatiotemporal consistency, that is, the surface areas covered by the two images should highly overlap, and the time interval between the two imaging should be as short as possible to reduce the registration error caused by target changes.

[0038] Based on the above embodiments, this method further includes:

[0039] S2. Perform grayscale and normalization processing on the optical image to obtain a first preprocessed image; perform logarithmic transformation and noise suppression processing on the radar amplitude image to obtain a second preprocessed image;

[0040] Specifically, step S2 includes:

[0041] S21. Convert the optical image to grayscale, transforming the color optical image into a single-channel grayscale image. Specifically, the grayscale conversion is performed using the following formula:

[0042] ;

[0043] In the formula, This represents the image after grayscale conversion. , , Representing images respectively pixel values, These are pixel coordinates.

[0044] S22. Normalize the optical image, linearly mapping its pixel intensity values ​​to... Within the interval, the first preprocessed image is obtained. :

[0045] S23. Perform a logarithmic transformation on the radar amplitude map to compress high-intensity areas and stretch low-intensity areas, making the image features more uniform; specifically, the logarithmic transformation is performed using the following formula:

[0046] , ;

[0047] In the formula, Represents the image after logarithmic transformation. This is a radar amplitude diagram. It is a very small constant.

[0048] S24. Perform noise suppression processing on the radar amplitude map. Preferably, this embodiment uses an adaptive filter for noise suppression, such as a Lee filter, specifically calculated using the following formula:

[0049] ;

[0050] In the formula, This is the image after noise suppression processing, i.e., the second preprocessed image. For The mean value of pixels within a local window centered on the pixel. For adaptive gain coefficients, This is the image after logarithmic transformation.

[0051] Based on the above embodiments, this method further includes:

[0052] S3. Obtain a set of corresponding points between the first preprocessed image and the second preprocessed image, wherein the set of corresponding points contains at least three non-collinear point pairs;

[0053] Specifically, step S3 includes:

[0054] S31. Perform corner detection on the first preprocessed image to generate an optical feature point set;

[0055] This embodiment uses the Harris corner detection algorithm to extract feature points. Specifically, it calculates the first preprocessed image. exist and gradient in direction and Construct a local structure tensor for each pixel. :

[0056] ;

[0057] In the formula, Represents the local structure tensor. and Indicates the first preprocessed image exist and Gradient in the direction.

[0058] Subsequently, based on the local structure tensor Calculate the corner response function for each pixel. ;

[0059] Finally, by all Pixels that are greater than a preset threshold and are local maxima constitute the optical feature point set. Preferably, the preset threshold value is 0.01.

[0060] S32. Perform edge detection on the second preprocessed image to generate a radar feature point set;

[0061] Specifically, this embodiment uses a Ratio of Averages (ROA) edge detector for edge detection. Edges are identified by comparing the average intensity ratio of different sub-regions within a pixel's neighborhood. Pixels with a ratio below a set threshold are determined to be edge points; preferably, the set threshold is 1.5. Non-maximum suppression is applied to the detected edge pixels, and the resulting edge pixels are used as a radar feature point set. .

[0062] S33. Based on the nearest neighbor distance ratio principle, the optical feature point set and the radar feature point set are initially matched to form an initial matching pair;

[0063] For optical feature point sets Each optical feature point in Calculate its relationship with the radar feature point set in sequence. Each radar feature point Euclidean distance, searching for the nearest neighbor distance among all calculated Euclidean distances. Distance to next nearest neighbor .

[0064] According to the nearest neighbor distance ratio principle, if the ratio of the nearest neighbor distance to the second nearest neighbor distance is less than a preset threshold, the optical feature point and the radar feature point with the nearest neighbor distance form an initial matching pair.

[0065] S34. Select at least three non-collinear point pairs from the initial matched point pairs to form the corresponding point set. ,in, , Indicates belonging to the optical feature point set Optical feature points, , and express coordinates Indicates belonging to the radar feature point set Radar feature points, , and express The coordinates.

[0066] Based on the above embodiments, this method further includes:

[0067] S4. Based on the set of corresponding points, estimate the affine transformation matrix from the first preprocessed image to the second preprocessed image using the random sampling consensus algorithm and the least squares method;

[0068] Specifically, step S4 includes:

[0069] S41. Randomly select a preset number of point pairs from the corresponding point set to construct a sample subset, wherein the preset number is the minimum number of point pairs required to solve the affine transformation matrix;

[0070] Specifically, from the set of corresponding points Random selection For each pair of points, construct a sample subset. ,in, The minimum number of point pairs required to solve for an affine transformation matrix. For example, for a two-dimensional affine transformation with 6 degrees of freedom, the minimum solution requires... Non-collinear point pairs. To ensure numerical stability and improve robustness, this embodiment can select [a specific pair] during actual calculations. .

[0071] S42. Based on the aforementioned sample subset, candidate affine transformation matrices are obtained through least squares fitting:

[0072] ;

[0073] In the formula, Denotes the candidate affine transformation matrix. , , , These are all parameters used to jointly control the rotation, scaling, and transformation of the image. This represents the amount of translation in the horizontal direction. This indicates the amount of translation in the vertical direction.

[0074] S43. Using the candidate affine transformation matrix, calculate the transformation error of all point pairs in the corresponding point set, and mark the point pairs with transformation errors less than a preset threshold as interior points of the candidate affine transformation matrix;

[0075] Specifically, the formula for calculating the transformation error is as follows:

[0076] ;

[0077] In the formula, Indicates the first Transformation error of a pair of points Let be the candidate affine transformation matrix. Indicates the first In a pair of points, the coordinates of the points located on the first preprocessed image can be represented as: , Indicates the first In a pair of points, the coordinates of the points located on the second preprocessed image can be represented as follows: , This represents the L2 norm.

[0078] When transformation error If the condition is met, then it is marked as an interior point of the candidate affine transformation matrix; otherwise, it is marked as an exterior point. To account for the preset error, the preferred option is... The value is 1.5.

[0079] S44. Repeat steps S41 to S43, and select the candidate affine transformation matrix with the most interior points as the optimal candidate matrix.

[0080] S45. Using the interior points corresponding to the optimal candidate matrix, the final affine transformation matrix is ​​recalculated using the least squares method;

[0081] Specifically, based on all point pairs corresponding to the optimal candidate matrix, the affine transformation matrix is ​​recalculated using the least squares method to obtain the final optimal affine transformation matrix. .

[0082] Based on the above embodiments, this method further includes:

[0083] S5. The first preprocessed image is subjected to coordinate transformation and pixel resampling using the affine transformation matrix to obtain the registered optical image;

[0084] Specifically, step S5 includes:

[0085] S51. Define the target image space based on the affine transformation matrix and the second preprocessed image;

[0086] In this embodiment, the size and spatial range of the second preprocessed image are used as a reference to define the target image space. Specifically, the defined target image space... The coordinate range in the x-axis direction is In the y-axis direction ,in, For height, For width.

[0087] S52. Calculate the inverse coordinate mapping relationship from the target image space to the first preprocessed image space;

[0088] Specifically, for the target image space Each integer coordinate point in The corresponding source point coordinates are found in the first preprocessed image, and the expression for the source point coordinates is:

[0089] ;

[0090] In the formula, Represents the coordinates of the source point, where , The source point's floating-point coordinates in the first preprocessed image coordinate system. for The inverse matrix, The optimal affine transformation matrix. The coordinates are integers.

[0091] S53. Determine the sub-pixel position of each coordinate point in the target image space in the first preprocessed image based on the reverse coordinate mapping relationship;

[0092] S54. Using bilinear interpolation, calculate the pixel values ​​of the first preprocessed image at each sub-pixel position to generate the registered optical image;

[0093] This embodiment uses bilinear interpolation. Search for the values ​​of the four nearest neighbor integer pixels to calculate the pixel value:

[0094] ;

[0095] In the formula, This indicates that the value is calculated based on sub-pixel positions. Pixel values ​​in the registered image for The decimal part, for The decimal part, , , as well as These represent the pixel values ​​of four adjacent integer pixels. for The integer obtained by rounding down. for The integer obtained by rounding down.

[0096] The pixel values ​​of the first preprocessed image at each sub-pixel position are sequentially traversed, and the calculated pixel values ​​are filled into the target image space in order to obtain the registered optical image. .

[0097] Based on the above embodiments, this method further includes:

[0098] S6. The registered optical image and the second preprocessed image are weighted and fused to generate a fused image; specifically, the following formula is used for fusion:

[0099] ;

[0100] In the formula, To merge images, To integrate the weighting coefficients, The registered optical image, This is the second preprocessed image. Preferably, let... This is to highlight the visual details of the optical image in the fusion result while preserving the feature information of the radar image.

[0101] Based on the above embodiments, this method further includes:

[0102] S7. Calculate the registration quality evaluation index based on the fused image and the second preprocessed image.

[0103] Specifically, step S7 includes:

[0104] S71. Transform the coordinates from the first preprocessed image in each point pair into predicted coordinates using the affine transformation matrix; specifically, using the affine transformation matrix... The coordinates of each optical image point are transformed to the radar image space to obtain the predicted coordinates:

[0105] ;

[0106] In the formula, Indicates the predicted coordinates. Denotes the optimal affine transformation matrix. The coordinates of the point located on the second preprocessed image, ( )for The predicted coordinates.

[0107] S72. Calculate the Euclidean distance between the predicted coordinates of each point pair and its corresponding coordinates in the second preprocessed image:

[0108] ;

[0109] In the formula, Indicates predicted coordinates Coordinates corresponding to real radar images ( The Euclidean distance between points is the registration error between point pairs.

[0110] S73. Calculate the arithmetic mean of the Euclidean distances for all point pairs to obtain the reprojection error, and use the reprojection error as an indicator of registration quality:

[0111] ;

[0112] In the formula, The arithmetic mean of the Euclidean distance represents the reprojection error of the registration. Indicates the number of point pairs. This represents the registration error between point pairs. In this embodiment, when the reprojection error... When the value is 0, it indicates that the registration result is good.

[0113] Alternatively, mutual information and structural similarity index can be used as registration quality evaluation indicators; specifically, mutual information is used to measure the degree of information association between two images, and structural similarity index is used to evaluate the similarity of structural features.

[0114] Example 2:

[0115] like Figure 2 As shown, this embodiment provides a multimodal optical and radar image registration device, the device comprising:

[0116] The data acquisition module is used to acquire optical images and radar amplitude maps of the same observation area;

[0117] The preprocessing module is used to perform grayscale and normalization processing on the optical image to obtain a first preprocessed image; and to perform logarithmic transformation and noise suppression processing on the radar amplitude map to obtain a second preprocessed image.

[0118] The corresponding point acquisition module is used to acquire a set of corresponding points between the first preprocessed image and the second preprocessed image, wherein the set of corresponding points contains at least three non-collinear point pairs.

[0119] The transformation estimation module is used to estimate the affine transformation matrix from the first preprocessed image to the second preprocessed image based on the corresponding point set, using a random sampling consensus algorithm and the least squares method.

[0120] The registration module is used to perform coordinate transformation and pixel resampling on the first preprocessed image using the affine transformation matrix to obtain the registered optical image.

[0121] Based on the above embodiments, the corresponding point acquisition module includes:

[0122] A corner detection unit is used to perform corner detection on the first preprocessed image and generate an optical feature point set;

[0123] An edge detection unit is used to perform edge detection on the second preprocessed image and generate a radar feature point set;

[0124] The feature matching unit is used to perform preliminary matching between the optical feature point set and the radar feature point set according to the nearest neighbor distance ratio principle to form an initial matching pair;

[0125] The filtering unit is used to filter out at least three non-collinear point pairs from the initial matched point pairs to form the corresponding point set.

[0126] Based on the above embodiments, the transformation estimation module includes:

[0127] A sampling unit is used to randomly select a preset number of point pairs from the corresponding point set to construct a sample subset, wherein the preset number is the minimum number of point pairs required to solve the affine transformation matrix;

[0128] A fitting unit is used to obtain a candidate affine transformation matrix by least squares fitting based on the sample subset.

[0129] The interior point marking unit is used to calculate the transformation error of all point pairs in the corresponding point set using the candidate affine transformation matrix, and mark the point pairs with transformation errors less than a preset threshold as interior points of the candidate affine transformation matrix.

[0130] The iterative optimization unit is used to repeatedly execute the operations of the sampling unit, the fitting unit, and the interior point marking unit, and to select the candidate affine transformation matrix with the most interior points as the optimal candidate matrix.

[0131] The recalculation unit is used to recalculate the final affine transformation matrix using the interior points corresponding to the optimal candidate matrix through the least squares method.

[0132] Based on the above embodiments, the registration module includes:

[0133] A spatial definition unit is used to define the target image space based on the affine transformation matrix and the second preprocessed image;

[0134] A mapping calculation unit is used to calculate the inverse coordinate mapping relationship from the target image space to the first preprocessed image space;

[0135] The coordinate determination unit is used to determine the sub-pixel position of each coordinate point in the target image space in the first preprocessed image according to the reverse coordinate mapping relationship.

[0136] A pixel resampling unit is used to calculate the pixel value of the first preprocessed image at each sub-pixel position using bilinear interpolation to generate the registered optical image.

[0137] Based on the above embodiments, the device further includes:

[0138] The fusion module is used to perform weighted fusion of the registered optical image and the second preprocessed image to generate a fused image;

[0139] An evaluation module is used to calculate a registration quality evaluation index based on the fused image and the second preprocessed image.

[0140] Based on the above embodiments, the evaluation module includes:

[0141] A coordinate transformation unit is used to transform the coordinates from the first preprocessed image in each point pair into predicted coordinates through the affine transformation matrix;

[0142] The distance calculation unit is used to calculate the Euclidean distance between the predicted coordinates of each point pair and its corresponding coordinates in the second preprocessed image.

[0143] The error calculation unit is used to calculate the arithmetic mean of the Euclidean distances of all point pairs to obtain the reprojection error, which is then used as a registration quality evaluation index.

[0144] It should be noted that the specific manner in which each module performs its operation in the apparatus described in the above embodiments has been described in detail in the embodiments of the method, and will not be elaborated here.

[0145] Example 3:

[0146] Corresponding to the above method embodiments, this embodiment also provides a multimodal optical and radar image registration device. The multimodal optical and radar image registration device described below and the multimodal optical and radar image registration method described above can be referred to in correspondence.

[0147] Figure 3 This is a block diagram illustrating a multimodal optical and radar image registration device 800 according to an exemplary embodiment. Figure 3 As shown, the multimodal optical and radar image registration device 800 may include a processor 801 and a memory 802. The multimodal optical and radar image registration device 800 may also include one or more of a multimedia component 803, an I / O interface 804, and a communication component 805.

[0148] The processor 801 controls the overall operation of the multimodal optical and radar image registration device 800 to complete all or part of the steps in the aforementioned multimodal optical and radar image registration method. The memory 802 stores various types of data to support the operation of the multimodal optical and radar image registration device 800. This data may include, for example, instructions for any application or method operating on the multimodal optical and radar image registration device 800, as well as application-related data such as contact data, sent and received messages, images, audio, video, etc. The memory 802 can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 803 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in the memory 802 or transmitted via the communication component 805. The audio component also includes at least one speaker for outputting audio signals. I / O interface 804 provides an interface between processor 801 and other interface modules, such as keyboards, mice, and buttons. These buttons can be virtual or physical. Communication component 805 is used for wired or wireless communication between the multimodal optical and radar image registration device 800 and other devices. Wireless communication includes Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination thereof. Therefore, the corresponding communication component 805 may include a Wi-Fi module, a Bluetooth module, or an NFC module.

[0149] In an exemplary embodiment, the multimodal optical and radar image registration device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the multimodal optical and radar image registration method described above.

[0150] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the multimodal optical and radar image registration method described above. For example, the computer-readable storage medium may be the memory 802 including the program instructions described above, which may be executed by the processor 801 of the multimodal optical and radar image registration device 800 to complete the multimodal optical and radar image registration method described above.

[0151] Example 4:

[0152] Corresponding to the above method embodiments, this embodiment also provides a readable storage medium. The readable storage medium described below can be referred to in conjunction with the multimodal optical and radar image registration method described above.

[0153] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the multimodal optical and radar image registration method described in the above method embodiments.

[0154] Specifically, the readable storage medium can be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other readable storage medium capable of storing program code.

[0155] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0156] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal optical and radar image registration method, characterized in that, include: Acquire optical images and radar amplitude maps of the same observation area; The optical image is subjected to grayscale and normalization processing to obtain a first preprocessed image; The radar amplitude image is subjected to logarithmic transformation and noise suppression to obtain a second preprocessed image; Obtain a set of corresponding points between the first preprocessed image and the second preprocessed image, wherein the set of corresponding points contains at least three non-collinear point pairs; Based on the corresponding point set, the affine transformation matrix from the first preprocessed image to the second preprocessed image is estimated using the random sampling consensus algorithm and the least squares method. The first preprocessed image is subjected to coordinate transformation and pixel resampling using the affine transformation matrix to obtain the registered optical image.

2. The multimodal optical and radar image registration method according to claim 1, characterized in that, Obtaining the set of corresponding points between the first preprocessed image and the second preprocessed image includes: Corner detection is performed on the first preprocessed image to generate an optical feature point set; Edge detection is performed on the second preprocessed image to generate a radar feature point set; Based on the nearest neighbor distance ratio principle, the optical feature point set and the radar feature point set are initially matched to form an initial matching pair; At least three non-collinear point pairs are selected from the initial matched point pairs to form the corresponding point set.

3. The multimodal optical and radar image registration method according to claim 1, characterized in that, Based on the corresponding point set, the affine transformation matrix from the first preprocessed image to the second preprocessed image is estimated using a random sampling consensus algorithm and the least squares method, including: A preset number of point pairs are randomly selected from the corresponding point set to construct a sample subset, wherein the preset number is the minimum number of point pairs required to solve the affine transformation matrix; Based on the sample subset, candidate affine transformation matrices are obtained by least squares fitting. Using the candidate affine transformation matrix, the transformation error of all point pairs in the corresponding point set is calculated, and point pairs with transformation errors less than a preset threshold are marked as interior points of the candidate affine transformation matrix. Repeatedly fit the candidate affine transformation matrix, and take the candidate affine transformation matrix with the most interior points as the optimal candidate matrix. Using the interior points corresponding to the optimal candidate matrix, the final affine transformation matrix is ​​recalculated using the least squares method.

4. The multimodal optical and radar image registration method according to claim 1, characterized in that, The first preprocessed image is geometrically transformed and resampled using the affine transformation matrix to obtain a registered optical image, including: Based on the affine transformation matrix and the second preprocessed image, a target image space is defined; Calculate the inverse coordinate mapping relationship from the target image space to the first preprocessed image space; Based on the reverse coordinate mapping relationship, determine the sub-pixel position of each coordinate point in the target image space in the first preprocessed image; Using bilinear interpolation, the pixel values ​​of the first preprocessed image at each sub-pixel position are calculated to generate the registered optical image.

5. A multimodal optical and radar image registration device, characterized in that, include: The data acquisition module is used to acquire optical images and radar amplitude maps of the same observation area; The preprocessing module is used to perform grayscale and normalization processing on the optical image to obtain a first preprocessed image; and to perform logarithmic transformation and noise suppression processing on the radar amplitude map to obtain a second preprocessed image. The corresponding point acquisition module is used to acquire a set of corresponding points between the first preprocessed image and the second preprocessed image, wherein the set of corresponding points contains at least three non-collinear point pairs. The transformation estimation module is used to estimate the affine transformation matrix from the first preprocessed image to the second preprocessed image based on the corresponding point set, using a random sampling consensus algorithm and the least squares method. The registration module is used to perform coordinate transformation and pixel resampling on the first preprocessed image using the affine transformation matrix to obtain the registered optical image.

6. The multimodal optical and radar image registration device according to claim 5, characterized in that, The corresponding point acquisition module includes: A corner detection unit is used to perform corner detection on the first preprocessed image and generate an optical feature point set; An edge detection unit is used to perform edge detection on the second preprocessed image and generate a radar feature point set; The feature matching unit is used to perform preliminary matching between the optical feature point set and the radar feature point set according to the nearest neighbor distance ratio principle to form an initial matching pair; The filtering unit is used to filter out at least three non-collinear point pairs from the initial matched point pairs to form the corresponding point set.

7. The multimodal optical and radar image registration device according to claim 5, characterized in that, The transformation estimation module includes: A sampling unit is used to randomly select a preset number of point pairs from the corresponding point set to construct a sample subset, wherein the preset number is the minimum number of point pairs required to solve the affine transformation matrix; A fitting unit is used to obtain a candidate affine transformation matrix by least squares fitting based on the sample subset. The interior point marking unit is used to calculate the transformation error of all point pairs in the corresponding point set using the candidate affine transformation matrix, and mark the point pairs with transformation errors less than a preset threshold as interior points of the candidate affine transformation matrix. The iterative optimization unit is used to repeatedly execute the operations of the sampling unit, the fitting unit, and the interior point marking unit, and to select the candidate affine transformation matrix with the most interior points as the optimal candidate matrix. The recalculation unit is used to recalculate the final affine transformation matrix using the interior points corresponding to the optimal candidate matrix through the least squares method.

8. The multimodal optical and radar image registration device according to claim 5, characterized in that, The registration module includes: A spatial definition unit is used to define the target image space based on the affine transformation matrix and the second preprocessed image; A mapping calculation unit is used to calculate the inverse coordinate mapping relationship from the target image space to the first preprocessed image space; The coordinate determination unit is used to determine the sub-pixel position of each coordinate point in the target image space in the first preprocessed image according to the reverse coordinate mapping relationship. A pixel resampling unit is used to calculate the pixel value of the first preprocessed image at each sub-pixel position using bilinear interpolation to generate the registered optical image.

9. A multimodal optical and radar image registration device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the multimodal optical and radar image registration method as described in any one of claims 1 to 4 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the multimodal optical and radar image registration method as described in any one of claims 1 to 4.

Images