Multimodal remote sensing image registration method, device, equipment and storage medium

Abstract

The present invention provides a multimodal remote sensing image registration method, device, equipment and storage medium, belonging to the field of image data processing technology, the method includes obtaining a multimodal remote sensing image to be registered; processing the multimodal remote sensing image based on a relative total variation algorithm, and determining a directional feature map of the multimodal remote sensing image based on a multi-directional and multi-scale logarithmic Gabor filter operator; extracting features from the directional feature map based on a phase consistency algorithm to determine key points of the multimodal remote sensing image; obtaining a descriptor of the multimodal remote sensing image based on the directional feature map and the key points; and performing feature matching on the descriptor to determine the registration result of the multimodal remote sensing image. The multimodal remote sensing image is subjected to image main structure enhancement and image texture suppression by a relative total variation algorithm, and feature extraction is performed by a phase consistency algorithm to extract main strong structural features, thereby avoiding modal differences between multi-source remote sensing images caused by nonlinear radiation distortion.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a multimodal remote sensing image registration method, device, equipment and storage medium. Background Art

[0002] Image registration is the process of aligning two or more images of the same scene taken at different times, from different viewpoints, or using different sensors. In the field of photogrammetry and remote sensing, image registration is essential for advanced tasks such as change detection, image fusion, and image stitching. In general, image registration methods can be divided into region-based, feature-based, and learning-based methods. There is little research on learning-based methods in the field of remote sensing, mainly because it is difficult to obtain enough multimodal remote sensing images for effective training and testing. Region-based methods are limited to two-dimensional translation searches, are sensitive to geometric transformations, and require geographic information for initialization. In contrast, feature-based methods are more flexible and robust, and are also the mainstream multimodal remote sensing image registration methods.

[0003] Existing feature-based image registration methods mainly capture gradient features through intensity information, are very sensitive to nonlinear radiation changes, and are prone to nonlinear radiation distortion. The effect of multimodal image registration needs to be improved. Summary of the invention

[0004] The present invention provides a multimodal remote sensing image registration method, device, equipment and storage medium to overcome the defects of the prior art that the gradient characteristics are mainly captured by intensity information, which is very sensitive to nonlinear radiation changes and prone to nonlinear radiation distortion, and the multimodal image registration effect needs to be improved.

[0005] The present invention provides a multimodal remote sensing image registration method, comprising:

[0006] Acquire multimodal remote sensing images to be registered;

[0007] Processing the multimodal remote sensing image based on a relative total variation algorithm, and determining a directional feature map of the multimodal remote sensing image based on a multi-directional and multi-scale logarithmic Gabor filter operator;

[0008] Extracting features from the directional feature map based on a phase consistency algorithm, and determining key points of the multimodal remote sensing image in the extracted image structure features;

[0009] Obtaining a descriptor of the multimodal remote sensing image according to the directional feature map and the key points;

[0010] Feature matching is performed on the descriptors of the multimodal remote sensing images to determine a registration result of the multimodal remote sensing images.

[0011] As an embodiment, obtaining the descriptor of the multimodal remote sensing image according to the directional feature map and the key points includes:

[0012] Generating a maximum directional mask image according to the directional feature image;

[0013] A descriptor of the multimodal remote sensing image is obtained according to the maximum directional mask image and the key points.

[0014] As an embodiment, obtaining the descriptor of the multimodal remote sensing image according to the maximum directional mask image and the key points includes:

[0015] Determine a circular description unit of the key point according to the maximum directional mask image;

[0016] The descriptor of the multimodal remote sensing image is obtained according to the features of the maximum directional mask image corresponding to the pixels in the circular description unit.

[0017] As an embodiment, performing feature matching on the descriptors of the multimodal remote sensing images to determine the registration result of the multimodal remote sensing images includes:

[0018] Performing feature matching on the descriptors of the multimodal remote sensing image based on a nearest neighbor distance matching method to obtain matching point pairs of the multimodal remote sensing image;

[0019] The matching point pairs of the multimodal remote sensing image are screened based on a preset screening algorithm, and the registration result of the multimodal remote sensing image is determined according to the matching point pairs obtained by the screening.

[0020] As an embodiment, the screening algorithm includes a gene sampling consistency algorithm. Correspondingly, the matching point pairs of the multimodal remote sensing image are screened based on the preset screening algorithm, and the registration result of the multimodal remote sensing image is determined according to the screened matching point pairs, including:

[0021] Initialize the number of iterations of the gene sampling consensus algorithm;

[0022] If the iteration round of the current iteration is less than a preset value, randomly selecting a preset number of matching point pairs of the multimodal remote sensing image as fitting point pairs;

[0023] Generate a transformation model of the current iteration according to the fitting point pairs, the transformation model is used to screen the matching point pairs of the multimodal remote sensing image, take the matching point pairs within a preset error range as supporting point pairs, take the supporting point pairs that appear repeatedly for a preset number of times or a preset proportion in multiple iterations as potential point pairs, and execute the next iteration;

[0024] If the iteration round of the current iteration is greater than or equal to a preset value, generating a self-intersection transformation model, a mutation transformation model and / or a hybrid transformation model of the current iteration according to one or more of the matching point pairs of the multimodal remote sensing image, the fitting point pairs of the historical iteration, the supporting point pairs of the historical iteration and the potential point pairs of the historical iteration;

[0025] According to the self-cross transformation model, the variation transformation model and / or the hybrid transformation model, the matching point pairs of the multimodal remote sensing image are screened, the matching point pairs within a preset error range are used as supporting point pairs, the supporting point pairs that appear repeatedly for a preset number of times or a preset proportion during multiple iterations are used as potential point pairs, and the next iteration is executed until the iteration round of the current iteration is equal to the iteration number;

[0026] Determine the target transformation model according to the number of supporting point pairs in each iteration;

[0027] The matching point pairs of the multimodal remote sensing image are screened according to the target transformation model, and the registration result of the multimodal remote sensing image is determined according to the matching point pairs within a preset error range.

[0028] As an embodiment, determining the registration result of the multimodal remote sensing image according to the screened matching point pairs includes:

[0029] The homography transformation matrix of the multimodal remote sensing image is obtained according to the screened matching point pairs, and the homography transformation matrix is ​​used as the registration result of the multimodal remote sensing image.

[0030] As an embodiment, the extracting features of the directional feature map based on a phase consistency algorithm and determining the key points of the multimodal remote sensing image in the extracted image structure features includes:

[0031] Extracting features from the directional feature map based on a phase consistency algorithm to obtain structural features of the multimodal remote sensing image;

[0032] The structural features of the multimodal remote sensing image are detected based on a preset detector to determine the key points of the multimodal remote sensing image.

[0033] The present invention also provides a multimodal remote sensing image registration device, comprising:

[0034] A key point detection module is used to obtain a multimodal remote sensing image to be registered, process the multimodal remote sensing image based on a relative total variation algorithm, and determine a directional feature map of the multimodal remote sensing image based on a multi-directional and multi-scale logarithmic Gabor filter operator; extract features from the directional feature map based on a phase consistency algorithm, and determine key points of the multimodal remote sensing image in the extracted image structure features;

[0035] A feature description module, used for obtaining a descriptor of the multimodal remote sensing image according to the directional feature map and the key points;

[0036] The feature matching module is used to perform feature matching on the descriptors of the multimodal remote sensing image to determine the registration result of the multimodal remote sensing image.

[0037] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multimodal remote sensing image registration method as described above is implemented.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the multi-modal remote sensing image registration method as described in any one of the above is implemented.

[0039] The multimodal remote sensing image registration method, device, equipment and storage medium provided by the present invention obtain a multimodal remote sensing image to be registered; process the multimodal remote sensing image based on a relative total variation algorithm, and determine the directional feature map of the multimodal remote sensing image based on a multi-directional and multi-scale logarithmic Gabor filter operator; extract features from the directional feature map based on a phase consistency algorithm, and determine the key points of the multimodal remote sensing image in the extracted image structure features; obtain the descriptor of the multimodal remote sensing image based on the directional feature map and the key points; perform feature matching on the descriptor of the multimodal remote sensing image to determine the registration result of the multimodal remote sensing image. The present invention performs image main structure enhancement and image texture suppression on the multimodal remote sensing image through a relative total variation algorithm, and then performs feature extraction through a phase consistency algorithm, so as to obtain strong structural information of the multimodal remote sensing image and avoid modal differences between multi-source remote sensing images caused by nonlinear radiation distortion. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 This is one of the flow charts of the multimodal remote sensing image registration method provided by the present invention.

[0042] Figure 2 It is a schematic diagram of the flow of generating a descriptor provided by the present invention.

[0043] Figure 3a is one of the iterative schematic diagrams of the gene sampling consensus algorithm provided by the present invention, Figure 3b This is the second iterative schematic diagram of the gene sampling consensus algorithm provided by the present invention. Figure 3c This is the third iterative schematic diagram of the gene sampling consensus algorithm provided by the present invention.

[0044] Figure 4 It is a schematic diagram comparing the accuracy and time of different feature matching strategies provided by the present invention.

[0045] Figure 5a is a schematic diagram of the matching results of the multimodal remote sensing image registration provided by the present invention, Figure 5b It is the chessboard result of the multimodal remote sensing image registration provided by the present invention.

[0046] Figure 6 It is a structural schematic diagram of the multimodal remote sensing image registration device provided by the present invention.

[0047] Figure 7 This is the second flow chart of the multimodal remote sensing image registration method provided by the present invention.

[0048] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] It should be noted that all actions of acquiring signals, information or data in the present invention are performed in compliance with the corresponding data protection laws and policies of the location and with the authorization given by the owner of the corresponding device.

[0051] Existing feature-based image registration methods include Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB) and their variants. These methods effectively solve the scale and rotation invariance problems, but their descriptors mainly capture gradient features through intensity information. For multimodal images with similar shapes or layouts, registration methods that focus on structural characteristics work better than intensity information methods when dealing with nonlinear radiation distortion. Phase Consistency (PC) is an image structural feature extraction method based on local phase information. It does not rely on intensity information and can effectively cope with changes in illumination and contrast. On this basis, a variety of feature-based image registration methods have been developed, such as Phase Congruency Structural Descriptor (PCSD), Radiation-Variation Insensitive Feature Transform 2 (RIFT2), and Position-Orientation-Scale guided Geometric and Intensity-invariant Feature Transformation (POS-GIFT). However, these methods directly use phase consistency to extract image structural features in images, which will be disturbed by local texture details and easily produce nonlinear radiation distortion. At the same time, some methods also face the problem of low computational efficiency.

[0052] In this regard, the present invention provides a new feature-based multimodal image registration method, which solves nonlinear radiation distortion by capturing structural information, reducing texture interference and enhancing structural information. The multimodal remote sensing image registration method, device, equipment and storage medium provided by the present invention are described in detail below in conjunction with the accompanying drawings.

[0053] Figure 1 This is one of the flow charts of the multimodal remote sensing image registration method provided by the present invention, such as Figure 1 As shown, the present invention provides a multimodal remote sensing image registration method, including steps S100 to S500.

[0054] Step S100, obtaining a multimodal remote sensing image to be registered. Multimodal remote sensing images include optical remote sensing images and synthetic aperture radar (SAR) remote sensing images. Optical remote sensing images are generated by electromagnetic radiation in the visible and near-infrared bands, and SAR remote sensing images use radar signals to illuminate the target from different angles and generate high-resolution images through subsequent signal processing technology.

[0055] Step S200, processing the multimodal remote sensing image based on a relative total variation algorithm, and determining a directional feature map of the multimodal remote sensing image based on a multi-directional and multi-scale logarithmic Gabor filter operator.

[0056] The relative total variation (RTV) algorithm maintains the edge features of the image by minimizing the difference between the generated image and the original image and the regularization term about the relative total variation, while blurring the texture details. Due to the nonlinear radiation intensity between multimodal remote sensing images, some local detail structures will not appear repeatedly between different modes. The present invention introduces a relative total variation algorithm to strengthen the main structure of the image and suppress the image texture. Processing the multimodal remote sensing image based on the relative total variation algorithm includes image processing of the optical remote sensing image and the SAR remote sensing image based on the relative total variation algorithm.

[0057] After the multimodal remote sensing image is processed by the relative total variation algorithm, the main structure of the image is enhanced and the image texture is suppressed. The directional feature map of the processed multimodal remote sensing image is calculated based on the filter or the preset algorithm. The directional feature map is a multi-scale and multi-directional feature map.

[0058] The embodiment of the present invention uses a logarithmic Gabor filter to construct feature maps at four scales and six directions for optical remote sensing images and SAR remote sensing images processed by a relative total variation algorithm, and generates six directional feature maps at each scale. Both the optical remote sensing image and the SAR remote sensing image have corresponding four-scale six-directional feature maps.

[0059] Step S300: extracting features from the directional feature map based on a phase consistency algorithm, and determining key points of the multimodal remote sensing image from the extracted image structure features.

[0060] Phase consistency theory assumes that the noise power spectrum is approximately constant, and uses an effective noise compensation method and a sigmoid-like frequency expansion function for weighting to achieve more accurate positioning. The present invention uses a phase consistency algorithm to extract features from the directional feature map, extract key points with high repeatability and highly distinctive feature descriptors between remote sensing images of different modalities, and can well extract structural features in the image. Correspondingly, based on the phase consistency algorithm, feature extraction is performed on the directional feature maps of the optical remote sensing image and the SAR remote sensing image, and the image structural features corresponding to the optical remote sensing image and the SAR remote sensing image can be obtained. By screening the image structural features, the key points corresponding to the optical remote sensing image and the SAR remote sensing image can be obtained.

[0061] Step S400, obtaining a descriptor of the multimodal remote sensing image according to the directional feature map and the key points. Obtaining descriptors corresponding to the optical remote sensing image and the SAR remote sensing image, respectively, according to the directional feature map and the key points corresponding to the optical remote sensing image and the SAR remote sensing image, respectively.

[0062] The present invention constructs a feature descriptor similar to DAISY based on the directional information provided by the directional feature map and the position information provided by the key points. The DAISY descriptor is a local descriptor for image feature extraction, and is particularly suitable for texture analysis and matching tasks. The DAISY descriptor can effectively capture the gradient and directional information of the local area of ​​the image, and is widely used in the field of computer vision, such as image retrieval, object recognition, and image stitching. Different from the existing DAISY descriptor, the descriptor constructed by the present invention combines directional information and position information, which can enhance uniqueness and improve computational efficiency.

[0063] Step S500, feature matching is performed on the descriptors of the multimodal remote sensing image to determine the registration result of the multimodal remote sensing image. Feature matching is performed on the descriptors corresponding to the optical remote sensing image and the SAR remote sensing image, and alignment of the optical remote sensing image and the SAR remote sensing image is achieved based on the result of feature matching.

[0064] It can be understood that the present invention uses a relative total variation algorithm to enhance the main structure of the multimodal remote sensing image and suppress the image texture, and then uses a phase consistency algorithm to extract features, so as to obtain strong structural information of the multimodal remote sensing image and avoid modal differences between multi-source remote sensing images caused by nonlinear radiation distortion.

[0065] Based on the above embodiment, as an optional embodiment, the calculation formula for processing the multimodal remote sensing image based on the relative total variation algorithm is as follows:

[0066] (1)

[0067] (2)

[0068] (3)

[0069] (4)

[0070] (5)

[0071] (6)

[0072] In formula (1) to formula (6), p,q Represents different pixel positions and , represents the target image obtained after relative total variation processing, represents the parameter for adjusting the regularization term, Represents a small positive number that avoids zero denominator, represents the original image, R ( p ) indicates position p A rectangular area centered on and Pixels p exist x and y The total variation of the window in the direction, and Pixels p exist x and y The intrinsic variation in the window in the direction, and The pixel pairs generate images S exist x and y The difference in direction, is a space scale of Gaussian weighting function.

[0073] It can be understood that the present invention introduces the relative total variation algorithm to strengthen the main structure of the image and suppress the image texture, which is beneficial to reduce texture interference and enhance structural information, and solve nonlinear radiation distortion.

[0074] Based on the above embodiment, as an optional embodiment, the feature extraction of the directional feature map based on the phase consistency algorithm and the determination of the key points of the multimodal remote sensing image in the extracted image structure features include the following steps.

[0075] Step S310: extracting features from the directional feature map based on a phase consistency algorithm to obtain structural features of the multimodal remote sensing image.

[0076] The present invention uses improved phase consistency theory calculation as the basis for extracting image edge structure features. The calculation formula of the improved phase consistency theory is as follows.

[0077] (7)

[0078] (8)

[0079] (9)

[0080] (10)

[0081] (11)

[0082] In formula (7) to formula (11), is at the image position The phase consistency value at is the direction selection weight function, which indicates the contribution in different directions. is the magnitude response at scale s and direction o, is the phase difference at scale s and direction o, T is the noise threshold, used to reduce the impact of noise, is a small constant that prevents the denominator from being zero; and are the responses of even and odd symmetric filters, respectively, is the normalized average phase value of the even symmetric filter response, is the normalized mean phase value of the odd-symmetric filter response; is the normalization factor used to normalize the phase response.

[0083] On the basis of improving the phase consistency theory, the present invention further expands the application of the improved phase consistency theory to images, and can calculate the maximum moment diagram or the minimum moment diagram of the image. The calculation formula is as follows.

[0084] (12)

[0085] (13)

[0086] (14)

[0087] (15)

[0088] (16)

[0089] In formula (12) to formula (16), a, b, and c are letter symbols for simplified formula expressions. Indicates that the direction angle is The phase consistency result, in the embodiment of the present invention, .

[0090] The maximum moment map represents the structural features of the image, the minimum moment map represents the corner features of the image, and the maximum moment map contains the minimum moment map, that is, the corner features must appear in the structural feature map. In the embodiment of the present invention, the optical remote sensing image and the SAR remote sensing image are calculated using formulas (12), (13), (14) and (15) to obtain the structural features (including corner features) of the optical remote sensing image and the SAR remote sensing image respectively.

[0091] Step S320: Detect the structural features of the multimodal remote sensing image based on a preset detector to determine the key points of the multimodal remote sensing image.

[0092] The preset detector may be a Harris corner detector, a Shi-Tomasi corner detector, a FAST (Features from Accelerated Segment Test) detector, etc. The embodiment of the present invention is described by taking the FAST detector as an example. Based on the FAST detector, the structural features of the optical remote sensing image and the SAR remote sensing image are detected respectively, and the 1000-5000 corner points with the highest confidence are used as the key points corresponding to the optical remote sensing image and the SAR remote sensing image respectively.

[0093] It can be understood that the present invention extracts features from the directional feature map through a phase consistency algorithm, and can obtain image features with strong structural information to avoid modal differences between multimodal remote sensing images. The structural features of the multimodal remote sensing images are detected based on a preset detector, and corner points with higher confidence and a greater likelihood of repetition can be obtained.

[0094] Figure 2 is a schematic diagram of a flow chart of generating a descriptor provided by the present invention, such as Figure 2 As shown, based on the above embodiment, as an optional embodiment, obtaining the descriptor of the multimodal remote sensing image according to the directional feature map and the key points includes the following steps.

[0095] Step S410: Generate a maximum directional mask map according to the directional feature map. Generate corresponding directional feature maps of the optical remote sensing image and the SAR remote sensing image according to their respective directional feature maps.

[0096] For each directional feature map, if the pixel value of a certain directional feature map at the image position (x, y) is the maximum value of the feature map value in all directions, it is assigned a value of 1; otherwise it is assigned a value of 0. After the assignment is completed, the maximum orientation mask map (MOMM) can be obtained. Both optical remote sensing images and SAR remote sensing images correspond to four-scale six-directional feature maps, so six maximum orientation mask maps can be generated for each, which is highly robust to the nonlinear radiation differences between multi-modal images.

[0097] Step S420, obtaining a descriptor of the multimodal remote sensing image according to the maximum directional mask image and the key points. Generating corresponding descriptors of the optical remote sensing image and the SAR remote sensing image according to the maximum directional mask image and the key points of each.

[0098] Optionally, obtaining the descriptor of the multimodal remote sensing image according to the maximum directional mask image and the key points includes the following steps.

[0099] Step S421, determining the circular description unit of the key point according to the maximum directional mask image. The descriptor generated by the present invention is similar to the DAISY descriptor. Each DAISY descriptor of the DAISY descriptor has a basic circle. Therefore, the present invention calculates the circular description unit of the key point based on the circular Gaussian weighted center of the maximum directional mask image, and effectively uses spatial information to generate a 6-dimensional feature vector with distinguishing properties, such as Figure 2 As shown in the Daisy-like descriptor, the basic circles are independent circular areas. Gaussian weighting refers to weighting in each circular area with the center of the circle as the center. The features of all directions corresponding to the circular area can be calculated (one direction and one value). MMOM has six directions, and each circle can generate a 6-dimensional feature vector. The descriptor of the embodiment of the present invention includes 37 circular units, that is, a 222-dimensional feature vector.

[0100] Step S422, according to the features of the maximum directional mask image corresponding to the pixels in the circular description unit, obtain the descriptor of the multimodal remote sensing image. In order to ensure rotation invariance, the gradient information in the previously extracted structural feature map is used to calculate the main direction, and the descriptor is rotated accordingly. The main direction refers to the starting position of the feature vector to avoid too large a difference in the feature vectors generated by the image rotation in the same area.

[0101] It can be understood that the present invention constructs a DAISY-like descriptor based on the maximum directional mask image, which combines directional information and position information to enhance uniqueness and improve computational efficiency, and has stronger rotational invariance; adjacent key points can share circular description units to achieve high efficiency, and all circular description units only need to be calculated once, without repeated calculations, thereby improving computational efficiency.

[0102] Based on the above embodiment, as an optional embodiment, the step of performing feature matching on the descriptors of the multimodal remote sensing images to determine the registration result of the multimodal remote sensing images includes the following steps.

[0103] Step S510: Perform feature matching on the descriptors of the multimodal remote sensing image based on the nearest neighbor distance matching method to obtain matching point pairs of the multimodal remote sensing image. Perform feature matching on the descriptors of the optical remote sensing image and the SAR remote sensing image based on the nearest neighbor distance matching method to obtain the corresponding matching point pairs.

[0104] Step S520, based on a preset screening algorithm, the matching point pairs of the multimodal remote sensing image are screened, and the registration result of the multimodal remote sensing image is determined according to the screened matching point pairs. Based on a preset screening algorithm, the matching point pairs of the optical remote sensing image and the SAR remote sensing image are screened respectively, and the registration result of the optical remote sensing image and the SAR remote sensing image is determined according to the screened matching point pairs.

[0105] Even if the feature point pairs are constructed by the most similar method, there may be false matches. For this reason, the present invention screens the matching point pairs after feature matching to improve the accuracy of matching. The screening algorithms include cross-validation, screening based on geometric constraints, gene sampling consistency algorithm, etc. The embodiment of the present invention takes the gene sampling consistency algorithm as an example for explanation.

[0106] Figure 3a is one of the iterative schematic diagrams of the gene sampling consensus algorithm provided by the present invention, Figure 3b This is the second iterative schematic diagram of the gene sampling consensus algorithm provided by the present invention. Figure 3c This is the third iterative schematic diagram of the gene sampling consensus algorithm provided by the present invention.

[0107] like Figure 3a-3cAs shown, optionally, the screening algorithm includes a genetic sampling consensus algorithm (Genetic Sample Consensuss, GSC), which is similar to a genetic algorithm. By utilizing the fitting information in the iterative process, the self-intersection, mutation and hybridization transformation models are constructed to maintain accurate, stable and efficient calculation of the homography transformation matrix for image alignment. Correspondingly, the matching point pairs of the multimodal remote sensing image are screened based on the preset screening algorithm, and the registration result of the multimodal remote sensing image is determined according to the screened matching point pairs, including the following steps.

[0108] Step S521, initializing the number of iterations of the gene sampling consensus algorithm; the number of iterations is within one hundred, preferably 50, which greatly reduces the number of iterations.

[0109] Step S522: If the iteration number of the current iteration is less than a preset value, a preset number of matching point pairs of the multimodal remote sensing images are randomly selected as fitting point pairs.

[0110] Step S523, generating a transformation model of the current iteration based on the fitting point pairs, wherein the transformation model is used to screen the matching point pairs of the multimodal remote sensing image, taking the matching point pairs within a preset error range as support point pairs, taking the support point pairs that appear repeatedly during multiple iterations to a preset number of times or a preset proportion as potential point pairs, and executing the next iteration.

[0111] Step S524, if the iteration round of the current iteration is greater than or equal to the preset value, the self-intersection transformation model, the variation transformation model and / or the hybrid transformation model of the current iteration are generated according to one or more of the matching point pairs of the multimodal remote sensing image, the fitting point pairs of the historical iteration, the supporting point pairs of the historical iteration and the potential point pairs of the historical iteration. For similar transformations, in each iteration, a self-intersection transformation is generated by selecting supporting point pairs from the historical optimal transformation model. At the same time, a variation transformation is generated by selecting from the potential point pairs based on fitting empirical statistics. In addition, a crossover transformation is generated by exchanging fitting point pairs between the current and previous optimal transformation models. In the newly generated transformation, the transformation with the largest number of supporting point pairs is selected, and the information such as the previous, current and historical optimal transformation models, potential point pairs, etc. is updated accordingly. Among them, the optimal transformation model refers to the transformation model with the largest number of supporting point pairs in all iterations before the current iteration.

[0112] Step S525, screening the matching point pairs of the multimodal remote sensing image according to the self-intersection transformation model, the variation transformation model and / or the hybrid transformation model, taking the matching point pairs within a preset error range as support point pairs, taking the support point pairs that appear repeatedly for a preset number of times or with a preset proportion during multiple iterations as potential point pairs, and executing the next iteration until the iteration round of the current iteration is equal to the iteration number.

[0113] Step S526: determine the target transformation model according to the number of supporting point pairs in each iteration.

[0114] Step S527: screening the matching point pairs of the multimodal remote sensing image according to the target transformation model, and determining the registration result of the multimodal remote sensing image according to the matching point pairs within a preset error range.

[0115] Optionally, determining the registration result of the multimodal remote sensing image based on the matching point pairs obtained through screening includes: obtaining the homography transformation matrix of the multimodal remote sensing image based on the matching point pairs obtained through screening, and using the homography transformation matrix as the registration result of the multimodal remote sensing image.

[0116] It can be understood that the present invention provides a gene sampling consistency algorithm, which significantly reduces the number of iterations required while maintaining the matching accuracy, thereby speeding up the calculation speed.

[0117] The performance of the present invention is described below in conjunction with experimental data.

[0118] Table 1 Performance comparison data of the present invention and the prior art.

[0119]

[0120] As shown in Table 1, the present invention combines phase consistency (PC) with relative total variation (RTV) to effectively utilize structural features to eliminate modal differences. The improved DAISY-like descriptor developed based on MOMM has the advantages of both the maximum index map and the DAISY descriptor, enhances the distinguishability, and performs better in registration performance. In comparison, OS-SIFT, RTV-SIFT and SRIF rely on intensity information, so their precision is not as good as the method that prioritizes structural information. RIFT2 and POS-GIFT directly use the phase consistency method, which is easily affected by texture interference, resulting in lower SR and higher RMSE than RTV-RIFT.

[0121] In addition, the present invention is several times faster than PC-based methods (such as POS-GIFT). The circular unit using shared descriptors in the feature description stage and the GSC strategy in the feature matching stage effectively improve the computational efficiency. In contrast, POS-GIFT takes more time because it uses a pyramid structure for feature point detection and a position-direction-scale guided interior point recovery strategy, while RIFT2 is also slower because it uses a descriptor similar to SIFT.

[0122] Figure 4 is a schematic diagram comparing the accuracy and time of different feature matching strategies provided by the present invention, such as Figure 4As shown in the figure, to evaluate the effectiveness of the GSC strategy, the present invention compares the GSC strategy and the Fast Sample Consensus (FSC) strategy based on 20 pairs of multimodal remote sensing images in the randomly selected data set. Although the accuracy curves of the two strategies are similar, the GSC strategy reduces the average calculation time to 1.60% of the time required for the FSC strategy.

[0123] Figure 5a is a schematic diagram of the matching results of the multimodal remote sensing image registration provided by the present invention, Figure 5b is the chessboard result of the multimodal remote sensing image registration provided by the present invention, such as Figure 5a and Figure 5b As shown, the present invention can also achieve accurate image registration under rotation conditions.

[0124] In summary, based on the good characteristics of image edge structure in different modes, the present invention provides a feature-based multimodal image registration method, which increases the repeatability of key points and enhances the robustness of feature descriptors by capturing the structural information of the image. First, by combining phase consistency (PC) with relative total variation (RTV), texture interference is reduced and structural information is enhanced to solve nonlinear radiation distortion. In addition, a Daisy-like descriptor based on the maximum directional mask image is provided to improve the distinguishability and robustness of the descriptor. At the same time, a more efficient outlier removal algorithm, namely the genetic sampling consistency algorithm, is proposed to speed up the calculation while maintaining the matching accuracy unchanged.

[0125] The multimodal remote sensing image registration device provided by the present invention is described below. The multimodal remote sensing image registration device described below and the multimodal remote sensing image registration method described above can be referred to each other.

[0126] Figure 6 is a schematic diagram of the structure of the multimodal remote sensing image registration device provided by the present invention, such as Figure 6 As shown, the present invention also provides a multimodal remote sensing image registration device, including the following modules.

[0127] The key point detection module 610 is used to obtain a multimodal remote sensing image to be registered, process the multimodal remote sensing image based on a relative total variation algorithm, and determine a directional feature map of the multimodal remote sensing image based on a multi-directional and multi-scale logarithmic Gabor filter operator; extract features from the directional feature map based on a phase consistency algorithm, and determine the key points of the multimodal remote sensing image in the extracted image structure features;

[0128] A feature description module 620, configured to obtain a descriptor of the multimodal remote sensing image according to the directional feature map and the key points;

[0129] The feature matching module 630 is used to perform feature matching on the descriptors of the multimodal remote sensing image to determine the registration result of the multimodal remote sensing image.

[0130] As an embodiment, the feature description module 620 is further used for:

[0131] Generating a maximum directional mask image according to the directional feature image;

[0132] A descriptor of the multimodal remote sensing image is obtained according to the maximum directional mask image and the key points.

[0133] As an embodiment, the feature description module 620 is further used for:

[0134] Determine a circular description unit of the key point according to the maximum directional mask image;

[0135] The descriptor of the multimodal remote sensing image is obtained according to the features of the maximum directional mask image corresponding to the pixels in the circular description unit.

[0136] As an embodiment, the feature matching module 630 is further used for:

[0137] Performing feature matching on the descriptors of the multimodal remote sensing image based on a nearest neighbor distance matching method to obtain matching point pairs of the multimodal remote sensing image;

[0138] The matching point pairs of the multimodal remote sensing image are screened based on a preset screening algorithm, and the registration result of the multimodal remote sensing image is determined according to the matching point pairs obtained by the screening.

[0139] As an embodiment, the screening algorithm includes a gene sampling consistency algorithm, and correspondingly, the feature matching module 630 is further used to:

[0140] Initialize the number of iterations of the gene sampling consensus algorithm;

[0141] If the iteration round of the current iteration is less than a preset value, randomly selecting a preset number of matching point pairs of the multimodal remote sensing image as fitting point pairs;

[0142] Generate a transformation model of the current iteration according to the fitting point pairs, the transformation model is used to screen the matching point pairs of the multimodal remote sensing image, take the matching point pairs within a preset error range as supporting point pairs, take the supporting point pairs that appear repeatedly for a preset number of times or a preset proportion in multiple iterations as potential point pairs, and execute the next iteration;

[0143] If the iteration round of the current iteration is greater than or equal to a preset value, generating a self-intersection transformation model, a mutation transformation model and / or a hybrid transformation model of the current iteration according to one or more of the matching point pairs of the multimodal remote sensing image, the fitting point pairs of the historical iteration, the supporting point pairs of the historical iteration and the potential point pairs of the historical iteration;

[0144] According to the self-cross transformation model, the variation transformation model and / or the hybrid transformation model, the matching point pairs of the multimodal remote sensing image are screened, the matching point pairs within a preset error range are used as supporting point pairs, the supporting point pairs that appear repeatedly for a preset number of times or a preset proportion during multiple iterations are used as potential point pairs, and the next iteration is executed until the iteration round of the current iteration is equal to the iteration number;

[0145] Determine the target transformation model according to the number of supporting point pairs in each iteration;

[0146] The matching point pairs of the multimodal remote sensing image are screened according to the target transformation model, and the registration result of the multimodal remote sensing image is determined according to the matching point pairs within a preset error range.

[0147] As an embodiment, the feature matching module 630 is further used for:

[0148] The homography transformation matrix of the multimodal remote sensing image is obtained according to the screened matching point pairs, and the homography transformation matrix is ​​used as the registration result of the multimodal remote sensing image.

[0149] As an embodiment, the key point detection module 610 is further used for:

[0150] Extracting features from the directional feature map based on a phase consistency algorithm to obtain structural features of the multimodal remote sensing image;

[0151] The structural features of the multimodal remote sensing image are detected based on a preset detector to determine the key points of the multimodal remote sensing image.

[0152] Figure 7 This is the second flow chart of the multimodal remote sensing image registration method provided by the present invention. It is implemented based on the multimodal remote sensing image registration device. The present invention aims at the pixel-level registration problem between multimodal remote sensing images of satellites. The remote sensing images to be registered of different modalities are input into the key point detection module, and potential key points can be obtained from the enhanced structural features. In the feature description module, based on the maximum directional mask map, a reusable unit circle is efficiently generated, and the improved DAISY-like descriptor is used in combination with the spatial position distance. Finally, the feature matching module first obtains a rough matching point pair through nearest neighbor matching, and then determines a more accurate matching point pair through the genetic sampling consistency GSC strategy, and finally generates a homography result.

[0153] It should be noted that the multimodal remote sensing image registration device provided by the present invention can execute the multimodal remote sensing image registration method described in any of the above embodiments during specific operation, and has the technical effect corresponding to the method, which will not be elaborated in this embodiment.

[0154] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the multimodal remote sensing image registration method, which includes: obtaining a multimodal remote sensing image to be registered; processing the multimodal remote sensing image based on a relative total variation algorithm, and determining a directional feature map of the multimodal remote sensing image based on a multi-directional and multi-scale logarithmic Gabor filter operator; extracting features from the directional feature map based on a phase consistency algorithm, and determining the key points of the multimodal remote sensing image in the extracted image structure features; obtaining a descriptor of the multimodal remote sensing image based on the directional feature map and the key points; and performing feature matching on the descriptor of the multimodal remote sensing image to determine the registration result of the multimodal remote sensing image.

[0155] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0156] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal remote sensing image registration method provided by the above methods, which includes: acquiring a multimodal remote sensing image to be registered; processing the multimodal remote sensing image based on a relative total variation algorithm, and determining a directional feature map of the multimodal remote sensing image based on a multi-directional and multi-scale log-Gabor filter operator; extracting features from the directional feature map based on a phase consistency algorithm, and determining key points of the multimodal remote sensing image in the extracted image structure features; obtaining a descriptor of the multimodal remote sensing image based on the directional feature map and the key points; and performing feature matching on the descriptors of the multimodal remote sensing image to determine the registration result of the multimodal remote sensing image.

[0157] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multimodal remote sensing image registration method provided by the above-mentioned methods, the method comprising: acquiring a multimodal remote sensing image to be registered; processing the multimodal remote sensing image based on a relative total variation algorithm, and determining a directional feature map of the multimodal remote sensing image based on a multi-directional and multi-scale log-Gabor filter operator; extracting features from the directional feature map based on a phase consistency algorithm, and determining key points of the multimodal remote sensing image in the extracted image structure features; obtaining a descriptor of the multimodal remote sensing image based on the directional feature map and the key points; and performing feature matching on the descriptors of the multimodal remote sensing image to determine the registration result of the multimodal remote sensing image.

[0158] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0159] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal remote sensing image registration method, characterized in that: include: Acquire multimodal remote sensing images to be registered; Processing the multimodal remote sensing image based on a relative total variation algorithm, and determining a directional feature map of the multimodal remote sensing image based on a multi-directional and multi-scale logarithmic Gabor filter operator; Extracting features from the directional feature map based on a phase consistency algorithm, and determining key points of the multimodal remote sensing image in the extracted image structure features; Obtaining a descriptor of the multimodal remote sensing image according to the directional feature map and the key points; Performing feature matching on the descriptors of the multimodal remote sensing images to determine a registration result of the multimodal remote sensing images; The step of obtaining a descriptor of the multimodal remote sensing image according to the directional feature map and the key points includes: Generating a maximum directional mask image according to the directional feature image; Obtaining a descriptor of the multimodal remote sensing image according to the maximum directional mask image and the key points; Generating a maximum directional mask image according to the directional feature image comprises: For each directional feature map, if the pixel value of a certain directional feature map at the image position (x, y) is the maximum value of the feature map value in all directions, it is assigned a value of 1; otherwise, it is assigned a value of 0. After completing the assignment, the maximum directional mask map can be obtained; The step of obtaining a descriptor of the multimodal remote sensing image according to the maximum directional mask image and the key points includes: Determine the circular description unit of the key point according to the maximum directional mask image; calculate the circular description unit of the key point based on the circular Gaussian weighted center of the maximum directional mask image, and generate a distinguishing feature vector using spatial information; According to the features of the maximum directional mask image corresponding to the pixels in the circular description unit, the descriptor of the multimodal remote sensing image is obtained, the main direction is calculated using the gradient information in the structural feature map, and the descriptor is rotated accordingly, the main direction refers to the starting position of the feature vector.

2. The multimodal remote sensing image registration method according to claim 1, characterized in that: The performing feature matching on the descriptors of the multimodal remote sensing images to determine the registration result of the multimodal remote sensing images includes: Performing feature matching on the descriptors of the multimodal remote sensing image based on a nearest neighbor distance matching method to obtain matching point pairs of the multimodal remote sensing image; The matching point pairs of the multimodal remote sensing image are screened based on a preset screening algorithm, and the registration result of the multimodal remote sensing image is determined according to the matching point pairs obtained by the screening.

3. The multimodal remote sensing image registration method according to claim 2, characterized in that: The screening algorithm includes a gene sampling consistency algorithm. Correspondingly, the preset screening algorithm is used to screen the matching point pairs of the multimodal remote sensing image, and the registration result of the multimodal remote sensing image is determined according to the screened matching point pairs, including: Initializing the number of iterations of the gene sampling consensus algorithm; If the iteration round of the current iteration is less than a preset value, randomly selecting a preset number of matching point pairs of the multimodal remote sensing image as fitting point pairs; Generate a transformation model of the current iteration according to the fitting point pairs, the transformation model is used to screen the matching point pairs of the multimodal remote sensing image, take the matching point pairs within a preset error range as supporting point pairs, take the supporting point pairs that appear repeatedly for a preset number of times or a preset proportion in multiple iterations as potential point pairs, and execute the next iteration; If the iteration round of the current iteration is greater than or equal to a preset value, generating a self-intersection transformation model, a mutation transformation model and / or a hybrid transformation model of the current iteration according to one or more of the matching point pairs of the multimodal remote sensing image, the fitting point pairs of the historical iteration, the supporting point pairs of the historical iteration and the potential point pairs of the historical iteration; According to the self-cross transformation model, the variation transformation model and / or the hybrid transformation model, the matching point pairs of the multimodal remote sensing image are screened, the matching point pairs within a preset error range are used as supporting point pairs, the supporting point pairs that appear repeatedly for a preset number of times or a preset proportion during multiple iterations are used as potential point pairs, and the next iteration is executed until the iteration round of the current iteration is equal to the iteration number; Determine the target transformation model according to the number of supporting point pairs in each iteration; The matching point pairs of the multimodal remote sensing image are screened according to the target transformation model, and the registration result of the multimodal remote sensing image is determined according to the matching point pairs within a preset error range.

4. The multimodal remote sensing image registration method according to claim 3, characterized in that: Determining the registration result of the multimodal remote sensing image according to the screened matching point pairs includes: The homography transformation matrix of the multimodal remote sensing image is obtained according to the screened matching point pairs, and the homography transformation matrix is ​​used as the registration result of the multimodal remote sensing image.

5. The multimodal remote sensing image registration method according to any one of claims 1, 3 and 4, characterized in that: The step of extracting features from the directional feature map based on a phase consistency algorithm and determining key points of the multimodal remote sensing image in the extracted image structure features includes: Extracting features from the directional feature map based on a phase consistency algorithm to obtain structural features of the multimodal remote sensing image; The structural features of the multimodal remote sensing image are detected based on a preset detector to determine the key points of the multimodal remote sensing image.

6. A multimodal remote sensing image registration device, characterized in that: include: A key point detection module is used to obtain a multimodal remote sensing image to be registered, process the multimodal remote sensing image based on a relative total variation algorithm, and determine a directional feature map of the multimodal remote sensing image based on a multi-directional and multi-scale logarithmic Gabor filter operator; extract features from the directional feature map based on a phase consistency algorithm, and determine key points of the multimodal remote sensing image in the extracted image structure features; A feature description module, used for obtaining a descriptor of the multimodal remote sensing image according to the directional feature map and the key points; A feature matching module, used for performing feature matching on the descriptors of the multimodal remote sensing image to determine the registration result of the multimodal remote sensing image; The step of obtaining a descriptor of the multimodal remote sensing image according to the directional feature map and the key points includes: Generating a maximum directional mask image according to the directional feature image; Obtaining a descriptor of the multimodal remote sensing image according to the maximum directional mask image and the key points; Generating a maximum directional mask image according to the directional feature image comprises: For each directional feature map, if the pixel value of a certain directional feature map at the image position (x, y) is the maximum value of the feature map value in all directions, it is assigned a value of 1; otherwise, it is assigned a value of 0. After completing the assignment, the maximum directional mask map can be obtained; The step of obtaining a descriptor of the multimodal remote sensing image according to the maximum directional mask image and the key points includes: Determine the circular description unit of the key point according to the maximum directional mask image; calculate the circular description unit of the key point based on the circular Gaussian weighted center of the maximum directional mask image, and generate a distinguishing feature vector using spatial information; According to the features of the maximum directional mask image corresponding to the pixels in the circular description unit, the descriptor of the multimodal remote sensing image is obtained, the main direction is calculated using the gradient information in the structural feature map, and the descriptor is rotated accordingly, the main direction refers to the starting position of the feature vector.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the multimodal remote sensing image registration method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal remote sensing image registration method according to any one of claims 1 to 5 is implemented.

Images