Registration operator for SAR image and optical remote sensing image

Abstract

The invention discloses a registration operator ROG (Region-Oriented Gradient) for an SAR (Synthetic Aperture Radar) image and an optical remote sensing image. The registration operator comprises the following steps of: inputting the SAR and the optical image and preprocessing the SAR and the optical image; extracting a rotation invariant gradient feature through RGT (Radial Gradient Transform) transformation; constructing a nine-dimensional feature vector based on a right feature direction index table; according to the operator, the problem that a traditional gradient is sensitive to image rotation is solved through RGT, a nine-dimensional gradient direction strategy is adopted to weaken the nonlinear gray difference between multi-modal images, and the matching efficiency is optimized by counting gradient modulus length maximum value distribution. Compared with a traditional method, ROG does not need training data, calculation complexity is low, registration time consumption is remarkably reduced while high precision is kept, and the method is suitable for multi-mode remote sensing image registration fusion application in a resource limited scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing and computer vision, and in particular to a technique named ROG (Region-Oriented Gradient) operator, which has broad application prospects in many computer vision tasks such as image feature extraction, cross-modal image registration, etc. BACKGROUND

[0002] In the field of earth observation and spatial information technology today, the application of multi-modal remote sensing images is becoming more and more widespread and important. Among them, synthetic aperture radar (SAR) images and optical remote sensing images are two main sources of remote sensing data, each with unique advantages and characteristics.

[0003] The basic principle of SAR is to emit microwave electromagnetic waves to the ground, receive the echo signals reflected from the ground, and thus obtain detailed information about the earth's surface. SAR images, with their active microwave imaging mechanism, can achieve all-weather, all-day data acquisition regardless of weather conditions (such as clouds, rain, and fog), and have an irreplaceable role in monitoring areas that are affected by bad weather for a long time (such as tropical rainforest areas, polar regions). For example, in tropical rainforest areas, frequent rainfall and thick clouds make it difficult for optical remote sensing to obtain clear images stably, while SAR images can continuously provide surface information in this area, which is of great significance for monitoring deforestation, vegetation growth changes, etc. Optical remote sensing images, with their high resolution and rich spectral information, can intuitively reflect the texture, color, and other characteristics of ground objects, and play a key role in urban construction monitoring, crop classification, etc. For example, in the process of urban development, high-resolution optical remote sensing images can clearly observe the expansion of urban buildings, the extension of road networks, and other changes.

[0004] In practical applications, it is often necessary to fuse SAR images with optical remote sensing images to fully exploit their respective advantages and obtain more comprehensive and accurate information. In the fields of environmental monitoring, urban planning, and disaster monitoring, the integrated use of optical and SAR remote sensing images is of great significance. In the field of environmental monitoring, monitoring coastal wetland ecosystems, optical remote sensing images can clearly present the distribution and growth of wetland vegetation, and SAR images can obtain information such as topography and soil moisture under the influence of clouds and tides. Accurate registration and fusion of the two can help to fully understand the changes in wetland ecology under different seasons and climates, providing a scientific basis for wetland protection policy formulation. In urban planning and management, rapid urban development leads to changes in form and function. Optical remote sensing images can visually display changes in the spatial layout of urban buildings and roads, while SAR images can detect underground structures and building structural features. Registration and fusion of the two provide urban planners with detailed and accurate spatial information, helping to plan and layout rationally, optimize infrastructure, enhance disaster prevention and mitigation capabilities, and promote sustainable urban development. In disaster monitoring and emergency response, during the initial stage of a disaster, SAR images can quickly obtain the overall situation of the affected area without being affected by adverse weather. After the disaster occurs, when the weather permits, optical remote sensing images can be used for detailed loss assessment with high resolution. Intelligent registration of the two can achieve seamless connection throughout the disaster monitoring process, accurately warning, real-time monitoring, and precise loss assessment, providing strong support for disaster relief and reconstruction.

[0005] This makes the registration of multi-modal remote sensing images a key link. Multi-modal remote sensing image registration, especially SAR image and optical remote sensing image registration, has extremely important significance and background.

[0006] In the field of computer vision, accurately and efficiently extracting image features is the key to achieving various tasks such as object recognition, image classification, and semantic segmentation. Traditional image feature extraction methods, such as gradient operators (e.g., Sobel operator, Prewitt operator), while to some extent can capture edge and texture information of images, have limitations in dealing with complex scene images. These traditional operators are usually based on fixed templates and simple calculation methods, and cannot fully consider the importance differences of different regions in the image, leading to inaccurate feature extraction when facing images containing multiple complex targets and background disturbances, thereby affecting the performance of subsequent tasks.

[0007] With the booming development of deep learning, Convolutional Neural Networks (CNNs) have emerged as a powerful tool for image feature extraction, achieving remarkable results. CNNs, with their multi-layered convolutional and pooling layers, can automatically learn features at different levels in images, from low-level edges and texture features to high-level semantic features. After training on large image datasets, CNNs have demonstrated superior performance to traditional methods in many image tasks. However, there are some issues with CNNs that cannot be ignored. On the one hand, the structure of CNNs is often very complex, containing a large number of convolutional, pooling, and fully connected layers, which results in a huge number of parameters. For example, some classic CNN models, such as VGG16 and ResNet, have millions or even tens of millions of parameters. Such a large number of parameters means that a large amount of computational resources are required for training and inference, which not only puts high demands on the computing power of hardware devices, but also consumes a lot of time and energy. On the other hand, the training of CNNs requires a large amount of labeled data to ensure that the model can learn enough feature patterns. However, in practical applications, it is not easy to obtain large-scale high-quality labeled data, which not only requires a lot of manpower, material resources, and time, but also may face problems such as data privacy. For some application scenarios that require real-time performance or have limited computing resources, such as image recognition on mobile devices and embedded vision systems, directly using deep learning models may not be suitable. In mobile devices, due to the relatively limited hardware resources (such as CPU performance and memory capacity), running complex CNN models can cause the device to overheat, consume power too quickly, and even cause lag, which cannot meet the real-time requirements. In embedded vision systems, due to the limitations of hardware cost and size, it is also difficult to support the running of large-scale CNN models. Therefore, it is of great practical significance to develop a method that can effectively extract image features while having low computational complexity.

[0008] In view of the above problems, the present application designs a ROG operator, a registration operator for SAR images and optical remote sensing images. Through this operator, image features can be more effectively extracted, and the problem of nonlinear gray difference between SAR images and optical remote sensing images and the problem of SAR image rotation caused by the imaging mechanism of SAR images can be solved. The ROG operator uses gradient as the basic method, adopts RGT gradient transformation, and gradient direction weighting to complete the construction of feature descriptors, and completes the matching task of feature descriptors through the maximum value principal direction matching method. Finally, more accurate and lower cost SAR image and optical remote sensing image registration is realized. SUMMARY

[0009] In view of the above problems, the application provides a registration operator for SAR images and optical remote sensing images.

[0010] The embodiment of the application provides a registration method for SAR images and optical remote sensing images, comprising the following steps:

[0011] Step one: data set preparation

[0012] In the step one, the data set preparation is performed in the following manner: a QXSLAB_SAROPT data set is used. The QXS-SAROPT data set is an open data set used in the field of SAR (synthetic aperture radar) and optical data registration or fusion. The data set contains 20,000 pairs of SAR and optical image patches, the SAR images are from the spotlight mode images of Gaofen-3 satellite, and the optical images are from Google Earth. The images cover three port cities: Santiago, Shanghai and Qingdao.

[0013] Step two: data preprocessing

[0014] In the step two, the data preprocessing is performed in the following manner: the optical images and the corresponding SAR images are selected, the SAR images are cropped and rotated by a certain angle to obtain a 128*128 size region, and the region is used as the input image of the system.

[0015] Step three: extracting the ROG features of the input SAR image

[0016] The ROG features of the SAR image obtained in the step two are extracted, which are used as the basis for subsequent feature matching.

[0017] Step four: traversing the optical remote sensing image

[0018] The SAR image obtained in the step two is used as a template size, the template size is traversed in the optical image, and the ROG features of the optical small image at each position are extracted.

[0019] Step five: calculating the distance between the features of the optical small image at each position and the SAR template image

[0020] In the step five, the distance between the features of the optical small image at each position and the SAR template image is calculated in the following manner: when the ROG features of the two are extracted, the fan-shaped region corresponding to the maximum value of the nine-dimensional feature vector module in each annular region is counted and recorded. In the matching process, the angle at which the maximum value region of the SAR image and the optical remote sensing image is considered to be the most coincident is regarded as the matching angle, and the sum of the square differences between the two is calculated as the distance between the two.

[0021] Step six: calculate the final matching position.

[0022] Wherein, the "calculate the final matching position" in step six is as follows: traversing the feature distance of the optical small image and the SAR template image at each position calculated in step five, the position with the minimum feature distance is taken as the final matching position, and the angle at which the SAR image is most overlapped with the maximum value region of the optical remote sensing image is the angle of rotation of the SAR image relative to the optical remote sensing image.

[0023] Through the above steps, the algorithm can obtain a change detection model with high precision and high generalization ability. The model can distinguish the range and category of changes in the dual-time-phase remote sensing images and output the results.

[0024] Compared with the prior art, the application has the following advantages:

[0025] The application provides a registration operator ROG for SAR images and optical remote sensing images. The registration operator ROG solves the nonlinear gray difference between the SAR images and the optical remote sensing images by using image gradients as a basic method, and solves the problem that traditional gradients do not have rotation invariance by the form of RGT transformation. In order to make the detection result more accurate, the gradient directions are divided into nine intervals for gradient amplitude weighting, further solving the nonlinear gray difference between the SAR images and the optical remote sensing images. In addition, a feature matching method based on the maximum distribution of a nine-dimensional gradient feature vector is designed to solve the problem of slow feature matching. Compared with the previous method, the ROG has the advantages of no training and high precision, and has extremely low generalization cost and extremely strong generalization ability.

[0026] Other features and advantages of the present application will be further described in the following specification, and some will become apparent from the specification, or will be learned by practice of the present application. The purposes and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the written specification and the accompanying drawings.

[0027] The technical solutions of the present application will be further described in detail below by means of the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0028] The accompanying drawings are used to provide a further understanding of the present application, and constitute a part of the specification, and are used to explain the present application together with embodiments of the present application, and do not constitute a limitation on the present application. In the drawings:

[0029] Figure 1 The framework diagram designed for the algorithm.

[0030] Figure 2 The RGT transformation method.

[0031] Figure 3 The image division method. Figure 4 The gradient maximum distribution contrast. DETAILED DESCRIPTION

[0032] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood, and so that the scope of the present disclosure can be completely conveyed to those skilled in the art.

[0033] The embodiment of the present application provides a registration operator for SAR images and optical remote sensing images, and specifically comprises the following steps:

[0034] S1, extraction of ROG features;

[0035] S2, matching of ROG features.

[0036] In the above S1, the extraction method of the ROG features is as follows:

[0037] Step one, input image data, the input image should be a gray image, for those three-channel color images, they should be converted into a gray image before subsequent operations.

[0038] Step two, calculate the image gradient g x and g y .

[0039] Step three, perform RGT transformation on the obtained gradient to obtain an RGT gradient.

[0040] Step four, divide the gradient according to the gradient direction.

[0041] Step five, construct a nine-dimensional gradient feature vector.

[0042] Next, the above steps one to five will be described in detail.

[0043] Step one, input image data, the input image should be a gray image, for those three-channel color images, they should be converted into a gray image before subsequent operations.

[0044] Step two, calculate the image gradient g x and g y

[0045] The Sobel operator is based on convolution operation, which calculates the gradient of the image in horizontal and vertical directions by convolving the image. It uses two 3x3 convolution kernels to convolve the image respectively, one for calculating the gradient g x in horizontal direction, and the other for calculating the gradient g y in vertical direction.

[0046] Step three, in the above steps, the image gradient map has been obtained, however, the traditional gradient is not rotation invariant, the gradient direction changes with the rotation of the image, therefore, on the basis of obtaining the traditional gradient, the obtained gradient is transformed by RGT (Radial Gradient Transform), it can be proved that the RGT gradient is rotation invariant. The transformation mode is as shown in Figure 2 .

[0047] As shown in Figure 2 , c represents the center point of the circular region, and p is an arbitrary point on the circumference. r and t are two orthogonal unit vectors, wherein r represents the unit vector of the radial direction of the p point, and t represents the unit vector of the tangential direction of the p point, the calculation formula of r and t is as follows, wherein R θ represents the rotation matrix of rotating θ angle:

[0048]

[0049] t=R π / 2 r

[0050] g represents the gradient of the image at the p point. Then the gradient g of the p point is projected in the r and t directions, we can decompose the gradient g into (g T r)r+(g T t)t, so the coordinates of the gradient g in the coordinate system with r and t as the basis vectors are (g T r)r+(g T t)t, this process is called RGT (Radial Gradient Transform) transformation, and the transformed value is called RGT gradient value. Replace the traditional gradient g x with (g T r)r and (g T t)t. y .

[0051] When the circular region rotates by θ angle, there are R θ p=p ’ , R θ r=r ’ , R θ t=t ’ . The rotation invariance proof process of the RGT gradient is as shown in the following formula:

[0052] (g T r′,g T t′)=((R θ g) T R θ r,(R θ g) T R θ t)=(g T R θ T R θ r,g T R θ T R θ t)

[0053] =(g T r,g T t)

[0054] After obtaining the RGT gradient, the gradient direction OG and the amplitude IG are calculated, and the calculation formula is as follows:

[0055]

[0056] Step four, dividing the gradient according to the gradient direction. Since there is a phenomenon of radiation intensity inversion between images obtained by different types of sensors, in order to weaken the influence caused by the radiation intensity inversion between SAR images and optical images, and improve the robustness of the descriptor, the gradient direction OG is normalized. The calculated gradient direction OG has a value range of [0°, 360°), and the values belonging to the interval [180°, 360°) are uniformly reduced by 180°, and the value range of OG is normalized to [0°, 180°). On this basis, the value range of OG is equally divided into 8 parts according to an angle interval of 22.5°, and the divided intervals are separated by 9 angle directions. Since the feature descriptor will be obtained by statistics in the 9 angle directions, they are called 9 feature directions, and the 9 feature directions are assigned index numbers 0-8 in order of angle from small to large, and the right feature direction index table IRF is calculated. The process is shown in Figure 3

[0057]

[0058] ​Step five, based on the calculated right feature direction index table IRF, the image is divided into a series of concentric circles. To ensure that the subsequent calculation of the order of magnitude of the feature vector is comparable, the number of pixels in each layer of the annulus needs to be basically the same when dividing the annulus. Therefore, the inner annulus spacing is large, and the outer annulus spacing is small. After obtaining a series of concentric annuli, each concentric annulus is divided into a plurality of small regions to obtain the small regions as a unit to count the feature vectors. As shown in the figure, the sum of the weighted gradient assignments of all pixels in the 3*3 sector neighborhood around each small sector is counted, each direction corresponds to a dimension, forming a nine-dimensional feature vector, which is the nine-dimensional feature vector corresponding to the middle sector region. After obtaining the feature vector, z-axis filtering and normalization are performed, that is, the construction of the feature vector descriptor is completed. For a 256*256 image, 128 concentric circles are constructed, each annulus is divided into 144 sector regions, which can process images rotated by 2.5 degrees and multiples thereof. Each sector region corresponds to a nine-dimensional feature vector. In this way, a 256*256 image corresponds to a 128*144*9 feature vector. Thus, the construction of the ROG feature vector descriptor is completed.

[0059] In the above S2, the matching method of the ROG feature is as follows:

[0060] Step one: extract the ROG feature of the input SAR image:

[0061] Step two: traverse the optical remote sensing image large image:

[0062] Step three: calculate the distance between the optical small image feature and the SAR template image at each position.

[0063] Step four: calculate the final matching position.

[0064] Next, steps two to five are described in detail.

[0065] Step two, traverse the optical image with the SAR image size as the template size, extract the ROG feature of the optical small image at each position, and the extraction method is the same as that of the SAR image feature extraction. After extraction, the optical image feature can be saved to a TXT file to avoid repeated extraction in the future.

[0066] Step 3: Calculate the distance between the SAR image features and the optical image features at each position. Conventional methods rotate the extracted ROG operator during matching to eliminate possible rotations in the SAR image. The sum of squared differences between the ROG operator in the optical remote sensing image at each rotation and the current position is calculated. The minimum sum of squared differences is used as the rotation angle, and this sum of squared differences is used as the sum of squared differences at that position. After the traversal is completed, the minimum sum of squared differences across the entire image is used as the matching position. However, this method is computationally intensive and inefficient, so we propose an improved matching method.

[0067] The ROG operator is a gradient-based operator. For areas with large gradient values, it means that the image here has rich edge information. For the matching areas of SAR images and optical remote sensing images, the image contents of the two are the same and contain similar edge information. Therefore, it is reasonable to assume that the gradient maximum distribution of the ROG operators extracted from the two should be similar. Based on this assumption, the matching strategy is optimized. Specifically, when extracting the ROG features of the two, the sectors with the maximum modulus length of the nine-dimensional feature vector corresponding to the sector areas in each circular area are counted and recorded. During matching, the angle where the maximum value area of ​​the SAR image and the optical remote sensing image overlap the most is considered the matching angle. At this angle, the sum of the square differences between the two is calculated. After the traversal is completed, the minimum sum of square differences of the entire image is used as the matching position. For example Figure 4 As shown in the figure, the highlighted area is the area with the maximum modulus of the nine-dimensional gradient vector.

[0068] Based on the above theory, the specific distance calculation method is as follows:

[0069] 1. Taking the SAR image as an example, the nine-dimensional gradient vector modulus of all small areas within the ring is calculated.

[0070] 2. Find the small sector-shaped area within the ring where the modulus of the nine-dimensional gradient vector is the maximum, and record the location.

[0071] 3. Repeat the same process for each ring, and finally obtain the regional distribution map of the nine-dimensional gradient vector modulus of the SAR image.

[0072] 4. Perform the same operation on the optical image at that position to obtain the nine-dimensional gradient vector modulus area distribution map of the optical image.

[0073] 5. Align the two nine-dimensional gradient vector modulus length area distribution maps to find the location where the maximum value areas of the two images overlap the most. Record the SAR image rotation angle at that location. This is the matching main direction, that is, the SAR image rotation angle.

[0074] 6. Calculate the sum of the squared differences between the features at that angle, which is the distance between the SAR image feature and the optical image feature at that position.

[0075] Step four, after traversing each position on the optical image, the position with the minimum distance between the SAR image feature and the optical sub-image feature is the registration result.

[0076] The process of using ROG features to register SAR images and optical remote sensing images is completed

[0077] The SAR image size tested in the application is 256*256, and the matching is performed in a 1024*1024 size optical image. For a 256*256 image, the image is divided into 128 annular rings, and each annular ring is further divided into 144 small regions, which correspond to the rotation of 2.5 degrees and multiples thereof. Each region corresponds to a 9-dimensional feature vector, and a 256*256 image corresponds to a 128*144*9 feature vector.

Claims

1. A registration operator ROG for SAR images and optical remote sensing images, characterized by: The following steps are involved: S1. Preprocess the input SAR image and optical remote sensing image, including size cropping and grayscale conversion; S2. Calculate the image gradient based on the Sobel operator and obtain the rotation-invariant radial gradient and tangential gradient through RGT (Radial Gradient Transform); S3. Normalize the gradient direction to the interval [0°, 180°) and divide it into 9 feature direction intervals. S4, dividing the image into concentric rings and fan-shaped areas, counting the nine-dimensional gradient feature vectors of each fan-shaped area, and constructing the ROG feature descriptor; S5. Traverse the matching positions in the optical image, determine the rotation angle through the maximum main direction distribution overlap, calculate the feature distance, and use the minimum distance position as the final registration result.

2. The registration operator according to claim 1, wherein: The specific method of RGT transformation in S2 is: decomposing the image gradient into radial components (g T r)r and tangential component (g T t)t, where the radial unit vector r and the tangential unit vector t satisfy the orthogonal relationship.

3. The registration operator according to claim 1, wherein: The gradient direction is divided into 9 intervals at equal intervals, the distribution of the gradient vector in the nine intervals is calculated, and the right feature direction index table IRF is constructed.

4. The registration operator according to claim 1, wherein: The method for constructing the feature descriptor in S4 is as follows: the image is divided into 128 layers of concentric rings, each layer of rings is divided into 144 sector-shaped areas, and the sum of the weighted gradient amplitudes in 9 directions in each sector-shaped area and its neighborhood is counted to form a nine-dimensional feature vector, which is then normalized.

5. The registration operator according to claim 1, wherein: The matching method in S5 includes: statistically analyzing the maximum distribution of the nine-dimensional gradient modulus within each circular ring of the SAR image and the optical image, determining the rotation angle based on the sector-shaped area with the most overlap, calculating the sum of the squared differences of the features at this angle, and selecting the position corresponding to the minimum distance as the registration result.

6. The registration operator according to claim 1, wherein: In the preprocessing, the cropping size of the SAR image is 128×128, the traversal template size of the optical image is 256×256, and the registration result supports rotation angle correction.

Images