U-net based optical and sar remote sensing image optical flow registration method

Abstract

The application discloses a U-Net-based optical and SAR remote sensing image optical flow registration method and relates to the technical field of heterogeneous image registration, and comprises the construction of an image data set, image preprocessing and the labeling of a region of interest; two U-Net network models are used to train an optical remote sensing image segmentation model and a SAR remote sensing image segmentation model respectively; U-Net network segmentation results of a to-be-registered image pair are acquired, and pixel point sets marked in specified channels of two segmentation images are recorded respectively; gradient operators are used to construct class GLOH descriptors of region-of-interest feature points and pixel points in specified neighborhoods of the region-of-interest feature points in the to-be-registered image pair; and the region-of-interest feature points in the to-be-registered image pair are registered by using a Gaussian pyramid LK optical flow method. The application can realize the registration of heterogeneous images with higher precision, stronger purpose and better timeliness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of heterologous image registration, and particularly relates to a visible light and SAR remote sensing image optical flow registration method based on U-Net. BACKGROUND

[0002] Due to different imaging mechanisms of visible light remote sensing images and SAR remote sensing images, the obtained image information has complementary characteristics, and the image information can be maximized by registration. However, there is a large geometric difference and nonlinear radiation difference between the visible light remote sensing image and the SAR remote sensing image, and the SAR remote sensing image has significant speckle noise, which makes it difficult to achieve ideal registration accuracy when directly applying existing visible light and SAR remote sensing image registration algorithms (such as Scale-invariant feature transform (SIFT), Speeded-Up Robust Features (SURF), etc.) to the registration problem of the two. Although the registration technology of visible light remote sensing images and SAR remote sensing images has developed rapidly in the past few decades, there are still problems in timeliness, registration accuracy, etc. SUMMARY

[0003] The purpose of the present application is to provide a visible light and SAR remote sensing image optical flow registration method based on U-Net, to solve the problems of existing registration methods for visible light remote sensing images and SAR remote sensing images in registration accuracy, purpose and timeliness.

[0004] The technical scheme adopted by the present application to solve the technical problems is as follows:

[0005] The visible light and SAR remote sensing image optical flow registration method based on U-Net of the present application comprises the following steps:

[0006] Step S1, construction of image data set, image preprocessing and annotation of region of interest;

[0007] Step S2, training a visible light remote sensing image segmentation model and a SAR remote sensing image segmentation model using two U-Net network models;

[0008] Step S3, obtaining the U-Net network segmentation result of the image pair to be registered, and recording the pixel point set labeled by the specified channel in the two segmentation images, respectively;

[0009] Step S4, constructing the class GLOH descriptor of the feature points in the region of interest and the pixel points in the specified neighborhood of the image pair to be registered using a gradient operator;

[0010] Step S5, registering the feature points of the region of interest in the image pair to be registered by the Gaussian pyramid LK optical flow method.

[0011] Further, the specific operation steps of step S1 are as follows:

[0012] S1.1, respectively constructing a visible light remote sensing image dataset and a SAR remote sensing image dataset;

[0013] A plurality of visible light remote sensing images and SAR remote sensing images are collected by a satellite to form a visible light remote sensing image dataset and a SAR remote sensing image dataset, respectively;

[0014] S1.2, converting the plurality of acquired visible light remote sensing images and SAR remote sensing images into gray scale images respectively by using formula (1), and performing pixel value consistency processing on the plurality of obtained gray scale images;

[0015]

[0016] In the formula, I'(x, y) represents the gray scale image after pixel value consistency processing, max(I(x, y)) represents the maximum pixel value in the original gray scale image without pixel value consistency processing, min(I(x, y)) represents the minimum pixel value in the original gray scale image without pixel value consistency processing, and x and y respectively represent the row and column where the pixel point is located;

[0017] S1.3, labeling the region of interest of the plurality of preprocessed gray scale images by using labelme software, and respectively constructing a visible light remote sensing image training dataset and a SAR remote sensing image training dataset.

[0018] Further, the specific operation steps of step 1.3 are as follows:

[0019] According to the registration purpose, the pixel points of the region of interest of the pre-processed multiple gray scale images are manually labeled using the labelme software, and the number of classes of the region of interest is the subsequent classification number of the pixel points; the multiple gray scale images corresponding to the visible light remote sensing image are randomly divided into opt_training data set, opt_test data set and opt_validation data set according to the proportion, the proportion of opt_training data set, opt_test data set and opt_validation data set is 70%, 20% and 10% respectively, the multiple gray scale images corresponding to the SAR remote sensing image are randomly divided into sar_training data set, sar_test data set and sar_validation data set according to the proportion, the proportion of sar_training data set, sar_test data set and sar_validation data set is 70%, 20% and 10% respectively; and the multiple gray scale images corresponding to the visible light remote sensing image after labeling the region of interest are recorded as opt_label data set, and the multiple gray scale images corresponding to the SAR remote sensing image after labeling the region of interest are recorded as sar_label data set.

[0020] Further, the specific operation steps of step S2 are as follows:

[0021] S2.1 build two U-Net network models;

[0022] S2.2 respectively train the visible light remote sensing image segmentation model and the SAR remote sensing image segmentation model;

[0023] The visible light remote sensing image training data set and the SAR remote sensing image training data set constructed in step S1 are respectively input into the two U-Net network models built above, wherein the opt_training data set and the sar_training data set are used for training the network, the opt_test data set and the sar_test data set are used for evaluating the network performance, the opt_validation data set and the sar_validation data set are used for setting the network parameters, and finally the trained visible light remote sensing image segmentation model and SAR remote sensing image segmentation model are obtained.

[0024] Further, the specific operation steps of step S3 are as follows:

[0025] S3.1 Acquire a pair of visible light remote sensing images and SAR remote sensing images outside of the visible light remote sensing image dataset and SAR remote sensing image dataset constructed in step S1. Preprocess this pair of images to be registered according to the method in step S1 to obtain the corresponding floating image opt and reference image sar. Input the floating image opt and reference image sar into the visible light remote sensing image segmentation model and the SAR remote sensing image segmentation model respectively to obtain the U-Net network segmentation result of the pair of images to be registered, i.e., two segmented images.

[0026] S3.2 In the U-Net network segmentation results above, each channel corresponds to a different category of region of interest. The pixel marked by the specified channel is the position of the region of interest in the corresponding segmented image. The set S of the pixels marked by the specified channel in the two segmented images is recorded respectively. opt and S sar .

[0027] Furthermore, the specific steps of step S4 are as follows:

[0028] S4.1 The gradient value of the floating image opt is calculated using the Canny operator, and the gradient value of the reference image sar is calculated using the ROEWA operator.

[0029] S4.2 Extract the DOG scale spatial feature point set F to obtain the region of interest feature point set F of the floating image opt. opt ={f1, f2, ..., f N}, where N is the number of feature points in the region of interest of the floating image opt, f1, f2, ..., f N For the feature points in the set, refer to the feature point set F of the region of interest in the image sar. sar ={f1, f2, ..., f M}, M is the number of feature points in the region of interest of the reference image sar, f1, f2, ..., f M These are the feature points in the set;

[0030] S4.3 Iterate N times, sequentially determining the set of feature points F of the region of interest in the floating image opt. opt Do the elements in the set belong to set S? opt If so, then in the set of feature points F of the region of interest opt If the feature point is retained, it is retained; otherwise, it is removed. The set of feature points in the region of interest of the updated floating image opt is denoted as FO. update ; Repeat the process M times, sequentially determining the feature point set F of the region of interest in the reference image sar. sar Do the elements in the set belong to set S? sar If so, then in the set of feature points F of the region of interestsar The feature point is retained, otherwise, the feature point is removed, and the updated feature point set of the region of interest of the reference image sar is denoted as FS update ;

[0031] S4.4, respectively, the feature points of the region of interest of the floating image opt and the feature points of the region of interest of the reference image sar and the pixel points in the specified neighborhood thereof are constructed.

[0032] Further, the specific operation steps of step S4.4 are as follows:

[0033] The neighborhood radius is set as r, and a logarithmic-polar coordinate three-ring is constructed with each processing pixel point as the center. The three rings have three values in the radial direction, from inside to outside, r1, r2, and r3, and satisfy r1=0.25×r3, r2=0.73×r3, and r3=4a. The value of a determines the size of the ring. The angle of the three rings is equally divided into 45°, so that the three rings have 17 subintervals. Mathematical statistics of image gradient amplitude accumulation is performed in each subinterval.

[0034] Further, the specific operation steps of step S5 are as follows:

[0035] S5.1, the original images, i.e., the floating image opt and the reference image sar in step S3 are down-sampled by Gaussian pyramid, and the value of the number of layers L of the Gaussian pyramid is related to the size of the original image, so that the small motion assumption of the LK optical flow method of the Gaussian pyramid is met;

[0036] S5.2, the set of the feature points of the region of interest and the pixel points in the specified neighborhood thereof is denoted as O, and the class GLOH descriptor of the pixel points in the set O is used to replace the pixel value at the corresponding position of each layer of the Gaussian pyramid, so that the brightness constant assumption of the LK optical flow method of the Gaussian pyramid is met;

[0037] S5.3, according to the brightness constant assumption, the following equation is obtained at the bottom layer of the Gaussian pyramid:

[0038] D1(x1, y1, t) = D2(x1+u, y1+v, t+1) (2)

[0039] In the formula, D1 and D2 respectively represent the class GLOH descriptors corresponding to the feature points of the regions of interest of the two original images at the bottom layer of the Gaussian pyramid; x1 and y1 respectively represent the row and column where the feature points of the region of interest are located; t represents the corresponding time of the floating image opt; u and v respectively represent the horizontal offset of the scale u and the vertical offset of the scale v of the feature points of the region of interest in the floating image opt; the right side of formula (2) is expanded by Taylor series to obtain the following equation:

[0040] D (x1+u, y1+v, t+1) = D (x1, y1, t) + D x u+D y v+D t +R2 (3)

[0041] In the formula, D x , D y respectively represent the gradient of the GLOH-like descriptor value corresponding to the feature point of the region of interest in the horizontal direction and the vertical direction, D t represents the partial derivative of the GLOH-like descriptor value corresponding to the feature point of the region of interest along the time t direction; and R2 represents a second-order infinitesimal term.

[0042] S5.4 According to the small motion assumption of the Gaussian pyramid LK optical flow method, the second-order infinitesimal term R2 is discarded to obtain the optical flow equation of the feature point of the region of interest in the bottom layer of the Gaussian pyramid:

[0043] D x u+D y v+D t =0 (4)

[0044] For each feature point of the region of interest in a specific layer of the Gaussian pyramid, the above optical flow equation can be established to obtain the feature point in the corresponding image that is closest to the GLOH-like descriptor value corresponding to the feature point of the region of interest. The neighborhood size is set to (2m+1) x (2m+1), and m represents the neighborhood radius. The neighborhood radius m is related to the layer number L of the Gaussian pyramid, that is, m=batch x 2 L-1 , and batch represents the neighborhood radius of the bottom layer of the Gaussian pyramid. The optical flow value of the pixel point in the specified neighborhood corresponding to the feature point of the region of interest is constant, and thus the optical flow equation of the pixel point in the specified neighborhood corresponding to the feature point of the region of interest is established.

[0045] S5.5 The optical flow equation of the pixel point in the specified neighborhood corresponding to the feature point of the region of interest is solved by using the least square method in the bottom layer of the Gaussian pyramid, the offset u and v of the feature point closest to the GLOH-like descriptor value of the region of interest are found, and then the information transmission between different layers of the Gaussian pyramid and the optical flow value updating are performed. In this way, the process is repeated until the highest layer of the Gaussian pyramid, and the calculation result of this layer is taken as the final optical flow value, so as to obtain the final registration result of the feature point of the region of interest of the floating image opt and the reference image sar.

[0046] The beneficial effects of the present application are as follows:

[0047] The U-Net-based optical and SAR remote sensing image optical flow registration method of the application adopts the constructed U-Net network model to demarcate the region of interest of the optical remote sensing image and the SAR remote sensing image to be registered, uses different gradient operators to calculate the descriptors of the selected region feature points and the designated neighborhood pixel points in the two images, and then uses the Gaussian pyramid LK optical flow method to register the feature points, so that the registration of the heterogeneous images with higher accuracy, stronger purpose and better timeliness is realized.

[0048] The application combines the feature point extraction, the designated region descriptor construction and the optical flow algorithm, improves the remote sensing image registration accuracy, guarantees the timeliness of the remote sensing image registration, and provides a new idea for the registration of the heterogeneous images. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 The flowchart of the U-Net-based optical and SAR remote sensing image optical flow registration method of the application.

[0050] Figure 2 The schematic diagram of the GLOH-like descriptor construction process of the region of interest feature points and the pixel points in the designated neighborhood.

[0051] Figure 3 The schematic diagram of the GLOH-like descriptor construction principle. DETAILED DESCRIPTION

[0052] The embodiments of the application are described in detail below, which are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the application, and cannot be explained as the limitation of the application.

[0053] The U-Net-based optical and SAR remote sensing image optical flow registration method of the application, as shown in Figure 1 specifically includes the following steps:

[0054] Step S1, construction of image data set, image preprocessing and labeling of region of interest;

[0055] S1.1 respectively constructing the optical remote sensing image data set and the SAR remote sensing image data set;

[0056] A plurality of optical remote sensing images and SAR remote sensing images are collected by a satellite to form the optical remote sensing image data set and the SAR remote sensing image data set respectively;

[0057] S1.2 respectively pre-processing the plurality of acquired optical remote sensing images and the plurality of SAR remote sensing images;

[0058] The obtained multiple visible light remote sensing images and SAR remote sensing images are converted into gray scale images by formula (1), and the obtained multiple gray scale images are subjected to pixel value consistency processing to reduce the influence of the amplitude of the heterogeneous images on the subsequent segmentation model.

[0059]

[0060] In the formula, I'(x, y) represents the gray scale image after pixel value consistency processing, max(I(x, y)) represents the maximum pixel value in the original gray scale image without pixel value consistency processing, min(I(x, y)) represents the minimum pixel value in the original gray scale image without pixel value consistency processing, and x and y represent the row and column of the pixel point, respectively.

[0061] S1.3 label the interested regions of the pre-processed multiple gray scale images using the labelme software, and construct the visible light remote sensing image training dataset and the SAR remote sensing image training dataset, respectively.

[0062] According to the registration purpose, the pixel points of the interested regions are manually labeled (such as mountains, rivers, etc.) using the labelme software, and the number of categories of the interested regions is the number of subsequent classifications of the pixel points; the multiple gray scale images corresponding to the visible light remote sensing images are randomly divided into an opt_training dataset (70%), an opt_test dataset (20%), and an opt_validation dataset (10%) according to a proportion, the multiple gray scale images corresponding to the SAR remote sensing images are randomly divided into a sar_training dataset (70%), a sar_test dataset (20%), and a sar_validation dataset (10%) according to a proportion; and the multiple gray scale images corresponding to the visible light remote sensing images after labeling the interested regions are recorded as an opt_label dataset, and the multiple gray scale images corresponding to the SAR remote sensing images after labeling the interested regions are recorded as a sar_label dataset.

[0063] Step S2, training the visible light remote sensing image segmentation model and the SAR remote sensing image segmentation model using two U-Net network models;

[0064] S2.1 building a U-Net network model;

[0065] S2.2 training the visible light remote sensing image segmentation model and the SAR remote sensing image segmentation model, respectively;

[0066] The visible light remote sensing image training data set and the SAR remote sensing image training data set constructed in step S1 are input into the two U-Net network models built above, wherein the opt_training data set and the sar_training data set are used for training the network, the opt_test data set and the sar_test data set are used for evaluating the network performance, the opt_validation data set and the sar_validation data set are used for setting the network parameters (including cuda, classification number, convolution layer number, etc.), and finally two trained visible light remote sensing image segmentation models and SAR remote sensing image segmentation models are obtained;

[0067] In step S3, the U-Net network segmentation results of the image pair to be registered are obtained, and the pixel point sets marked by the specified channel in the two segmentation images are recorded respectively;

[0068] In step S3.1, a pair of visible light remote sensing images and SAR remote sensing images other than the visible light remote sensing image data set and the SAR remote sensing image data set constructed in step S1 are collected, and then the pair of images to be registered are preprocessed according to the method in step S1, and further the corresponding floating image opt and reference image sar are obtained. The floating image opt and the reference image sar are input into the visible light remote sensing image segmentation model and the SAR remote sensing image segmentation model trained in step S2, and then the U-Net network segmentation results of the image pair to be registered, i.e. two segmentation images, are obtained.

[0069] In step S3.2, each channel in the U-Net network segmentation result corresponds to a different class of interest, and the pixel points marked by the specified channel are the positions of the regions of interest in the corresponding segmentation images. Therefore, the pixel point sets marked by the specified channel in the two segmentation images are recorded respectively, which are S opt and S sar respectively.

[0070] In step S4, as shown in Figure 2 , two different gradient operators are selected according to the different characteristics of the two kinds of remote sensing images to calculate the gradient values of the pixel points, the DOG scale space feature point set is constructed and updated, and the GLOH descriptor of the region of interest feature point and the pixel points in its specified neighborhood is constructed on this basis.

[0071] In step S4.1, the gradient values corresponding to the floating image opt are calculated by using the Canny operator, and the gradient values corresponding to the reference image sar are calculated by using the ROEWA operator.

[0072] In step S4.2, the DOG scale space feature point set F is extracted, and the region of interest feature point set F opt of the floating image opt is obtained. N}(N is the number of feature points in the region of interest of the floating image opt, f1, f2,... f N is the set of feature points in the region of interest of the reference image sar) and the set of feature points F sar = {f1, f2,... f M}(M is the number of feature points in the region of interest of the reference image sar, f1, f2,... f M is the set of feature points in the region of interest of the reference image sar) ;

[0073] S4.3 Loop N times, judge whether the elements in the set of feature points F opt of the region of interest of the floating image opt belong to the set S opt of marked pixel points in step S3, if yes, keep the feature points in the set of feature points F opt of the region of interest of the floating image opt, otherwise, eliminate the feature points in the set of feature points F opt of the region of interest of the floating image opt, and the updated set of feature points of the region of interest of the floating image opt is denoted as FO upaate ; Loop M times, judge whether the elements in the set of feature points F sar of the region of interest of the reference image sar belong to the set S sar of marked pixel points in step S3, if yes, keep the feature points in the set of feature points F sar of the region of interest of the reference image sar, otherwise, eliminate the feature points in the set of feature points F sar of the region of interest of the reference image sar, and the updated set of feature points of the region of interest of the reference image sar is denoted as FS update ;

[0074] S4.4 Construct the class GLOH descriptor of the feature points of the region of interest of the floating image opt and the feature points of the region of interest of the reference image sar and the pixel points in their designated neighborhood respectively;

[0075] According to the above feature point set update results, construct the class GLOH descriptor of the feature points of the region of interest of the floating image opt and the feature points of the region of interest of the reference image sar and the pixel points in their designated neighborhood respectively; the specific operation steps are as follows:

[0076] For example, Figure 3As shown, the neighborhood radius is set as r, a log-polar coordinate three-ring is drawn with each processing pixel point as the center, the three rings have three values in the radial direction, from inside to outside, r1, r2, r3, and satisfy r1=0.25×r3, r2=0.73×r3, r3=4a, the value of a determines the size of the ring, if the value of a is too large, it will cause confusion of information, if the value of a is too small, it will cause the loss of information; the angle of the three rings is equally divided into 45°, so that the three rings have 17 subintervals, and then the mathematical statistics of image gradient is performed in each subinterval; since the edge pixel intensity of the two obtained remote sensing images is difficult to avoid peak value, thereby causing gradient reversal phenomenon, therefore, when performing the mathematical statistics of image gradient, the gradient direction is discarded, and only the accumulation of gradient response amplitude is performed;

[0077] Step S5, registering the feature points of the region of interest in the image pair to be registered by the Gaussian pyramid LK optical flow method;

[0078] S5.1, satisfying the small motion assumption of the Gaussian pyramid LK optical flow method by Gaussian pyramid down-sampling;

[0079] The original images (floating image opt and reference image sar) in step S3 are subjected to Gaussian pyramid down-sampling, and the value of the Gaussian pyramid layer L is closely related to the size of the original image (floating image opt and reference image sar), which can be set by itself according to the need, and this operation can satisfy the small motion assumption of the Gaussian pyramid LK optical flow method;

[0080] S5.2, constructing the brightness constant assumption of the Gaussian pyramid LK optical flow method by the region of interest feature point extraction in step S4 and the GLOH-like descriptor of the pixel points in the specified neighborhood;

[0081] Since the construction of the GLOH-like descriptor of the region of interest feature points and the pixel points in the specified neighborhood in step S4 reduces the pixel difference of the original image, the GLOH-like descriptor of the pixel points in the set O is used instead of the pixel value of the corresponding position of each layer of the Gaussian pyramid, and this operation can satisfy the brightness constant assumption of the Gaussian pyramid LK optical flow method;

[0082] S5.3, according to the brightness constant assumption, the following equation can be obtained at the bottom layer (the smallest scale layer) of the Gaussian pyramid:

[0083] D1(x1, y1, t) = D2(x1+u, y1+v, t+1) (2)

[0084] In the formula, D1 and D2 respectively represent the GLOH-like descriptor corresponding to the feature point of the region of interest in the two original images (floating image opt and reference image sar) at the bottom layer of the Gaussian pyramid, since the GLOH-like descriptor is a 17-dimensional vector, the L2 norm is used to compare the GLOH-like descriptor values; x1 and y1 respectively represent the row and column where the feature point of the region of interest is located; t represents the corresponding time of the floating image opt; u and v respectively represent the horizontal offset of the scale u and the vertical offset of the scale v of the feature point of the region of interest in the floating image opt, so that the GLOH-like descriptor value corresponding to the feature point of the region of interest in the reference image sar can be kept consistent; the right side of formula (2) is expanded by Taylor series to obtain:

[0085] D (x1+u, y1+v, t+1) = D (x1, y1, t) + D x u+D y v+D t +R2 (3)

[0086] In the formula, D x , D y respectively represent the gradient of the GLOH-like descriptor value corresponding to the feature point of the region of interest in the horizontal direction and the vertical direction, D t represents the partial derivative of the GLOH-like descriptor value corresponding to the feature point of the region of interest along the time t direction, and the above parameters can be obtained according to the floating image opt and the reference image sar themselves; R2 represents the second-order infinitesimal;

[0087] S5.4 Limiting the local area optical flow value to be constant by selecting a neighborhood value of appropriate size;

[0088] According to the small motion assumption of the Gaussian pyramid LK optical flow method, the second-order infinitesimal R2 is discarded, and the optical flow equation of the feature point of the region of interest at the bottom layer of the Gaussian pyramid is obtained, as follows:

[0089] D x u+D y v+D t =0 (4)

[0090] For each feature point of the region of interest at a specific layer of the Gaussian pyramid, the above optical flow equation can be established to obtain the feature point in the corresponding image which is closest to the GLOH-like descriptor value corresponding to the feature point of the region of interest; since the number of unknown quantities (u, v) is greater than the number of equations, a neighborhood of appropriate size (2m+1)×(2m+1) needs to be set, where m represents the neighborhood radius, and the neighborhood radius m is related to the layer number L of the Gaussian pyramid, that is, m=batch×2 L-1, batch represents the radius of the bottom layer of the Gaussian pyramid, so that the optical flow value of the pixel points in the specified neighborhood corresponding to the feature points of the region of interest is constant, and then the optical flow equation of the pixel points in the specified neighborhood corresponding to the feature points of the region of interest is established;

[0091] S5.5. Solving the optical flow equation of the pixel points in the specified neighborhood corresponding to the feature points of the region of interest at the bottom layer (the smallest scale layer) of the Gaussian pyramid by using the least square method, and then finding the offset situation u, v of the feature point closest to the GLOH descriptor value of the feature point of the region of interest, and then performing information transmission between different layers of the Gaussian pyramid and optical flow value updating, and so on, until the highest layer (the largest scale layer) of the Gaussian pyramid, and taking the calculation result of this layer as the final optical flow value, so as to obtain the final registration result of the feature points of the region of interest of the floating image opt and the reference image sar.

[0092] The above only describes the preferred embodiments of the present application, and it should be noted that for those skilled in the art, some improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A method for optical flow registration of optical and SAR remote sensing images based on U-Net, characterized in that, The method comprises the following steps: Step S1, construction of image data set, image preprocessing and labeling of region of interest; Step S2, training of visible light remote sensing image segmentation model and SAR remote sensing image segmentation model by using two U-Net network models respectively; Step S3, obtaining of U-Net network segmentation results of the image pair to be registered, and recording of pixel point sets labeled in specified channels in the two segmentation images respectively; Step S4, construction of region of interest feature points and class GLOH descriptors of pixel points in the specified neighborhood of the region of interest feature points of the image pair to be registered by using a gradient operator respectively; Step S5, registration of the region of interest feature points in the image pair to be registered by using a Gaussian pyramid LK optical flow method; S5.1, Gaussian pyramid down-sampling of the original image, i.e., the floating image opt and the reference image sar in step S3, and the value of the number of layers L of the Gaussian pyramid is related to the size of the original image, so that the small motion assumption of the Gaussian pyramid LK optical flow method is met; S5.2, the region of interest feature points and the pixel point set in the specified neighborhood thereof are recorded as O, and the class GLOH descriptors of the pixel points in the set O are used to replace the pixel values at the corresponding positions of each layer of the Gaussian pyramid, so that the brightness constant assumption of the Gaussian pyramid LK optical flow method is met; S5.3, according to the brightness constant assumption, the following equation is obtained at the bottom layer of the Gaussian pyramid: D1(x1, y1, t) = D2(x1 + u, y1 + v, t + 1) (2) In the formula, D1 and D2 respectively represent the class GLOH descriptors corresponding to the region of interest feature points of the two original images at the bottom layer of the Gaussian pyramid; x1 and y1 respectively represent the row and column where the region of interest feature points are located; t represents the corresponding time of the floating image opt; u and v respectively represent the horizontal offset of the scale u and the vertical offset of the scale v of the region of interest feature points in the floating image opt; the right side of formula (2) is expanded by Taylor series to obtain the following equation: D(x1+u, y1+v, t+1) = D(x1, y1, t) + D x u + D y v + D t + R2 (3) where D x , D y respectively represent the gradient of the GLOH-like descriptor value corresponding to the feature point of the region of interest in the horizontal direction and the vertical direction, respectively, D t represents the partial derivative of the GLOH-like descriptor value corresponding to the feature point of the region of interest in the direction of time t; R2represents the second-order infinitesimal term. S5.4, according to the small motion assumption of the Gaussian pyramid LK optical flow method, the second-order infinitesimal R2 is discarded to obtain the region of interest feature point optical flow equation at the bottom layer of the Gaussian pyramid: D x u+D y v+D t =0(4) For each feature point of the region of interest of the specific layer of the Gaussian pyramid, the above optical flow equation can be established to obtain the feature point in the corresponding image which is closest to the GLOH descriptor value corresponding to the feature point of the region of interest; the neighborhood size is set to (2m+1)×(2m+1), m represents the neighborhood radius, the neighborhood radius m is related to the layer number L of the Gaussian pyramid, that is, m=batch×2 L-1 , batch represents the neighborhood radius of the bottom layer of the Gaussian pyramid, so that the optical flow value of the pixel point in the specified neighborhood corresponding to the feature point of the region of interest is a constant, thereby establishing the optical flow equation of the pixel point in the specified neighborhood corresponding to the feature point of the region of interest. S5.5, the least square method is used to solve the optical flow equation of the pixel points in the specified neighborhood corresponding to the region of interest feature points at the bottom layer of the Gaussian pyramid, the offset u and v of the feature point with the closest class GLOH descriptor value to the region of interest feature points are found, and then the information transmission between different layers of the Gaussian pyramid and the optical flow value updating are performed, and the process is repeated until the highest layer of the Gaussian pyramid, and the calculation result of this layer is taken as the final optical flow value, so that the final registration result of the region of interest feature points of the floating image opt and the reference image sar is obtained.

2. The U-Net based optical and SAR remote sensing image optical flow registration method of claim 1, wherein, The specific operation steps of step S1 are as follows: S1.1, construction of visible light remote sensing image data set and SAR remote sensing image data set respectively; A plurality of visible light remote sensing images and SAR remote sensing images are collected by a satellite to form a visible light remote sensing image data set and a SAR remote sensing image data set respectively; S1.2, the acquired multiple visible light remote sensing images and SAR remote sensing images are converted into gray scale images respectively by using formula (1), and the obtained multiple gray scale images are subjected to pixel value consistency processing; In the formula, I'(x, y) represents the gray scale image after pixel value consistency processing, max(I(x, y)) represents the maximum pixel value in the original gray scale image without pixel value consistency processing, min(I(x, y)) represents the minimum pixel value in the original gray scale image without pixel value consistency processing, and x and y represent the row and column of the pixel point respectively; S1.3, the pre-processed multiple gray scale images are labeled with their interested regions by using labelme software, and visible light remote sensing image training data set and SAR remote sensing image training data set are constructed respectively.

3. The U-Net based optical and SAR remote sensing image optical flow registration method of claim 2, wherein, The specific operation steps of step 1.3 are as follows: According to the registration purpose, the pixel points of the interested regions of the pre-processed multiple gray scale images are manually labeled by using labelme software, and the number of classes of the interested regions is the subsequent classification number of the pixel points; the multiple gray scale images corresponding to the visible light remote sensing images are randomly divided into opt_training data set, opt_test data set and opt_validation data set according to the proportion, and the proportions of the opt_training data set, the opt_test data set and the opt_validation data set are 70%, 20% and 10% respectively; the multiple gray scale images corresponding to the SAR remote sensing images are randomly divided into sar_training data set, sar_test data set and sar_validation data set according to the proportion, and the proportions of the sar_training data set, the sar_test data set and the sar_validation data set are 70%, 20% and 10% respectively; and the multiple gray scale images corresponding to the visible light remote sensing images after labeling the interested regions are recorded as opt_label data set, and the multiple gray scale images corresponding to the SAR remote sensing images after labeling the interested regions are recorded as sar_label data set.

4. The U-Net based optical and SAR remote sensing image optical flow registration method of claim 3, wherein, The specific operation steps of step S2 are as follows: S2.1, two U-Net network models are built; S2.2, the visible light remote sensing image segmentation model and the SAR remote sensing image segmentation model are trained respectively; The visible light remote sensing image training data set and the SAR remote sensing image training data set constructed in step S1 are input into the two U-Net network models built above, wherein the opt_training data set and the sar_training data set are used for training the network, the opt_test data set and the sar_test data set are used for evaluating the network performance, the opt_validation data set and the sar_validation data set are used for setting the network parameters, and finally the trained visible light remote sensing image segmentation model and SAR remote sensing image segmentation model are obtained.

5. The U-Net based optical and SAR remote sensing image optical flow registration method of claim 4, wherein, The specific operation steps of step S3 are as follows: S3.1 Collecting a pair of visible remote sensing image and SAR remote sensing image in addition to the visible remote sensing image dataset and the SAR remote sensing image dataset constructed in step S1, pre-processing the to-be-registered image pair according to the method in step S1 to obtain the corresponding floating image opt and the reference image sar, inputting the floating image opt and the reference image sar into the visible remote sensing image segmentation model and the SAR remote sensing image segmentation model respectively, and obtaining the U-Net network segmentation result of the to-be-registered image pair, i.e. two segmentation images; S3.2Each channel in the segmentation result of the above U-Net network corresponds to a different class of region of interest, and the pixel points marked by the channel are specified as the location of the region of interest in the corresponding segmentation image. The set S of pixel points marked by the specified channel in the two segmentation images is recorded respectively opt and S sar .

6. The U-Net based optical and SAR remote sensing image optical flow registration method of claim 5, wherein, The specific operation steps of step S4 are as follows: S4.1 calculating the gradient value corresponding to the floating image opt by using the Canny operator, and calculating the gradient value corresponding to the reference image sar by using the ROEWA operator; S4.2 extracts the DOG scale space feature point set F, to obtain the feature point set F of the region of interest of the floating image opt opt = {f1, f2…fN}, N is the number of feature points of the region of interest of the floating image opt, f1, f2…f N N are feature points in the set N The feature point set F of the region of interest of the reference image sar sar = {f1, f2…fM}, M is the number of feature points of the region of interest of the reference image sar, f1, f2…f M M are feature points in the set M ​ S4.3 Iterate N times, sequentially determining the set of feature points F of the region of interest in the floating image opt. opt Do the elements in the set belong to set S? opt If so, then in the set of feature points F of the region of interest opt If the feature point is retained, it is retained; otherwise, it is removed. The set of feature points in the region of interest of the updated floating image opt is denoted as FO. update ; Repeat the process M times, sequentially determining the feature point set F of the region of interest in the reference image sar. sar Do the elements in the set belong to set S? sar If so, then in the set of feature points F of the region of interest sar The feature point is retained if it is not present, and removed otherwise. The set of feature points in the region of interest of the updated reference image sar is denoted as FS. update ; S4.4 constructing the region of interest feature points of the floating image opt and the region of interest feature points of the reference image sar and the class GLOH descriptor of the pixel points in the specified neighborhood thereof.

7. The U-Net based optical and SAR remote sensing image optical flow registration method of claim 6, wherein, The specific operation steps of step S4.4 are as follows: The neighborhood radius is set as r, a logarithmic-polar coordinate three-ring is drawn with each processing pixel point as the center, the three rings have three values in the radial direction, and from inside to outside, they are r1, r2 and r3, and satisfy r1=0.25×r3, r2=0.73×r3, r3=4a, the value of a determines the size of the ring; the angle of the three rings is equally divided into 45°, so that the three rings have 17 subintervals, and mathematical statistics of image gradient amplitude accumulation is performed in each subinterval.

Images