High-precision control point precise extraction method based on cooperation of SAR image and optical image

Abstract

The application provides a high-precision control point precise extraction method based on SAR images and optical images, (1) point-like ground feature type characteristics are identified and selected on the SAR images, and a feature point identification network is used to accurately position the feature center pixel, which is used as a primary selected control point; (2) a stereoscopic SAR image intersection is constructed, and the pixel coordinates of the primary selected control point are solved; (3) through optical-SAR image matching, the pixel coordinates of the primary selected control point are determined on the optical images, and the primary selected control points that are successfully matched are used as final control points. The application fuses spaceborne optical and SAR satellite images, develops a high-precision joint control point extraction method, effectively solves the positioning problem of space mapping technology in a large range and a difficult-to-reach area, provides reliable and efficient new type technical support for global infrastructure construction, resource exploration, disaster monitoring and the like, and has remarkable economic benefits and social benefits.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote sensing image processing, and particularly relates to a high-precision control point accurate extraction method based on cooperation of SAR images and optical images. BACKGROUND

[0002] In recent years, with the rapid development of domestic optical and synthetic aperture radar (SAR) satellites and related technologies, it has become a top priority to build self-controllable SAR and optical load high-precision control points.

[0003] The current research mainly focuses on the extraction of high-precision control points of stereo SAR under optical assistance, and few applications consider the joint extraction of control points of optical and interferometric SAR payloads. For example: 1. A stereo SAR and optical joint photogrammetry method and computer readable medium (CN 118463934A): a stereo SAR and optical joint photogrammetry method and computer readable medium are proposed. First, the original optical image, SAR image and corresponding auxiliary file, digital elevation model are obtained, and the SAR image is subjected to Lee filtering; the optical image is determined to be a city or non-city area according to the ground object type; if the optical image is a city area, a coarse-to-fine target point analysis method is used to obtain the accurate image coordinates of the control points on the SAR image; then the forward intersection method is used to calculate the geographic coordinates of the control points; if the optical image is a non-city area, an artificial-assisted matching prediction method is used to obtain the accurate image coordinates of the control points on the SAR image; then the forward intersection method is used to calculate the geographic coordinates of the control points; the extracted SAR control points are converted to the optical image; according to the extracted control points, a region network adjustment is carried out using a rational function model compensation method based on the image plane; the adjustment results are used for optical image positioning model updating, thereby improving the positioning accuracy of the optical image. This patent only uses optical image assistance to determine the ground object type of the SAR image, realizes the extraction of stereo SAR and optical image control points and the improvement of optical image positioning, and does not comprehensively consider the joint extraction of control points of optical and interferometric SAR payloads. 2. A SAR image control point screening and extraction method and system coordinated with optical images (CN 118191821A): a SAR image control point screening and extraction method and system coordinated with optical images are proposed. The optical orthographic image, digital elevation model, SAR left image and right image and their corresponding auxiliary files are obtained; the planar position of the metal structure on the optical orthographic image is measured, and the digital elevation model is combined to obtain the pixel coordinates of the metal structure as the pixel coordinates of the scatterer-like points; the left image and right image geometric positioning models are constructed according to the SAR left image and right image and their auxiliary files, respectively; the pixel coordinates of the scatterer-like points are mapped into the accurate image coordinates in the left image and right image using the left image and right image geometric positioning models, respectively; the pixel coordinates corresponding to the accurate image coordinates in the left image and right image are calculated and used as the pixel coordinates of the control points; the pixel coordinates of the control points are used to participate in aerial triangulation to realize the positioning accuracy improvement of the micro SAR satellite image or optical satellite image with low geometric positioning accuracy. This patent only uses the measurement of the planar position of the metal structure on the optical orthographic image to extract the control points of stereo SAR by constructing a geometric positioning model, realizes the positioning accuracy improvement of the micro SAR satellite image or optical satellite image with low geometric positioning accuracy, but also does not comprehensively consider the joint extraction of control points of optical and interferometric SAR payloads. SUMMARY

[0004] The application provides a high-precision control point precise extraction method based on SAR images and optical images, which is improved in that the method comprises the following steps:

[0005] (1) identifying and selecting point feature types on the SAR image, using a feature point recognition network to accurately position the feature center pixel as a preliminary control point;

[0006] (2) constructing a stereoscopic SAR image intersection and solving the pixel coordinates of the preliminary control point;

[0007] (3) determining the pixel coordinates of the preliminary control point on the optical image through optical-SAR image matching, and taking the preliminary control point with successful matching as the final control point.

[0008] Further, the step (1) comprises

[0009] (1.1) labeling the region of the point feature in the SAR image of multiple perspectives as a positive sample;

[0010] (1.2) selecting a region without point features in the SAR image of multiple perspectives as a negative sample;

[0011] (1.3) using a ResNet34 and a feature pyramid network mixed deep network to construct a SAR image control point extraction network;

[0012] (1.4) using a focal loss function to optimize the positive and negative samples in the SAR image of multiple perspectives through the SAR image control point extraction network, and extracting the point feature as the preliminary control point pixel coordinates.

[0013] Further, the step (1.3) comprises combining ResNet34 and a feature pyramid network to construct a mixed deep network structure for SAR image point feature extraction, which is used to identify the control point feature and extract the feature after identification; wherein the features of five different spatial scales of the convolution block of ResNet34 are fused with the FPN features to improve the extraction of control points of different scales.

[0014] Further, the step (1.4) comprises calculating the speed of weight reduction of the positive and negative samples in the SAR image of multiple perspectives through the focal loss and learning the control point feature, and extracting the point feature as the preliminary control point pixel.

[0015] Further, the step (2) comprises

[0016] (2.1) based on the total least squares estimation of the distance-Doppler positioning model and the corresponding error equation, realizing the construction of the stereoscopic SAR image intersection positioning equation.

[0017] (2.2) On the primary selected control point pixels, extract the stereo SAR image intersection points, linearize them according to the range-Doppler model, and establish the intersection positioning equation formula, and according to the positioning equation formula, solve the pixel coordinates of the control points.

[0018] Further, the step (2.1) comprises

[0019] The intersection positioning equation of the multi-piece stereo SAR image is constructed by the range-Doppler positioning model, and the ground point coordinates are P T , the orbit position of the SAR satellite when acquiring the stereo SAR image is P Si , the velocity is V Si , and i represents the number of observations:

[0020] R i = |P Si -P T | (1)

[0021]

[0022] Wherein, R i and f DCi are the distance value and Doppler frequency measured by the radar respectively, and λ is the radar wavelength; through two or more sets of range-Doppler equations, the equations are linearized, and solved by Newton iteration and least squares.

[0023] Further, the step (2.2) comprises

[0024] Extract the stereo SAR image intersection points, linearize them according to formula (1) range-Doppler model, and establish the stereo SAR image intersection positioning equation formula (2), and solve the pixel coordinates of the control points.

[0025] Further, formula (2) is converted into least squares estimation for solving, and numerical weather model and ionosphere model are introduced to correct radar wave atmospheric transmission delay.

[0026] Further,

[0027] (3.1) Collect image data sets containing optical-SAR images, and obtain the primary control point pixels in the SAR image through the feature points identified in step (1); each pair of data sets will include a SAR image block centered on the feature points, and find the corresponding matching area of the SAR image block in the optical image;

[0028] (3.2) For image registration tasks, the final control points are obtained through the mixed feature matching network of the local feature extraction module and the global feature alignment module.

[0029] Further, the step (3.2) comprises matching the optical and SAR images by using a local feature extraction module and a global feature alignment module hybrid network for the image registration task through the distance intersection of the stereo SAR image, determining the pixel corresponding to the primary control point from the optical image, and taking the primary control point matched successfully as the final control point.

[0030] Advantages:

[0031] The application adopts the ResNet34 and feature pyramid network (FPN) to construct a hybrid deep network architecture specially for SAR image point feature extraction. The architecture deeply integrates the features of five different spatial resolution convolution modules on the network level, and realizes the effective coupling of multi-scale features by means of the bottom-up feature aggregation path of FPN, so that the point target can be accurately captured and recognized at different spatial resolutions. The architecture shows significant technical innovation and engineering application value in point target feature saliency modeling and multi-resolution adaptability, and provides a new technical idea for the SAR image analysis field.

[0032] The application constructs a new control point precise three-dimensional coordinate inversion mode through the single SAR image primary control point extraction and stereo SAR image distance intersection technology. The feature information of the single SAR image primary control point is deeply mined, and the multi-view geometric observation advantage of the stereo SAR image is fully utilized, and the distance intersection algorithm is used to realize the high-precision inversion of the spatial position of the control point.

[0033] The application innovatively integrates the HardNet module and the Transformer encoder to construct a hybrid feature matching network, which aims to accurately locate the pixels in the optical image corresponding to the SAR control points. The HardNet module can sensitively capture the subtle differences between different modal images due to its excellent local feature extraction performance, and provides strong support for detail discrimination. The Transformer encoder is good at deep modeling of global context modeling in feature dimension, effectively enhances the consistency of cross-modal feature representation by mining the long-range dependency relationship between features, solves the significant difference problem between SAR images and optical images in feature representation, greatly reduces the complexity of cross-modal feature matching, and provides an efficient solution for accurate matching of cross-modal images.

[0034] The application fuses the spaceborne optical and SAR satellite images, and carries out a high-precision joint control point extraction method; through constructing a cross-modal image collaborative processing framework, the complementary advantages of the high-resolution texture features of the optical image and the all-weather, multi-polarized electromagnetic scattering characteristics of the SAR image are fully tapped, a new type of technical support system for overseas uncontrolled area mapping is created, the positioning problem of the space mapping technology in a large range and a difficult-to-reach area is effectively solved, reliable and efficient new type of technical support is provided for global infrastructure construction, resource exploration, disaster monitoring and other projects, and remarkable economic benefits and social benefits are obtained.

[0035] It should be understood that the foregoing general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the scope of the application claimed. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 A flow chart of a high-precision control point precise extraction method based on SAR image and optical image collaboration according to the application;

[0037] Figure 2 A SAR image feature point recognition network architecture diagram in a high-precision control point precise extraction method based on SAR image and optical image collaboration according to the application;

[0038] Figure 3 An optical and SAR image registration network architecture diagram in a high-precision control point precise extraction method based on SAR image and optical image collaboration according to the application.

[0039] It should be understood that the drawings are not necessarily to scale, presenting a slightly simplified representation of various features illustrating the basic principles of the present disclosure. Specific design features of the application as disclosed herein, including for example, specific dimensions, orientations, locations, and shapes, will be determined in part by the particular intended application and use environment.

[0040] In the drawings, like reference numerals refer to the same or similar functionalities throughout the several views of the drawings. DETAILED DESCRIPTION

[0041] Reference will now be made in detail to various embodiments of the application, examples of which are illustrated in the accompanying drawings and described below. While the application will be described in conjunction with the example embodiments, it should be understood that the application is not intended to be limited to those example embodiments. On the contrary, the application is intended to cover alternatives, modifications, equivalents, and other embodiments, which can be included within the spirit and scope of the application as defined by the appended claims.

[0042] Hereinafter, exemplary embodiments of the present application will be described in detail with reference to the accompanying drawings. The specific configurations and functions described in the exemplary embodiments of the present application are merely for illustrative purposes. Embodiments in accordance with the concept of the present application can be implemented in various forms, and it should be understood that they should not be interpreted as being limited by the exemplary embodiments described in the exemplary embodiments, but include all modifications, equivalents, or alternatives included in the spirit and scope of the present application.

[0043] Throughout the specification, the professional terms used herein are merely for describing various exemplary embodiments, and are not intended to be limiting. It will be further understood that the terms "include", "comprise", "have", and the like, when used in the exemplary embodiments, specifically refer to the existence of the stated components, steps, operations, or elements, but do not exclude the existence or addition of one or more other components, steps, operations, or elements.

[0044] Control points are the key to georeferencing data. They are landmarks of known locations, usually physical features on the ground such as markers, geographic coordinate points, or GPS measurement points. The role of control points is to calibrate or georeference remote sensing images to ensure consistency between the images and the geographic coordinate system. By identifying and matching control points in the image with control points of known geographic location, the image can be accurately positioned on the map or the earth's surface. Currently, the main methods of obtaining high-precision control points include satellite GPS positioning technology, remote sensing technology, and laser radar scanning, etc. However, satellite GPS and laser radar scanning are not suitable for difficult-to-reach or large-scale areas. Satellite remote sensing technology can remotely obtain high-resolution images of the ground to assist in determining the location of control points, and this method is also suitable for large-scale, difficult-to-reach areas.

[0045] As shown in Figure 1 The present application is based on a high-precision control point extraction method based on SAR images and optical images, which comprises:

[0046] 1. According to the selected point-like ground object type, identify the selected point-like ground object type feature on the SAR image, and use the feature point recognition network to accurately position the feature center pixel as the preliminary control point.

[0047] (1.1) Accurately identify feature points with high representativeness from SAR images, such as streetlights, tower tips, fence posts, building corners, etc. The feature points serve as reliable reference points for registration with optical images. In SAR images, feature points often appear as small and local salient structures in the image. By accurately positioning the feature points, a corresponding relationship between images of different modalities is established.

[0048] (1.2) In the selection process of feature points, multiple stable detected structures under multiple perspectives are adopted, such as object edges with obvious geometric features or significant reflection points.

[0049] (1.3) The region of the point feature is labeled as a positive sample in the SAR image under multiple perspectives; the region without point features is selected as a negative sample in the SAR image under multiple perspectives; the manually labeled SAR image feature point sample is extracted to ensure that the model has good adaptability in various actual scenes. The selected scenes include cities, mountains and other complex environments, ensuring that the data set has high diversity; the manually labeled data set can cover the identification task of small-scale local feature points, enhancing the identification ability of the model in complex environments.

[0050] (1.4) A hybrid deep network structure for SAR image point feature extraction is constructed by combining ResNet34 and feature pyramid network (FPN), and the feature points are identified; five different spatial scale convolution block features are fused at the network structure level, and the multi-layer feature aggregation from bottom to top is realized through FPN, so that the point control target can be effectively identified under different resolutions. The structure has significant innovation and adaptability in the expression of point target significance.

[0051] As shown in Figure 2 , the integration of ResNet34 and feature pyramid network (FPN) is as follows:

[0052] FPN effectively integrates high-level semantic information and low-level spatial details by constructing a top-down path and a horizontal connection; this structure is suitable for detecting small and local feature points in SAR images, avoiding the risk of missing subtle features; FPN can fully exert its advantages when processing features of different scales and resolutions, greatly improving the robustness and adaptability of the model.

[0053] (1.5) The focal loss function (Focal Loss) is used to optimize the positive and negative samples in the SAR image under multiple perspectives through the SAR image control point extraction network, extract the point features as the initial control point pixel coordinates, and solve the problem of significant class imbalance between positive and negative samples in the SAR image; the focal loss effectively reduces the weight of easy-to-classify samples by introducing an adjustment factor, and focuses on difficult-to-classify samples, so that the model can effectively learn under the condition of class imbalance, especially when the background samples are dominant and the feature points are rare.

[0054] The focal loss expression is as follows:

[0055] L focal =-α(1-p t )γ log(p t )-(1-α)(p t ) γ log(1-p t )

[0056] where p t is the predicted probability of the correct class, α is the balance factor of positive and negative samples, and γ is the degree parameter for controlling the speed of down-weighting easy-to-classify samples. In this way, the focal loss can effectively guide the model to preferentially learn the features of those difficult-to-classify samples, thereby improving the recognition ability of the model in complex scenes.

[0057] 2. High-precision stereo SAR image intersection to solve the initial control point pixel coordinates;

[0058] (2.1) Based on the total least squares estimation of the distance-Doppler positioning model and the corresponding error equation, the stereo SAR image intersection positioning equation is constructed.

[0059] The intersection positioning equation of multi-piece stereo SAR image is constructed by the distance-Doppler positioning model, assuming that the ground point coordinates are P T , the orbital position of the SAR satellite when obtaining the stereo SAR image is P Si , the velocity is V Si , and i represents the observation number:

[0060] R i = |P Si -P T | (1)

[0061]

[0062] where R i and f DCi are the distance value and Doppler frequency measured by the radar, respectively, and λ is the radar wavelength. Two sets (four) or more than two sets of distance-Doppler equations can be solved. The solution can linearize the equation and solve it by Newton iteration and least squares.

[0063] (2.2) After extracting the SAR image intersection points, the intersection positioning equation (formula (2)) is established according to the distance-Doppler model (formula (1)) after linearization, and the pixel coordinates of the control points are solved.

[0064] In the technical solution, formula (2) can be converted into a least square problem for solving, and intermediate error terms including atmospheric delay, transceiver channel delay, radar clock error, and ephemeris error are contained, and if direct solving is performed, the influence of the error terms cannot be overcome. The total least square method is used for solving, and a numerical weather model and an ionosphere model are introduced to estimate an initial value of atmospheric delay.

[0065] 3. Determine the pixel coordinates of the preliminary control points on the optical image by optical-SAR image matching, and take the successfully matched preliminary control points as the final control points.

[0066] (3.1) Collect a set of image data sets containing SAR and optical images (optical-SAR images), and ensure that the feature points identified in the first step are preliminary control point pixels that can be obtained in the SAR image; each pair of data sets will include a SAR image block centered on the feature points, and a corresponding matching area of the SAR image block found in the optical image;

[0067] (3.2) Design a fusion network architecture for image registration tasks, which includes a local feature extraction module (HardNet) and a global feature alignment module (Transformer), and a hybrid feature matching network combining HardNet and Transformer is used to accurately locate the pixels corresponding to the SAR control points in the optical image;

[0068] The preliminary control points extracted from the stereo SAR image are combined with the distance intersection process of the stereo SAR image to invert the precise three-dimensional coordinates of the control points, the same features of the optical and SAR images are matched by using the HardNet and Transformer hybrid network, the pixels corresponding to the preliminary control points are determined on the optical image, and the final control points are generated after successful matching;

[0069] The HardNet module has excellent local feature extraction capability and is suitable for distinguishing details between different modalities; the Transformer encoder models the global context information at the feature level and improves the representation consistency between cross-modal features. Through the combined structure, the present application effectively solves the key technical problems such as large feature representation difference and difficult matching between SAR and optical images.

[0070] The local feature extraction module (HardNet) extracts high-recognition local features from SAR and optical images by using a HardNet algorithm; the local feature extraction module uses two L2Net networks sharing weights, each of which is used for processing feature extraction of SAR and optical images, and improves the calculation efficiency of the model by retaining the convolution layer; the global feature alignment module (Transformer) uses a Transformer encoder to capture the global context information between images and accurately align the features in SAR and optical images; the multi-head attention mechanism of the Transformer can effectively focus on the most relevant features and align them, ensuring cross-modal feature matching.

[0071] As shown in Figure 3 For the image registration task, the local feature extraction module and the global feature alignment module are used to match the corresponding features of optical and SAR images through the distance intersection of stereo SAR images, and the corresponding pixels of the initial control points are determined from the optical images, and the matching successful initial control points are used as the final control points.

[0072] The above embodiments of the present application can be implemented in various hardware, software coding or combination of both. For example, the embodiments of the present application can also represent program codes for executing the above methods in a data signal processor. The present application can also relate to various functions performed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array. The above processor can be configured to perform specific tasks by executing machine-readable software code or firmware code defining the specific methods disclosed in the present application. The software code or firmware code can be developed in different programming languages and different formats or forms, such as source code, compiled code, interpreted code, scripts, machine code, and the like. The software code or firmware code can be stored in different types of computer-readable storage media, such as computer-readable storage medium 1300. Different types of computer-readable storage media can be used in the implementation of the present application, including magnetic storage media, optical storage media, hardwired or circuitry components, semiconductor memory components, etc.

[0073] The foregoing description of specific exemplary embodiments of the application has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the application to the precise forms disclosed, and obviously many modifications and variations are possible in light of the above teaching. The exemplary embodiments were chosen and described in order to explain certain principles of the application and their practical application, to enable others skilled in the art to make or utilize the application in various embodiments and with various modifications as are suited to the particular use contemplated, as well as to enable others to understand the application to appreciate the principles and concepts of the application for themselves. It is intended that the scope of the application be defined by the claims appended hereto and their equivalents.

[0074] It is understood that the above embodiments are only exemplary for illustrating the principles of the present application, and the present application is not limited thereto. Various modifications and improvements can be made by those skilled in the art without departing from the spirit and scope of the present application, and these modifications and improvements are also considered as the protection scope of the present application.

Claims

1.A high-precision control point precise extraction method based on SAR image and optical image cooperation, characterized in that, The method comprises: (1) identifying and selecting point feature type on SAR image, using feature point recognition network to accurately locate feature center pixel as preliminary control point; (1.1) labeling the region of point feature in multiple view SAR images as positive sample; (1.2) selecting the region without point feature in multiple view SAR images as negative sample; (1.3) using ResNet34 and feature pyramid network mixed deep network to construct SAR image control point extraction network; (1.4) using focal loss function to optimize positive and negative samples in multiple view SAR images through SAR image control point extraction network, extracting point feature as preliminary control point pixel coordinate; (2) constructing stereo SAR image intersection and solving pixel coordinate of preliminary control point; (2.1) based on distance-Doppler positioning model and overall least squares estimation of corresponding error equation, realizing stereo SAR image intersection positioning equation construction; (2.2) extracting stereo SAR image intersection point on the preliminary control point pixel, linearizing according to the distance-Doppler model to establish intersection positioning equation formula, and solving the pixel coordinate of the control point according to the positioning equation formula; (3) determining the pixel coordinate of the preliminary control point on the optical image through optical-SAR image matching, and taking the matched preliminary control point as the final control point. 2.The method of claim 1, wherein, The step (1.3) comprises ResNet34 combined with feature pyramid network to construct a mixed deep network structure for SAR image point feature extraction, which is used to identify and extract control point features after identification through the mixed deep network; wherein the five different spatial scale convolution block features of ResNet34 are fused with FPN features to improve the extraction of control points of different scales. 3.The method of claim 2, wherein, The step (1.4) comprises calculating the speed of down-regulating the weight of positive and negative samples in multiple view SAR images and learning control point features through focal loss, and extracting point features as preliminary control point pixels. 4.The method of claim 1, wherein, The step (2.1) comprises; The intersection positioning equation of multi-piece stereo SAR images is constructed by the range-Doppler positioning model, and the ground point coordinates are P T , the orbit position of the SAR satellite when obtaining stereo SAR images is P Si , the velocity is V Si , i represents the observation quantity: (1)； (2)； wherein, and are the radar measured range and Doppler frequency, respectively, The step (2.2) comprises is the radar wavelength; the equations are linearized and solved by Newton iteration and least squares. 5.The method of claim 4, wherein, extracting stereo SAR image intersection point, linearizing according to formula (1) distance-Doppler model to establish stereo SAR image intersection positioning equation formula (2), and solving the pixel coordinate of the control point. Convert formula (2) into least squares estimation for solving, and introduce numerical weather model and ionosphere model to correct radar wave atmospheric transmission delay. 6.The method of claim 1, wherein, 7. The high-precision control point precise extraction method based on SAR image and optical image cooperation according to claim 1, characterized in that (3.1) collecting image data sets containing optical-SAR images, obtaining preliminary control point pixels in SAR images through identified feature points in step (1); each pair of data sets will include a SAR image block centered on the feature point, and the corresponding matching region found in the optical image; (3.2) for image registration task, using a mixed feature matching network of local feature extraction module and global feature alignment module to obtain the final control point. ​ 8.The method of claim 7, wherein, The step (3.2) comprises matching the homonymic features of the optical and SAR images by using a local feature extraction module and a global feature alignment module hybrid network for the image registration task through the distance intersection of the stereo SAR image, determining the pixel corresponding to the primary selected control point from the optical image, and taking the primary selected control point with successful matching as the final control point.

Images