Multi-modal image common visible region extraction method, system, equipment and medium

Abstract

The invention discloses a multi-modal image common visible region extraction method, system and device and a medium, and relates to the technical field of image processing. Comprising the following steps: performing scale transformation processing of different magnification times on an original image pair for multiple times to obtain a plurality of initial processing image pairs; carrying out key point matching processing on the original image pair to obtain a first matched key point pair and a second matched key point pair; respectively carrying out key point matching processing on the original image pair and the initial processing image pair to obtain a third matched key point pair and a fourth matched key point pair; combining and splicing the first matching key point pair and the third matching key point pair to obtain a first matching point pair; projecting the second matching key point pair and the fourth matching key point pair to the position of the original image pair, and then combining and splicing to obtain a second matching point pair; removing abnormal key points of the two matching point pairs to obtain final matching key points; and determining a common visible area. According to the invention, the matching precision of the common visible region of the multi-modal image can be improved, and a better registration effect is realized.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, system, device and medium for extracting a common visible area of ​​a multimodal image. Background Art

[0002] SAR (Synthetic Aperture Radar) images and optical images each have unique advantages in the field of remote sensing, but their different imaging mechanisms lead to differences in information coverage and representation. It is extremely necessary to extract the common visible areas of SAR images and optical images, because this can not only integrate the advantages of both and improve the richness and accuracy of information, but also provide a more comprehensive observation perspective in disaster monitoring, environmental change assessment, resource exploration, navigation, guidance and control, etc. Through the precise extraction of the common visible area, it is possible to effectively remove mismatched information and reduce data redundancy, thereby improving the efficiency and accuracy of image interpretation and providing a more reliable scientific basis for decision support.

[0003] In the prior art, deep learning-based registration methods have been widely used to extract the co-visible area of ​​SAR images and optical images. The specific implementation process includes: preprocessing the SAR image and the optical image, extracting features from the preprocessed image using a deep learning model, matching the extracted features using a deep learning framework, and obtaining preliminary matching results. The preliminary matching results are screened using the Random Sample Consensus (RANSAC) algorithm to remove erroneous matching points. Based on the screened matching point pairs, the SAR image and the optical image are aligned through an image transformation model, and the co-visible area of ​​the two images is extracted based on the position information of the matching point pairs.

[0004] The defects of the above-mentioned prior art are: there are significant differences between SAR images and optical images, including brightness, contrast, texture, color information, etc. These differences make it difficult to capture enough common features in the feature extraction stage, thereby affecting the subsequent matching and registration effects. Summary of the invention

[0005] Based on this, it is necessary to provide a method, system, device and medium for extracting the common visible area of ​​a multimodal image in response to the above technical problems.

[0006] An embodiment of the present invention provides a method for extracting a common visible area of ​​a multimodal image, comprising:

[0007] Obtaining an original image pair consisting of a SAR image and an optical image to be registered;

[0008] Performing multiple scale transformations with different magnifications on the original image pair to obtain multiple initial processed image pairs;

[0009] The LoFTR model is used to perform key point matching on the original image pair to obtain a first matching key point pair KeyptLA and a second matching key point pair KeyptLB;

[0010] The original image pair and the initial processed image pair are respectively matched with key points by the SuperGlue model to obtain a third matching key point pair KeyptSA and a fourth matching key point pair KeyptSB;

[0011] The first matching key point pair KeyptLA and the third matching key point pair KeyptSA are merged and spliced ​​to obtain a first matching point pair mkptA; the second matching key point pair KeyptSB and the fourth matching key point pair KeyptSB are projected to the position of the original image pair and then merged and spliced ​​to obtain a second matching point pair mkptB;

[0012] Abnormal key points in the first matching point pair mkptA and the second matching point pair mkptB are cleared by using the DBSCAN clustering algorithm to obtain final matching key points; and the common visible area in the original image pair is determined according to the final matching key points.

[0013] Optionally, the image matching key points represent image position information of a target area in an image, including:

[0014] The upper left corner vertex of the image is taken as the coordinate origin, the horizontal right direction in the image is taken as the X axis, and the vertical downward direction in the image is taken as the Y axis; the image position information includes X-axis coordinate information and Y-axis coordinate information; the position information corresponding to the i-th image matching key point is expressed as (x 0i ,y 0i ), where x 0i Indicates the x-axis coordinate information corresponding to the i-th image matching key point, y 0i Represents the y-axis coordinate information corresponding to the i-th image matching key point, i = 1, 2, 3, ..., N, N is the number of image matching key points.

[0015] Optionally, performing key point matching processing on the original image pair by using the LoFTR model includes:

[0016] Establish pixel-level correspondence between image pairs to obtain joint information of the two images;

[0017] Based on the joint information of the two images, the self-attention and cross-attention layers in the Transformer are used to generate feature descriptors, and dense matching features are generated in the images of the original image pair through the feature descriptors;

[0018] The dense matching features are refined and classified to obtain the first matching key point pair KeyptLA and the second matching key point pair KeyptLB.

[0019] Optionally, performing key point matching processing on the original image pair and the initial processed image pair respectively by using the SuperGlue model includes:

[0020] Extract feature points and descriptors corresponding to the feature points from the original image pair, the first initial processed image pair, and the second initial processed image pair using the SuperPoint network of the SuperGlue model;

[0021] The extracted feature points are constructed into a graph structure, where each feature point is a node of the graph; the edges between the nodes are connected according to the geometric relationship of the feature points and the similarity of the descriptors;

[0022] Use graph neural network GNN to process the constructed graph structure and obtain updated features;

[0023] The matching layer of the SuperGlue model uses bilinear interpolation and attention mechanism to calculate the matching probability between feature points on the updated features to obtain the third matching key point pair KeyptSA and the fourth matching key point pair KeyptSB.

[0024] Optionally, obtaining the first matching point pair mkptA and the second matching point pair mkptB specifically includes:

[0025] For each scale-transformed initial processed image, the position coordinates of the initial processed image relative to the original image are determined according to the scaling ratio of the scale transformation. The calculation formula is:

[0026]

[0027] Where scale is the scaling ratio, x1 is the x-axis coordinate information corresponding to the image matching key point, and y1 is the y-axis coordinate information corresponding to the matching key point;

[0028] The coordinates of the first matching key point pair KeyptLA and the third matching key point pair KeyptSA are transformed to the coordinates of the original image position according to the scaling ratio, and then merged and spliced ​​to obtain the first matching point pair mkptA;

[0029] The coordinates of the second matching key point pair KeyptSB and the fourth matching key point pair KeyptSB are transformed to the coordinates of the original image position according to the scaling ratio, and then merged and spliced ​​to obtain the second matching point pair mkptB.

[0030] Optionally, the specific process of determining the common visible area in the image according to the final matching key points is as follows:

[0031] Set the neighborhood parameter ε and the minimum number of key points MinPts, where the neighborhood parameter ε is the neighborhood radius eps, and the point set consisting of all key points in the matching key point set mkpt = [mkptA, mkptB] is recorded as cluster C j ;

[0032] If the key point k j The ε neighborhood of contains at least MinPts key points, namely:

[0033] N ε (k j )≥MinPts;

[0034] Then k j is a core point in the matching key point set mkpt, where:

[0035] N ε (k j )={k i ∈mkpt|dist(k i ,k j )≤ε};

[0036]

[0037] Among them, dist(k i ,k j ) represents the key point k i and k j The Euclidean distance between

[0038] The matching key point set mkpt does not belong to any cluster C = {C1, C2…, C n The key point of} is the abnormal key point;

[0039] After removing abnormal key points, the outer boundary formed by the retained key points is the co-visible area in the original image pair.

[0040] An embodiment of the present invention further provides a multimodal image co-visible area extraction system, comprising:

[0041] An image pair acquisition module is used to acquire an original image pair consisting of a SAR image and an optical image to be registered;

[0042] An image pair processing module is used to perform multiple scale transformation processes with different magnifications on the original image pair to obtain multiple initial processed image pairs;

[0043] The first processing module is used to perform key point matching processing on the original image pair through the LoFTR model to obtain a first matching key point pair KeyptLA and a second matching key point pair KeyptLB;

[0044] The second processing module is used to perform key point matching processing on the original image pair and the initial processed image pair respectively through the SuperGlue model to obtain a third matching key point pair KeyptSA and a fourth matching key point pair KeyptSB;

[0045] A splicing module is used to merge and splice the first matching key point pair KeyptLA and the third matching key point pair KeyptSA to obtain a first matching point pair mkptA; and merge and splice the second matching key point pair KeyptSB and the fourth matching key point pair KeyptSB after projecting them to the position of the original image pair to obtain a second matching point pair mkptB;

[0046] The region extraction module is used to remove abnormal key points in the first matching point pair mkptA and the second matching point pair mkptB through the DBSCAN clustering algorithm to obtain the final matching key points; and determine the common visible area in the original image pair according to the final matching key points.

[0047] An embodiment of the present invention further provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the method for extracting a common visible area of ​​a multimodal image are implemented.

[0048] An embodiment of the present invention further provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for extracting a common visible area of ​​a multimodal image are implemented.

[0049] Compared with the prior art, the above-mentioned multimodal image common visible area extraction method, system, device and medium provided by the embodiments of the present invention have the following beneficial effects:

[0050] The present invention aims to significantly improve the matching accuracy and registration effect of multimodal images (such as SAR images and optical images) by using two models, LoFTR and SuperGlue. First, the LoFTR model is used to perform key point matching on the original image pair, and the first and second matching key point pairs (KeyptLA, KeyptLB) are extracted. This step effectively captures the preliminary feature associations between images. Next, the SuperGlue model is introduced to perform key point matching on the image pairs of the original image pair and the initial processed image pair to generate the third and fourth matching key point pairs (KeyptSA, KeyptSB). The powerful feature learning ability of the model further enhances the accuracy and robustness of the matching.

[0051] Finally, the matching results of LoFTR and SuperGlue are spliced ​​and integrated to generate the first and second matching point pairs (mkptA, mkptB). These matching point pairs are not only expanded in quantity, but also significantly improved in quality, and can more accurately capture the common features between SAR images and optical images. This dual-model collaborative strategy effectively solves the problem of insufficient common feature extraction in traditional methods and lays a solid foundation for achieving high-precision multimodal image registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The figure is a flowchart of a method for extracting a common visible area of ​​a multimodal image provided in an embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0054] In one embodiment, a method for extracting a common visible area of ​​a multimodal image is provided, the method comprising:

[0055] Step 1: Image scaling

[0056] An original image pair consisting of a SAR image and an optical image to be registered is obtained, and the original image pair is scaled for multiple times at different magnifications to obtain multiple initial processed image pairs.

[0057] Furthermore, as shown in Table 1, it is a table of registration data of original image pairs of the SuperGlue model at different scales (256×256, 288×288, 312×312 and 512×512).

[0058] Scale NIM RMSE Time(s) 256×256 29 64.9909 0.4462 288×288 24 62.3025 0.4554 312×312 28 61.3757 0.4628 512×512 38 70.2727 0.6692

[0059] In the table, Scale is the scaling ratio, NIM is the number of matching key point pairs extracted, RMSE is the root mean square error, and Time is the processing time of the SuperGlue model on the RTX 3080Ti GPU (in s).

[0060] As can be seen from Table 1, at the scales of 256x256, 288x288, and 312x312, the SuperGlue model extracted 29, 24, and 28 matching keypoint pairs, respectively. As the scale increases, the root mean square error (RMSE) gradually decreases, indicating that the quality of the extracted matching keypoint pairs has improved. However, the number of extracted matching keypoint pairs does decrease slightly. In addition, the processing time of the SuperGlue model increases slightly to 0.4628 seconds. When the scale reaches 512x512, the number of extracted matching keypoint pairs increases to 38. However, the RMSE also increases to 70.2727, and the processing time increases significantly to 0.6692 seconds. Therefore, the magnification of the scale transformation is determined by the matching degree of the images in the original image pair, the hardware facilities, and the model processing speed.

[0061] Exemplarily, the original image SAR-optical image pair is read, and the image is scaled to enlarge the image size to 1.1 times and 1.2 times, respectively, to obtain a first initial processed image pair and a second initial processed image pair.

[0062] Taking the initial processed image pair with an image size of 256×256 (the height and width of the image are both 256 pixels) as an example, the images in the initial processed image pair are scaled twice with different magnifications, and the size of the original image is enlarged from 256×256 to 288×288 and 312×312. The processed image pair with an image size of 288×288 is the first initial processed image pair, and the processed image pair with an image size of 312×312 is the second initial processed image pair.

[0063] Step 2: Use the LoFTR model to perform key point matching on the original image pair to obtain the first matching key point pair KeyptLA and the second matching key point pair KeyptLB. Use the SuperGlue model to perform key point matching on the original image pair and the initial processed image pair to obtain the third matching key point pair KeyptSA and the fourth matching key point pair KeyptSB.

[0064] Exemplarily, LoFTR is used to perform key point matching processing on an image pair with an image size of 256×256 (original image pair), and the first matching key point pair KeyptLA and the second matching key point pair KeyptLB are extracted; SuperGlue is used to perform key point matching processing on the original image pair with an image size of 256×256, the first initial processed image pair with an image size of 288×288, and the second initial processed image pair with an image size of 312×312, and the third matching key point pair KeyptSA and the fourth matching key point pair KeyptSB are extracted.

[0065] Step 2.1 performs key point matching processing on the original image pair through the LoFTR model to obtain the first matching key point pair KeyptLA and the second matching key point pair KeyptLB.

[0066] (1) Loading LoFTR model: Use the pre-trained LoFTR model. LOFTR (Local Feature TRansformer) is a Transformer-based local feature matching method that aims to solve the problem that traditional feature detectors have difficulty in repeatedly detecting points of interest in low-texture areas. Unlike traditional methods, LOFTR does not rely on feature detectors, but instead uses the Transformer's self-attention and cross-attention mechanisms to achieve feature matching.

[0067] (2) Matching Process

[0068] A. Pixel-level dense matching:

[0069] LOFTR first establishes dense pixel-level matching at a coarse-grained level. This step does not rely on traditional feature detection and description, but directly establishes pixel-level correspondences between image pairs to obtain joint information of the two images.

[0070] B. Acquisition of feature descriptors:

[0071] Based on the joint information of two images, feature descriptors are generated using self-attention and cross-attention layers in Transformer.

[0072] C. Utilization of global receptive field:

[0073] The global receptive field provided by the Transformer enables LOFTR to generate dense matching features in the image of the original image pair through feature descriptors.

[0074] D. Refinement of matching:

[0075] After the initial coarse-grained matching, LOFTR refines and classifies the dense matching features to obtain the first matching key point pair KeyptLA and the second matching key point pair KeyptLB to improve the matching accuracy and reliability. This step ensures that the final matching results have high accuracy in various scenarios.

[0076] Step 2.2 performs key point matching processing on the original image pair through the LoFTR model to obtain the first matching key point pair KeyptLA and the second matching key point pair KeyptLB.

[0077] (1) Load SuperGlue model: Use the pre-trained SuperGlue model.

[0078] The SuperGlue model is a feature matching method based on graph neural networks, which aims to improve the accuracy of feature matching between images.

[0079] (2) Matching Process

[0080] A. Feature extraction:

[0081] First, the SuperPoint network of the SuperGlue model is used to extract feature points and descriptors corresponding to the feature points from the original image pair, the first initial processed image pair, and the second initial processed image pair. The SuperPoint network can detect feature points and generate their descriptors, which will be used in the subsequent matching process.

[0082] B. Graph structure construction:

[0083] The extracted feature points are constructed into a graph structure, where each feature point is a node of the graph. The edges between nodes are connected according to the geometric relationship of the feature points and the similarity of the descriptors.

[0084] C. Graph Neural Network Processing:

[0085] The constructed graph is processed using a graph neural network (GNN) to obtain updated features. GNN propagates information in the graph through a multi-layer message passing mechanism to update the feature representation of each node. This step enables feature points to use information from their neighborhood for more accurate matching.

[0086] D. Matching inference:

[0087] The matching layer of the SuperGlue model uses bilinear interpolation and attention mechanism to calculate the matching probability between feature points on the updated features to obtain the third matching key point pair KeyptSA and the fourth matching key point pair KeyptSB.

[0088] In this embodiment, the role of the LoFTR model and the SuperGlue model is to process key point matching and extract matching key point pairs. According to the specific registration image pairs, any matching model can be replaced, such as the DKM model, Alike (Aliked Light Glue) model, MatchFormer model, RoMa model, SGM model and other deep learning models.

[0089] In this embodiment, each image matching key point corresponds to corresponding position information, and the position information is expressed as two-dimensional coordinate information. In some embodiments, the number of image matching key points is N; each image matching key point represents the image position information of the target area in the image of the co-visible area to be extracted; each image position information includes X-axis coordinate information and Y-axis coordinate information; the position information corresponding to the i-th image matching key point is expressed as (x 0i ,y 0i ), where x 0i Indicates the x-axis coordinate information corresponding to the i-th image matching key point, y 0i Represents the y-axis coordinate information corresponding to the i-th image matching key point, where i = 1, 2, 3, …, N.

[0090] For example, with N matching key point pairs as input, the coordinates of the i-th matching key point pair are (x 0i,y0i ), accordingly, the coordinates of the image mapping points on the optical image according to the matching key point pair are expressed as (x 1i ,y 1i ), where i=1,2,3,…,N.

[0091] Step 3: Merge and splice the first matching key point pair KeyptLA and the third matching key point pair KeyptSA to obtain the first matching point pair mkptA; project the second matching key point pair KeyptSB and the fourth matching key point pair KeyptSB to the position of the original image pair and then merge and splice them to obtain the second matching point pair mkptB.

[0092] The specific steps are as follows:

[0093] (1) Obtaining the scaling ratio: For each scaled image, calculate its scaling ratio relative to the original image.

[0094] For example, for a 282×282 image, the scaling ratio is 1.1. The extracted matching key point pair coordinates are set to (x1, y1), and are projected back to the original image using the following formula.

[0095]

[0096] Among them, scale is the scaling ratio.

[0097] (2) Transform the coordinates of the matching key point pairs to the original image positions according to the scaling ratio.

[0098] (3) The first matching key point pair KeyptLA and the third matching key point pair KeyptSA are merged and spliced ​​to obtain the first matching point pair mkptA; the second matching key point pair KeyptSB and the fourth matching key point pair KeyptSB are projected to the position of the original image pair and then merged and spliced ​​to obtain the second matching point pair mkptB.

[0099] Step 4: Clear the abnormal key points in the first matching point pair mkptA and the second matching point pair mkptB to obtain the final matching key points.

[0100] According to the DBSCAN clustering algorithm, the abnormal key points in the obtained matching point pairs mkptA and mkptB are cleared to obtain the matching key points corresponding to the SAR-optical image.

[0101] In this embodiment, the DBSCAN clustering algorithm is a typical density-based spatial clustering algorithm, which uses the concept of density-based clustering, that is, it requires that the number of objects (points or other spatial objects) contained in a certain area in the clustering space is not less than a given threshold. The significant advantages of the DBSCAN algorithm are fast clustering speed and the ability to effectively process noise points and discover spatial clusters of arbitrary shapes.

[0102] DBSCAN algorithm (Density-Based Spatial Clustering of Applications with Noise) is a density-based clustering algorithm. The specific execution process and steps are as follows:

[0103] (1) Parameter setting

[0104] A parameter settings follow the following guidelines:

[0105] a. Nature of the data: The distribution and density of keypoints in the feature space affect the selection of features. SAR and optical images usually exhibit different keypoint densities due to the differences in textures and features captured by their respective imaging modalities.

[0106] b. Image scale: The spatial resolution and scale of the image play an important role in determining the neighborhood parameter ε. A high-resolution image may require a smaller neighborhood parameter ε to capture details, while a low-resolution image may require a larger neighborhood parameter ε to include a wider range of features.

[0107] BSpecific settings

[0108] eps (neighborhood radius): defines the neighborhood range of a point.

[0109] minPts (minimum number of points): defines the minimum number of points for a cluster.

[0110] Definition of core point: If the neighborhood of a point contains at least minPts points, then the point is a core point.

[0111] Cluster expansion: Starting from a core point, add all points in its neighborhood to the same cluster, and recursively add points in the neighborhood of these points to the cluster until the cluster no longer expands.

[0112] Identification of outliers: Points that do not belong to any cluster are marked as outliers (noise points).

[0113] (2) Specific implementation steps

[0114] Import the DBSCAN library;

[0115] Perform DBSCAN clustering;

[0116] Eliminate outliers;

[0117] Get the final matching key points.

[0118] Step 5: Determine the co-visible area in the image based on the coordinates of the final matching key points

[0119] According to the coordinates of the matching key points after clustering processing, the upper left position and the lower right position of the matching key point coordinates in the SAR image and the optical image are searched respectively, and then the common visible area in the image is determined.

[0120] In this embodiment, the top left corner of the image is taken as the coordinate origin, the horizontal right direction is taken as the X axis, and the vertical downward direction is taken as the Y axis. By comparing the coordinates of the matching key points extracted from the image, the minimum X-axis coordinate and the minimum Y-axis coordinate of the key point coordinate axis are found; the maximum X-axis coordinate and the maximum Y-axis coordinate of the key point coordinate axis are found. Thus, the boundary of the visible area in the image is determined.

[0121] For example, when the DBSCAN clustering algorithm is used to cluster the key points in the matching key point set, the neighborhood parameter ε (neighborhood radius eps) and the minimum number of key points MinPts need to be given, and the point set consisting of all key points in the matching key point set mkpt = [mkptA, mkptB] is recorded as cluster C j If the key point k j The ε neighborhood of contains at least MinPts key points, namely:

[0122] N ε (k j )≥MinPts;

[0123] Then k j is a core point in the matching key point set mkpt.

[0124] Nε (k j )={k i ∈mkpt|dist(k i ,k j )≤ε};

[0125]

[0126] Among them, dist(k i ,k j ) represents the key point k i and k j The Euclidean distance between them.

[0127] After clustering, the matching key point set mkpt does not belong to any cluster C = {C1, C2···, C n The key points of the extracted key points are the abnormal key points. After removing the abnormal key points, the outer boundary formed by the remaining key points is the extracted common visible area.

[0128] Based on the same inventive concept, the present invention also provides a multimodal image common visible area extraction system, comprising:

[0129] The image pair acquisition module is used to acquire an original image pair consisting of a SAR image and an optical image to be registered.

[0130] The image pair processing module is used to perform multiple scale transformation processes with different magnifications on the original image pair to obtain multiple initial processed image pairs.

[0131] The first processing module is used to perform key point matching processing on the original image pair through the LoFTR model to obtain a first matching key point pair KeyptLA and a second matching key point pair KeyptLB.

[0132] The second processing module is used to perform key point matching processing on the original image pair and the initial processed image pair respectively through the SuperGlue model to obtain a third matching key point pair KeyptSA and a fourth matching key point pair KeyptSB.

[0133] The splicing module is used to merge and splice the first matching key point pair KeyptLA and the third matching key point pair KeyptSA to obtain the first matching point pair mkptA; and merge and splice the second matching key point pair KeyptSB and the fourth matching key point pair KeyptSB after projecting them to the position of the original image pair to obtain the second matching point pair mkptB.

[0134] The region extraction module is used to remove abnormal key points in the first matching point pair mkptA and the second matching point pair mkptB through the DBSCAN clustering algorithm to obtain the final matching key points; and determine the common visible area in the original image pair according to the final matching key points.

[0135] Based on the same inventive concept, the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the method for extracting the common visible area of ​​a multimodal image are implemented.

[0136] Based on the same inventive concept, the present invention also provides a computer-readable storage medium containing instructions, and a computer program is stored on the storage medium. For example, a memory containing instructions, the above instructions can be executed by a processor of a computer device to complete the above method. For example, a non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a tape, a floppy disk, and an optical data storage device. When the computer program is executed by the processor, the steps in the embodiment of the method for extracting the common visible area of ​​a multimodal image can be implemented. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0137] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for extracting a common visible region of a multimodal image, characterized in that: include: Obtaining an original image pair consisting of a SAR image and an optical image to be registered; Performing multiple scale transformations with different magnifications on the original image pairs to obtain multiple initial processed image pairs; The LoFTR model is used to perform key point matching on the original image pair to obtain a first matching key point pair KeyptLA and a second matching key point pair KeyptLB; The original image pair and the initial processed image pair are respectively matched with key points by the SuperGlue model to obtain a third matching key point pair KeyptSA and a fourth matching key point pair KeyptSB; The first matching key point pair KeyptLA and the third matching key point pair KeyptSA are merged and spliced ​​to obtain a first matching point pair mkptA; the second matching key point pair KeyptSB and the fourth matching key point pair KeyptSB are projected to the position of the original image pair and then merged and spliced ​​to obtain a second matching point pair mkptB; Abnormal key points in the first matching point pair mkptA and the second matching point pair mkptB are cleared by using the DBSCAN clustering algorithm to obtain final matching key points; and the common visible area in the original image pair is determined according to the final matching key points.

2. The method for extracting a common visible region of a multimodal image according to claim 1, characterized in that: The image matching key points represent the image position information of the target area in the image, including: The upper left corner vertex of the image is taken as the coordinate origin, the horizontal rightward direction in the image is taken as the X-axis, and the vertical downward direction in the image is taken as the Y-axis; the image position information includes the X-axis coordinate information and the Y-axis coordinate information; the position information corresponding to the i-th image matching key point is expressed as (x 0i ,y 0i ), where x 0i Indicates the x-axis coordinate information corresponding to the i-th image matching key point, y 0i Represents the y-axis coordinate information corresponding to the i-th image matching key point, i = 1, 2, 3, ..., N, N is the number of image matching key points.

3. The method for extracting a common visible region of a multimodal image according to claim 1, wherein: The key point matching process of the original image pair is performed by the LoFTR model, including: Establish pixel-level correspondence between image pairs to obtain joint information of the two images; Based on the joint information of the two images, the self-attention and cross-attention layers in the Transformer are used to generate feature descriptors, and dense matching features are generated in the images of the original image pair through the feature descriptors; The dense matching features are refined and classified to obtain the first matching key point pair KeyptLA and the second matching key point pair KeyptLB.

4. The method for extracting a common visible region of a multimodal image according to claim 1, wherein: The method of performing key point matching processing on the original image pair and the initial processed image pair respectively by using the SuperGlue model includes: Extract feature points and descriptors corresponding to the feature points from the original image pair, the first initial processed image pair, and the second initial processed image pair using the SuperPoint network of the SuperGlue model; The extracted feature points are constructed into a graph structure, where each feature point is a node of the graph; the edges between the nodes are connected according to the geometric relationship of the feature points and the similarity of the descriptors; Use graph neural network GNN to process the constructed graph structure and obtain updated features; The matching layer of the SuperGlue model uses bilinear interpolation and attention mechanism to calculate the matching probability between feature points on the updated features to obtain the third matching key point pair KeyptSA and the fourth matching key point pair KeyptSB.

5. The method for extracting a common visible region of a multimodal image according to claim 1, characterized in that: The obtaining of the first matching point pair mkptA and the second matching point pair mkptB specifically includes: For each scale-transformed initial processed image, the position coordinates of the initial processed image relative to the original image are determined according to the scaling ratio of the scale transformation. The calculation formula is: Where scale is the scaling ratio, x1 is the x-axis coordinate information corresponding to the image matching key point, and y1 is the y-axis coordinate information corresponding to the matching key point; The coordinates of the first matching key point pair KeyptLA and the third matching key point pair KeyptSA are transformed to the coordinates of the original image position according to the scaling ratio, and then merged and spliced ​​to obtain the first matching point pair mkptA; The coordinates of the second matching key point pair KeyptSB and the fourth matching key point pair KeyptSB are transformed to the coordinates of the original image position according to the scaling ratio, and then merged and spliced ​​to obtain the second matching point pair mkptB.

6. The method for extracting a common visible region of a multimodal image according to claim 1, characterized in that: The specific process of determining the common visible area in the image according to the final matching key points is as follows: Set the neighborhood parameter ε and the minimum number of key points MinPts, where the neighborhood parameter ε is the neighborhood radius eps, and the point set consisting of all key points in the matching key point set mkpt = [mkptA, mkptB] is recorded as cluster C j ; If the key point k j The ε neighborhood of contains at least MinPts key points, namely: N ε (k j )≥MinPts; Then k j is a core point in the matching key point set mkpt, where: N ε (k j )={k i ∈mkptdist(k i ,k j )≤ε}; Among them, dist(k i ,k j ) represents the key point k i and k j The Euclidean distance between The matching key point set mkpt does not belong to any cluster C = {C1, C2..., C n The key point of} is the abnormal key point; After removing abnormal key points, the outer boundary formed by the retained key points is the co-visible area in the original image pair.

7. A multimodal image common visible area extraction system, characterized in that: include: An image pair acquisition module is used to acquire an original image pair consisting of a SAR image and an optical image to be registered; An image pair processing module is used to perform multiple scale transformation processes with different magnifications on the original image pair to obtain multiple initial processed image pairs; The first processing module is used to perform key point matching processing on the original image pair through the LoFTR model to obtain a first matching key point pair KeyptLA and a second matching key point pair KeyptLB; The second processing module is used to perform key point matching processing on the original image pair and the initial processed image pair respectively through the SuperGlue model to obtain a third matching key point pair KeyptSA and a fourth matching key point pair KeyptSB; A splicing module is used to merge and splice the first matching key point pair KeyptLA and the third matching key point pair KeyptSA to obtain a first matching point pair mkptA; and merge and splice the second matching key point pair KeyptSB and the fourth matching key point pair KeyptSB after projecting them to the position of the original image pair to obtain a second matching point pair mkptB; The region extraction module is used to remove abnormal key points in the first matching point pair mkptA and the second matching point pair mkptB through the DBSCAN clustering algorithm to obtain the final matching key points; and determine the common visible area in the original image pair according to the final matching key points.

8. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the multimodal image co-visible area extraction method described in any one of claims 1 to 6 are implemented.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal image co-visible area extraction method according to any one of claims 1 to 6 are implemented.

Images