A phase consistency guided public feature map extraction algorithm

Abstract

The application discloses a common feature map extraction algorithm guided by phase congruency, and comprises the following steps: S1, extracting a phase congruency guided map of a multispectral image; S2, training the multispectral image; and S3, extracting a common feature map; the application extracts edge features of the multispectral image by using phase congruency, carries out binary processing on the edge feature map and other information, takes common edge features of the multispectral image, and retains a binary mask of common features of the two aligned images; the mask is used as a guided map for training of multi-modal common feature extraction, attention of a network to a common region is enhanced, and thus common information of different spectral bands can be effectively aligned.

Description

Technical Field

[0001] This invention relates to the field of common feature extraction technology, and specifically to a phase consistency-guided common feature map extraction algorithm. Background Technology

[0002] In traditional manual feature matching, the main steps include feature point extraction, principal orientation calculation, and descriptor extraction. These steps involve finding salient feature points in an image and extracting discriminative descriptors based on their local neighborhoods to perform image matching and registration. Feature point selection and descriptor extraction are core components of multimodal registration tasks. Traditional gradient-based feature descriptors (such as SIFT, SURF, and ORB) are widely used in image matching. These methods describe the local texture and shape around feature points by calculating gradient information in local regions and extracting principal orientations.

[0003] However, the effectiveness of these methods often depends on image contrast and lighting conditions, and they perform poorly for nonlinear radiation distortion (NRD) in multispectral images. In multispectral images, due to the nonlinear mapping relationship between the imaging mechanisms of different spectral bands, traditional gradient-based feature descriptors may not be able to effectively handle the large differences between different spectral bands. In particular, between visible light images and infrared images, the gradient response may be severely shifted due to the presence of radiation distortion, resulting in a significant decrease in similarity when matching descriptors.

[0004] To overcome this problem, some emerging image feature extraction methods have gradually gained attention in recent years, especially those based on phase congruency. Furthermore, with the rapid development of deep learning, contrastive learning, as a self-supervised learning method, has been introduced into the task of multimodal image registration.

[0005] The existing technology has the following problems:

[0006] (1) Using only self-supervised contrastive learning algorithms without label guidance, when the spectral differences of multispectral images are large (such as visible light and far infrared), the common features of the same scene in each spectral band are difficult to be accurately aligned by the network, resulting in poor common feature extraction.

[0007] (2) In multimodal images, there may be offset positions of edge or texture response of different spectral images, resulting in different common feature regions. The neural network cannot accurately focus on the common region to learn the common feature extraction map of the image.

[0008] (3) Edge-based registration only focuses on the contour information in the image, while ignoring rich features such as texture and grayscale details. In particular, it is difficult to provide enough registration cues in areas with rich texture or low contrast, resulting in insufficient accuracy of the final transformation estimation. Summary of the Invention

[0009] The present invention aims to provide a phase consistency-guided common feature map extraction algorithm to solve the above problems.

[0010] The technical solution of this invention is: a phase consistency-guided common feature map extraction algorithm, comprising:

[0011] S1, Phase consistency guided map extraction of multispectral images, that is, extracting stable edge features with spectral invariance from different spectral images, generating a binary mask map of common edge regions, providing clear and reliable supervision guidance for subsequent neural network training;

[0012] S2, Multispectral Image Training, which uses the extracted mask as a guide to train a neural network so that it can learn common feature representations in multispectral images for subsequent extraction of multispectral image common feature maps;

[0013] S3, common feature map extraction, uses a trained neural network to extract common feature maps from new multispectral image pairs that have not been used during training, providing a foundation for subsequent image registration, fusion and other tasks.

[0014] Preferably, S1 includes:

[0015] Guide map extraction based on multispectral image pairs, specifically including:

[0016] S11, Calculate the phase consistency response value for each pixel.

[0017]

[0018] S12, based on phase consistency, extract the maximum moment map of the edge of any two images with different spectra respectively;

[0019] S13, binarize the edge maximum moment map;

[0020] S14, take the intersection of the binarized images of the two edge images as the feature guide map (Mask).

[0021] Preferably, S2 includes:

[0022] S21, pairing multispectral image pairs (I 1 ,I 2 The neural network f is fed in batches.θ (e.g., in UNet) the output feature map pairs (F) are obtained 1 F 2 ):

[0023] F 1 =f θ (I 1 ), F 2 =f θ (I 2 )

[0024] S22, multiply the output feature image pair pixel-by-pixel with the generated feature guide map (mask) to obtain a new feature output image pair (y). 1 y 2 ):

[0025] y 1 =F 1 ⊙Mask,y 2 =F 2 ⊙Malk

[0026] S23, the new feature output map pair represents the common features between the input multispectral image pairs, and the new feature output map pair (y 1 y 2 The InfoNCE Loss function is used to learn and obtain a common feature map for image edge enhancement. Simultaneously, a background suppression method is employed to treat the non-common parts of the feature map as background for suppression. The loss function is expressed as follows:

[0027]

[0028] Where h(y) 1 ,y 2 ) is the similarity function, and τ is the temperature parameter.

[0029] Preferably, in S3,

[0030] After training the common feature map extraction network, the multispectral images of the test set are input into the network, and the output image feature map is the common feature map of the input multispectral image.

[0031] The test set refers to data that differs from the data used during training.

[0032] Preferably, in S12,

[0033]

[0034] in,

[0035] In the above formula, A n(x,y) represents the magnitude response of the image in each direction, φ n (x,y) represents the phase response of the image in each direction.

[0036] Preferably, in S13, the binarization process is as follows:

[0037]

[0038] Where τ represents the binarization threshold;

[0039] Obtain the binary edge map B of the visible light image. vis (x, y) and the edge binary map B of the infrared image ir (x, y).

[0040] Preferably, in S14,

[0041] The intersection of the two binarized edge images is used to obtain a common feature guidance map;

[0042] G(x, y) = B vis (x, y) ∩ B ir (x, y) = B vis (x, y)·B ir (x, y)

[0043] The intersection is calculated by dot product to remove edges that are significant only in a single spectral band, leaving only the regions that respond to both spectral bands. These regions serve as the common areas that the subsequent network focuses on. G(x,y) is used as the common feature guide map to guide the neural network in learning cross-spectral features.

[0044] Preferably, the multispectral image pair includes a visible-infrared image pair.

[0045] The beneficial effects of this invention are as follows:

[0046] 1. It solves the problem that when using only self-supervised contrastive learning algorithms without label guidance, the common feature extraction effect is poor if the spectral distance of the multispectral images is large during the training process.

[0047] 2. Phase congruency is used to extract edge features from multispectral images. After binarizing the edge feature images and other information, the common edge features of the multispectral images are extracted. The binarized mask of the common features of the two aligned images is retained. This mask is used as a guide map for training multimodal common feature extraction, which enhances the network’s attention to common regions and thus effectively aligns the common information of different spectral bands. Attached Figure Description

[0048] Figure 1A schematic flowchart of a phase consistency-guided common feature map extraction algorithm provided in an embodiment of the present invention;

[0049] Figure 2 The infrared image in step S1 provided in this embodiment of the invention;

[0050] Figure 3 The visible light image in step S1 provided in this embodiment of the invention;

[0051] Figure 4 The feature consistency mask extracted from the infrared image in step S1 of this embodiment of the invention;

[0052] Figure 5 The phase-consistency mask extracted from the visible light image in step S1 of this embodiment of the invention;

[0053] Figure 6 The common image guiding mask in step S1 provided in the embodiment of the present invention. Detailed Implementation

[0054] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. The embodiments of the present invention are not limited thereto.

[0055] Two-dimensional phase congruency is a method for feature detection that determines the saliency of features based on local structural information in an image.

[0056] Contrastive learning: A self-supervised learning method that learns representations by comparing different views or variations of data, such that similar samples are closer together in the representation space and dissimilar samples are further apart.

[0057] Example 1

[0058] like Figure 1 As shown, a phase consistency-guided common feature map extraction algorithm includes:

[0059] S1, Phase consistency guided map extraction of multispectral images, that is, extracting stable edge features with spectral invariance from different spectral images, generating a binary mask map of common edge regions, providing clear and reliable supervision guidance for subsequent neural network training;

[0060] S2, Multispectral Image Training, which uses the extracted mask as a guide to train a neural network so that it can learn common feature representations in multispectral images for subsequent extraction of multispectral image common feature maps;

[0061] S3, common feature map extraction, uses a trained neural network to extract common feature maps from new multispectral image pairs that have not been used during training, providing a foundation for subsequent image registration, fusion and other tasks.

[0062] S1 includes:

[0063] Guide map extraction based on multispectral image pairs, specifically including:

[0064] S11, Calculate the phase consistency response value for each pixel.

[0065]

[0066] S12, based on phase consistency, extract the maximum moment map of the edge of any two images with different spectra respectively;

[0067] S13, binarize the edge maximum moment map;

[0068] S14, take the intersection of the binarized images of the two edge images as the feature guide map (Mask).

[0069] In S12,

[0070]

[0071] in,

[0072] In the above formula, A n (x,y) represents the magnitude response of the image in each direction, φ n (x, y) represents the phase response of the image in each direction.

[0073] In S13, the binarization process is as follows:

[0074]

[0075] Where τ represents the binarization threshold;

[0076] Obtain the edge binary map B of the visible light image. vis (x, y) and the edge binary map B of the infrared image ir (x, y).

[0077] In S14,

[0078] The intersection of the two binarized edge images is used to obtain a common feature guidance map;

[0079] G(x, y) = B vis (x, y) ∩ B ir (x, y) = B vis(x, y)·B ir (x, y)

[0080] The intersection is calculated by dot product to remove edges that are significant only in a single spectral band, leaving only the regions that respond to both spectral bands. These regions serve as the common areas that the subsequent network focuses on. G(x,y) is used as the common feature guide map to guide the neural network in learning cross-spectral features.

[0081] Multispectral image pairs include visible-infrared image pairs.

[0082] Among them, such as Figure 2 and Figure 3 As shown, this is an aligned pair of infrared and visible light images used to train the network.

[0083] like Figure 4 and Figure 5 The image shows the corresponding usage described above. Figure 2 and Figure 3 Edge maximum moments extracted based on phase consistency Figure 2 The mask after valueization.

[0084] like Figure 6 As shown, it is based on Figure 4 Figure 5 The final common feature mask is generated by taking the intersection.

[0085] S2 includes:

[0086] S21, pairing multispectral image pairs (I 1 I 2 The neural network f is fed in batches. θ (e.g., in UNet) the output feature map pairs (F) are obtained 1 F 2 ):

[0087] F 1 =f θ (I 1 ), F 2 =f θ (I 2 )

[0088] S22, multiply the output feature image pair pixel-by-pixel with the generated feature guide map (mask) to obtain a new feature output image pair (y). 1 y 2 ):

[0089] y 1 =F 1 ⊙Mask, y 2 =F 2 ⊙Mask

[0090] S23, the new feature output map pair represents the common features between the input multispectral image pairs, and the new feature output map pair (y 1 y 2 The InfoNCE Loss function is used to learn and obtain a common feature map for image edge enhancement. Simultaneously, a background suppression method is employed to treat the non-common parts of the feature map as background for suppression. The loss function is expressed as follows:

[0091]

[0092] Where h(y) 1 y 2 ) is a similarity function (such as MSE or cosine similarity), and τ is a temperature parameter.

[0093] In S23, the background suppression method uses a feature guide map that is a matrix containing only 0s and 1s, with the feature part being 1s and the non-feature part being 0s. Therefore, after the feature guide map is multiplied by the output feature pair, the background part is reduced to zero. This process is called background suppression.

[0094] In S3,

[0095] After training the common feature map extraction network, the multispectral images of the test set are input into the network, and the output image feature map is the common feature map of the input multispectral image.

[0096] The test set refers to data that differs from the data used during training.

[0097] In S3, the trained network can be used directly. For example, a network trained with infrared and visible light can directly extract the common feature maps of infrared and visible light image pairs, and then it can be used for normal subsequent registration.

[0098] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A phase-consistency-guided common feature map extraction algorithm, characterized in that, include: S1, Phase consistency guided map extraction of multispectral images, that is, extracting stable edge features with spectral invariance from different spectral images, generating a binary mask map of common edge regions, providing clear and reliable supervision guidance for subsequent neural network training; S2, Multispectral Image Training, which uses the extracted mask as a guide to train a neural network so that it can learn common feature representations in multispectral images for subsequent extraction of multispectral image common feature maps; S3, common feature map extraction, is to use a trained neural network to extract common feature maps from new multispectral image pairs that have not been used during training, thus providing a foundation for subsequent image registration and fusion tasks; S1 includes: Guide map extraction based on multispectral image pairs, specifically including: S11, Calculate the phase consistency response value for each pixel. ； S12, based on phase consistency, extract the maximum moment map of the edge of any two images with different spectra respectively; S13, binarize the edge maximum moment map; S14, take the intersection of the binarized images of the two edge images as the feature guide map; S2 includes: S21, pairing multispectral images Batch input into neural network In the process, the output feature map pairs are obtained. : S22, multiply the output feature image pairs pixel-by-pixel with the generated feature guidance map to obtain new feature output image pairs. : S23, the new feature output map pairs represent the common features between the input multispectral image pairs, and the new feature output map pairs A loss function is used for learning to obtain a common feature map for image edge enhancement. Simultaneously, a background suppression method is employed to treat the non-common parts of the feature map as background for suppression. The loss function is expressed as follows: in, Let τ be a similarity function, and τ be a temperature parameter.

2. The phase consistency-guided common feature map extraction algorithm according to claim 1, characterized in that, In S3, After training the common feature map extraction network, the multispectral images of the test set are input into the network, and the output image feature map is the common feature map of the input multispectral image. The test set refers to data that differs from the data used during training.

3. The phase consistency-guided common feature map extraction algorithm according to claim 1, characterized in that, In S12, in, In the above formula, This represents the magnitude response of the image in various directions. This represents the phase response of the image in each direction.

4. The phase consistency-guided common feature map extraction algorithm according to claim 1, characterized in that, In S13, the binarization process is as follows: in Indicates the binarization threshold; Obtain the edge binary map of the visible light image. Binary edge map of infrared image .

5. The phase consistency-guided common feature map extraction algorithm according to claim 1, characterized in that, In S14, The intersection of the two binarized edge images is used to obtain a common feature guidance map; The intersection is calculated using dot product to remove edges that are significant only in a single spectral band, retaining only regions where both spectral bands respond simultaneously. These regions serve as the primary focus of the subsequent network. As a common feature guide map, it can guide the neural network to learn cross-spectral features.

6. The phase-consistency-guided common feature map extraction algorithm according to claim 1, characterized in that, Multispectral image pairs include visible-infrared image pairs.

Images