A polarized image enhancement method based on feature point and signal-to-noise ratio double enhancement

Abstract

The application provides a polarized image enhancement method based on feature point and signal-to-noise ratio double enhancement, and belongs to the technical field of polarized vision detection, and comprises the following steps: obtaining two types of image information by using a polarized camera: a polarized image and a visible light image; constructing a polarized image signal-to-noise ratio enhancement network, separating noise from an original scene by learning the correlation between multi-angle polarized information, and realizing signal-to-noise ratio enhancement; constructing a polarized image feature point enhancement network, improving the number and quality of detected feature points by learning the feature fusion relationship between a total light intensity image and a polarized degree image, and realizing feature point enhancement; finally, the outputs of the two networks are crossed as inputs to form a double-loop closed-loop learning network. The method constructs a polarized image signal-to-noise ratio enhancement network and a feature point enhancement network, proposes an integrated enhancement method of signal-to-noise ratio and feature point loop empowerment, and can improve the precision and environmental adaptability of a polarized vision detection system in an interference environment.

Description

Technical Field

[0001] This invention belongs to the field of polarization vision detection technology, and particularly relates to a polarization image enhancement method based on dual enhancement of feature points and signal-to-noise ratio. Background Technology

[0002] Navigation and detection systems provide spatial motion and target feature information, serving as core components of unmanned systems. Research has found that organisms in nature possess remarkable navigation and detection capabilities, utilizing polarization vision to perceive environmental and target information. These systems offer advantages such as high autonomy, high target recognition accuracy, and rich image information, providing a technological means for autonomous navigation and detection in complex and disruptive environments. A common technology in polarization vision and detection is the processing and extraction of information from polarization images. Therefore, improving the accuracy of information extraction from polarization images becomes a crucial key to polarization vision-based navigation and detection.

[0003] In recent years, many research institutions have conducted extensive research on polarization image processing and information extraction methods. For example, existing technologies have proposed using convolutional feature enhancement modules, which enhance image feature representation capabilities and effectively capture local details and edge information by combining central difference convolution with activation followed by pooling. The invention application "Infrared Image Dynamic Range Adaptive Enhancement Method and System Based on Signal-to-Noise Ratio Sensing" (application number: CN202410359705.6) utilizes an encoder to extract features from the original image and calculate the signal-to-noise ratio distribution map of the original infrared image. The feature map is divided into high signal-to-noise ratio regions and low signal-to-noise ratio regions and then fused, which can enhance the image signal-to-noise ratio and recover more image detail information. Existing technologies also propose designing texture enhancement preprocessing methods to analyze the statistical information of image texture features, thereby improving the expression of image texture information. Furthermore, ORB algorithms are applied to the texture and color enhanced images to extract feature points, resulting in more prominent and diverse texture feature points. In the invention application "A Visual SLAM Method and Device Based on Feature Enhancement Network" (application number: CN202311404737.5), the geometric information and descriptors of the extracted feature points are used as network inputs. Enhanced feature descriptors are obtained through the feature enhancement network, and feature point matching is performed using these enhanced descriptors, improving the detectability of feature points. In the invention application "A Combined Navigation Method for Unmanned Aerial Vehicles with Analogous Complex Eye Polarization Vision in Low-Light Environments" (application number: CN202310474059.3), addressing the problem of indistinct image feature points in low-light scenes, mean filtering and opening operations are performed on the polarization degree image and polarization angle image, followed by weighted fusion to enhance image features, thereby increasing the number of detected image feature points in low-light scenes.

[0004] The aforementioned existing technologies only target the enhancement of feature points or signal-to-noise ratio, failing to fully combine the advantages of both. Furthermore, they do not effectively utilize the polarization information of the image, resulting in limited accuracy in feature point and signal-to-noise ratio extraction under complex environments such as clouds, fog, and uneven lighting. Summary of the Invention

[0005] To address the aforementioned problems and overcome the shortcomings of existing technologies, this invention proposes a polarization image enhancement method based on dual enhancement of feature points and signal-to-noise ratio (SNR). It utilizes a polarization camera to obtain two types of image information: polarization images and visible light images. A polarization image SNR enhancement network is constructed, which learns the correlation between multi-angle polarization information to separate noise from the original scene, thus enhancing the SNR. A polarization image feature point enhancement network is also constructed, which learns the feature fusion relationship between the total light intensity image and the polarization degree image to improve the quantity and quality of detected feature points, thus enhancing the feature point. Finally, the outputs of the two networks are cross-referenced as inputs to form a dual-loop closed-loop learning network, achieving integrated loop-based enhancement of both SNR and feature points.

[0006] The technical solution of this invention is: a polarization image enhancement method based on dual enhancement of feature points and signal-to-noise ratio, the implementation steps of which are as follows:

[0007] Step (1): Obtain two types of image information using a polarization camera: polarization image and visible light image; the polarization image includes the degree of polarization. Images and polarized light intensity images; visible light images include total light intensity images;

[0008] Step (2): Construct a polarization image signal-to-noise ratio enhancement network; The input of the signal-to-noise ratio enhancement network is a polarization light intensity image. The clear image is used as the ground truth constraint. The correlation between multi-angle polarization information is learned by using image sequences containing polarization information. The polarization light intensity image is reconstructed, and the background noise containing polarization characteristics of the scattered light is separated from the original scene.

[0009] Step (3): Construct a polarization image feature point enhancement network; the input to the feature point enhancement network is the total light intensity image and the degree of polarization. The image is subjected to dual constraints on the network output through a feature point loss function and a multi-scale image structure similarity loss function. The feature point loss function is used to fuse the original scene information and polarization degree in the total intensity image. Edge feature information of the image and a multi-scale image structure similarity loss function are used to improve the quantity and quality of feature point detection, thereby learning the total light intensity image and polarization degree. Feature fusion relationships between images;

[0010] Step (4): The outputs of the signal-to-noise ratio enhancement network and the feature point enhancement network are cross-inputted into the two networks to form a dual-loop closed-loop learning network for signal-to-noise ratio and feature points, generating a dual-enhanced image for feature points and signal-to-noise ratio.

[0011] The advantages of this invention compared to the prior art are:

[0012] This invention proposes for the first time a polarization image enhancement method based on dual enhancement of feature points and signal-to-noise ratio (SNR). Based on polarization image information, a polarization image SNR enhancement network and a feature point enhancement network are constructed. Furthermore, combining the input and output information of the two networks, an integrated enhancement method of polarization SNR / feature point loop closure empowerment is proposed, improving the detection accuracy of motion and target information under interference environments such as clouds, fog, and uneven lighting, providing technical support for high-precision polarization vision detection and navigation. Attached Figure Description

[0013] Figure 1 This is a flowchart of a polarization image enhancement method based on dual enhancement of feature points and signal-to-noise ratio according to the present invention.

[0014] Figure 2 The original image after SURF feature point detection;

[0015] Figure 3 This is a double-enhanced image after SURF feature point detection. Detailed Implementation

[0016] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0017] This invention proposes a polarization image enhancement method based on dual enhancement of feature points and signal-to-noise ratio, such as... Figure 1 As shown, the specific implementation steps of the present invention are as follows:

[0018] Step 1: Using a polarization camera and calculating based on Stokes vectors, obtain two types of image information: polarization image and visible light image; the polarization image includes the degree of polarization. image Polarized light intensity image Visible light images include total light intensity images. ;

[0019] The Stokes vector consists of four components and can be represented as:

[0020] ,

[0021] The four components of the Stokes vector can be determined by the following formula relating to the intensity of polarized light, thus enabling a description of the polarization state of light:

[0022] ,

[0023] in, , , and These represent four images of polarized light intensity along different directions. This represents an image of right-handed circularly polarized light intensity. This represents an image of the intensity of left-handed circularly polarized light. Indicates total light intensity; This indicates the difference in light intensity between the horizontal and vertical polarization directions. This indicates the difference in light intensity between polarization directions of 45° and 135°. This represents the circularly polarized component of light, which can be considered as 0 for linearly polarized light.

[0024] The degree of polarization of the scene can be calculated using the Stokes vector. :

[0025] ,

[0026] Step 2: Construct a polarization image signal-to-noise ratio (SNR) enhancement network. The input to the polarization image SNR enhancement network is a polarized light intensity image. Using a clear image with high SNR as the ground truth constraint, the network learns the correlation between multi-angle polarization information using image sequences containing polarization information, reconstructs the polarized light intensity image, and separates background noise containing polarization characteristics, such as scattered light, from the original scene.

[0027] The polarization image signal-to-noise ratio (SNR) enhancement network is primarily an encoder-decoder structure, with an embedded SNR enhancement module. The encoder extracts multi-level features from the input image, while the decoder recovers details by progressively reconstructing the image. The SNR enhancement module first divides the input feature map into high-frequency and low-frequency components using multi-scale convolution, extracting spatial polarization features at different scales. Subsequently, it weights the importance of different channels and spatial distributions of the image through channel attention and spatial attention mechanisms. Then, multi-scale convolutional layers learn the correlation between multi-angle polarization information, separating background noise containing polarization characteristics from the original scene, enhancing the feature representation of the signal portion, and obtaining the image with enhanced SNR. The channel attention mechanism can be represented as:

[0028] ,

[0029] in, and These represent the input and output of the channel attention mechanism, respectively. and These represent the height and width of the image, respectively. Represents the Sigmoid activation function. Represents the Hadamard product. This represents a mapping operation consisting of two fully connected layers in a channel attention mechanism network. represent No. Line number Liede The values ​​of each channel. The spatial attention mechanism can be represented as:

[0030] ,

[0031] in, and These represent the input and output of the spatial attention mechanism, respectively. Represents the Sigmoid activation function. Represents the Hadamard product. Represents the convolution operation. This represents the max pooling operation.

[0032] The polarization image signal-to-noise ratio (SNR) enhancement network employs supervised learning. The network ensures effective SNR enhancement by comparing the enhanced image with the labeled ground truth image (high SNR). The loss function for the polarization image SNR enhancement network is constructed as follows:

[0033] ,

[0034] in, The loss function representing the polarization image signal-to-noise ratio enhancement network. This represents the input polarized light intensity image. A ground truth image representing a high signal-to-noise ratio. The image with enhanced signal-to-noise ratio (SNR) output by the polarization image SNR enhancement network represents the image with enhanced SNR.

[0035] Step 3: Construct a polarization image feature point enhancement network. The network input consists of the total light intensity image and the degree of polarization. The image is subjected to dual constraints on the network output through a feature point loss function and a multi-scale image structure similarity loss function. The feature point loss function is used to fuse the original scene information and polarization degree in the total intensity image. Image edge feature information and multi-scale structural similarity loss are used to further improve the quantity and quality of feature point detection, thereby learning the total light intensity image and polarization degree. The feature fusion relationship between images improves the quantity and quality of feature points in the network output image.

[0036] The polarization image feature point enhancement network is primarily an encoder-decoder structure, with an embedded feature point enhancement module. The encoder extracts multi-level features from the input image, while the decoder recovers details by progressively reconstructing the image. The feature point enhancement module first extracts spatial features of the image at multiple scales through a series of convolutional layers, capturing feature points at different scales to ensure sensitivity to both large-scale and local feature points. Subsequently, under the constraint of the feature point loss function, it fully integrates the original scene information from the total intensity image and the key edge contour information from the polarization degree image through operations such as convolution and self-attention mechanisms. This enhances the detectability and robustness of feature points without compromising the original scene information. Simultaneously, a multi-scale image structure similarity loss function is incorporated to improve the quality of detected feature points. The self-attention mechanism can be represented as:

[0037] ,

[0038] in, and They represent input and output respectively. Represents the dimension of the input. The normalized exponential function can be expressed as:

[0039] ,

[0040] The polarization image feature point enhancement network employs unsupervised learning. To ensure that the enhanced image output by the network maintains structural consistency with the original image at different resolutions and levels of detail, and to ensure that the network can effectively capture significant features in the polarization image while preserving the original scene information, and to ensure that the enhanced image can detect more effective feature points, the loss function of the polarization image feature point enhancement network is constructed as follows:

[0041] ,

[0042] in, The loss function representing the polarization image feature point enhancement network. and These are weighting coefficients. The multi-scale weighted structural similarity loss function can be expressed as:

[0043] ,

[0044] in, These are the weighting coefficients. Indicates window size. Representative in window The image above and The similarity between them can be expressed as:

[0045] ,

[0046] in, Indicates the image is in the window The inner part, , These represent the mean and variance of the image, respectively. and It is a constant.

[0047] The polarization intensity loss function can be expressed as:

[0048] ,

[0049] in, and These represent the height and width of the image, respectively. Representing a matrix Norm.

[0050] The feature point loss function can be expressed as:

[0051] ,

[0052] in, This indicates the number of feature points detected in the image using the SURF feature point detection method.

[0053] Step 4: The outputs of the signal-to-noise ratio (SNR) enhancement network and the feature point enhancement network are cross-inputted into the two networks to form a dual-loop closed-loop learning network for SNR and feature points, achieving integrated loop-based empowerment of polarization SNR and feature points. Finally, a dual-enhanced image of feature points and SNR is generated, providing high-precision target and motion perception information for polarization detection and navigation.

[0054] Example:

[0055] The example uses a biomimetic polarization vision / inertial navigation detection system. Considering that interference environments such as clouds, dust, and uneven lighting can affect the system's detection accuracy of motion and target information, it is necessary to design a polarization image enhancement method based on feature point / signal-to-noise ratio enhancement to provide technical support for high-precision polarization vision navigation and detection.

[0056] The specific implementation process includes:

[0057] Step 1: Use a polarization camera to obtain two types of image information: polarization image and visible light image; the polarization image includes polarization degree image and polarization intensity image, and the visible light image includes total intensity image;

[0058] Step 2: Construct a polarization image signal-to-noise ratio (SNR) enhancement network; The input of the SNR enhancement network is a polarization light intensity image. The clear image is used as the ground truth constraint. The correlation between multi-angle polarization information is learned by using image sequences containing polarization information to reconstruct the polarization light intensity image and separate the background noise containing polarization characteristics of the scattered light from the original scene.

[0059] Step 3: Construct a polarization image feature point enhancement network. The input to the feature point enhancement network is the total intensity image and the polarization degree image. The network output is subject to dual constraints through a feature point loss function and a multi-scale image structure similarity loss function. The feature point loss function is used to fuse the original scene information in the total intensity image and the edge feature information in the polarization degree image. The multi-scale image structure similarity loss function is used to improve the number and quality of feature point detection, thereby learning the feature fusion relationship between the total intensity image and the polarization degree image.

[0060] Finally, the outputs of the signal-to-noise ratio enhancement network and the feature point enhancement network are cross-inputted into the two networks to form a dual-loop closed-loop learning network for signal-to-noise ratio and feature points, generating a dual-enhanced image for both feature points and signal-to-noise ratio.

[0061] To demonstrate the effectiveness of this method in improving system performance under environmental interference, the Cityscapes dataset was selected for simulation verification. Polarization and visible light images were calculated using the designed method. A polarization image signal-to-noise ratio (SNR) enhancement network and a feature point enhancement network were then constructed. The polarization and visible light images were input into the networks, and the network outputs were cross-inputted into the two networks, forming a dual-loop closed-loop learning network. This achieves integrated loop-based enhancement of polarization SNR and feature points. The original image after SURF feature point detection is shown below. Figure 2 As shown, the double-enhanced image after SURF feature point detection is as follows: Figure 3 As shown.

[0062] Image analysis revealed that the original polarization image was affected by cloud cover and dust, significantly impacting the system's accuracy in detecting motion and target information. However, after performing feature point / signal-to-noise ratio (SNR) enhancement, both the SNR and the number of detectable feature points were significantly improved. Further analysis using image evaluation metrics to calculate the PNSR (Peak Signal-to-Noise Ratio) between the original polarization image, the enhanced image, and the ground truth image showed a PSNR of 17.49 for the original polarization image and 31.60 for the enhanced image, representing a 44.65% improvement. Using the SURF feature point detection method, the number of feature points in the original and enhanced images was found to be 145 for the original and 334 for the enhanced image, representing a 56.69% improvement. Therefore, the polarization image enhancement method based on feature point / SNR enhancement successfully enhanced both the feature points and the SNR of the polarization image.

[0063] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes will be obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A polarization image enhancement method based on feature point and signal-to-noise ratio double enhancement, characterized in that, The implementation steps are as follows: Step (1), obtaining two kinds of image information by using polarization camera: polarization image and visible light image; the polarization image includes polarization degree image and polarization light intensity image, the visible light image includes total light intensity image; Step (2), constructing a polarization image signal-to-noise ratio enhancement network; the input of the signal-to-noise ratio enhancement network is a polarized light intensity image, and a clear image is used as a true value constraint to learn the correlation between multi-angle polarization information by using an image sequence containing polarization information, to reconstruct the polarized light intensity image and separate the background noise containing polarization characteristics of scattered light from the original scene; Step (3), constructing a polarized image feature point enhancement network; the input of the feature point enhancement network is a total light intensity image and a polarization degree The network output is doubly constrained by a feature point loss function and a multi-scale image structural similarity loss function, wherein the feature point loss function is used for fusing original scene information in the total light intensity image and the polarization degree Edge feature information of the image, and the multi-scale image structural similarity loss function is used for improving the detection quantity and quality of the feature points, so as to learn the total light intensity image and the polarization degree Feature fusion relationship between the images; Step (4), the outputs of the signal-to-noise ratio enhancement network and the feature point enhancement network are cross-input to the two networks to form a signal-to-noise ratio and feature point double-loop closed-loop learning network to generate a feature point and signal-to-noise ratio double-enhanced image.

2. The polarization image enhancement method based on feature point and signal-to-noise ratio double enhancement according to claim 1, characterized in that: In step (1), two types of image information are obtained by using a polarization camera and according to Stokes vector calculation: a polarization image and a visible light image; the polarization image includes a polarization degree image and a polarization light intensity image ; the visible light image includes a total light intensity image ; Stokes vector comprises four components, denoted as: ， The four components of the Stokes vector are determined by the following formula in relation to the polarized light intensity, thereby realizing the description of the polarization state of light: ， wherein, , , and represent 4 images of intensity of polarized light in different directions; represents the right circular polarized light intensity image, represents the left circular polarized light intensity image; represents the total light intensity; represents the difference in intensity of light between horizontal and vertical polarization directions; represents the difference in intensity of light between 45° and 135° polarization directions; represents the circular polarization component of light, which is considered 0 for linearly polarized light; The polarization degree of the scene is calculated according to the Stokes vector solution : 。 3. The polarization image enhancement method based on feature point and signal-to-noise ratio double enhancement according to claim 2, characterized in that: In step (2), the polarization image signal-to-noise ratio enhancement network is an encoder-decoder structure, which internally embeds a signal-to-noise ratio enhancement module; the encoder is used to extract multi-level features of the input image, and the decoder restores the details by gradually reconstructing the image; the signal-to-noise ratio enhancement module first divides the input feature map into high-frequency and low-frequency parts through multi-scale convolution, and extracts the spatial polarization features of the image at different scales; Then, through the channel attention mechanism and the spatial attention mechanism, the importance of different channels and spatial distributions of the image is weighted, and then through the multi-scale convolution layer, the correlation between multi-angle polarization information is learned, the background noise containing polarization characteristics is separated from the original scene, and the signal-to-noise ratio enhanced image is obtained.

4. The polarization image enhancement method based on feature point and signal-to-noise ratio double enhancement according to claim 3, characterized in that: The channel attention mechanism is represented as: ， wherein, and represent the input image and the output image of the channel attention mechanism, respectively, and represent the height and the width of the image, respectively, represents the Sigmoid activation function, represents the Hadamard product, represents the mapping operation consisting of two fully connected layers in the channel attention mechanism, represents the value of the column of the channel.

5. The polarization image enhancement method based on feature point and signal-to-noise ratio double enhancement according to claim 3, characterized in that: The spatial attention mechanism is represented as: ， wherein, and represent the input image and the output image of the spatial attention mechanism, respectively, represents the Sigmoid activation function, represents the Hadamard product, represents the convolution operation, represents the max-pooling operation.

6. The polarization image enhancement method based on feature point and signal-to-noise ratio double enhancement according to claim 3, characterized in that: The polarization image signal-to-noise ratio enhancement network adopts supervised learning, and by comparing the enhanced image with the labeled high signal-to-noise ratio true value image, the loss function of the polarization image signal-to-noise ratio enhancement network is constructed as follows: ， wherein, a loss function representing the polarization image signal-to-noise ratio enhancement network, an input polarization intensity image, a high signal-to-noise ratio ground truth image, a signal-to-noise ratio enhanced image output by the polarization image signal-to-noise ratio enhancement network.

7. The polarization image enhancement method based on feature point and signal-to-noise ratio double enhancement according to claim 2, characterized in that: In the step (3), the polarized image feature point enhancement network is an encoder-decoder structure, and a feature point enhancement module is embedded in the encoder-decoder structure; the encoder is used for extracting multi-level features of an input image, the decoder is used for recovering details by gradually reconstructing the image, the feature point enhancement module is used for firstly extracting spatial features of the image at multiple scales through a series of convolution layers, and feature points of the image are captured at different scales, so that the extraction result is sensitive to large-scale and local feature points; and then, under the constraint of a feature point loss function, original scene information in a total light intensity image and key edge contour information in a polarization degree image are fully fused through convolution and self-attention mechanism operations, and a multi-scale image structural similarity loss function constraint is added.

8. The polarized image enhancement method based on feature point and signal-to-noise ratio double enhancement according to claim 7, wherein: The self-attention mechanism is represented as: ， wherein, and represent the input image and the output image of the self-attention mechanism, respectively, represents the dimension of the input, represents the normalized exponential function, denoted as: 。 9. The polarized image enhancement method based on feature point and signal-to-noise ratio double enhancement according to claim 7, wherein: The polarized image feature point enhancement network adopts unsupervised learning, and a loss function for constructing the polarized image feature point enhancement network is as follows: ， wherein, a loss function representing a polarized image feature point enhancement network, and is a weighting coefficient, a multi-scale weighted structural similarity loss function, a polarized light intensity loss function, a feature point loss function.

10. The polarized image enhancement method based on feature point and signal-to-noise ratio double enhancement according to claim 9, wherein: is represented by: ， wherein, is a weight coefficient, denotes a window scale, represents a similarity between the image and and is expressed as: ， wherein, represents the portion of the image within the window , , represent the image mean and variance, respectively, and are constants.

11. The polarized image enhancement method based on feature point and signal-to-noise ratio double enhancement according to claim 9, wherein: is represented as: ， wherein, and denote the height and width of the image, respectively, denotes the norm of a matrix.

12. The polarized image enhancement method based on feature point and signal-to-noise ratio double enhancement according to claim 9, wherein: is represented as: ， wherein, represents the number of feature points detected by the SURF feature point detection method for the image.

Images