Multi-scale target detection method based on spectrum polarization information fusion
By fusing spectral and polarization information, a pseudo-color fused image is generated and feature selection and fusion are performed, which solves the problem of insufficient feature richness in multi-scale target detection and improves the detection effect of small targets and occluded targets.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-21
- Publication Date
- 2026-04-03
AI Technical Summary
Existing target detection methods suffer from insufficient feature richness and poor performance in multi-scale detection, especially in the detection of small targets and occluded targets.
By combining spectral and polarization information, principal component analysis and Stokes' theorem are used for image fusion to generate a pseudo-color fused image. Multi-scale feature optimization and fusion are then performed through a feature selection network and a shallow feature fusion network to enhance target detection capabilities.
It effectively integrates multi-dimensional features, enhances target details and edge features, improves the ability to detect small targets, reduces interference from occluded targets, and improves the overall performance of target detection.
Smart Images
Figure CN121788979A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of target detection technology, specifically relating to a multi-scale target detection method based on spectral polarization information fusion. Background Technology
[0002] Spectral and polarization images, as powerful tools for target detection, can reveal material composition and surface features. Their combination forms spectral polarization imaging technology, providing four-dimensional information (spatial, spectral, polarization, and radiation information). This method deepens the understanding of the target, enhances the contrast with the background, and improves detection capabilities. Spectral polarization imaging has significant advantages in remote sensing, and the significant differences between spectral and polarization characteristics highlight its superiority over traditional methods.
[0003] By combining the advantages of polarization and hyperspectral imaging, this technology provides detailed information, amplifies the contrast between the target and the background, and extracts polarization spectral curves to enhance detection capabilities.
[0004] Object detection is one of the most important tasks in the field of computer vision. In recent years, with the rapid development of artificial intelligence theory, object detection methods based on convolutional neural networks (CNNs) have completely replaced traditional methods that rely on manually designed features for detection. Single-stage object detectors generally use feature layers with different resolutions in the feature extraction network to detect objects of different scales. Typically, the SSD network uses VGG-16 as the feature extraction network, which uses the feature map of Conv4_3 layers and a total of 6 feature maps generated by additional networks for detection. The basic theory of this multi-scale feature map-based object detection method is that feature maps of different levels have different receptive fields, and a feature map of a certain scale can only describe the target size corresponding to that scale. High-resolution feature maps have smaller receptive fields and are used to detect smaller targets. Conversely, low-resolution feature maps often have larger receptive fields and are used to detect larger targets. Feature maps of different scales work together to cover and describe targets of different sizes in the image. To make the features of each level of the multi-scale detector more expressive, the Adaptive Spatial Feature Fusion (ASFF) network uses an adaptive weighted fusion method to generate new level features from the features of each level. The feature fusion method of the ASFF network enables adaptive connections between features at different levels, enhancing feature consistency. The Neighborhood Feature Erasure and Transformation (NETNet) network erases features from deeper layers in shallow feature maps to enhance their expressiveness, thereby reducing interference and strengthening features. Summary of the Invention
[0005] The purpose of this invention is to provide a multi-scale target detection method based on spectral and polarization information fusion. This method enhances image information by fusing the spectral and polarization information of an image, that is, by capturing the spectral and polarization information reflected from the object surface. Then, it enhances the target recognition and tracking capabilities through a multi-scale feature selection and fusion target detection algorithm.
[0006] To achieve the above objectives, this invention provides a multi-scale target detection method based on spectral polarization information fusion, comprising the following steps:
[0007] S1. Acquire a multispectral polarization image of the reflection from the object's surface;
[0008] S2. For the obtained single-channel polarization image, spectral information fusion is performed using principal component analysis plus spectral energy weighting, and after processing based on Stokes' theorem, a pseudo-color fused image of the HSI three channels is obtained.
[0009] S3. The obtained pseudo-color fused image is subjected to multi-scale feature optimization and fusion through the structure of the feature selection network module FSN plus the shallow feature fusion network module LFFM, and finally the target detection image is obtained.
[0010] Furthermore, in step S1, four sets of single-channel polarization images are acquired.
[0011] Furthermore, step S2 specifically includes the following steps:
[0012] S21. Construct a spectral data matrix from the four sets of single-channel polarization images, reduce the dimensionality using principal component analysis (PCA), and select the first principal component image as the output feature.
[0013] S22. Based on the sum of squared gray values at each pixel location, calculate the spectral energy weight, and perform weighted processing on the first principal component image to obtain the weighted principal component image;
[0014] S23. Calculate the Stokes parameters and polarization parameters, and transform the weighted principal component image after weighting into a Stokes fusion image, a degree of polarization (DoLP) image, and an angle of polarization (AoP) image.
[0015] S24. Map the Stokes fusion image, degree of polarization (DoLP) image, and angle of polarization (AoP) image into the HSI color space to obtain a pseudo-color fusion image.
[0016] Furthermore, in step S21, the dimensionality reduction process using principal component analysis (PCA) specifically includes:
[0017] S211. Construct a data matrix A with an M×N structure;
[0018] S212. Calculate the covariance matrix C.
[0019] S213. Solve for the eigenvectors E of the covariance matrix C, and select the eigenvectors corresponding to the N largest eigenvalues to form a subset K of the transformation matrix W. k ;
[0020] S214. Output the dimensionality-reduced image. The dimensionality reduction process satisfies P. c =K k A(3), in formula (3), P c To capture the principal components with the most significant variance in the data, K k It is a matrix containing the first "k" eigenvectors.
[0021] Furthermore, the calculation of the Stokes parameters and polarization parameters in step S23 satisfies:
[0022]
[0023] In formulas (5), (6), and (7), S λ (x,y) is the Stokes vector at coordinates (x,y) in the image when the wavelength is λ. It is used to fully describe the polarization state of light and contains four components: S 0,λ (x,y),S 1,λ (x,y),S 2,λ (x,y),S 3,λ (x,y); I 0,λ (x,y), I 45,λ (x,y), I 90,λ (x,y), I 135,λ (x, y) represent the image grayscale values (light intensity) measured at wavelength λ with polarization directions of 0°, 45°, 90°, and 135°, respectively; I R,λ (x,y), I L,λ (x,y) represent the light intensities of right-handed and left-handed circularly polarized light at wavelength λ, respectively.
[0024] The generation of Stokes fused images satisfies the Stokes weighting formula:
[0025] in, The sum of squares of the gray values of the image corresponding to the polarization direction calculated in formula (13) is represented by S0, S1, and S2, which represent the Stokes vectors corresponding to the polarization direction.
[0026] Furthermore, the specific rules for mapping the fused image and polarization to the HSI color space obtained in step S24 are as follows:
[0027] Map the AoP image to the H channel;
[0028] Map the DoLP image to the S channel and then threshold it;
[0029] Map the spectral polarization fusion image onto the I channel.
[0030] Furthermore, the shallow feature fusion network module LFFM described in step S3 is composed of multiple feature fusion modules FFM;
[0031] The Feature Selection Network Module (FSN) consists of multiple FSM modules.
[0032] Furthermore, step S3 specifically includes the following steps:
[0033] S301. The pseudo-color fused image is subjected to a 3*3 convolution operation and then processed by the attention mechanism module ECA to obtain the first feature map F. q1 ;
[0034] S302, First Feature Map F q1 After a 4x3 convolution operation and processing by the attention mechanism module ECA, the second feature map F is obtained. q2 ;
[0035] S303, Second Feature Map F q2 After convolutional operation with conv7 and processing by the attention mechanism module ECA, the third feature map F is obtained. q3 ;
[0036] S304, Third Feature Map F q3 After an 8*2 convolution operation and processing by the attention mechanism module ECA, the fourth feature map F is obtained. q4 ;
[0037] S305, Fourth Feature Map F q4 After a 9*2 convolution operation and processing by the attention mechanism module ECA, the fifth feature map F is obtained. q5 ;
[0038] S306, Fifth Feature Map F q5 After a 10*2 convolution operation and processing by the attention mechanism module ECA, the sixth feature map F is obtained. q6 ;
[0039] S307, First Feature Map F q1 With the second feature map F q2 The first feature fusion map y1′ is obtained by fusion using the feature fusion module FFM.
[0040] S308, First feature fusion map y1′ and third feature map F q3 The second feature fusion map y2′ is obtained by fusion using the feature fusion module FFM.
[0041] S309, Second Feature Fusion Map y2′ and Fourth Feature Map F q4 The third feature fusion map y3′ is obtained by the feature fusion module FFM.
[0042] S310, the third feature fusion map y3′ and the fifth feature map F q5 The fourth feature fusion map y4′ is obtained by fusion using the feature fusion module FFM.
[0043] S311, Sixth Feature Map F q6 After being filtered directly by the Feature Selection Module (FSM), the first feature enhancement map Y1′ is obtained.
[0044] S312. After the fourth feature fusion image y4′ is filtered by the feature selection module FSM, the second feature enhancement image Y2′ is obtained through upsampling.
[0045] S313. After the third feature fusion image y3′ is filtered by the feature selection module FSM, the third feature enhancement image Y3′ is obtained through upsampling.
[0046] S314. After the second feature fusion image y2′ is filtered by the feature selection module FSM, the fourth feature enhancement image Y4′ is obtained through upsampling.
[0047] S315. After the first feature fusion image y1′ is filtered by the feature selection module FSM, the fifth feature enhancement image Y5′ is obtained through upsampling.
[0048] S316. After the first feature enhancement image Y1′ undergoes a convolution operation, it is fused with the second feature enhancement image Y2′ and then convolved. Subsequently, it is fused with the third feature enhancement image Y3′ and then convolved. After the third feature enhancement image Y4′ and then convolved, it is fused with the fifth feature enhancement image Y5′ and then convolved, finally obtaining the target detection result image.
[0049] Compared to traditional methods, this method has the following advantages:
[0050] 1. This algorithm effectively integrates multi-dimensional features from a scene into a single image, enhancing target details and enriching edge features. Then, a feature selection module separates irrelevant features, and a feature pyramid network is combined to form a feature selection network structure, reducing the interference of local features of targets at different scales in the feature map on the current scale features.
[0051] 2. By using a shallow feature fusion method, shallow features are gradually fused into deeper features, thus solving the problem of insufficient feature richness in the feature map;
[0052] 3. This method can detect more small targets in the image. For occluded targets, the feature selection module filters out the features of larger targets in the foreground, thereby enhancing their features and enabling them to be detected by the detection network.
[0053] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. Attached Figure Description
[0054] Figure 1 This is a schematic diagram of the structure of the present invention.
[0055] Figure 2 This is a flowchart of polarization fusion.
[0056] Figure 3 This is a schematic diagram of the FSM structure.
[0057] Figure 4 This is a diagram of the feature fusion network architecture.
[0058] Figure 5 It is the HSI representation of the spectral polarization image.
[0059] Figure 6 This is a diagram of the ECA structure.
[0060] Figure 7 This is a diagram of the FFM structure. Detailed Implementation
[0061] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the specific implementation methods, structural features and effects of the present invention are described in detail below with reference to the accompanying drawings and embodiments.
[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0063] In the description of this invention, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "aligned", "overlapping", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.
[0064] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature; in the description of this invention, unless otherwise stated, "a plurality of" means two or more.
[0065] Example 1
[0066] This embodiment provides, for example Figures 1 to 7 The method shown is a multi-scale target detection method based on spectral polarization information fusion, which includes the following steps:
[0067] S1. Acquire a multispectral polarization image of the reflection from the object's surface;
[0068] That is, firstly, the polarization image of the original image is obtained. Step S1 provides the necessary information basis for subsequent steps.
[0069] S2. For the obtained single-channel polarization image, spectral information fusion is performed using principal component analysis plus spectral energy weighting. After processing based on Stokes' theorem, it is mapped to the HSI space to obtain a pseudo-color fused image of the HSI three channels. Step S2 processes the original polarization image and uses the pseudo-color fusion method: HSI to map the image, finally obtaining a single three-channel image, i.e., a pseudo-color fused image.
[0070] Step S2 is the core step of this invention. The single-channel polarization image obtained in step S1 is reduced in dimensionality using the PCA method, namely principal component analysis, and the first principal component image is extracted as the output feature to retain the spectral features of the direction of maximum variance. On this basis, spectral energy weighting is performed, and the weights are allocated by the sum of squared pixel gray levels to further highlight the features of high-energy regions. Then, Stokes images and their derived DoLP and AoP images are generated based on the Stokes formula. These three images serve as the core carriers for polarization information processing and are mapped to the HSI color space through the HSI algorithm. This innovative method not only enhances the visual appeal of the image but also ensures a more comprehensive understanding of spectral polarization information, thereby achieving optimized fusion results. The HSI space contains more polarization information, which will allow for better contrast at the edges of objects.
[0071] S3. The obtained pseudo-color fused image is subjected to multi-scale feature optimization and fusion through the structure of the feature selection network module FSN plus the shallow feature fusion network module LFFM, and finally the target detection image is obtained.
[0072] Step S3 introduces the Feature Pyramid Network architecture FPN and adds the Feature Selection Module (FSM) to prevent the FPN network from introducing confusing features again when passing high-level semantic features to shallow feature maps, thus forming the Feature Selection Network structure FSN; that is, Feature Pyramid Network architecture FPN + Feature Selection Module FSM = Feature Selection Network structure FSN.
[0073] The pseudo-color fused image obtained in step S2, i.e. the image after fusing spectral and polarization information, is optimized by the feature selection network structure FSN. At the same time, the shallow feature fusion network module LFFM is used to fuse shallow features into deep feature maps, which effectively improves the detection performance, thereby achieving a significant improvement in the detection effect of some small targets and occluded targets.
[0074] Furthermore, in step S1, four sets of single-channel polarization images are acquired. Since the present invention processes the images based on the Stokes formula in subsequent steps, the Stokes parameters require light intensity data at four angles: 0°, 45°, 90°, and 135°. Therefore, four sets of single-channel images are acquired, corresponding to four sets of single-channel grayscale images at linear polarization angles of 0°, 45°, 90°, and 135°, respectively.
[0075] Furthermore, such as Figure 2 As shown, step S2 specifically includes the following steps:
[0076] S21. Construct a spectral data matrix from four sets of single-channel polarization images, reduce the dimensionality using principal component analysis (PCA), and select the first principal component image as the output feature. To address issues related to information redundancy and spectral replication, this invention introduces the PCA method, which reduces data complexity and improves computational efficiency by converting the original spectral data into principal components. PCA achieves effective feature extraction by selecting the direction with the maximum variance to retain the most representative spectral features.
[0077] S22. Based on the sum of squared gray values at each pixel location, calculate the spectral energy weight and perform weighted processing on the first principal component image to obtain a weighted principal component image. This calculation involves determining the sum of squared gray values at each pixel location in all images, and then assigning weights to each pixel by dividing the square of its gray value by the sum of squared gray values corresponding to that location, which can better highlight the features of high-energy regions.
[0078] The calculation of spectral energy weighting involves the sum of squared gray values at each pixel location in all images. Weights are assigned to each pixel by dividing the square of the gray value by the sum of squares corresponding to that location. The final fusion result is achieved by multiplying the pixel values in each image by the weights assigned to their corresponding locations. The mathematical expression for the pixel gray values of the fused image is shown in the following formula:
[0079] In formula (13), λ represents the wavelength of light, and a λ (i,j) represents the gray value of the pixel at position (i,j) with wavelength λ in the first principal component image, and the sum of the squares of the gray values.
[0080] It should be noted that PCA dimensionality reduction reduces the dimensionality of each of the four graphs separately, and subsequent operations are only performed after all four graphs have been reduced in dimensionality; energy weighting is to apply energy weighting to each of the four graphs after PCA. Since each graph has more than one graph, energy weighting is also a fusion, and after energy weighting, there are still four graphs.
[0081] S23. Based on Stokes' theorem, calculate the Stokes parameters and polarization parameters, and transform the four weighted principal component images after weighting into Stokes fusion images, degree of polarization (DoLP) images, and angle of polarization (AoP) images.
[0082] Step S23 calculates the four Stokes parameters, two polarization parameters, and the Stokes fusion vector F. stokes Then, based on these parameters, the polarization channels are fused and energy weighting of the Stokes vector is performed to finally obtain the Stokes fused image, the degree of polarization (DoLP) image, and the angle of polarization (AoP) image;
[0083] The process adjusts the relative importance of different Stokes parameters based on their energy contributions, aiming to enhance valuable information while suppressing noise. This method refines the representation of polarization information by prioritizing higher-energy components, emphasizing important polarization features and weakening irrelevant polarization features.
[0084] S24. Map the Stokes fusion image, DoLP image, and AoP image to the HSI color space to obtain a pseudo-color fusion image. For the Stokes fusion image, DoLP image, and AoP image obtained in step S23, a new pseudo-color fusion algorithm (HSI) is used to map them to the HSI color space to include more polarization information. This will allow for better contrast at the edges of objects. Step S24 finally compresses the multi-dimensional information into a three-channel image, that is, generates a three-channel pseudo-color fusion image to better solve the data redundancy problem.
[0085] Furthermore, in step S21, the dimensionality reduction process using principal component analysis (PCA) specifically includes: S211, constructing a data matrix A with an M×N structure; during the PCA process, a data matrix A with an M×N structure is constructed, where matrix A represents information captured from N pixels across M original spectral dimensions from four sets of single-channel polarization images in the original image, and each row of A corresponds to a different frequency band of the spectral data;
[0086] PCA is mathematically derived from equation P c =WA represents, where W is the transformation matrix, A is the original image matrix, and P... c A matrix representing the dimensionality-reduced image;
[0087] S212. Calculate the covariance matrix C; the covariance matrix C is calculated as follows: In formula (1), A is the original M×N data matrix. T It is the transpose of A, and the denominator N-1 is used to make an unbiased estimate of the covariance. By calculating the covariance matrix, we can analyze the eigenvalues λ and eigenvectors E of C, thereby determining which directions (band combinations) in the data contain the most information (maximum variance), providing a mathematical basis for subsequent dimensionality reduction.
[0088] S213. Calculate the eigenvectors E of the covariance matrix C, and select the eigenvectors corresponding to the k largest eigenvalues to form a subset K of the transformation matrix W. k Step S2 requires obtaining the final dimensionality-reduced image P. c Therefore, based on the covariance matrix C obtained in step S212, step S213 calculates the eigenvector E and eigenvalue λ of matrix C, thereby obtaining the transformation matrix W;
[0089] The transformation matrix W is calculated by solving the eigenvalue problem: (λI-C)E = 0, where λ is the eigenvalue, I is the identity matrix used to ensure dimensionality consistency in matrix operations, and C is the covariance matrix of the input image. In Principal Component Analysis (PCA), the core role of the eigenvalue problem is to extract the most representative directions (principal components) in the data, and these directions directly determine the transformation matrix W. The eigenvector E is a unit vector representing the direction with the greatest variation in the data (principal component direction); the corresponding eigenvalue λ represents the magnitude of variation (variance) in that direction. The transformation matrix W is composed of eigenvectors (usually the eigenvectors corresponding to the k largest eigenvalues), and its function is to project the original data onto these principal component directions, achieving dimensionality reduction while retaining key information.
[0090] For (λI-C)E = 0, if vector E is an eigenvector of covariance matrix C, then the product of C and E is equal to the product of λ and E, i.e., CE = λE. This shows that when data is projected along the direction of eigenvector E, the variance is quantized by the eigenvalue λ, and the direction remains unchanged. By solving this equation, the most important direction of variation in the data can be obtained, providing a basis for dimensionality reduction.
[0091] The eigenvalues λ can be arranged in descending order: λ1≥λ2≥…≥λ m The eigenvectors E satisfy the orthogonality condition to facilitate the formation of the transformation matrix W, where each column represents an eigenvector; the transformation matrix W is the inverse of the eigenvector matrix E: W = E -1 (2); In formula (2), E represents the eigenvector matrix of the covariance matrix C. The eigenvector E is obtained from the covariance matrix C of the data. Each eigenvector E represents a major direction of change in the data (i.e., the principal component direction), and different eigenvectors satisfy orthogonality; W represents the transformation matrix, which is used to map the original high-dimensional data to the low-dimensional principal component space; Formula (2) is the key step in principal component analysis (PCA) to achieve data dimensionality reduction. Its function is to achieve the mapping from the original high-dimensional space to the principal component space through the inverse transformation of the eigenvectors, thereby extracting the most representative spectral features.
[0092] S214. Output the dimensionality-reduced image. The dimensionality reduction process satisfies P. c =K k A(3), in formula (3), P c This reflects the principal components that capture the most significant variance in the data; in fact, for the dimensionality-reduced image P c Only the first "k" eigenvectors corresponding to the "k" largest eigenvalues are selected, therefore a subset of W can be represented as K. k Combining with the PCA equation: P c =WA, therefore the dimensionality-reduced image can be represented as: P c =K kA(3), in formula (3), K k It is a matrix containing the first "k" eigenvectors, P c This reflects the principal components that capture the most significant variance in the data;
[0093] In a concise representation, the essence of the PCA method is encapsulated in this expression: [P c ,W]=PCA(A,k)(4), in formula (4), P c By encapsulating principal components, W represents the transformation matrix that projects the original data onto the space defined by these components; this method effectively extracts key information from polarization images, paving the way for augmentation analysis or subsequent image processing tasks.
[0094] PCA achieves effective feature extraction by selecting the direction with the maximum variance to retain the most representative spectral features. At the same time, it reduces the impact of noise in the data through correlation elimination, ensuring that the processing results are more stable and reliable.
[0095] Furthermore, the calculation of the Stokes parameters and polarization parameters in step S23 satisfies:
[0096]
[0097] In formulas (5), (6), and (7), S λ (x,y) is the Stokes vector at coordinates (x,y) in the image when the wavelength is λ. It is used to fully describe the polarization state of light and contains four components: S 0,λ (x,y),S 1,λ (x,y),S 2,λ (x,y),S 3,λ (x,y); I 0,λ (x,y), I 45,λ (x,y), I 90,λ (x,y), I 135,λ (x, y) represent the image grayscale values (light intensity) measured at wavelength λ with polarization directions of 0°, 45°, 90°, and 135°, respectively; I R,λ (x,y), I L,λ (x, y) represent the light intensities of right-handed and left-handed circularly polarized light at wavelength λ, respectively. In practical applications, the intensity of right-handed circularly polarized light is negligible and is often simplified. 0,λ (x,y) represents the total radiant intensity, reflecting the total energy of light, S 1,λ (x,y) represents the light intensity difference between the 0° and 90° polarization directions, reflecting the difference between horizontal and vertical polarization. 2,λ (x,y) represents the light intensity difference between the 45° and 135° polarization directions, reflecting the difference in diagonal polarization. 3,λ(x,y) represents the intensity difference between right-handed and left-handed circularly polarized light, reflecting the circular polarization characteristics.
[0098] Based on formulas (5), (6), and (7), the DoLP image and AoP image are then subjected to energy weighting of the Stokes vectors. This process adjusts the relative importance of different Stokes parameters based on their energy contributions, aiming to enhance valuable information while suppressing noise. This method refines the representation of polarization information by prioritizing higher-energy components, emphasizing important polarization features and weakening irrelevant polarization features. The generation of the Stokes fused image satisfies the Stokes weighting formula: in, The sum of squares of the gray values of the image corresponding to the polarization direction calculated in formula (13) is represented by S0, S1, and S2, which represent the Stokes vectors of the corresponding polarization directions. Based on formula (8), an optimized Stokes fusion image that contains both light intensity information and highlights significant polarization information is finally obtained for subsequent HSI mapping.
[0099] Furthermore, the specific rules for mapping the obtained fused image and polarization to the HSI color space in step S24 are as follows:
[0100] Map the AoP image containing wavelength information to the H channel;
[0101] The DoLP image, which also contains wavelength information, is mapped to the S channel and then thresholded.
[0102] Map the Stokes fusion image to the I channel;
[0103] like Figure 5 As shown, the AoP image containing wavelength information is mapped to the H channel to determine the color of the pixels, the DoLP image, which also contains wavelength information, is mapped to the S channel, and the pixel values in the S channel are thresholded. The Stokes fusion image is mapped to the I channel.
[0104] The H channel is assigned a value based on the polarization angle distribution of the AoP image. AoP represents the vibration direction of the electric field vector of the light wave (which is also an angle). Therefore, when the AoP image is mapped to the H channel, the hue angle is (0°-360°).
[0105] Thresholding the DoLP image can maximize the saturation value of the target area and minimize the saturation value of the non-target area by setting an appropriate threshold range, thereby significantly increasing the saturation difference between different targets. This difference can make the target stand out more visually. DoLP represents the proportion of polarization components in light waves (0-1). Therefore, the DoLP image is mapped to the S channel. The value of the S channel is determined by the thresholded DoLP image. The saturation value, that is, the purity of the color, is (0-1).
[0106] The purpose of I-channel fusion is to improve the overall brightness of the image. Therefore, Stokes fusion is used for image mapping. The specific mapping parameters depend on the different images. Initially, the three mapping parameters H, S, and I can be randomly set, and then the parameters can be adjusted according to the visual effect of the mapped image.
[0107] Furthermore, in step S3, the shallow feature fusion network module LFFM is composed of multiple feature fusion modules FFM; the feature selection network module FSN is composed of multiple feature selection modules FSM.
[0108] Specifically, in this embodiment, the shallow feature fusion network module LFFM consists of four feature fusion modules FFM; the feature selection network module FSN consists of five feature selection modules FSM. The pseudo-color fused image is fused into deeper features by the downsampled shallow features through the shallow feature fusion module LFFM. Then, the interference features of the fused image are eliminated by constructing the feature selection network module FSN.
[0109] Furthermore, step S3 specifically includes the following steps:
[0110] S301. The pseudo-color fused image is subjected to a 3*3 convolution operation and then processed by the attention mechanism module ECA to obtain the first feature map F. q1 ;
[0111] S302, First Feature Map F q1 After a 4x3 convolution operation and processing by the attention mechanism module ECA, the second feature map F is obtained. q2 ;
[0112] S303, Second Feature Map F q2 After a 7x7 convolution operation and processing by the attention mechanism module ECA, the third feature map F is obtained. q3 ;
[0113] S304, Third Feature Map F q3 After an 8*2 convolution operation and processing by the attention mechanism module ECA, the fourth feature map F is obtained. q4 ;
[0114] S305, Fourth Feature Map F q4 After a 9*2 convolution operation and processing by the attention mechanism module ECA, the fifth feature map F is obtained. q5 ;
[0115] S306, Fifth Feature Map F q5 After a 10*2 convolution operation and processing by the attention mechanism module ECA, the sixth feature map F is obtained. q6 ;
[0116] S307, First Feature Map F q1 With the second feature map F q2 The first feature fusion map y1′ is obtained by fusion using the feature fusion module FFM.
[0117] S308, First feature fusion map y1′ and third feature map F q3 The second feature fusion map y2′ is obtained by fusion using the feature fusion module FFM.
[0118] S309, Second Feature Fusion Map y2′ and Fourth Feature Map F q4 The third feature fusion map y3′ is obtained by the feature fusion module FFM.
[0119] S310, the third feature fusion map y3′ and the fifth feature map F q5 The fourth feature fusion map y4′ is obtained by fusion using the feature fusion module FFM.
[0120] S311, Sixth Feature Map F q6 After being filtered directly by the Feature Selection Module (FSM), the first feature enhancement map Y1′ is obtained.
[0121] S312. After the fourth feature fusion image y4′ is filtered by the feature selection module FSM, the second feature enhancement image Y2′ is obtained through upsampling.
[0122] S313. After the third feature fusion image y3′ is filtered by the feature selection module FSM, the third feature enhancement image Y3′ is obtained through upsampling.
[0123] S314. After the second feature fusion image y2′ is filtered by the feature selection module FSM, the fourth feature enhancement image Y4′ is obtained through upsampling.
[0124] S315. After the first feature fusion image y1′ is filtered by the feature selection module FSM, the fifth feature enhancement image Y5′ is obtained through upsampling.
[0125] S316. After the first feature enhancement image Y1′ undergoes a convolution operation, it is fused with the second feature enhancement image Y2′ and then convolved. Subsequently, it is fused with the third feature enhancement image Y3′ and then convolved. After the third feature enhancement image Y4′ and then convolved, it is fused with the fifth feature enhancement image Y5′ and then convolved to finally obtain the target detection result image.
[0126] For the pseudo-color fused image obtained in step S2, the present invention designs a shallow feature fusion module LFFM based on the SSD architecture. In order to utilize shallower feature maps, a feature map with a resolution of 75×75 from the Conv3_3 layer is first selected. This high-resolution feature map has only undergone two pooling operations compared to the original image, thus retaining a large number of small detail features.
[0127] Subsequently, convolution operations are performed through six convolutional layers at different scales. Each convolutional layer is followed by an efficient channel attention mechanism (ECA) module, which extracts low-frequency component features at each scale to obtain the feature map F. q F q =H ECA (Conv a×b (F))(9), in formula (9), a and b are the height and width of the convolution kernel, respectively, and F is the feature before convolution. q These are features processed by a convolutional module and an efficient channel attention mechanism (ECA). The channel attention mechanism allows the model to consider the relationships between different channels when processing feature maps, thereby enhancing the focus on important features. The structure of the channel attention mechanism (ECA) module is as follows: Figure 6 As shown; the process is as follows:
[0128] First, channel-level global average pooling (GAP) is performed on the feature map output by the convolutional block. Let the feature map output by the convolutional block be χ∈R. W×H×C Where W (width) and H (height) are spatial dimensions, and C (number of channels) is the channel dimension; GAP performs global average pooling independently on each channel, compressing the spatial dimension to 1, as shown in the following formula:
[0129]
[0130] Where g(·) represents the GAP operation, χ i,j It is the aggregated feature of a channel at spatial location (x,j) by summing all channel values of the feature map and taking the average.
[0131] Then, through a non-linear mapping between the channel dimension C and the kernel size k, the kernel size of the 1D convolution (i.e., the "coverage" of local cross-channel interaction) is adaptively determined, avoiding manual parameter tuning. The formula is as follows:
[0132]
[0133] Where γ = 2, b = 1 (fixed parameters), A nonlinear correlation is established using the power-law property of the number of channels, [t] odd This means taking the nearest odd number to t.
[0134] Then, the channel feature vector y obtained from GAP is processed by 1D convolution to efficiently capture local cross-channel dependencies and generate preliminary channel attention weights. The preliminary weights are normalized to the [0,1] interval by an activation function to obtain the final channel attention weights, which are then multiplied element-wise with the original input feature map to enhance important channels and suppress unimportant channels.
[0135] The difference from traditional methods: Traditional channel attention methods reduce the dimensionality of channel features through fully connected layers, but this operation will destroy the direct correspondence between "channel and weight", resulting in a decrease in attention prediction accuracy.
[0136] Specifically, such as Figure 4 As shown, the shallow feature fusion module LFFM consists of four feature fusion modules FFM. FFM fuses adjacent feature maps and generates a fused feature map Y′. Based on feature fusion between adjacent shallow feature maps, it avoids the problem of non-adjacent feature maps being unable to fuse due to significant feature differences. Therefore, it can progressively fuse shallow feature maps F′. q When passed to the deep feature map Y, FFM can be expressed as: In formula (10), Y represents the feature map of a specific level, and is related to the feature map F of the previous level. q To merge, i.e., Y = F q+1 The structure of FFM is as follows Figure 7 As shown.
[0137] As shown in formula (10), the feature map F must be processed before feature fusion. q Downsampling is performed, followed by activation using an activation function. Then, the feature map is fused element-wise with the deep feature map Y. The result of this element-wise fusion is the fused feature map Y′. Here, the deep feature map Y represents the current selected feature map relative to the previous level feature map F. q A deeper meaning;
[0138] The downsampling method employs depthwise separable convolution downsampling; during the fusion process, depthwise separable convolution is used as a means of feature map downsampling, effectively achieving adaptive downsampling of shallow feature maps.
[0139] The Feature Selection Network (FSN) module is used to eliminate interfering features in the fused feature map. As the resolution of the feature map decreases, the target that the feature map can represent also increases, while the features of small targets gradually disappear in this process. For a feature map of a certain level, its feature representation is as follows:
[0140] Y = f(x1, x2, x3, ..., x s ), s∈{1,2,3,…,S}(11), formula (11)
[0141] In the formula (11), s represents a specific scale, Y represents the feature representation of the feature map at that level, f represents the feature extraction algorithm at that level, and x represents the target at different scales. Formula (11) indicates that there is a feature mixture of targets at multiple scales in the same feature layer. For features with larger resolution, it extracts features from targets at multiple scales. These unwanted target features are serious interference terms for subsequent classification and regression tasks.
[0142] For high-resolution features, it extracts features from targets at multiple scales. These unwanted target features are significant interference for subsequent classification and regression tasks. Therefore, to eliminate interference features, this invention designs a Feature Selection Network (FSN) module, which consists of multiple Feature Selection Modules (FSMs). The structure of the FSM is as follows: Figure 3 As shown;
[0143] For feature level Y, a convolution operation is performed with a kernel size of 5. The result of the convolution is a feature map with 1 channel. Finally, a two-dimensional weight matrix is obtained by applying the ReLU activation function. The feature selection result is the product of the original feature map and this weight matrix, expressed as:
[0144]
[0145] Where σ represents the ReLU activation process. The core function of the ReLU activation function after convolution is to introduce non-linear feature expression capability into the network. At the same time, the ReLU activation function can also alleviate the gradient vanishing problem to a certain extent.
[0146] After the element-wise multiplication operation, a convolution operation with a kernel size of 1 was performed as a feature adaptation layer.
[0147] The Feature Selection Module (FSM) is embedded into the Feature Pyramid Network (FPN) to form the Feature Selection Network Module (FSN). The FSN uses convolution operations to generate a spatial weight matrix to filter hierarchical features and thus filter out interfering features, thereby solving the confusion problem during shallow feature propagation.
[0148] Specifically, the method of this invention mainly consists of two major steps: information fusion and target detection. The key to the information fusion part is to process four sets of single-channel polarization images, and to perform effective weighting through PCA dimensionality reduction and spectral energy weighting. Then, based on the Stokes formula, the required Stokes fusion image and two polarization images are generated and mapped to the HSI color space to obtain the HSI three-channel pseudo-color fusion image. The Stokes parameters require light intensity data at four angles: 0°, 45°, 90°, and 135°. The PCA dimensionality reduction targets the spectral data matrix composed of these four polarization angles, while HSI achieves effective integration of spectral and polarization information.
[0149] The target detection method employs a structure of Feature Selection Network (FSN) and Shallow Feature Fusion Network (LFFM). The Feature Selection Module (FSM) within the FSN effectively filters out large target interference, while the LFFM plays a synergistic role in supplementing shallow details. Ultimately, this method incorporates multi-scale thinking throughout the information fusion into the detection network. Compared to traditional methods, it can obtain detection result images with target location information, making it more adaptable to complex scenes.
[0150] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.
Claims
1. A multi-scale target detection method based on spectral polarization information fusion, characterized in that: The method includes the following steps: S1. Acquire a multispectral polarization image of the reflection from the object's surface; S2. For the obtained single-channel polarization image, spectral information fusion is performed using principal component analysis plus spectral energy weighting, and after processing based on Stokes' theorem, a pseudo-color fused image of the HSI three channels is obtained. S3. The obtained pseudo-color fused image is subjected to multi-scale feature optimization and fusion through the structure of the feature selection network module FSN plus the shallow feature fusion network module LFFM, and finally the target detection image is obtained.
2. The multi-scale target detection method based on spectral polarization information fusion as described in claim 1, characterized in that: In step S1, four sets of single-channel polarization images are acquired.
3. The multi-scale target detection method based on spectral polarization information fusion as described in claim 1, characterized in that: Step S2 specifically includes the following steps: S21. Construct a spectral data matrix from the four sets of single-channel polarization images, reduce the dimensionality using principal component analysis (PCA), and select the first principal component image as the output feature. S22. Based on the sum of squared gray values at each pixel location, calculate the spectral energy weight, and perform weighted processing on the first principal component image to obtain the weighted principal component image; S23. Calculate the Stokes parameters and polarization parameters, and transform the weighted principal component image after weighting into a Stokes fusion image, a degree of polarization (DoLP) image, and an angle of polarization (AoP) image. S24. Map the Stokes fusion image, degree of polarization (DoLP) image, and angle of polarization (AoP) image into the HSI color space to obtain a pseudo-color fusion image.
4. The multi-scale target detection method based on spectral polarization information fusion as described in claim 3, characterized in that, In step S21, the dimensionality reduction process using principal component analysis (PCA) specifically includes: S211. Construct a data matrix A with an M×N structure; S212. Calculate the covariance matrix C. S213. Solve for the eigenvectors E of the covariance matrix C, and select the eigenvectors corresponding to the N largest eigenvalues to form a subset K of the transformation matrix W. k ; S214. Output the dimensionality-reduced image. The dimensionality reduction process satisfies P. c =K k A(3), in formula (3), P c To capture the principal components with the most significant variance in the data, K k It is a matrix containing the first "k" eigenvectors.
5. The multi-scale target detection method based on spectral polarization information fusion as described in claim 3, characterized in that, The calculation of Stokes parameters and polarization parameters in step S23 satisfies: In formulas (5), (6), and (7), S λ (x,y) is the Stokes vector at coordinates (x,y) in the image when the wavelength is λ. It is used to fully describe the polarization state of light and contains four components: S 0,λ (x,y),S 1,λ (x,y),S 2,λ (x,y),S 3,λ (x,y); I 0,λ (x,y), I 45,λ (x,y), I 90,λ (x,y), I 135,λ (x, y) represent the image grayscale values (light intensity) measured at wavelength λ with polarization directions of 0°, 45°, 90°, and 135°, respectively; I R,λ (x,y), I L,λ (x,y) represent the light intensities of right-handed and left-handed circularly polarized light at wavelength λ, respectively. The generation of Stokes fused images satisfies the Stokes weighting formula: in, The sum of squares of the gray values of the image corresponding to the polarization direction calculated in formula (13) is represented by S0, S1, and S2, which represent the Stokes vectors corresponding to the polarization direction.
6. The multi-scale target detection method based on spectral polarization information fusion as described in claim 3, characterized in that: The specific rules for obtaining the fused image and polarization mapping to the HSI color space in step S24 are as follows: Map the AoP image to the H channel; Map the DoLP image to the S channel and then threshold it; Map the spectral polarization fusion image onto the I channel.
7. The multi-scale target detection method based on spectral polarization information fusion as described in claim 1, characterized in that: The shallow feature fusion network module LFFM described in step S3 is composed of multiple feature fusion modules FFM; The Feature Selection Network Module (FSN) consists of multiple FSM modules.
8. The multi-scale target detection method based on spectral polarization information fusion as described in claim 7, characterized in that, Step S3 specifically includes the following steps: S301. The pseudo-color fused image is subjected to a 3*3 convolution operation and then processed by the attention mechanism module ECA to obtain the first feature map F. q1 ; S302, First Feature Map F q1 After a 4x3 convolution operation and processing by the attention mechanism module ECA, the second feature map F is obtained. q2 ; S303, Second Feature Map F q2 After convolutional operation with conv7 and processing by the attention mechanism module ECA, the third feature map F is obtained. q3 ; S304, Third Feature Map F q3 After an 8*2 convolution operation and processing by the attention mechanism module ECA, the fourth feature map F is obtained. q4 ; S305, Fourth Feature Map F q4 After a 9*2 convolution operation and processing by the attention mechanism module ECA, the fifth feature map F is obtained. q5 ; S306, Fifth Feature Map F q5 After a 10*2 convolution operation and processing by the attention mechanism module ECA, the sixth feature map F is obtained. q6 ; S307, First Feature Map F q1 With the second feature map F q2 The first feature fusion map y1 is obtained by the feature fusion module FFM. ′ ; S308, First Feature Fusion Image y1 ′ With the third feature map F q3 The second feature fusion map y2 is obtained by the feature fusion module FFM. ′ ; S309, Second Feature Fusion Map y2 ′ With the fourth feature map F q4 After fusion by the feature fusion module FFM, the third feature fusion map y3 is obtained. ′ ; S310, Third Feature Fusion Map y3 ′ With the fifth feature map F q5 After fusion by the feature fusion module FFM, the fourth feature fusion map y4 is obtained. ′ ; S311, Sixth Feature Map F q6 After direct filtering by the Feature Selection Module (FSM), the first feature enhancement map Y1 is obtained. ′ ; S312, Fourth Feature Fusion Map y4 ′ After filtering by the Feature Selection Module (FSM), the second feature enhancement map Y2 is obtained through an upsampling operation. ′ ; S313, Third Feature Fusion Map y3 ′ After filtering by the Feature Selection Module (FSM), the third feature enhancement image Y3 is obtained through upsampling. ′ ; S314, Second Feature Fusion Map y2 ′ After filtering by the Feature Selection Module (FSM), the fourth feature enhancement image Y4 is obtained through upsampling. ′ ; S315, First Feature Fusion Map y1 ′ After filtering by the Feature Selection Module (FSM), the fifth feature enhancement image Y5 is obtained through upsampling. ′ ; S316, First Feature Enhancement Map Y1 ′ After convolution, it is compared with the second feature enhancement map Y2. ′ After fusion and convolution, it is then combined with the third feature enhancement map Y3. ′ After fusion and convolution, it is combined with the fourth feature enhancement map Y4. ′ After fusion and convolution, it is combined with the fifth feature enhancement map Y5. ′ The images are fused and then convolutional to obtain the final target detection result image.