A vehicle detection method using linear polarization information and RGB image fusion

Abstract

The application provides a vehicle detection method using linear polarization information and RGB image fusion, comprising inputting the linear polarization information into a linear polarization adaptive fusion module to extract linear polarization shallow features, inputting the linear polarization shallow features into a linear polarization material attention module group for feature enhancement, and generating three linear polarization feature maps; performing down-sampling on the RGB image to obtain visible light shallow features; performing three times of multi-modal fusion to generate three different levels of fused features; inputting the three different levels of fused features into a neck network and a detection head in sequence, and outputting a vehicle detection result; the application adopts a polarization and visual interactive fusion module to realize deep semantic interaction and complementary alignment of linear polarization information and visible light images at different receptive field scales; and multi-level feature fusion is adopted to improve the detection recall rate and positioning accuracy of small targets and far targets through a cross-scale up-sampling and fusion mechanism.

Description

Technical Field

[0001] This invention relates to the fields of image processing and vehicle detection technology, and in particular to a vehicle detection method that utilizes linear polarization information and RGB image fusion. Background Technology

[0002] The safety and reliability of autonomous driving and advanced driver assistance systems (ADAS) heavily rely on accurate analysis of road scenes. As a fundamental step in achieving reliable road scene understanding, object detection, especially vehicle detection, has received widespread attention in recent years, particularly driven by deep learning frameworks such as YOLO, Faster R-CNN, and DETR, which have led to significant advancements in object detection technology. However, due to the vast structural and appearance variations of vehicles in different scenarios, vehicle detection remains a highly challenging problem.

[0003] Existing advanced vehicle detection methods typically rely on RGB cameras, LiDAR, or infrared sensors, often assuming optimal image quality. To improve detection accuracy, these methods usually explore and fuse rich contextual information from multiple modalities. However, in real-world complex road scenarios (such as low light, rain, fog, and other adverse weather conditions), these conventional multimodal fusion perception systems are often based on an idealized assumption: that image quality is always optimal. When faced with harsh real-world imaging conditions, such as low light, rain, fog, and backlighting, the extremely scarce and damaged scene information acquired by the sensors makes it difficult for existing algorithms to extract effective discriminative features of the vehicle. The acquired color and perception information becomes unstable and insufficient, leading to a precipitous drop in detection accuracy. Existing algorithms struggle to extract meaningful discriminative features from damaged image signals that can effectively distinguish the vehicle from the surrounding background, failing to meet the safety requirements of high-precision perception for autonomous driving.

[0004] The polarization characteristics of light waves, such as the angle of linear polarization map (AoLP) and degree of linear polarization map (DoLP), can robustly reflect the intrinsic physical and material properties of objects, such as surface geometry, roughness, and material composition, unaffected by lighting and weather. However, the prominence and amount of polarization information are not uniform across different scenes and image regions. If traditional direct fusion methods are used, those invalid or redundant polarization features not only fail to improve performance but may also interfere with or confuse the originally effective RGB visual cues.

[0005] In summary, there is an urgent need to address key challenges in existing vehicle detection technologies, such as the reduced detection accuracy due to limited sensor capabilities in complex lighting conditions, and the ease with which information interference can occur during multimodal feature fusion. Summary of the Invention

[0006] To overcome the problems existing in related technologies, the purpose of this invention is to provide a vehicle detection method that utilizes linear polarization information and RGB image fusion to complete a closed loop of local feature compression, global cross-modal interaction, spatial detail restoration and smooth reconstruction, and achieves highly robust vehicle detection for scenarios with complex lighting and drastic weather changes.

[0007] Vehicle detection methods utilizing the fusion of linear polarization information and RGB images include: The linear polarization information and RGB image of the vehicle under test are acquired, and the linear polarization information is input into the linear polarization adaptive fusion module to extract the shallow linear polarization features. The shallow linear polarization features are input into the linear polarization material attention module group for feature enhancement, generating a first linear polarization feature map, a second linear polarization feature map, and a third linear polarization feature map; The RGB image is downsampled to obtain shallow visible light features; Multimodal fusion of the first linear polarization feature map and the visible light shallow layer feature map is performed to obtain the fused shallow layer feature map; Multimodal fusion of the fused shallow layer features and the second linear polarization feature map is performed to obtain the fused middle layer features; The fused middle-layer features and the third linear polarization feature map are subjected to multimodal fusion to obtain the fused high-layer features; The fused shallow features, fused middle features, and fused high features are sequentially input into the neck network and the detection head to perform vehicle detection and obtain the vehicle detection results.

[0008] In a preferred embodiment of the present invention, the step of multimodal fusion of the first linear polarization feature map and the visible light shallow layer feature to obtain the fused shallow layer feature includes: The visible light shallow layer features are preprocessed to obtain the preprocessed visible light shallow layer features; The first linear polarization feature map is preprocessed to obtain the preprocessed first linear polarization feature map; the preprocessing includes average pooling, max pooling and weighted summation. Multiply the preprocessed visible light shallow layer features by the query weight matrix to obtain the first query matrix; multiply the preprocessed first linear polarization feature map by the bond weight matrix and the value weight matrix respectively to obtain the first bond matrix and the first value matrix; Multiply the preprocessed first linear polarization feature map by the query weight matrix to obtain the second query matrix; multiply the preprocessed visible light shallow layer features by the bond weight matrix and the value weight matrix respectively to obtain the second bond matrix and the second value matrix. The first query matrix, the first key matrix, and the first value matrix are subjected to scaling dot product cross attention calculation to obtain the polarization mode sequence. Scaling dot product cross attention calculation is performed on the second query matrix, the second key matrix, and the second value matrix to obtain the visible light mode sequence; The fused shallow features are calculated based on the polarization mode sequence and the visible light mode sequence.

[0009] In a preferred embodiment of the present invention, the step of calculating the fused shallow features based on the polarization mode sequence and the visible light mode sequence includes: The following formula is used to calculate the fused shallow features: ; ; ; in, Indicates enhanced visible light characteristics, This indicates shallow features of the visible light layer. This represents depthwise separable convolution. Indicates upsampling, Represents the visible light mode sequence; Indicates enhanced polarization characteristics. This represents the first linear polarization feature map. Represents a polarization mode sequence; This indicates shallow features after fusion. This represents a 1×1 convolution.

[0010] In a preferred embodiment of the present invention, the step of inputting the shallow linear polarization features into the linear polarization material attention module group for feature enhancement to generate a first linear polarization feature map, a second linear polarization feature map, and a third linear polarization feature map includes: The linearly polarized shallow layer features are input into the first linearly polarized material attention module for feature enhancement to obtain the first linearly polarized feature map. The first linear polarization feature map is input into the second linear polarization material attention module for feature enhancement to obtain the second linear polarization feature map. The second linear polarization feature map is input into the third linear polarization material attention module for feature enhancement to obtain the third linear polarization feature map.

[0011] In a preferred embodiment of the present invention, the step of inputting the shallow linear polarization features into a first linear polarization material attention module for feature enhancement to obtain a first linear polarization feature map includes: The shallow linear polarization features are sequentially subjected to depth-separable convolution and activation to obtain the first refined polarization features; The first refined polarization feature is subjected to channel enhancement to obtain a channel enhancement feature; the channel enhancement feature is used to characterize the channel material. The first refined polarization feature is spatially enhanced to obtain a spatially enhanced feature; the spatially enhanced feature is used to characterize spatial positioning. The channel enhancement features and spatial enhancement features are added element by element to obtain the first linear polarization feature map.

[0012] In a preferred embodiment of the present invention, the linear polarization adaptive fusion module includes a maximum mapping layer, an average mapping layer, an edge extraction layer, a maximum pooling layer, multiple convolutional layers, and multiple activation layers; the outputs of the maximum mapping layer, the average mapping layer, and the edge extraction layer are all connected to the input of the first stitching layer, the output of the first stitching layer is connected to the input of the first convolutional layer, and the output of the first convolutional layer is connected to the input of the first activation layer.

[0013] In a preferred embodiment of the present invention, the step of inputting the linear polarization information into the linear polarization adaptive fusion module to extract shallow linear polarization features includes: The original linear polarization map is input into the edge extraction layer, and the edge confidence map of the original linear polarization map is extracted using Scharr edge filtering. The original linear polarization map is input into the maximum mapping layer, and the original linear polarization map is mapped to the maximum value to obtain the maximum feature; The original linear polarization map is input into the average value mapping layer to perform average value mapping on the original linear polarization map, thereby obtaining the average feature; The edge confidence map, the maximum feature, and the average feature are sequentially concatenated, convolved, and activated to obtain a multi-dimensional spatial attention weight map. The original linear polarization map is sequentially convolved, max-pooled, and activated to obtain a morphological weight map; Convolve the original linear polarization angle map to obtain the initial linear polarization angle map; convolve the original linear polarization degree map to obtain the initial linear polarization degree map. The multidimensional spatial attention weight map and the morphological weight map are added together to obtain the mask multiplier; the initial linear polarization angle map and the mask multiplier are multiplied element by element to obtain the denoised linear polarization angle map. The edge confidence map and the initial linear polarization degree map are multiplied element by element to obtain the enhanced linear polarization degree map; the shallow linear polarization features are calculated based on the denoised linear polarization angle map and the enhanced linear polarization degree map.

[0014] In a preferred embodiment of the present invention, the step of calculating the shallow linear polarization features based on the denoised linear polarization angle map and the enhanced linear polarization degree map includes: By stitching together the denoised linear polarization angle map and the enhanced linear polarization degree map, dual-path features are obtained; The dual-channel feature input channel attention module dynamically assigns channel weights to obtain a channel-weighted feature map; The linearly polarized shallow layer features are obtained by multiplying the dual-path features and the channel-weighted feature map element by element.

[0015] In a preferred embodiment of the present invention, the step of sequentially performing channel concatenation, convolution, and activation on the edge confidence map, the maximum feature, and the average feature to obtain a multi-dimensional spatial attention weight map includes: The multi-dimensional spatial attention weight map is calculated using the following formula: ; in, This represents the multi-dimensional spatial attention weight map. This represents the Sigmoid activation function. Represents a 7×7 convolution; Represents the maximum value mapping. Represents the average value mapping. This represents the original linear polarization degree diagram. Indicates the most significant feature. Indicates average characteristics; E [ ] represents the edge confidence map, and [ ] represents channel splicing.

[0016] In a preferred embodiment of the present invention, the step of extracting the edge confidence map of the original linear polarization map using Scharr edge filtering includes: The edge confidence plot of the original linear polarization plot is calculated using the following formula: ; in, E This represents the edge confidence plot of the original linear polarization map, and Norm represents the normalization operation. This represents the original linear polarization map after grayscale conversion. express x Convolution kernels in the direction, express y The convolution kernel in the direction, * indicates the convolution operation.

[0017] The beneficial effects of this invention are as follows: The vehicle detection method utilizing linear polarization information and RGB image fusion provided by this invention includes acquiring the linear polarization information and RGB image of the vehicle under test, and inputting the linear polarization information into a linear polarization adaptive fusion module to extract shallow linear polarization features. The linear polarization information includes a linear polarization angle map and a linear polarization degree map, which can effectively capture the inherent physical properties of the vehicle and its environment, overcome the limitations of color information, and supplement the RGB image data. The linear polarization adaptive fusion module uses physical priors and edge-aware attention to dynamically suppress background noise and enhance the target contour. The shallow linear polarization features are input into a linear polarization material attention module group for feature enhancement, generating a first linear polarization feature map, a second linear polarization feature map, and a third linear polarization feature map. A dual-path attention mechanism is constructed, focusing on spatial and channel features respectively, to deeply extract and enhance the unique physical priors contained in the linear polarization information, such as material differences. Multimodal fusion is performed on the first linear polarization feature map and the shallow visible light features to obtain the fused shallow features. The fused shallow features and the second linear polarization feature map are multimodally fused to obtain the fused mid-level features. The fused mid-level features and the third linear polarization feature map are then multimodally fused to obtain the fused high-level features. These three fused features are sequentially input into the neck network and the detection head for vehicle detection, yielding the vehicle detection results. This invention employs a polarization and visual interaction fusion module to achieve deep semantic interaction and complementary alignment between linear polarization information and visible light images at different receptive field scales. Multi-level feature fusion is used, employing cross-scale upsampling and fusion mechanisms to improve the detection recall and localization accuracy of small and distant targets, avoiding interference from direct fusion. Finally, the fused multi-level features are aggregated, and the detection head is used to accurately locate and classify vehicle targets, outputting the final vehicle detection results. Attached Figure Description

[0018] Figure 1 This is a flowchart of the vehicle detection method of the present invention that utilizes the fusion of linear polarization information and RGB images; Figure 2 This is an overall architecture diagram of the PMF network of the present invention; Figure 3 This is a structural diagram of the linear polarization adaptive fusion module of the present invention; Figure 4 This is a structural diagram of the linearly polarized material attention module of the present invention; Figure 5 This is a structural diagram of the polarization and visual interaction fusion module of the present invention; Figure 6 This is an RGB image of a daytime scene according to the present invention; Figure 7 The method of this invention is for Figure 6 The results of the inspection of the vehicles in the middle; Figure 8 This is an RGB image of a nighttime scene according to the present invention; Figure 9 The method of this invention is for Figure 8 The results of the inspection of the vehicles in the middle. Detailed Implementation

[0019] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0020] Example 1 like Figure 1 As shown, this embodiment provides a vehicle detection method that utilizes linear polarization information and RGB image fusion, including: S1: Collect the linear polarization information and RGB image of the vehicle under test, and input the linear polarization information into the linear polarization adaptive fusion module to extract the shallow linear polarization features; S2: Input the linearly polarized shallow layer features into the linearly polarized material attention module group for feature enhancement to generate a first linearly polarized feature map, a second linearly polarized feature map, and a third linearly polarized feature map; S3: Downsample the RGB image to obtain visible light shallow layer features; S4: Perform multimodal fusion of the first linear polarization feature map and the visible light shallow layer feature to obtain the fused shallow layer feature; S5: Perform multimodal fusion on the fused shallow layer features and the second linear polarization feature map to obtain the fused middle layer features; S6: Perform multimodal fusion of the fused middle-layer features and the third linear polarization feature map to obtain the fused high-layer features; S7: The fused shallow features, fused middle features, and fused high features are sequentially input into the neck network and the detection head to perform vehicle detection and obtain the vehicle detection results.

[0021] The vehicle recognition framework of this invention is constructed based on a YOLO model backbone network, such as the YOLOv5 model backbone network. It simultaneously receives visible light images and three-channel linear polarization information from the same scene. The linear polarization information includes a linear polarization angle map and a linear polarization degree map, reflecting the differences in the linear polarization angle and degree of polarization of different pixels. For vehicle recognition, the linear polarization angle map can encode the vehicle's geometric features, enhancing the clarity of the vehicle's outline. The linear polarization degree map can distinguish the material differences between the vehicle and the road surface and suppress reflective interference.

[0022] like Figure 2As shown, this invention constructs a PMF network, which is divided into two branches: a polarization branch that processes linear polarization information and a visible light branch that simultaneously processes RGB images. The two branches perform downsampling independently. Specifically, the downsampling method utilizes the five feature extraction layers of the YOLO model's backbone network. The convolution operation of each feature extraction layer is a downsampling process, reducing resolution and increasing channel dimensions to extract deeper features with stronger semantic meaning.

[0023] The PMF network employs one linear polarization adaptive fusion module (PAFM), three linear polarization material attention modules (PMAB), and three polarization and visual interaction fusion modules (PVI-Fusion). Combined with the five feature extraction layers of the YOLO model, it constructs multi-scale contextual features. These multi-scale contextual features are then sequentially input into the neck network and the detection head, outputting the vehicle detection result, namely the location and confidence level of the vehicle's bounding box in the image.

[0024] The structure of the linear polarization adaptive fusion module is as follows: Figure 3 As shown, the linear polarization degree map input of the linear polarization adaptive fusion module is connected to the maximum mapping layer, the average mapping layer, the edge extraction layer, the second convolutional layer, and the third convolutional layer, while the linear polarization angle map input is connected to the fourth convolutional layer. The outputs of the maximum mapping layer, the average mapping layer, and the edge extraction layer are all connected to the input of the first stitching layer, which is followed by the first convolutional layer and the first activation layer, respectively.

[0025] The polarization adaptive fusion module PAFM utilizes physical priors and edge-aware attention to dynamically suppress background noise and enhance target contours, providing clean material features for cross-modal fusion of RGB images and polarization information.

[0026] The step of inputting the linear polarization information into the linear polarization adaptive fusion module to extract shallow linear polarization features includes: S12: Input the original linear polarization map into the edge extraction layer, and use Scharr edge filtering to extract the edge confidence map of the original linear polarization map; S13: Input the original linear polarization map into the maximum mapping layer, perform maximum value mapping on the original linear polarization map, and obtain the maximum feature; S14: Input the original linear polarization degree map into the average value mapping layer, perform average value mapping on the original linear polarization degree map, and obtain the average feature; S15: Perform channel concatenation, convolution, and activation on the edge confidence map, the maximum feature, and the average feature in sequence to obtain a multi-dimensional spatial attention weight map; S16: Perform convolution, max pooling, and activation sequentially on the original linear polarization map to obtain a morphological weight map; S17: Convolve the original linear polarization angle map to obtain the initial linear polarization angle map; convolve the original linear polarization degree map to obtain the initial linear polarization degree map; S18: Add the multi-dimensional spatial attention weight map and the morphological weight map to obtain the mask multiplier; multiply the initial linear polarization angle map and the mask multiplier element by element to obtain the denoised linear polarization angle map; S19: Multiply the edge confidence map and the initial linear polarization degree map element by element to obtain the enhanced linear polarization degree map; calculate the shallow linear polarization features based on the denoised linear polarization angle map and the enhanced linear polarization degree map.

[0027] Before S12, step S11 is included: acquiring the linear polarization information and RGB image of the vehicle under test. The step of extracting the edge confidence map of the original linear polarization map using Scharr edge filtering includes: The edge confidence plot of the original linear polarization plot is calculated using the following formula: (1) Where E represents the edge confidence plot of the original linear polarization map, and Norm represents the normalization operation. This represents the original linear polarization map after grayscale conversion. express x Convolution kernels in the direction, express y The convolution kernel in the direction, * indicates the convolution operation.

[0028] Scharr edge filtering in x The convolution kernel in the direction is: (2) Scharr edge filtering in y The convolution kernel in the direction is: (3) The core function of the polarization adaptive fusion module is to purify linear polarization information and enhance high-frequency structures before cross-modal fusion. Since linear polarization angle maps typically exhibit extremely high randomness and noise interference in low-polarization regions, direct use would severely contaminate the feature fusion process. The polarization adaptive fusion module innovatively utilizes the physical prior confidence contained in the original linear polarization map as guiding information. First, the original linear polarization map is grayscaled and then subjected to Scharr edge filtering. The pixel values ​​of the filtered images are extracted and normalized to the range [0,1] to obtain the edge confidence map. The edge confidence map is a grayscale image with the same resolution as the original linear polarization map. Each pixel value represents the probability that the pixel belongs to a true physical edge, thus accurately capturing high-frequency features such as the metal edges of vehicles. Compared to the Sobel operator of the same size, the Scharr operator significantly improves the accuracy of gradient calculation, especially excelling at capturing fine and weak edges. Its improved kernel coefficients have better rotation invariance.

[0029] In the channel dimension, the original linear polarization degree map is mapped to the maximum value and the average value respectively. The maximum feature, average feature and edge confidence map are concatenated in the channel dimension. Then, a multi-dimensional spatial attention weight map is generated by using a 7×7 large receptive field convolution and a sigmoid activation function.

[0030] The edge confidence map, maximum feature, and average feature are sequentially concatenated, convolved, and activated to obtain a multi-dimensional spatial attention weight map, including: The multi-dimensional spatial attention weight map is calculated using the following formula: (4) in, This represents the multi-dimensional spatial attention weight map. This represents the Sigmoid activation function. Represents a 7×7 convolution; Represents the maximum value mapping. Represents the average value mapping. This represents the original linear polarization degree diagram. Indicates the most significant feature. Indicates average characteristics; E [ ] represents the edge confidence map, and [ ] represents channel splicing.

[0031] The original linear polarization map is input into the second convolutional layer of the linear polarization adaptive fusion module for convolution, followed by max pooling and sigmoid activation to expand the receptive field of the high-confidence region of the original linear polarization map, thus obtaining the morphological weight map. The formula for calculating the morphological weight map is as follows: (5) in, Represents the morphological weighting diagram. This represents the Sigmoid activation function. This indicates max pooling. This represents the convolution operation.

[0032] The multi-dimensional spatial attention weight map and the morphological weight map are added together and used as a mask multiplier on the initial linear polarization angle map to calculate the denoised linear polarization angle map: (6) in, This shows the linear polarization angle diagram after denoising. This represents the initial linear polarization angle map, which is obtained by convolving the original linear polarization angle map. This indicates element-wise multiplication.

[0033] Convolve the original linear polarization map to obtain the initial linear polarization map. The initial linear polarization degree map and the morphological weight map are multiplied element-wise to obtain the enhanced linear polarization degree map, thus significantly enhancing the edge region of the target at the feature level. The denoised linear polarization angle map and the enhanced linear polarization degree map are then concatenated to obtain dual-path features. (7) in, This indicates a dual-path feature; [] indicates a concatenation operation. This shows the linear polarization angle diagram after denoising. This represents the enhanced linear polarization degree diagram.

[0034] After dynamically allocating weights through channel attention, the formula for the output linearly polarized shallow layer features is: (8) in, Indicates shallow linear polarization characteristics. CA This represents channel attention, and Conv represents the convolution operation. This indicates element-wise multiplication.

[0035] The vehicle detection method utilizing linear polarization information and RGB image fusion provided in this embodiment includes acquiring the linear polarization information and RGB image of the vehicle under test, and inputting the linear polarization information into a linear polarization adaptive fusion module to extract shallow linear polarization features. The linear polarization information includes a linear polarization angle map and a linear polarization degree map, which can effectively capture the inherent physical properties of the vehicle and its environment, overcome the limitations of color information, and supplement the RGB image data. The linear polarization adaptive fusion module uses physical priors and edge-aware attention to dynamically suppress background noise and enhance the target contour. The shallow linear polarization features are input into a linear polarization material attention module group for feature enhancement, generating a first linear polarization feature map, a second linear polarization feature map, and a third linear polarization feature map. A dual-path attention mechanism is constructed, focusing on spatial and channel features respectively, to deeply extract and enhance the unique physical priors contained in the linear polarization information, such as material differences. Multimodal fusion is performed on the first linear polarization feature map and the shallow visible light features to obtain the fused shallow features. The fused shallow features and the second linear polarization feature map are multimodally fused to obtain the fused mid-level features. The fused mid-level features and the third linear polarization feature map are then multimodally fused to obtain the fused high-level features. These three fused features are sequentially input into the neck network and the detection head for vehicle detection, yielding the vehicle detection results. This invention employs a polarization and visual interaction fusion module to achieve deep semantic interaction and complementary alignment between linear polarization information and visible light images at different receptive field scales. Multi-level feature fusion is used, employing cross-scale upsampling and fusion mechanisms to improve the detection recall and localization accuracy of small and distant targets, avoiding interference from direct fusion. Finally, the fused multi-level features are aggregated, and the detection head is used to accurately locate and classify vehicle targets, outputting the final vehicle detection results.

[0036] Example 2 This embodiment provides a vehicle detection method that utilizes linear polarization information and RGB image fusion. This embodiment further describes the scheme of Embodiment 1 based on Embodiment 1.

[0037] like Figure 2 As shown, P1 represents the first feature extraction layer of the YOLO model backbone network, P2 corresponds to the second feature extraction layer, and so on, with P5 corresponding to the fifth feature extraction layer. After the linear polarization adaptive fusion module PAFM outputs the linear polarization shallow features, these features are sequentially input into the first, second, and third feature extraction layers of the YOLO model backbone network. The linear polarization shallow features are then subjected to three consecutive convolutions (downsampling) to enhance their high-level features and improve classification representation capabilities.

[0038] The step of inputting the shallow linearly polarized features into the linearly polarized material attention module group for feature enhancement to generate a first linearly polarized feature map, a second linearly polarized feature map, and a third linearly polarized feature map includes: S21: Input the linearly polarized shallow layer features into the first linearly polarized material attention module for feature enhancement to obtain the first linearly polarized feature map; S22: Input the first linear polarization feature map into the second linear polarization material attention module for feature enhancement to obtain the second linear polarization feature map; S23: Input the second linear polarization feature map into the third linear polarization material attention module for feature enhancement to obtain the third linear polarization feature map.

[0039] The shallow linear polarization features, after convolution through layers P1-P3, are input into the first linear polarization material attention module. The output of the first linear polarization material attention module sequentially passes through the fourth feature extraction layer of the YOLO model backbone network, the second linear polarization material attention module, the fifth feature extraction layer of the YOLO model backbone network, and the third linear polarization material attention module. The linear polarization material attention module is... Figure 2 In PMAB, the structure of the linearly polarized material attention module is as follows: Figure 4 As shown.

[0040] This embodiment takes the feature enhancement process of the first polarization material attention module as an example: First, depthwise separable convolution and the LeakyReLU activation function are used to perform preliminary refinement and reconstruction of the shallow linear polarization features across channels and spatial dimensions, generating refined polarization features: (9) in, This indicates a refined polarization feature. This represents depthwise separable convolution. This represents the LeakyReLU activation function. X This represents the input to the polarization material attention module. For the first polarization material attention module, X As in Example 1 .

[0041] Channel attention calculation based on multidimensional feature aggregation is performed on the refined polarization features to obtain channel enhancement features. : (10) (11) in, Indicates global average pooling. Indicates global max pooling. This represents the first fully connected layer of the polarization material attention module. This represents the second fully connected layer of the polarization material attention module; This represents the SiLU activation function. This represents the Sigmoid activation function. This represents the channel attention map.

[0042] Spatial enhancement features are calculated using formulas (12)-(14): (12) (13) (14) in, The expression represents refined polarization features. MeanPool represents mean pooling, MaxPool represents max pooling, MinPool represents min pooling, and SumPool represents summation pooling. Pconcat represents the features obtained after concatenating the four pooling methods. This represents the spatial attention weight map of the attention module for linearly polarized materials. This represents spatial enhancement features.

[0043] Four aggregation operations—mean pooling, max pooling, min pooling, and summation pooling—are independently performed on the refined polarization features. The results of these four pooling operations are then concatenated along the channel dimension to generate a comprehensive spatial descriptor containing rich local and global statistical information. Subsequently, convolutional layers are used to extract broader contextual spatial information and generate a spatial attention weight map. This enables spatial attention calculation and weighting based on global pooling concatenation, resulting in spatially enhanced features. .

[0044] Finally, a parallel fusion strategy is adopted to add the channel enhancement features and spatial enhancement features element-wise to generate the final output polarization feature map, which has both strong channel material characterization capabilities and precise spatial positioning capabilities. (15) in, Y To output the polarization feature map, This indicates element-wise addition.

[0045] The three linearly polarized material attention modules have identical structures, differing only in their inputs and outputs. The first linearly polarized material attention module takes shallow linearly polarized features as input and outputs a first linearly polarized feature map. The second linearly polarized material attention module takes the first linearly polarized feature map downsampled by layer P4 as input and outputs a second linearly polarized feature map. The third linearly polarized material attention module takes the second linearly polarized feature map downsampled by layer P5 as input and outputs a third linearly polarized feature map. The core function of the polarization material attention module is to selectively extract and enhance material features that reflect the physical priors of the target at a deep dimension of the feature space. The PMAB polarization material attention module constructs a dual-path attention mechanism that focuses on spatial and channel features respectively, deeply extracting and enhancing the unique physical priors contained in the polarization information, such as material differences.

[0046] Example 3 This embodiment provides a vehicle detection method that utilizes linear polarization information and RGB image fusion. This embodiment further describes the scheme of Embodiment 1 based on Embodiment 1.

[0047] The multimodal fusion of the first linear polarization feature map and the visible light shallow layer feature to obtain the fused shallow layer feature includes: S41: Preprocess the visible light shallow layer features to obtain the preprocessed visible light shallow layer features; S42: Preprocess the first linear polarization feature map to obtain a preprocessed first linear polarization feature map; the preprocessing includes average pooling, max pooling and weighted summation; S43: Multiply the preprocessed visible light shallow layer features by the query weight matrix to obtain the first query matrix; multiply the preprocessed first linear polarization feature map by the bond weight matrix and the value weight matrix respectively to obtain the first bond matrix and the first value matrix; S44: Multiply the preprocessed first linear polarization feature map by the query weight matrix to obtain the second query matrix; multiply the preprocessed visible light shallow layer features by the bond weight matrix and the value weight matrix respectively to obtain the second bond matrix and the second value matrix; S45: Perform scaling dot product cross attention calculation on the first query matrix, the first key matrix, and the first value matrix to obtain the polarization mode sequence; S46: Perform scaling dot product cross-attention calculation on the second query matrix, the second key matrix, and the second value matrix to obtain the visible light mode sequence; S47: Calculate the fused shallow features based on the polarization mode sequence and the visible light mode sequence.

[0048] The core function of the polarization and visual interaction fusion module is to construct the global contextual dependency between visible light and polarization modes at different receptive field scales, realize deep semantic interaction of cross-modal features, and overcome the limitations of traditional local convolution in cross-modal fusion.

[0049] The polarization and visual interaction fusion module is Figure 2 The PVI-Fusion synchronous reception in the image features the characteristics of the visible light branch and polarization branch, such as... Figure 5 As shown, the visible light features and polarization feature maps are preprocessed. Then, the low-resolution preprocessed visible light features and preprocessed polarization features are flattened in space dimension and independent learnable position codes are added to generate modal sequences containing global spatial distribution information. A cross-attention mechanism is then used to enable the features of the two modalities to guide and learn from each other. Finally, after upsampling, smoothing, concatenation and convolution operations, different levels of fused features are output.

[0050] Preprocessing of visible light features and polarization feature maps is performed using formulas (16)-(17): (16) (17) in, The visible light characteristics after preprocessing. This is a preprocessed linear polarization feature map.

[0051] For the first polarization and visual interaction fusion module, the input is the visible light shallow layer feature and the first linear polarization feature map, and the output is the fused shallow layer feature. For the second polarization and visual interaction fusion module, the input is the fused shallow layer feature downsampled by the P4 layer, and the output of the second linear polarization material attention module, and the output is the fused mid-layer feature. For the third polarization and visual interaction fusion module, the input is the fused mid-layer feature downsampled by the P5 layer, and the output of the third linear polarization material attention module, and the output is the fused high-layer feature.

[0052] This embodiment takes the processing of the first polarization and visual interaction fusion module as an example, and uses formulas (18)-(19) to calculate the polarization mode sequence and the visible light mode sequence: (18) (19) in, Represents the polarization mode sequence. Represents the visible light mode sequence; Softmax represents the normalized exponential function. Represents the first-valued matrix. This represents the transpose of the first bond matrix. This represents the first query matrix. This indicates the length of the first bond matrix. Represents the second-valued matrix. This represents the transpose of the second bond matrix. This represents the second query matrix. Calculate the scaled dot product cross-attention, enabling the two modal features to retrieve complementary information from each other.

[0053] Bilinear interpolation is used to upsample the two modal sequences to the original spatial resolution of the initial input. Depthwise separable convolution is used to smooth the upsampled visible light modal sequence and polarization modal sequence. Then, the smoothed features are added to the input visible light features and polarization features by residual addition to obtain enhanced visible light features and enhanced polarization features. Finally, channel concatenation and convolution are used to perform cross-channel information interaction and dimensionality compression to output the fused shallow features. The formulas for calculating the fused shallow features are (20)-(22): (20) ;(twenty one) ;(twenty two) in, Indicates enhanced visible light characteristics, This indicates shallow features of the visible light layer. This represents depthwise separable convolution. Indicates upsampling, Represents the visible light mode sequence; Indicates enhanced polarization characteristics. This represents the first linear polarization feature map. Represents a polarization mode sequence; This indicates shallow features after fusion. This represents a 1×1 convolution.

[0054] The process of calculating the fused mid-level and high-level features is the same as described above, only the inputs differ, and will not be repeated here. Through the above process, the PVI-Fusion module constructs cross-modal global context dependencies, realizing deep semantic interaction and complementary alignment between polarization images and visible light images at different receptive field scales. This completes the closed loop of local feature compression, global cross-modal interaction, spatial detail restoration, and smooth reconstruction, significantly improving the detection head's performance in locating and classifying vehicle targets in complex environments. Through cross-scale upsampling and fusion mechanisms, the detection recall and localization accuracy of small and distant targets are improved.

[0055] The fused shallow features, fused mid-level features, and fused high-level features are respectively input into the neck network of the YOLO model. The neck network of the YOLO model integrates and strengthens the fused features at different scales, enabling the YOLO model to recognize both large and small targets.

[0056] Finally, the outputs of the three branches of the neck network of the YOLO model are provided to the detection head. The detection head classifies objects in the scene based on the inputs of three different levels, detects whether there are vehicles in the scene, and if there are vehicles, outputs the position and confidence of the vehicles.

[0057] Example 4 The experimental platform for this invention is a deep learning server running Linux. Its CPU is an Intel(R) Xeon(R) Platinum 8336C with a clock speed of 2.3GHz, its GPU is an NVIDIA GeForce RTX A6000 with 48GB of video memory, and its RAM is 64GB. The programming language is Python, and all deep learning frameworks are based on PyTorch. The development environment is PyTorch 2.5.1, CUDA 12.1, and Python 3.10.20.

[0058] The experiments were conducted on the RGB-P Car dataset, which contains 2601 sets of RGB, AoLP, and DoLP images. Each image is accompanied by a manually annotated bounding box indicating the location and size of each vehicle. Linear polarization information and RGB images were acquired under different weather conditions (sunny and rainy), different lighting conditions (day and night), different driving environments (indoor, outdoor, road, and parking lot), and different vehicle densities.

[0059] To verify the superiority of the proposed multimodal vehicle detection method, a comparative experiment was conducted between the proposed method and a baseline model under the same hardware and software testing environment and dataset. This experiment used the average precision (AP), a standard metric in the target detection field, as an objective evaluation indicator. A higher AP value indicates better overall target detection and localization performance of the model. The experimental results are shown in the table below: Table 1 Comparative Experimental Results ; The traditional single-modal target detection model (Yolov5-rgb) achieved an AP value of 46.2% on the test set. When the single input was replaced with a linear polarization degree map (Yolov5-DoLP), the detection accuracy slightly improved to 47% due to the introduction of physical priors that characterize the high-frequency contours of the object. However, when using only a linear polarization angle map (Yolov5-AoLP), the detection accuracy dropped to 45.6% due to the presence of more random noise in the low-polarization region. This indicates that although polarization information contains unique physical material properties, without a reasonable extraction and filtering mechanism, it is difficult to obtain optimal performance when used directly as a single input.

[0060] Compared to single-modal baseline models, the multimodal fusion detection method proposed in this invention achieves a highest AP value of 52.1%. Specifically, the method of this invention achieves a significant performance improvement of 5.9% compared to the traditional pure visual baseline model (Yolov5-rgb); and also improves by 5.1% compared to the best-performing single polarization baseline model (Yolov5-DoLP).

[0061] Figure 6 This is an RGB image of a daytime scene. Figure 7 The method of the present invention is for Figure 6 The results of the inspection of the vehicles in the middle. Figure 8 RGB image of a nighttime scene. Figure 9 The method of the present invention is for Figure 8 The results of the inspection of the vehicles in the middle.

[0062] This invention also provides an electronic device, which includes a processor, a memory, a network interface, and a database connected via a system bus. The processor in this electronic device is designed to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores operating devices, computer programs, and a database. When the computer program is executed by the processor, it implements the vehicle detection methods of Embodiments 1-3, which utilize linear polarization information and RGB image fusion.

[0063] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0064] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A vehicle detection method utilizing the fusion of linear polarization information and RGB images, characterized in that, include: The linear polarization information and RGB image of the vehicle under test are acquired, and the linear polarization information is input into the linear polarization adaptive fusion module to extract the shallow linear polarization features. The shallow linear polarization features are input into the linear polarization material attention module group for feature enhancement, generating a first linear polarization feature map, a second linear polarization feature map, and a third linear polarization feature map; The RGB image is downsampled to obtain shallow visible light features; Multimodal fusion of the first linear polarization feature map and the visible light shallow layer feature map is performed to obtain the fused shallow layer feature map; Multimodal fusion of the fused shallow layer features and the second linear polarization feature map is performed to obtain the fused middle layer features; The fused middle-layer features and the third linear polarization feature map are subjected to multimodal fusion to obtain the fused high-layer features; The fused shallow features, fused middle features, and fused high features are sequentially input into the neck network and the detection head to perform vehicle detection and obtain the vehicle detection results. The linear polarization adaptive fusion module includes a maximum mapping layer, an average mapping layer, an edge extraction layer, a maximum pooling layer, multiple convolutional layers, and multiple activation layers; the outputs of the maximum mapping layer, the average mapping layer, and the edge extraction layer are all connected to the input of the first splicing layer, the output of the first splicing layer is connected to the input of the first convolutional layer, and the output of the first convolutional layer is connected to the input of the first activation layer. The step of inputting the linear polarization information into the linear polarization adaptive fusion module to extract shallow linear polarization features includes: The original linear polarization map is input into the edge extraction layer, and the edge confidence map of the original linear polarization map is extracted using Scharr edge filtering. The original linear polarization map is input into the maximum mapping layer, and the original linear polarization map is mapped to the maximum value to obtain the maximum feature; The original linear polarization map is input into the average value mapping layer to perform average value mapping on the original linear polarization map, thereby obtaining the average feature; The edge confidence map, the maximum feature, and the average feature are sequentially concatenated, convolved, and activated to obtain a multi-dimensional spatial attention weight map. The original linear polarization map is sequentially convolved, max-pooled, and activated to obtain a morphological weight map; Convolve the original linear polarization angle map to obtain the initial linear polarization angle map; convolve the original linear polarization degree map to obtain the initial linear polarization degree map. The multidimensional spatial attention weight map and the morphological weight map are added together to obtain the mask multiplier; the initial linear polarization angle map and the mask multiplier are multiplied element by element to obtain the denoised linear polarization angle map. The edge confidence map and the initial linear polarization degree map are multiplied element-wise to obtain the enhanced linear polarization degree map; the shallow linear polarization features are calculated based on the denoised linear polarization angle map and the enhanced linear polarization degree map.

2. The vehicle detection method based on the fusion of linear polarization information and RGB images according to claim 1, characterized in that, The multimodal fusion of the first linear polarization feature map and the visible light shallow layer feature to obtain the fused shallow layer feature includes: The visible light shallow layer features are preprocessed to obtain the preprocessed visible light shallow layer features; The first linear polarization feature map is preprocessed to obtain the preprocessed first linear polarization feature map; the preprocessing includes average pooling, max pooling and weighted summation. Multiply the preprocessed visible light shallow layer features by the query weight matrix to obtain the first query matrix; multiply the preprocessed first linear polarization feature map by the bond weight matrix and the value weight matrix respectively to obtain the first bond matrix and the first value matrix; Multiply the preprocessed first linear polarization feature map by the query weight matrix to obtain the second query matrix; multiply the preprocessed visible light shallow layer features by the bond weight matrix and the value weight matrix respectively to obtain the second bond matrix and the second value matrix. The first query matrix, the first key matrix, and the first value matrix are subjected to scaling dot product cross attention calculation to obtain the polarization mode sequence; Scaling dot product cross attention calculation is performed on the second query matrix, the second key matrix, and the second value matrix to obtain the visible light mode sequence; The fused shallow features are calculated based on the polarization mode sequence and the visible light mode sequence.

3. The vehicle detection method using linear polarization information and RGB image fusion according to claim 2, characterized in that, The calculation of the fused shallow features based on the polarization mode sequence and the visible light mode sequence includes: The following formula is used to calculate the fused shallow features: ； ； ； in, Indicates enhanced visible light characteristics, This indicates shallow features of the visible light layer. This represents depthwise separable convolution. Indicates upsampling, Represents the visible light mode sequence; Indicates enhanced polarization characteristics. This represents the first linear polarization feature map. Represents a polarization mode sequence; This indicates shallow features after fusion. This represents a 1×1 convolution.

4. The vehicle detection method using linear polarization information and RGB image fusion according to claim 1, characterized in that, The step of inputting the shallow linearly polarized features into the linearly polarized material attention module group for feature enhancement to generate a first linearly polarized feature map, a second linearly polarized feature map, and a third linearly polarized feature map includes: The linearly polarized shallow layer features are input into the first linearly polarized material attention module for feature enhancement to obtain the first linearly polarized feature map. The first linear polarization feature map is input into the second linear polarization material attention module for feature enhancement to obtain the second linear polarization feature map. The second linear polarization feature map is input into the third linear polarization material attention module for feature enhancement to obtain the third linear polarization feature map.

5. The vehicle detection method using linear polarization information and RGB image fusion according to claim 4, characterized in that, The step of inputting the shallow linear polarization features into the first linear polarization material attention module for feature enhancement to obtain the first linear polarization feature map includes: The shallow linear polarization features are sequentially subjected to depth-separable convolution and activation to obtain the first refined polarization features; The first refined polarization feature is subjected to channel enhancement to obtain a channel enhancement feature; the channel enhancement feature is used to characterize the channel material. The first refined polarization feature is spatially enhanced to obtain a spatially enhanced feature; the spatially enhanced feature is used to characterize spatial positioning. The first linear polarization feature map is obtained by adding the channel enhancement feature and the spatial enhancement feature element by element.

6. The vehicle detection method based on the fusion of linear polarization information and RGB images according to claim 1, characterized in that, The step of calculating the shallow linear polarization features based on the denoised linear polarization angle map and the enhanced linear polarization degree map includes: By stitching together the denoised linear polarization angle map and the enhanced linear polarization degree map, dual-path features are obtained; The dual-channel feature input channel attention module dynamically assigns channel weights to obtain a channel-weighted feature map; The linearly polarized shallow layer features are obtained by multiplying the dual-path features and the channel-weighted feature map element by element.

7. The vehicle detection method based on the fusion of linear polarization information and RGB images according to claim 1, characterized in that, The edge confidence map, maximum feature, and average feature are sequentially concatenated, convolved, and activated to obtain a multi-dimensional spatial attention weight map, including: The multi-dimensional spatial attention weight map is calculated using the following formula: ； in, This represents the multi-dimensional spatial attention weight map. This represents the Sigmoid activation function. Represents a 7×7 convolution; Represents the maximum value mapping. Represents the average value mapping. This represents the original linear polarization degree diagram. Indicates the most significant feature. Indicates average characteristics; E [ ] represents the edge confidence map, and [ ] represents channel splicing.

8. The vehicle detection method using linear polarization information and RGB image fusion according to claim 1, characterized in that, The step of extracting the edge confidence map of the original linear polarization map using Scharr edge filtering includes: The edge confidence plot of the original linear polarization plot is calculated using the following formula: ； in, E This represents the edge confidence plot of the original linear polarization map, and Norm represents the normalization operation. This represents the original linear polarization map after grayscale conversion. express x Convolution kernels in the direction, express y The convolution kernel in the direction, * indicates the convolution operation.

Images