A vehicle target detection method based on RGB and polarization modality fusion

Abstract

The present invention belongs to the field of target detection in the field of computer vision, and proposes a method for automobile target detection based on the fusion of RGB and polarization modalities. This method is the first to simultaneously utilize RGB and three-color AoLP / DoLP cues to achieve automobile detection in challenging scenarios. It utilizes polarization modalities to perceive automobile materials, and designs a material perception memory module and a dynamic weight distribution mechanism for multimodal feature fusion. The PCDNet algorithm model solves the problem of automobile detection in challenging traffic scenarios. As the first method to use RGB and polarization information in automobile detection methods, the present invention solves the problem of difficulty in detecting automobile instances under challenging degraded scene conditions. The unique polarization performance of the automobile area is learned and recorded, and the two modalities of RGB and polarization are dynamically combined to generate comprehensive, comprehensive, and reliable contextual semantic features. Comparative experiments have demonstrated the advantages of this method in detection accuracy and detection speed.

Description

Technical Field

[0001] The present invention relates to the field of target detection in the field of computer vision, and in particular to a method for detecting automobile targets based on the fusion of RGB and polarization modes. Background Art

[0002] Object detection in two-dimensional images is a core problem in computer vision. The goal is to find all instances of objects of interest in an image and determine their categories and locations. However, due to the differences in appearance, shape, and pose of different objects, as well as interference from factors such as illumination and occlusion during imaging, object detection remains a challenging task.

[0003] Vehicle detection methods are a critical task and a fundamental prerequisite for realizing many autonomous driving and intelligent transportation functions. Significant differences in lighting, weather, and vehicle density across different traffic scenarios can still pose challenges to existing general-purpose detectors or dedicated vehicle detection algorithms. Unstable and limited color information hinders the extraction of key contextual semantic features of vehicles, reducing the accuracy and reliability of vehicle detection, further failing to meet the extremely demanding safety requirements of autonomous driving and intelligent transportation. Therefore, enabling vision systems to discover and identify vehicle instances in a scene is of practical significance and application value.

[0004] The present invention mainly uses relevant technologies such as the polarization principle of light, target detection method and multimodal fusion method.

[0005] (a) Principle of polarization of light

[0006] The polarization phenomenon of light refers to the light in which the electric field vector in the light wave has the same vibration direction in space but different amplitudes. Unpolarized light has a randomly fluctuating electric field, while the electric field direction of polarized light has a clear bias. Common ordinary light sources, such as the sun, incandescent lamps and LED spotlights, emit unpolarized light. When unpolarized light passes through a linear polarizer, reflects on certain material surfaces, or undergoes certain types of scattering, it will be converted into partially polarized light or fully polarized light. Polarization has been widely used in tasks such as reflection removal, normal / shape estimation, and semantic segmentation. In this invention, we only focus on linearly polarized light and use the angle of linear polarization (AoLP) and degree of linear polarization (DoLP) to describe it. The degree of polarization of light reflected from the glass, rubber and plastic parts of a car is usually higher than that from trees, roads and the sky. This physical clue inspired us to perform robust car detection based on the perception of the polarization properties of the material.

[0007] (b) Target detection methods

[0008] Thanks to advances in deep learning, object detection methods have made significant progress. Many state-of-the-art (SOTA) methods have adopted powerful encoders such as ResNet, VGG, and EfficientNet, including two-stage (region-based) Faster R-CNN and EfficientDet, one-stage YOLO and SSD, anchor-based Cascade R-CNN and TOOD, and anchor-free FCOS and FSAF. At the same time, target detection methods are also developing towards the trend of large parameter models. The Vision Transformer (ViT) structure derived from the field of natural language processing has brought significant improvements to the field of target detection. "Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision (ECCV).", "Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020). Deformable DETR: Deformable transformers for end-to-end object detection. arXiv preprint arXiv: 2010.04159." and "Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang.2022.DAB-DETR:Dynamic Anchor Boxes are Better QueriesforDETR.In International Conference on Learning Representations(ICLR).” and a series of DETR methods have achieved remarkable performance.However, most of these methods rely on clear RGB images, which are difficult to obtain under degraded conditions. Restoring and enhancing low-quality images requires additional hardware and time. Furthermore, the overhead of increasingly parameterized detection models often exceeds the hardware load of end devices and the speed requirements of real-time tasks. Our material property perception based on polarization imaging can further improve the robustness of detection algorithms while limiting the increase in algorithm runtime overhead.

[0009] (c) Multimodal fusion method

[0010] Multimodal fusion features can provide rich contextual information for robust object detection. In their paper, "Rachel Blin, Samia Ainouz, Stéphane Canu, and Fabrice Meriaudeau. 2019. Roadscenes analysis in adverse weather conditions by polarization-encoded images and adapted deep learning. In ITSC," Blin et al. simply superimposed multimodal data in the channel dimension to use as fused features to replace the original input. “Ankith Manjunath, Ying Liu, Bernardo Henriques, and Armin Engstle. 2018. Radar based object detection and tracking for autonomous driving. In Proceedings of the IEEE MTT-S International Conference on Microwaves for Intelligent Mobility (ICMIM).” Manjunath et al. and “Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. 2017. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition (CVPR).” Chen et al. used concatenation and element-wise addition to fuse low-level LiDAR and RGB features, respectively. "Peng Zhang, Peijun Du, Cong Lin, Xin Wang, Erzhu Li, Zhaohui Xue, and Xuyu Bai. 2020. A hybrid attention-aware fusion network (HAFNet) for building extraction from high-resolution imagery and LiDAR data. Remote Sensing (2020)." HAFNet uses a cross-modal attention mechanism for feature fusion.“Haiyang Mei, Bo Dong, Wen Dong, Pieter Peers, Xin Yang, Qiang Zhang, and Xiaopeng Wei. 2021. Depth-aware mirror segmentation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.” Mei et al. compute dynamic fusion weights for RGB and depth modalities based on the quality of modality features. Similarly, “Wei Ji, Jingjing Li, Shuang Yu, Miao Zhang, Yongri Piao, Shunyu Yao, Qi Bi, Kai Ma, Yefeng Zheng, Huchuan Lu, et al. 2021. Calibrated RGB-D salient object detection. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.” Ji et al. use global average pooling followed by fully connected layers to compute channel attention weights for each modality. While these methods achieve some performance improvements, they only passively combine information from additional modalities and fail to fully exploit the inherent properties of specific modalities, leading to performance degradation in challenging degraded scenarios. The present invention's vehicle detection method, based on material polarization feature perception, learns the inherent characteristics of the various materials that make up a vehicle by memory, and implements a dynamic fusion strategy for multimodal features with active compensation, enhancing the robustness of vehicle detection in challenging scenarios.

[0011] This paper is the first car detection algorithm to use RGB, three-color AoLP and three-color DoLP as algorithm inputs. There is no publicly available dataset. To further study material polarization information and implement effective car detection algorithms, this paper constructs the first large-scale car detection dataset RGB-PCar that includes RGB, three-color AoLP and three-color DoLP, and implements PCDNet (RGB-Polarization car detection network) car detection method that perceives material polarization properties. Summary of the Invention

[0012] The purpose of the present invention is to achieve a more efficient and reliable vehicle detection method based on RGB and corresponding AoLP and DoLP data through an effective target instance material memory learning method and a dynamic multimodal feature compensation strategy. The present invention is applicable to a variety of real-world scenarios and vehicles of different types and densities. To achieve the above objectives, the present invention first constructs the first multi-scene, multi-shooting condition, and multi-vehicle type vehicle detection dataset RGB-PCar that contains RGB, AoLP, and DoLP data, and designs and implements a vehicle detection method in real-world scenarios. Given a single RGB and polarization image group, the position and size results of the vehicles present therein are obtained.

[0013] The technical solution of the present invention is: a car target detection method based on RGB and polarization mode fusion, establishing a PCDNet network, and the car detection dataset RGB-P Car includes RGB information, AoLP information and DoLP information, which is input into the PCDNet network;

[0014] The PCDNet network includes a polarization integration module (PI), two feature extraction branches, a material perception memory module (MPM), a cross-domain demand query module (CDDQ), a feature pyramid (FPN), and a detection head. AoLP information and DoLP information are integrated into polarization information by the polarization integration module (PI). The two feature extraction branches are a polarization processing branch and an RGB processing branch, which are used to extract polarization features and RGB features. After the feature extraction branch extracts multi-level polarization features, the polarization features are passed through the material perception memory module (MPM) to obtain the differences and connections in the polarization property performance of different material regions. The RGB features are guided by the cross-domain demand query module (CDDQ), and compensation is obtained from the polarization features in a query-response manner. The RGB features themselves are used to map the correlation of all channels in the channel dimension to assign weights for multimodal feature fusion. Finally, the features passed through the cross-domain demand query module (CDDQ) are fed into the feature pyramid (FPN) structure to exchange semantic features of different levels, and are respectively sent to the detection head to generate the final detection results.

[0015] The construction process of the car detection dataset RGB-P Car is as follows:

[0016] A color polarization camera was used to record cars in different traffic scenarios. The color polarization camera simultaneously captured pixel-aligned polarization data in three color channels and four directions, and each car in each image had a corresponding professionally annotated bounding box. The three color channels were R, G, and B; the four directions were angles of 0°, 45°, 90°, and 135° with the camera plane. The scenes in the car detection dataset included different weather conditions, lighting conditions, traffic environments, and vehicle density scenarios.

[0017] The polarization integration module PI includes a maximum function, an average function, a cascade operation layer, a boundary extraction operation layer, an addition operation, a multiplication operation, a maximum pooling layer and a convolution layer; the DoLP information is cascaded with the values ​​obtained by the average function and the maximum function, and then input into the first convolution layer; the DoLP information is added to the output of the first convolution layer after passing through the second convolution layer and the maximum pooling layer, and multiplied with the AoLP information before inputting into the first 3×3 convolution layer; the original value of the DoLP information is added to the output of the boundary extraction operation layer and input into the second 3×3 convolution layer; the outputs of the first 3×3 convolution layer and the second 3×3 convolution layer are cascaded and input into the third 3×3 convolution layer to obtain the polarization information F pol ;

[0018] The PI module is expressed as:

[0019]

[0020]

[0021] Where φ represents AoLP information and ρ represents DoLP information; denotes a convolutional layer with a kernel size of k×k, stride s, batch normalization, and SiLU activation function; [·] refers to the cascade operation in the channel dimension; MP is the maximum pooling with a kernel size of 5; E represents the edge extraction operation using the Schur operator; Represents pixel-level multiplication operation; avg and max refer to the average function and maximum function respectively; σ refers to the sigmoid activation function.

[0022] The material perception memory module MPM includes a material space perception memory module MSPM and a material channel perception memory module MCPM;

[0023] The material space perception memory module MSPM includes a continuous convolution layer and a deconvolution layer with a step size of 2; the polarization information F pol Input into the material space perception memory module MSPM, and judge the connection and difference of each area in the spatial dimension through the memory matrix in the spatial perception memory module to obtain the polarization feature representation F′ of the enhanced space pol ;

[0024] The material channel perception memory module MCPM includes convolution operation, deconvolution operation, global average pooling layer and two fully connected layers; strengthen the polarization feature representation F′ of the space pol Input into the material channel perception memory module MCPM, and judge the connection and difference of each area in the channel dimension through the memory matrix in the material channel perception memory module to obtain the polarization feature representation F″ of the enhanced channel pol ;

[0025] The MPM module is represented as:

[0026]

[0027]

[0028] in, Represents deconvolution with kernel size k×k, stride s, batch normalization, and SiLU activation function; are the memory matrices in the two fully connected layers, named m1 and m2 respectively; GAP is the global average pooling operation; F is the polarization information F pol ; F' is the polarization characteristic representation of the enhanced space F' pol .

[0029] The cross-domain demand query module CDDQ includes a spatial demand graph delivery block SDMD and a channel weight dynamic allocation block CWDA;

[0030] The spatial demand graph transfer block SDMD includes channel self-attention layer, spatial self-attention layer, average pooling layer and convolution layer, through RGB information F rgb Polarization characteristics after processing by the material perception memory module Calculating a spatial attention map, and querying and responding to features in the polarization processing branch through the spatial attention map; Including F′ pol and F″ pol ;

[0031] RGB information F rgb Polarization information processed by the material perception memory module Input to the spatial demand map transfer block SDMD, RGB information F rgb Obtain channel features through the channel self-attention layer F rgb and After the multiplication operation, the spatial features are obtained through the spatial self-attention layer. F rgb and The multiplication value of After multiplication, the same as F rgb After addition Input to the channel weight dynamic allocation block CWDA;

[0032] Spatial characteristics After the average pooling layer and the convolution layer, the polarization feature processed by the material perception memory module is multiplied, and the multiplied value is multiplied with the polarization information processed by the material perception memory module. After addition Input to the channel weight dynamic allocation block CWDA;

[0033]

[0034]

[0035]

[0036]

[0037]

[0038] Among them, the superscript ca represents the channel features obtained by the channel self-attention layer; the superscript sa represents the spatial features obtained by the spatial self-attention layer; AP represents the average pooling operation with a kernel size of 3; GMP and GAP represent global maximum pooling and global average pooling, respectively;

[0039] The channel weight dynamic allocation block CWDA includes two global average pooling layers, a fully connected layer and a convolutional layer; and After the output of the global average pooling layer is cascaded, it passes through two fully connected layers and is divided into two channels. After the two channels pass through the activation function, the first channel is connected to Multiply, the second channel and Multiplication; the multiplication value of the two channels is cascaded and input into the convolution layer to obtain the weight fusion F fused ;

[0040]

[0041]

[0042]

[0043] in, and For the output of the SDMD block, α and β are assigned to and The two weights are processed by the fully connected layer and constrained by the Softmax function to be non-negative values ​​whose sum is 1 at the channel position; δ is the softmax function.

[0044] Beneficial results of this invention: This is the first method to simultaneously utilize RGB and three-color AoLP / DoLP cues to achieve vehicle detection in challenging traffic scenarios. This method leverages the unique ability of polarization modalities to perceive vehicle materials. Through a specially designed material perception memory module and a dynamic weighting mechanism for multimodal feature fusion, the PCDNet algorithm model solves the problem of vehicle detection in challenging traffic scenarios.

[0045] This paper constructs the first large-scale vehicle detection dataset containing RGB and its corresponding three-color AoLP and DoLP images. This dataset covers images in various weather, lighting, and traffic scenarios, presents a diverse and challenging image quality, and boasts accurate human annotation results. This ensures the dataset's usability and effectiveness for algorithm training.

[0046] As the first method to utilize both RGB and polarization information in vehicle detection, this invention addresses the challenging challenge of detecting vehicle instances in degraded scene conditions. Leveraging polarization information to reflect the material properties of each object in the scene and high-dimensional feature mapping from a deep neural network, the method learns and records the unique polarization behavior of vehicle regions and dynamically combines the RGB and polarization modalities to generate comprehensive, comprehensive, and reliable contextual semantic features. Comparative experiments demonstrate the advantages of PCDNet in detection accuracy and speed, and further visualization solutions demonstrate its outstanding performance under various adverse conditions and in densely populated vehicle scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 (a) is a PCDNet network structure diagram of the present invention;

[0048] FIG1( b ) is a diagram of the PI network structure of the polarization integration module of the present invention;

[0049] FIG1( c ) is a network structure diagram of the material space perception memory module MSPM of the present invention;

[0050] Figure 1(d) is a network structure diagram of the material channel perception memory module MCPM of the present invention;

[0051] FIG1(e) is a network structure diagram of the cross-domain demand query module CDDQ of the present invention.

[0052] FIG2 shows the results of seven groups of examples of the present invention and the results of comparative experiments;

[0053] Figure 2(a) is an RGB image, and Figure 2(b) is a three-color polarization image.

[0054] Figure 2(c) is a schematic diagram of the Dynamic R-CNN network results;

[0055] Figure 2(d) is a schematic diagram of the Deformable DETR network results;

[0056] Figure 2(e) is a schematic diagram of the YOLO v7 network results;

[0057] Figure 2(f) is a schematic diagram of the DINO network results;

[0058] Figure 2(g) is a schematic diagram of the results of the present invention;

[0059] Figure 2(h) shows the true value. DETAILED DESCRIPTION

[0060] A method for detecting automobile targets based on RGB and polarization modal fusion, the method comprising the following steps:

[0061] Step 1: Build the car detection dataset RGB-PCar

[0062] When constructing the RGB-PCar dataset, the present invention uses a color polarization camera to record cars in different traffic scenarios. The camera can simultaneously capture pixel-aligned polarization data in three color channels (R, G, and B) and four directions (0°, 45°, 90°, 135°), and each car in each image has a corresponding professionally labeled bounding box. The scenes in this dataset cover scenes with different weather conditions (sunny and rainy days), lighting conditions (day and night), traffic environments (indoor / outdoor parking lots and roads), and different vehicle densities. On the one hand, it ensures the diversity and challenge of the dataset, and on the other hand, it also ensures the effectiveness of the algorithms based on these data. When training the deep learning algorithm network model, the present invention randomly divides the RGB-PCar dataset into a training set and a test set.

[0063] Step 2: Build the polarization integration module PI

[0064] Because AoLP and DoLP capture two aspects of the description simultaneously, it is necessary to build a polarization integration module (PI) to integrate the two into a comprehensive and integrated polarization description. The PI module generates feature maps and edge enhancement information based on DoLP and filters the corresponding AoLP information to enhance the distinction between regions of different materials. The PI module can be expressed as:

[0065]

[0066]

[0067] Here, φ and ρ represent AoLP and DoLP respectively. denotes a convolution with kernel size k×k and stride s, followed by batch normalization and SiLU activation. [·] refers to concatenation along the channel dimension. MP is max pooling with kernel size 5. Operation E refers to edge extraction using the Scharroperator. Pixel-level multiplication operation. avg and max refer to the average and maximum functions, respectively. σ refers to the sigmoid activation function. The PI module integrates the two aspects of polarization information (AoLP and DoLP) into a unified and complete description.

[0068] Step 3: Build the material perception memory module MPM

[0069] Polarization features can effectively describe the inherent physical properties of a car, such as surface structure, roughness, and material. The present invention constructs a material memory module (Material Perception Memory, MPM) to learn and memorize the polarization and material properties of the car across samples in the entire data set. In the early stage of the algorithm network model, a special material spatial perception memory (Material Spatial Perception Memory, MSPM) module is designed to learn and memorize the differences between different materials in a spatial dimension where more information is distributed. The module contains continuous convolution and deconvolution (transposed convolution) operations with a step size of 2; and in the later stage of the model, the present invention uses a special material channel perception memory (Material Channel Perception Memory, MCPM) module to learn the connection between different material areas in the channel dimension. In addition to convolution and deconvolution operations, this module also contains continuous fully connected operations. The MPM module can be generally expressed as:

[0070]

[0071]

[0072]

[0073] in, represents deconvolution with kernel size k×k, stride s, followed by batch normalization and SiLU activation function. These are the memory matrices in two independent fully connected layers, named m1 and m2. GAP is the global average pooling operation. The MPM module learns the differences and connections between regions of different materials across all training samples, forming a characteristic memory of the vehicle instance.

[0074] Step 4: Build a cross-domain demand query module

[0075] In order to avoid the dilution of powerful clues when simply fusing multi-domain information, the strategy for multimodal feature fusion should be context-sensitive. The present invention constructs a Cross-Domain Demand Query (CDDQ) module to solve the problem of robust feature fusion that considers the quality differences of multi-domain features under different environmental conditions. The CDDQ module consists of a Spatial Demand Map Delivery (SDMD) block and a Channel Weight Dynamic Assignment (CWDA) block.

[0076] The SDMD block enhances the features in the RGB domain through multi-dimensional self-attention and uses its spatial attention map to initiate queries and responses to the feature data in the polarization processing branch to return and supplement the necessary contextual semantic features. The formula description of this part can be expressed as follows:

[0077]

[0078]

[0079]

[0080]

[0081]

[0082] The superscripts ca and sa indicate that feature F is the feature to which channel self-attention and spatial self-attention are applied, respectively. AP represents the average pooling operation with a kernel size of 3. GMP and GAP represent global maximum pooling and global average pooling, respectively.

[0083] The CWDA block receives the self-attentioned RGB features and the polarization features that respond to the RGB feature query. These two features are then globally average-pooled to obtain a channel-wise representation of the information from both modalities. A fully connected layer then associates a single channel with all other channels (including feature channels from adjacent modalities) and generates fusion weights for each channel, achieving multimodal data fusion with its corresponding weights.

[0084]

[0085]

[0086]

[0087] in, and is the output of the SDMD block, α and β are the weights assigned to the RGB features and polarization features respectively. After being processed by the fully connected layer, the two weights are constrained by the Softmax function (δ) to be non-negative values ​​whose sum is 1 at the channel position.

[0088] The role of the CDDQ module is to dynamically fuse multimodal features in a query-response manner to generate robust and complete cross-domain contextual semantic feature representations.

[0089] Step 5: Build a car detection network PCDNet based on material polarization property perception

[0090] The present invention integrates the local modules constructed in the above steps into an overall vehicle detection network model, PCDNet. During the feature extraction phase, this network model consists of two branches that process RGB and polarization information in parallel. The inputs are RGB and its corresponding AoLP and DoLP images, respectively. The AoLP and DoLP images are integrated into a unified and complete polarization information description through the PI module. The RGB and polarization information are then fed into two similarly structured Darknet53 encoders to extract multi-level features. Starting from the third level, the polarization features are fed into the MPM module to learn and memorize the distribution of vehicle instances in the training samples, recording the differences and connections in the polarization properties of different material regions. The RGB features then guide the polarization features, obtaining compensation from the polarization modality through a query-response approach and collaborating with the RGB features themselves to map the correlation of all channels in the channel dimension to assign weights for multimodal feature fusion. These weights guide the fusion of features on the two branches, and the fused features serve as the output of feature extraction for the current level. Finally, the third and subsequent levels of features are fed into the FPN structure to exchange semantic features at different levels, and are then fed into the detection head to generate the final detection results.

[0091] Step 6: Vehicle inspection result verification

[0092] In order to verify the effectiveness of PCDNet in detecting car instances, this paper compares it with other SOTA methods in the field of target detection. For fairness in comparison, these methods all use their public codes and parameter settings recommended in the paper, and are retrained on the RGB-PCar dataset. They are also tested on the test set, and the average precision (AP) value of the detection results is statistically analyzed for unified comparison. Related methods include: FasterR-CNN, SSD, CascadeR-CNN

[15] , CornerNet, FCOS, DoubleHeadR-CNN, DynamicR-CNN, EfficientDet, VarifocalNet, DeformableDETR, DDOD, TOOD, YOLOX, YOLOv7, RTMDet, YOLOv8 and DINO.

[0093] The RGB-PCar vehicle detection dataset with polarization information used in this invention contains 2,601 RGB images captured from various traffic scenarios under different conditions, along with corresponding linear polarization angle and degree images. Each image is manually annotated with a bounding box representing the position and size of the vehicle in the image. The dataset is randomly divided into a training set of 1,611 images and a test set of 990 images. A deep neural network algorithm model is iteratively optimized on the training set, and the method's performance and effectiveness are verified on the test set. The network parameters are randomly initialized, with the feature extraction parameters initialized using a pretrained Darknet53 network.

[0094] In the PCDNet network, AoLP and DoLP images first pass through the polarization integration module to obtain a complete and comprehensive polarization feature representation. The RGB image and the integrated polarization feature are respectively sent to two multi-level feature extraction branches. The memory module learns and memorizes the features of the polarization mode from the third-level features to obtain the connection between the polarization properties of the car material and the difference with the polarization properties of different object materials. The cross-domain demand query module supplements and improves the multimodal feature representation from the third-level features from RGB features to polarization features in a request-response manner and globally allocates dynamic fusion weights in the channel dimension. Finally, the fused multimodal features are sent to the subsequent feature pyramid and detection head to obtain the final detection results. PCDNet is implemented based on PyTorch and trained for three hundred rounds with a data batch size of 32 on two NVIDIA GeForce RTX 3090 graphics processing units. The present invention uses a stochastic gradient descent algorithm when training the algorithm model, with a momentum set to 0.937 and a weight decay of 5×10 -4 , the initial learning rate is 0.01, and the cosine annealing weight size is 0.001.

[0095] Figure 1(a) illustrates the PCDNet network architecture. Overall, it has three inputs: RGB, AoLP, and DoLP. It consists of two feature extraction branches that process RGB and polarization information, respectively; a material-aware memory module in the polarization branch; a cross-domain demand query module spanning the two branches; a feature pyramid; and a detection head. Figure 2(b)-Figure 2(e) The implementation details of the polarization integration module, the two material-aware memory modules, and the cross-domain demand query module are presented. As shown in Figure 1, the PCDNet feature extraction stage consists of two branches that process RGB and polarization information in parallel. The inputs are RGB and its corresponding AoLP and DoLP images, respectively. The AoLP and DoLP images are integrated into a unified and complete polarization description by the PI module. The RGB and polarization information are then fed into two similarly structured Darknet53 encoders to extract multi-level features. Starting from the third level, the polarization features are fed into the MPM module to learn and memorize the distribution of car instances in the training samples, recording the differences and connections in the polarization properties of different material regions. The RGB features then guide the polarization features. Through a query-response approach, compensation is obtained from the polarization modality and the RGB features themselves are used to map the correlations across all channels in the channel dimension to assign weights for multimodal feature fusion. These weights guide the fusion of features from the two branches, and the fused features serve as the output of the feature extraction for the current level. Finally, the third and subsequent levels of features are fed into the FPN architecture to exchange semantic features at different levels and are then fed into the detection head to generate the final detection results.

[0096] Figure 2 shows the results of the embodiment of the present invention and the comparative experimental results. Figure 2(g) shows the output results of the algorithm of the present invention, and Figure 2(h) shows the manually labeled ground truth (GT) results. Figure 2(c)-Figure 2(f) The results of other advanced methods (Dynamic R-CNN, DeformableDETR, YOLOv7, and DINO) are presented to verify the effectiveness of car detection based on material polarization properties. To ensure fair comparison, the methods presented here use the publicly available code and parameter settings recommended in the paper. All methods are retrained on the RGB-PCar dataset and tested on the test set.

[0097] Table 1 Quantitative results of the present invention and comparative experiments

[0098]

[0099]

[0100] Table 1 is a quantitative comparison of the experimental results of the present invention and the comparative experimental results. The table uses the average precision (AP) to indicate the accuracy of the detection method. AP is a model evaluation indicator widely used in the field of target detection. Specifically, it refers to the area under the PR (precision-recall) curve drawn at different thresholds. The higher the AP value, the better the performance of the model. The table uses the average time (unit: milliseconds) taken for one inference of the algorithm model under the same conditions to measure the running speed of the detection method. It can be seen from the table that the detection accuracy of the PCDNet of the present invention is the highest, and the detection speed is second only to the YOLOv7 method.

Claims

1. A vehicle target detection method based on RGB and polarization mode fusion, characterized in that: Establish the PCDNet network. The car detection dataset RGB-PCar includes RGB information, AoLP information, and DoLP information, which is input into the PCDNet network. The PCDNet network includes a polarization integration module (PI), two feature extraction branches, a material perception memory module (MPM), a cross-domain demand query module (CDDQ), a feature pyramid (FPN), and a detection head (Head). AoLP information and DoLP information are integrated into polarization information by the polarization integration module (PI). The two feature extraction branches are a polarization processing branch and an RGB processing branch, respectively, for extracting polarization features and RGB features. After extracting multi-level polarization features through the feature extraction branch, the polarization features are passed through the Material Perception Memory Module (MPM) to capture the differences and connections between polarization properties of different material regions. The RGB features are then used to guide the polarization features through the Cross-Domain Demand Query (CDDQ) module. This query-response approach compensates for the polarization features and coordinates the RGB features themselves to map the correlations across all channels in the channel dimension, assigning weights for multimodal feature fusion. Finally, the features passed through the CDDQ module are fed into the feature pyramid FPN structure to exchange semantic features at different levels, and are then fed into the detection head to generate the final detection results. The construction process of the RGB-PCar vehicle detection dataset is as follows: A color polarization camera is used to record cars in different traffic scenes. The color polarization camera simultaneously captures pixel-aligned polarization data in three color channels and four directions, and each car in each image has a corresponding professionally labeled bounding box; the three color channels are R, G and B; the four directions are at angles of 0°, 45°, 90°, and 135° to the plane where the camera is located; the scenes in the car detection dataset include different weather conditions, lighting conditions, traffic environments, and vehicle density scenes; the polarization integration module PI includes a maximum function, an average function, a cascade operation layer, and an edge function. The DoLP information is cascaded with the values ​​obtained by the average function and the maximum function and then input into the first convolutional layer. After the DoLP information passes through the second convolutional layer and the maximum pooling layer, it is added to the output of the first convolutional layer and multiplied with the AoLP information before being input into the first 3×3 convolutional layer. The original value of the DoLP information is added to the output of the boundary extraction operation layer and input into the second 3×3 convolutional layer. The outputs of the first 3×3 convolutional layer and the second 3×3 convolutional layer are cascaded and input into the third 3×3 convolutional layer to obtain the polarization information F. pol ; The PI module is expressed as: in, represents AoLP information, ρ represents DoLP information; denotes a convolutional layer with a kernel size of k×k, stride s, batch normalization, and SiLU activation function; [·] refers to the cascade operation in the channel dimension; MP is the maximum pooling with a kernel size of 5; E represents the edge extraction operation using the Schur operator; Represents pixel-level multiplication operation; avg and max refer to the average function and the maximum function respectively; σ refers to the sigmoid activation function; the material perception memory module MPM includes a material space perception memory module MSPM and a material channel perception memory module MCPM; The material space perception memory module MSPM includes a continuous convolution layer and a deconvolution layer with a step size of 2; the polarization information F pol Input into the material space perception memory module MSPM, and judge the connection and difference of each area in the spatial dimension through the memory matrix in the spatial perception memory module to obtain the polarization feature representation F′ of the enhanced space pol ; The material channel perception memory module MCPM includes a convolution operation, a deconvolution operation, a global average pooling layer, and two fully connected layers. The polarization feature representation of the enhanced space is input into the material channel perception memory module MCPM. The connection and difference of each region in the channel dimension are judged by the memory matrix in the material channel perception memory module to obtain the polarization feature representation F″ of the enhanced channel. pol ; The MPM module is represented as: in, Represents deconvolution with kernel size k×k, stride s, batch normalization, and SiLU activation function; are the memory matrices in the two fully connected layers, named m1 and m2 respectively; GAP is the global average pooling operation; F is the polarization information F pol ; F' is the polarization characteristic representation of the enhanced space F' pol ; The cross-domain demand query module CDDQ includes a spatial demand graph transfer block SDMD and a channel weight dynamic allocation block CWDA; The spatial demand graph transfer block SDMD includes channel self-attention layer, spatial self-attention layer, average pooling layer and convolution layer, through RGB information F rgb Polarization characteristics after processing by the material perception memory module Calculating a spatial attention map, and querying and responding to features in the polarization processing branch through the spatial attention map; Including F′ pol and F″ pol ; RGB information F rgb Polarization information processed by the material perception memory module Input to the spatial demand map transfer block SDMD, RGB information F rgb Obtain channel features through the channel self-attention layer F rgb and After the multiplication operation, the spatial features are obtained through the spatial self-attention layer. F rgb and The multiplication value of After multiplication, the same as F rgb After addition Input to the channel weight dynamic allocation block CWDA; Spatial characteristics After the average pooling layer and the convolution layer, the polarization feature processed by the material perception memory module is multiplied, and the multiplied value is multiplied with the polarization information processed by the material perception memory module. After addition Input to the channel weight dynamic allocation block CWDA; Among them, the superscript ca represents the channel features obtained by the channel self-attention layer; the superscript sa represents the spatial features obtained by the spatial self-attention layer; AP represents the average pooling operation with a kernel size of 3; GMP and GAP represent global maximum pooling and global average pooling, respectively; The channel weight dynamic allocation block CWDA includes two global average pooling layers, a fully connected layer and a convolutional layer; and After the output of the global average pooling layer is cascaded, it passes through two fully connected layers and is divided into two channels. After the two channels pass through the activation function, the first channel is connected to Multiply, the second channel and Multiplication; the multiplication value of the two channels is cascaded and input into the convolution layer to obtain the weight fusion F fused ; in, and For the output of the SDMD block, α and β are assigned to and The two weights are processed by the fully connected layer and constrained by the Softmax function to be non-negative values ​​whose sum is 1 at the channel position; δ is the softmax function.

Images