Road target detection method and device based on bimodal feature fusion and weak light enhancement

Abstract

The invention discloses a road target detection method and device based on bimodal feature fusion and weak light enhancement, and relates to the field of target detection. And the weak light enhancement target detection model performs enhancement and target detection on the road image under the weak light condition. And the bimodal target detection model performs target detection on the road image under the normal light condition. The weak light enhancement target detection model carries out feature extraction through a multi-scale feature attention module, and carries out image feature brightness enhancement through a brightness enhancement module. The image quality is improved, the image contrast is kept, and the target detection accuracy of the weak light image is greatly improved. The bimodal target detection model adopts a double-branch training strategy, the rich capture capability of visible light on target color and texture features under a normal illumination condition is fully exerted, the advantage of a target contour is highlighted in combination with the penetration characteristic of infrared light, and the target recognition capability under the conditions of complex background, shielding and long-distance blurring is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and more specifically, to a method and apparatus for road target detection using dual-modal feature fusion and low-light enhancement. Background Technology

[0002] With the accelerating pace of urbanization and the continuous expansion of urban areas, the number of vehicles on the road has exploded. Frequent traffic congestion and accidents have placed a heavy burden on urban management and traffic control, making it a hot research topic and attracting widespread attention. Intelligent transportation systems can effectively alleviate urban traffic congestion.

[0003] With the development and advancement of artificial intelligence, computer vision, and deep learning technologies, deep learning, with its powerful capabilities in automatic feature extraction and model building, has made the processing of massive traffic data in intelligent transportation systems increasingly efficient and accurate. Road object detection is one of the key technologies in the field of intelligent transportation. It can provide drivers with real-time information on road conditions, promptly alerting them to the distance to vehicles ahead, pedestrian movements, and potential road obstacles, thereby significantly reducing the probability of traffic accidents and effectively protecting the lives and property of pedestrians and drivers.

[0004] In practical applications, complex road scenes can bring many thorny problems to target detection. For example: (1) The background elements of real roads are complex and diverse. Background objects such as billboards and trees have similar features to target objects, which can easily interfere with the detection model's accurate identification of the target, leading to false detection or missed detection. Moreover, when the target is partially occluded, whether by other vehicles, pedestrians, or road facilities, the detection model often cannot fully acquire the target features, further increasing the detection difficulty and making it difficult to accurately determine the target's category, location, and state. (2) In low light environments, the quality of visible light imaging is severely limited, the target details are blurred, color information is lost, and traditional visible light-based detection methods are difficult to play a role, resulting in a significant decrease in detection accuracy. Even in cloudy or overcast weather conditions, the uneven distribution of light will cause the target to exhibit different degrees of brightness, increasing the recognition difficulty of the detection model. Summary of the Invention

[0005] In view of this, the present invention provides a road target detection method and apparatus with dual-modal feature fusion and low-light enhancement, which is used to solve the problem of low target detection accuracy of existing road target detection methods in complex backgrounds, occlusion and low-light environments.

[0006] To achieve the above objectives, the following solution is proposed:

[0007] A road target detection method based on dual-modal feature fusion and low-light enhancement includes:

[0008] Acquire road images and infrared images;

[0009] Determine whether the road image is a low-light image;

[0010] If so, the road image is input into a pre-trained low-light enhanced target detection model for target detection; the low-light enhanced target detection model includes a multi-scale feature attention module, a brightness enhancement module, and a target detection network; the brightness enhancement module performs nonlinear transformation on the image;

[0011] If not, the road image and infrared image are input into a pre-trained bimodal target detection model for target detection; the bimodal target detection model includes a feature extraction module and a feature fusion module; the feature extraction module has a dual-branch structure, which extracts features from the road image and the infrared image respectively, and the feature fusion module performs feature fusion through a BiFPN module and a C2PSA module.

[0012] Preferably, the C2PSA module includes a convolutional layer and a PSA module;

[0013] The BiFPN module includes learnable weight parameters.

[0014] Preferably, the feature fusion process using the BiFPN module and the C2PSA module includes:

[0015] The BiFPN module includes multiplying the features of different modalities element-wise with the weight parameters to obtain an initial fused feature map;

[0016] The C2PSA module extracts local features from the initial fused feature map through convolutional layers;

[0017] The PSA module performs weighted combination of local features based on their spatial location and importance to the current task to obtain the target fusion feature map.

[0018] Preferably, the process by which the PSA module performs a weighted combination of local features based on their spatial location and importance to the current task includes:

[0019] The SENetV2 module is used to adjust the channel attention of local features to adaptively calibrate the channel weights of local features.

[0020] The target local features are obtained by performing nonlinear transformation on the local features through a feedforward network.

[0021] The local features of each target are stitched together by convolutional layers to obtain a target fusion feature map.

[0022] Preferably, the dual-modal target detection model is trained based on the Focaler-ShapeIoU loss function, which is expressed as follows:

[0023] L Shape-IoU =1-IoU+distance shape +0.5×Ω shape ;

[0024]

[0025] Where IOU is the traditional intersection-union ratio metric, c is the diagonal length of the minimum bounding box, (x c y c ) represents the true coordinates of the target to be detected, and scale is the scale factor. Let h be the predicted coordinates of the target object, and h and w be the height and width of the ground truth bounding box, respectively. gt w gt These represent the height and width of the predicted bounding box, respectively.

[0026] Preferably, the multi-scale feature attention module includes an Inception module, a Sobel module, and a Laplacian module;

[0027] Preferably, the target detection process performed by the low-light enhanced target detection model includes:

[0028] The Inception module uses convolutional kernels of different sizes and max pooling layers to extract multi-scale features from road images, and then merges them using a concat operation to obtain an initial feature map.

[0029] The Sobel and Laplacian modules extract edge features from road images and enhance and stitch the edge features and initial feature maps through a channel attention mechanism to obtain the merged target feature map.

[0030] The brightness enhancement module segments the input target feature map and enhances each sub-feature block step by step with nonlinear terms to obtain an enhanced road image;

[0031] The object detection network performs object detection based on enhanced road images.

[0032] Preferably, the target detection network performs target detection based on enhanced road images, including:

[0033] The enhanced road image is divided into feature maps of different scales by using progressively downsampled convolutional layers and spatial attention mechanisms;

[0034] The spatial pyramid pooling module is used to fuse feature maps of different scales to obtain a fused feature map.

[0035] The RepVGGBlocks module is used to optimize the features of the fused feature map to obtain an optimized feature map.

[0036] The detection module performs predictions based on the optimized feature map.

[0037] Preferably, the low-light enhanced target detection model is trained based on a comprehensive loss function, which is:

[0038] L total =L spa +L col +L mae +L perceptual ;

[0039]

[0040]

[0041] L perceptual =||Φ ij (I pred )-Φ ij (I gt )||1;

[0042] Where K is the number of pixel blocks, Ω(i) represents the set of pixel blocks in the four neighboring regions of pixel i, and Y i and Y j O represents the pixel values ​​at the i-th and j-th pixels in the original image, respectively. i and O j These represent the pixel values ​​at the i-th and j-th pixels in the output image, respectively; J p and J q ε represents the average intensity of the p-th and q-th channels in the image, respectively; ε is the set of color channel pairs, including three combinations: red-green (R, G), red-blue (R, B), and green-blue (G, B); n represents the total number of pixels in the image; Φ ij This represents the feature extraction function obtained, with subscripts i and j used to identify the specific positions in the feature mapping. pred To predict the image, I gt For the real image, ||·|| represents the L1 norm, and represents the sum of the absolute differences between the two feature maps.

[0043] A road target detection device with dual-modal feature fusion and low-light enhancement includes:

[0044] Image acquisition unit, used to acquire road images and infrared images;

[0045] An image determination unit is used to determine whether a road image is a low-light image;

[0046] The low-light image detection unit is used to input the road image into a pre-trained low-light enhancement target detection model for target detection when the road image is a low-light image. The low-light enhancement target detection model includes a multi-scale feature attention module, a brightness enhancement module, and a target detection network. The brightness enhancement module performs nonlinear transformation on the image.

[0047] A dual-modal image detection unit is used to input road images and infrared images into a pre-trained dual-modal target detection model for target detection when the road image is a normal light image. The dual-modal target detection model includes a feature extraction module and a feature fusion module. The feature extraction module has a dual-branch structure and extracts features from the road image and the infrared image respectively. The feature fusion module performs feature fusion through a BiFPN module and a C2PSA module.

[0048] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0049] This invention provides a road target detection method based on dual-modal feature fusion and low-light enhancement. It enhances road images and detects targets in low-light conditions using a low-light enhanced target detection model. The low-light enhanced target detection model includes a multi-scale feature attention module, a brightness enhancement module, and a target detection network; the brightness enhancement module performs nonlinear transformations on the image. This invention uses the multi-scale feature attention module for feature extraction to restore clear details of the target. Simultaneously, the brightness enhancement module enhances the brightness of image features, effectively improving the quality of low-light images while maintaining image contrast, thus significantly improving the accuracy of road target detection in low-light images.

[0050] This invention provides a road target detection method with dual-modal feature fusion and low-light enhancement. The method inputs road images and infrared images into a pre-trained dual-modal target detection model for target detection. The dual-modal target detection model has a two-branch structure, extracting features from the road image and infrared image separately, and then fusing these features using a BiFPN module. The dual-modal target detection model of this invention employs a two-branch training strategy, fully leveraging the rich ability of visible light to capture target color and texture features under normal lighting conditions. Simultaneously, it combines the penetrating characteristics of infrared light to highlight target contours. These two modalities are used for feature extraction separately, significantly improving target recognition capabilities in complex backgrounds, occlusion, and long-distance blurring conditions. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1 This invention provides a method for road target detection using dual-modal feature fusion and low-light enhancement.

[0053] Figure 2 This is a schematic diagram of the network structure of a low-light enhanced target detection model provided in an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of a multi-scale feature attention module structure provided in an embodiment of the present invention;

[0055] Figure 4 This is a schematic diagram of a dual-modal target detection model structure provided in an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of a brightness enhancement module structure provided in an embodiment of the present invention;

[0057] Figure 6 This is a schematic diagram of a spatial pyramid pooling module structure provided in an embodiment of the present invention;

[0058] Figure 7 A schematic diagram of an RCRep2A module network structure is provided in an embodiment of the present invention;

[0059] Figure 8 A flowchart of RepVGGBlock module reparameterization provided in this embodiment of the invention;

[0060] Figure 9 This is a schematic diagram of a C2f-SCConv module structure provided in an embodiment of the present invention;

[0061] Figure 10 This is a schematic diagram of a BiFPN weighted fusion process provided in an embodiment of the present invention;

[0062] Figure 11 This is a schematic diagram of a C2SPA-SENetV2 module structure provided in an embodiment of the present invention;

[0063] Figure 12 This is a schematic diagram of the Focaler-Shape IoU loss function calculation provided in an embodiment of the present invention;

[0064] Figures 13a-13bThe comparison results of the PR curves of each model provided in the embodiments of the present invention on different datasets are shown respectively.

[0065] Figure 14 This is a schematic diagram of a road target detection device with dual-modal feature fusion and low-light enhancement provided in an embodiment of the present invention. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] First, combined Figure 1 The road target detection method with dual-modal feature fusion and low-light enhancement provided in the embodiments of the present invention will be introduced, such as... Figure 1 As shown, the method includes:

[0068] Step S01: Acquire road images and infrared images.

[0069] Step S02: Determine whether the road image is a low-light image.

[0070] Specifically, the OpenCV library can be used to read the dataset images for data preprocessing, convert the images to grayscale, and calculate the average brightness of the images to determine whether they are low-light images. When the average brightness is lower than a preset brightness threshold, the road image is considered a low-light image.

[0071] If the road image is a low-light image, then proceed to step S03 below; if the road image is not a low-light image, then proceed to step S04 below.

[0072] Step S03: Input the road image into the pre-trained low-light enhanced target detection model for target detection.

[0073] Specifically, to address the problem of feature loss in low-light images, a low-light enhancement target detection model for low-light image enhancement is proposed, such as... Figure 2 As shown, the low-light enhanced target detection model mainly consists of a Fast Scale InceptNet (FSINet) and a lightweight target detection network. The Fast Scale InceptNet includes a Multi-Scale Feature Attention (MFAM) module and a brightness enhancement module. The brightness enhancement module performs a non-linear transformation on the image. The Multi-Scale Feature Attention module includes an Inception module, a Sobel module, and a Laplacian module.

[0074] In fast multi-scale residual networks, the main structure is composed of multiple overlapping multi-scale feature attention modules (MFAM). For example... Figure 3 As shown, the core of the MFAM module is the Inception module. The Inception module extracts features at different scales through multiple parallel branches, which is used to enhance the feature representation of low-light images and improve the performance of the network.

[0075] The Fast Multi-Scale Residual Network (FML) first receives a low-light image as input. It then extracts multi-scale features using the Inception module, extracts edge features using Sobel and Laplacian operators, and further enhances important features through a channel attention mechanism. Next, a brightness enhancement module uses an iterative function and MFAM to enhance the brightness and contrast of the merged feature map. Finally, an object detection network performs object detection based on the enhanced feature map.

[0076] Step S04: Input the road image and infrared image into the pre-trained dual-modal target detection model for target detection.

[0077] Specifically, the YOLO11 structure is improved to construct a dual-modal target detection model based on the BFF-YOLO network. For example... Figure 4 As shown, BFF-YOLO mainly consists of three parts: Backbone, Neck, and Head. The feature extraction module (Backbone part) is responsible for extracting basic features from the input image. Since the training of the bimodal object detection model requires learning from both modalities, the Backbone structure of the bimodal object detection model is designed as two parallel dual-branch structures, with each branch performing feature extraction on images of different modalities.

[0078] During the mid-stage fusion phase of the model, the feature fusion module utilizes a combination of the BiFPN feature fusion module and the C2SPA-SENetV2 module to perform feature fusion, thereby enhancing the integration effect of multimodal information. Simultaneously, an inverted residual mobile block is introduced to enhance the model's ability to capture key features.

[0079] The road target detection method with dual-modal feature fusion and low-light enhancement provided in this invention enhances and detects targets in road images under low-light conditions using a low-light enhanced target detection model. The low-light enhanced target detection model includes a multi-scale feature attention module, a brightness enhancement module, and a target detection network; the brightness enhancement module performs nonlinear transformations on the image. This invention uses the multi-scale feature attention module for feature extraction to restore clear details of the target. Simultaneously, the brightness enhancement module enhances the brightness of image features, effectively improving the quality of low-light images while maintaining image contrast, thus significantly improving the accuracy of road target detection in low-light images.

[0080] The road target detection method with dual-modal feature fusion and low-light enhancement provided in this invention inputs road images and infrared images into a pre-trained dual-modal target detection model for target detection. The dual-modal target detection model has a two-branch structure, extracting features from the road image and the infrared image separately, and then fusing the features through a BiFPN module. The dual-modal target detection model of this invention employs a two-branch training strategy, fully leveraging the rich ability of visible light to capture target color and texture features under normal lighting conditions, while simultaneously combining the penetrating characteristics of infrared light to highlight target contours. These two modalities are used for feature extraction separately, significantly improving target recognition capabilities in complex backgrounds, occlusion, and long-distance blurring conditions.

[0081] Next, in conjunction with the embodiments of the present invention Figure 2-3 This paper introduces a target detection model for enhanced low-light performance.

[0082] (1) Multi-scale feature attention module

[0083] First, the input low-light image is processed by the Inception module using convolutional kernels of different sizes (1x1, 3x3, 5x5) and a max-pooling layer (3x3 MaxPool2d) to extract multi-scale features of the road image, which are then merged using a concat operation to obtain an initial feature map. After entering the Inception module, the low-light image is split into four parallel branches, each responsible for extracting feature information of different scales or types. Specifically, the 1x1 convolutional branch extracts low-order features using small convolutional kernels while reducing dimensionality and preserving key information; the 3x3 convolutional branch first performs dimensionality reduction using 1x1 convolutions, then uses 3x3 convolutional kernels to capture local texture and details, extracting medium-scale features from the image; the 5x5 convolutional branch further extracts a wider range of structural information using larger convolutional kernels, capturing macroscopic features from the image; and the pooling branch extracts global features through max-pooling operations and adjusts channels through 1x1 convolutions to enhance the perception of global information. Each branch introduces non-linear characteristics through the LeakyReLU activation function, further enhancing the expressive power of the features. The outputs of these four branches are concatenated along the channel dimension using Concat to form an initial feature map that incorporates multi-scale information.

[0084] Then, the Sobel and Laplacian modules perform edge detection to extract edge features from the road image. The Laplacian module detects second-derivative edges in the image using the Laplacian operator, which can highlight sharp changes and details in the image; the Sobel module detects first-derivative edges in the horizontal and vertical directions using the Sobel operator, capturing gradient changes in the image.

[0085] Finally, the edge features extracted by the two modules are compared with the initial features extracted by the Inception module. Figure 1 The system then enters the Channel Attention (CA) module. The channel attention mechanism enhances and concatenates edge features and the initial feature map to obtain the merged target feature map. Specifically, dynamic channel attention weights are generated through global average pooling and one-dimensional convolution to weight the channels of the feature map, highlighting important information and suppressing redundant information, thereby achieving refined feature enhancement.

[0086] (2) Brightness enhancement module

[0087] The brightness enhancement module in the FSINet network enhances the image by performing multiple nonlinear transformations and feature adjustments. For example... Figure 5As shown, the brightness enhancement module segments the input target feature map into 8 sub-feature blocks (r1~r8) according to the channel dimension, with each sub-feature block having 9 channels. Then, starting from the initial input, feature enhancement is performed on the sub-feature blocks through a step-by-step iterative process. Each enhancement operation involves multiplying the result of the previous step with the first 3 channels of the current residual block and incorporating a nonlinear term to obtain the enhanced road image. The specific mathematical formula for step-by-step enhancement is expressed as follows:

[0088]

[0089] For the update of the i-th layer, Indicates from r i The first three channels extracted, namely Indicates the relationship between y i-1 Each element is squared element by element, and ⊙ represents element-wise multiplication.

[0090] (3) Target Detection Network

[0091] Object detection in enhanced road images is performed using the lightweight road object detection network LODNet. The network employs a hierarchical feature extraction structure, dividing the input image into feature maps of different scales through progressively downsampled convolutional layers, thereby capturing feature information from global to local perspectives. At the highest layer, a Spatial Pyramid Pooling module (SPPFLite) is introduced to fuse multi-scale features into a fused feature map, significantly improving the detection capability for objects of different sizes.

[0092] like Figure 6 As shown, SPPFLite employs smaller pooling kernels and replaces traditional large pooling kernels with multiple parallel pooling operations, which not only reduces redundant computation but also avoids feature loss layer by layer. Furthermore, SPPFLite introduces the lightweight convolution LightConv, further reducing the number of parameters and computational complexity, making the model more suitable for deployment on resource-constrained devices. Through the design of parallel pooling layers, SPPFLite can independently process input features and efficiently fuse features at different scales, thereby reducing computational cost while preserving rich multi-scale information.

[0093] LODNet's feature extraction module introduces the RCRep2A (Repeated Attention Feature Extraction) module, which combines repeated convolution and attention mechanisms. This not only enhances the non-linear expressiveness of features but also dynamically adjusts feature weights, improving the model's sensitivity to key features. The RCRep2A module enhances the model's ability to express features through multi-layer feature extraction and fusion, thereby improving detection accuracy and the model's robustness in different scenarios. Figure 7As shown, the RCRep2A module first extracts features using GhostConv, then further enhances the expressive power of the features through the stacking of multiple Rep2ABlocks, and finally outputs the final features through a Conv layer. RCRep2A captures richer feature information and improves the model's detection accuracy through multi-layer feature extraction and fusion. Furthermore, it enhances the model's ability to fuse different features by leveraging spatial attention and residual connections, maintaining good performance even in complex scenes. In addition, the GhostConv and RepConv modules reduce computational complexity and the number of parameters while maintaining high performance, thus improving the model's efficiency.

[0094] In the feature fusion stage, LODNet aligns and fuses features at different scales using the TriScaleAlign module, and further optimizes the feature representation using upsampling operations and the RepVGGBlocks module to obtain an optimized feature map, ensuring the consistency and integrity of features in the spatial dimension. Finally, the model uses the detection module to predict the fused feature map, outputting detection results including bounding boxes, class labels, and confidence scores.

[0095] like Figure 8 As shown, RepVGGBlocks are modules composed of multiple RepVGGBlocks, each containing multiple convolutional branches, such as 3×3 convolutions, 1×1 convolutions, and identity mappings. These branches provide multi-scale feature learning capabilities during the training phase, enabling the network to capture richer feature information. During the inference phase, these multi-branch structures are fused into a single 3×3 convolutional layer through reparameterization techniques, thereby improving inference efficiency.

[0096] Reparameterization of the RepVGGBlock module offers several advantages, primarily in improving inference speed, saving memory, and simplifying model deployment. During model training, complex multi-branch structures are typically built to enhance the model's expressive power. However, these complex branches incur additional computational burden during inference. Reparameterization technology integrates multi-branch structures into single-branch structures, effectively reducing computation and significantly accelerating inference.

[0097] like Figure 8 As shown, at the structural level, after reparameterization, the RepVGGBlock module is simplified into a single 3x3 convolutional layer. That is, by integrating the parameters of the 1x1 convolutional layer and the batch normalization layer into the 3x3 convolutional layer, the computational cost and the number of parameters are reduced without sacrificing model performance.

[0098] Figure 8The right side shows the parameter reparameterization process. The upper part shows the convolution parameters, including the mean μ, standard deviation σ, scaling factor γ, and offset factor β. The lower part shows the batch normalization (BN) parameters. Through a series of operations, these parameters are finally fused into the parameters of a 3x3 convolution.

[0099] To further improve the model's performance and feature extraction capabilities in real-world road scenarios, the C2f module in YOLOv8 is combined with the spatial and channel reconstruction convolution SCConv, resulting in the following... Figure 9 The C2f-SCConv module is shown. Unlike the Bottleneck module in the traditional C2f structure, the Bottleneck part in the C2f-SCConv module is replaced by an SCConv module, introducing spatial and channel reconstruction mechanisms. In the C2f-SCConv structure, the input features first pass through a convolutional layer, then are divided into multiple branches. The SCConv module in each branch performs feature processing separately. Finally, the processed features are concatenated and convolved to generate the final output.

[0100] The core of SCConv consists of cascaded Spatial Refinement Units (SRUs) and Channel Refinement Units (CRUs). After compressing the feature dimensions generated by the previous convolutional block, the SRU uses a spatial attention mechanism to perform pixel-level weight calibration on the feature map, effectively suppressing interference from complex backgrounds in road scenes. Simultaneously, the CRU uses a channel attention mechanism to dynamically adjust the feature channel weights, enhancing the feature response to key road targets. This cascaded structure achieves bidirectional feature refinement in both the spatial and channel domains.

[0101] (4) Loss Function

[0102] In the field of image enhancement, the loss function is crucial for optimizing model performance. Four loss functions are used during the training of the FSINet network, which together constitute the total loss function to guide the network's training to improve the performance of low-light image enhancement. The comprehensive loss function is as follows:

[0103] L total =L spa +L col +L mae +L perceptual

[0104]

[0105] L perceptual =||Φ ij (I pred )-Φ ij (I gt)||1;

[0106] Among them, L spa For spatial consistency loss, L col For color consistency loss, L mae For the average absolute error loss, L perceptual For perceptual loss; K is the number of pixel blocks, Ω(i) represents the set of pixel blocks in the four neighboring regions of pixel i, Y i and Y j O represents the pixel values ​​at the i-th and j-th pixels in the original image, respectively. i and O j These represent the pixel values ​​at the i-th and j-th pixels in the output image, respectively; J p and J q ε represents the average intensity of the p-th and q-th channels in the image, respectively; ε is the set of color channel pairs, including three combinations: red-green (R, G), red-blue (R, B), and green-blue (G, B); n represents the total number of pixels in the image; Φ ij This represents the feature extraction function obtained, with subscripts i and j used to identify the specific positions in the feature mapping. pred For the predicted image, i.e. the image generated by the model after low-light enhancement, I gt The real image is the original normal light image or reference image. ||·|| represents the L1 norm, which represents the sum of the absolute differences between the two feature maps.

[0107] Next, the performance of the low-light enhanced target detection model was verified in this embodiment of the invention. RetinexNet

[66] , MBLLEN

[67] , TBEFN

[68] , Zero-DCE

[69] , UREtinexNet

[70] , SCI

[76] and RUAS

[75] were selected as comparison models. All models were trained on the LOLv1 dataset. The comparison results are shown in Table 1.

[0108] Table 1

[0109]

[0110]

[0111] As shown in Table 1, a quantitative comparison was made between the low-light enhanced target detection model of this invention and the comparative models, focusing on the key indicator of the total number of network model parameters. The results show that the low-light enhanced target detection model has a significantly lower total number of parameters compared to other comparative models. This advantage enables the model to have faster single-frame image processing capabilities, effectively meeting the practical needs of real-time detection.

[0112] Furthermore, the embodiments of the present invention introduce a dual-modal target detection model, as follows:

[0113] (1) BiFPN module

[0114] BiFPN inherits PANet's bidirectional path and adds skip connections and repeated bidirectional feature fusion operations, allowing feature fusion at the same level to enhance feature representation. Furthermore, as... Figure 10 As shown, the BiFPN module also introduces learnable weight parameters. After normalizing the parameters, the features of different modalities are multiplied element-wise with the weight parameters to obtain the initial fused feature map. The weights of different features are dynamically adjusted according to different task requirements to determine the importance of features at each scale during feature fusion. For example, during periods of good road lighting, the visible light features are dynamically increased, which is beneficial for the model to extract detailed features.

[0115] (2) C2PSA module

[0116] In the original YOLO11 model architecture, the C2PSA module is integrated into the backbone network, following the SPPF module. The C2PSA module consists of convolutional layers and a PSA module. It utilizes an attention mechanism to optimize feature representation. More precise attention management is achieved by stacking multiple Pyramid Slice Attention (PSA) layers. The PSA module acquires local features extracted by the convolutional layers, reweights and combines these features based on their spatial location and importance to the current task, resulting in a target fusion feature map.

[0117] Considering that insufficient feature extraction may occur due to the target being far away, small in size, or occluded, and is easily affected by background interference, leading to missed or false detections, this embodiment of the invention can also integrate the channel attention mechanism of SENetV2 into C2SPA, enabling the model to adaptively adjust the feature weights according to the importance of the channels. Figure 11 As shown, the original convolutional layers, feature segmentation, and PSA module of C2SPA work together with SENetV2 to obtain the C2SPA-SENetV2 module.

[0118] The SENetV2 module performs dimensionality reduction on the input feature map using adaptive average pooling, followed by non-linear transformation through multiple linear layers and the ReLU activation function. Finally, after feature concatenation and processing with the sigmoid activation function, the final output is generated. The process can be expressed in functional form as follows:

[0119] F senetv2 (x)=x⊙σ(MLP(AvgPool(x)));

[0120] Where ⊙ represents element-level multiplication, σ represents the Sigmoid activation function, MLP represents the multilayer perceptron, and AvgPool represents global average pooling.

[0121] The PSABlock module is a core component of C2SPA-SENetV2. After receiving local features extracted from the convolutional layers, it first performs channel attention adjustment through the SENetV2 module. This process adaptively recalibrates the channel weights of the feature map, highlighting channel features that play a crucial role in object recognition. Next, the feature map undergoes a non-linear transformation through a feedforward network (FFN) to further enhance the model's expressive power. Finally, the adjusted feature map is output to the next layer. The process can be expressed as a function:

[0122] F PSA (x)=FFN(F senetv2 (x))+x;

[0123] The C2SPA-SENetV2 module first receives the initial fused feature map as input and performs preliminary feature extraction through a convolutional layer. Subsequently, the data is segmented into multiple parts, which are processed in parallel by multiple PSABlocks. Each PSABlock utilizes Pyramid Spatial Attention (PSA) to capture features at different scales, thereby enhancing the model's ability to recognize multi-scale targets. After n iterations, these features are reassembled and integrated through another convolutional layer to finally obtain the target fused feature map. This process can be expressed as a function: F out (x)=Conv(C(F PSA (x1),F PSA (x2),F PSA (x n ))).

[0124] (3) Loss Function

[0125] In object detection training, the loss function is a crucial component for accurate model recognition, playing an indispensable role in model performance measurement, training guidance, and overfitting control. In road object detection tasks, occluded and small objects are prevalent, often leading to slow convergence and inaccurate regression results during model training. Therefore, a novel loss function, Focaler-ShapeIoU, was used for training. The ShapeIoU loss function optimizes the model's regression accuracy by combining traditional IoU loss, the shape distance of the predicted bounding box, and shape consistency. Figure 12 As shown, the expression for the Focaler-ShapeIoU function is as follows:

[0126] L Shape-IoU =1-IoU+distance shape +0.5×Ω shape ;

[0127]

[0128] Where IOU is the traditional intersection-union ratio metric, c is the diagonal length of the minimum bounding box, (x c y c () represents the true coordinates of the target to be detected. Let h be the predicted coordinates of the target object, and h and w be the height and width of the ground truth bounding box, respectively. gt w gt These represent the height and width of the predicted bounding box, respectively. Scale is a scaling factor that is related to the scale of the targets in the dataset. It can adjust the size of the shape to a uniform range, making the loss function more fair and stable in handling data of different shapes.

[0129] Next, the performance of the bimodal target detection model is verified in this embodiment of the invention. In order to verify the effectiveness of the bimodal target detection model BFF-YOLO of this application, comparative experiments were carried out on the LLVIP and MF3D datasets. All comparative algorithms (including Faster R-CNN

[24] , SSD

[23] , FCOS

[22] and YOLO series) were trained and verified based on visible light single-modal data, while BFF-YOLO achieved feature enhancement through bimodal feature fusion. The results are shown in Table 2.

[0130] BFF-YOLO significantly outperforms the single-modal method across all key metrics on both datasets: On the LLVIP dataset, its Precision (0.945), Recall (0.902), and mAP@0.5 (0.954) are improved by 6.6%, 7.0%, and 6.9% respectively compared to the best single-modal method (YOLO11); on the MF3D dataset, the three metrics (0.861, 0.719, 0.798) also lead the second-place YOLO11 (0.814, 0.683, 0.762). The experimental results clearly demonstrate that the bimodal fusion strategy effectively enhances the model's adaptability to complex scenes through complementary feature extraction, validating the superiority of the BFF-YOLO architecture.

[0131] Table 2

[0132]

[0133] To further verify the robustness and generalization ability of the bimodal BFF-YOLO, this paper dynamically analyzes its performance using precision-recall (PR) curves, such as... Figure 13a As shown. On the LLVIP dataset, BFF-YOLO's PR curve is generally closer to the upper right, indicating that it can maintain a higher precision while maintaining a higher recall. Its significant advantage in area under the curve (AUC) further confirms the leading performance of mAP@0.5 (0.954) in Table 2, highlighting the model's balance between target localization and classification capabilities. Meanwhile, as... Figure 13b As shown, on the more complex MF3D dataset, the PR curve of BFF-YOLO still maintains a smooth and steep downward trend, indicating that the model's precision decays more slowly as recall increases, and it is more adaptable to interference such as occlusion and lighting changes. In contrast, the precision of single-modal methods such as YOLO11 drops sharply after recall exceeds 0.7, exposing the limitations of feature extraction from a single data source. Comprehensive quantitative metrics and dynamic performance analysis show that the dual-modal fusion strategy effectively alleviates the information loss problem of single-modal methods in complex scenes through complementary feature extraction from infrared and visible light data.

[0134] The road target detection device with dual-modal feature fusion and low-light enhancement provided in the embodiments of the present invention will be described below. The road target detection device with dual-modal feature fusion and low-light enhancement described below can be referred to in correspondence with the road target detection method with dual-modal feature fusion and low-light enhancement described above.

[0135] First, combine Figure 14 This paper introduces a road target detection device that combines dual-modal feature fusion and low-light enhancement, such as... Figure 14 As shown, the road target detection device with dual-modal feature fusion and low-light enhancement may include:

[0136] Image acquisition unit 100 is used to acquire road images and infrared images;

[0137] The image judgment unit 200 is used to determine whether the road image is a low-light image;

[0138] The low-light image detection unit 300 is used to input the road image into a pre-trained low-light enhancement target detection model for target detection when the road image is a low-light image; the low-light enhancement target detection model includes a multi-scale feature attention module, a brightness enhancement module, and a target detection network; the brightness enhancement module performs nonlinear transformation on the image;

[0139] The dual-modal image detection unit 400 is used to input the road image and infrared image into a pre-trained dual-modal target detection model for target detection when the road image is a normal light image. The dual-modal target detection model includes a feature extraction module and a feature fusion module. The feature extraction module has a dual-branch structure and extracts features from the road image and the infrared image respectively. The feature fusion module performs feature fusion through a BiFPN module and a C2PSA module.

[0140] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0141] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0142] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for road target detection using dual-modal feature fusion and low-light enhancement, characterized in that, include: Acquire road images and infrared images; Determine whether the road image is a low-light image; If so, the road image is input into a pre-trained low-light enhanced target detection model for target detection; the low-light enhanced target detection model includes a multi-scale feature attention module, a brightness enhancement module, and a target detection network; the brightness enhancement module performs nonlinear transformation on the image; If not, the road image and infrared image are input into a pre-trained bimodal target detection model for target detection; the bimodal target detection model includes a feature extraction module and a feature fusion module; The feature extraction module has a dual-branch structure, which extracts features from road images and infrared images respectively. The feature fusion module performs feature fusion through BiFPN and C2PSA modules.

2. The road target detection method based on dual-modal feature fusion and low-light enhancement according to claim 1, characterized in that, The C2PSA module includes a convolutional layer and a PSA module; The BiFPN module includes learnable weight parameters.

3. The road target detection method based on dual-modal feature fusion and low-light enhancement according to claim 2, characterized in that, The feature fusion process using the BiFPN module and the C2PSA module includes: The BiFPN module includes multiplying the features of different modalities element-wise with the weight parameters to obtain an initial fused feature map; The C2PSA module extracts local features from the initial fused feature map through convolutional layers; The PSA module performs weighted combination of local features based on their spatial location and importance to the current task to obtain the target fusion feature map.

4. The road target detection method based on dual-modal feature fusion and low-light enhancement according to claim 3, characterized in that, The process by which the PSA module weights and combines local features based on their spatial location and importance to the current task includes: The SENetV2 module is used to adjust the channel attention of local features to adaptively calibrate the channel weights of local features. The target local features are obtained by performing nonlinear transformation on the local features through a feedforward network. The local features of each target are stitched together by convolutional layers to obtain a target fusion feature map.

5. The road target detection method based on dual-modal feature fusion and low-light enhancement according to claim 1, characterized in that, The dual-modal target detection model is trained based on the Focaler-ShapeIoU loss function, which is expressed as follows: L Shape-IoU =1-IoU+distance shape +0.5×Ω shape ; Where IOU is the traditional intersection-union ratio metric, c is the diagonal length of the minimum bounding box, (x c y c ) represents the true coordinates of the target to be detected, and scale is the scale factor. Let h be the predicted coordinates of the target object, and h and w be the height and width of the ground truth bounding box, respectively. gt w gt These represent the height and width of the predicted bounding box, respectively.

6. The road target detection method based on dual-modal feature fusion and low-light enhancement according to claim 1, characterized in that, The multi-scale feature attention module includes the Inception module, Sobel, and Laplacian module.

7. The road target detection method based on dual-modal feature fusion and low-light enhancement according to claim 6, characterized in that, The target detection process of the low-light enhanced target detection model includes: The Inception module uses convolutional kernels of different sizes and max pooling layers to extract multi-scale features from road images, and then merges them using a concat operation to obtain an initial feature map. The Sobel and Laplacian modules extract edge features from road images and enhance and stitch the edge features and initial feature maps through a channel attention mechanism to obtain the merged target feature map. The brightness enhancement module segments the input target feature map and enhances each sub-feature block step by step with nonlinear terms to obtain an enhanced road image; The object detection network performs object detection based on enhanced road images.

8. The road target detection method based on dual-modal feature fusion and low-light enhancement according to claim 6, characterized in that, The target detection network performs target detection based on enhanced road images, including: The enhanced road image is divided into feature maps of different scales by using progressively downsampled convolutional layers and spatial attention mechanisms; The spatial pyramid pooling module is used to fuse feature maps of different scales to obtain a fused feature map. The RepVGGBlocks module is used to optimize the features of the fused feature map to obtain an optimized feature map. The detection module performs predictions based on the optimized feature map.

9. The road target detection method based on dual-modal feature fusion and low-light enhancement according to claim 1, characterized in that, The low-light enhanced target detection model is trained based on a comprehensive loss function, which is: L total L spa +L col +L mae +L perceptual ; L perceptual =||Φ ij (I pred )-F ij (I gt )||1; Where K is the number of pixel blocks, Ω(i) represents the set of pixel blocks in the four neighboring regions of pixel i, and Y i and Y j O represents the pixel values ​​at the i-th and j-th pixels in the original image, respectively. i and O j These represent the pixel values ​​at the i-th and j-th pixels in the output image, respectively; J p and J q ε represents the average intensity of the p-th and q-th channels in the image, respectively; ε is the set of color channel pairs, including three combinations: red-green (R, G), red-blue (R, B), and green-blue (G, B); n represents the total number of pixels in the image; Φ ij This represents the feature extraction function obtained, with subscripts i and j used to identify the specific positions in the feature mapping. pred To predict the image, I gt For the real image, ||·|| represents the L1 norm, and represents the sum of the absolute differences between the two feature maps.

10. A road target detection device with dual-modal feature fusion and low-light enhancement, characterized in that, include: Image acquisition unit, used to acquire road images and infrared images; An image determination unit is used to determine whether a road image is a low-light image; The low-light image detection unit is used to input the road image into a pre-trained low-light enhanced target detection model for target detection when the road image is a low-light image. The low-light enhanced target detection model includes a multi-scale feature attention module, a brightness enhancement module, and a target detection network. The brightness enhancement module performs non-linear transformations on the image; A dual-modal image detection unit is used to input the road image and infrared image into a pre-trained dual-modal target detection model for target detection when the road image is a normal light image; the dual-modal target detection model includes a feature extraction module and a feature fusion module; The feature extraction module has a dual-branch structure, which extracts features from road images and infrared images respectively. The feature fusion module performs feature fusion through BiFPN and C2PSA modules.

Images