Roadside weak light target detection method and device based on multi-branch feature fusion

Abstract

The application provides a roadside weak light target detection method and device based on multi-branch feature fusion, and relates to the technical field of target detection and intelligent transportation. Real-time road image data collected by a roadside monitoring device is acquired; and the real-time road image data is input into a pre-trained roadside weak light target detection model DBRW-DETR model to perform detection reasoning and output a result. The DBRW-DETR model integrates a DBB multi-branch structure and a wavelet pooling layer in a backbone network to retain texture; a wavelet inverse pooling layer and a wavelet pooling layer are used for sampling in an encoder, and a Ret-RepC3 module integrating a space perception mechanism is combined to strengthen feature fusion. The application can solve the problem of feature loss under low illumination, retain high-frequency edges while reducing dimensions, significantly improve the accuracy and robustness of target detection under complex illumination, and meet the real-time requirement of roadside perception.

Description

Technical Field

[0001] This invention relates to the fields of target detection and intelligent transportation technology, and more specifically, to a method and apparatus for detecting roadside targets in low light based on multi-branch feature fusion. Background Technology

[0002] With the deepening of intelligent transportation systems and smart city construction, higher demands are being placed on the real-time performance and accuracy of road traffic safety management. Traditional methods relying on manual monitoring or simple sensors are no longer sufficient to meet the needs of all-weather, large-scale supervision. Therefore, more and more traffic management departments are deploying roadside sensing devices, using intelligent target detection technology to identify and analyze motor vehicles, non-motor vehicles, and pedestrians on the road in real time, in order to achieve accurate perception and efficient management of traffic flow. Against this backdrop, target detection technology based on visible light images has become the mainstream approach, especially real-time detection algorithms represented by deep learning models such as the YOLO series and RT-DETR, which have demonstrated excellent performance under sufficient lighting conditions.

[0003] However, in real-world road scenarios, low-light environments (such as nighttime, dusk, rainy days, or tunnels) are common. Images acquired under these conditions often suffer from insufficient illumination, severe noise interference, low contrast, and blurred texture details, making it difficult to effectively distinguish targets from the background. Existing general-purpose target detection models often employ traditional max-pooling or average-pooling operations during feature extraction, which can easily lead to the loss of key edge and structural information. Furthermore, their feature fusion mechanisms lack effective modeling of frequency domain information, easily introducing aliasing and blurring during upsampling, further weakening their ability to perceive small targets. In addition, conventional attention mechanisms do not adequately consider the spatial geometry of images when handling long-distance dependencies, making it difficult to accurately model the contextual relationships between widely distributed and scale-variable targets in roadside scenes. These factors collectively lead to a significant decrease in detection accuracy of existing methods under low-light conditions, with frequent false negatives and missed detections, failing to meet the core requirement of intelligent transportation systems for reliable all-weather perception.

[0004] In view of the above, this application is hereby submitted. Summary of the Invention

[0005] The present invention aims to provide a method, device, equipment and medium for roadside low light target detection based on multi-branch feature fusion, so as to solve the technical defects of the prior art when performing target detection in roadside low light environment, such as serious feature loss and low detection accuracy due to insufficient image illumination, serious noise interference and low contrast.

[0006] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0007] A method for detecting roadside low-light targets based on multi-branch feature fusion includes: S101, acquire real-time road image data collected by roadside monitoring equipment; S102, the real-time road image data is input into a pre-trained roadside low-light target detection model to perform detection inference, and the detection result including target category, bounding box and confidence level is output; The roadside low-light target detection model includes a backbone network, an encoder, and a decoder. The backbone network is a backbone feature extraction network that integrates a DBB multi-branch structure and a wavelet pooling layer, used to extract multi-scale feature maps. The encoder is a hybrid feature encoder based on wavelet transform. It performs feature sampling through wavelet inverse pooling layer and wavelet pooling layer, and performs feature fusion using Ret-RepC3 module with integrated two-dimensional spatial preservation mechanism. The Ret-RepC3 module embeds spatial awareness attention module RetBlock in the bottleneck layer of cross-stage local structure to capture long-distance dependencies in roadside scene. The decoder is used to predict the fused features output by the encoder to generate the detection result.

[0008] Preferably, the backbone network includes several convolutional normalization modules, wavelet pooling layers, and DBB multi-branch modules; The convolution normalization module is used to perform convolution and normalization processing on the input image to extract shallow feature maps; The wavelet pooling layer is used to decompose the input shallow feature map into four orthogonal frequency band components, namely one low-frequency approximation component and three high-frequency detail components, based on the discrete wavelet transform theory and using low-pass and high-pass filters of the Haar wavelet basis. The DBB multi-branch module consists of multiple cascaded multi-branch convolutional modules DBranchBlock, used to extract the input low-frequency approximation components through multi-branch convolution. Except for the first DBranchBlock, the output of each DBranchBlock is processed by 1... The number of channels in the first convolutional layer is adjusted. The output of the last DBranchBlock is then input into the AIFI attention feature enhancement module after the number of channels is adjusted. This module performs weighted enhancement and redundancy filtering of spatial and channel attention dimensions, and finally outputs multi-scale feature maps of different levels, which are then input into the encoder. Each DBranchBlock contains four parallel convolutional paths, and after fusing the outputs of each convolutional path, the fused features of each DBranchBlock are output through a nonlinear transformation. Each convolutional path contains multi-scale convolutional layers and batch normalization layers, used to extract multi-scale deep features from the low-frequency approximation components of the input. The high-frequency detail components are input into the encoder for upsampling reconstruction.

[0009] Preferably, the calculation formula for the four orthogonal frequency band components is as follows: ; ; ; ; in, This is a low-frequency approximation component. , , These are the high-frequency detail components in the horizontal, vertical, and diagonal directions, respectively. This is a shallow feature map; This represents the convolution operation; , These are low-pass and high-pass filters for the horizontal direction, respectively. , These are column-direction low-pass and high-pass filters, respectively.

[0010] Preferably, the encoder adopts a bidirectional path structure of top-down and bottom-up, including several wavelet inverse pooling layer upsampling modules, several wavelet pooling layer downsampling modules, and several Ret-RepC3 modules; In the wavelet inverse pooling layer upsampling module, the high-level enhanced features of the backbone network after AIFI processing are received as low-frequency approximate feature maps. Discrete wavelet inverse transform is performed, and then the high-frequency detail components transmitted by the corresponding level skip connections are combined to reconstruct the high-resolution feature map. The high-resolution feature map and the corresponding shallow feature map output by the backbone network are spliced ​​and fused in the channel dimension to output cross-scale spliced ​​and fused features. The cross-scale splicing and fusion features are fed into the Ret-RepC3 module for long-distance dependency modeling and spatial awareness feature calibration, and the fused features after Ret-RepC3 calibration are output. The fused features after Ret-RepC3 calibration enter the encoder's bottom-up path. The features are downsampled by the wavelet pooling layer downsampling module and fused with the deep features of the corresponding level for a second time, outputting bidirectional path fused features, i.e., the fused features output by the encoder; wherein, the deep features of the corresponding level are either the fused features after Ret-RepC3 calibration or the fused features output from the previous bottom-up round.

[0011] Preferably, the formula for calculating the high-resolution feature map is: ; in, This refers to the high-resolution feature map; This is the inverse discrete wavelet transform; This is a low-frequency approximation feature map; These are the high-frequency detail components in the horizontal, vertical, and diagonal directions, respectively.

[0012] Preferably, the features input to the Ret-RepC3 module are processed via a two-branch 1 After dimensionality reduction by convolution, they are fed into a RetConv layer containing a RetBlock; In the RetConv layer, the input features are first used to generate a query matrix through linear projection. Key matrix Sum matrix Then, a two-dimensional space preservation mechanism is introduced to calculate the spatially constrained context aggregation features. The calculation formula is as follows: ; in, Features are aggregated for context; It represents the Hadamardi (or Hadama) stack; It is the transpose symbol; This is a spatial location attenuation matrix based on Manhattan distance; Contextual aggregation features The output is after linear transformation and normalization. Wherein, the spatial position attenuation matrix Used to weight the interaction between visual tokens in different spatial locations, its elements For the first The feature point and the first The Manhattan distance between each feature point on the feature map is calculated using an exponential decay formula: ; in, , Two feature points respectively , Two-dimensional spatial coordinates; The preset spatial attenuation factor and .

[0013] Preferably, the method further includes: training the roadside low-light target detection model in an end-to-end manner, iteratively updating the model parameters by calculating the difference between the predicted bounding box and the real bounding box, until the model converges and meets the evaluation criteria, thereby obtaining the trained roadside low-light target detection model.

[0014] Preferably, a composite loss function is used to calculate the difference between the predicted bounding box and the true bounding box; The composite loss function is composed of a weighted average of classification loss, bounding box L1 regression loss, and generalized intersection-union (IoU) loss; wherein, the classification loss employs Varifocal Loss, using the predicted IoU score as a soft label to weight the binary cross-entropy loss; the bounding box L1 regression loss is calculated by... coordinates of the ground truth bounding box The absolute distance between the predicted and actual values ​​is used to measure the difference between them. The formula is as follows: ; ; ; in, It is a composite loss function; For classification loss; The bounding box L1 regression loss; For generalized intersection and comparison of losses; , , These are the weighting coefficients for the corresponding loss terms; The number of positive examples; It is an L1 norm; The minimum circumscribed convex closure area; Indicates area calculation; Indicates intersection; Represents the union; The difference operation represents the set difference; When evaluating the model, a judgment is made by calculating the detection evaluation index on the test set and combining it with the loss fluctuation. Specifically: First, calculate the predicted bounding box. With the true bounding box intersection ratio The calculation formula is: ; in, Indicates the area of ​​the calculation region; Next, set the IoU threshold. Determine the true case False positives False negatives : when And if the predicted category is correct, it is determined to be ; when Or, if the predicted category is incorrect, it is determined to be ; The real target that is not covered by the predicted bounding box is judged as ; Then calculate specific categories At IoU threshold Precision rate With recall rate The expressions are as follows: ; ; in, , , They are respectively , , Quantity; Based on accuracy With recall rate Construct a precision-recall curve and calculate the category using integration. Average accuracy The formula is: ; in, For category At IoU threshold The precision-recall curve below; This indicates that the recall rate is integrally calculated; Calculate all categories at the IoU threshold Mean precision The calculation formula is: ; in, Total number of categories; Based on average precision mean Calculate separately and ;in, For when The mean of the accuracy over time; The average value is calculated as the IoU threshold changes from 0.50 to 0.95 in steps of 0.05. ; in, This represents the average precision as the IoU threshold changes in steps of 0.05, starting from 0.50. When the model is on the test set and All exceeded the preset accuracy threshold. and continuous Fluctuation of the loss function over each training epoch Less than the convergence threshold When the model meets the evaluation criteria, it is determined that the model is in compliance with the evaluation criteria.

[0015] Preferably, it further includes: enriching the receptive field of feature extraction by utilizing the DBB multi-branch structure during the model training stage, and converting the convolution kernel parameters and batch normalization parameters of the DBB multi-branch structure into equivalent weights and biases of a single-path convolution by using structural reparameterization technology during the model inference stage, so as to improve the feature extraction capability without increasing the inference time.

[0016] The present invention also provides a roadside low-light target detection device based on multi-branch feature fusion, comprising: Image data acquisition unit, used to acquire real-time road image data collected by roadside monitoring equipment; The detection inference unit is used to input the real-time road image data into a pre-trained roadside low-light target detection model to perform detection inference and output detection results including target category, bounding box and confidence level.

[0017] The present invention also provides a roadside low-light target detection device based on multi-branch feature fusion, including a processor and a memory. The memory stores a computer program that can be executed by the processor to realize the roadside low-light target detection method based on multi-branch feature fusion as described above.

[0018] The present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor of the device on which the computer-readable storage medium is located, implement the roadside low-light target detection method based on multi-branch feature fusion as described above.

[0019] In summary, compared with the prior art, the present invention has the following beneficial effects: First, by introducing a DBB multi-branch structure and combining it with structure reparameterization technology, this invention can fully exploit the weak semantic features in low-light images by combining multi-scale and multi-path convolutions. While ensuring detection accuracy, it meets the real-time processing performance requirements of roadside embedded devices, achieving a balance between algorithm accuracy and speed.

[0020] Second, by employing a sampling module based on discrete wavelet transform, this invention changes the traditional deep learning model's practice of brute-force discarding information during downsampling. By utilizing the frequency domain decomposition characteristics of wavelet transform, the structural information and detail information of the image are separated and processed, and the edge features of the target are preserved to the greatest extent during feature transfer. This effectively solves the problem of inaccurate positioning caused by low contrast between the target and the background and blurred texture in low-light environments.

[0021] Third, by integrating the Ret-RepC3 module and introducing a two-dimensional space preservation mechanism, this invention enhances the model's ability to perceive the spatial geometry of images. Compared with the traditional global attention mechanism, this invention can more scientifically model the contextual relationships between targets at different locations in roadside scenes, suppress the interference of distant background noise, and significantly improve the ability to recognize complex occluded scenes and multi-scale targets.

[0022] Fourth, this invention uses a composite loss function for end-to-end optimization testing. It has made specific optimizations to address the characteristics of extremely imbalanced positive and negative samples and unclear target features in low-light scenarios. This enables the model to learn more discriminative classification boundaries and more accurate bounding box regression logic, ensuring the reliability of perception under complex lighting conditions in all weather conditions. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of a roadside low-light target detection method based on multi-branch feature fusion provided in Example 1.

[0025] Figure 2 The training flowchart for the roadside low-light target detection model provided in Example 1 is shown.

[0026] Figure 3 This is a schematic diagram of the roadside low-light target detection model provided in Example 1.

[0027] Figure 4 The graph showing the changes in the loss function and evaluation metrics during the training process is provided for Example 1.

[0028] Figure 5 Figure (a) is a comparison diagram of the model detection effect in a low-light scene provided in Example 1, and Figure (b) is a real-time image obtained and a result image after detection using the model of the present invention.

[0029] Figure 6 This is a schematic diagram of a roadside low-light target detection device based on multi-branch feature fusion, provided in Embodiment 2.

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0032] Example 1 Embodiment 1 of the present invention provides a roadside low-light target detection method based on multi-branch feature fusion, which can be implemented by a roadside low-light target detection device based on multi-branch feature fusion (hereinafter referred to as the detection device), specifically, executed by one or more processors within the detection device.

[0033] In this embodiment, the detection device may be an electronic device equipped with a processor, which carries a computer program for the roadside low-light target detection method based on multi-branch feature fusion and the computer program can be executed, such as a computer, smartphone, smart tablet, workstation, etc., which are not limited here.

[0034] like Figures 1-2 As shown, a roadside low-light target detection method based on multi-branch feature fusion includes steps S101 to S102.

[0035] S101, acquire real-time road image data collected by roadside monitoring equipment.

[0036] When the roadside monitoring equipment is in operation, it continuously collects video stream image data of the road monitoring area, including low-light road scene images under weather conditions such as night and dusk, as well as visual feature information of motor vehicles, non-motor vehicles and pedestrians contained in the images.

[0037] Based on the real-time road image data collected by the roadside monitoring equipment in low-light environment, the images are preprocessed.

[0038] S102, the real-time road image data is input into a pre-trained roadside low-light target detection model to perform detection inference, and the detection result including target category, bounding box and confidence level is output.

[0039] The roadside low-light target detection model includes a backbone network, an encoder, and a decoder. The backbone network is a backbone feature extraction network that integrates a DBB multi-branch structure and a wavelet pooling layer, used to extract multi-scale feature maps. The encoder is a hybrid feature encoder based on wavelet transform. It performs feature sampling through wavelet inverse pooling layer and wavelet pooling layer, and performs feature fusion using Ret-RepC3 module with integrated two-dimensional spatial preservation mechanism. The Ret-RepC3 module embeds spatial awareness attention module RetBlock in the bottleneck layer of cross-stage local structure to capture long-distance dependencies in roadside scene. The decoder is used to predict the fused features output by the encoder to generate the detection result.

[0040] like Figure 2 As shown, the pre-trained roadside low-light target detection model is constructed and trained through steps A1 to A3.

[0041] A1. Based on the real-time road image data collected by the roadside monitoring equipment in low-light environment, the images are preprocessed to obtain the training sample set.

[0042] In this embodiment, preprocessing includes manually annotating the acquired low-light road images using an image annotation tool, determining the true category and true bounding box coordinates of the target object, and generating the corresponding label file; The labeled image data is divided into training, validation, and test sets according to a preset ratio; the expression for dataset partitioning is: ; ; in, For the original image dataset, These are the training set, validation set, and test set, which are non-overlapping. These represent the number of samples in each set; This is the preset division ratio coefficient.

[0043] A2. Construct a roadside low-light target detection model (DBRW-DETR), which includes an improved backbone feature extraction network, a wavelet transform-based hybrid feature encoder and decoder.

[0044] A21, such as Figure 3 As shown, the backbone network includes several convolutional normalization modules, wavelet pooling layers, and DBB multi-branch modules.

[0045] The Convolutional Normalization (ConvNormLayer) module is used to perform convolution and normalization processing on the input image to extract shallow feature maps. Figure 3The ConvNormLayer module in the program has three layers and is used to extract basic texture and edge features.

[0046] The wavelet pooling layer (WaveletPool) is used to decompose the input shallow feature map into four orthogonal frequency band components based on discrete wavelet transform theory, utilizing low-pass and high-pass filters of the Haar wavelet basis: one low-frequency approximation component and three high-frequency detail components. The calculation formulas for the four orthogonal frequency band components are as follows: ; ; ; ; in, This is a low-frequency approximation component. , , These are the high-frequency detail components in the horizontal, vertical, and diagonal directions, respectively. This is a shallow feature map; This represents the convolution operation; , These are low-pass and high-pass filters for the horizontal direction, respectively. , These are column-direction low-pass and high-pass filters, respectively.

[0047] The low-frequency approximation component in this step Used to preserve target contours and suppress low-light noise; high-frequency detail component. , , Temporarily store (e.g., in a skip connection) for subsequent encoder upsampling reconstruction, in order to suppress background noise through frequency domain filtering while achieving downsampling.

[0048] The DBB multi-branch module is composed of multiple cascaded multi-branch convolutional modules DBranchBlock (e.g., ...). Figure 3 DBranchBlocks S2-S5 in the middle are used to perform multi-branch convolution extraction on the input low-frequency approximation components, except for the first DBranchBlock (such as...). Figure 3 Except for DBranchBlockS2, the output of each other DBranchBlock is processed by 1 1 convolutional layer (Conv 1) 1) Adjusting the number of channels: The output of the last DBranchBlock, after adjusting the number of channels, is then input into the AIFI attention feature enhancement module for weighted enhancement and redundancy filtering of spatial and channel attention dimensions, further strengthening the expressive power of the target features, and finally outputting multi-scale feature maps of different levels (e.g., ...). Figure 3 The P3-P5 features corresponding to different levels are input into the encoder.

[0049] Specifically, DBranchBlockS3 output: The output of DBranchBlock S3 is processed by 1... 1. Convolution adjusts the number of channels, corresponding to shallow features at the second-highest resolution (P3 level).

[0050] DBranchBlock S4 Output: The output of DBranchBlock S4 is processed by 1... 1. Convolution adjusts the number of channels, corresponding to shallow features with higher resolution (P4 level).

[0051] DBranchBlock S5 Output: The output of DBranchBlock S5 is processed by 1... After adjusting the number of channels in the convolution, the input is then fed into the AIFI attention feature enhancement module for feature enhancement, and the high-level enhanced features are output (P5 level).

[0052] Each DBranchBlock contains four parallel convolutional paths, and after fusing the outputs of each convolutional path, it outputs the fused features of each DBranchBlock through a nonlinear transformation.

[0053] Each convolutional path contains multi-scale convolutional layers (Conv 1) 1. Conv k k) and batch normalization layer (BatchNorm) (e.g. Figure 3 Middle 1 1. Convolution + Batch Normalization (BatchNorm, k) k-convolution + batch normalization (BatchNorm), 1 1+k k-concatenated convolutions + batch normalization (BatchNorm), 1 1. Convolution + Average Pooling + Batch Normalization (BatchNorm) is used to extract deep features at multiple scales from the low-frequency approximation components of the input.

[0054] The high-frequency detail components are input into the encoder for upsampling reconstruction.

[0055] To further enhance the system's robustness, this embodiment employs a refined parameter fusion strategy during the reparameterization process of the DBB multi-branch structure. During the inference phase, trained convolutional kernel parameters (such as convolutional layer weights and biases) and batch normalization layer parameters (such as the mean, variance, and scaling factor of the batch normalization layer) are merged into the weights and biases of the convolutional layers. In other words, structural reparameterization technology is used to combine the convolutional kernel parameters and batch normalization parameters of multiple branches into the weights and biases of a single convolution, achieving lossless inference acceleration. This mathematical transformation reduces the number of computational operators during hardware deployment, further shortening the inference latency per frame.

[0056] This invention introduces a DBB multi-branch structure into the convolutional module of the backbone network. During the training phase, multi-scale convolutional branches are used to enrich the receptive field of feature extraction. During the inference phase, the structure reparameterization technique is used to convert it into a single-path convolution, thereby improving the feature extraction capability without increasing the inference time. At the same time, in the shallow feature downsampling stage of the backbone network, a wavelet pooling layer is used. The discrete wavelet transform is used to decompose the feature map into low-frequency approximate components and high-frequency detail components. Only the low-frequency components are retained to enter the next layer. This effectively preserves the texture and edge structure information of the low-light image while reducing the resolution, avoiding the loss of details caused by traditional max pooling or average pooling.

[0057] A22, a hybrid feature encoder based on wavelet transform.

[0058] This module achieves deep fusion of multi-scale features through a bidirectional path. The encoder adopts a bidirectional path structure of top-down and bottom-up, including several wavelet inverse pooling layer upsampling modules, several wavelet pooling layer downsampling modules, and several Ret-RepC3 modules.

[0059] Top-down approach: Using wavelet inverse pooling layers, combined with the corresponding high-frequency components (LH / HL / HH) temporarily stored in the backbone, discrete wavelet inverse transform is performed to reconstruct high-resolution feature maps, solving the blurring and aliasing problems of upsampling in low-light images.

[0060] Bottom-up approach: Wavelet pooling layers are used for downsampling to maintain consistency in frequency domain processing logic.

[0061] In the wavelet inverse pooling layer upsampling module, the high-level enhanced features of the backbone network after AIFI processing are received as a low-frequency approximate feature map, and a discrete wavelet inverse transform operation is performed, which is then combined with the corresponding level skip connections ( Figure 3 The high-frequency detail components (transmitted by "C" in the figure represent skip connections) are reconstructed to restore a high-resolution feature map; the calculation formula for the high-resolution feature map is: ; in, This refers to the high-resolution feature map; This is the inverse discrete wavelet transform; This is a low-frequency approximation feature map; These are the high-frequency detail components in the horizontal, vertical, and diagonal directions, respectively.

[0062] The high-level enhancement features output by AIFI are the low-frequency approximate feature maps required by the wavelet inverse pooling layer.

[0063] Furthermore, in the bottom-up path of the hybrid feature encoder, the new low-frequency components generated by wavelet pooling downsampling of features will also serve as the low-frequency approximate feature map of the next wavelet inverse pooling layer, realizing iterative reconstruction at multiple scale levels.

[0064] The low-frequency approximate feature map not only preserves the global semantic information of the low-frequency components, but also enhances the target features through attention enhancement, providing a high-quality low-frequency foundation for high-resolution reconstruction using wavelet inverse pooling.

[0065] The high-resolution feature map and the corresponding shallow feature map output by the backbone network ( Figure 3 The shallow feature maps output by the backbone network DBranchBlock S3-S4 are fed into the encoder via skip connections and spliced ​​and fused along the channel dimension to output cross-scale spliced ​​and fused features.

[0066] The cross-scale splicing and fusion features are fed into the Ret-RepC3 module (spatial perception feature enhancement module) for long-distance dependency modeling and spatial perception feature calibration, and output the fused features after Ret-RepC3 calibration.

[0067] Specifically, the features input to the Ret-RepC3 module are processed through a two-branch 1 After dimensionality reduction by convolution, the two layers are fed into a RetConv layer containing a RetBlock.

[0068] In the RetConv layer, the input features are first used to generate a query matrix through linear projection. Key matrix Sum matrix Then, a two-dimensional space preservation mechanism is introduced to calculate the spatially constrained context aggregation features. The calculation formula is as follows: ; in, Features are aggregated for context; It represents the Hadamardi (or Hadama) stack; It is the transpose symbol; This is a spatial location attenuation matrix based on Manhattan distance; Contextual aggregation features The output after linear transformation and normalization is the fused feature after Ret-RepC3 calibration.

[0069] Wherein, the spatial position attenuation matrix Used to weight the interaction between visual tokens in different spatial locations, its elements For the first The feature point and the first The Manhattan distance between each feature point on the feature map is calculated using an exponential decay formula: ; in, , Two feature points respectively , Two-dimensional spatial coordinates; The preset spatial attenuation factor and .

[0070] For any two points in the feature map, the larger the Manhattan distance, the smaller the corresponding attenuation value. This means that when updating the feature representation of a pixel, the model will refer more to features within its physical neighborhood, while giving less attention to distant regions that may belong to background noise. This weight allocation mechanism based on physical distance explicitly models the two-dimensional geometry of the roadside scene without relying on complex location encoding, resulting in a significant performance improvement for recognizing rows of parked vehicles or groups of pedestrians on the roadside.

[0071] The fused features after Ret-RepC3 calibration enter the encoder's bottom-up path. The features are downsampled by the wavelet pooling layer downsampling module (this step can generate new low-frequency approximate components) and fused a second time with the deep features of the corresponding level (the fused features after Ret-RepC3 calibration or the fused features output from the previous bottom-up round). The output is the bidirectional path fused features, which are the fused features output by the encoder. These fused features integrate multi-scale semantics and detailed information.

[0072] This step replaces traditional bilinear interpolation or nearest-neighbor interpolation upsampling with WaveletUnPool wavelet inverse pooling layers in the top-down path of the feature pyramid network. High-resolution feature maps are reconstructed through wavelet inverse transform, reducing feature aliasing and blurring during upsampling. In the bottom-up path of the path aggregation network of this module, WaveletPool wavelet pooling layers replace traditional strided convolutions for downsampling to maintain the consistency of frequency domain information during feature fusion. Simultaneously, at the connection nodes of feature fusion, the RepC3 module, improved based on RetBlock preservation blocks, processes the fused features. Utilizing the two-dimensional spatial preservation mechanism of RetBlock, weights are dynamically adjusted based on the spatial distance between features, enhancing the model's ability to capture long-distance dependencies and global contextual information in roadside scenes.

[0073] A23, Transformer Decoder.

[0074] This module is responsible for predicting target attributes and outputting results.

[0075] The decoder receives the fused features output by the encoder, updates the query vector through a multi-head self-attention mechanism and a cross-attention mechanism, and finally outputs the detection results of roadside targets through the prediction head.

[0076] The detection results include: category labels (such as cars, trucks, bicycles, pedestrians), bounding box coordinates (expressed as the coordinates of the center point of the rectangle and the aspect ratio), and confidence score (reflecting the reliability of the detection results).

[0077] A3. Based on the training sample set, the constructed roadside low-light target detection model is trained in an end-to-end manner. By calculating the difference between the predicted box and the real box, the model parameters are iteratively updated until the model converges and meets the evaluation criteria, thus obtaining the trained roadside low-light target detection model.

[0078] When calculating the difference between the predicted bounding box and the ground truth bounding box, a composite loss function is used. The composite loss function is composed of a weighted average of classification loss, bounding box L1 regression loss, and generalized intersection-union (IoU) loss; wherein, the classification loss employs Varifocal Loss, using the predicted IoU score as a soft label to weight the binary cross-entropy loss; the bounding box L1 regression loss is calculated by... coordinates of the ground truth bounding box The absolute distance between the predicted and actual values ​​is used to measure the difference between them. The formula is as follows: ; ; ; in, It is a composite loss function; For classification loss; The bounding box L1 regression loss; For generalized intersection and comparison of losses; , , These are the weighting coefficients for the corresponding loss terms; The number of positive examples; It is an L1 norm; The minimum circumscribed convex closure area; Indicates area calculation; Indicates intersection; Represents the union; The difference operation of sets.

[0079] The bounding box L1 regression loss accurately calculates the absolute deviation between the predicted box and the ground truth box in terms of center coordinates and width and height. The generalized intersection-union loss optimizes the localization accuracy of the predicted box by considering the overlap area between the predicted box and the ground truth box and its minimum circumscribed convex closure. The backpropagation algorithm continuously updates the weights of each level of the model until the model's performance on the validation set reaches the preset performance threshold.

[0080] When evaluating the model, a judgment is made by calculating the detection evaluation index on the test set and combining it with the loss fluctuation. Specifically: First, calculate the predicted bounding box. With the true bounding box intersection ratio The calculation formula is: ; in, Indicates the area of ​​the calculation region; Next, set the IoU threshold. Determine the true case False positives False negatives : when And if the predicted category is correct, it is determined to be ; when Or, if the predicted category is incorrect, it is determined to be ; The real target that is not covered by the predicted bounding box is judged as ; Then calculate specific categories At IoU threshold Precision rate With recall rate The expressions are as follows: ; ; in, , , They are respectively , , Quantity; Based on accuracy With recall rate Construct a precision-recall curve and calculate the category using integration. Average accuracy The formula is: ; in, For category At IoU threshold The precision-recall curve below; This indicates that the recall rate is integrally calculated; Calculate all categories at the IoU threshold Mean precision The calculation formula is: ; in, Total number of categories; Based on average precision mean Calculate separately and ;in, For when The mean of the accuracy over time; The average value is calculated as the IoU threshold changes from 0.50 to 0.95 in steps of 0.05. ; in, This represents the average precision as the IoU threshold changes in steps of 0.05, starting from 0.50. When the model is on the test set and All exceeded the preset accuracy threshold. and continuous Fluctuation of the loss function over each training epoch Less than the convergence threshold When the model meets the evaluation criteria, it is determined that the model is in compliance with the evaluation criteria.

[0081] like Figure 4As shown, the loss convergence trend and performance index changes of the roadside low-light target detection model of the present invention are fully demonstrated during 200 training rounds. All losses and indices change rapidly within 20 rounds and converge completely at 200 rounds, indicating that the feature extraction module of the model of the present invention can efficiently mine the features of low-light targets and has high learning efficiency; moreover, the curve trends of the training set and the validation set are highly consistent, with no obvious overfitting, which is suitable for the actual needs of roadside scenes with changes in lighting and complex backgrounds. Approximately 0.9 A score close to 0.5 is excellent in the field of low-light target detection, indicating that the model can effectively solve the problems of weak target features and high background noise in low light.

[0082] Once the model meets the standards, the final object detection task is performed. The low-light image to be tested is input into the model, and the output tensor set is obtained. ,in The normalized coordinates of the prediction box. For confidence level, M represents the category index, and M represents the total number of predicted boxes in the initial output of the model.

[0083] Using confidence threshold Filter out The system calculates the predicted terms and maps the coordinates of the remaining predicted boxes back to the original image resolution, ultimately outputting the detection results.

[0084] Once trained, the model is embedded into an inference engine and deployed in roadside edge computing nodes to provide real-time and accurate target perception data for the intelligent traffic management platform, thereby enabling precise monitoring of traffic flow and accident early warning.

[0085] In actual operation, when a roadside monitoring device captures a frame of a nighttime road image, it is first converted into a digital matrix for detection and inference. During sample preprocessing, the image is first scaled to the input size required by the model. Then, the backbone network extracts feature maps containing the contours of weak-light targets through multi-branch convolution and frequency domain decomposition. The encoder uses inverse wavelet transform and the Ret-RepC3 module to deeply fuse these multi-scale features, enhancing the contrast between the target and the background. Finally, the decoder generates a sequence of candidate targets. Post-processing logic is also included before outputting the final result. By setting a confidence threshold, such as 0.5, low-scoring interference items are eliminated. Subsequently, a scaling algorithm is used to restore the normalized coordinates of the remaining candidate boxes to the physical pixel space of the original image.

[0086] like Figure 5The diagram shows a comparison of the performance of a trained model for target detection in a low-light roadside scene. This invention enables clear labeling of occluded vehicles and distant pedestrians in low-light environments. The labeling information includes text labels for the target category and colored bounding boxes closely following the target's outline.

[0087] In summary, compared with the prior art, the present invention has the following beneficial effects: This invention achieves enhanced feature mining capabilities by specifically modifying the underlying operators of a deep learning detection framework, deeply integrating DBB multi-branch reparameterization technology, wavelet frequency domain analysis technology, and spatial awareness attention mechanism. In the backbone network, the DBB multi-branch structure enhances feature mining capabilities, while wavelet pooling layers effectively preserve frequency domain information. In the encoder, wavelet inverse pooling layers and the Ret-RepC3 module enable high-fidelity feature fusion and global context modeling.

[0088] This invention enables reliable detection of traffic targets in extreme environments such as low roadside light, providing a solid technical guarantee for the realization of all-weather intelligent traffic perception.

[0089] Those skilled in the art can fine-tune hyperparameters such as the number of channels and attenuation factor in the model based on the actual road width, traffic density, and installation height of the monitoring equipment, in order to achieve the best perception effect in a specific scenario.

[0090] Example 2 like Figure 6 As shown, the second embodiment of the present invention also provides a roadside low-light target detection device based on multi-branch feature fusion, comprising: The image data acquisition unit S201 is used to acquire real-time road image data collected by the roadside monitoring equipment; The detection inference unit S202 is used to input the real-time road image data into a pre-trained roadside low-light target detection model to perform detection inference and output detection results including target category, bounding box and confidence level.

[0091] Example 3 The third embodiment of the present invention also provides a roadside low-light target detection device based on multi-branch feature fusion, which includes a memory and a processor. The memory stores a computer program, which can be executed by the processor to realize the roadside low-light target detection method based on multi-branch feature fusion as described above.

[0092] Example 4 The fourth embodiment of the present invention also provides a computer-readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by the processor of the device where the computer-readable storage medium is located, the roadside low-light target detection method based on multi-branch feature fusion as described above is implemented.

[0093] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting roadside low-light targets based on multi-branch feature fusion, characterized in that, include: Acquire real-time road image data collected by roadside monitoring equipment; The real-time road image data is input into a pre-trained roadside low-light target detection model to perform detection inference and output detection results including target category, bounding box and confidence level. The roadside low-light target detection model includes a backbone network, an encoder, and a decoder. The backbone network is a backbone feature extraction network that integrates a DBB multi-branch structure and a wavelet pooling layer, used to extract multi-scale feature maps. The encoder is a hybrid feature encoder based on wavelet transform. It performs feature sampling through wavelet inverse pooling layer and wavelet pooling layer, and performs feature fusion using Ret-RepC3 module with integrated two-dimensional spatial preservation mechanism. The Ret-RepC3 module embeds spatial awareness attention module RetBlock in the bottleneck layer of cross-stage local structure to capture long-distance dependencies in roadside scene. The decoder is used to predict the fused features output by the encoder to generate the detection result.

2. The method for detecting roadside low-light targets based on multi-branch feature fusion according to claim 1, characterized in that... The backbone network includes several convolutional normalization modules, wavelet pooling layers, and DBB multi-branch modules; The convolution normalization module is used to perform convolution and normalization processing on the input image to extract shallow feature maps; The wavelet pooling layer is used to decompose the input shallow feature map into four orthogonal frequency band components, namely one low-frequency approximation component and three high-frequency detail components, based on the discrete wavelet transform theory and using low-pass and high-pass filters of the Haar wavelet basis. The DBB multi-branch module consists of multiple cascaded multi-branch convolutional modules DBranchBlock, used to extract the input low-frequency approximation components through multi-branch convolution. Except for the first DBranchBlock, the output of each DBranchBlock is processed by 1... The number of channels in the first convolutional layer is adjusted. The output of the last DBranchBlock is then input into the AIFI attention feature enhancement module after the number of channels is adjusted. This module performs weighted enhancement and redundancy filtering of spatial and channel attention dimensions, and finally outputs multi-scale feature maps of different levels, which are then input into the encoder. Each DBranchBlock contains four parallel convolutional paths, and after fusing the outputs of each convolutional path, the fused features of each DBranchBlock are output through a nonlinear transformation. Each convolutional path contains multi-scale convolutional layers and batch normalization layers, used to extract multi-scale deep features from the low-frequency approximation components of the input. The high-frequency detail components are input into the encoder for upsampling reconstruction.

3. The method for detecting roadside low-light targets based on multi-branch feature fusion according to claim 2, characterized in that... The formulas for calculating the four orthogonal frequency band components are as follows: ； ； ； ； in, This is a low-frequency approximation component. , , These are the high-frequency detail components in the horizontal, vertical, and diagonal directions, respectively. This is a shallow feature map; This represents the convolution operation; , These are low-pass and high-pass filters for the horizontal direction, respectively. , These are column-direction low-pass and high-pass filters, respectively.

4. The method for detecting roadside low-light targets based on multi-branch feature fusion according to claim 2, characterized in that... The encoder adopts a bidirectional path structure of top-down and bottom-up, including several wavelet inverse pooling layer upsampling modules, several wavelet pooling layer downsampling modules, and several Ret-RepC3 modules. In the wavelet inverse pooling layer upsampling module, the high-level enhanced features of the backbone network after AIFI processing are received as low-frequency approximate feature maps. Discrete wavelet inverse transform is performed, and then the high-frequency detail components transmitted by the corresponding level skip connections are combined to reconstruct the high-resolution feature map. The high-resolution feature map and the corresponding shallow feature map output by the backbone network are spliced ​​and fused in the channel dimension to output cross-scale spliced ​​and fused features. The cross-scale splicing and fusion features are fed into the Ret-RepC3 module for long-distance dependency modeling and spatial awareness feature calibration, and the fused features after Ret-RepC3 calibration are output. The fused features after Ret-RepC3 calibration enter the encoder's bottom-up path. The features are downsampled by the wavelet pooling layer downsampling module and fused with the deep features of the corresponding level for a second time, outputting bidirectional path fused features, i.e., the fused features output by the encoder; wherein, the deep features of the corresponding level are either the fused features after Ret-RepC3 calibration or the fused features output from the previous bottom-up round.

5. The method for detecting roadside low-light targets based on multi-branch feature fusion according to claim 4, characterized in that... The formula for calculating the high-resolution feature map is as follows: ； in, This refers to the high-resolution feature map; This is the inverse discrete wavelet transform; This is a low-frequency approximation feature map; These are the high-frequency detail components in the horizontal, vertical, and diagonal directions, respectively.

6. The method for detecting roadside low-light targets based on multi-branch feature fusion according to claim 4, characterized in that... The features input to the Ret-RepC3 module are processed through a two-branch 1 After dimensionality reduction by convolution, they are fed into a RetConv layer containing a RetBlock; In the RetConv layer, the input features are first used to generate a query matrix through linear projection. Key matrix Sum matrix Then, a two-dimensional space preservation mechanism is introduced to calculate the spatially constrained context aggregation features. The calculation formula is as follows: ； in, Features are aggregated for context; It represents the Hadamardi (or Hadama) stack; It is the transpose symbol; This is a spatial location attenuation matrix based on Manhattan distance; Contextual aggregation features The output is after linear transformation and normalization. Wherein, the spatial position attenuation matrix Used to weight the interaction between visual tokens in different spatial locations, its elements For the first The feature point and the first The Manhattan distance between each feature point on the feature map is calculated using an exponential decay formula: ； in, , Two feature points respectively , Two-dimensional spatial coordinates; The preset spatial attenuation factor and .

7. The method for detecting roadside low-light targets based on multi-branch feature fusion according to claim 1, characterized in that... It also includes: training the roadside low-light target detection model in an end-to-end manner, iteratively updating the model parameters by calculating the difference between the predicted box and the real box, until the model converges and meets the evaluation criteria, thus obtaining the trained roadside low-light target detection model.

8. The method for detecting roadside low-light targets based on multi-branch feature fusion according to claim 7, characterized in that... When calculating the difference between the predicted bounding box and the ground truth bounding box, a composite loss function is used. The composite loss function is composed of a weighted average of classification loss, bounding box L1 regression loss, and generalized intersection-union (IoU) loss; wherein, the classification loss employs Varifocal Loss, using the predicted IoU score as a soft label to weight the binary cross-entropy loss; the bounding box L1 regression loss is calculated by... coordinates of the ground truth bounding box The absolute distance between the predicted and actual values ​​is used to measure the difference between them. The formula is as follows: ； ； ； in, It is a composite loss function; For classification loss; The bounding box L1 regression loss; For generalized intersection and comparison of losses; , , These are the weighting coefficients for the corresponding loss terms; The number of positive examples; It is an L1 norm; The minimum circumscribed convex closure area; Indicates area calculation; Indicates intersection; Represents the union; The difference operation represents the set difference; When evaluating the model, a judgment is made by calculating the detection evaluation index on the test set and combining it with the loss fluctuation. Specifically: First, calculate the predicted bounding box. With the true bounding box intersection ratio The calculation formula is: ； in, Indicates the area of ​​the calculation region; Next, set the IoU threshold. Determine the true case False positives False negatives : when And if the predicted category is correct, it is determined to be ; when Or, if the predicted category is incorrect, it is determined to be ; The real target that is not covered by the predicted bounding box is judged as ; Then calculate specific categories At IoU threshold Precision rate With recall rate The expressions are as follows: ； ； in, , , They are respectively , , Quantity; Based on accuracy With recall rate Construct a precision-recall curve and calculate the category using integration. Average accuracy The formula is: ； in, For category At IoU threshold The precision-recall curve below; This indicates that the recall rate is integrally calculated; Calculate all categories at the IoU threshold Mean precision The calculation formula is: ； in, Total number of categories; Based on average precision mean Calculate separately and ;in, For when The mean of the accuracy over time; The average value is calculated as the IoU threshold changes from 0.50 to 0.95 in steps of 0.

05. ； in, This represents the average precision as the IoU threshold changes in steps of 0.05, starting from 0.

50. When the model is on the test set and All exceeded the preset accuracy threshold. and continuous Fluctuation of the loss function over each training epoch Less than the convergence threshold When the model meets the evaluation criteria, it is determined that the model is in compliance with the evaluation criteria.

9. A method for detecting roadside low-light targets based on multi-branch feature fusion according to claim 2, characterized in that... It also includes: enriching the receptive field of feature extraction by utilizing the DBB multi-branch structure during the model training phase, and converting the convolution kernel parameters and batch normalization parameters of the DBB multi-branch structure into equivalent weights and biases of single-path convolutions through structural reparameterization technology during the model inference phase, so as to improve the feature extraction capability without increasing the inference time.

10. A roadside low-light target detection device based on multi-branch feature fusion, used to implement the roadside low-light target detection method based on multi-branch feature fusion as described in any one of claims 1-9, characterized in that, include: Image data acquisition unit, used to acquire real-time road image data collected by roadside monitoring equipment; The detection inference unit is used to input the real-time road image data into a pre-trained roadside low-light target detection model to perform detection inference and output detection results including target category, bounding box and confidence level.

Images