A road object detection method based on brain-like dual-pathway feature fusion

Abstract

The present invention discloses a road target detection method based on brain-like dual-pathway feature fusion, which belongs to the field of image recognition technology. The method proposes a dual-pathway mechanism based on target recognition and spatial positioning of the human brain visual system, converts the traffic scene video stream into a pulse density map, inputs it into a bionic dual-pathway network, extracts features from the target recognition path and the spatial positioning path respectively, and then performs adaptive dynamic feature fusion through a dynamic fusion module. By strictly mapping the human brain visual pathway, the theoretical reliability of the technical solution is enhanced. At the same time, pulse convolution is introduced to ensure high real-time performance and computational efficiency, balance the accuracy, speed and power consumption of traffic target detection, and is suitable for on-board traffic target detection tasks, meeting the target detection requirements of automatic driving systems. Experiments have shown that this method improves the mAP of small target detection by 15.8% and reduces energy consumption by 40%.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition technology and relates to a method for detecting traffic targets, and in particular to a road target detection method based on brain-like dual-pathway feature fusion. Background Art

[0002] Traffic object detection is a core technology in autonomous driving and intelligent transportation systems. Traffic object detection involves the real-time identification and location of various road objects, including vehicles, pedestrians, traffic signs, and traffic lights, using on-board sensors. Traditional object detection methods typically rely on deep convolutional neural networks (CNNs) or Transformer models, extracting visual features from large amounts of image data. However, CNNs rely on fixed-size convolutional kernels, making them ineffective at processing multi-scale objects (e.g., small vehicles at a distance versus large trucks at close range). Furthermore, pooling operations result in spatial information loss, leading to a high rate of missed detection of small objects. While Transformers can model temporal dependencies, their high computational complexity makes them difficult to meet the real-time requirements of on-board equipment. To alleviate these issues, existing technologies employ methods such as multi-scale feature fusion, lightweight design, attention mechanisms, and multimodal data fusion. Multi-scale feature fusion employs feature pyramid networks or dilated convolutions to enhance multi-scale detection, but static weight allocation struggles to adapt to dynamic object changes and increases computational complexity. Lightweight designs employ network pruning, quantization, or knowledge distillation to compress models (e.g., MobileNet and YOLO-Lite), but this results in significant accuracy losses. Attention mechanisms introduce channel / spatial attention to enhance responses to key regions, but lack a dynamic triggering mechanism for sudden targets. Multimodal data fusion methods integrate data from multiple sources, such as cameras, lidar, and infrared sensors, to improve robustness. However, feature-level fusion (e.g., early / late fusion) can easily introduce redundant computations, and cross-modal alignment errors can lead to performance fluctuations (e.g., spatial and temporal asynchrony).

[0003] Complex traffic scenarios involve multiple objects, multiple scales, high-density occlusion, extreme lighting (backlight / nighttime), and unusually shaped objects (non-motorized vehicles, special vehicles). Existing technologies can partially alleviate these issues through multimodal fusion and lightweight design. However, in complex traffic scenarios and dynamic mitigation, there are still drawbacks such as redundant multimodal feature extraction, insufficient dynamic weight allocation, and a conflict between model complexity and energy efficiency.

[0004] Current traffic target detection technology faces core bottlenecks such as poor multi-scale adaptability, low computational efficiency, and insufficient dynamic fusion mechanism in complex scenarios and dynamic environments. An innovative solution that breaks through the inherent limitations of existing deep learning frameworks is urgently needed. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention proposes a road target detection method based on brain-like dual-pathway feature fusion, which introduces the dual-pathway mechanism of the human brain's visual system into traffic target detection. Combined with the pulse neural network, the target recognition and spatial positioning tasks are respectively handled by two pathways, and then feature fusion technology is used to combine them, providing a new, efficient and accurate target detection method for intelligent transportation systems.

[0006] A road object detection method based on brain-like dual-pathway feature fusion specifically includes the following steps:

[0007] Step 1: Use the vehicle-mounted camera to capture the traffic scene video stream in real time, simulate the information processing process of the retina, convert the traffic scene video stream from continuous frame data to pulse data, and generate a pulse density map. The specific method is as follows:

[0008] Calculate the pixel intensity change between adjacent frame images :

[0009]

[0010] in, 、 Represents the pixel value at coordinate (m, n) of the t-th frame and t-1 frame respectively. | | represents the absolute value.

[0011] Set when the pixel intensity of adjacent frame images changes Generates a pulse when the threshold is exceeded :

[0012]

[0013] Among them, θ base represents the basic pulse threshold, Entropy( ) represents the information entropy. η represents the adjustment coefficient, which is used to control the threshold Sensitivity to scene changes.

[0014] Count the number of pulses per unit time and generate a pulse density map.

[0015] Preferably, the V1 area in the visual cortex of the brain is simulated, and the outline edge, color contrast and local texture of objects are extracted from the traffic scene video stream using a brain-like Gabor filter. These basic features are used to trigger high-frequency pulses in the high-contrast edge areas in the pulse density map and low-frequency pulses in the low-texture background areas, thereby suppressing static background noise and completing the data enhancement operation.

[0016] Step 2: Construct a bionic dual-pathway network, extract features from the target recognition pathway and the spatial positioning pathway of the pulse density map generated in step 1, and then perform adaptive dynamic feature fusion through the dynamic fusion module.

[0017] s2.1. The target recognition pathway corresponds to the temporal lobe of the human brain and is responsible for identifying object details. It processes the pulse density map through edge feature extraction, component-level feature aggregation, target classification, and attention layers.

[0018] The edge feature extraction layer extracts local edge features from the pulse density map through pulse convolution. The component-level feature aggregation layer integrates component-level features from edge features through pulse convolution. The object classification and attention layer further extracts and integrates component-level features through a sparse pulse feature extraction network, adaptively assigns dynamic weights, and outputs a target detail feature tensor containing semantic category information.

[0019] The sparse spike feature extraction network includes multiple spike convolution kernels of varying sizes, a spike attention module, and a fully spike connection layer. Smaller spike convolution kernels are used to extract local features from the spike density map. Larger spike convolution kernels use lateral connections to simulate the long-range interactions of temporal cortical neurons for feature aggregation. The spike attention module simulates the response enhancement mechanism of temporal lobe neurons to highly significant targets. Dynamic weights are adaptively assigned based on the regional spike density D(m, n), outputting a weighted feature map:

[0020]

[0021] in, represents the dynamic weight at coordinate (m, n). θ2 represents the pulse density threshold, and Sigmoid represents the activation function.

[0022] The fully pulse-connected layer classifies the weighted feature maps, generates category probabilities through pulse emission frequency statistics, and outputs target detail feature tensors containing semantic category information.

[0023] s2.2. The spatial positioning pathway corresponds to the parietal lobe area of ​​the human brain and is responsible for spatial perception. It processes the pulse density map in sequence through the multi-scale spatial encoding layer, spatial selection layer, and environmental adaptation layer.

[0024] The multi-scale spatial coding layer first convolves the pulse density map and then fuses the multi-scale features through dynamic pyramid pooling to output a multi-scale spatial heat map. The dynamic pyramid pooling includes two pooling schemes of different sizes, selected based on the entropy value in the pulse density map.

[0025] The spatial selection layer performs pulse-driven dilated convolution and non-maximum suppression on the multi-scale spatial heat map to expand the receptive field, and performs sparse output of the spatial coordinate map of the redundant candidate boxes, including the normalized center coordinates, width, and height of the candidate boxes.

[0026] The environment adaptation layer dynamically adjusts the coordinate regression weights in the spatial coordinate map based on the global average pulse frequency, and outputs the final spatial positioning map (x, y, w, h), where (x, y) is the center coordinate of the candidate box and (w, h) is the width and height of the candidate box.

[0027] s2.3. The dynamic fusion module simulates the prefrontal region of the brain and adaptively fuses the multimodal information output by the target recognition pathway and the spatial positioning pathway to output a unified representation of both semantic and spatial information.

[0028] The dynamic fusion module first solves the resolution difference of the two paths through the feature alignment layer and outputs the aligned semantic feature map and spatial feature map. Then, the predicted target bounding box area A is obtained from the spatial feature map, so as to adaptively allocate the weights W of the target recognition path and the spatial positioning path. what 、W where , fuse the semantic feature map and the spatial feature map:

[0029]

[0030]

[0031] Among them, S small 、S large They represent the set small target area threshold and large target area threshold respectively.

[0032] Step 3: Use the hybrid decoder to perform pulse density decoding on the fused feature map output in step 2 to obtain the category probability confidence:

[0033]

[0034] Where K represents the number of target categories. The K-dimensional vector pi cls represents the confidence of the i-th category. max Represents the maximum pulse frequency of the neuron. Ni spike represents the spike count of the i-th type output neuron.

[0035] The category with the highest confidence is taken as the recognition result, and the target category, confidence and bounding box information are output.

[0036] The present invention has the following beneficial effects:

[0037] 1. This method proposes a dual-pathway mechanism based on the human visual system: object recognition (what) and spatial localization (where). It balances traffic object detection accuracy, speed, and power consumption, making it suitable for on-board traffic object detection tasks. By rigorously mapping the human visual pathway, the theoretical reliability of the technical solution is enhanced. Experimental verification has shown a 15.8% increase in mean average precision (mAP) for small object detection, while reducing energy consumption by 40%.

[0038] 2. This method can accurately identify traffic targets in complex traffic scenarios. At the same time, the processing method based on the pulse neural network can ensure that the method has high real-time performance and computational efficiency, meeting the target detection requirements of the autonomous driving system. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The mapping relationship between data processing and visual cortex;

[0040] Figure 2 This is a schematic diagram of the bionic dual-pathway network structure;

[0041] Figure 3 This is a test example in a dim scene in Example 2;

[0042] Figure 4 This is a test example in the dense scene in Example 2;

[0043] Figure 5 This is a test example under the exposure scenario in Example 2;

[0044] Figure 6 This is a test example under the occlusion scenario in Example 2;

[0045] Figure 7 This is a test example in the rainy day scene in Example 2;

[0046] Figure 8 This is a test example under the blurred scene in Example 2. DETAILED DESCRIPTION

[0047] The present invention will be further explained below with reference to the accompanying drawings;

[0048] Example 1

[0049] like Figure 1 As shown in FIG, a road target detection method based on brain-like dual-pathway feature fusion specifically includes the following steps:

[0050] Step 1: This example uses the BDD100K dataset as the source of traffic scene video streams, which includes 13 different target types. Simulating the retinal sparse pulse coding, the traffic scene video streams in the BDD100K dataset are converted from continuous frame data to spatiotemporal differential pulse signals to generate a pulse density map. The specific method is as follows:

[0051] Calculate the pixel intensity change between adjacent frame images :

[0052]

[0053] in, 、 Represents the pixel value at coordinate (m, n) of the t-th frame and t-1 frame respectively. | | represents the absolute value.

[0054] Set when the pixel intensity of adjacent frame images changes Generates a pulse when the threshold is exceeded :

[0055]

[0056] Among them, θ base represents the basic pulse threshold, Entropy( ) represents the information entropy. η represents the adjustment coefficient, which is used to control the threshold Sensitivity to scene changes. In this embodiment, θ base =30mV,η=0.1.

[0057] The number of pulses per unit time is counted to generate a pulse density map. The pulse frequency of each pixel in the pulse density map represents the significance of the feature at that location. The higher the frequency, the greater the contribution of the feature to the detection task.

[0058] Step 2: Build Figure 2 The bionic dual-pathway network shown in FIG1 extracts features from the target recognition pathway and the spatial positioning pathway of the pulse density map generated in step 1, and then performs adaptive dynamic feature fusion through the dynamic fusion module.

[0059] s2.1. The object recognition pathway corresponds to the temporal lobe of the human brain and is the core area for visual object recognition. Its neurons integrate representations from edge features to complex objects through hierarchical integration, and are invariant to viewpoint and scale. The object recognition pathway uses sparse spike convolution to identify object details, such as vehicle type and pedestrian posture. The spike density map is processed sequentially through edge feature extraction, component-level feature aggregation, object classification, and an attention layer.

[0060] The edge feature extraction layer extracts local edge features from the pulse density map through pulse convolution. The component-level feature aggregation layer integrates component-level features from edge features through pulse convolution. The object classification and attention layer further extracts and integrates component-level features through a sparse pulse feature extraction network, adaptively assigns dynamic weights, and outputs a target detail feature tensor containing semantic category information.

[0061] The sparse spike feature extraction network includes a cascade of 3×3 spike convolution kernels, 5×5 spike convolution kernels, a spike attention module, and a fully spike connection layer. The 3×3 spike convolution kernel activates neurons only when the spike density is greater than 20Hz, retains target contour details through phase encoding, and extracts basic features such as edges and corners. The 5×5 spike convolution kernel uses horizontal links to simulate the long-range interaction of temporal cortical neurons to form a component-level representation, and introduces dynamic threshold adjustment to suppress redundant spikes in the background area. The spike attention module is used to simulate the response enhancement mechanism of temporal lobe neurons to highly significant targets, adaptively allocates dynamic weights according to the regional spike density D(m, n), generates spatial mask weighted features, simulates the response gain of the temporal lobe under attention regulation, improves detection accuracy, and reduces false detection rate:

[0062]

[0063] in, represents the dynamic weight at coordinate (m, n). θ2 = 40 Hz represents the pulse density threshold. Sigmoid represents the activation function.

[0064] The full pulse connection layer classifies the weighted feature map, generates category probability through pulse emission frequency statistics, and outputs the target detail feature tensor T∈R containing semantic category information H / 8×W / 8×D , where H and W represent the height and width of the input pulse density map, and D represents the feature dimension.

[0065] s2.2. The spatial positioning pathway corresponds to the parietal lobe region of the human brain. It uses dynamic pyramid pooling to achieve spatial positioning and motion trajectory prediction. The pulse density map is processed in sequence through a multi-scale spatial encoding layer, a spatial selection layer, and an environmental adaptation layer.

[0066] The multi-scale spatial coding layer first convolves the pulse density map, and then performs multi-scale feature fusion through dynamic pyramid pooling to output a multi-scale spatial heat map. The dynamic pyramid pooling includes two different levels of pooling schemes. The first level is 5×5 grid pooling to capture the position of large coarse-grained targets in the vicinity, and the second level is 10×10 grid pooling to locate the coordinates of small fine-grained targets in the distance. According to the entropy value of the pulse density map Automatically select pooling levels:

[0067]

[0068] Among them, p k It represents the probability of occurrence of the kth grayscale level in the pulse density map. When the entropy value is less than 2.5 bits, it indicates that the target distribution is concentrated and the first level is selected for pooling; otherwise, it indicates that the entropy value is high and the second level is used for pooling.

[0069] The spatial selection layer performs pulse-driven dilated convolution and non-maximum suppression on the multi-scale spatial heat map, wherein the pulse-driven dilated convolution is used to expand the receptive field to cover the entire spatial range of the target, and the non-maximum suppression is used to thin out redundant candidate boxes, thereby outputting a spatial coordinate map containing the normalized center coordinates, width, and height of the candidate boxes.

[0070] The environment adaptation layer dynamically adjusts the coordinate regression weights in the spatial coordinate map based on the global average pulse frequency, and outputs the final spatial positioning map (x, y, w, h), where (x, y) is the center coordinate of the candidate box and (w, h) is the width and height of the candidate box.

[0071] s2.3. The dynamic fusion module simulates the decision-making integration mechanism of the prefrontal region of the brain, adaptively allocates dual-pathway weights according to the target scale, and combines brain-like lateral inhibition technology to eliminate background interference, and adaptively fuses the multimodal information output by the target recognition pathway and the spatial positioning pathway.

[0072] The dynamic fusion module first solves the resolution difference of the two paths through the feature alignment layer and outputs the aligned semantic feature map and spatial feature map. Then, the predicted target bounding box area A is obtained from the spatial feature map, so as to adaptively allocate the weights W of the target recognition path and the spatial positioning path. what 、W where , fuse the semantic feature map and the spatial feature map:

[0073]

[0074]

[0075] Among them, S small 、S large They represent the set small target area threshold and large target area threshold respectively. In the BDD100K dataset, the average area of ​​traffic signs is 900. In this embodiment, S is defined as small =1,500, so as to better cover small targets; the area of ​​nearby vehicles generally exceeds 10,000, so define S large =20000, to distinguish medium and large targets.

[0076] Step 3: Use the pulse-ANN hybrid decoder to perform pulse density decoding on the fused feature map output in step 2 to obtain the category probability confidence:

[0077]

[0078] Among them, the vector pi cls represents the confidence of the i-th type of target. maxRepresents the maximum pulse frequency of the neuron. Nispike represents the pulse count of the i-th type output neuron.

[0079] The category with the highest confidence is taken as the recognition result, and the target category, confidence and bounding box information are output.

[0080] Example 2

[0081] Based on Example 1, this example simulates the V1 area in the visual cortex of the brain, uses a brain-like Gabor filter to extract the contour edges, color contrast, and local texture of objects from a traffic scene video stream, uses these basic features to trigger high-frequency pulses in high-contrast edge areas in the pulse density map, and low-frequency pulses in low-texture background areas, suppresses static background noise, completes data enhancement, and inputs the data-enhanced location pulse density map into a bionic dual-pathway network for feature extraction and fusion, and outputs recognition results.

[0082] Figure 3 、 4 Figures 5, 6, 7, and 8 are test examples of this embodiment, showing the target detection results in dim, dense, exposed, occluded, rainy, and blurred scenes. As can be seen from the figures, this method can achieve good detection results in different scenes.

[0083] To further illustrate the advantages of this method, we compare it with the existing target detection network on the BDD100K dataset and select FLOPs (floating point operations), operating power consumption, and AP. 50 As an evaluation indicator, FLOPs is used to evaluate the computational workload. The test results are shown in Table 1:

[0084] Table 1

[0085] method FLOPs Power consumption (J) <![CDATA[AP 50 (%)]]> Cascade R-CNN 2.94E+11 0.45 57.2 Sparse R-CNN 1.56E+11 0.36 57.9 YOLOv6 1.97E+10 4.54E-2 43.3 YOLOv7 1.20E+11 0.28 43.5 YOLOv8 1.56E+11 1.56E+11 44.8 FCOS 8.60E+10 0.20 47.8 YOLOX 2.68E+10 6.16E-2 49.7 This method 2.62E+09 4.73E-03 56.7

[0086] From the data in Table 1, we can see that compared with the existing target detection network, the proposed method has an AP 50 The detection accuracy and power consumption are 56.7% and 4.73E-03J respectively, which shows a good trade-off between detection accuracy and operating power consumption.

[0087] The above specific embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A road object detection method based on brain-inspired dual-pathway feature fusion, characterized by: Simulating the information processing process of the retina, the traffic scene video stream is converted from continuous frame data to pulse data to generate a pulse density map; Simulating the human brain's temporal lobe's ability to recognize object details and the parietal lobe's spatial perception function, a bionic dual-pathway network was constructed to extract features from the pulse density map from the target recognition pathway and the spatial positioning pathway, respectively. Then, based on the predicted target bounding box area A, the weights W of the target recognition pathway and the spatial positioning pathway were adaptively assigned. what 、W where , to achieve adaptive dynamic feature fusion: ； ； Among them, S small 、S large Respectively represent the set small target area threshold and large target area threshold; Use the pulse-ANN hybrid decoder to perform pulse density decoding on the fused feature map to obtain the category probability confidence: ； in, Represents the confidence of the i-th category target; represents the maximum pulse frequency of the neuron; represents the pulse count of the i-th type output neuron; T represents the time step of the spiking neural network conversion; Based on the category probability confidence calculation results, the road target detection results are output.

2. The method for road object detection based on brain-inspired dual-pathway feature fusion as claimed in claim 1, characterized in that: The method of converting traffic scene video stream into pulse density map is as follows: Calculate the pixel intensity change between adjacent frame images : ； in, 、 Respectively represent the pixel values ​​of the t-th frame and t-1 frame images at coordinates (m, n); | | represents the absolute value; Set when the pixel intensity of adjacent frame images changes When the threshold is exceeded, a pulse is generated, the number of pulses per unit time is counted, and a pulse density map is generated.

3. A road object detection method based on brain-inspired dual-pathway feature fusion as claimed in claim 1 or 2, characterized in that: Simulating the V1 area in the brain's visual cortex, the brain-like Gabor filter is used to extract the contour edges, color contrast and local texture of objects from the traffic scene video stream. High-frequency pulses are triggered in the high-contrast edge areas in the pulse density map, and low-frequency pulses are emitted in the low-texture background areas to suppress static background noise and complete the data enhancement operation.

4. The method for road object detection based on brain-inspired dual-pathway feature fusion as claimed in claim 1, characterized in that: The target recognition pathway processes the pulse density map sequentially through edge feature extraction, component-level feature aggregation layer, and target classification and attention layer; The edge feature extraction layer extracts local edge features from the pulse density map through pulse convolution; the component-level feature aggregation layer integrates component-level features from edge features through pulse convolution; the target classification and attention layer further extracts and integrates component-level features through a sparse pulse feature extraction network, adaptively assigns dynamic weights, and outputs a target detail feature tensor containing semantic category information.

5. The method for road object detection based on brain-inspired dual-pathway feature fusion as claimed in claim 4, characterized in that: The sparse spike feature extraction network includes a cascade of 3×3 spike convolution kernels, 5×5 spike convolution kernels, a spike attention module and a full spike connection layer; The 3×3 spike convolution kernel activates neurons only when the spike density is greater than 20Hz, and the 5×5 spike convolution kernel uses horizontal links to simulate the long-range interaction of temporal cortical neurons. The spike attention module is used to simulate the response enhancement mechanism of temporal lobe neurons to highly significant targets. It adaptively allocates dynamic weights according to the regional spike density D(m, n) to generate spatial mask weighted features: ； in, represents the dynamic weight at the coordinate (m, n); θ2 represents the pulse density threshold; Sigmoid represents the activation function; The fully pulse-connected layer classifies the weighted feature maps, generates category probabilities through pulse emission frequency statistics, and outputs target detail feature tensors containing semantic category information.

6. The method for road object detection based on brain-inspired dual-pathway feature fusion as claimed in claim 1, characterized in that: The spatial positioning pathway processes the pulse density map in sequence through a multi-scale spatial coding layer, a spatial selection layer, and an environment adaptation layer; The multi-scale spatial coding layer first convolves the pulse density map, and then uses dynamic pyramid pooling to select different pooling levels according to the entropy value in the pulse density map, performs multi-scale feature fusion, and outputs a multi-scale spatial heat map; The spatial selection layer performs pulse-driven dilated convolution and non-maximum suppression on the multi-scale spatial heat map, and outputs a spatial coordinate map containing the normalized center coordinates, width, and height of the candidate box; The environment adaptation layer dynamically adjusts the coordinate regression weights in the spatial coordinate map according to the global average pulse frequency, and outputs the final spatial positioning map (x, y, w, h); where (x, y) is the center coordinate of the candidate box, and (w, h) is the width and height of the candidate box.

7. The method for road object detection based on brain-inspired dual-pathway feature fusion as claimed in claim 6, characterized in that: The dynamic pyramid pooling performs 5×5 grid pooling when the entropy value of the pulse density map is lower than 2.5 bits, and performs 10×10 grid pooling in other cases.

8. The method for road object detection based on brain-inspired dual-pathway feature fusion as claimed in claim 1, characterized in that: The dynamic feature fusion simulates the frontal lobe area of ​​the brain. First, through the feature alignment layer, the resolution of the output feature maps of the target recognition pathway and the spatial positioning pathway are unified to obtain the aligned semantic feature map and the spatial feature map. Then, the predicted target bounding box area A is obtained from the spatial feature map, and the weights W of the target recognition pathway and the spatial positioning pathway are adaptively assigned. what 、W where , the semantic feature map and the spatial feature map are fused to output a unified representation that has both semantic information and spatial information.

9. The method for road object detection based on brain-inspired dual-pathway feature fusion as claimed in claim 8, characterized in that: S small 、S large The specific value is set according to the average area of ​​small and large targets in the recognition scene.

Images