Road target detection method based on brain-like dual-channel feature fusion

Abstract

The invention discloses a road target detection method based on brain-like dual-channel feature fusion, and belongs to the technical field of image recognition. According to the method, a target recognition and spatial positioning dual-channel mechanism based on a human brain vision system is provided, a traffic scene video stream is converted into a pulse density map, the pulse density map is input into a bionic dual-channel network, feature extraction is performed from a target recognition channel and a spatial positioning channel, then adaptive dynamic feature fusion is performed through a dynamic fusion module, and a traffic scene video stream is obtained. By strictly mapping a human brain visual path, enhancing the theoretical reliability of the technical scheme and introducing pulse convolution, high real-time performance and calculation efficiency are ensured, the precision, speed and power consumption of traffic target detection are balanced, and the method is suitable for a vehicle-mounted traffic target detection task and meets the requirement of an automatic driving system for target detection. Experiments prove that the mAP of the method for small target detection is improved by 15.8%, and the energy consumption is reduced by 40%.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, relates to a method for detecting traffic targets, and particularly relates to a method for detecting road targets based on brain-inspired dual-pathway feature fusion. Background Art

[0002] Traffic target detection is one of the core technologies of autonomous driving technology and intelligent transportation systems. Traffic target detection refers to the real-time recognition and positioning of various traffic targets on the road through in-vehicle sensors, including vehicles, pedestrians, traffic signs, traffic lights, etc. Traditional target detection methods usually rely on deep convolutional neural networks (CNNs) or Transformer models to extract visual features of targets by learning a large amount of image data. However, CNNs rely on convolutional kernels of fixed scales and are difficult to effectively process multi-scale targets (such as small vehicles at a distance and large trucks at close range), and pooling operations result in loss of spatial information and a high missed detection rate for small targets. Although Transformer can model temporal dependencies, its computational complexity is high, making it difficult to meet the real-time requirements of in-vehicle devices. To alleviate the above problems, existing technologies use methods such as multi-scale feature fusion, lightweight design, attention mechanisms, and multi-modal data fusion to solve them. Multi-scale feature fusion uses feature pyramid networks or dilated convolutions to enhance multi-scale detection, but static weight allocation is difficult to adapt to the dynamic changes of targets and increases the computational amount. Lightweight design compresses models (such as MobileNet, YOLO-Lite) through network pruning, quantization, or knowledge distillation, but the accuracy loss is significant. Attention mechanisms introduce channel / space attention to enhance the response of key regions, but lack a dynamic triggering mechanism for sudden targets. Multi-modal data fusion methods fuse multi-source data such as cameras, lidar, and infrared to improve robustness, but feature-level fusion (such as early / late fusion) is prone to introducing redundant calculations, and cross-modal alignment errors lead to performance fluctuations (such as spatio-temporal asynchrony problems).

[0003] Complex traffic scenarios include multiple targets, multiple scales, high-density occlusions, extreme lighting (backlighting / nighttime), and irregular targets (non-motor vehicles, special vehicles). Existing technologies can partially alleviate the above problems through multi-modal fusion, lightweight design, etc., but there are still defects such as redundant multi-modal feature extraction, insufficient dynamic weight allocation, and the contradiction between model complexity and energy efficiency in complex traffic scenarios and dynamic scenarios.

[0004] Current traffic target detection technologies face core bottlenecks such as poor multi-scale adaptability, low computational efficiency, and insufficient dynamic fusion mechanisms in complex scenarios and dynamic environments, and there is an urgent need for an innovative solution that breaks through the inherent limitations of existing deep learning frameworks. Summary of the Invention

[0005] In view of the deficiencies of the prior art, the present invention proposes a road target detection method based on brain-inspired dual-pathway feature fusion, which introduces the dual-pathway mechanism of the human brain visual system into traffic target detection. Combining with spiking neural networks, the target recognition and spatial localization tasks are respectively handed over to two pathways for processing, and then feature fusion technology is used for combination, providing a new efficient and accurate target detection method for intelligent transportation systems.

[0006] A road target detection method based on brain-inspired dual-pathway feature fusion specifically includes the following steps:

[0007] Step 1: Real-time capture the traffic scene video stream through an in-vehicle camera, simulate the information processing process of the retina, convert the traffic scene video stream from continuous frame data into pulse data, and generate a pulse density map. The specific method is as follows:

[0008] Calculate the pixel intensity change between adjacent frame images :

[0009]

[0010] Among them, , respectively represent the pixel values of the t-th frame and the (t - 1)-th frame images at the coordinates (m, n). || represents the absolute value.

[0011] Set that when the pixel intensity change between adjacent frame images exceeds the threshold, a pulse is generated :

[0012]

[0013] Among them, θ base represents the basic pulse threshold, Entropy( ) represents the information entropy. η represents the adjustment coefficient, which is used to control the sensitivity of the threshold to scene changes.

[0014] Statistical the number of pulses within a unit time to generate a pulse density map.

[0015] Preferably, simulate the V1 region in the brain visual cortex, use a brain-inspired Gabor filter to extract the contour edges, color contrast, and local texture of objects from the traffic scene video stream, and use these basic features to trigger high-frequency pulses in the high-contrast edge regions of the pulse density map, and emit low-frequency pulses in the low-texture background regions to suppress static background noise and complete the data enhancement operation.

[0016] Step 2: Construct a bionic dual-pathway network, extract features from the pulse density map generated in Step 1 from the target recognition pathway and the spatial localization pathway respectively, and then perform adaptive dynamic feature fusion through a dynamic fusion module.

[0017] S2.1. The target recognition pathway corresponds to the temporal lobe of the human brain and is responsible for object detail recognition. It processes the pulse density map through edge feature extraction, part-level feature aggregation, and target classification and attention layers in sequence.

[0018] The edge feature extraction extracts local edge features from the pulse density map through pulse convolution. The part-level feature aggregation layer integrates part-level features from the edge features through pulse convolution. The target classification and attention layer further extracts and integrates the part-level features through a sparse pulse feature extraction network, adaptively assigns dynamic weights, and outputs a target detail feature tensor containing semantic category information.

[0019] The sparse pulse feature extraction network includes multiple pulse convolution kernels of different sizes, a pulse attention module, and a fully pulsed connection layer. Among them, the smaller pulse convolution kernels are used to extract local features in the pulse density map. The larger pulse convolution kernels use lateral connections to simulate the long-range interactions of neurons in the temporal lobe cortex for feature aggregation. The pulse attention module is used to simulate the response enhancement mechanism of temporal lobe neurons to highly salient targets, adaptively assigns dynamic weights according to the regional pulse density D(m, n), and outputs a weighted feature map:

[0020]

[0021] where represents the dynamic weight at coordinates (m, n). θ 2 represents the pulse density threshold, and Sigmoid represents the activation function.

[0022] The fully pulsed connection layer classifies the weighted feature map, generates class probabilities through pulse firing frequency statistics, and outputs a target detail feature tensor containing semantic category information.

[0023] S2.2. The spatial localization pathway corresponds to the parietal region of the human brain and is responsible for spatial perception. It processes the pulse density map through a multi-scale spatial encoding layer, a spatial selection layer, and an environment adaptation layer in sequence.

[0024] The multi-scale spatial encoding layer first convolves the pulse density map and then performs multi-scale feature fusion through dynamic pyramid pooling, outputting a multi-scale spatial heat map. The dynamic pyramid pooling includes two pooling schemes of different sizes, which are selected according to the entropy value in the pulse density map.

[0025] The spatial selection layer performs pulse-driven dilated convolution and non-maximum suppression on the multi-scale spatial heat map, expands the receptive field, and sparsely outputs a spatial coordinate map for redundant candidate boxes, including the normalized center coordinates, width, and height of the candidate boxes.

[0026] The environment adaptation layer dynamically adjusts the coordinate regression weights in the spatial coordinate map according to the global average pulse frequency, and the output is the final spatial localization map (x, y, w, h). Among them, (x, y) are the center coordinates of the candidate box, and (w, h) are the width and height of the candidate box.

[0027] S2.3. The dynamic fusion module simulates the prefrontal region of the brain and adaptively fuses the multimodal information output by the target recognition pathway and the spatial localization pathway, and outputs a unified representation with both semantic information and spatial information.

[0028] The dynamic fusion module first solves the resolution difference between the two pathways through the feature alignment layer, and outputs the aligned semantic feature map and spatial feature map. Then, the predicted target bounding box area A is obtained from the spatial feature map, so as to adaptively allocate the weights W what 、W where , and fuses the semantic feature map and the spatial feature map:

[0029]

[0030]

[0031] Among them, S small 、S large respectively represent the set small target area threshold and large target area threshold.

[0032] Step 3. Use the hybrid decoder to perform pulse density decoding on the fused feature map output in step 2 to obtain the class probability confidence:

[0033]

[0034] Among them, K represents the number of target categories. The K-dimensional vector pi cls represents the confidence of the i-th class. f max represents the maximum pulse frequency of the neuron. Ni spike represents the pulse count of the i-th class output neuron.

[0035] Take the class with the highest confidence as the recognition result, and output the target category, confidence, and bounding box information.

[0036] The present invention has the following beneficial effects:

[0037] 1. This method proposes a dual-path mechanism for target recognition (what) and spatial localization (where) based on the human brain visual system, which balances the accuracy, speed, and power consumption of vehicle traffic target detection and is applicable to vehicle traffic target detection tasks. By strictly mapping the human brain visual pathway, the theoretical reliability of the technical solution is enhanced, and it is experimentally verified that the mAP for small target detection is increased by 15.8% and the energy consumption is reduced by 40%.

[0038] 2. This method can accurately identify traffic targets in complex traffic scenarios. At the same time, based on the processing method of the spiking neural network, it can ensure that this method has high real-time performance and computational efficiency, meeting the requirements of the autonomous driving system for target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is the mapping relationship between data processing and the visual cortex;

[0040] Figure 2 is a schematic diagram of the bionic dual-path network structure;

[0041] Figure 3 is a test sample in the dim scene in Embodiment 2;

[0042] Figure 4 is a test sample in the dense scene in Embodiment 2;

[0043] Figure 5 is a test sample in the overexposed scene in Embodiment 2;

[0044] Figure 6 is a test sample in the occlusion scene in Embodiment 2;

[0045] Figure 7 is a test sample in the rainy day scene in Embodiment 2;

[0046] Figure 8 is a test sample in the blurred scene in Embodiment 2. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] The present invention will be further explained below with reference to the accompanying drawings;

[0048] Embodiment 1

[0049] As Figure 1 shown, a road target detection method based on brain-inspired dual-path feature fusion specifically includes the following steps:

[0050] Step 1. In this embodiment, the BDD100K dataset is used as the source of the traffic scene video stream, which includes a total of 13 different target types. Simulating the sparse spike coding of the retina, the traffic scene video stream in the BDD100K dataset is converted from continuous frame data into spatio-temporal differential spike signals to generate a spike density map. The specific method is:

[0051] Calculate the pixel intensity change of adjacent frame images :

[0052]

[0053] Among them, , They respectively represent the pixel values of the t-th frame and the (t - 1)-th frame images at the coordinates (m, n). | | represents the absolute value.

[0054] Set to generate a pulse when the pixel intensity change of adjacent frame images exceeds the threshold :

[0055]

[0056] where, θ base represents the basic pulse threshold, Entropy( ) represents the information entropy. η represents the adjustment coefficient, which is used to control the threshold sensitivity to scene changes. In this embodiment, θ base = 30mV, η = 0.1.

[0057] Count the number of pulses within a unit time to generate a pulse density map. The pulse frequency of each pixel point in the pulse density map characterizes the significance of the feature at that position. The higher the frequency, the greater the contribution of the feature to the detection task.

[0058] Step 2, construct a bionic dual-channel network as Figure 2 shown, and perform feature extraction on the pulse density map generated in Step 1 from the target recognition channel and the spatial localization channel respectively, and then perform adaptive dynamic feature fusion through the dynamic fusion module.

[0059] s2.1. The target recognition channel corresponds to the temporal lobe of the human brain, which is the core area for visual object recognition. Its neurons integrate the representations from edge features to complex objects hierarchically and have view and scale invariance. The target recognition channel recognizes target details, such as vehicle models and pedestrian postures, through sparse pulse convolution, and processes the pulse density map through edge feature extraction, part-level feature aggregation, and target classification and attention layers in sequence.

[0060] The edge feature extraction extracts local edge features from the pulse density map through pulse convolution. The part-level feature aggregation layer integrates part-level features from the edge features through pulse convolution. The target classification and attention layer further extracts and integrates the part-level features through a sparse pulse feature extraction network, and adaptively assigns dynamic weights to output a target detail feature tensor containing semantic category information.

[0061] The sparse pulse feature extraction network includes cascaded 3×3 pulse convolutional kernels, 5×5 pulse convolutional kernels, a pulse attention module, and a fully pulse-connected layer. Among them, the 3×3 pulse convolutional kernel activates neurons only when the pulse density is greater than 20Hz, retains the details of the target contour through phase encoding, and extracts basic features such as edges and corners. The 5×5 pulse convolutional kernel uses lateral connections to simulate the long-range interactions of neurons in the temporal cortex, forms part-level representations, and introduces dynamic threshold adjustment to suppress redundant pulses in the background area. The pulse attention module is used to simulate the response enhancement mechanism of temporal lobe neurons to highly salient targets, adaptively allocate dynamic weights according to the regional pulse density D(m, n), generate spatially masked weighted features, simulate the response gain of the temporal lobe under attention regulation, improve the detection accuracy, and reduce the false detection rate:

[0062]

[0063] Among them, represents the dynamic weight at the coordinate (m, n). θ 2 =40Hz, representing the pulse density threshold. Sigmoid represents the activation function.

[0064] The fully pulse-connected layer classifies the weighted feature map, generates class probabilities through pulse firing frequency statistics, and outputs the target detail feature tensor T∈R H / 8×W / 8×D , where H and W represent the height and width of the input pulse density map, and D represents the feature dimension.

[0065] s2.2. The spatial localization pathway corresponds to the parietal lobe region of the human brain, and realizes spatial localization and motion trajectory prediction by means of dynamic pyramid pooling. The pulse density map is processed sequentially through a multi-scale spatial encoding layer, a spatial selection layer, and an environment adaptation layer.

[0066] The multi-scale spatial encoding layer first convolves the pulse density map, and then performs multi-scale feature fusion through dynamic pyramid pooling to output a multi-scale spatial heat map. The dynamic pyramid pooling includes two different levels of pooling schemes. The first level is 5×5 grid pooling, which captures the positions of large nearby targets with coarse granularity. The second level is 10×10 grid pooling, which locates the coordinates of small distant targets with fine granularity. According to the entropy value of the pulse density map, the pooling level is automatically selected:

[0067]

[0068] Among them, p k represents the occurrence probability of the k-th gray level in the pulse density map. When the entropy value is lower than 2.5bit, it means that the target distribution is concentrated, and the first level is selected for pooling; otherwise, it means that the entropy value is higher, and the second level is used for pooling.

[0069] The spatial selection layer performs dilated convolution with pulse driving and non-maximum suppression on the multi-scale spatial heat map. The dilated convolution with pulse driving is used to expand the receptive field to cover the overall spatial range of the target, and the non-maximum suppression is used to sparsify the redundant candidate boxes, thereby outputting a spatial coordinate map containing the normalized central coordinates, width, and height of the candidate boxes.

[0070] The environment adaptation layer dynamically adjusts the coordinate regression weights in the spatial coordinate map according to the global average pulse frequency, and the output is the final spatial localization map (x, y, w, h). Where (x, y) are the central coordinates of the candidate box, and (w, h) are the width and height of the candidate box.

[0071] S2.3. The dynamic fusion module simulates the decision integration mechanism of the prefrontal region of the brain, adaptively allocates dual-pathway weights according to the target scale, and combines the brain-like lateral inhibition technology to eliminate background interference, and adaptively fuses the multi-modal information output by the target recognition pathway and the spatial localization pathway.

[0072] The dynamic fusion module first solves the resolution difference between the dual pathways through the feature alignment layer, and outputs the aligned semantic feature map and spatial feature map. Then, the predicted target bounding box area A is obtained from the spatial feature map, so as to adaptively allocate the weights W what 、W where , and fuse the semantic feature map and the spatial feature map:

[0073]

[0074]

[0075] Among them, S small 、S large respectively represent the set small target area threshold and large target area threshold. In the BDD100K dataset, the average area of traffic signs is 900. In this embodiment, S small = 1,500 is defined to better cover small targets; the area of nearby vehicles generally exceeds 10,000, so S large = 20,000 is defined to distinguish medium and large targets.

[0076] Step 3. Through the pulse-ANN hybrid decoder, perform pulse density decoding on the fused feature map output in step 2 to obtain the class probability confidence:

[0077]

[0078] Among them, the vector pi cls represents the confidence of the i-th class target. f maxIt represents the maximum firing frequency of neurons. Nispike represents the spike count of the i-th type of output neurons.

[0079] The category with the highest confidence is taken as the recognition result, and the target category, confidence, and bounding box information are output.

[0080] Embodiment 2

[0081] Based on Embodiment 1, this embodiment simulates the V1 region in the visual cortex of the brain, uses a brain-inspired Gabor filter to extract the contour edges, color contrast, and local texture of objects from the traffic scene video stream, uses these basic features to trigger high-frequency pulses in the high-contrast edge regions of the pulse density map, emit low-frequency pulses in the low-texture background regions, suppress static background noise, complete the data augmentation operation, and input the pulse density map after data augmentation into the bionic dual-path network for feature extraction and fusion, and output the recognition result.

[0082] Figure 3 、 4 Figures 5, 6, 7, and 8 are the test examples of this embodiment, showing the target detection results in dim, dense, overexposed, occluded, rainy, and blurred scenes. It can be seen from the figures that this method can achieve good detection results in the face of different scenes.

[0083] To further illustrate the advantages of this method, a comparison is made with existing object detection networks on the BDD100K dataset, and FLOPs (floating point operations), running power consumption, and AP 50 are selected as evaluation metrics. FLOPs is used to evaluate the amount of computation. The test results are shown in Table 1:

[0084] Table 1

[0085] Method FLOPs Power consumption (J) <![CDATA[AP 50 (%)]]> Cascade R-CNN 2.94E+11 0.45 57.2 Sparse R-CNN 1.56E+11 0.36 57.9 YOLOv6 1.97E+10 4.54E-2 43.3 YOLOv7 1.20E+11 0.28 43.5 YOLOv8 1.56E+11 1.56E+11 44.8 FCOS 8.60E+10 0.20 47.8 YOLOX 2.68E+10 6.16E-2 49.7 This method 2.62E+09 4.73E-03 56.7

[0086] It can be seen from the data in Table 1 that compared with existing object detection networks, the AP 50 and power consumption of this method are 56.7% and 4.73E-03 J respectively, and there is a better trade-off between detection accuracy and running power consumption.

[0087] The above specific implementation manners are used to explain and illustrate the present invention, rather than limiting the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims of the present invention fall within the protection scope of the present invention.

Claims

1. A road target detection method based on brain-like dual-pathway feature fusion, characterized by: Simulate the information processing process of the retina, convert the traffic scene video stream from continuous frame data to pulse data, and generate a pulse density map; By simulating the human brain's temporal lobe's ability to recognize object details and the parietal lobe's spatial perception function, a bionic dual-pathway network is constructed to extract features from the pulse density map from the target recognition pathway and the spatial positioning pathway, and then weights are assigned to the target recognition pathway and the spatial positioning pathway based on the predicted target bounding box area A to achieve adaptive dynamic feature fusion. A hybrid decoder is used to perform pulse density decoding on the fused feature map to obtain the category probability confidence and output the road object detection result.

2. A road target detection method based on brain-like dual-pathway feature fusion as claimed in claim 1, characterized in that: The method of converting traffic scene video stream into pulse density map is: Calculate the pixel intensity change between adjacent frame images : ； in, , Respectively represent the pixel values ​​of the t-th frame and t-1-frame images at the coordinates (m, n); | | represents the absolute value; Set when the pixel intensity of adjacent frame images changes Generates a pulse when the threshold is exceeded : ； Among them, θ base represents the basic pulse threshold, Entropy ( ) represents the information entropy; η represents the adjustment coefficient; Count the number of pulses per unit time and generate a pulse density map.

3. A road target detection method based on brain-like dual-pathway feature fusion as claimed in claim 1 or 2, characterized in that: The V1 area in the visual cortex of the brain is simulated, and the brain-like Gabor filter is used to extract the contour edges, color contrast and local texture of objects from the traffic scene video stream. High-frequency pulses are triggered in the high-contrast edge areas in the pulse density map, and low-frequency pulses are emitted in the low-texture background areas to suppress static background noise and complete the data enhancement operation.

4. A road target detection method based on brain-like dual-pathway feature fusion as claimed in claim 1, characterized in that: The target recognition pathway processes the pulse density map sequentially through edge feature extraction, component-level feature aggregation, and target classification and attention layers; The edge feature extraction extracts local edge features from the pulse density map through pulse convolution; the component-level feature aggregation layer integrates component-level features from edge features through pulse convolution; the target classification and attention layer further extracts and integrates component-level features through a sparse pulse feature extraction network, and adaptively assigns dynamic weights to output a target detail feature tensor containing semantic category information.

5. A road target detection method based on brain-like dual-pathway feature fusion as claimed in claim 4, characterized in that: The sparse spike feature extraction network includes a cascaded 3×3 spike convolution kernel, a 5×5 spike convolution kernel, a spike attention module and a full spike connection layer; The 3×3 pulse convolution kernel activates neurons only when the pulse density is greater than 20Hz, and the 5×5 pulse convolution kernel uses horizontal links to simulate the long-range interaction of temporal cortical neurons. The pulse attention module is used to simulate the response enhancement mechanism of temporal lobe neurons to highly significant targets, and adaptively allocates dynamic weights according to the regional pulse density D(m, n) to generate spatial mask weighted features: ； in, represents the dynamic weight at the coordinate (m, n); θ2 represents the pulse density threshold; Sigmoid represents the activation function; The fully pulse-connected layer classifies the weighted feature maps, generates category probabilities through pulse emission frequency statistics, and outputs the target detail feature tensor T containing semantic category information.

6. A road target detection method based on brain-like dual-pathway feature fusion as claimed in claim 1, characterized in that: The spatial positioning pathway processes the pulse density map in sequence through a multi-scale spatial coding layer, a spatial selection layer, and an environment adaptation layer; The multi-scale spatial coding layer first convolves the pulse density map, and then selects different pooling levels according to the entropy value in the pulse density map through dynamic pyramid pooling, performs multi-scale feature fusion, and outputs a multi-scale spatial heat map; The spatial selection layer performs pulse-driven dilated convolution and non-maximum suppression on the multi-scale spatial heat map, and outputs a spatial coordinate map containing the normalized center coordinates, width, and height of the candidate box; The environment adaptation layer dynamically adjusts the coordinate regression weights in the spatial coordinate map according to the global average pulse frequency, and the output is the final spatial positioning map (x, y, w, h); where (x, y) is the center coordinate of the candidate box, and (w, h) is the width and height of the candidate box.

7. A road target detection method based on brain-like dual-pathway feature fusion as claimed in claim 6, characterized in that: The dynamic pyramid pooling performs 5×5 grid pooling when the entropy value of the pulse density map is lower than 2.5 bits, and performs 10×10 grid pooling in other cases.

8. A road target detection method based on brain-like dual-pathway feature fusion as claimed in claim 1, characterized in that: The dynamic fusion module simulates the frontal lobe area of ​​the brain. First, through the feature alignment layer, the resolution of the output feature map of the target recognition pathway and the spatial positioning pathway is unified to obtain the aligned semantic feature map and the spatial feature map. Then, the predicted target bounding box area A is obtained from the spatial feature map, and the weight W of the target recognition pathway and the spatial positioning pathway is adaptively allocated. what , W where , the semantic feature map and the spatial feature map are fused to output a unified representation of both semantic information and spatial information: ； ； Among them, S small , S large They represent the set small target area threshold and large target area threshold respectively.

9. A road target detection method based on brain-like dual-pathway feature fusion as claimed in claim 8, characterized in that: S small , S large The specific value is set according to the average area of ​​small targets and large targets in the recognition scene.

10. A road target detection method based on brain-like dual-pathway feature fusion as claimed in claim 1, characterized in that: The pulse-ANN hybrid decoder performs pulse density decoding on the fused feature map to obtain the category probability confidence: ； Where K represents the number of target categories; pi cls represents the confidence of the i-th target; f max represents the maximum pulse frequency of the neuron; Ni spike represents the pulse count of the i-th type output neuron.

Images