A vehicle detection method and system with multi-scale saliency hybrid enhancement

Abstract

This invention proposes a multi-scale saliency hybrid enhancement method and system for vehicle detection, improving upon the existing DETR (DEtection Transformer) object detection framework. It includes a saliency feature selection mechanism and a multi-scale hybrid proposal module, aiming to build an efficient vehicle detection model. Compared to traditional road testing algorithms, this module significantly improves detection accuracy in complex scenes (such as motion blur and dense traffic conditions) and effectively refines features. The key innovation of this invention lies in combining a multi-scale hybrid proposal module with images of different resolutions, overcoming the feature decay problem of high-level prior information in traditional methods and fully utilizing inter-layer relationships to improve feature extraction accuracy. This method integrates the semantic information of low-resolution images with the detailed information of high-resolution images, reducing the number of model parameters, decreasing computational resource requirements, and improving overall detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of target detection algorithms in computer vision, specifically to a vehicle detection method and system with multi-scale saliency hybrid enhancement. Background Technology

[0002] Deep learning-based object detection is generally divided into two categories based on its detection method: two-stage detection and single-stage detection. The former first generates predicted bounding boxes, which may contain the target to be detected. These boxes are then analyzed and fine-tuned to obtain the desired detection result. Representative algorithms include Region Convolutional Neural Networks (R-CNN), Fast R-CNN, Faster R-CNN, and Mask R-CNN. The latter directly extracts image features within the network and directly classifies and regresses the positions of potential targets within predefined anchor boxes on the feature map, thereby detecting all targets. Representative algorithms include SSD and the YOLO series.

[0003] Convolutional Neural Networks (CNNs) are the mainstream deep neural networks used in deep learning tasks such as semantic segmentation and 3D object detection. CNNs can efficiently extract feature tensors from input data. However, at each convolutional step, a CNN can only capture the feature tensor of a local patch. To expand the receptive field of a deep neural network, CNNs require several rounds of convolutional operations. However, hierarchical convolutional operations still affect the extraction of global-scale feature tensors. Unlike CNNs, transformers utilize self-attention modules for feature tensor extraction. The transformer first divides the input data into a series of smaller tensors, and then runs several rounds of self-attention modules to extract the features of the input. In this way, the transformer has a much larger receptive field and can better capture the spatial information of the input data. Furthermore, the transformer can better express the relationships between different patches of the input data using formulas.

[0004] In the field of object detection, effectively representing and processing objects at different scales remains a core challenge. Traditional CNN-based detectors widely employ multi-scale feature extraction techniques, such as FPN, BiFPN, PANet, and NAS-FPN, which have been proven to significantly improve model efficiency and performance. In contrast, the efficiency problem of the DETR model (an end-to-end object detection model based on Transformer) in handling multi-scale features has not been fully resolved. To address the limitations of existing technologies, a DETR-based urban traffic vehicle detection module is needed, which can effectively utilize multi-scale features in Transformer-based object detectors. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention proposes a vehicle detection method with multi-scale saliency hybrid enhancement, which can effectively utilize multi-scale features in Transformer-based target detectors. The method includes the following steps:

[0006] Step 1: Acquire real scan image data from the road test camera;

[0007] Step 2: Use a pre-convolutional neural network to obtain multi-scale feature images corresponding to the data;

[0008] Step 3: Filter multi-scale saliency features by calculating saliency scores;

[0009] Step 4: Through the multi-scale hybrid proposal module, effective features are selected from the highest layer (i.e., the low-resolution layer) and mapped to the low-level feature images to obtain a multi-scale hybrid feature group.

[0010] Step 5: Feed the multi-scale mixed feature group into the encoder based on the transformer structure to obtain the enhanced multi-scale mixed feature group, and use the decoder to predict and output the category and probability of the detected target.

[0011] Furthermore, in step 2, the input image data is processed by a pre-convolutional neural network ResNet to obtain a series of feature maps F. l Each feature map Representing different scale levels of resolution, where C is the number of channels in each feature map, and H... l and W l These are the height and width of the feature map at layer l, respectively.

[0012] Furthermore, the specific implementation method of step 3 is as follows:

[0013] Using scale-independent saliency scores as the supervision target, salient foreground features are extracted. For each feature at position (i, j) in the feature map... The saliency score of the feature corresponds to the coordinates c = (x, y) in the original image. Determined according to the following rules:

[0014]

[0015] in The text represents the true bounding box, where w and h represent the width and height of the true bounding box, respectively. The saliency score is calculated based on the relative distance to the center of the true bounding box.

[0016]

[0017] Where Δx and Δy represent the distance between c = (x, y) and the center of the corresponding ground truth box.

[0018] Furthermore, the specific implementation of the multi-scale hybrid proposal module in step 4 is as follows:

[0019] In step 3, obtain the saliency scores corresponding to all features, and then use a dynamic threshold to confirm the effective features, that is, adjust the threshold of the saliency score so that the effective region occupies a specific percentage of the entire map area; next, starting from the effective features of the highest layer, construct a feature group layer by layer downward, define a circular neighborhood N(p, r) with radius r, and select saliency features within this circular neighborhood;

[0020] N(p, r) = {q|||qp||≤r}

[0021] Here, p is the position mapped from the previous layer, i.e., the center of the circular neighborhood corresponding to the current layer, and q is the position of the salient feature in the current layer. To ensure that high-level information is accurately transmitted to lower layers, a circular convolution is used to simulate the construction process of this circular neighborhood. r corresponds to the size of a circular convolution kernel, which upsamples and maps the position encoding of the salient feature in layer i to layer j (i, -1), and its position in layer j (i, -1) is denoted as P. l-1 In P l-1 Within the circular neighborhood N(p, r), if a salient feature of layer l-1 exists, it is considered valid and selected for encoding by the encoder mentioned in step 5. This process continues until layer 1.

[0022] S l ={F l (q)|q∈N(p,r),p∈P l-1}

[0023] Where F l (q) represents the feature in the , layer; q is the salient feature position in the neighborhood N(p, r), S l The corresponding layer is the set of effective salient features that conform to the mapping relationship.

[0024] Furthermore, the encoder in step 5 is implemented as follows:

[0025] Within the local neighborhood of the multi-scale mixed feature group, the multi-scale mixed feature group is fed into an encoder based on a transformer structure for encoding: First, a self-attention operation is performed on the salient mixed feature group. After capturing global long-distance connections through ordinary self-attention processing, it is then put into deformable attention to enhance local focusing processing, thereby obtaining enhanced multi-scale mixed features.

[0026] Furthermore, the specific implementation of self-attention operations is as follows:

[0027] For a given feature, there exists a query vector Q. i Pointing to itself, other feature vectors are referred to as key vectors relative to it. For each feature, there exists a value vector. Refers to itself; first, select a query vector Q corresponding to a feature. i The key vector within its multi-scale mixed feature group range Similarity is calculated and normalized using the softmax function to generate attention weights, which represent the correlation between two features. Finally, these weights are used to adjust the value vectors within the neighborhood. Perform weighted summation to generate the enhanced feature representation;

[0028]

[0029] Where Qi, This is achieved by applying three different linear transformations to the original feature x; d k It represents the dimension of the query and key vectors, used to prevent the dot product result from becoming too large.

[0030] Furthermore, to represent the order relationship, a dimension of d is added to the original features. model The location encoding is specifically expressed as follows:

[0031]

[0032]

[0033] Where PE represents the encoding position, pos represents the position of the feature in the input sequence, i represents the dimension index of the feature vector of the input sequence, and d model This represents the dimension of each input vector.

[0034] Furthermore, the specific implementation of deformable attention is as follows:

[0035] Given an input feature map, the formula for calculating deformable attention is:

[0036]

[0037] Among them, Q i Viewed as a query vector, it is generated from the original feature x through a linear transformation, where q is the corresponding index and p q Represents Q i The position of Δp is a two-dimensional vector, called the reference point. mqk It is an offset, representing the position from the query position Q. i A certain sampling position at the offset; ΔA mqk It is the attention weight, which represents the query Q. i Correlation with a specific sampling location; weight ΔA mqk Through query Q i The attention is calculated; k is the index of the key vector, K is the total number of sampled keys, and m represents the m-th attention head. There are a total of M attention heads, calculated through W. m The matrix further linearly transforms the feature representations of multiple heads and integrates the results of all heads, W′. m This is used to unify the calculation of different local features within a single attention head, making it better integrated with the query vector.

[0038] The present invention also provides a vehicle detection system with multi-scale saliency hybrid enhancement, comprising:

[0039] One or more processors;

[0040] A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement a multi-scale saliency hybrid enhancement vehicle detection method as described above.

[0041] Compared with existing technologies, the advantages of this invention are: the multi-scale hybrid feature tunnel designed in this invention has a simple and efficient structure. A significant advantage of this network is that it achieves the effect of a 6-layer encoder used in traditional baseline models with only one encoder layer. This design achieves refined feature processing by efficiently utilizing cross-layer features, significantly improving the detection performance and efficiency of the model. Attached Figure Description

[0042] Figure 1 This is a flowchart of an embodiment of the present invention;

[0043] Figure 2 This is a flowchart of the overall model processing of the present invention;

[0044] Figure 3This is a flowchart of the multi-scale hybrid suggestion module processing according to an embodiment of the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0046] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0047] The present invention will be further described below with reference to specific embodiments, but these are not intended to limit the scope of the invention.

[0048] like Figure 1 As shown in the figure, this invention discloses a vehicle detection method with multi-scale saliency hybrid enhancement, comprising the following steps:

[0049] Step 1: By integrating ResNet as the feature extraction network, introducing feature tunneling, and combining a dual attention mechanism, a multi-scale hybrid proposal module for urban traffic vehicle detection based on DETR is constructed.

[0050] Step 2: Obtain the training image dataset in traffic, perform image annotation processing, and then randomly select a certain number of images and label data to be fed into the detection model that integrates the multi-scale hybrid proposal module.

[0051] Step 3: The input image is processed by ResNet to extract features, resulting in a series of feature maps F. l Each feature map Representing different resolution scale levels, a total of n feature maps are obtained (l∈[1,n]), in this example n=4, C is the number of channels in each feature map, H l and W l These are the height and width of the l-th layer feature map, respectively. Generally, the highest layer image is set to the lowest resolution to provide semantic information, while the resolution is doubled for each subsequent layer, meaning one pixel in the upper layer corresponds to four pixels in the lower layer, and so on, to obtain four feature images that follow the feature pyramid structure.

[0052] Step 4: Extract salient foreground features using scale-independent saliency scores as the supervision target, not just labeling the foreground as 1 and the background as 0. Specifically, for each feature at position (i,j) in the l-th feature map... The saliency score of the feature corresponds to the coordinates c = (x, y) in the original image. Determined according to the following rules:

[0053]

[0054] in The text represents the true bounding box, where w and h represent the width and height of the true bounding box, respectively. The saliency score is calculated based on the relative distance to the center of the true bounding box.

[0055]

[0056] Where Δx and Δy represent the distance between c = (x, y) and the center of the corresponding ground truth box.

[0057] Step 4: Obtain the saliency scores corresponding to all features in Step 3, and then confirm the effective features through dynamic thresholding, that is, adjust the threshold of the saliency score so that the effective region occupies a specific percentage of the entire map area; next, starting from the effective features of the highest layer, construct a feature group layer by layer downwards, define a circular neighborhood N(p, r) with radius r, and select saliency features within this circular neighborhood;

[0058] N(p, r) = {q|||qp||≤r}

[0059] Here, p is the position mapped from the previous layer, i.e., the center of the circular neighborhood corresponding to the current layer, and q is the position of the salient feature in the current layer. To ensure that high-level information is accurately transmitted to lower layers, a circular convolution is used to simulate the construction process of this circular neighborhood. r corresponds to the size of a circular convolution kernel, which upsamples and maps the position encoding of the salient feature in layer i to layer j (i, -1), and its position in layer j (i, -1) is denoted as P. l-1 In P l-1 Within the circular neighborhood N(p, r), if a salient feature of layer l-1 exists, it is considered valid and selected for encoding by the encoder mentioned in step 5. This process continues until layer 1.

[0060] S l ={F l (q)|q∈N(p,r),p∈P l-1}

[0061] Where F l (q) represents the feature in the , layer; q is the salient feature position in the neighborhood N(p, r), S l The set of effective salient features corresponding to the l-th layer that conform to the mapping relationship.

[0062] Step 5: In the defined S lWithin the local neighborhood, the multi-scale mixed feature group is fed into the transformer-structured encoder and decoder for analysis and prediction. First, a self-attention operation is performed on the salient mixed feature group; specifically, for a feature, there exists a query vector Q. i Pointing to itself, other feature vectors are referred to as key vectors relative to it. For each feature, there exists a value vector. Refers to itself; first, select a query vector Q corresponding to a feature. i The key vectors of the four layers within its feature group range Similarity is calculated and normalized using the softmax function to generate attention weights, which represent the correlation between two features. Finally, these weights are used to adjust the value vectors within the neighborhood. Perform weighted summation to generate the enhanced feature representation; where Q i , All are achieved by applying three different linear transformations to the original feature x, d k This is the dimension of the query and key vectors, used to prevent the dot product result from being too large;

[0063]

[0064] This process involves multiple operations running in parallel, known as multi-head attention, which can improve the efficiency of model prediction.

[0065] Since the Transformer processes all elements in a sequence in parallel, it inherently lacks the ability to perceive sequence order or spatial location. By incorporating position embedding (PE), the Transformer can effectively capture the sequential information or spatial structure of the input sequence, thereby improving its understanding of temporal or spatially related tasks (such as natural language processing and image processing). The specific equation is as follows:

[0066]

[0067]

[0068] `pos` represents the position of the feature in the input sequence. `i` represents the dimension index of the input feature vector. `d` model The dimension of each input vector in the model is represented by sine and cosine functions (e.g., 512 or 1024). These functions generate periodic variations at different frequencies, which change as the position pos increases, allowing the model to perceive the relative positions between inputs through different frequency ranges.

[0069] Step Six: After capturing global long-range connections through standard self-attention processing, deformable attention is then applied to enhance local focusing. For each query Q, deformable attention (DeformAttn) no longer calculates the similarity of all keys K as standard self-attention does. Instead, it selects sparse sampling points from the feature map for attention computation by predicting a set of offsets. The positions of these sampling points are relative to the query position and are learnable, allowing for adaptive adjustment during training.

[0070] Deformable attention features are calculated using the following formula:

[0071]

[0072] Among them, Q i Viewed as a query vector, it is generated from the original feature x through a linear transformation, where q is the corresponding index and p q Represents Q i The position of Δp is a two-dimensional vector, called the reference point. mqk It is an offset, representing the position from the query position Q. i The offset is a sampling position. The offset is dynamically learned by the model and represents the query Q. i Key feature points (or regions) that should be focused on. ΔA mqk It is the attention weight, which represents the query Q. i Correlation (similarity) with a specific sampling location. Weight ΔA mqk Through query Q i The attention calculation is similar to the weight calculation process in standard attention mechanisms. A larger weight value indicates higher importance of the sampling position. k is the index of the key vector, K is the total number of sampling keys, and m represents the m-th attention head. There are a total of M attention heads, calculated through W... m The matrix W′ further linearly transforms the feature representations of multiple heads and integrates the results from all heads. m This is used to unify the calculation of different local features within a single attention head, making it better integrated with the query vector, where W m and W′ m These are trainable parameters.

[0073] After the encoder passes through the image, the image features have undergone sufficient global and local information extraction through self-attention and deformable attention. Next, the decoder utilizes these encoded features, combined with the query vector, to predict the final category and bounding box. The specific process is as follows:

[0074] The input consists of feature maps output from the encoder, which contain information processed globally and locally. The learnable target query vectors are not features extracted directly from the image, but rather a set of vectors learned by the model. Each query represents a "clue" the model is trying to find for a target, such as the location or category of different objects.

[0075] The number of query vectors is fixed, typically 100, meaning the model can predict a maximum of 100 targets at a time. These query vectors interact with the feature maps output by the encoder to generate the final object detection result. Similar to the self-attention mechanism, our target query Q' undergoes cross-attention with the key vector K and value vector V output by the encoder to calculate their relevance. The Q corresponding to the K,V with the highest relevance is then predicted as the category of Q'.

[0076] Compared to the state-of-the-art salience DETR technology, this invention reduces the number of encoder parameters from 27M to 19M, a reduction of approximately 29.63%. Simultaneously, the number of floating-point operations (FLOPs) decreases from 0.208T to 0.176T, a reduction of approximately 15.38%. Furthermore, the inference speed is improved from 0.0614 seconds (16.27 FPS) per image to 0.0532 seconds (19 fps), an increase of approximately 13.36%. This significant performance improvement not only reduces the computational complexity of the model but also increases processing speed, making it more efficient and practical in real-world applications.

[0077] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] The above are merely preferred embodiments of the present invention and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should recognize that any equivalent substitutions and obvious changes made based on the content of this specification should be included within the protection scope of the present invention.

Claims

1. A vehicle detection method with multi-scale saliency hybrid enhancement, characterized in that, The steps include: Step 1: Acquire real scan image data from the road test camera; Step 2: Use a pre-convolutional neural network to obtain multi-scale feature images corresponding to the data; Step 3: Filter multi-scale saliency features by calculating saliency scores; Step 4: Through the multi-scale hybrid proposal module, effective features are selected from the highest layer (i.e., the low-resolution layer) and mapped to the low-level feature images to obtain a multi-scale hybrid feature group. The specific implementation of the multi-scale hybrid proposal module in step 4 is as follows: In step 3, obtain the saliency scores corresponding to all features, and then use a dynamic threshold to confirm the effective features, that is, adjust the threshold of the saliency score so that the effective region occupies a specific percentage of the entire map area; next, starting from the effective features of the highest layer, construct a feature group layer by layer downward, define a circular neighborhood N(p,r) with radius r, and select saliency features within this circular neighborhood; N(p,r)={q∣||qp||≤r} Here, p is the position mapped from the previous layer, i.e., the center of the circular neighborhood corresponding to the current layer, and q is the position of the salient feature in the current layer. To ensure that high-level information is accurately transmitted to low-level layers, a circular convolution is used to simulate the construction process of this circular neighborhood. r corresponds to the size of a circular convolution kernel, which upsamples and maps the position encoding of the salient feature in layer l to layer (l-1), and its position in layer (l-1) is denoted as P. l-1 In P l-1 If a salient feature of layer (l-1) exists within the circular neighborhood N(p,r), it is considered valid and selected for encoding by the encoder mentioned in step 5. This process continues until layer 1. S l ={F l (q)∣q∈N(p,r),p∈P l-1 } Where F l (q) represents the feature in the l-th layer; q is the salient feature position in the neighborhood N(p,r), S l The set of effective salient features corresponding to the l-th layer that conform to the mapping relationship; Step 5: Feed the multi-scale mixed feature group into the encoder based on the transformer structure to obtain the enhanced multi-scale mixed feature group, and use the decoder to predict and output the category and probability of the detected target. The encoder is implemented in step 5 as follows: Within the local neighborhood of the multi-scale mixed feature group, the multi-scale mixed feature group is fed into an encoder based on a transformer structure for encoding: First, a self-attention operation is performed on the salient mixed feature group. After capturing global long-distance connections through ordinary self-attention processing, it is then put into deformable attention to enhance local focusing processing, thereby obtaining enhanced multi-scale mixed features.

2. The vehicle detection method with multi-scale saliency hybrid enhancement as described in claim 1, characterized in that: In step 2, the input image data is processed by a pre-convolutional neural network ResNet to obtain a series of feature maps F. l Each feature map Representing different scale levels of resolution, where C is the number of channels in each feature map, and H... l and W l These are the height and width of the feature map at layer l, respectively.

3. The vehicle detection method with multi-scale saliency hybrid enhancement as described in claim 1, characterized in that: The specific implementation method of step 3 is as follows: Scalar-independent saliency scores are used as the supervision target to extract salient foreground features. Each feature at position (i,j) in the l-th feature map is... The saliency score of the feature corresponding to the coordinates c = (x, y) in the original image. Determined according to the following rules: in The text represents the true bounding box, where w and h represent the width and height of the true bounding box, respectively. The saliency score is calculated based on the relative distance to the center of the true bounding box. Where Δx and Δy represent the distance between c = (x, y) and the center of the corresponding ground truth box.

4. The vehicle detection method with multi-scale saliency hybrid enhancement as described in claim 1, characterized in that: The specific implementation of self-attention operations is as follows: For a given feature, there exists a query vector Q. i Pointing to itself, other feature vectors are referred to as key vectors relative to it. For each feature, there exists a value vector. Refers to itself; first, select a query vector Q corresponding to a feature. i The key vector within its multi-scale mixed feature group range Similarity is calculated and normalized using the softmax function to generate attention weights, which represent the correlation between two features. Finally, these weights are used to adjust the value vectors within the neighborhood. Perform weighted summation to generate the enhanced feature representation; where Q i , This is achieved by applying three different linear transformations to the original feature x; d k It represents the dimension of the query and key vectors, used to prevent the dot product result from becoming too large.

5. The vehicle detection method with multi-scale saliency hybrid enhancement as described in claim 4, characterized in that: To represent the order relationship, a dimension of d is added to the original features. model The location encoding is specifically expressed as follows: Where PE represents the encoding position, pos represents the position of the feature in the input sequence, i represents the dimension index of the feature vector of the input sequence, and d model This represents the dimension of each input vector.

6. The vehicle detection method with multi-scale saliency hybrid enhancement as described in claim 4, characterized in that: The specific implementation of deformable attention is as follows: Given an input feature map, the formula for calculating deformable attention is: Among them, Q i Viewed as a query vector, it is generated from the original feature x through a linear transformation, where q is the corresponding index and p q Represents Q i The position of Δp is a two-dimensional vector, called the reference point. mqk It is an offset, representing the position from the query position Q. i A certain sampling position at the offset; ΔA mqk It is the attention weight, which represents the query Q. i Correlation with a specific sampling location; weight ΔA mqk Through query Q i The attention is calculated; k is the index of the key vector, K is the total number of sampled keys, and m represents the m-th attention head. There are a total of M attention heads, calculated through W. m The matrix further linearly transforms the feature representations of multiple heads and integrates the results of all heads, W′. m This is used to unify the calculation of different local features within a single attention head, making it better integrated with the query vector.

7. A vehicle detection system with multi-scale saliency hybrid enhancement, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement a multi-scale saliency hybrid enhancement vehicle detection method as described in any one of claims 1 to 6.

Images