Multi-scale saliency hybrid enhanced vehicle detection method and system

Abstract

The invention provides a multi-scale saliency hybrid enhanced vehicle detection method and system, which is used for improving an existing DETR (DEection TRannform) target detection framework, comprises a saliency feature screening mechanism and a multi-scale hybrid suggestion module, and aims to construct an efficient vehicle detection model. Compared with a traditional road measurement algorithm, the module has the advantages that the detection precision is remarkably improved under complex scenes (such as smear and dense vehicle conditions), and features are effectively purified. The key innovation of the method is that a multi-scale mixed suggestion module of images with different resolutions is combined, the problem of feature attenuation of advanced prior information in a traditional method is solved, and the feature extraction precision is improved by fully utilizing the interlayer relationship. According to the method, the semantic information of the low-resolution image and the detail information of the high-resolution image are fused, the model parameter quantity is reduced, the computing resource demand is reduced, and the overall detection precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of target detection algorithms in computer vision, and in particular to a vehicle detection method and system with multi-scale saliency hybrid enhancement. Background Art

[0002] Object detection based on deep learning is generally divided into two categories according to the detection method: two-stage detection and single-stage detection. The former first generates a prediction box, which may contain the object to be detected, and then analyzes and fine-tunes the prediction box to obtain an ideal detection result. Its representative algorithms include regional convolutional neural network R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN, etc. The latter directly extracts image features in the network, and directly classifies and regresses the potential targets in the preset anchor box on the feature map, thereby detecting all targets. Representative algorithms include SSD, YOLO series, etc.

[0003] Convolutional Neural Network (CNN) is the mainstream deep neural network used in deep learning tasks such as semantic segmentation and 3D object detection. CNN can effectively extract feature tensors of input data. However, at each convolution step, CNN can only capture the feature tensor of a local patch. In order to expand the receptive field of deep neural networks, CNN requires several rounds of convolution operations. However, the hierarchical convolution operation still affects the extraction of global scale feature tensors. Unlike CNN, transformer uses self-attention module for feature tensor extraction. Transformer first divides the input data into a series of smaller tensors, and then runs several rounds of self-attention modules to extract the features of the input. In this way, transformer has a much larger receptive field and can better capture the spatial information of the input data. In addition, transformer can better express the relationship between different patches of input data in a formula.

[0004] In the field of object detection, effectively representing and processing objects of different scales has always been one of the core challenges. Traditional CNN-based detectors widely adopt multi-scale feature extraction techniques, such as FPN, BiFPN, PANet, and NAS-FPN, which have been proven to significantly improve the efficiency and effectiveness of the models. In contrast, the efficiency problem of the DETR model (end-to-end object detection model based on Transformer) in processing multi-scale features has not been fully addressed. In view of the limitations of existing technologies, a DETR-based urban traffic vehicle detection module is needed, which can effectively use multi-scale features in Transformer-based object detectors. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention proposes a vehicle detection method with multi-scale saliency hybrid enhancement, which can effectively use multi-scale features in a Transformer-based object detector, and comprises the following steps:

[0006] Step 1, obtaining real scan image data of the road test camera;

[0007] Step 2, using the pre-convolutional neural network to obtain the multi-scale feature images corresponding to the data;

[0008] Step 3: Screen multi-scale significant features by calculating the significance score;

[0009] Step 4: Through the multi-scale hybrid proposal module, effective features are screened from the highest layer, i.e., the low-resolution layer, and mapped to the low-layer feature image to obtain a multi-scale hybrid feature group;

[0010] In step 5, the multi-scale mixed feature group is sent to the encoder based on the transformer structure to obtain the enhanced multi-scale mixed feature group, and the decoder is used to predict and output the category and probability of the detected target.

[0011] Furthermore, in step 2, the input image data is processed by the convolutional neural network ResNet to obtain a series of feature maps F l , each feature map Represents the scale levels of different resolutions, where C is the number of channels in each feature map, and H l and W l are the height and width of the feature map of the lth layer respectively.

[0012] Furthermore, the specific implementation of step 3 is as follows:

[0013] Use the scale-independent saliency score as the supervision target to extract salient foreground features. Each feature at position (i, j) in the feature map Corresponding to the coordinate c = (x, y) in the original image, the saliency score of the feature Determined according to the following rules:

[0014]

[0015] in represents the true box, w and h correspond to the width and height of the true box respectively, and the significance score is calculated by the relative distance to the center of the true box:

[0016]

[0017] where Δx and Δy represent the distance between c=(x, y) and the center of the corresponding ground-truth box.

[0018] Furthermore, the specific implementation of the multi-scale hybrid proposal module in step 4 is as follows:

[0019] In step 3, the significance scores corresponding to all features are obtained, and then the effective features are confirmed by dynamic threshold, that is, the threshold of the significance score is adjusted so that the effective area occupies a specific percentage of the entire image area; next, starting from the effective features of the highest layer, a feature group is constructed layer by layer, and a circular neighborhood N(p, r) with a radius of r is defined, and significant features are selected within the circular neighborhood;

[0020] N(p, r)={q|||qp||≤r}

[0021] Among them, p is the position obtained by mapping in the previous layer, that is, the center of the circle corresponding to the circular neighborhood of the current layer, and q is the position of the significant feature of the current layer. In order to ensure that high-level information is accurately transmitted to lower layers, circular convolution is used to simulate the construction process of this circular neighborhood. r corresponds to the size of a circular convolution kernel, and the position encoding of the significant feature of the ,th layer is upsampled and mapped to the ,-1th layer. Its position in the ,-1th layer is recorded as P l-1 , in P l-1 If there is a significant feature of the l-1th layer in the circular neighborhood N(p, r), it is recorded as valid and selected to enter the encoder encoding mentioned in step 5, and so on until the first layer:

[0022] S l ={F l (q)|q∈N(p,r),p∈P l-1}

[0023] where F l (q) represents the feature in the,th layer; q is the significant feature position in the neighborhood N(p, r), S l Corresponding to the,th layer, the set of effective salient features that conform to the mapping relationship.

[0024] Furthermore, the specific implementation of the encoder in step 5 is as follows:

[0025] In the local neighborhood of the multi-scale mixed feature group, the multi-scale mixed feature group is sent to the encoder based on the transformer structure for encoding: first, a self-attention operation is performed on the salient mixed feature group. After ordinary self-attention processing to capture the global long-distance connection, it is placed in the deformable attention to strengthen the local focus processing, thereby obtaining enhanced multi-scale mixed features.

[0026] Furthermore, the specific implementation of the self-attention operation is as follows:

[0027] For a feature, there is a query vector Q i points to itself, and other eigenvectors are called key vectors relative to it For all features, there is a value vector Refers to itself; first select a query vector Q corresponding to a feature i , and its key vector in the range of multi-scale hybrid feature groups Calculate the similarity and use the softmax function to normalize it to generate attention weights, which refer to the correlation between the current two features. Finally, use these weights to adjust the value vector in the neighborhood Perform weighted summation to generate enhanced feature representation;

[0028]

[0029] Among them, Qi This is achieved by applying three different linear transformations to the original feature x; d k is the dimension of the query and key vectors, used to prevent the dot product result from being too large.

[0030] Furthermore, in order to express the sequential relationship, the dimension d is added to the original feature embedding. model The position encoding is as follows:

[0031]

[0032]

[0033] Among them, PE represents the encoding position, pos represents the position of the feature in the input sequence, i represents the dimension index of the input sequence feature vector, and d model Represents the dimension of each input vector.

[0034] Furthermore, the specific implementation of deformable attention is as follows:

[0035] Given an input feature map, the deformable attention is calculated as:

[0036]

[0037] Among them, Q i As a query vector, it is generated by linear transformation of the original feature x, q is the corresponding index, and p q Representative Q i The position of is a two-dimensional vector, called the reference point, Δp mqk is an offset from the query position Q i A sampling position shifted; ΔA mqk is the attention weight, which represents the query Q i Correlation with a specific sampling location; weight ΔA mqk is obtained by querying Q i The attention is calculated by ; k is the index of the key vector, K is the total number of sampled keys, m represents the mth attention head, there are M attention heads in total, through W m Matrix, further linearly transforms the feature representations of multiple heads and integrates the results of all heads, W′ m It is used to unify the calculation of different local features within a single attention head so that it can be better combined with the query vector.

[0038] The present invention also provides a vehicle detection system with multi-scale saliency hybrid enhancement, comprising:

[0039] one or more processors;

[0040] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement a vehicle detection method with multi-scale saliency hybrid enhancement as described in the above scheme.

[0041] Compared with the prior art, the beneficial effects of the present invention are as follows: in the multi-scale hybrid feature tunnel designed by the present invention, the structure is simple and efficient. A significant advantage of the network is that only one encoder layer is required to achieve the effect of using 6 layers of encoders in the traditional baseline model. This design achieves refined processing of features by efficiently utilizing cross-layer features, significantly improving the detection performance and efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of an embodiment of the present invention;

[0043] Figure 2 It is a flowchart of the overall model processing of the present invention;

[0044] Figure 3This is a processing flow chart of a multi-scale hybrid suggestion module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present invention will be described clearly and completely below in combination with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0047] The present invention will be further described below with reference to specific embodiments, but they are not intended to be limiting of the present invention.

[0048] like Figure 1 As shown, the embodiment of the present invention discloses a vehicle detection method with multi-scale saliency hybrid enhancement, comprising the following steps:

[0049] Step 1: By integrating ResNet as the feature extraction network, introducing feature tunneling, and combining the dual attention mechanism, a multi-scale hybrid proposal module for urban traffic vehicle detection based on DETR is constructed.

[0050] Step 2: Obtain a training image dataset in traffic, perform image annotation processing on it, and randomly select certain images and label data into the detection model that integrates the multi-scale hybrid proposal module.

[0051] Step 3: The input image is subjected to feature extraction by ResNet to obtain a series of feature maps F l , each feature map Represents the scale levels of different resolutions, and a total of n feature maps (l∈[1,n]) are obtained. In this example, n=4, C is the number of channels of each feature map, and H l and W l are the height and width of the feature map of the first layer, respectively. Generally, the highest layer image is made to have the lowest resolution to provide semantic information, and the resolution is doubled for each layer below, that is, one pixel in the upper layer corresponds to four pixels in the next layer, and so on, to obtain four feature images that follow the feature pyramid structure.

[0052] Step 4: Use the scale-independent saliency score as the supervision target to extract salient foreground features, not just labeling the foreground as 1 and the background as 0. Specifically, each feature at position (i, j) in the lth feature map Corresponding to the coordinate c = (x, y) in the original image, the saliency score of the feature Determined according to the following rules:

[0053]

[0054] in represents the true box, w and h correspond to the width and height of the true box respectively, and the significance score is calculated by the relative distance to the center of the true box:

[0055]

[0056] where Δx and Δy represent the distance between c=(x, y) and the center of the corresponding ground-truth box.

[0057] Step 4: Obtain the significance scores corresponding to all features in step 3, and then confirm the effective features through dynamic thresholds, that is, adjust the threshold of the significance score so that the effective area occupies a specific percentage of the entire image area; next, starting from the effective features of the highest layer, build a feature group layer by layer, define a circular neighborhood N(p, r) with a radius of r, and select significant features in the circular neighborhood;

[0058] N(p, r)={q|||qp||≤r}

[0059] Among them, p is the position obtained by mapping in the previous layer, that is, the center of the circle corresponding to the circular neighborhood of the current layer, and q is the position of the significant feature of the current layer. In order to ensure that high-level information is accurately transmitted to lower layers, circular convolution is used to simulate the construction process of this circular neighborhood. r corresponds to the size of a circular convolution kernel, and the position encoding of the significant feature of the ,th layer is upsampled and mapped to the ,-1th layer. Its position in the ,-1th layer is recorded as P l-1 , in P l-1 If there is a significant feature of the l-1th layer in the circular neighborhood N(p, r), it is recorded as valid and selected to enter the encoder encoding mentioned in step 5, and so on until the first layer:

[0060] S l ={F l (q)|q∈N(p,r),p∈P l-1}

[0061] Among them, F l (q) represents the feature in the ,th layer; q is the significant feature position in the neighborhood N(p, r), S l The corresponding effective salient feature set that conforms to the mapping relationship at the lth layer.

[0062] Step 5: Define S lIn the local neighborhood, the multi-scale mixed feature group is sent to the encoder-decoder of the transformer structure for analysis and prediction. First, a self-attention operation is performed on the significant mixed feature group. Specifically, for a feature, there is a query vector Q i points to itself, and other eigenvectors are called key vectors relative to it For all features, there is a value vector Refers to itself; first select a query vector Q corresponding to a feature i , and the key vectors of the four layers in its feature group range Calculate the similarity and use the softmax function to normalize it to generate attention weights, which refer to the correlation between the current two features. Finally, use these weights to adjust the value vector in the neighborhood Perform weighted summation to generate enhanced feature representation; where Q i , All of them are realized by three different linear transformations of the original feature x, d k is the dimension of the query and key vectors, used to prevent the dot product result from being too large;

[0063]

[0064] This process performs multiple operations in parallel, called multi-head attention, which can improve the efficiency of model prediction.

[0065] Since Transformer processes all elements in a sequence in parallel, it does not have the ability to perceive sequence order or spatial position. By adding position encoding (PE), Transformer can effectively capture the sequential information or spatial structure in the input sequence, thereby improving the ability to understand time-series or space-related tasks (such as natural language processing, image processing, etc.). The specific equation is as follows:

[0066]

[0067]

[0068] pos represents the position of the feature in the input sequence. i represents the dimension index of the input feature vector. model Representing the dimension of each input vector in the model (such as 512 or 1024, etc.), the sine and cosine functions generate periodic changes of different frequencies, which change as the position pos increases, allowing the model to perceive the relative positions between inputs through different frequency ranges.

[0069] Step 6: After capturing the global long-distance connection through ordinary self-attention processing, deformable attention is added to strengthen the local focus processing. For each query Q, deformable attention (DeformAttn) no longer calculates the similarity of all keys K like standard self-attention, but selects some sparse sampling points from the feature map for attention calculation by predicting a set of offsets. The positions of these sampling points are relative to the query position and are learnable and can be adaptively adjusted during training.

[0070] The deformable attention feature is calculated by the following formula:

[0071]

[0072] Among them, Q i As a query vector, it is generated by linear transformation of the original feature x, q is the corresponding index, and p q Representative Q i The position of is a two-dimensional vector, called the reference point, Δp mqk is an offset from the query position Q i The offset is learned dynamically by the model and represents the query Q i Feature points (or areas) that should be focused on. ΔA mqk is the attention weight, which represents the query Q i The correlation (similarity) with a specific sampling location. Weight ΔA mqk is obtained by querying Q i The weight calculation is similar to the weight calculation process in the standard attention mechanism. The larger the weight value, the more important the sampling position is. k is the index of the key vector, K is the total number of sampling keys, and m represents the mth attention head. There are M attention heads in total. m Matrix, further linearly transforms the feature representations of multiple heads and integrates the results of all heads, matrix W′ m It is used to unify the calculation of different local features in a single attention head so that it can be better combined with the query vector, where W m and W′ m are trainable parameters.

[0073] After passing through the encoder, the features of the image have been fully extracted through self-attention and deformable attention. Next, the decoder will use these encoded features and combine them with the query vector to complete the final category and bounding box prediction. The specific process is as follows:

[0074] Input feature maps output from the encoder, which contain information after global and local processing. Learnable target query vectors, which are not features extracted directly from the image, but a set of vectors learned by the model. Each query represents the model's attempt to find a "clue" for a target, such as the location, category, etc. of different objects.

[0075] The number of query vectors is fixed, usually 100, which means that the model can predict up to 100 targets at a time. These query vectors interact with the feature map output by the encoder to generate the final target detection result. Similar to the self-attention mechanism, our target query Q', cross-attention is performed with the key vector K and value vector V output by the encoder to calculate the correlation. The Q corresponding to the K and V with the highest correlation will be predicted as the category of Q'.

[0076] Compared with the most advanced technology, salience DETR, the encoder parameters of this invention are reduced from 27M to 19M, a reduction of about 29.63%. At the same time, the number of floating-point operations (FLOPs) is reduced from 0.208T to 0.176T, a reduction of about 15.38%. In addition, the inference speed is also increased from 0.0614 seconds per photo (16.27FPS) to 0.0532 seconds (19fps), an increase of about 13.36%. This significant performance improvement not only reduces the computational complexity of the model, but also increases the processing speed, making it more efficient and practical in practical applications.

[0077] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD ROM, optical storage, etc.) containing computer-usable program code.

[0078] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the contents of the specification of the present invention should be included in the protection scope of the present invention.

Claims

1. A vehicle detection method with multi-scale saliency hybrid enhancement, characterized in that: The steps include: Step 1, obtaining real scan image data of the road test camera; Step 2, using the pre-convolutional neural network to obtain the multi-scale feature images corresponding to the data; Step 3: Screen multi-scale significant features by calculating the significance score; Step 4: Through the multi-scale hybrid proposal module, effective features are screened from the highest layer, i.e., the low-resolution layer, and mapped to the low-layer feature image to obtain a multi-scale hybrid feature group; In step 5, the multi-scale mixed feature group is sent to the encoder based on the transformer structure to obtain the enhanced multi-scale mixed feature group, and the decoder is used to predict and output the category and probability of the detected target.

2. The vehicle detection method with multi-scale saliency hybrid enhancement as claimed in claim 1, characterized in that: In step 2, the input image data is processed by the convolutional neural network ResNet to obtain a series of feature maps F l , each feature map Represents the scale levels of different resolutions, where C is the number of channels in each feature map, and H l and W l are the height and width of the feature map of the lth layer respectively.

3. The vehicle detection method with multi-scale saliency hybrid enhancement as claimed in claim 1, characterized in that: The specific implementation of step 3 is as follows: Using the scale-independent saliency score as the supervision target to extract salient foreground features, each feature at position (i, j) in the lth feature map Corresponding to the coordinates c = (x, y) in the original image, the saliency score of the feature Determined according to the following rules: in represents the true box, w and h correspond to the width and height of the true box respectively, and the significance score is calculated by the relative distance to the center of the true box: where Δx and Δy represent the distance between c = (x, y) and the center of the corresponding ground-truth box.

4. The vehicle detection method with multi-scale saliency hybrid enhancement as claimed in claim 1, characterized in that: The specific implementation of the multi-scale hybrid proposal module in step 4 is as follows: In step 3, the significance scores corresponding to all features are obtained, and then the effective features are confirmed by dynamic threshold, that is, the threshold of the significance score is adjusted so that the effective area occupies a specific percentage of the entire image area; next, starting from the effective features of the highest layer, a feature group is constructed layer by layer, and a circular neighborhood N(p,r) with a radius of r is defined, and significant features are selected within the circular neighborhood; N(p,r)={q||qp|≤r} Among them, p is the position obtained by mapping in the previous layer, that is, the center of the circle corresponding to the circular neighborhood of the current layer, and q is the position of the salient feature in the current layer. In order to ensure that high-level information is accurately transmitted to lower layers, circular convolution is used to simulate the construction process of this circular neighborhood. r corresponds to the size of a circular convolution kernel, and the position encoding of the salient feature of the lth layer is upsampled and mapped to the l-1th layer. Its position in the l-1th layer is recorded as P l-1 , in P l-1 If there is a significant feature of the l-1th layer in the circular neighborhood N(p,r), it is recorded as valid and selected to enter the encoder encoding mentioned in step 5, and so on until the first layer: S l ={F l (q)∣q∈N(p,r),p∈P l-1 } where F l (q) represents the feature in the lth layer; q is the significant feature position in the neighborhood N(p,r), S l The corresponding effective salient feature set that conforms to the mapping relationship at the lth layer.

5. The vehicle detection method with multi-scale saliency hybrid enhancement as claimed in claim 1, characterized in that: The specific implementation of the encoder in step 5 is as follows: In the local neighborhood of the multi-scale mixed feature group, the multi-scale mixed feature group is sent to the encoder based on the transformer structure for encoding: first, a self-attention operation is performed on the salient mixed feature group. After ordinary self-attention processing to capture the global long-distance connection, it is placed in the deformable attention to strengthen the local focus processing, thereby obtaining enhanced multi-scale mixed features.

6. The vehicle detection method with multi-scale saliency hybrid enhancement as claimed in claim 5, characterized in that: The specific implementation of the self-attention operation is as follows: For a feature, there is a query vector Q i points to itself, and other eigenvectors are called key vectors relative to it For all features, there is a value vector Refers to itself; first select a query vector Q corresponding to a feature i , and its key vector in the range of multi-scale hybrid feature groups Calculate the similarity and use the softmax function to normalize it to generate attention weights, which refer to the correlation between the current two features. Finally, use these weights to adjust the value vector in the neighborhood Perform weighted summation to generate enhanced feature representation; Where Q i , This is achieved by applying three different linear transformations to the original feature x; d k is the dimension of the query and key vectors, used to prevent the dot product result from being too large.

7. The vehicle detection method with multi-scale saliency hybrid enhancement as claimed in claim 6, characterized in that: In order to express the sequential relationship, the embedding adds a dimension d to the original feature. model The position encoding is as follows: Among them, PE represents the encoding position, pos represents the position of the feature in the input sequence, i represents the dimension index of the input sequence feature vector, and d model Represents the dimension of each input vector.

8. The vehicle detection method with multi-scale saliency hybrid enhancement as claimed in claim 1, characterized in that: The specific implementation of deformable attention is as follows: Given an input feature map, the deformable attention is calculated as: Among them, Q i As a query vector, it is generated by linear transformation of the original feature x, q is the corresponding index, and p q Representative Q i The position of is a two-dimensional vector, called the reference point, Δp mqk is an offset from the query position Q i A sampling position shifted; ΔA mqk is the attention weight, which represents the query Q i Correlation with a specific sampling location; weight ΔA mqk is obtained by querying Q i The attention is calculated by ; k is the index of the key vector, K is the total number of sampled keys, m represents the mth attention head, there are M attention heads in total, through W m Matrix, further linearly transforms the feature representations of multiple heads and integrates the results of all heads, W′ m It is used to unify the calculation of different local features within a single attention head so that it can be better combined with the query vector.

9. A vehicle detection system with multi-scale saliency hybrid enhancement, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement a vehicle detection method with multi-scale saliency hybrid enhancement as described in any one of claims 1 to 8.

Images