Remote sensing vehicle detection method based on multi-model fusion

Abstract

The invention discloses a remote sensing vehicle detection method based on multi-model fusion, and the method comprises the steps: carrying out the preprocessing of obtained vehicle remote sensing image data, and obtaining a data set; the method comprises the following steps: by taking an original YOLOv12 model as a benchmark, introducing a DSCWHConv convolution module into a backbone network to replace standard convolution in original C3k2 and A2C2f modules; in the neck network, an Area AgentAttention attention mechanism module is adopted to replace an original attention mechanism module; and inputting pictures in the data set into the improved YOLOv12 model for training to obtain a trained model. In the feature extraction stage, a multi-frequency information fusion lightweight convolution module is introduced, traditional convolution is replaced, multi-scale features are extracted and a receptive field is expanded, in the feature fusion stage, a context-aware attention mechanism is designed, and combined modeling of local and global semantic information is realized in combination with an agent token and a region aggregation strategy. The detection precision is remarkably improved, 70.1% mAP50 and 45.5% mAP50-95 are obtained, and the effectiveness and practicability of the method in a remote sensing vehicle detection task are verified.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image target detection, and in particular to a remote sensing vehicle target detection method based on multi-model fusion. Background Art

[0002] With the continuous advancement of remote sensing technology and imaging sensors, platforms such as satellites, aircraft, and drones are able to acquire high-spatial-resolution, multispectral, and even hyperspectral remote sensing images. These images are widely used in traffic monitoring, smart city planning, disaster relief, and national defense reconnaissance. Vehicle detection, a key task in remote sensing image analysis, is crucial for traffic flow statistics and road congestion prediction.

[0003] Early vehicle detection in remote sensing images relied primarily on manually designed target features and sliding window classifier frameworks, such as support vector machine (SVM) / Adaboost detection methods based on HOG, LBP, or Haar features. These methods were less robust to complex and changing backgrounds, changes in vehicle appearance, and variations in lighting, and were heavily reliant on feature design and parameter tuning.

[0004] In recent years, deep convolutional neural networks have rapidly developed in the field of object detection. Two-stage detectors identify and localize objects in images by generating candidate boxes, classifying them, and fine-tuning their bounding boxes. Single-stage detectors, on the other hand, balance detection speed and accuracy in an end-to-end manner, becoming the mainstream approach for practical deployment. However, when applied to remote sensing imagery, these methods often face computational resource bottlenecks, feature loss of small objects, and complex background interference.

[0005] In response to the characteristics of vehicle targets in remote sensing images - extremely small size relative to the image size, common dense distribution and diverse orientations, existing studies have attempted to improve detection performance through multi-scale feature pyramids, rotated box regression or contextual attention mechanisms, but there are still deficiencies in model lightweighting and large receptive field expansion methods.

[0006] To address these issues, this paper proposes a remote sensing vehicle detection method based on multi-model fusion. This method utilizes the proprietary DSCWHConv convolution in the backbone network to balance multi-frequency feature fusion with computational efficiency. It also incorporates the AreaAgentAttention mechanism in the neck network, achieving a dual-stage attention model: first aggregating within a region and then broadcasting globally. This method effectively improves vehicle detection accuracy in complex backgrounds and high-resolution scenarios. Summary of the Invention

[0007] To address the problems of low small target detection accuracy, strong background interference, and insufficient multi-scale perception in the existing technology, the present invention proposes a remote sensing vehicle detection method based on multi-model fusion. This method can achieve a logarithmically growing multi-frequency large receptive field while ensuring lightweight, thereby improving the detection accuracy of small target information in complex backgrounds.

[0008] The technical solution adopted by the present invention is: a remote sensing vehicle detection algorithm based on multi-model fusion, comprising the following steps:

[0009] Step 1: Obtain vehicle remote sensing image data, preprocess the dataset image, and obtain the dataset;

[0010] Step 2: Using the original YOLOv12 model as a benchmark, introduce the DSCWHConv convolution module into the backbone network to replace the standard convolution in the original C3k2 and A2C2f modules;

[0011] Step 3: In the neck network, the AreaAgentAttention attention mechanism module is used to replace the original attention mechanism module;

[0012] Step 4: Input the images in the dataset into the improved YOLOv12 model for training to obtain a trained model.

[0013] Furthermore, in step 2, first, four filters used in the two-dimensional Haar wavelet transform are introduced, and then the input feature map F is first subjected to the L-level Haar wavelet transform (WT) to obtain the low-frequency component of the i-th level. And three groups of high frequency components The size of the decomposition space is halved each time, but the number of channels remains unchanged.

[0014] Among them, the four filter formulas used in the two-dimensional Haar wavelet transform are:

[0015]

[0016] Among them, f LL is a low-pass filter, f LH ,f HL ,f HH Horizontal, vertical and diagonal high-pass filters respectively.

[0017] Furthermore, in step 2, the low-frequency and high-frequency components of each level are calculated. The formula is as follows:

[0018] [F LL ,F LH ,F HL ,F HH ]=↓Conv([f LL,f LH ,f HL ,f HH ],F)

[0019] Among them, ↓ represents downsampling.

[0020] Furthermore, in step 2, deep convolution is performed on the low-frequency and high-frequency components of the i-th level, and then the low-frequency and high-frequency components are scaled to avoid gradient explosion, where the weight is recorded as W (i) The formula is:

[0021]

[0022] Among them, scale(·) represents scaling and dwconv(·) represents depthwise convolution.

[0023] Furthermore, in step 2, the inverse wavelet transform (IWT) is finally used to linearly reconstruct the convolution results of all levels and all frequency bands to obtain the fused feature map, which is first up-sampled and then transposed convolution is performed. The formula is as follows:

[0024] [F LL ,F LH ,F HL ,F HH ]=ConvT(up(F),f -1 )

[0025] Among them, ConvT stands for transposed convolution, up stands for upsampling, and f -1 Filter representing the inverse wavelet transform.

[0026] Furthermore, in step 3, the feature map X is divided into M non-overlapping regions in the horizontal or vertical direction. For each region i of the feature map X, a linear mapping of query, key, and value is performed:

[0027] Q=X i W q +b q ,[K,V]=X i W kv +b kv

[0028] Where W q , W kv is the weight matrix, b q and b kv is the bias vector

[0029] Furthermore, in step 3, for each region Q i Generate regional proxy tokens using adaptive average pooling:

[0030] A i=Pool(Q i )

[0031] Among them A i As Q i Proxy tokens.

[0032] Furthermore, in step 3, for each region i, A i For query, K i , V i Calculate region-level aggregate features for keys and values:

[0033]

[0034] All regional features are concatenated to obtain the global proxy feature:

[0035]

[0036] Among them A i It's Q i Proxy tokens, K i , V i is the original key value of region i, and σ(·) represents the Softmax function.

[0037] Furthermore, in step 3, the proxy feature V after the above formula aggregation is A Then perform a second attention broadcast with the original query Q to obtain the output O A :

[0038]

[0039] Where σ(·) represents the Softmax function, Q is the original query (i.e., the query vector after input feature map mapping), O A is the proxy feature V after aggregation A The output obtained in .

[0040] Furthermore, in step 3, finally, the broadcast result is added to the bias and dwc results to perform residual addition to obtain the final output O.

[0041] O=σ(QA T +B2)σ(AK T +B1)V+dwc(V)

[0042] Among them, B1 and B2 are proxy biases, each of which is composed of three bias components; dwc is depth convolution, which uses depth convolution to maintain feature diversity.

[0043] Furthermore, in step 4, the traditional IoU / GIoU loss function is replaced by NWD to enhance the accurate detection of small objects.

[0044] Replace the bounding box loss function with the NWD loss function. The existing IoU loss function is often unable to adapt to different target tasks when facing small remote sensing targets with drastic scale changes and extremely small overlapping areas. It cannot provide effective gradients for non-overlapping or slightly overlapping boxes and has insufficient generalization ability. To this end, we introduce the normalized Wasserstein distance as a new similarity metric and convert it into a regression loss to reflect slight position deviations more smoothly and robustly. NWD-Loss first models the predicted box and the true box as two-dimensional Gaussian distributions, then calculates the Wasserstein distance between them and exponentially normalizes them to obtain a similarity index between 0 and 1. Finally, the difference between it and 1 is taken as the regression loss, which can still maintain gradient information when the overlap disappears, accelerate convergence and improve positioning accuracy. The calculation formula of NWD-Loss is as follows:

[0045]

[0046] where u p ,∑ p with u g ,∑ g are the Gaussian distribution parameters corresponding to the predicted box and the true box, respectively, and C is the normalization constant set according to the average box scale of the dataset.

[0047] The beneficial effects of the present invention are:

[0048] 1. Using the DSCWHConv convolution module in the backbone network enables multi-band feature extraction and large receptive field coverage without significantly increasing the number of parameters and computational overhead. This convolution uses the wavelet transform to decompose the input feature map into different frequency bands, enhancing the model's response to details and edge information, thereby improving the semantic expression ability of small vehicle targets in remote sensing images. The depthwise separable convolution significantly reduces the amount of convolution calculations and video memory usage, allowing the model to maintain efficient inference when processing high-resolution images. In addition, the inherent denoising properties of the wavelet transform further enhance feature robustness, enabling the network to stably detect small targets even in complex backgrounds and noise interference.

[0049] 2. Compared to traditional attention mechanisms, AreaAgentAttention divides the feature map into several spatial regions and generates agent tokens within each region, implementing a two-stage attention mechanism: local aggregation followed by global broadcasting. The first stage aggregates local information within the region, avoiding the direct calculation of Q–K similarity across the entire image. The second stage distributes the aggregated global semantics back to each pixel through broadcast attention, effectively integrating global and local context. This design not only significantly reduces computational and memory overhead while maintaining a large receptive field, but also provides finer-grained semantic features and more precise spatial positioning for small vehicle detection in remote sensing images, significantly improving detection accuracy and inference efficiency.

[0050] 3. Replace the traditional IoU / GIoU loss function with NWD to enhance the accurate detection of small objects BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is the detection flow chart of the present invention;

[0052] Figure 2 Improved YOLOv12 network model architecture diagram for the present invention;

[0053] Figure 3 The DSCWHConv structure diagram is customized for the present invention;

[0054] Figure 4 Improved AreaAgentAttention structure diagram for this invention;

[0055] Figure 5 This is a diagram of the experimental results of the present invention. DETAILED DESCRIPTION

[0056] The present invention is described clearly and completely below with reference to the accompanying drawings and specific embodiments. The following embodiments are intended only to illustrate the technical concepts and features of the present invention and are not intended to limit the scope of protection of the present invention. Any equivalent or modified versions based on the spirit of the present invention shall be included within the scope of protection of the present invention.

[0057] like Figure 1-5 As shown, the present invention discloses a remote sensing image vehicle detection method based on multi-model fusion, as follows:

[0058] Using the DOTA dataset, this paper describes in detail the implementation of a multi-model fusion-based vehicle detection method for remote sensing images. This method includes a complete flow of data processing, feature extraction, feature fusion, and prediction output. The technical solution strictly adheres to claims 1-6 of the patent and is intended for illustrative purposes only and does not limit the scope of protection of this invention.

[0059] Step 1: Acquire remote sensing vehicle image data and construct a training dataset. The dataset is randomly divided into training and validation sets in a ratio of 3:1:2, with image sizes ranging from 800x800 to 2000x2000. Subsequently, the dataset images are preprocessed and each image is cropped using a fixed 640x640 sliding window. The cropped sub-images are further enhanced using color jittering and Gaussian blurring, and the corresponding bounding box coordinates are updated simultaneously.

[0060] Step 2: Using the original YOLOv12 model as a baseline, the DSCWHConv convolution module is introduced into the backbone network to improve the standard convolution in the original C3k2 and A2C2f modules. DSCWHConv combines the advantages of wavelet transform and depthwise separable convolution. It can achieve multi-band feature fusion and large receptive field expansion while maintaining computational efficiency, laying the foundation for subsequent remote sensing small target detection.

[0061] (1) First, the four filters used in the two-dimensional Haar wavelet transform are introduced, and then the L-level Haar wavelet transform (WT) is performed on the input feature map F to obtain the low-frequency component of the i-th level And three groups of high frequency components The size of the decomposition space is halved each time, but the number of channels remains unchanged.

[0062] Among them, the four filter formulas used in the two-dimensional Haar wavelet transform are:

[0063]

[0064] Among them, f LL is a low-pass filter, f LH ,f HL ,f HH Horizontal, vertical and diagonal high-pass filters respectively.

[0065] (2) Then, calculate the low-frequency and high-frequency components of each level. The formula is as follows:

[0066]

[0067] Among them, ↓ represents downsampling.

[0068] (3) Perform deep convolution on the low-frequency and high-frequency components of the i-th level, and then scale the low-frequency and high-frequency components to avoid gradient explosion, where the weight is recorded as W (i) The formula is:

[0069]

[0070] Among them, scale(·) represents scaling and dwconv(·) represents depthwise convolution.

[0071] (4) Finally, the inverse wavelet transform (IWT) is used to linearly reconstruct the convolution results of all levels and all frequency bands to obtain the fused feature map. Upsampling is performed first, and then transposed convolution is performed. The formula is as follows:

[0072] [F LL ,F LH ,F HL ,F HH ]=ConvT(up(F),f -1 )

[0073] Among them, ConvT stands for transposed convolution, up stands for upsampling, and f -1 Filter representing the inverse wavelet transform.

[0074] Step 3: In the neck network, integrate the AreaAgentAttention mechanism into each A2C2f module. This mechanism effectively models global and local context by introducing agent tokens and combining them with a regional aggregation strategy, thereby improving the feature representation and spatial positioning capabilities of small vehicle targets.

[0075] (1) Divide the feature map X into M non-overlapping regions in the horizontal or vertical direction, with each region size being (height / 4, width) or (height, width / 4). For each region i of the feature map X, perform a linear mapping of query, key, and value:

[0076] Q=X i W q +b q ,[K,V]=X i W kv +b kv

[0077] Where W q , W kv is the weight matrix, b q and b kv is the bias vector.

[0078] (2) For each region Q i Generate regional proxy tokens using adaptive average pooling:

[0079] A i =Pool(Q i )

[0080] Among them A i As Q i Proxy tokens.

[0081] (3) For each region i, Ai For query, K i , V i Calculate region-level aggregate features for keys and values:

[0082]

[0083] All regional features are concatenated to obtain the global proxy feature:

[0084]

[0085] Among them A i It is the proxy token of Q, K i , V i is the original key value of region i, and σ(·) represents the Softmax function.

[0086] (4) The proxy feature V after aggregation of the above formula A Then perform a second attention broadcast with the original query Q to obtain the output O A :

[0087]

[0088] Where σ(·) represents the Softmax function, Q is the original query (i.e., the query vector after input feature map mapping), O A is the proxy feature V after aggregation A The output obtained in .

[0089] (5) Finally, the broadcast result is added to the bias and dwc results to obtain the final output O.

[0090] O=σ(QA T +B2)σ(AK T +B1)V+dwc(V)

[0091] Among them, B1 and B2 are proxy biases, each of which is composed of three bias components; dwc is depth convolution, which uses depth convolution to maintain feature diversity.

[0092] Step 4: Replace the traditional IoU / GIoU loss function with NWD to enhance the accurate detection of small objects.

[0093] The bounding box loss function is replaced by the NWD loss function. The existing IoU loss function is often unable to adapt to different target tasks when facing small remote sensing targets with drastic scale changes and extremely small overlapping areas. It cannot provide effective gradients for non-overlapping or slightly overlapping boxes and has insufficient generalization ability. For this reason, the normalized Wasserstein distance (NWD) is introduced as a new similarity metric, and it is converted into a regression loss to smoothly and more robustly reflect weak position deviations. NWD-Loss first models the predicted box and the true box as two-dimensional Gaussian distributions, then calculates the Wasserstein distance between them and normalizes them exponentially to obtain a similarity index between 0 and 1. Finally, the difference between it and 1 is taken as the regression loss. It can still maintain gradient information when the overlap disappears, accelerate convergence and improve positioning accuracy. The calculation formula of NWD-Loss is as follows:

[0094]

[0095] where u p ,∑ p with u g ,∑ g are the Gaussian distribution parameters corresponding to the predicted box and the true box, respectively, and C is the normalization constant set according to the average box scale of the dataset.

[0096] Step 5: Model training. By setting the training hyperparameters, the preprocessed dataset from step 2 is input into the improved model for training. The test hardware platform environment is Python 3.11, CUDA 12.1, NVIDIA RTX 4090 GPU, and 24G memory. The specific configuration is shown in Table 1 below:

[0097] Table 1

[0098]

[0099] The training hyperparameters are set as follows: 200 epochs, BatchSize 32, and SGD optimizer. The best training weights are saved as best.pt.

[0100] Step 6: Based on the best model weights (best.pt) saved after training, the improved model's performance was verified on the validation set. The results show that the improved model achieved 8.8% and 10.3% improvements in mAP@0.5 and mAP@0.5:0.95, respectively, compared to YOLOv12, as shown in Table 2. This demonstrates that the proposed method has a significant advantage in detection accuracy.

[0101] Table 2

[0102]

[0103] In order to enhance the reliability of the experimental results, the proposed model was further compared with the mainstream comparison model under multiple evaluation indicators. The main performance indicators include precision (Precision), recall (Recall), average precision (mAP) and computational complexity (Gflops). Among them, precision reflects the proportion of true positive samples in the samples predicted by the model as positive, and recall reflects the model's ability to identify actual positive samples. The higher the Recall value, the stronger the model's ability to capture positive samples; mAP@0.5 represents the average precision when the IoU threshold is 0.5, and mAP@0.5:0.95 is the arithmetic mean of the mAP values ​​in the IoU threshold range from 0.5 to 0.95. The higher this indicator is, the higher the overlap between the predicted box and the true box, that is, the model has more accurate positioning ability. The average precision mAP is obtained by averaging the AP of the detection results of each category, and AP refers to the area of ​​the curve enclosed by the horizontal and vertical axes with the calculated precision and recall. The calculation formulas for Precision and Recall are as follows:

[0104]

[0105] Among them, TP is the number of correct targets predicted as correct, and FP is the number of incorrect targets predicted as correct.

[0106] The calculation formula for average precision mAP is as follows:

[0107]

[0108] Where C is the number of categories, AP i is the average precision of the i-th category.

[0109] As shown in Table 2, while maintaining low computational complexity, the proposed method outperforms the original model in all detection indicators. In particular, the mAP@0.5 is improved by 8.8% and the mAP@0.5:0.95 is improved by 10.3%, which significantly improves the detection accuracy. This verifies the practicality and effectiveness of the proposed method in the task of vehicle detection in remote sensing images.

[0110] The above embodiment is one implementation of the present invention, but the implementation of the present invention is not limited thereto. Any modifications, replacements, and improvements made by those skilled in the art without departing from the principles and spirit of the present invention are included in the scope of protection of the present invention.

Claims

1. A remote sensing vehicle detection method based on multi-model fusion, characterized in that: The following steps are involved: Step 1: Obtain vehicle remote sensing image data, preprocess the dataset image, and obtain the dataset; Step 2: Using the original YOLOv12 model as a benchmark, introduce the DSCWHConv convolution module into the backbone network to replace the standard convolution in the original C3k2 and A2C2f modules; Step 3: In the neck network, the AreaAgentAttention attention mechanism module is used to replace the original attention mechanism module; Step 4: Input the images in the dataset into the improved YOLOv12 model for training to obtain a trained model.

2. The remote sensing vehicle detection method based on multi-model fusion according to claim 1 is characterized in that: The DSCWHConv module described in step 2 includes: (2.1) Introduce the four filters used in the two-dimensional Haar wavelet transform, and then perform L-level Haar wavelet transform (WT) on the input feature map F to obtain the low-frequency component of the i-th level And three groups of high frequency components Each time the decomposition space size is halved, but the number of channels remains unchanged; (2.2) Calculate the low-frequency and high-frequency components of each level; (2.3) Perform depthwise convolution on the low-frequency and high-frequency components of the i-th level respectively, and adjust the feature amplitude by channel-by-channel scaling to enhance training stability; (2.4) Using the inverse wavelet transform (IWT), the convolution results of all levels and all frequency bands are linearly reconstructed to obtain the fused feature map, which is first upsampled and then transposed convolution is performed.

3. The remote sensing vehicle detection method based on multi-model fusion according to claim 2 is characterized in that: In step 2.1, the four filter formulas used for the two-dimensional Haar wavelet transform are: Among them, f LL is a low-pass filter, f LH ,f HL ,f HH Horizontal, vertical and diagonal high-pass filters respectively.

4. The remote sensing vehicle detection method based on multi-model fusion according to claim 2 is characterized in that: The formula used in step 2.2 is as follows: Among them, ↓ represents downsampling.

5. The remote sensing vehicle detection method based on multi-model fusion according to claim 2 is characterized in that: The formula used in step 2.3 is: Among them, scale(·) represents scaling and dwconv(·) represents depthwise convolution.

6. The remote sensing vehicle detection method based on multi-model fusion according to claim 2 is characterized in that: The formula used in step 2.4 is as follows: [F LL ,F LH ,F HL ,F HH ]=ConvT(up(F),f -1 ) Among them, ConvT stands for transposed convolution, up stands for upsampling, and f -1 Filter representing the inverse wavelet transform.

7. The remote sensing vehicle detection method based on multi-model fusion according to claim 1 is characterized in that: The AreaAgentAttention module in step 3 includes: (3.1) Divide the feature map X into M non-overlapping regions in the horizontal and vertical directions respectively. For each region i of the feature map X, perform a linear mapping of query, key, and value: Q i =X i W q +b q ,[K i ,V i ]=X i W kv +b kv Where W q , W kv is the weight matrix, b q and b kv is the bias vector; (3.2) For each region Q i Generate regional proxy tokens using adaptive average pooling: A i =Pool(Q i ) Among them A i As Q i Proxy tokens; (3.3) For each region i, A i For query, K i , V i Calculate region-level aggregate features for keys and values: All regional features are concatenated to obtain the global proxy feature: Among them A i It's Q i Proxy tokens, K i , V i is the original key value of region i, σ(·) represents the Softmax function; (3.4) The proxy feature V after aggregation of the above formula A Then perform a second attention broadcast with the original query Q to obtain the output O A : Among them, σ(·) represents the Softmax function, Q is the original query, that is, the query vector after the input feature map is mapped, O A is the proxy feature V after aggregation A Get the final output; (3.5) Finally, add the residual of the broadcast result to the bias and dwc result to get the final output O. O=σ(QA T +B2)σ(AK T +B1)V+dwc(V) Among them, B1 and B2 are proxy biases, each of which is composed of three bias components; dwc is depth convolution, which uses depth convolution to maintain feature diversity.

8. The remote sensing vehicle detection method based on multi-model fusion according to claim 1 is characterized in that ,The traditional boundary regression loss is replaced by the NWD loss function to enhance the accurate detection of small targets.

Images