An unmanned aerial vehicle image target detection method and system based on a remote sensing basic model auxiliary framework
Patent Information
- Application Number
- CN202410734750.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-06
- Publication Date
- 2026-10-30
- Estimated Expiration
- 2044-06-06
AI Technical Summary
[0008]为了至少能够部分地解决轻量化目标检测网络在航空遥感影像目标检测领域检测性能低、构建航空遥感影像目标检测领域专用算法困难，以及未充分复用现有成果、迁移学习性能提升有限和需要耗费加大资源和样本数据量的问题，本发明提供一种基于遥感基础模型辅助框架的无人机图像目标检测方法和系统，通过大规模遥感基础模型高效提取表征能力强的航空遥感图像特征，与轻量级主干网络的特征提取进行融合，从而弥补各类在自然场景图像上训练的轻量级主干网络在提取航空遥感图像特征性能的不足，并将融合后的特征进行增强，进一步提升航空遥感图像目标检测领域能力
[0064](1)本发明构建了一种融合遥感基础模型的面向航空遥感图像目标检测的辅助架构，能够在基本不改变轻量化主干网络结构的情况下，实现航空遥感图像目标检测性能极大提升。
Smart Images
Figure CN118505980B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of aerial remote sensing image target detection technology, and in particular to a method and system for UAV image target detection based on a remote sensing basic model-assisted framework. Background Technology
[0002] Aerial remote sensing imagery has been widely used in urban planning, resource exploration, disaster relief, traffic management, and other fields. Among these applications, extracting target category information from images through target detection is an important research and application direction. Especially with the rapid development of deep learning technology, deep learning-based aerial remote sensing image target detection technology has become a popular research area.
[0003] In the field of object detection, numerous models are pre-trained on massive amounts of natural scene data (such as ImageNet (a commonly used image recognition dataset), COCO (a large image dataset developed and maintained by Microsoft), and Objects365 (a large-scale, high-quality dataset), with parameter counts ranging from millions to tens of billions, and their application areas also vary. When these pre-trained models, trained on natural scene data, are applied to downstream tasks in aerial remote sensing image object detection, they require transfer training on aerial remote sensing images to achieve good performance. Compared to training from scratch on aerial remote sensing image datasets, initializing model parameters through transfer training yields better aerial remote sensing image object detection results. However, due to differences in data domain, data quantity, data quality, and model application areas, this process still faces the following challenges:
[0004] (1) In order to achieve higher performance in target detection tasks in the field of airborne remote sensing, it can be achieved by increasing the number of dataset instances, increasing training time, and increasing the number of model parameters. However, this leads to higher demand for training resources and training costs, and may even affect the running speed of the model.
[0005] (2) The performance of target detection can be improved by optimizing pipeline structure, innovating backbone network and building efficient optimizer. However, there are problems such as time consumption, high requirements for professional skills and insufficient reuse of existing results. At the same time, targeted optimization operations are also required.
[0006] (3) In downstream tasks in the field of airborne remote sensing, the use of pre-trained models to initialize parameters can improve the final target detection accuracy. However, due to the domain gap between natural scene data and airborne remote sensing image data and the different domains targeted by the models, the performance improvement is limited, especially for lightweight models with a small number of parameters, the effect is even worse.
[0007] (4) Researchers in the field of remote sensing have built foundation models (FM) with huge parameters for specific needs, such as RingMo, SatMAE, RSP, etc. The number of parameters ranges from 27M to 2.4B. However, due to the huge number of parameters, when these models are applied to various downstream tasks in the field of remote sensing, training requires a lot of computing resources, training time and sample data. Summary of the Invention
[0008] To at least partially address the issues of low detection performance of lightweight target detection networks in aerial remote sensing image target detection, difficulty in constructing dedicated algorithms for aerial remote sensing image target detection, insufficient reuse of existing achievements, limited performance improvement through transfer learning, and the need for increased resources and sample data, this invention provides a UAV image target detection method and system based on a remote sensing basic model-assisted framework. This method efficiently extracts aerial remote sensing image features with strong representational capabilities using a large-scale remote sensing basic model, and fuses these features with the feature extraction capabilities of a lightweight backbone network. This compensates for the shortcomings of various lightweight backbone networks trained on natural scene images in extracting aerial remote sensing image features. Furthermore, the fused features are enhanced to further improve the capability of aerial remote sensing image target detection.
[0009] To achieve the above objectives, the technical solution of the present invention is as follows:
[0010] The first aspect of this invention proposes a method for UAV image target detection based on a remote sensing fundamental model-assisted framework, comprising the following steps:
[0011] Step 1: Construct a remote sensing basic model auxiliary framework, which includes a lightweight backbone network unit, a remote sensing basic model unit, a basic feature fusion unit, and a feature aggregation and extension unit, to facilitate UAV image target detection through the constructed remote sensing basic model auxiliary framework;
[0012] Step 2: Input the UAV remote sensing images into the lightweight backbone network unit and the remote sensing basic model unit respectively for data processing and feature extraction to obtain the processed output features;
[0013] Specifically, the lightweight backbone network unit can process the input remote sensing image and the feature data output by the basic feature fusion unit;
[0014] Step 3: Input the output features processed by the lightweight backbone network unit and the remote sensing basic model unit into the basic feature fusion unit for feature fusion to obtain the fused features;
[0015] Step 4: Input the fused features from Step 3 into the lightweight backbone network unit as the input features for the next stage;
[0016] Simultaneously repeat steps three and four until all stages of the lightweight backbone network unit are completed;
[0017] Step 5: Input the features fused in Step 3 into the feature aggregation and expansion unit for feature aggregation and expansion to obtain the processed features, which facilitates feature enhancement;
[0018] Step 6: Complete image target detection based on the features processed in Step 5.
[0019] Furthermore, in step two, the lightweight backbone network unit includes a preprocessing subunit and a lightweight four-stage processing subunit. The preprocessing subunit is used to preprocess the remote sensing image, and the lightweight four-stage processing subunit consists of four stages of feature processing in the lightweight backbone network. Each stage is used to perform step-by-step feature processing on the preprocessed image features.
[0020] Specifically, the preprocessing subunit can select different subunits according to the needs of the lightweight four-stage processing subunit, such as the sampling subunit, the patch embedding subunit, or empty, to facilitate better feature processing.
[0021] The lightweight four-stage processing subunit includes multiple sub-stages.
[0022] Furthermore, in step two, the remote sensing basic model unit includes a visual converter subunit and a feature converter subunit;
[0023] The visual converter subunit is an improved visual converter constructed using a masked autoencoder pre-training algorithm, used to extract rich features from remote sensing images.
[0024] The feature converter subunit resamples the output features of the last layer of the visual converter subunit according to the ratio of 4, 2, 1, and 0.5 to generate four sets of features, which are set to correspond to the output features of the lightweight four-stage processing subunit in step two, so as to facilitate feature fusion.
[0025] Furthermore, in step three, the basic feature fusion unit includes a basic model feature adaptation subunit and a linear cross-attention subunit;
[0026] The feature adaptation subunit includes a convolutional layer, a normalization layer, and an activation layer, which are used to process the number of channels of the input basic feature fusion unit to facilitate feature fusion.
[0027] The linear cross-attention subunit uses a decomposable similarity metric function instead of the normalized exponential function in the standard multi-head attention mechanism, sets the number of attention heads to the number of features, and swaps the calculation order of the standard multi-head attention mechanism to obtain an efficient linear cross-attention calculation method, so as to efficiently calculate multi-head attention and reduce the computational complexity.
[0028] Furthermore, in the linear cross-attention subunit, multi-head efficient attention is calculated according to the following formula:
[0029]
[0030] Where φ(Q) is the decomposition function, Q is the query vector, K is the key vector, and V is the numerical vector, and Q, K, and V all belong to R. N×d , H represents the number of heads in the multi-head attention mechanism, and is element-wise multiplication.
[0031] Furthermore, in step four, the feature aggregation and expansion unit includes upsampling / downsampling subunits, aggregation subunits, expansion subunits, and residual subunits;
[0032] Upsampling and downsampling sub-units are used to amplify or reduce the features fused by the basic feature fusion unit;
[0033] The aggregation subunit is used to aggregate the features output by the upsampling and downsampling subunits or the features directly copied, which facilitates subsequent processing.
[0034] The extended subunit is used to expand the aggregated features, and to convert the number of aggregated feature channels to be the same as the number of feature channels output by the basic feature fusion unit;
[0035] The residual sub-unit is used to add the extended features to the feature residuals of different levels output by the basic feature fusion unit, thereby enhancing the features.
[0036] A second aspect of this invention proposes a UAV image target detection system based on a remote sensing fundamental model-assisted framework, comprising:
[0037] The framework construction module is used to construct the remote sensing basic model auxiliary framework, which includes a lightweight backbone network unit, a remote sensing basic model unit, a basic feature fusion unit, and a feature aggregation and expansion unit.
[0038] The first processing module is used to input the UAV remote sensing images into the lightweight backbone network unit and the remote sensing basic model unit respectively for data processing and feature extraction, and obtain the processed output features.
[0039] Specifically, the lightweight backbone network unit can process the input remote sensing image and the feature data output by the basic feature fusion unit;
[0040] The parameters in the remote sensing basic model unit are frozen and are not used for training but only for feature extraction.
[0041] The fusion module is used to input the output features processed by the lightweight backbone network unit and the remote sensing basic model unit into the basic feature fusion unit for feature fusion to obtain the fused features.
[0042] The loop input module is used to input the features fused by the fusion module as the input features for the next stage into the lightweight backbone network unit;
[0043] Simultaneously, the fusion module and the loop input module are executed repeatedly until all stages of the lightweight backbone network unit are completed;
[0044] The second processing module is used to input the features fused by the fusion module into the feature aggregation and expansion unit for feature aggregation and expansion, so as to obtain the processed features, which facilitates feature enhancement.
[0045] The third processing module is used to complete image target detection based on the features processed by the second processing module.
[0046] Furthermore, in the first processing module, the lightweight backbone network unit includes a preprocessing subunit and a lightweight four-stage processing subunit. The preprocessing subunit is used to preprocess the remote sensing image, and the lightweight four-stage processing subunit consists of four stages of feature processing by the lightweight backbone network. Each stage is used to perform step-by-step feature processing on the preprocessed image features.
[0047] Specifically, the preprocessing subunit can select different subunits according to the needs of the lightweight four-stage processing subunit, such as the sampling subunit, the patch embedding subunit, or empty, to facilitate better feature processing.
[0048] The lightweight four-stage processing subunit includes multiple sub-stages.
[0049] Furthermore, in the first processing module, the remote sensing basic model unit includes a visual converter subunit and a feature converter subunit;
[0050] The visual converter subunit is an improved visual converter constructed using a masked autoencoder pre-training algorithm, used to extract rich features from remote sensing images.
[0051] The feature converter subunit resamples the output features of the last layer of the visual converter subunit according to the ratio of 4, 2, 1, and 0.5 to generate four sets of features, which are set to correspond to the output features of the lightweight four-stage processing subunit in step two, so as to facilitate feature fusion.
[0052] Furthermore, in the fusion module, the basic feature fusion unit includes a basic model feature adaptation subunit and a linear cross-attention subunit;
[0053] The feature adaptation subunit includes a convolutional layer, a normalization layer, and an activation layer, which are used to process the number of channels of the input basic feature fusion unit to facilitate feature fusion.
[0054] The linear cross-attention subunit uses a decomposable similarity metric function instead of the normalized exponential function in the standard multi-head attention mechanism, sets the number of attention heads to the number of features, and swaps the calculation order of the standard multi-head attention mechanism to obtain an efficient linear cross-attention calculation method, so as to efficiently calculate multi-head attention and reduce the computational complexity.
[0055] Furthermore, in the linear cross-attention subunit, multi-head efficient attention is calculated according to the following formula:
[0056]
[0057] Where φ(Q) is the decomposition function, Q is the query vector, K is the key vector, and V is the numerical vector, and Q, K, and V all belong to R. N×d , H represents the number of heads in the multi-head attention mechanism, and is element-wise multiplication.
[0058] Furthermore, in the second processing module, the feature aggregation and expansion unit includes an upsampling / downsampling subunit, an aggregation subunit, an expansion subunit, and a residual subunit;
[0059] Upsampling and downsampling sub-units are used to amplify or reduce the features fused by the basic feature fusion unit;
[0060] The aggregation subunit is used to aggregate the features output by the upsampling and downsampling subunits or the features directly copied, which facilitates subsequent processing.
[0061] The extended subunit is used to expand the aggregated features, and to convert the number of aggregated feature channels to be the same as the number of feature channels output by the basic feature fusion unit;
[0062] The residual sub-unit is used to add the extended features to the feature residuals of different levels output by the basic feature fusion unit, thereby enhancing the features.
[0063] The beneficial effects of this invention are:
[0064] (1) This invention constructs an auxiliary architecture for target detection in aerial remote sensing images that integrates a basic remote sensing model. It can greatly improve the performance of target detection in aerial remote sensing images without significantly changing the lightweight backbone network structure.
[0065] (2) This invention proposes a basic feature fusion unit based on linear attention, which integrates the features of the lightweight target backbone network and the features extracted by the remote sensing basic model. This unit achieves efficient fusion of the two types of features and effectively promotes the model's acquisition of remote sensing image features.
[0066] (3) This invention utilizes the fusion-segmentation structural feature method to achieve further fusion and mutual learning of the acquired multi-scale features, thereby further improving the model's ability to utilize features and improving the target detection accuracy.
[0067] (4) This invention demonstrates the effectiveness of the proposed auxiliary architecture by combining different remote sensing basic models and lightweight target backbone networks and testing them on datasets of different sizes. It can provide high-performance target detection results for subsequent downstream tasks targeting aerial remote sensing images. Attached Figure Description
[0068] Figure 1 A flowchart illustrating a UAV image target detection method based on a remote sensing basic model-assisted framework, provided for embodiments of the present invention.
[0069] Figure 2 This is an architecture diagram of a remote sensing basic model-assisted framework for a UAV image target detection method based on a remote sensing basic model-assisted framework, provided in an embodiment of the present invention.
[0070] Figure 3 This is a schematic diagram of a remote sensing basic model unit for a UAV image target detection method based on a remote sensing basic model auxiliary framework, provided in an embodiment of the present invention.
[0071] Figure 4 This is a schematic diagram of the basic feature fusion unit of a UAV image target detection method based on a remote sensing basic model auxiliary framework, provided in an embodiment of the present invention.
[0072] Figure 5 This is a schematic diagram of a linear cross-attention subunit in a UAV image target detection method based on a remote sensing basic model auxiliary framework, provided in an embodiment of the present invention.
[0073] Figure 6 This is a schematic diagram of the multi-head attention mechanism of a UAV image target detection method based on a remote sensing basic model-assisted framework, provided in an embodiment of the present invention.
[0074] Figure 7This is a schematic diagram of the feature aggregation and expansion unit of a UAV image target detection method based on a remote sensing basic model auxiliary framework, provided in an embodiment of the present invention.
[0075] Figure 8 This is an architecture diagram of an unmanned aerial vehicle (UAV) image target detection system based on a remote sensing basic model-assisted framework, provided in an embodiment of the present invention. Detailed Implementation
[0076] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0077] like Figure 1 As shown, a UAV image target detection method based on a remote sensing basic model-assisted framework includes the following steps:
[0078] S1: Construct a remote sensing basic model auxiliary framework, which includes a lightweight backbone network unit, a remote sensing basic model unit, a basic feature fusion unit, and a feature aggregation and expansion unit;
[0079] S2: Input the UAV remote sensing images into the lightweight backbone network unit and the remote sensing basic model unit respectively for data processing and feature extraction to obtain the processed output features;
[0080] S3: Input the output features processed by the lightweight backbone network unit and the remote sensing basic model unit into the basic feature fusion unit to perform feature fusion and obtain the fused features;
[0081] S4: Input the features fused from S3 into the lightweight backbone network unit as the input features for the next stage;
[0082] Simultaneously repeat S3 and S4 until all stages of the lightweight backbone network unit are completed;
[0083] S5: Input the features fused from S3 into the feature aggregation and expansion unit to perform feature aggregation and expansion, and obtain the processed features;
[0084] S6: Perform image target detection based on the features processed by S5.
[0085] The method specifically includes:
[0086] Construct a remote sensing basic model auxiliary framework, such as Figure 2As shown, the remote sensing basic model auxiliary framework includes a lightweight backbone network unit, a remote sensing basic model unit, a basic feature fusion unit, and a feature aggregation and expansion unit.
[0087] The remote sensing images are input into the lightweight backbone network unit and the remote sensing basic model unit respectively for data processing and feature extraction to obtain the processed output features.
[0088] Based on a review of existing research, most lightweight backbone network designs currently follow a hierarchical structure approach, allowing for the output of results from different stages and their fusion with features extracted from the base model. However, for lightweight backbone networks with simple, non-hierarchical designs, the output channels can be divided into different stages, and the outputs from each stage can be fused with features extracted from the base model.
[0089] Specifically, the lightweight backbone network unit includes a preprocessing subunit and a lightweight four-stage processing subunit. The preprocessing subunit can be selected according to the needs of the lightweight four-stage processing subunit, such as a sampling subunit, a patch embedding subunit, or an empty subunit, to facilitate feature processing at different stages in the subsequent lightweight four-stage processing subunit. The lightweight four-stage processing subunit employs various types of lightweight backbone networks; therefore, X-Backbone is used to represent the selected lightweight backbone network. Its feature processing is divided into four sub-stages according to level, with each sub-stage represented by X-Stage i (i = 1, 2, 3, 4).
[0090] The visual converter subunit in the remote sensing basic model is constructed using a masked autoencoder pre-training algorithm on the large-scale remote sensing dataset MillionAID. The classic visual converter subunit has a simple non-hierarchical structure, belonging to the category of simple visual converters. Since the visual converter subunit needs to compute the relationships between all patches, it places high demands on computational performance and training time. Therefore, local and global attention are used in the cross-interval of the original visual converter to reduce computational and GPU consumption without sacrificing performance. Furthermore, the output of the last layer is resampled to form a simple pyramid structure output.
[0091] like Figure 3 As shown, units 3, 6, 9, and 12 in the visual converter subunit employ global attention mechanisms, while the rest use local attention mechanisms, and the feature size of the output of each visual converter layer is the same.
[0092] The feature converter subunit in the remote sensing basic model unit also divides the processed image features into four sub-stages, each denoted by F-stage j (j = 1, 2, 3, 4). The outputs of the corresponding sub-stages in the lightweight four-stage processing subunit and the feature converter subunit in the remote sensing basic model unit are fused. The fusion result serves as the input to the next sub-stage in the lightweight four-stage processing subunit and is also output to the feature aggregation and expansion unit for further aggregation and expansion.
[0093] like Figure 3 As shown, the input original image data has channel dimension C, height dimension H, and width dimension W. After embedding patching, the output has 768 channels, and both the length and width are reduced to 1 / 16 of the original image. After passing through 12 visual transformer layers, the dimensions remain unchanged. Because this model has a simple, non-hierarchical structure, when applied to object detection tasks, the features output from the last layer need to be resampled by each sub-stage (i.e., F-Stage j (j = 1, 2, 3, 4)) in the feature transformer subunit at a ratio of 4, 2, 1, and 0.5, and used as the final output.
[0094] The remote sensing basic model unit has been pre-trained on various aerospace remote sensing images, and it can effectively extract representational features from remote sensing image data. In order to fuse these extracted features with various lightweight target detection backbone networks, a basic feature fusion unit is proposed to improve the feature processing capability of the lightweight feature extraction backbone network.
[0095] Details of the basic feature fusion unit and the method of feature processing, such as Figure 4 As shown, C is the channel dimension, H is the height dimension, and W is the width dimension. F The feature channels are processed by the patch embedding unit. HS represents the scaled height dimension, WS represents the scaled width dimension, and S represents the scaling dimension. FMFA is the feature adaptation unit for the basic model, and LCAHA is the constructed linear cross-attention subunit based on multi-head efficient attention.
[0096] The input image data is fed into the remote sensing base model unit and the lightweight backbone network unit, respectively. The remote sensing base model unit processes the features through the F-stage j (j = 1, 2, 3, 4) subunit of the feature converter, while the lightweight backbone network unit outputs features using the lightweight four-stage processing subunit X-stage i (i = 1, 2, 3, 4). When i = j, both the F-stage j (j = 1, 2, 3, 4) and X-stage i (i = 1, 2, 3, 4) features are input into the same base model feature adaptation unit for processing.
[0097] In the basic feature fusion unit, when i = j, because the output features of F-stage j (j = 1, 2, 3, 4) in the feature converter subunit differ from the output features of the corresponding lightweight four-stage processing subunit X-stage i (i = 1, 2, 3, 4) in terms of channel number and size, affecting subsequent fusion, when i = j, F-stage j (j = 1, 2, 3, 4) will first be processed by a basic model feature adaptation unit before fusing with the features of X-stage i (i = 1, 2, 3, 4) to obtain features with the same number of channels as X-stage i (i = 1, 2, 3, 4). This subunit includes a convolutional layer (Conv2D), a layer normalization layer (LN), and an activation layer (GELU). The basic model feature adaptation unit only processes the number of input channels and does not process the feature size.
[0098] Features output from the feature adaptation unit of the basic model can be fused with features in F-stage j (j=1,2,3,4) of the feature transformer subunit through a linear cross-attention subunit. This cross-attention mechanism can fuse advantageous information across different backbone networks. However, constructing a cross-attention mechanism based on the original attention mechanism suffers from high computational complexity and time consumption, leading to increased computational complexity and excessive resource requirements when directly using it. Therefore, to reduce the computational complexity of attention, a highly efficient linear cross-attention subunit based on Hydra Attention is constructed to achieve feature fusion, such as... Figure 5 As shown.
[0099] The standard multi-head attention mechanism can be represented by the following formula:
[0100]
[0101] Among them, Q h Query vector with h header, K h V is the key vector when the number of heads is h. h The numerical vector when the h-head is present. H represents the number of heads in the multi-head attention mechanism, and d represents the feature dimension of the label. The computational complexity is O(N). 2 d).
[0102] By replacing the normalized exponential function softmax with a decomposable similarity metric function, and by swapping the computation order of the standard multi-head attention mechanism, we can obtain a linear attention computation method:
[0103] Attn(Q h ,K h V h;φ)=sim(Q,K,V)=(φ(Q h )φ(K h ) T V h =φ(Q h )(φ(K h ) T V h (2)
[0104] Where sim(Q,K,V) represents the decomposable similarity measure function, φ(Q h ) is a function of decomposition.
[0105] In linear attention, increasing the number of attention heads in the model keeps the computational cost constant. Furthermore, when the order of operations is changed in linear attention, adding more heads can actually reduce the computational cost of the layer. Therefore, setting the number of attention heads in the linear cross-attention subunit to the number of features creates an attention unit where both the computation of the flag and the feature are linear, thus maximizing the aforementioned characteristics. Figure 6 As shown. Based on linear attention, the number of attention heads is maximized to equal the number of features, thereby reducing its computational complexity. That is, H is set as the feature dimension d of the flag, and since Q... h K h and V h Since they are all 1×N vectors, they can be transformed into element-wise multiplication operations instead of matrix multiplication operations:
[0106]
[0107] Where Q is the query vector, K is the key vector, and V is the value vector, and Q, K, and V all belong to R. N×d This is element-wise multiplication, and the computational complexity is O(Nd).
[0108] like Figure 6 As shown, in the linear cross-attention subunit based on Hydra Attention, when i = j, the output of F-stage j (j = 1, 2, 3, 4) in the feature converter subunit is used as the query (Q), and the output of X-stage i (i = 1, 2, 3, 4) in the lightweight four-stage processing subunit is used as the key (K) and value (V). First, by adjusting the size R... N×d Multiplying K and V, we get a size R. 1×d K T V; then multiply it with the input query Q to obtain the output. Finally, perform a residual join with the query Q to obtain the final output result.
[0109] The features fused using the basic feature fusion unit are directly used as input to the next sub-stage (X-Stage i+1) of the lightweight four-stage processing sub-unit. Meanwhile, in feature extraction networks, shallow features typically contain more specific and detailed information, while deep features usually contain more abstract semantic information. To better utilize both shallow and deep features, a Feature Aggregation and Expansion Unit (FAEM) is constructed, using the features fused by the basic model feature adaptation units at each level as input, such as... Figure 7 As shown, the feature aggregation and expansion unit mainly includes upsampling / downsampling sub-units, aggregation sub-units, expansion sub-units, and residual sub-units.
[0110] First, the features output by the basic feature fusion units at each level are integrated. Since the feature sizes output by the basic feature fusion units at each level are different, the features at different levels are converted into features with the same size as a certain intermediate layer after passing through upsampling and downsampling sub-units throughout the process.
[0111] Then, the features of the same size after being processed by the upsampling and downsampling subunits or directly copied are aggregated together by the aggregation subunit. The number of channels of the resulting feature is the sum of the input feature channels.
[0112] Subsequently, the features processed by the aggregation subunit need to be fused with the different levels of features output by the basic feature fusion unit to enhance the capabilities of each level of features. Since the aggregated features have changed in terms of the number of channels and size, an extension unit is constructed to convert the number of channels of the aggregated features to the same number of channels as the different levels of features output by the basic feature fusion unit. After passing through upsampling and downsampling subunits or direct copying, the extended features are obtained.
[0113] Finally, the extended features and the different levels of features output by the basic feature fusion unit are added together by the residual sub-unit to enhance the features. The results are then input into the subsequent head and neck units for processing to obtain the final result and complete the image target detection.
[0114] like Figure 8 As shown, a UAV image target detection system based on a remote sensing fundamental model-assisted framework includes:
[0115] The framework construction module is used to construct the remote sensing basic model auxiliary framework, which includes a lightweight backbone network unit, a remote sensing basic model unit, a basic feature fusion unit, and a feature aggregation and expansion unit.
[0116] The first processing module is used to input the UAV remote sensing images into the lightweight backbone network unit and the remote sensing basic model unit respectively for data processing and feature extraction, and obtain the processed output features.
[0117] The fusion module is used to input the output features processed by the lightweight backbone network unit and the remote sensing basic model unit into the basic feature fusion unit for feature fusion to obtain the fused features.
[0118] The loop input module is used to input the features fused by the fusion module as the input features for the next stage into the lightweight backbone network unit;
[0119] Simultaneously, the fusion module and the loop input module are executed repeatedly until all stages of the lightweight backbone network unit are completed;
[0120] The second processing module is used to input the features fused by the fusion module into the feature aggregation and expansion unit for feature aggregation and expansion to obtain the processed features.
[0121] The third processing module is used to complete image target detection based on the features processed by the second processing module.
[0122] It should be noted that the UAV image target detection system based on a remote sensing basic model auxiliary framework provided in this embodiment of the invention is to implement the above-mentioned UAV image target detection method based on a remote sensing basic model auxiliary framework. Its specific functions can be referred to in the above-mentioned method embodiments, and will not be repeated here.
[0123] In summary, this invention constructs an auxiliary architecture for target detection in aerial remote sensing images that integrates a basic remote sensing model. This architecture significantly improves target detection performance in aerial remote sensing images without altering the lightweight backbone network structure. The invention proposes a feature fusion unit based on linear attention, combining features from a lightweight target backbone network and features extracted from the basic remote sensing model. This achieves efficient fusion of the two types of features, effectively promoting the model's acquisition of remote sensing image features. Furthermore, the invention utilizes a fusion-segmentation structure feature to further fuse and learn from the acquired multi-scale features, further enhancing the model's ability to utilize features and improving target detection accuracy. By combining different basic remote sensing models and a lightweight target backbone network, and testing on datasets of varying sizes, the effectiveness of the proposed auxiliary network is demonstrated, providing high-performance target detection results for subsequent downstream tasks involving aerial remote sensing images.
[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for target detection in UAV images based on a remote sensing fundamental model-assisted framework, characterized in that, Includes the following steps: Step 1: Construct a remote sensing basic model auxiliary framework, which includes a lightweight backbone network unit, a remote sensing basic model unit, a basic feature fusion unit, and a feature aggregation and expansion unit; Step 2: Input the UAV remote sensing images into the lightweight backbone network unit and the remote sensing basic model unit respectively for data processing and feature extraction to obtain the processed output features; Step 3: Input the output features processed by the lightweight backbone network unit and the remote sensing basic model unit into the basic feature fusion unit for feature fusion to obtain the fused features; Step 4: Input the fused features from Step 3 into the lightweight backbone network unit as the input features for the next stage; Simultaneously repeat steps three and four until all stages of the lightweight backbone network unit are completed; Step 5: Input the features fused in Step 3 into the feature aggregation and expansion unit for feature aggregation and expansion to obtain the processed features; Step Six: Perform image target detection based on the features processed in Step Five; In step two, the lightweight backbone network unit includes a preprocessing subunit and a lightweight four-stage processing subunit. The preprocessing subunit is used to preprocess the remote sensing image, and the lightweight four-stage processing subunit is the four stages of feature processing of the lightweight backbone network. Each stage is used to perform step-by-step feature processing on the preprocessed image features. In step two, the remote sensing basic model unit includes a visual converter subunit and a feature converter subunit; The visual converter subunit is an improved visual converter constructed using a masked autoencoder pre-training algorithm, used to extract rich features from remote sensing images. The feature converter subunit resamples the output features of the last layer of the visual converter subunit according to the ratios of 4, 2, 1, and 0.5 to generate four sets of features, which are set to correspond to the output features of the lightweight four-stage processing subunit in step two.
2. The UAV image target detection method based on a remote sensing basic model-assisted framework according to claim 1, characterized in that, In step three, the basic feature fusion unit includes a basic model feature adaptation subunit and a linear cross-attention subunit; The feature adaptation subunit includes a convolutional layer, a normalization layer, and an activation layer, which are used to process the number of channels of the input basic feature fusion unit; The linear cross-attention subunit uses a decomposable similarity metric function instead of the normalized exponential function in the standard multi-head attention mechanism, sets the number of attention heads to the number of features, and swaps the calculation order of the standard multi-head attention mechanism to obtain an efficient linear cross-attention calculation method for efficient multi-head attention calculation.
3. The UAV image target detection method based on a remote sensing basic model-assisted framework according to claim 2, characterized in that, In the linear cross-attention subunit, multi-head attention is calculated according to the following formula: in, Let Q be the function to decompose the vector, K be the key vector, and V be the value vector. Q, K, and V all belong to the function. , H represents the number of heads in the multi-head attention mechanism. This is element-wise multiplication.
4. The UAV image target detection method based on a remote sensing basic model-assisted framework according to claim 1, characterized in that, In step five, the feature aggregation and expansion unit includes upsampling and downsampling subunits, aggregation subunits, expansion subunits, and residual subunits; Upsampling and downsampling sub-units are used to amplify or reduce the features fused by the basic feature fusion unit; The aggregation subunit is used to aggregate the features output by the upsampling subunit or the features directly copied. The extended subunit is used to extend the aggregated features; The residual sub-unit is used to add the extended features to the residuals of different levels of features output by the basic feature fusion unit.
5. A UAV image target detection system based on a remote sensing fundamental model-assisted framework, characterized in that, include: The framework construction module is used to construct the remote sensing basic model auxiliary framework, which includes a lightweight backbone network unit, a remote sensing basic model unit, a basic feature fusion unit, and a feature aggregation and expansion unit. The first processing module is used to input the UAV remote sensing images into the lightweight backbone network unit and the remote sensing basic model unit respectively for data processing and feature extraction, and obtain the processed output features. The fusion module is used to input the output features processed by the lightweight backbone network unit and the remote sensing basic model unit into the basic feature fusion unit for feature fusion to obtain the fused features. The loop input module is used to input the features fused by the fusion module as the input features for the next stage into the lightweight backbone network unit; Simultaneously, the fusion module and the loop input module are executed repeatedly until all stages of the lightweight backbone network unit are completed; The second processing module is used to input the features fused by the fusion module into the feature aggregation and expansion unit for feature aggregation and expansion to obtain the processed features. The third processing module is used to complete image target detection based on the features processed by the second processing module. In the first processing module, the lightweight backbone network unit includes a preprocessing subunit and a lightweight four-stage processing subunit. The preprocessing subunit is used to preprocess the remote sensing image, and the lightweight four-stage processing subunit consists of four stages of feature processing in the lightweight backbone network. Each stage is used to perform step-by-step feature processing on the preprocessed image features. In the first processing module, the remote sensing basic model unit includes a visual converter subunit and a feature converter subunit; The visual converter subunit is an improved visual converter constructed using a masked autoencoder pre-training algorithm, used to extract rich features from remote sensing images. The feature converter subunit resamples the output features of the last layer of the visual converter subunit according to the ratios of 4, 2, 1, and 0.5 to generate four sets of features, which are set to correspond to the output features of the lightweight four-stage processing subunit in step two.
6. The UAV image target detection system based on a remote sensing fundamental model-assisted framework according to claim 5, characterized in that, In the fusion module, the basic feature fusion unit includes a basic model feature adaptation subunit and a linear cross-attention subunit; The feature adaptation subunit includes a convolutional layer, a normalization layer, and an activation layer, which are used to process the number of channels of the input basic feature fusion unit; The linear cross-attention subunit uses a decomposable similarity metric function instead of the normalized exponential function in the standard multi-head attention mechanism, sets the number of attention heads to the number of features, and swaps the calculation order of the standard multi-head attention mechanism to obtain an efficient linear cross-attention calculation method for efficient multi-head attention calculation.