Lightweight aerial image target detection method based on fine-grained information and related equipment

Abstract

The invention provides a lightweight aerial image target detection method based on fine-grained information and related equipment. The method comprises the following steps: constructing and training an aerial image target detection network; inputting the aerial image to be detected into the aerial image target detection model to obtain a detection result; wherein a feature extraction network in the aerial image target detection network is constructed and obtained based on a multifunctional convolution module Mconv, and a feature map processing process of the Mconv comprises the steps of firstly processing an input feature map by using two branches, and then multiplying outputs of the two branches element by element to obtain a final output; wherein in one branch, the input feature map is split into two sub-feature maps with the same size according to channels; processing one of the sub-feature maps by using dilated convolution to obtain a feature map with an enlarged receptive field, and then splicing the feature map with the other sub-feature map; in another branch, the significant features of the input feature map are extracted channel by channel using adaptive maximum pooling.

Description

Technical Field

[0001] The present invention relates to the field of image detection technology, and in particular to a lightweight aerial image target detection method based on fine-grained information and related equipment. Background Art

[0002] Due to the characteristics of drones such as ease of use, low cost, and high flexibility, they are widely used in military reconnaissance, civil aerial photography, agricultural monitoring, traffic management, logistics and transportation, etc. However, due to the particularity of drone images (referred to as drone images or aerial images in this invention), most target detection methods cannot be well applied to drone images. Compared with target detection on natural images, drone image target detection has the following two main problems: (1) Due to the different positions and heights of drone shooting, there are many small targets in drone images and they are easily disturbed by complex backgrounds, resulting in the loss of fine-grained target information. (2) The current mainstream detection method is to improve accuracy through complex iterative convolution operations and attention mechanisms. However, while these methods improve accuracy, they also bring high memory overhead and feature redundancy, which brings unbearable load pressure to the drone platform. Summary of the invention

[0003] In order to better utilize the fine-grained information in aerial images and achieve a balance between the detection accuracy and memory overhead of drone images, the present invention provides a lightweight aerial image target detection method based on fine-grained information and related equipment.

[0004] In a first aspect, the present invention provides a lightweight aerial image target detection method based on fine-grained information, comprising:

[0005] Step 1: construct an aerial image target detection network; the aerial image target detection network includes a feature extraction network and a feature fusion network; wherein the feature extraction network extracts a multi-scale feature map of the input aerial image, and the feature fusion network fuses the extracted multi-scale feature map and outputs a target detection result;

[0006] Step 2: training the aerial image target detection network to obtain an aerial image target detection model;

[0007] Step 3: Input the aerial image to be detected into the aerial image target detection model to obtain the detection result;

[0008] Among them, the feature extraction network is constructed based on the multifunctional convolution module Mconv, and the feature map processing process of Mconv includes first using two branches to process the input feature map respectively, and then multiplying the outputs of the two branches element by element to obtain the final output; wherein, in one branch, the input feature map is split into two sub-feature maps of the same size according to the channel; one of the sub-feature maps is processed by using the hole convolution to obtain a feature map with an increased receptive field; the other sub-feature map is spliced ​​with the obtained feature map with an increased receptive field to obtain the output of the branch; in another branch, the adaptive maximum pooling is used to extract the significant features of the input feature map channel by channel to obtain the output of the branch.

[0009] Furthermore, the feature extraction network is constructed based on the multifunctional convolution module Mconv, specifically comprising: firstly constructing an MConvBottleNeck module using Mconv, and then stacking multiple MConvBottleNeck modules and a hierarchical feature aggregation module to form the feature extraction network; wherein the feature processing process of the constructed MConvBottleNeck module comprises:

[0010] First, the input feature map is split into two sub-feature maps of the same size according to the channel; then one of the sub-feature maps is processed by Mconv, and the other sub-feature map is processed by the CBL module; then the outputs of the Mconv and CBL modules are spliced ​​to obtain the first feature map; then the first feature map is subjected to channel shuffling operation; finally, the input feature map and the feature map after the channel shuffling operation are summed using residual connection, and then the feature map after the summation operation is subjected to maximum pooling and processed by the CBL module in turn to obtain the final output.

[0011] Furthermore, the hierarchical feature aggregation module adopts an inter-layer cascade information aggregation pooling ICIAP module; wherein the feature graph processing process of the ICIAP module includes:

[0012] First, use the convolution layer to optimize the channel of the input feature map to obtain the second feature map;

[0013] The first pooling cascade module and the second pooling cascade module are respectively used to perform pooling processing on the second feature map; wherein the first pooling cascade module includes m layers of average pooling and m-1 layers of maximum pooling that are alternately cascaded in sequence, and the second pooling cascade module includes m layers of maximum pooling and m-1 layers of average pooling that are alternately cascaded in sequence; m is an integer greater than 1.

[0014] The pooling result of the current pooling layer corresponding to the two pooling cascade modules is summed with the correlation result of the previous pooling layer to obtain the correlation result of the current pooling layer; wherein the feature map obtained by summing the pooling result of the first pooling layer corresponding to the two pooling cascade modules and the second feature map is the correlation result of the first pooling layer;

[0015] The second feature map is sequentially concatenated with the association results of all pooling layers and processed by convolutional layers to obtain the final output.

[0016] Furthermore, the feature extraction network adopts a feature recombination enhancement network FRENet, and the FRENet is constructed based on the FRE module; wherein the feature processing process of the FRE module includes:

[0017] The input feature map is extracted channel by channel according to the following formula to obtain two sub-feature maps f1 and f2 of the same size;

[0018] f1=Concat(c0,c2,…,c n )

[0019] f2=Concat(c1,c3,…,c n-1 )

[0020] The sub-feature map f1 is processed by a convolution layer, and the sub-feature map f2 is processed by a dilated convolution layer. At the same time, adaptive maximum pooling is used to extract the salient features of the sub-feature maps f1 and f2 channel by channel respectively;

[0021] The outputs of the convolution layer and the dilated convolution are concatenated to obtain the third feature map.

[0022] Using the significant features of one of the sub-feature maps to modulate another sub-feature map, and performing a splicing operation on the modulation results to obtain a fourth feature map;

[0023] The third feature map and the fourth feature map are summed up and processed by the convolution layer in sequence to obtain the final output.

[0024] Furthermore, the feature fusion process and detection process of the constructed FRENet include:

[0025] The feature maps of each scale output by the feature extraction module are processed using the FRE module respectively;

[0026] For feature maps of each scale other than the feature map with the largest scale output by the FRE module, feature enhancement processing is performed using the feature enhancement module respectively, so as to sum the feature maps processed by all feature enhancement modules with the feature map with the largest scale output by the FRE module to obtain the first-level fused feature map;

[0027] The feature enhancement module is used to perform feature enhancement processing on the l-th level fused feature map, and then the obtained feature map is concatenated with the feature map output by the FRE module of the adjacent layer to obtain the l+1-th level fused feature map; where l is an integer not less than 1;

[0028] The feature maps obtained after the feature enhancement processing of the fused feature maps at all levels by the feature enhancement module are first processed by the convolution layer, and then detected by the detection head to obtain the detection results.

[0029] Furthermore, the feature enhancement module adopts an ELAN-R module; the feature enhancement processing process of the ELAN-R module includes:

[0030] First, use two branches to process the input feature map separately, then splice the outputs of the two branches and process them through a 1×1 convolution layer; then use a residual connection to sum the input feature map and the output of the 1×1 convolution layer, and then process it through a 1×1 convolution layer to get the final output; in one branch, the input feature map is processed by a 1×1 convolution layer; in the other branch, the input feature map is first processed by a 1×1 convolution layer and two 3×3 convolution layers in sequence, and then the output of the 1×1 convolution layer is spliced ​​with the output of the last 3×3 convolution layer using a residual connection.

[0031] In a second aspect, the present invention provides a lightweight aerial image target detection system based on fine-grained information, comprising:

[0032] A network construction module is used to construct an aerial image target detection network; the aerial image target detection network includes a feature extraction network and a feature fusion network; wherein the feature extraction network extracts a multi-scale feature map of the input aerial image, and the feature fusion network fuses the extracted multi-scale feature map and outputs a target detection result;

[0033] A training module, used to train the aerial image target detection network to obtain an aerial image target detection model;

[0034] A detection module, used to input the aerial image to be detected into the aerial image target detection model to obtain a detection result;

[0035] Among them, the feature extraction network is constructed based on the multifunctional convolution module Mconv, and the feature map processing process of Mconv includes first using two branches to process the input feature map respectively, and then multiplying the outputs of the two branches element by element to obtain the final output; wherein, in one branch, the input feature map is split into two sub-feature maps of the same size according to the channel; one of the sub-feature maps is processed by using the hole convolution to obtain a feature map with an increased receptive field; the other sub-feature map is spliced ​​with the obtained feature map with an increased receptive field to obtain the output of the branch; in another branch, the adaptive maximum pooling is used to extract the significant features of the input feature map channel by channel to obtain the output of the branch.

[0036] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0037] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0038] Beneficial effects of the present invention:

[0039] (1) A multifunctional convolution (MConv) module is designed, and a feature extraction network is constructed based on the module. The MConv module uses the effectiveness of multi-type features in understanding the target during hierarchical processing to aggregate and weight multi-receptive field information, overcoming the problem that the conventional convolution receptive field is single and easy to lose the fine-grained information mentioned above. Therefore, by applying MConv to the feature extraction network, the focus is on extracting multi-receptive field features while retaining fine-grained information, improving the model's sensitivity to multi-scale targets in drone images, thereby achieving a balance between the detection accuracy and memory overhead of the model to a certain extent.

[0040] (2) Based on the diverse information provided by MConv, a new MConvBottleNeck architecture is constructed by combining MConv with channel shuffle operations and residual connection operations to achieve channel information flow, further enhance the expression ability of target feature information, and strengthen the extraction of target representation information in UAV images.

[0041] (3) In order to effectively capture the contextual information in high-level semantic information and preserve the original fine-grained information as much as possible, an inter-layer cascade information aggregation pooling (ICIAP) module is also designed. This module, together with MConvBottleNeck, constitutes a new feature extraction network MCBNet, which can further improve the detection accuracy of the model.

[0042] (4) A feature recombination enhancement (FRE) module was designed. FRE splits the original feature map and uses the two sets of features after the split to enhance the information between each other and aggregate the original information, further improving the information exchange between adjacent channels. Based on the FRE module, a feature fusion network (FRENet) based on the feature recombination enhancement (FRE) module was further designed. It can use the complementary characteristics of information in different channel layers to obtain the result after overall channel enhancement, effectively improving the detection capability of targets of different scales.

[0043] (5) The present invention takes into account the characteristics of the drone platform, effectively utilizes the fine-grained information of the target and realizes the interaction between channel information. Finally, these feature maps will be sent to the prediction head for detection to obtain the final detection results. Experiments on the VisDrone dataset show that compared with other algorithms, the method of the present invention has better performance in drone image target detection. It can be seen from the ablation experiment that each improved module proposed in the present invention can effectively improve the detection accuracy. Therefore, the method proposed in the present invention is more suitable for drone image target detection than the existing target detection algorithm, and has important practical significance for the research on drone image target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic flow chart of a lightweight aerial image target detection method based on fine-grained information provided by an embodiment of the present invention;

[0045] Figure 2 A diagram of an aerial image detection network architecture provided by an embodiment of the present invention;

[0046] Figure 3 A structural diagram of a multifunctional convolution module provided by an embodiment of the present invention;

[0047] Figure 4 The MConvBottleNeck module structure diagram provided by the embodiment of the present invention;

[0048] Figure 5 A structural diagram of an inter-layer cascade information aggregation and pooling module provided in an embodiment of the present invention;

[0049] Figure 6 A structural diagram of a feature reorganization and enhancement module provided in an embodiment of the present invention;

[0050] Figure 7 The structure diagram of ELAN-R provided by the embodiment of the present invention;

[0051] Figure 8 Visual comparison results of the method of the present invention provided in the embodiment of the present invention on the Visdrone data set;

[0052] Fig. 9 A schematic diagram of the structure of a lightweight aerial image target detection system based on fine-grained information provided by an embodiment of the present invention;

[0053] Fig.10 A structural diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Example 1

[0056] like Figure 1 As shown, an embodiment of the present invention provides a lightweight aerial image target detection method based on fine-grained information, comprising the following steps:

[0057] S101: constructing an aerial image target detection network; the aerial image target detection network includes a feature extraction network and a feature fusion network; wherein the feature extraction network extracts a multi-scale feature map of the input aerial image, and the feature fusion network fuses the extracted multi-scale feature map and outputs a target detection result;

[0058] In this embodiment, the feature extraction network is constructed based on the multifunctional convolution module Mconv, for example, multiple Mconvs are directly stacked. Or on the basis of the above, a hierarchical feature aggregation module is stacked after the last Mconv, and the hierarchical feature aggregation module refers to a module having the function of fusing multi-scale information on feature maps at different levels (or also referred to as having the function of fusing multi-scale contextual information), which can be an existing module such as SPP and SPPF. This is because as the depth of the target detection network expands, the semantic information of the target becomes richer, which helps to understand the overall concept of the target. However, the targets in the drone images are generally surrounded by complex background environments, which causes environmental factors to be included as part of the target semantic information. Therefore, pure high-order semantic information generally cannot fully meet the needs of drone image target detection, so at the end of the feature extraction network, it is necessary to aggregate high and low-order semantic information of different levels through a hierarchical feature aggregation module. The feature fusion network can use existing modules such as PAN, BIFPN, etc.

[0059] S102: training the aerial image target detection network to obtain an aerial image target detection model;

[0060] S103: Inputting the aerial image to be detected into the aerial image target detection model to obtain a detection result;

[0061] Specifically, in this embodiment, the structure of the multifunctional convolution module Mconv is as follows: Figure 3 The feature map processing process of Mconv includes: first, the input feature map F in ∈R C×H×W Split into sub-feature maps of the same size by channel and In order to improve the receptive field of the model, a dilated convolution is used on the sub-feature map x1 to obtain a feature map with an increased receptive field. In order to retain fine-grained information, no additional operations are performed on the sub-feature map x2, and the location information of the target is retained to the maximum extent. This design follows the style of the Inception-style structure, which not only reduces information redundancy but also retains more detailed information. After obtaining multi-receptive field information, the feature concatenation operation Concat is used to restore the size of the input feature map. The calculation process is shown in formulas (1) and (2):

[0062] x1,x2=Split(F in ,dim=1) (1)

[0063] Y=Concat(DC 3×3 (x1),x2) (2)

[0064] where Y∈R C×H×W Represents the feature map obtained after feature concatenation, DC 3×3 () represents a dilated convolution with a kernel of 3 and a dilation rate of 2.

[0065] Secondly, adaptive max pooling is used to extract the input feature map F channel by channel in The significant features of this group of feature vectors are recorded as a∈R C×1×1 .

[0066] Finally, a is used to multiply Y element by element to enhance the channel feature representation capability of the multi-receptive field feature Y. The calculation process is shown in formulas (3) and (4).

[0067] a=AdaptMaxpool(F in ) (3)

[0068]

[0069] Among them, F out ∈R C×H×WRepresents the final output result of MConv, Stands for element-wise multiplication.

[0070] From the above process, we can see that the MConv module expands the receptive field based on the idea of ​​layered processing while retaining more fine-grained information of the target. In addition, MConv uses the channel attention mechanism to learn the importance of each channel of the input feature. By applying MConv to the backbone network, focusing on extracting multi-receptive field features while retaining fine-grained information, the model's sensitivity to multi-scale targets in drone images is improved, thereby achieving a balance between the detection accuracy and memory overhead of the model to a certain extent.

[0071] Example 2

[0072] Although the multi-type features provided by MConv are richer than those of traditional convolution, the information depth of these features extracted by directly using stacked MConv may not meet the higher requirements of drone image detection. Therefore, based on MConv, the embodiment of the present invention also designs a new feature extraction layer MConvBottleNeck to expand the richness and depth of features. Its structure diagram is shown in the figure. Figure 4 shown.

[0073] In this embodiment, the feature processing process of the feature extraction layer MConvBottleNeck includes: first, the input feature map F in ∈R C×H×W Split into sub-feature maps of the same size by channel and MConv is used to expand the receptive field of the sub-feature map f1 and retain fine-grained information, and 3×3 convolution is used to extract local information in f2.

[0074] Secondly, the Concat operation is used to combine the feature information extracted by MConv and 3×3 convolution to obtain F1∈R C×H×W In order to strengthen the information exchange and transfer between f1 and f2 and enhance the generalization ability of the model, a channel shuffle operation is used to divide F1 into multiple channel groups, and these channel groups are recombined to promote information exchange between different channels.

[0075] Finally, in order to solve the gradient vanishing and gradient exploding problems, residual connections are used to help the model converge faster. In addition, the maximum pooling is used to extract significant features in 2×2 grid units, and the 1×1 convolution is used to widen the channel depth. The calculation process is shown in formulas (5), (6), and (7).

[0076] f1,f2=Split(F in , dim=1) (5)

[0077] F1=Concat(MConv(f1),CBL 3×3 (f2)) (6)

[0078]

[0079] in, Represents the output feature map of MConvBottleNeck, Stands for element-wise addition. CBL i×i () represents the existing CBL module, which includes the traditional convolution Conv with convolution kernel size i, normalization and LeakyReLu activation function.

[0080] The embodiment of the present invention can further enhance the information exchange in different layer features and expand the richness of information by applying MConvBottleNeck to the feature extraction network, thereby generating a global information exchange feature F out , further improving the detection accuracy of the model.

[0081] It is understandable that the other steps may refer to the above-mentioned embodiment 1 and will not be described in detail here.

[0082] Example 3

[0083] Different from the hierarchical feature aggregation module in the above embodiments, which uses an existing module, in this embodiment, in order to further improve the ability to accurately focus on the target's own information, the embodiment of the present invention proposes an inter-layer cascade information aggregation pooling (ICIAP) module as a hierarchical feature aggregation module to extract contextual information of different scales of the target, and combine high-level semantic information to improve the information extraction ability of the model. The structure of the ICIAP module is as follows: Figure 5 shown.

[0084] The feature processing of the ICIAP module includes: First, in order to further alleviate the model's dependence on computing resources, that is, to reduce the model's computing resource overhead, a 1×1 convolution is used to adjust the input feature map F in ∈R C×H×W The number of channels is the original The feature map F1 is obtained. Secondly, the module explores the features of different receptive fields by utilizing cascaded pooling layers. Next, in order to construct the contextual relationship around the target, two strategies are adopted. The first strategy is to use alternating maximum pooling and average pooling to extract key information around the target. The second strategy is to use multi-level association to retain detailed information in the feature map. The specific implementation method is to associate the output result of the current pooling layer with the association result of the previous pooling layer, and add the elements of their corresponding positions. Finally, multi-level association is used to splice the feature maps of each level together, and 1×1 convolution is used to fuse the information of each level and restore the number of channels. The final feature map not only retains the information of multiple receptive fields, but also constructs the relationship between the target and the surrounding environment. The calculation process is shown in formulas (8), (9), and (10):

[0085] F1=CBL 1×1 (F in ) (8)

[0086]

[0087] F out =CBL 1×1 (Concat(F1,Z1,Z2,Z3)) (10)

[0088] in Represents the result after channel optimization, Represents the association results of each layer, Represents the output of ICIAP.

[0089] The ICIAP proposed in the embodiment of the present invention can capture the target context information of multiple receptive fields, which is combined with the rich semantic information of the target provided by MConvBottleNeck to form a new feature extraction network, such as Figure 2 The MCBNet shown in can jointly enhance the model's sensitivity to target location and category.

[0090] Example 4

[0091] The scales of targets in drone images vary due to changes in shooting height and angle, ranging from targets with a size of only a few pixels to targets with a size of dozens of pixels. These multi-scale targets have different information representations in different feature layers. In order to further match the feature correspondence between different layers, although the FPN network or its variants can be used to fuse information from different feature layers, these methods only consider the information interaction between different feature layers, and do not consider the information exchange between channels within the feature layers. Therefore, unlike the feature fusion network in the above-mentioned embodiments that uses existing modules, in this embodiment, in order to better exchange and fuse the fine-grained information between the internal channels of each scale feature map output by the feature extraction module, a new feature fusion network is designed in the embodiment of the present invention, namely, the Feature Recombination and Enhancement Network (FRENet). The FRENet is constructed based on the FRE (Feature Recombination and Enhancement) module. The FRE module is committed to using information exchange between adjacent channels to enhance the information expression of the overall feature map, and its structure is as follows: Figure 6 shown.

[0092] Specifically, the feature processing process of the FRE module includes: first, the input feature map F in ∈R C×H×W Extract each channel into two sub-feature vectors of equal size and The calculation process is shown in formulas (11) and (12):

[0093] f1=Concat(c0,c2,…,c n ) (11)

[0094] f2=Concat(c1,c3,…,c n-1 ) (12)

[0095] Among them, c i Representative feature map F in The i-th channel of , n=C-1.

[0096] Secondly, perform 3×3 convolution on f1 to modulate the local information, perform 3×3 hole convolution on f2 to expand the receptive range of feature information, and use the Concat operation to connect the information of the two to obtain the feature map F1∈R C×H×W The calculation process is shown in formula (13):

[0097] F1=Concat(CBL 3×3 (f1),DC 3×3(f2)) (13)

[0098] At the same time, adaptive maximum pooling is used to extract the most significant features of f1 and f2 channel by channel, and the extracted feature vectors are recorded as

[0099] Then, f2 and f1 are modulated by a1 and a2 respectively, and the results of information modulation are connected to obtain the feature map F2∈R C×H×W The information complementary enhancement between adjacent channels within the feature map is realized. The calculation process is shown in formulas (14) and (15):

[0100] a i =AdaptMaxpool(f i ),i=1,2 (14)

[0101]

[0102] Finally, the sum operation is performed on F1 and F2, and the information of the two is integrated using 1×1 convolution. The calculation process is shown in formula (16):

[0103]

[0104] Among them, F out ∈R C×H×W Represents the output result of the FRE module.

[0105] The FRE module can align and interact local context information across channels and generate cross-channel embedding features containing different details and semantics.

[0106] Furthermore, the structure of the constructed FRENet is as follows Figure 2 As shown. The feature fusion process and detection process of FRENet include:

[0107] The feature maps of each scale output by the feature extraction module are processed using the FRE module respectively;

[0108] For feature maps of each scale other than the feature map with the largest scale output by the FRE module, feature enhancement processing is performed using the feature enhancement module respectively, so as to sum the feature maps processed by all feature enhancement modules with the feature map with the largest scale output by the FRE module to obtain the first-level fused feature map;

[0109] The feature enhancement module is used to perform feature enhancement processing on the l-th level fused feature map, and then the obtained feature map is concatenated with the feature map output by the FRE module of the adjacent layer to obtain the l+1-th level fused feature map; where l is an integer not less than 1;

[0110] The feature maps obtained after the feature enhancement processing of the fused feature maps at all levels by the feature enhancement module are first processed by the convolution layer, and then detected by the detection head to obtain the detection results.

[0111] The FRENet provided in this embodiment first uses the FRE module to realize feature exchange within the main feature layer, and then uses the classic feature fusion structure ("V"-shaped feature fusion structure) to realize feature exchange between main feature layers.

[0112] Furthermore, in addition to using existing modules (such as the ELAN module and the existing improved modules based on the ELAN module) as the feature enhancement module, the embodiment of the present invention also provides a new improved module based on the ELAN module, called the ELAN-R module. Figure 7 shown.

[0113] The feature enhancement processing process of the ELAN-R module includes: first use two branches to process the input feature map respectively, then splice the outputs of the two branches and process them through a 1×1 convolution layer; then use a residual connection to sum the input feature map and the output of the 1×1 convolution layer, and then process it through a 1×1 convolution layer to obtain the final output; wherein, in one branch, the input feature map is processed by a 1×1 convolution layer; in another branch, the input feature map is first processed by a 1×1 convolution layer and two 3×3 convolution layers in sequence, and then the output of the 1×1 convolution layer is spliced ​​with the output of the last 3×3 convolution layer using a residual connection. In the ELAN-R module of this embodiment, residual connections and 1×1 convolutions are additionally added to improve the convergence speed of the model, which can prevent the occurrence of gradient vanishing and explosion problems.

[0114] The network model formed by combining all the improved modules of the present invention is recorded as UAVDNet. Figure 2 In order to verify the effectiveness of the entire network model and each improved module in the model, the present invention also provides the following experimental data.

[0115] The experimental environment used is as follows: the server operating system is Ubuntu 18.04.6LTS system, the GPU model of the server is NVIDIA GeForce RTX 3090 (24G), and the CPU processor is Intel (R) Xeon (R) Silver 4114 CPU @ 2.20GHz. The Python deep learning framework is used, where the PyTorch version is 1.10.2, the Python version is 3.6, and the CUDA version is 11.7. In the training phase of the experiment, the stochastic gradient descent (SGD) optimizer with momentum is used. The initial learning rate of the training is set to 0.021, the weight decay coefficient is set to 0.0005, the momentum parameter is set to 0.937, and the batch size is set to 16. In the training and testing phases of the entire experiment, all image input sizes are 640×640.

[0116] Dataset used: Experiments were conducted on the UAV public dataset VisDrone dataset and compared with advanced target detection methods, with 300 training iterations.

[0117] The experimental results are shown in Table 1. The data in the table show that the method of the present invention achieved the highest mAP0.5 and mAP0.95 values, reaching 48.1% and 27.0% respectively. Although UAVDNet did not achieve the best in other indicators, the value of each item is close to the highest value achieved in that item. For example, the mAP0.75 value of UAVDNet is only 0.5% different from the highest value of that item. In addition, UAVDNet has a good performance in AP S , A.P. M and AP L The proposed method has achieved excellent performance in all indicators, achieving 18.9%, 37.4% and 41.7% performance in each indicator respectively. However, the reason why the proposed method failed to achieve comprehensive leadership is that the proposed method focuses on reducing the model's dependence on computing resources and adopts a lightweight design to serve the UAV platform. The memory consumption is lower than that of most algorithms in Table 1.

[0118] Table 3 shows a comprehensive comparison between UAVDNet and similar lightweight algorithms. Among the algorithms of the same level, UAVDNet has a clear advantage. Compared with the YOLOv8 series of algorithms, the overall performance of UAVDNet is better than all versions except YOLOv8x. Compared with YOLOv8x, the advantages of YOLOv8x are mainly reflected in mAP0.75, AP M and AP L In terms of indicators, however, UAVDNet's mAP0.5 value, mAP0.95 value and AP SThe value is more advanced. In terms of memory usage, the number of parameters of UAVDNet is only about 6.8% of YOLOv8x. Compared with the YOLOv10 series algorithms, the method of the present invention has achieved comprehensive leadership. In general, the method of the present invention achieves a balance between detection efficiency and memory usage, and has better comprehensive performance.

[0119] Table 1 Performance comparison of the proposed method and existing advanced detection methods on the VisDrone dataset

[0120]

[0121]

[0122] Figure 8 Some visual detection results of various scenes on the Visdrone dataset are shown. The first column in the figure shows the true value labeling results of the dataset, and the second column shows the detection results of UAVDNet. In the first group of comparison pictures, the dataset did not mark the smaller vehicle targets above the picture, however, UAVDNet accurately detected the targets above the picture after learning the target features. In the second group of comparison pictures, although UAVDNet failed to completely detect the small targets far above the image, it detected a certain amount of distant targets and all targets in the main area of ​​the image. In the third and fourth groups of comparison pictures, the dataset did not mark the vehicles in the parking lot above the image, but UAVDNet effectively detected some valid targets in the parking lot. This proves that the method of the present invention has a strong learning ability and has a good target detection effect in drone images.

[0123] At the same time, some advanced lightweight single-stage target detection methods are also compared. Comprehensive consideration is made from the aspects of multi-scale target detection accuracy, parameter quantity, and processing time. The experimental results are shown in Table 2. In terms of accuracy, UAVDNet has achieved L The highest value of the remaining indicators in the indicator appearance 3. In terms of memory usage, the parameter volume of UAVDNet is only 4.4M, which is in an advantageous position among a number of lightweight models. In terms of processing speed, UAVDNet can detect 50 pictures per second, which fully meets the requirements of real-time detection. Although YOLOv5s and YOLO-UAVlite have the fastest speed and the least memory consumption respectively, their detection capabilities are far behind UAVDNet. Overall, the method of the present invention has achieved a better balance between detection accuracy and processing speed.

[0124] Table 2 Comparison of detection results of lightweight methods on VisDrone dataset

[0125]

[0126] In order to verify the effectiveness of the improved modules proposed in this paper, this study further uses ablation experiments to explore each method. Table 3 shows the results of the ablation experiment, and the analysis of the results is as follows:

[0127] 1) MCBNet: To verify the effectiveness of MCBNet, we selected three excellent lightweight backbone networks and compared them with the classic PANet feature fusion network. The comparison results show that the combination of MCBNet and PANet achieved the best performance in all indicators shown in the table, with the mAP0.5 value exceeding the second place by 4.7%. In addition, the memory consumption after the introduction of MCBNet was the lowest among the four.

[0128] 2) ICIAP: To verify the effectiveness of ICIAP, we selected three spatial pyramid pooling modules with the same function as ICIAP for comparison. From the data in the table, we can see that although the four modules are all designed to extract contextual information, ICIAP has better detection capabilities based on the same resource consumption. This proves that ICIAP has a stronger ability to extract contextual information from high-order semantic feature maps, and it can achieve the best detection effect when combined with the UAVDNet we proposed.

[0129] 3) FRENet: In order to verify the effectiveness of FRENet, we selected three mainstream feature fusion networks and combined them with the MCBNet proposed by us for experiments. The experimental results show that FRENet with the FRE module added has achieved the best performance in all indicators in the table. The combination of PANet and MCBNet also achieved good detection performance with the least memory consumption. However, our method has improved the mAP0.5 value of the former by 4%, while the additional parameters introduced are only 0.8M. This shows that the FRE module based on channel splitting and reorganization proposed by us can effectively improve the feature perception ability of the model without consuming too many resources.

[0130] Table 3 Ablation test results of each improved module of UAVDNet on VisDrone dataset

[0131]

[0132] Fig. 9 An embodiment of the present invention provides a lightweight aerial image target detection system based on fine-grained information, including a network construction module, a training module and a detection module.

[0133] Specifically, the network construction module is used to construct an aerial image target detection network; the aerial image target detection network includes a feature extraction network and a feature fusion network; wherein the feature extraction network extracts a multi-scale feature map of the input aerial image, and the feature fusion network fuses the extracted multi-scale feature map and outputs the target detection result. The training module is used to train the aerial image target detection network to obtain an aerial image target detection model. The detection module is used to input the aerial image to be detected into the aerial image target detection model to obtain the detection result. Among them, the feature extraction network is constructed based on the multifunctional convolution module Mconv, and the feature map processing process of Mconv includes first using two branches to process the input feature map respectively, and then multiplying the outputs of the two branches element by element to obtain the final output; wherein, in one branch, the input feature map is split into two sub-feature maps of the same size according to the channel; one of the sub-feature maps is processed by using the hole convolution to obtain a feature map with an increased receptive field; the other sub-feature map is spliced ​​with the obtained feature map with an increased receptive field to obtain the output of the branch; in another branch, the adaptive maximum pooling is used to extract the significant features of the input feature map channel by channel to obtain the output of the branch.

[0134] It should be noted that the detection system provided in the embodiments of the present invention is for implementing the above-mentioned methods. The specific functions thereof can be referred to the above-mentioned method embodiments, which will not be described in detail here.

[0135] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10 As shown, the electronic device may include: a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002 and the memory 1003 communicate with each other through the communication bus 1004. The processor 1001 may call the logic instructions in the memory 1003 to execute the aerial image target detection method, which includes: constructing an aerial image target detection network; the aerial image target detection network includes a feature extraction network and a feature fusion network; wherein the feature extraction network extracts a multi-scale feature map of the input aerial image, and the feature fusion network fuses the extracted multi-scale feature map and outputs a target detection result; the aerial image target detection network is trained to obtain an aerial image target detection model; the aerial image to be detected is input into the aerial image target detection model to obtain a detection result.

[0136] In addition, when the logic instructions in the above-mentioned memory 1003 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0137] An embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: constructing an aerial image target detection network; the aerial image target detection network includes a feature extraction network and a feature fusion network; wherein the feature extraction network extracts a multi-scale feature map of the input aerial image, and the feature fusion network fuses the extracted multi-scale feature map and outputs a target detection result; the aerial image target detection network is trained to obtain an aerial image target detection model; the aerial image to be detected is input into the aerial image target detection model to obtain a detection result.

[0138] An embodiment of the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the methods provided by the above-mentioned method embodiments are implemented, for example, including: constructing an aerial image target detection network; the aerial image target detection network includes a feature extraction network and a feature fusion network; wherein the feature extraction network extracts a multi-scale feature map of the input aerial image, and the feature fusion network fuses the extracted multi-scale feature map and outputs a target detection result; the aerial image target detection network is trained to obtain an aerial image target detection model; the aerial image to be detected is input into the aerial image target detection model to obtain a detection result.

[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lightweight aerial image target detection method based on fine-grained information, characterized in that: include: Step 1: construct an aerial image target detection network; the aerial image target detection network includes a feature extraction network and a feature fusion network; wherein the feature extraction network extracts a multi-scale feature map of the input aerial image, and the feature fusion network fuses the extracted multi-scale feature map and outputs a target detection result; Step 2: training the aerial image target detection network to obtain an aerial image target detection model; Step 3: Input the aerial image to be detected into the aerial image target detection model to obtain the detection result; Among them, the feature extraction network is constructed based on the multifunctional convolution module Mconv, and the feature map processing process of Mconv includes first using two branches to process the input feature map respectively, and then multiplying the outputs of the two branches element by element to obtain the final output; wherein, in one branch, the input feature map is split into two sub-feature maps of the same size according to the channel; one of the sub-feature maps is processed by using the hole convolution to obtain a feature map with an increased receptive field; the other sub-feature map is spliced ​​with the obtained feature map with an increased receptive field to obtain the output of the branch; in another branch, the adaptive maximum pooling is used to extract the significant features of the input feature map channel by channel to obtain the output of the branch.

2. According to claim 1, a lightweight aerial image target detection method based on fine-grained information is characterized in that: The feature extraction network is constructed based on the multifunctional convolution module Mconv, specifically comprising: firstly constructing an MConvBottleNeck module using Mconv, and then stacking multiple MConvBottleNeck modules and a hierarchical feature aggregation module to form the feature extraction network; wherein the feature processing process of the constructed MConvBottleNeck module comprises: First, the input feature map is split into two sub-feature maps of the same size according to the channel; then one of the sub-feature maps is processed by Mconv, and the other sub-feature map is processed by the CBL module; then the outputs of the Mconv and CBL modules are spliced ​​to obtain the first feature map; then the first feature map is subjected to channel shuffling operation; finally, the input feature map and the feature map after the channel shuffling operation are summed using residual connection, and then the feature map after the summation operation is subjected to maximum pooling and processed by the CBL module in turn to obtain the final output.

3. The lightweight aerial image target detection method based on fine-grained information according to claim 2 is characterized in that: The hierarchical feature aggregation module adopts an inter-layer cascade information aggregation pooling ICIAP module; wherein the feature graph processing process of the ICIAP module includes: First, use the convolution layer to optimize the channel of the input feature map to obtain the second feature map; The first pooling cascade module and the second pooling cascade module are respectively used to perform pooling processing on the second feature map; wherein the first pooling cascade module includes m layers of average pooling and m-1 layers of maximum pooling that are alternately cascaded in sequence, and the second pooling cascade module includes m layers of maximum pooling and m-1 layers of average pooling that are alternately cascaded in sequence; m is an integer greater than 1; The pooling result of the current pooling layer corresponding to the two pooling cascade modules is summed with the correlation result of the previous pooling layer to obtain the correlation result of the current pooling layer; wherein the feature map obtained by summing the pooling result of the first pooling layer corresponding to the two pooling cascade modules and the second feature map is the correlation result of the first pooling layer; The second feature map is sequentially concatenated with the association results of all pooling layers and processed by convolutional layers to obtain the final output.

4. A lightweight aerial image target detection method based on fine-grained information according to any one of claims 1 to 3, characterized in that: The feature extraction network adopts a feature recombination enhancement network FRENet, and the FRENet is constructed based on the FRE module; wherein the feature processing process of the FRE module includes: The input feature map is extracted channel by channel according to the following formula to obtain two sub-feature maps f1 and f2 of the same size; f1=Concat(c0,c2,…,c n ) f2=Concat(c1,c3,…,c n-1 ) The sub-feature map f1 is processed by a convolution layer, and the sub-feature map f2 is processed by a dilated convolution layer. At the same time, adaptive maximum pooling is used to extract the salient features of the sub-feature maps f1 and f2 channel by channel respectively; The outputs of the convolution layer and the dilated convolution are concatenated to obtain the third feature map. Using the significant features of one of the sub-feature maps to modulate another sub-feature map, and performing a splicing operation on the modulation results to obtain a fourth feature map; The third feature map and the fourth feature map are summed up and processed by the convolution layer in sequence to obtain the final output.

5. The lightweight aerial image target detection method based on fine-grained information according to claim 4 is characterized in that: The feature fusion process and detection process of the constructed FRENet include: The feature maps of each scale output by the feature extraction module are processed using the FRE module respectively; For feature maps of each scale other than the feature map with the largest scale output by the FRE module, feature enhancement processing is performed using the feature enhancement module respectively, so as to sum the feature maps processed by all feature enhancement modules with the feature map with the largest scale output by the FRE module to obtain the first-level fused feature map; The feature enhancement module is used to perform feature enhancement processing on the l-th level fused feature map, and then the obtained feature map is concatenated with the feature map output by the FRE module of the adjacent layer to obtain the l+1-th level fused feature map; where l is an integer not less than 1; The feature maps obtained after the feature enhancement processing of the fused feature maps at all levels by the feature enhancement module are first processed by the convolution layer, and then detected by the detection head to obtain the detection results.

6. The lightweight aerial image target detection method based on fine-grained information according to claim 5 is characterized in that: The feature enhancement module adopts the ELAN-R module; the feature enhancement processing process of the ELAN-R module includes: First, use two branches to process the input feature map separately, then splice the outputs of the two branches and process them through a 1×1 convolution layer; then use a residual connection to sum the input feature map and the output of the 1×1 convolution layer, and then process it through a 1×1 convolution layer to get the final output; in one branch, the input feature map is processed by a 1×1 convolution layer; in the other branch, the input feature map is first processed by a 1×1 convolution layer and two 3×3 convolution layers in sequence, and then the output of the 1×1 convolution layer is spliced ​​with the output of the last 3×3 convolution layer using a residual connection.

7. A lightweight aerial image target detection system based on fine-grained information, characterized in that: include: A network construction module is used to construct an aerial image target detection network; the aerial image target detection network includes a feature extraction network and a feature fusion network; wherein the feature extraction network extracts a multi-scale feature map of the input aerial image, and the feature fusion network fuses the extracted multi-scale feature map and outputs a target detection result; A training module, used for training the aerial image target detection network to obtain an aerial image target detection model; A detection module, used to input the aerial image to be detected into the aerial image target detection model to obtain a detection result; Among them, the feature extraction network is constructed based on the multifunctional convolution module Mconv, and the feature map processing process of Mconv includes first using two branches to process the input feature map respectively, and then multiplying the outputs of the two branches element by element to obtain the final output; wherein, in one branch, the input feature map is split into two sub-feature maps of the same size according to the channel; one of the sub-feature maps is processed by using the hole convolution to obtain a feature map with an increased receptive field; the other sub-feature map is spliced ​​with the obtained feature map with an increased receptive field to obtain the output of the branch; in another branch, the adaptive maximum pooling is used to extract the significant features of the input feature map channel by channel to obtain the output of the branch.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images