Lightweight remote sensing target detection method and related equipment

Abstract

The invention provides a lightweight remote sensing target detection method and related equipment, and relates to the technical field of remote sensing image target detection. The method comprises the steps of obtaining a to-be-measured remote sensing image; inputting the to-be-detected remote sensing image into a preset remote sensing target detection model, and outputting a detection result; the detection result comprises a target type and a corresponding confidence coefficient; the remote sensing target detection model comprises a backbone network, a neck network and a detection head; wherein the backbone network is used for extracting multi-level features of a to-be-detected remote sensing image from a spatial domain and a frequency domain; the neck network is used for enhancing the relation among the multi-level features output by the backbone network so as to fuse the semantic information of the high-level features and the detail information of the low-level features and generate multi-scale fusion features; and the detection head is used for performing target detection on each scale fusion feature. According to the method, the light weight of the model is kept, and meanwhile, the detection precision higher than that of other mainstream target detection algorithms is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image target detection, and particularly to a lightweight remote sensing target detection method and related devices. Background Art

[0002] When processing remote sensing data, due to the scale variation of targets, complex and diverse backgrounds, and dense distribution of target objects, these factors increase the difficulty of remote sensing image target detection. There are significant scale differences in the targets in remote sensing images, especially between small targets (such as vehicles and buildings) and large targets (such as airports and lakes), which makes it easy for existing methods to have false detections or missed detections when dealing with targets of different scales in images, thereby affecting the overall accuracy of detection.

[0003] Feature extraction is a key link in target detection. By downsampling to expand the receptive field and capture more global information, the target can be more accurately identified and located. For example, researchers such as Chun Liu and Yao Teng use pooling operations to reduce the resolution of the feature map, reduce the amount of data, and extract higher-level semantic information. Gong Chen and Xin Xie et al. use cross-row convolution, which not only expands the horizontal and vertical ranges of the receptive field but also captures the structural information in the image while reducing the size of the feature map. Although existing downsampling methods can expand the receptive field and aggregate features, they may cause some key information to be ignored or lost.

[0004] By capturing the global semantic information in remote sensing images, the overall layout and environmental background of the target can be better understood. For example, Yin Zhang et al. proposed a (FEM) feature enhancement module, which increases the perception range by using a multi-branch convolution structure and enhances the global semantic information. Shilong Zhou et al. introduced the DualC2f structure, which broadens the perception domain through double convolution and group convolution techniques, thereby improving the feature extraction ability. The convolution structure has a local perception range and has certain limitations in extracting global features, which restricts its ability to obtain overall semantic information.

[0005] In addition, fusing semantic information from different sources can better capture the details and overall layout in remote sensing images. Zhang Jiaqi et al. proposed a multi-level semantic fusion network, which combines the shallow edge detail information with the deep global semantic information, significantly improving the performance of the model in multi-scale scenarios. Li Can et al. designed a context-aware multi-scale feature fusion method, which can dynamically adjust the feature weights at different semantic levels to enhance the adaptability of the model when dealing with complex backgrounds and multi-scale targets. Existing methods have improved the overall accuracy of remote sensing image target detection, but they ignore the connection between different levels of semantics and lack an effective mechanism to coordinate the interaction between semantics.

[0006] In the field of remote sensing image object detection, a large number of studies in recent years have focused on how to improve the performance of algorithms in complex backgrounds and small object detection. However, current remote sensing image object detection algorithms still have drawbacks such as loss of downsampled information, insufficient ability to capture context information, and inadequate feature fusion. The self-attention mechanism has been widely used in the field of remote sensing image object detection, especially in enhancing the performance of small object detection. Researchers have significantly improved the robustness and accuracy of the model by introducing the self-attention mechanism and multi-scale feature fusion technology. However, the use of the self-attention mechanism also brings a huge computational cost. Summary of the Invention

[0007] In order to reduce information loss and improve the ability to extract and fuse feature information, the present invention provides a lightweight remote sensing object detection method and related devices.

[0008] In a first aspect, the present invention provides a lightweight remote sensing object detection method, including:

[0009] Obtain a remote sensing image to be measured;

[0010] Input the remote sensing image to be measured into a preset remote sensing object detection model to output a detection result; the detection result includes the target type and the corresponding confidence; the remote sensing object detection model includes a backbone network, a neck network, and a detection head; wherein, the backbone network is used to extract multi-level features of the remote sensing image to be measured from the spatial domain and the frequency domain respectively; the neck network is used to strengthen the connection between the multi-level features output by the backbone network to fuse the semantic information of the high-level features and the detail information of the low-level features, and generate multi-scale fusion features; the detection head is used to perform object detection on each scale of the fusion features.

[0011] Further, a low-frequency enhancement downsampling module is used in the backbone network and the neck network to downsample the input feature map; wherein, the feature processing process of the low-frequency enhancement downsampling module includes:

[0012] Decompose the input feature map into a low-frequency part, a diagonal part, a vertical part, and a high-frequency part by using the Haar wavelet transform;

[0013] Enhance the low-frequency part by using a feature transformation operation and re-stitch the enhanced result with the diagonal part and the vertical part to obtain an enhanced feature map.

[0014] Further, a spatial information interaction module is used in the neck network to interact and fuse multi-level features; wherein, the feature processing process of the spatial information interaction module includes:

[0015] The input feature map is subjected to channel expansion using a convolution operation to obtain a first feature map, a second feature map, a third feature map, and a fourth feature map;

[0016] The first feature map and the second feature map are concatenated to obtain a first intermediate feature map; the third feature map and the fourth feature map are concatenated to obtain a second intermediate feature map;

[0017] The global semantic information of the first feature map is extracted using wavelet transform and self-attention mechanism to obtain a fifth feature map; the second feature map and the fifth feature map are multiplied pointwise, feature extraction is performed on the pointwise multiplication result, and the feature extraction result is concatenated with the fifth feature map to obtain a third intermediate feature map;

[0018] The first intermediate feature map is enhanced using a feature transformation operation and the enhancement result is added to the third intermediate feature map to obtain a fourth intermediate feature map;

[0019] The second intermediate feature map and the fourth intermediate feature map are added to obtain a fifth intermediate feature map;

[0020] The fifth intermediate feature map is enhanced using a feature transformation operation and the enhancement result is added to the second intermediate feature map to obtain a sixth intermediate feature map;

[0021] The fourth intermediate feature map and the sixth intermediate feature map are concatenated to obtain an output feature map.

[0022] Further, the process of enhancing the input feature map by the feature transformation operation specifically includes:

[0023] The input feature map is segmented into multiple sub-feature maps by channel splitting, feature displacement operations are performed on the multiple sub-feature maps in different directions, and all the feature maps generated after the feature displacement are concatenated with the input feature map.

[0024] Further, the extraction of the global semantic information of the first feature map using wavelet transform and self-attention mechanism specifically includes:

[0025] The first feature map is decomposed into a low-frequency part, a diagonal part, a vertical part, and a high-frequency part using Haar wavelet transform;

[0026] Generate a distance attenuation matrix corresponding to the low-frequency part of the first feature map, and use the self-attention mechanism to process the low-frequency part of the first feature map. Introduce the distance attenuation matrix in the self-attention mechanism to balance local features and global features. Among them, the distance attenuation matrix corresponding to the low-frequency part of the first feature map is generated according to the following process: generate a patch sequence of the low-frequency part of the first feature map, calculate the distance between any two patches in the patch sequence, and weight each distance using an attenuation factor. All the calculated weighted values constitute the distance attenuation matrix. The attenuation factor is determined according to the number of input channels, the number of attention heads, and the corresponding patch number. The closer the distance, the larger the corresponding attenuation factor.

[0027] Further, introduce the distance attenuation matrix in the self-attention mechanism according to the following formula to balance local features and global features:

[0028] f MASA (X) = (Softmax(QK T ) ⊙ D)V

[0029] where, f MASA (X) represents the combination of the distance attenuation matrix and the self-attention mechanism, X represents the patch sequence of the low-frequency part of the first feature map, Q, K, and V represent the query matrix, key matrix, and value matrix generated by linearly transforming X, and D represents the distance attenuation matrix.

[0030] Further, perform feature extraction on the dot product result using the GB operation. The GB refers to the GSConv Bottleneck module.

[0031] In a second aspect, the present invention provides a lightweight remote sensing target detection device, including:

[0032] An acquisition module for acquiring a remote sensing image to be measured;

[0033] A detection module for inputting the remote sensing image to be measured into a preset remote sensing target detection model and outputting a detection result. The detection result includes the target type and the corresponding confidence. The remote sensing target detection model includes a backbone network, a neck network, and a detection head. Among them, the backbone network is used to extract multi-level features of the remote sensing image to be measured from the spatial domain and the frequency domain respectively. The neck network is used to strengthen the connection between the multi-level features output by the backbone network to fuse the semantic information of the high-level features and the detail information of the low-level features, and generate multi-scale fusion features. The detection head is used to perform target detection on each scale of fusion features.

[0034] In a third aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the program, the method described in the first aspect is implemented.

[0035] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0036] The beneficial effects of the present invention are as follows:

[0037] (1) Starting from the perspective of information extraction and fusion, the present invention extracts the features of the target object in the image from both the spatial domain and the frequency domain in the backbone network, alleviating the information loss of the feature map during the transmission in the neural network. In the neck network, by adopting a method of enhancing the connection between different levels of semantic features to coordinate the interaction between semantics, the effective extraction and fusion of high-level abstract semantics and low-level detailed information are ensured.

[0038] (2) The present invention designs a low-frequency enhancement downsampling module (LEM), and uses LEM as the downsampling module in the backbone network and the neck network. This module decomposes the feature map through Haar wavelet transform, performs image downsampling in the frequency domain, and enhances the low-frequency part through feature displacement operation, so that the model can retain more image information during the downsampling process, reduce noise interference, and improve the robustness of the model in complex backgrounds. It solves the problem of information loss caused by using pooling and strided convolution for downsampling in traditional networks; optimizes the information retention ability of the model and makes the model more lightweight.

[0039] (3) The present invention designs a spatial information interaction module (Spatial Information Interaction Module, SIIM), and models the connection between different levels of semantic features by adopting SIIM in the neck network. SIIM uses convolution operation to expand the channels of the input feature map, which can provide a broader processing basis for feature fusion. Features of different scales can be better fused in the expanded channels, improving the target detection ability of the model. By converting the feature maps of some channels to the frequency domain and applying the self-attention mechanism, global features can be efficiently extracted. This method not only enhances the expression ability of the features, but also improves the detection accuracy of the target in complex backgrounds.

[0040] (4) The present invention designs a Haar Frequency Domain Self-Attention (HFDSA). HFDSA uses the characteristics of the Haar wavelet transform to reduce the size of the feature map, greatly reducing the computational resource occupancy of the self-attention mechanism and reducing the model's computational volume and parameter quantity. Secondly, a distance attenuation matrix is introduced to introduce an explicit spatial prior for the feature map, balance global information and local details, better extract global semantic information, and alleviate the problems of global semantic extraction and long-distance dependence in remote sensing image object detection tasks. Description of the Drawings

[0041] Figure 1 It is a schematic flow chart of a lightweight remote sensing object detection method provided by an embodiment of the present invention;

[0042] Figure 2 It is the specific structure of the Low-Frequency Enhancement Downsampling Module (LEM) provided by an embodiment of the present invention;

[0043] Figure 3 It is the specific structure of the Feature Shift FS provided by an embodiment of the present invention;

[0044] Figure 4 It is the specific structure of the Spatial Information Interaction Module (SIIM) provided by an embodiment of the present invention;

[0045] Figure 5 It is the specific structure of the Haar Frequency Domain Self-Attention Mechanism (HFDSA) provided by an embodiment of the present invention;

[0046] Figure 6 It is the specific structure of SL-YOLO provided by an embodiment of the present invention;

[0047] Figures 7-1 to 7-4 It is the experimental result diagram provided by an embodiment of the present invention: (a) is the original image, (b) is the detection result of YOLOV8, (c) is the detection result of FFCA-YOLO, and (d) is the detection result of the present invention;

[0048] Figure 8 It is a schematic structural diagram of a lightweight remote sensing object detection device provided by an embodiment of the present invention;

[0049] Figure 9 It is a structural block diagram of an electronic device provided by an embodiment of the present invention. Detailed Embodiments

[0050] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the protection scope of the present invention.

[0051] The scales of objects in remote sensing images usually vary greatly. When identifying these objects, problems such as loss of downsampling information, insufficient extraction of context information, and insufficient feature fusion are faced. To solve these problems, the present invention proposes a lightweight remote sensing object detection method, device, electronic device, and storage medium.

[0052] As Figure 1 shown, the embodiments of the present invention provide a lightweight remote sensing object detection method, including:

[0053] S101: Obtain a remote sensing image to be measured;

[0054] Specifically, the remote sensing image to be measured refers to a remote sensing image for which object detection is required. The remote sensing images in this embodiment may be satellite remote sensing images, images taken by unmanned aerial vehicles, aerial images taken by airplanes, or images obtained by other platforms using remote sensing technology. It can be obtained from a user terminal and an image processing device, or directly obtain a remote sensing image input by the user.

[0055] S102: Input the remote sensing image to be measured into a preset remote sensing object detection model and output a detection result; the detection result includes the object type and the corresponding confidence level; the remote sensing object detection model includes a backbone network, a neck network, and a detection head; wherein, the backbone network is used to extract multi-level features of the remote sensing image to be measured from the spatial domain and the frequency domain respectively; the neck network is used to strengthen the connection between the multi-level features output by the backbone network to fuse the semantic information of the high-level features and the detail information of the low-level features, and generate multi-scale fusion features; the detection head is used to perform object detection on each scale fusion feature.

[0056] Specifically, the remote sensing object detection model can use the YOLO series of networks (including YOLOv8 and YOLOv10) as the benchmark framework, and then improve the selected YOLO network. Generally, there are many objects in remote sensing images. The traditional YOLO network generally only extracts multi-level features of the input remote sensing image in the spatial domain. The feature representation information of many objects is insufficient. Especially, some small objects may lose more information during the forward propagation of the neural network. To solve this problem, the object detection model proposed in this embodiment improves the traditional YOLO network. On the one hand, it extracts the features of the objects in the image in the spatial domain in the backbone network. On the other hand, it transforms the image from the spatial domain to the frequency domain and further extracts the features of the objects in the image in the frequency domain, thereby reducing information loss. After extracting multi-level features, better information fusion is required when identifying these objects. In particular, it is necessary to better fuse the global semantic information. Therefore, in this embodiment, the neck network strengthens the connection between the multi-level features output by the backbone network to fuse the semantic information of the high-level features and the detailed information of the low-level features.

[0057] In one embodiment, in order to extract the features of the objects in the image in the frequency domain, as Figure 2 shown, this embodiment proposes a low-frequency enhancement downsampling module (LEM) based on Haar wavelet transform in the backbone network to replace the traditional downsampling module, which can enable the neural network to effectively retain the key information in the remote sensing image, thereby reducing information loss during the downsampling process.

[0058] Specifically, in order to detect objects of different scales in the image, it is necessary to rely on downsampling operations to expand the receptive field of the image. However, currently, the downsampling modules of most networks mainly rely on transposed convolution or pooling to achieve. Although transposed convolution and pooling perform well in reducing the size of the feature map, during their operation, they will cause the feature map to lose some key information. For remote sensing images, due to the significant differences in the scales of objects, this information loss problem is particularly serious. Whether it is small objects or large objects, they may lose key features during the downsampling process, thus affecting their accurate representation and analysis. Small objects may become more blurred during the downsampling process due to the small number of pixels, while large objects may lose their edge details and internal structures. Compared with ordinary optical images, the low-frequency part of the feature map of remote sensing images after Haar transform processing is more important because it reflects the overall shape and layout of the ground objects and contains key feature information. Therefore, it is very important to retain and enhance the low-frequency information. Therefore, the embodiment of the present invention designs a low-frequency enhancement downsampling module based on Haar wavelet transform.

[0059] In an exemplary embodiment, the input feature map X is transformed using Haar wavelet transform InputDecomposed into low-frequency X LL , diagonal X LH , vertical X HL and high-frequency X HH into four parts, and the specific operations are as follows:

[0060] Haar(X Input ) = X LL , X LH , X HL , X HH

[0061] The low-frequency part X LL contains the general outline and main information of the image; the vertical part X HL contains the detailed information of the image in the vertical direction; the diagonal part X LH contains the detailed information of the image in the horizontal direction; the high-frequency part X HH contains the detailed information of the image in the diagonal direction, usually corresponding to the noise and texture details of the image.

[0062] It can be understood that the low-frequency part carries the main information of the feature map. A rich low-frequency information feature representation can effectively reduce the interference of high-frequency noise on target detection, ensure that the model focuses on key information, and at the same time enhance the feature representation of the target object, reducing the information loss of the target object information during the transmission process in the neural network. The high-frequency part usually carries noise, and in this embodiment, it is directly discarded, which has multiple benefits. Discarding high-frequency noise can reduce interference, improve the purity of the feature map, and make the model more stable and accurate during training and inference. Secondly, this approach can effectively highlight the low-frequency key information in the image, strengthen the main structure and important features of the remote sensing image, so as to better identify and analyze the target. In addition, reducing the high-frequency data volume can also improve the calculation efficiency and speed up the processing speed.

[0063] Next, the low-frequency enhancement downsampling module processes the remaining three parts as follows: using a feature transformation operation to enhance the low-frequency part and splicing the enhanced result with the diagonal part and the vertical part to obtain an enhanced feature map.

[0064] In one embodiment, as Figure 3 shown, this embodiment provides a feature transformation operation, which specifically includes: first, evenly dividing the input feature map into multiple sub-feature maps through a channel splitting operation, then performing feature displacement operations on the multiple sub-feature maps in different directions, and splicing all the feature maps generated after the feature displacement with the input feature map to obtain an output feature map.

[0065] Based on the above feature transformation operation, the enhancement processing process of the low-frequency part of the remote sensing image is specifically as follows:

[0066] X s1 , X s2 , X s3 , X s4 , X s5 , X s6 , X s7 , X s8 = f Split (X LL )

[0067] X Shift = f Shift (X si )(i = 1, 2, 3, 4, 5, 6, 7, 8)

[0068] X LL + δ × X Shift = X ELL

[0069] Specifically, through the channel splitting operation f Split divide X LL into X si(i＝1,2,3,4,5,6,7,8) on average, and then perform the feature displacement operation f si on X Shift in 8 directions (up, down, left, right, lower right, upper right, upper left, lower left) to obtain X Shift . δ is a hyperparameter, and X LL and X Shift finally obtain the enhanced feature map X ELL through the addition operation. It should be noted that in this embodiment, the direction of feature displacement is not specifically limited, and 4, 8 or more directions can be selected according to needs.

[0070] In the remote sensing target detection method provided in this embodiment, the low-frequency part is enhanced through the feature displacement operation without increasing the number of model parameters. Specifically, since remote sensing images usually involve a large range of surface information, displacements in different directions can capture different spatial information, thus representing the features of terrain, vegetation, buildings and other ground objects more comprehensively. At the same time, it can also enhance the model's ability to capture fine spatial features. Low-frequency enhancement in the remote sensing image target detection task enables the model to capture image information more accurately; reduces noise interference and improves the purity of the feature map; at the same time, by more effectively retaining the structure and features of the image, it enhances the detection ability for small and large targets.

[0071] Correspondingly, the specific operation of LEM can be expressed by the following formula:

[0072] LEM(X Input ) = Conv 1×1 (Concat(X HL , X LH, X ELL ))

[0073] wherein, X LH , X HL respectively represent diagonal and vertical features. The above formula means to concatenate X HL , X LH , X ELL along the channel dimension, and then perform a convolution operation to obtain an output feature map.

[0074] In one embodiment, in order to better fuse the semantic information of high-level features and the detailed information of low-level features, as Figure 4 shown, the embodiment of the present invention provides a Spatial Information Interaction Module (SIIM).

[0075] Specifically, in remote sensing image target recognition, it is crucial to efficiently fuse and interact with semantic information at different levels. High-level features extract abstract semantic information, while low-level features capture details such as edge textures. However, current deep learning models often ignore the connections between these hierarchical features during the feature extraction and fusion process, resulting in feature mismatches. Especially when dealing with complex backgrounds and targets with significant scale differences, this kind of neglect may lead to feature conflicts or information loss, thereby reducing the detection accuracy. To solve this problem, this embodiment proposes a Spatial Information Interaction Module.

[0076] In an exemplary embodiment, the feature processing process of the Spatial Information Interaction Module includes: using a convolution operation to perform channel expansion on the input feature map to obtain a first feature map E1, a second feature map E2, a third feature map E3, and a fourth feature map E4; concatenating the first feature map E1 and the second feature map E2 to obtain a first intermediate feature map I1; concatenating the third feature map E3 and the fourth feature map E4 to obtain a second intermediate feature map I2; using wavelet transform and self-attention mechanism to extract global semantic information from the first feature map E1 to obtain a fifth feature map E5; multiplying the second feature map E2 and the fifth feature map E5, performing feature extraction on the multiplication result, and concatenating the feature extraction result with the fifth feature map E5 to obtain a third intermediate feature map I3; using a feature transformation operation to enhance the first intermediate feature map I1 and adding the enhanced result to the third intermediate feature map I3 to obtain a fourth intermediate feature map I4; adding the second intermediate feature map I2 and the fourth intermediate feature map I4 to obtain a fifth intermediate feature map I5; using a feature transformation operation to enhance the fifth intermediate feature map I5 and adding the enhanced result to the second intermediate feature map I2 to obtain a sixth intermediate feature map I6; concatenating the fourth intermediate feature map I4 and the sixth intermediate feature map I6 to obtain an output feature map.

[0077] In this embodiment, by strengthening the connection between features at different levels, the effective fusion of high-level semantics and low-level details is ensured, thereby improving the model's accurate recognition ability of the target. SIIM first uses a convolution operation on the input feature map X Input to perform channel expansion, which can provide a broader processing basis for feature fusion. Features of different scales can be better fused in the expanded channels, improving the model's detection ability for the target. In SIIM, only global semantic information is extracted from E1, and then E5 is obtained, which greatly reduces the computational cost. By converting the feature maps of some channels to the frequency domain and applying the self-attention mechanism, global features can be efficiently extracted. This method not only enhances the expressive ability of the features but also improves the detection accuracy of targets in complex backgrounds. Then, a dot product operation is performed between E5 and E2, which can preserve and enhance local detail information and is very beneficial for target detection. Secondly, the dot product operation can strengthen specific features, helping the model to more accurately detect and identify the target object in the presence of background interference.

[0078] In an exemplary embodiment, a feature transformation (FS) operation as shown in Figure 3 is used to process the first intermediate feature map I1 (I1 = Concat(E1, E2)). By performing displacements in eight different directions, the subtle changes and position information, as well as texture information, of the feature map in these directions can be captured. This multi-directional displacement operation can enrich the spatial representation of the features, enabling the model to more comprehensively understand the information in the image. Finally, the feature map processed by the FS operation is added to I3 to obtain the high-level semantic information feature I4, and the expression is I4 = Conv 1×1 (FS(I1)) + I3. The FS operation is also used for the fifth intermediate feature map I5, so that the extraction of the texture information of the feature map is achieved without significantly increasing the number of parameters, obtaining the feature map I6, and finally the output feature map X is obtained through a concatenation operation Output .

[0079] In the above remote sensing target detection method, SIIM mines the connection between different semantic information by interacting different levels of semantic information, effectively fuses low-level details and high-level semantic information, enhances the model's detection ability for targets of different scales, improves the detection accuracy and robustness, and at the same time improves the gradient flow, making the training process more stable and improving the generalization ability of the model.

[0080] In one embodiment, as shown in Figure 5 , this embodiment provides a lightweight Haar frequency domain self-attention mechanism HFDSA to extract global semantic information from the first feature map E1.

[0081] Specifically, remote sensing images usually have high resolution and complex scenes, containing rich details and diverse target types. The self-attention mechanism has a powerful ability to extract global semantic information, which is particularly important in remote sensing image object detection. When traditional convolutional neural networks (CNNs) process these images, it may be difficult to capture global information due to the limitation of the local receptive field, thus affecting the detection effect. However, at the same time, self-attention also brings a huge consumption of computing resources. This is because the self-attention mechanism needs to calculate the relationship between each position in the input sequence and all other positions. When processing high-resolution remote sensing images, the length of the input sequence is often very large, which significantly increases the computational cost and memory consumption. In addition, the self-attention mechanism does not consider the spatial distance between input sequences, which means that the model pays equal attention to nearby and distant pixels, thus ignoring the actual spatial structure in the image. To address the above problems, this embodiment designs the Haar frequency domain self-attention mechanism HFDSA. The Haar frequency domain self-attention mechanism HFDSA uses the characteristics of the Haar wavelet transform to greatly reduce the computing resource consumption of the self-attention mechanism. Secondly, a distance decay matrix is introduced to introduce an explicit spatial prior for the feature map. Specifically, it includes: decomposing the first feature map into a low-frequency part, a diagonal part, a vertical part, and a high-frequency part by using the Haar wavelet transform; generating a distance decay matrix corresponding to the low-frequency part of the first feature map, and using the self-attention mechanism to process the low-frequency part of the first feature map, and introducing the distance decay matrix into the self-attention mechanism to balance local features and global features; wherein, the distance decay matrix corresponding to the low-frequency part of the first feature map is generated according to the following process: generating a patch sequence of the low-frequency part of the first feature map, calculating the distance between any two patches in the patch sequence and weighting each distance with an attenuation factor, and all the calculated weighted values constitute the distance decay matrix; the attenuation factor is determined according to the number of input channels, the number of attention heads, and the corresponding patch number, and the closer the distance, the larger the corresponding attenuation factor.

[0082] For remote sensing images, macroscopic features such as terrain, vegetation cover, and urban structure are very important. The Haar wavelet processes the input image through filtering and divides the input image into four parts: a low-frequency part, a horizontal part, a vertical part, and a high-frequency part, where the low-frequency part retains the main structural information of the remote sensing image, including the shape and position of ground objects.

[0083] X LL = HaarL(X input )

[0084] The above formula represents performing the Haar wavelet transform on the input feature X input ∈R C×H×W and outputting the low-frequency part Operations. Applying the self-attention mechanism to the low-frequency part can effectively reduce the computational complexity while retaining the main information of the image. Specifically, first perform the Haar wavelet transform on the input features to obtain the low-frequency part (LL part). The low-frequency part contains the main structural information and low-frequency features of the image, which are crucial for understanding and analyzing the global features of the image. Next, apply the self-attention mechanism to the low-frequency part. The self-attention mechanism can capture the long-range dependencies and global context information between features, further enhancing the richness and accuracy of feature representation.

[0085] In the above remote sensing target detection method, by introducing a distance attenuation matrix, an explicit spatial prior is provided for the feature map, enabling the model to incorporate spatial position information into the attention calculation. Specifically, the distance attenuation matrix weights the distances between different patches, enabling the model to better capture spatial relationships when processing images. The distance attenuation matrix makes the model more focused on local features, and by weakening the influence of distant pixels, it can more effectively extract and utilize local information. In addition, the distance attenuation matrix allows the model to perform different degrees of attenuation on features of different scales for different attention heads. By adjusting the parameters of the distance attenuation matrix, the model can balance between local and global features, thereby better capturing multi-scale information. Specifically, it is represented as follows:

[0086] f MASA (X) = (Softmax(QK T ) ⊙ D)V

[0087]

[0088] Q = XW Q ,K = XW K ,V = XW V

[0089] Each patch in the image has a unique coordinate representation (x, y), D is the distance attenuation matrix generated based on each patch; λ is the attenuation factor generated according to the custom initial value j, the number of input channels c of HFDSA, the number of attention heads N, and the patch serial number i; f MASA (X) is the combination of the distance attenuation matrix and the self-attention mechanism. By calculating the relative importance between different positions in the feature map, it emphasizes the correlation of adjacent positions, thereby effectively capturing the spatial relationships and context information of the target, improving the attention to the target area, and enhancing the recognition ability of the target in complex backgrounds.

[0090] Finally, the Haar frequency domain self-attention mechanism HFDSA upsamples the feature map back to the size at the input. The Haar frequency domain self-attention mechanism HFDSA can not only better identify the spatial relationship between targets using low-frequency components, but also significantly reduce the computational resource consumption of the self-attention mechanism.

[0091] In one embodiment, as Figure 4 shown, for the result after dot product, the GB (GSConv Bottleneck) operation is used for feature extraction. GB is a lightweight feature extraction module. By performing a convolution operation on the input features, the number of channels is reduced by half, and depthwise separable convolution (Dwconv) is applied to further extract features. Then, these features are concatenated (Concat) with E5 to obtain I3, forming a rich feature representation. This not only improves the feature expression ability but also effectively reduces the computational cost. The specific operations are shown as follows:

[0092] HFDSA(E1) = E5

[0093] Concat(GB(E2⊙E5), E5) = I3

[0094] To verify the effectiveness of the remote sensing target detection scheme provided by the present invention, the following experimental data are also provided by the present invention.

[0095] (I) Experimental environment

[0096] YOLOv8 is selected as the basic model, and the low-frequency enhancement downsampling module (LEM), Haar frequency domain self-attention mechanism (HFDSA), and spatial information interaction module (SIIM) designed by the present invention are respectively introduced into the backbone network and neck network of YOLOv8. The improved network is denoted as SL-YOLO, as Figure 6 shown. Experiments are carried out on the NWPU-VHR-10 and SAR-Airport-1.0 datasets. All experiments are completed on RTX4060. During the training process, a unified input size of 640×640 is adopted, the training cycle is 300, and the batch size is 4.

[0097] (II) Datasets

[0098] (1) NWPU-VHR-10

[0099] The NWPU-VHR-10 dataset is a benchmark dataset designed specifically for object detection in high-resolution remote sensing images. It consists of 800 images with resolutions ranging from 0.5 to 2 meters, which contain 10 object categories, including airplanes, ships, storage tanks, swimming pools, bridges, vehicles, helicopters, tennis courts, soccer fields, and containers. The objects in each image are accurately labeled with bounding boxes and class information. This dataset is widely used in the field of remote sensing image analysis to support the development and evaluation of automated object detection algorithms, especially suitable for studying object detection tasks in high-resolution environments.

[0100] (2) SAR-Airport-1.0

[0101] The SAR-Airport-1.0 dataset is a high-quality benchmark dataset specifically for object detection in synthetic aperture radar (SAR) images. This dataset contains SAR images of multiple airports, covering different environments and conditions, providing rich object information such as airplanes, aprons, runways, etc. The objects in each image are detailedly labeled, including the bounding boxes and class information of the objects. The diversity and high-resolution characteristics of the SAR-Airport-1.0 dataset make it an ideal dataset for studying object detection and recognition.

[0102] (III) Evaluation Metrics

[0103] In the object detection task of remote sensing images, mAP50 and mAP50-95 are important metrics for evaluating the performance of the model. mAP50 is the mean of the average precisions of all object categories calculated when the intersection over union (IoU) threshold is 0.5. This metric mainly measures the detection effect of the model when the requirement for the overlap degree is relatively low, and is suitable for scenarios where the accuracy requirement is not very strict. While mAP50-95 calculates the average precision at each threshold in the range of IoU thresholds from 0.5 to 0.95 (with a step of 0.05) and takes the mean of these average precisions. mAP50-95 provides a comprehensive performance evaluation of the model under different overlap degree requirements, and more comprehensively reflects the detection ability and robustness of the model. For remote sensing image processing tasks, these metrics can effectively evaluate the performance of the model in complex environments and ensure the accurate identification and localization of objects in high-resolution images. mAP is shown in the following formula, where P is the precision, R is the recall rate, and n represents the number of categories.

[0104]

[0105] (IV) Method Evaluation

[0106] (1) Comparison with Previous Models

[0107] Table 1 Experimental Comparison with Other Models on the NWPU Dataset

[0108]

[0109]

[0110] On the NWPU-VHR-10 dataset, this paper compared the performance of multiple models in terms of the number of parameters, computational complexity (GFLOPs), and object detection performance (mAP50 and mAP50-95). Specifically, as shown in Table 1, the SL-YOLO model has only 2.75M parameters, which is the fewest among all the compared models. At the same time, its computational complexity is also very low, only 7.90 GFLOPs. In terms of the mAP50 metric, SL-YOLO achieved an accuracy of 93.1%, exceeding other models with more parameters and higher computational complexity, such as Faster R-CNN (parameters: 41.17M, GFLOPs: 127.7, mAP50: 77.8%) and MobileNetV2 (parameters: 10.29M, GFLOPs: 71.49, mAP50: 76.9%). In the more challenging mAP50-95 metric, SL-YOLO also achieved excellent performance, reaching 61.0%, leading all models. Compared with classical models such as YOLOv8n and YOLOv8s, SL-YOLO significantly reduces the number of parameters and computational volume while maintaining high accuracy. For example, the YOLOv8n model has 3.00M parameters and a computational complexity of 8.10 GFLOPs, but its mAP50-95 is 59.7%, lower than 61.0% of SL-YOLO. This indicates that SL-YOLO can effectively improve the accuracy while maintaining the lightweight of the model in multi-object detection tasks, with high practicality and application potential.

[0111] Table 2 Experimental comparison with other models on the SAR-Airport-1.0 dataset

[0112]

[0113] Table 2 shows the comparison results with multiple object detection models on SAR-Airport-1.0. Specifically, the SL-YOLO model has only 2.75M parameters, which is the lowest among all models. At the same time, its computational complexity is 7.9 GFLOPs, which is also significantly lower than other models. Despite the lightweight of the model, SL-YOLO reached 99.1% in the mAP50 metric, performing comparably to some models with much higher computational complexity and number of parameters, such as TPH-YOLO (41.51M parameters, 160.6 GFLOPs). In addition, in the mAP50-95 metric, SL-YOLO also achieved a score of 77.3%, exceeding most of the compared models.

[0114] By introducing an efficient feature processing strategy, SL-YOLO still maintains good performance when detecting objects in remote sensing images. Generally speaking, while maintaining a lightweight design, SL-YOLO successfully achieves efficient detection of complex targets. Figures 7-1 to 7-4 The figure shows the schematic diagram of the detection results of the present invention and some existing detection algorithms.

[0115] (2) Ablation experiment

[0116] Table 3 shows the ablation experiment conducted on the NWPU dataset.

[0117]

[0118] To verify the effectiveness of the method proposed in this paper, an ablation experiment was conducted on the NWPU dataset. Table 3 shows the impact of using different modules on the experimental results. The baseline model of this experiment is YOLOv8n. By gradually adding different modules, the impact on the number of model parameters, computational complexity, and mAP50 performance was observed. The number of parameters of the baseline model (YOLOv8n) is 3.00M, the computational complexity is 8.1 GFLOPs, and it achieved a score of 91.6% in the mAP50 metric. When adding LEM to the baseline model, both the number of parameters and computational complexity of the model decreased, to 2.67M and 7.6 GFLOPs respectively, and the mAP50 reached 92.4. When adding SIIM alone, the number of parameters slightly increased to 3.08M, the computational complexity increased to 8.4 GFLOPs, but the mAP50 increased to 92.7%, showing the effectiveness of the SIIM module in improving the detection performance. Finally, the model combining the LEM and SIIM modules performed the best, with the number of parameters being 2.75M, the computational complexity being 7.9 GFLOPs, and the mAP50 reaching 93.1%, maintaining a low computational cost while improving the performance.

[0119] Table 4 Comparison of the number of parameters of the downsampling modules of YOLOv8 and SL-YOLO

[0120]

[0121] In Table 4, this paper compares the number of parameters of the downsampling modules at corresponding positions between YOLOv8 and SL-YOLO. The downsampling module of YOLOv8 mainly uses standard cross-row convolutions, and the numbers of its parameters are 18,560, 295,424, 36,992, and 147,712 in sequence. In contrast, SL-YOLO uses lightweight depthwise separable convolution (LWD) modules, which significantly reduces the number of parameters. The numbers of parameters at the same positions are 6,272, 9,881, 12,416, and 49,408 respectively. This comparison shows that SL-YOLO has successfully reduced the number of parameters of the model by using LEM in network design. This lightweight design not only helps reduce computational complexity but also improves inference efficiency.

[0122] Table 5 Influence of HFDSA on the Number of Parameters

[0123]

[0124] In Table 5, this paper compares the influence of HFDSA on the number of parameters of the SL-YOLO model. The specific data shows that the number of parameters of the SL-YOLO model containing HFDSA is 2.75M, while without using HFDSA, the number of parameters of the model slightly increases to 2.77M.

[0125] The SL-YOLO of the present invention can be used to solve the problem of target scale diversity in remote sensing image target detection tasks. The present invention mainly starts from the perspective of information extraction and fusion, and proposes Haar frequency domain self-attention (HFDSA), spatial information interaction module (SIIM), and low-frequency enhanced downsampling module (LEM). The low-frequency enhanced downsampling module relies on the advantages of Haar wavelet transform to alleviate the information loss of the feature map during the transmission process in the neural network. HFDSA extracts global semantic information with fewer computational resources and provides spatial priors for the model. The spatial information interaction module (SIIM) enables better fusion of features by modeling the connections between semantic information. The performance of SL-YOLO was tested on the public datasets NWPU-VHR-10 and SAR-Airport-1.0. Through comparative experiments, it can be seen that SL-YOLO not only outperforms existing mainstream target detection algorithms in detection accuracy but is also more efficient in terms of computational resources, that is, while maintaining the lightweight of the model, it achieves higher detection accuracy than other mainstream target detection algorithms. The design idea and module innovation of SL-YOLO provide a new solution for remote sensing image target detection tasks.

[0126] As Figure 8 shown, corresponding to the above method, based on the same inventive concept, an embodiment of the present invention also provides a lightweight remote sensing target detection device, including an acquisition module and a detection module.

[0127] The acquisition module is used to acquire the remote sensing image to be measured; the detection module is used to input the remote sensing image to be measured into a preset remote sensing target detection model and output a detection result; the detection result includes the target type and the corresponding confidence level; the remote sensing target detection model includes a backbone network, a neck network, and a detection head; wherein, the backbone network is used to extract multi-level features of the remote sensing image to be measured from the spatial domain and the frequency domain respectively; the neck network is used to strengthen the connection between the multi-level features output by the backbone network to fuse the semantic information of the high-level features and the detailed information of the low-level features, and generate multi-scale fusion features; the detection head is used to perform target detection on each scale of fusion features.

[0128] In the remote sensing target detection device provided by the embodiment of the present invention, on the one hand, the backbone network extracts the features of the target object in the image in the spatial domain, and on the other hand, it converts the image from the spatial domain to the frequency domain and further extracts the features of the target object in the image in the frequency domain, thereby reducing information loss. After extracting multi-level features, better information fusion is required when identifying these target objects, especially better fusion of global semantic information. Therefore, in this embodiment, the neck network strengthens the connection between the multi-level features output by the backbone network to fuse the semantic information of the high-level features and the detailed information of the low-level features.

[0129] It should be noted that the remote sensing target detection device provided by the embodiment of the present invention is to implement the above method, and its functions can be specifically referred to in the above method embodiments, which will not be elaborated here.

[0130] Figure 9 An example of the physical structure diagram of an electronic device is shown in Figure 9 As shown, the electronic device may include: a processor 901, a communication interface 902, a memory 903, and a communication bus 904. Among them, the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904. The processor 901 can call the logical instructions in the memory 903 to execute the remote sensing target detection method, which includes: acquiring the remote sensing image to be measured; inputting the remote sensing image to be measured into a preset remote sensing target detection model and outputting a detection result; the detection result includes the target type and the corresponding confidence level; the remote sensing target detection model includes a backbone network, a neck network, and a detection head; wherein, the backbone network is used to extract multi-level features of the remote sensing image to be measured from the spatial domain and the frequency domain respectively; the neck network is used to strengthen the connection between the multi-level features output by the backbone network to fuse the semantic information of the high-level features and the detailed information of the low-level features, and generate multi-scale fusion features; the detection head is used to perform target detection on each scale of fusion features.

[0131] In addition, when the logical instructions in the above-mentioned memory 903 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0132] The embodiments of the present invention also provide a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the remote sensing target detection method provided in each of the above method embodiments.

[0133] The embodiments of the present invention also provide a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the remote sensing target detection method provided in each of the above method embodiments.

[0134] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disks, optical discs, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lightweight remote sensing target detection method, characterized in that, Including: Obtain the remote sensing image to be measured; Input the remote sensing image to be measured into a preset remote sensing target detection model and output the detection result; The detection result includes the target type and the corresponding confidence level; the remote sensing target detection model includes a backbone network, a neck network, and a detection head; wherein, the backbone network is used to extract multi-level features of the remote sensing image to be measured from the spatial domain and the frequency domain respectively; the neck network is used to strengthen the connection between the multi-level features output by the backbone network to fuse the semantic information of the high-level features and the detail information of the low-level features, and generate multi-scale fusion features; the detection head is used to perform target detection on each scale fusion feature.

2. The lightweight remote sensing target detection method according to claim 1, characterized in that, Use a low-frequency enhancement downsampling module in the backbone network and the neck network to downsample the input feature map; wherein, the feature processing process of the low-frequency enhancement downsampling module includes: Decompose the input feature map into a low-frequency part, a diagonal part, a vertical part, and a high-frequency part by using Haar wavelet transform; Enhance the low-frequency part by using a feature transformation operation and splice the enhanced result with the diagonal part and the vertical part to obtain an enhanced feature map.

3. A lightweight remote sensing target detection method according to claim 1, characterized in that, Use a spatial information interaction module in the neck network to interact and fuse multi-level features; wherein, the feature processing process of the spatial information interaction module includes: Use a convolution operation to expand the channels of the input feature map to obtain a first feature map, a second feature map, a third feature map, and a fourth feature map; Splice the first feature map and the second feature map to obtain a first intermediate feature map; splice the third feature map and the fourth feature map to obtain a second intermediate feature map; Extract global semantic information from the first feature map by using wavelet transform and self-attention mechanism to obtain a fifth feature map; perform dot multiplication on the second feature map and the fifth feature map, perform feature extraction on the dot multiplication result, and splice the feature extraction result with the fifth feature map to obtain a third intermediate feature map; Enhance the first intermediate feature map by using a feature transformation operation and add the enhanced result to the third intermediate feature map to obtain a fourth intermediate feature map; Add the second intermediate feature map to the fourth intermediate feature map to obtain a fifth intermediate feature map; Enhance the fifth intermediate feature map by using a feature transformation operation and add the enhanced result to the second intermediate feature map to obtain a sixth intermediate feature map; Splice the fourth intermediate feature map and the sixth intermediate feature map to obtain an output feature map.

4. A lightweight remote sensing target detection method according to claim 2 or 3, characterized in that The process of enhancing the input feature map by the feature transformation operation specifically includes: Split the input feature map into multiple sub-feature maps by channel splitting, perform feature displacement operations on the multiple sub-feature maps in different directions, and splice all the feature maps generated after feature displacement with the input feature map.

5. A lightweight remote sensing target detection method according to claim 3, characterized in that Extracting global semantic information from the first feature map by using wavelet transform and self-attention mechanism specifically includes: Decompose the first feature map into a low-frequency part, a diagonal part, a vertical part, and a high-frequency part by using Haar wavelet transform; Generate a distance attenuation matrix corresponding to the low-frequency part of the first feature map, and use the self-attention mechanism to process the low-frequency part of the first feature map. Introduce the distance attenuation matrix in the self-attention mechanism to balance local features and global features. Among them, the distance attenuation matrix corresponding to the low-frequency part of the first feature map is generated according to the following process: generate a patch sequence of the low-frequency part of the first feature map, calculate the distance between any two patches in the patch sequence, and weight each distance using an attenuation factor. All the calculated weighted values constitute the distance attenuation matrix. The attenuation factor is determined according to the number of input channels, the number of attention heads, and the corresponding patch number. The closer the distance, the larger the corresponding attenuation factor.

6. A lightweight remote sensing target detection method according to claim 5, characterized in that Introduce the distance attenuation matrix in the self-attention mechanism according to the following formula to balance local features and global features: f MASA (X) = (Softmax(QK T )) ⊙ D)V Among them, f MASA (X) represents the combination of the distance attenuation matrix and the self-attention mechanism, X represents the patch sequence of the low-frequency part of the first feature map, Q, K, and V represent the query matrix, key matrix, and value matrix generated by linearly transforming X, and D represents the distance attenuation matrix.

7. A lightweight remote sensing target detection method according to claim 3, characterized in that Use the GB operation to extract features from the dot product result. The GB refers to the GSConv Bottleneck module.

8. A lightweight remote sensing target detection device, characterized in that, Include: An acquisition module for acquiring a remote sensing image to be measured. A detection module for inputting the remote sensing image to be measured into a preset remote sensing target detection model and outputting a detection result. The detection result includes the target type and the corresponding confidence level. The remote sensing target detection model includes a backbone network, a neck network, and a detection head. Among them, the backbone network is used to extract multi-level features of the remote sensing image to be measured from the spatial domain and the frequency domain respectively. The neck network is used to strengthen the connection between the multi-level features output by the backbone network to fuse the semantic information of the high-level features and the detailed information of the low-level features, and generate multi-scale fusion features. The detection head is used to perform target detection on each scale of the fusion features.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1 to 7.

Images