Remote sensing image target detection method based on wavelet convolutional sequence network

Abstract

This invention provides a remote sensing image target detection method based on wavelet convolutional sequence network. Wavelet transform is applied to the backbone network of the target detection model of remote sensing image. By leveraging the ability of wavelet transform to extract features of different frequencies, different dimensional attribute features are extracted for the same target. Wavelet convolutional sequence module and convolutional sequence fusion module are used to reduce the computational load.

Description

Technical Field

[0001] This invention belongs to the technical fields of information processing and remote sensing image target detection, specifically relating to a remote sensing image target detection method based on wavelet convolutional sequence networks, which can be applied to fields such as agricultural water and forest resource management, urban planning management, and environmental and marine monitoring. Background Technology

[0002] Remote sensing image target detection, a key technology in remote sensing image applications, focuses on accurately identifying and locating specific objects within images. This is crucial not only for enhancing the application value of remote sensing data but also for effectively supporting decision-making in areas such as resource management, environmental monitoring, and disaster response, thereby providing vital support for sustainable development and environmental security.

[0003] Chinese patent "A Remote Sensing Image Target Detection Method Based on Fusion Convolutional Attention Mechanism" (CN116229295A) addresses the problems of excessively long training time due to the difficulty in convergence of the attention mechanism in the DETR-based Transformer remote sensing image end-to-end target detection framework, and the low accuracy of the model for small targets due to the inability of the attention mechanism to effectively acquire local information. It proposes a remote sensing image target detection method based on a fusion convolutional attention mechanism, which has stronger capabilities in acquiring local image features and faster model convergence speed. However, it is still a Transformer-based target detection model, and the number of parameters and computational complexity of the model makes it unsuitable for target detection requirements in remote sensing images in scenarios with high real-time demands. Xie et al., in their paper "Xingxing Xie, Gong Cheng, Jiabao Wang, Xiwen Yao, Junwei Han, 'Oriented R-CNN for Object Detection,'"

[0004] The International Conference on Computer Vision (ICCV), 2021, pp. 3520-3529, updated the regression method for rotated bounding boxes. It changed the original 4-point, 8-coordinate regression method to a 6-value method, adding 2 values ​​for the midpoint coordinate offset to the 4 bounding box values. This reduced the computational cost of bounding box representation, freeing up computational resources previously reliant on placing numerous rotated anchor points, and improving computational efficiency when regressing bounding boxes. However, it still doesn't solve the problem of extracting features from small targets in remote sensing images, making it difficult to meet the target detection requirements of remote sensing images in scenarios with high accuracy requirements. Feng et al., in their paper "Mingkui Feng, Hancheng Yu, Xiaoyu Dang and Ming Zhou. 'Category-Aware Dynamic Label Assignment with High-Quality Oriented Proposal.' ArXiv...", addressed this issue. The method in abs / 2407.03205(2024):n.pag. introduces the angular representation of the rotated bounding box into the complex plane, reducing the possibility of discontinuities in the label regression values ​​at the boundary due to the periodicity of the angle, and avoiding sudden fluctuations in the loss function. However, this method does not solve the problem of decreased target detection accuracy caused by interference information such as occlusion and blurring, and is difficult to meet the target detection requirements of remote sensing images in scenarios with high anti-interference capabilities. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings and deficiencies of existing solutions by proposing a remote sensing image target detection method based on a wavelet convolutional sequence network. This method applies wavelet transform to the backbone network of a target detector using remote sensing images, primarily solving the problems of low detection accuracy and slow detection speed caused by target interference such as occlusion and noise in complex environments. The detection implementation steps include: (1) constructing a training dataset, a validation dataset, and a test dataset; (2) constructing a target detection model based on a wavelet convolutional sequence network; (3) training the model using a large training dataset; and (4) detecting targets using the test dataset and outputting the final prediction result. The network model constructed in this invention uses a wavelet convolution module and a convolutional sequence fusion module, reducing computational load, significantly improving target detection speed, and achieving high detection accuracy.

[0006] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0007] A remote sensing image target detection method based on wavelet convolutional sequence network is proposed. The wavelet transform is applied to the backbone network of the target detection model of remote sensing image. By leveraging the wavelet transform's ability to extract features of different frequencies, different dimensional attribute features are extracted for the same target. A wavelet convolutional sequence module and a convolutional sequence fusion module are used to reduce the computational load.

[0008] Furthermore, the process of constructing training, validation, and test datasets for the object detection model includes:

[0009] The bilinear interpolation algorithm is used to scale all images in the database with a size of [R1, R2, R3] to obtain image patches of different sizes.

[0010] Using a rectangular window of size Hc×Hc and an overlap gap of size O, the optical remote sensing image containing labeled targets in the database is divided into blocks to obtain data blocks of size Hc×Hc×3 after cutting, and the overlap pixels between the data blocks are O.

[0011] All the cut data blocks are divided into three parts according to the size ratio [R4, R5, R6]: a training dataset T consisting of N1 data blocks, a validation dataset V consisting of N2 data blocks, and a test dataset U consisting of N3 data blocks.

[0012] Furthermore, in the target detection model, without increasing the number of parameters or expanding the receptive field, a wavelet convolutional sequence network is constructed by fusing a wavelet convolutional sequence module and a convolutional sequence fusion module.

[0013] Furthermore, the wavelet convolution sequence module includes two wavelet convolution operations WTconv; the wavelet convolution operation WTconv includes at least two branches, wherein the first branch only performs convolution operation on the image, and the second branch obtains four images through discrete wavelet transform, which are then added element-wise with the image of the first branch after convolution and inverse wavelet transform operations; if the number of branches exceeds 2, the processing of the nth branch (n>2) is the same as that of the second branch, and it is added element-wise with the (n-1)th branch before the inverse wavelet transform operation of the (n-1)th branch, and the outputs of all branches are output after the element-wise addition operation is completed.

[0014] Furthermore, the discrete wavelet transform is a Haar wavelet transform.

[0015] Furthermore, in the wavelet convolutional sequence network, for a given image X, it is input into a wavelet convolution operation WTconv_1 to obtain feature map F1. Then, feature map F1 is input into another wavelet convolution operation WTconv_2 to obtain feature map F2. F1 and F2 are input into the convolutional sequence fusion module to obtain feature map S. Finally, feature map S is added to the input image X to obtain the final feature map Y.

[0016] Furthermore, the specific process of the convolutional sequence fusion module is as follows: feature map F1 and feature map F2 are respectively processed by a 1×1 convolution. Obtain feature maps of the same dimension Will The layers are connected, pass through an average pooling layer, then a max pooling layer, and then input into a weight allocation layer to obtain the feature map SA. The feature map SA is then compared with the feature map. and Multiply them and add the results to obtain the feature map S.

[0017] Furthermore, in the target detection model, the backbone network extracts target features from remote sensing images using a wavelet convolutional sequence network, the neck network integrates different dimensional feature images extracted by the backbone network using an FPN network, and the head network obtains candidate boxes for the target region and the final classification score using Oriented R-CNN.

[0018] Furthermore, the detection process is as follows: data blocks in the test dataset are sequentially input into a trained target detection model based on a wavelet convolutional sequence network to obtain candidate boxes, scores corresponding to candidate boxes, and target categories corresponding to candidate boxes for each data block in the test dataset; all candidate boxes for target categories with scores higher than 0.5 are retained, and the remaining candidate boxes are discarded; finally, non-maximum suppression is applied to the coordinates and confidence scores of all retained target bounding boxes to obtain the detection results of the remote sensing images in the test dataset.

[0019] And an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the remote sensing image target detection method based on wavelet convolutional sequence networks as described above.

[0020] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the remote sensing image target detection method based on wavelet convolutional sequence networks as described above.

[0021] Compared to existing technologies, this invention and its preferred embodiment introduce wavelet transform into the feature extraction network for target detection. By leveraging the wavelet transform's ability to extract features of different frequencies, different dimensional attribute features can be extracted for the same target. Furthermore, the size of the receptive field in target detection is guaranteed without increasing the computational resources used, thereby achieving better results. Attached Figure Description

[0022] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0023] Figure 1 This is a flowchart illustrating the overall process of an embodiment of the present invention.

[0024] Figure 2 This is a diagram of the wavelet convolution module and wavelet convolution sequence network in an embodiment of the present invention;

[0025] Figure 3 This is a framework diagram of the target detection model based on wavelet convolutional sequence network in an embodiment of the present invention. Detailed Implementation

[0026] To make the features and advantages of the present invention more apparent and understandable, specific embodiments are described below in detail:

[0027] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0028] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0029] like Figure 1 As shown, the steps for implementing this invention are as follows:

[0030] Step 1: Construct the training dataset, validation dataset, and test dataset:

[0031] a. Scale all images in the database using bilinear interpolation with a size ratio of [R1, R2, R3] to obtain image patches of different sizes. In this embodiment, the database is provided by the DOTA optical remote sensing image dataset, R1 = 0.5, R2 = 1.0, R3 = 1.5, and the target bounding box coordinates consist of the coordinates of the top-left corner and the bottom-right corner of the target.

[0032] b. Using a rectangular window of size Hc×Hc and an overlap gap of size 0, the optical remote sensing image containing labeled targets in the database is segmented into blocks, resulting in data blocks of size Hc×Hc×3 after segmentation, with an overlap of 0 pixels between the data blocks. In this embodiment of the invention, Hc = 1024, 0 = 500;

[0033] c. Name each cut image patch according to the dataset naming rules. The dataset naming rules refer to connecting the file name of each remote sensing image to be cut, the corresponding sliding window step number of the cut data patch, the initial position of the sliding window on the x-axis, and the initial position of the sliding window on the y-axis using the English hyphen "--" symbol to generate a .jpg format file.

[0034] d. Divide all the segmented data blocks into images in the database according to a size ratio of [R4, R5, R6], resulting in a training dataset T consisting of N1 data blocks, a validation dataset V consisting of N2 data blocks, and a test dataset U consisting of N3 data blocks. In this embodiment of the invention, R4 = 0.5, R5 = 0.17, R6 = 0.33, N1 = 138883, N2 = 35462, and N3 = 71888.

[0035] Step 2, construct a target detection model based on wavelet convolutional sequence networks:

[0036] a. Construct a wavelet convolution module for extracting image features;

[0037] In the wavelet convolution sequence module of this embodiment, the Haar wavelet transform operation is preferably used because it is computationally efficient and simple. However, it should be noted that the implementation of the present invention is not limited to the Haar wavelet; other types of discrete wavelet bases can also be used, although this will increase the computational cost.

[0038] like Figure 2 As shown, firstly, image X needs to go through two branches. The first branch performs a convolution operation only on image X. The second branch uses Haar wavelets to obtain four images X. LL (1) X LH (1) X HL (1) X HH (1) The four images need to undergo convolution and inverse wavelet transform operations before being added to the image from the first branch. If the second branch requires two Haar wavelet operations, then X will be obtained. LL (2) …and so on, four images are then subjected to the same convolution and inverse wavelet transform operations, and the images after the convolution operation in the two branches are added together.

[0039] In general, a third or even a fourth branch can be derived after the second branch, depending on the number of Haar wavelets performed.

[0040] Taking Haar wavelets as an example, the process of performing a Haar wavelet (WT) includes: given an image X, a first-order Haar WT in one spatial dimension (width or height) is generated by the kernel. and The depthwise convolution is given, followed by a standard downsampling operator with a factor of 2. To perform 2D Haar WT, the operation is combined in two dimensions using the following four filters to produce a depthwise convolution with a stride of 2:

[0041]

[0042] Among them, f LL It is a low-pass filter, and f LH f HL f HH It is a set of high-pass filters. For each input channel, the output of the convolution satisfies the following formula:

[0043] [X LL ,X LH ,X HL ,X HH ] = Conv([f LL ,f LH ,f HL ,f HH ],X) (2)

[0044] The left side of the equation above contains four channels, each with a resolution of half X (in each spatial dimension). LL This represents the low-frequency component of X, while X... LH X HL X HH These represent its horizontal, vertical, and diagonal high-frequency components, respectively.

[0045] Since the kernel in the above formula constitutes an orthonormal basis, the inverse wavelet transform (IWT) is obtained through transpose convolution:

[0046] X = Conv-transposed([f LL ,f LH ,f HL ,f HH ],[X LL ,X LH ,X HL ,X HH (3)

[0047] Then, the cascaded wavelet decomposition is given by recursively decomposing the low-frequency components. Each level of the decomposition is as follows:

[0048]

[0049] in i is the index of the current level. This process results in an increase in frequency resolution and a decrease in low-frequency spatial resolution.

[0050] Increasing the kernel size of a spatial convolutional layer quadratically increases the number of parameters (degrees of freedom). To mitigate this, in this embodiment, wavelet transform is first used to filter and reduce the low-frequency and high-frequency content of the input. Then, before constructing the output using IWT, a small kernel depthwise convolution is performed on different frequency maps. The process is as follows:

[0051] Y = IWT(Conv(W,WT(X))) (5)

[0052] Here, X is the input tensor, and W is the weight tensor of a k×k depth kernel, which has four times the number of input channels as X. This operation not only separates the convolutions between frequency components but also allows a smaller kernel to operate over a larger area of ​​the original input, i.e., increasing its receptive field input.

[0053] like Figure 2 As shown. The derivation process for performing multiple Haar wavelets in wavelet convolution operations, and the process for calculating each component are as follows: In this embodiment of the invention, the wavelet convolution module adopts a first-level combination operation of WT and convolution, and further increases the number of Haar wavelets by using the same cascading principle as formula (4). The process is given by the following formula:

[0054]

[0055] in It is the input of this layer. This represents all three high-frequency mappings of the i-th layer described above. To combine outputs of different frequencies, this embodiment utilizes the property that WT and its inverse are linear operations, i.e.

[0056] IWT(X+Y) = IWT(X) + IWT(Y). Therefore, the following formula is applied:

[0057]

[0058] To obtain the sum of convolutions at different levels, where Z (i) This is the summation output starting from the i-th layer. The output is the sum of the outputs of two convolutions of different sizes.

[0059] In theory, we would like to normalize after convolution of each frequency component. However, since we are working in the wavelet domain, individual normalization of each frequency map does not correspond to normalization in the original domain. Therefore, we find that instead of normalizing, we can achieve the desired effect by simply performing channel scaling to measure the contribution of each frequency component.

[0060] b. Without increasing the number of parameters or expanding the receptive field, the wavelet convolution sequence module and the convolution sequence fusion module are fused to construct a wavelet convolution sequence network;

[0061] like Figure 2 As shown, in this embodiment of the wavelet convolutional sequence network, for a given image X, it is input into a wavelet convolution operation WTconv_1 to obtain feature map F1. Then, feature map F1 is input into another wavelet convolution operation WTconv_2 to obtain feature map F2. F1 and F2 are input into the convolutional sequence fusion module to obtain feature map S. Feature map S is added to the input image X to obtain the final feature map Y.

[0062] The specific steps within the convolutional sequence fusion module are as follows: F1 and F2 are respectively processed by a 1×1 convolution. Obtain feature maps of the same dimension Will The layers are connected, pass through an average pooling layer, then a max pooling layer, and finally input into a weight allocation layer to obtain the feature map SA. The feature map SA is then compared with the feature map... and Multiply them and add the results to obtain the feature map S.

[0063] In summary, the wavelet convolution sequence network is formed by the wavelet convolution sequence module consisting of the connected WTconv_1 and WTconv_2 and the convolution sequence fusion module.

[0064] c. Construct a complete target detection model;

[0065] like Figure 3As shown, in the target detection model of this invention, the backbone network extracts target features from remote sensing images using a wavelet convolutional sequence network. The neck network integrates feature images of different dimensions extracted by the backbone network using an FPN network. Finally, the head network obtains candidate boxes for the target region and the final classification score using Oriented R-CNN, i.e., the candidate box coordinates, the corresponding score, and the corresponding category are obtained by the regression classification layer. Both the FPN network and the Oriented R-CNN network are existing technologies. The general idea of ​​the FPN method is to integrate feature maps of different scales that have passed through convolutional layers consecutively in the backbone network, thereby obtaining target feature information at different scales. The general idea of ​​the Oriented R-CNN method is to increase the parameters required for prediction by the regression classification layer, transforming the prediction of bounding boxes into a method of predicting rotated bounding boxes within the bounding boxes, thus reducing the low detection rate of remote sensing image target detection models caused by preset anchor points.

[0066] Step 3: Train the model based on a large training dataset;

[0067] Step 4: Detect the target based on the test dataset and output the final prediction result:

[0068] a. Input the data blocks in the test dataset into the trained target detection model based on wavelet convolutional sequence network in sequence to obtain the candidate box, the score corresponding to the candidate box, and the target category corresponding to the candidate box for each data block in the test dataset;

[0069] b. Keep all candidate boxes for the target category with a score higher than 0.5, and discard the rest;

[0070] c. Perform non-maximum suppression on all the retained target bounding box coordinates and confidence scores to obtain the detection results of the remote sensing images in the test dataset. Non-maximum suppression is an existing technique. The general idea of ​​this method is to sort all bounding box coordinates from high to low confidence, retain bounding boxes with low overlap with other bounding boxes and high confidence, and discard bounding boxes with high overlap with other bounding boxes and low confidence.

[0071] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0072] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0073] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0074] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0075] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0076] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0077] This invention is not limited to the preferred embodiment described above. Anyone inspired by this invention can derive other forms of remote sensing image target detection methods based on wavelet convolutional sequence networks. All equivalent changes and modifications made within the scope of this invention should be considered within the scope of this invention.

Claims

1. A method for remote sensing image target detection based on a wavelet convolution sequence network, characterized in that: Wavelet transform is applied to the backbone network of the target detection model of the remote sensing image, different dimensional attribute features of the same target are extracted through the extraction ability of the wavelet transform for different frequency features, and a wavelet convolution sequence module and a convolution sequence fusion module are used to reduce the calculation amount; The process of constructing the training data set, the verification data set and the test data set for the target detection model comprises The scaling operation is performed on all the images in the database by using the bilinear interpolation algorithm with the size ratio of [R1, R2, R3] to obtain image blocks of different sizes; The optical remote sensing images containing the labeled targets in the database are cut into blocks by using a rectangular window with the size of Hc*Hc and an overlapping gap with the size of O to obtain data blocks with the size of Hc*Hc*3 after cutting, and the overlapping pixels between the data blocks are O; All the cut data blocks are divided into the training data set T composed of N1 data blocks, the verification data set V composed of N2 data blocks and the test data set U composed of N3 data blocks according to the size ratio of [R4, R5, R6] to the cut image blocks; In the target detection model, the wavelet convolution sequence network is constructed by fusing the wavelet convolution sequence module and the convolution sequence fusion module without increasing the parameter amount and expanding the receptive field; The wavelet convolution sequence module comprises two wavelet convolution operations WTconv, the wavelet convolution operation WTconv comprises at least two branches, wherein the first branch only performs convolution operation on the image, the second branch obtains four images through discrete wavelet transform, and the images are elementally added to the image of the first branch after convolution operation and inverse wavelet transform operation; if the number of branches is more than 2, the processing process of the nth branch is the same as that of the second branch, n>2, and the nth branch is elementally added to the (n-1)th branch before the inverse wavelet transform operation of the (n-1)th branch, and the output of all branches is output after the element addition operation is completed; The discrete wavelet transform is Haar wavelet transform; In the wavelet convolution sequence network, the given image X is input into a wavelet convolution operation WTconv_1 to obtain a feature map F1, then the feature map F1 is input into another wavelet convolution operation WTconv_2 to obtain a feature map F2, the F1 and F2 are input into the convolution sequence fusion module to obtain a feature map S, and the feature map S is added to the input image X to obtain a final feature map Y.

2. The method of claim 1, wherein the method comprises: The specific process of the convolution sequence fusion module is as follows: the feature map F1 and the feature map F2 are respectively processed by a 1x1 convolution F1 1×1 、 to obtain feature maps of the same dimension The are connected, and after passing through an average pooling layer and a maximum pooling layer and then inputting into a weight distribution layer, a feature map SA is obtained. The feature map SA is multiplied by the feature map and respectively, and the multiplied results are added to obtain the feature map S. 3.The method of claim 1, wherein the method further comprises: In the target detection model, the backbone network extracts the target feature of the remote sensing image by the wavelet convolution sequence network, the neck network integrates the different dimensional feature images extracted by the backbone network by the FPN network, and the head network obtains the target region candidate frame and the final classification score by the OrientedRCNN.

4. The method of claim 3, wherein the method further comprises: The detection process specifically comprises: the data blocks in the test data set are sequentially input into the trained target detection model based on the wavelet convolution sequence network to obtain the candidate frame of each data block in the test data set, the score corresponding to the candidate frame and the target category corresponding to the candidate frame; all the candidate frames of the target category with the score higher than 0.5 are retained, and the remaining candidate frames are discarded. Finally, all the target bounding box coordinates and confidence obtained after reservation are executed non-maximum suppression to obtain the detection results of the remote sensing image in the test data set.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the remote sensing image target detection method based on the wavelet convolution sequence network according to any one of claims 1-4 when executing the program.

Images