Target detection method and device for remote sensing image, medium and equipment

Abstract

The invention discloses a target detection method and device for a remote sensing image, a medium and equipment, and relates to the technical field of target detection. Comprising the following steps: constructing a deep learning network comprising a coding module, a multi-head cross attention module, a weighting balance module and a decoding module; the coding module comprises a convolutional neural network and a Transformer which are parallel, and a cross module which is connected with the output end of the convolutional neural network and the output end of the Transformer at the same time. According to the method, the cross module and the multi-head cross attention module are combined, so that the model can be more flexibly adapted to the scale change of a salient target in a remote sensing image, and the situation that foreground details are insufficiently captured due to excessive global features or the background is insufficiently understood due to excessive local features is avoided; and the accuracy of the target detection task in the remote sensing image is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a method, device, medium and equipment for detecting a target from a remote sensing image. Background Art

[0002] SOD (Salient Object Detection) is a research direction in the field of computer vision. Its main purpose is to automatically detect the most eye-catching part from the image, that is, the salient object in the image that is significantly different from the surrounding environment in color, brightness, texture or shape. Since SOD can effectively identify the most eye-catching part of the image, it can assist image editing software to automatically identify salient objects, thereby guiding automatic cropping or scaling operations; or assist video compression software to retain salient areas and use higher compression rates in non-salient areas.

[0003] In the field of SOD research, deep learning methods are currently a hot research direction. Considering that CNN (Convolutional Neural Network) tends to capture local information and Transformer is good at modeling global dependencies, we can design a method such as Figure 1 The parallel architecture-based fusion strategy shown in the figure realizes the complementarity of the two models. The parallel architecture-based fusion strategy processes the image through multiple stages and fuses the outputs of CNN and Transformer in each stage, making full use of the characteristics of CNN and Transformer so that the features of each stage can complement and interact.

[0004] However, due to the significant scale variations of target objects in remote sensing images, the fusion strategy based on parallel architecture in existing technologies is difficult to balance between CNN and Transformer, which may lead to too many global features resulting in insufficient capture of foreground details or too many local features resulting in insufficient understanding of the background, resulting in inaccurate target detection results in remote sensing images. Summary of the invention

[0005] Based on this, in order to solve the technical problem that the prior art has inaccurate detection results of significant targets in remote sensing images, the present invention provides a method, device, medium and equipment for target detection in remote sensing images.

[0006] The present invention provides a method for detecting a target in a remote sensing image, comprising:

[0007] Constructing a deep learning network including an encoding module, a multi-head cross attention module and a decoding module connected in sequence; the encoding module includes a parallel convolutional neural network and a Transformer, and a cross module connected to the output end of the convolutional neural network and the output end of the Transformer at the same time;

[0008] Collecting remote sensing images containing targets to construct a data set; using the data set to train the deep learning network to obtain a salient target detection model;

[0009] The remote sensing image to be detected is input into the salient target detection model, and the features of the remote sensing image are extracted through convolutional neural network and Transformer to obtain the local features and global features of the remote sensing image; the local features and global features are cross-interchanged along different dimensions through the cross module to cross-complementarily perceive the local features and global features of the remote sensing image to obtain cross-local features and cross-global features; the contribution of cross-local features and cross-global features to the targets in the remote sensing image is learned from multiple aspects through multiple heads in the multi-head cross attention module, and the weights of the cross-local features and cross-global features that respond to the targets in the remote sensing image are dynamically expanded according to the learning results to obtain balanced features; the balanced features are decoded by the decoder to obtain the salient targets in the remote sensing image.

[0010] Furthermore, the convolutional neural network includes: 4 sequentially connected first residual convolution modules, second residual convolution modules, third residual convolution modules and fourth residual convolution modules; the residual convolution modules are used to extract local features of different sizes in remote sensing images;

[0011] The Transformer includes: 4 sequentially connected first pyramid vision Transformer modules, second pyramid vision Transformer modules, third pyramid vision Transformer modules and fourth pyramid vision Transformer modules; the pyramid vision Transformer modules are used to extract global features of different sizes in remote sensing images;

[0012] The cross module includes: a first spatial cross module connected to the output end of the first residual convolution module and the output end of the first pyramid vision Transformer module, a second spatial cross module connected to the output end of the second residual convolution module and the output end of the second pyramid vision Transformer module, a first channel cross module connected to the output end of the third residual convolution module and the output end of the third pyramid vision Transformer module, and a second channel cross module connected to the output end of the fourth residual convolution module and the output end of the fourth pyramid vision Transformer module; the spatial cross module is used to exchange local features and global features along the spatial dimension; the channel cross module is used to exchange local features and global features along the channel dimension.

[0013] Furthermore, the first spatial cross module and the second spatial cross module structure perform feature interaction operations on the output of the first residual convolution module and the output of the first pyramid vision Transformer module, and the output of the second residual convolution module and the output of the second pyramid vision Transformer module in the spatial dimension respectively;

[0014] The first channel cross module and the second channel cross module respectively perform interactive operations on the output of the third residual convolution module and the output of the third pyramid vision Transformer module and the output of the fourth residual convolution module and the output of the fourth pyramid vision Transformer module in the channel dimension, specifically including:

[0015] Perform channel normalization on the feature map of the input cross module to align the number of channels of local features and global features:

[0016]

[0017] in, Represents the global features output by the i-th residual convolution module; represents the local features output by the i-th pyramid vision Transformer module; where i = 1, 2, 3, 4; CBR(·) means performing convolution, batch normalization, and activation function operations in sequence;

[0018] Perform channel enhancement on local features and global features after channel normalization:

[0019]

[0020] in, Represents the global features after channel normalization operation; represents the local features after channel normalization operation; ⊙ represents the multiplication of channel elements; FC represents the fully connected layer; GMP(·) represents the global maximum pooling operation; GAP(·) represents the global average pooling operation; σ(·) represents the Sigmoid activation function;

[0021] Cross-exchange the local features and global features after channel enhancement:

[0022]

[0023]

[0024] in, Represents the global features after cross-exchange; represents the local features after cross-exchange; SEx(·) represents the spatial interaction operation; CEx(·) represents the channel interaction operation.

[0025] Furthermore, a parallel local feature input branch and a global feature input branch are connected between the output end of the cross module and the input end of the multi-head cross attention module;

[0026] The global feature input branch is used to perform cross-interchange on the global feature after the output of the second channel cross module is Add positional coding and based on the Constructing a query Q in the multi-head cross-attention module;

[0027] The local feature input branch is used to perform cross-interchange on the local features output by the second channel cross module. Add positional coding and based on the Construct the key K and value V in the multi-head criss-cross attention module.

[0028] Furthermore, the decoding module includes a first decoding submodule, a second decoding submodule, a third decoding submodule, and a fourth decoding submodule connected in sequence; wherein the input end of the first decoding submodule is connected to the output end of the cross module; the input end of the second decoding submodule is connected to the output end of the first channel cross module; the input ends of the third decoding submodule and the fourth decoding submodule are connected to the output end of the second spatial cross module; the input ends of the fourth decoding submodule and the fourth decoding submodule are connected to the output end of the first spatial cross module; and the decoding submodule is used to reconstruct the output of the cross module.

[0029] The present invention provides a remote sensing image target detection device, comprising:

[0030] A model building module, for building a deep learning network including an encoding module, a multi-head cross attention module and a decoding module connected in sequence; the encoding module includes a parallel convolutional neural network and a Transformer, and a cross module connected to the output end of the convolutional neural network and the output end of the Transformer at the same time;

[0031] A model training module is used to collect remote sensing images containing targets to construct a data set; the deep learning network is trained using the data set to obtain a salient target detection model;

[0032] The target detection module is used to input the remote sensing image to be detected into the salient target detection model, extract the features of the remote sensing image through the convolutional neural network and Transformer, and obtain the local features and global features of the remote sensing image; the local features and the global features are cross-interchanged along different dimensions through the cross module to cross-complementarily perceive the local features and the global features of the remote sensing image, and obtain cross-local features and cross-global features; the contribution of the cross-local features and the cross-global features to the targets in the remote sensing image is learned from multiple aspects through the multiple heads in the multi-head cross attention module, and the weights of the cross-local features and the cross-global features that respond to the targets in the remote sensing image are dynamically expanded according to the learning results to obtain balanced features; the balanced features are decoded by the decoder to obtain the salient targets in the remote sensing image.

[0033] The present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned remote sensing image target detection method is implemented.

[0034] The present invention provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned remote sensing image target detection method when executing the program.

[0035] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0036] In the target detection method of remote sensing images provided by the present invention, local features and global features are cross-exchanged along different dimensions through a cross module, thereby realizing information exchange between local features and global features in different dimensions, so that the model outputs the detection result after balancing the information of local features and global features in different dimensions, avoiding the model being more biased towards a certain feature for output, and improving the sensitivity of the model to the scale transformation of significant targets; then, using multiple heads in a multi-head cross attention module, the contribution of the cross-exchanged local features and global features to the significant targets in the remote sensing images is learned from multiple aspects, and the feature weights that respond to the significant targets in the remote sensing images are dynamically expanded according to the learning results, and the local features and global features are further balanced; by combining the cross module with the multi-head cross attention module, the model can more flexibly adapt to the scale changes of significant targets in remote sensing images, avoiding the situation where too many global features lead to insufficient capture of foreground details or too many local features lead to insufficient understanding of the background, thereby improving the accuracy of the significant target detection task in remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0038] Figure 1 A schematic diagram of the CNN and Transformer fusion strategy based on a parallel architecture provided by the present invention;

[0039] Figure 2 A schematic diagram of a dual-stream feature fusion perception network framework provided by the present invention; Figure 2 (a) is the overall network framework; Figure 2 (b) is a schematic diagram of the structure of PvtBlock; Figure 2 (c) is a schematic diagram of the structure of ResBlock; Figure 2 (d) is the structural diagram of DeBlock;

[0040] Figure 3 A schematic diagram of the structure of the MCIM provided by the present invention;

[0041] Figure 4 This is a schematic diagram of the structure of the FWBM provided by the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in combination with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] After years of development, SOD has made significant progress in natural images. However, since remote sensing images are taken from a bird's-eye view, their structure is significantly different from that of natural images, which makes SOD in remote sensing still face many challenges. First, the background of remote sensing images is complex, the spectral information is diverse, and the images contain a large number of natural landscapes and man-made structures, making the detection of salient targets more difficult. Second, remote sensing images are usually taken from high altitudes. Unlike natural images, they are not affected by the single direction of gravity, so the directions of salient target objects are variable and the scales vary significantly. This multi-scale and multi-directional feature increases the complexity of detection. In addition, salient targets in remote sensing images include many different types of objects such as buildings, roads, and water bodies. These salient targets vary greatly in size, shape, color, etc. In this case, the SOD method in natural images is difficult to adapt to the characteristics of remote sensing images. In summary, salient targets are easily affected by complex imaging conditions and interference from similar objects, making it difficult to accurately extract information.

[0044] Based on this, in order to solve the problem that a single branch (convolutional neural network or transformer) cannot completely extract significant salient targets, the present invention constructs a dual-stream feature fusion perception framework for remote sensing significant salient target detection. First, dual-branch feature extractors with good local bias and long-range dependency characteristics are used to extract multi-scale features of remote sensing images. Then, the MCIM (Multi-Path Complementarity-aware Interaction Module) is constructed to refine and fuse the representations of significant targets in different channels, and realize the fine-grained fusion and interactive alignment of dual-branch features. Finally, in order to prevent the insufficient capture of foreground details due to too many global features or insufficient understanding of the background due to too many local features in the process of dual-branch feature complementation, a feature weighted balance module is proposed to balance the foreground and background features of the significant area, refine and enhance the features of different areas, and further improve the perception ability of significant features.

[0045] The following is a detailed description of the process of the method for identifying salient targets in remote sensing images provided by this embodiment, which specifically includes the following steps:

[0046] S1: Construct a deep learning network including an encoding module, a feature weighted balancing module, and a decoding module.

[0047] CNN is subject to the spatial constraints of the receptive field and lacks the ability to model global features. It is inadequate in capturing complex semantic information and dealing with scale changes, occlusions, and shadows. This limits its ability to identify and locate remote sensing images with diverse salient targets and backgrounds. On the other hand, there are irrelevant objects with different shapes and distributions in the background of remote sensing images. These complex backgrounds have large spectral differences, which increases the difficulty of scene parsing. Although Transformer has a strong global feature aggregation capability, it will introduce a large amount of background noise, which will have a negative impact on the accuracy of salient target detection.

[0048] Therefore, a dual-stream feature fusion perception network is constructed to combine the advantages of CNN and Transformer to build a more robust and accurate remote sensing image salient target detection model. The dual-stream feature fusion perception network can effectively improve the target detection performance of remote sensing images by utilizing the local bias capability of CNN and the global feature aggregation capability of Transformer through a dual-branch feature fusion strategy. The constructed dual-stream feature fusion perception network framework includes: encoding module, feature weighted balance module and decoding module. Among them, the encoding module includes a parallel convolutional neural network and a Transformer, and a cross module connected to the output end of the convolutional neural network and the output end of the Transformer at the same time; the convolutional neural network includes 4 first residual convolution modules, a second residual convolution module, a third residual convolution module and a fourth residual convolution module connected in sequence; the Transformer includes 4 first pyramid visual Transformer modules, a second pyramid visual Transformer module, a third pyramid visual Transformer module and a fourth pyramid visual Transformer module connected in sequence; the cross module includes a first spatial cross module connected to the output end of the first residual convolution module and the output end of the first pyramid visual Transformer module, a second spatial cross module connected to the output end of the second residual convolution module and the output end of the second pyramid visual Transformer module, a first channel cross module connected to the output end of the third residual convolution module and the output end of the third pyramid visual Transformer module, and a second channel cross module connected to the output end of the fourth residual convolution module and the output end of the fourth pyramid visual Transformer module. The feature weighted balance module is connected to the output end of the cross module, including parallel local feature input branches and global feature input branches, and a multi-head cross attention module connected to the output ends of the local feature input branches and the global feature input branches. The decoding module includes a first decoding submodule, a second decoding submodule, a third decoding submodule, and a fourth decoding submodule connected in sequence; wherein the input end of the first decoding submodule is connected to the output end of the cross module; the input end of the second decoding submodule is connected to the output end of the first channel cross module; the input ends of the third decoding submodule and the fourth decoding submodule are connected to the output end of the second spatial cross module; the input ends of the fourth decoding submodule and the fourth decoding submodule are connected to the output end of the first spatial cross module.

[0049] The constructed dual-stream feature fusion perception network framework is as follows Figure 2As shown in (a), first, the two-stream feature extractor consists of a complementary global encoder and a local encoder to achieve the complementarity of global contextual dependency and local fine-grained features. The global encoder is built on PVTv2 (Pyramid Vision Transformervision 2), and the local encoder is built on ResNet50. PVTv2 consists of four stages, each of which contains PatchEmbedding and multi-head attention blocks. ResNet50 retains the pooling layer and four residual layers for feature extraction. When the input image x∈3×352×352 is input into the feature extractor, 4 sets of multi-scale complementary features are output in the two encoders respectively. The global features are The local features are

[0050] Then the same-level features are sent to MCIM for fine-grained feature fusion and interactive feature alignment operations. MCIM integrates the global features and local details of RSI and carefully fuses the salient information of different channels. Then, FWBW (Feature Weighted Balancing Module) is used to maintain the balance of semantic information of foreground and background in deep features, adaptively enhance the feature representation of different regions, and prevent too many global features or local features from drowning out salient features in the dual-branch feature fusion process. Specifically, too many global features may lead to insufficient capture of details of salient foreground objects, while too many local features may lead to insufficient understanding of background information, thus affecting the distinction between foreground and background. Finally, multi-level features are cascaded and upsampled in the decoder to output high-quality segmentation results. The decoder consists of DeBlock1, DeBlock2, DeBlock3 and SalHead. DeBlock integrates convolution and deconvolution operations, where deconvolution is used to upsample low-resolution feature maps to solve the problem of inconsistent resolution of multi-level features. SalHead is a 1×1 convolution used to generate a segmentation map with 1 channel. Figure 2 The structures of PvtBlock, ResBlock and DeBlock in (a) are as follows Figure 2 (b) Figure 2 (c) and Figure 2 As shown in (d) in .

[0051] Figure 2 The structure of MCIM is as follows Figure 3As shown in the figure. In the task of salient object detection in remote sensing images, methods that rely solely on CNN or Transformer have certain limitations. The respective advantages and disadvantages of CNN and Transformer determine that a single feature extraction method alone cannot fully realize their potential, thus limiting the improvement of salient object detection performance. CNN gradually extracts local features through a sliding window mechanism. Due to the large correlation between pixels within the window, CNN performs well in extracting local details of salient objects. However, the sliding window mechanism also limits CNN's ability to capture global features, and the connection to pixels outside the window is weak, resulting in its poor performance in reflecting the global structural features of the image. In contrast, Transformer can closely connect all pixel information globally by converting all pixels in the entire image into query (Q), key (K) and value (V) triplets for self-attention interaction. Therefore, Transformer has a significant advantage in aggregating global features. However, this global attention mechanism will also introduce too much background noise, interfere with the feature extraction of salient objects, and reduce the accuracy of detection.

[0052] To address these limitations, a multi-path complementary perception interaction module is proposed to deeply explore the complementary information in CNN and Transformer branches for multi-layer feature fusion and achieve feature alignment between global and local branches. This module uses spatial and channel interaction strategies to fuse and complement the features of the two branches at different feature extraction stages.

[0053] Spatial Interaction: In shallow networks (stage 1 and stage 2), since the feature map has a higher resolution, the features are mainly distributed in the spatial dimension. A spatial interaction strategy is used to achieve feature alignment and fusion. Feature exchange is performed in the spatial dimension to embed contextual information and local area features into each other. On the one hand, the feature information from the transformer branch can assist the model to better capture the spatial relationship and structural information in the image; on the other hand, the feature information from the CNN branch enables the model to capture clearer local context.

[0054] Channel Interaction: In deep networks (stage 3 and stage 4), salient features are mainly concentrated in the channel dimension, and it is more suitable to adopt the channel interaction strategy to fuse information between different feature channels. The global information from the Transformer branch enhances the model's perception of large-scale features, while the local details from the CNN branch further refine the feature representation of salient targets.

[0055] Therefore, by adopting the spatial interaction strategy in the shallow layer, the complementarity of local and global features can be fully utilized in the initial stage to improve the expressiveness of features and the recognition accuracy of salient targets. By adopting the channel interaction strategy in the deep layer, the feature information of different channels can be effectively integrated when processing complex scenes, improving the richness and accuracy of the expression of salient features.

[0056] First, perform channel normalization on the input feature map to align the number of channels of the feature maps of the CNN and Transformer branches:

[0057]

[0058] in, Represents the global features output by the i-th residual convolution module; Represents the local features output by the i-th pyramid vision Transformer module; where i = 1, 2, 3, 4; CBR(·) represents the sequence of convolution (Conv), batch normalization (Batch Normalization) and activation function ReLU.

[0059] Then, channel attention is used to enhance the representation ability of important feature channels and improve the recognition of significant features:

[0060]

[0061] in, Represents the global features after channel normalization operation; represents the local features after channel normalization; ⊙ represents the multiplication of channel elements; FC represents the fully connected layer; GMP(·) represents the global maximum pooling operation; GAP(·) represents the global average pooling operation; σ(·) represents the sigmoid activation function

[0062] Next, different ExChange strategies are adopted according to the distribution of the features of the input image in the spatial dimension and channel dimension. Spatial feature exchange is adopted in the shallow layer, and channel feature exchange is adopted in the deep layer:

[0063]

[0064] in, Represents the global features after cross-exchange; Indicates the local features after cross-exchange; SEx indicates spatial interaction; CEx indicates channel interaction. Cross-exchange in different dimensions can ensure that features of different scales are integrated. Interchange in the spatial dimension can integrate information from different regions, and interchange in the channel dimension can integrate different types of information.

[0065] After the channel interaction, the number of channels of the feature map is adjusted to the original size again through 1×1 convolution to ensure that the interacted features can be smoothly passed to the next stage of the backbone network, further promoting the fusion of dual-branch features.

[0066]

[0067] in, and They are the global features and local features of the input of the i+1th stage respectively.

[0068] Figure 2 The structure of FWBM is as follows Figure 4 As shown in the figure. For salient objects in complex scenes (such as buildings, roads, and water bodies), the extracted semantic features vary significantly due to their different sizes, shapes, colors, lighting conditions, and even imaging conditions. Although improved by global and local feature complementation and fusion strategies, the imbalance between global semantics and local information can still lead to the submergence of important features. Specifically, too many global features may lead to insufficient capture of details of foreground salient targets, while too many local features may lead to insufficient understanding of background information, thus affecting the distinction between foreground and background. Therefore, a feature weighted balancing module is proposed to balance the global and local features of salient regions and refine and enhance the features of different regions. Through this module, the model can adaptively weight the features of foreground and background regions. When the ground objects in the background region are misjudged as salient targets, it means that the background features are relatively strong, and the module will weaken the feature representation of the background region to achieve more accurate salient target separation; on the contrary, when the salient target features in the salient region are disturbed by the background features, the module can enhance the feature expression of the salient target to ensure its clarity and recognition in a complex background.

[0069] This module integrates global context and local detail information, uses self-attention and feature reweighting strategies, and dynamically adjusts the importance of features to ensure accurate detection and detailed characterization of salient targets in diverse remote sensing scenarios, thereby significantly improving the detection performance of the model.

[0070] The deep feature maps f4x and f4y of the Transformer branch and CNN branch are used as input to achieve feature refinement and enhancement. First, the input features are embedded through the convolution layer and trainable position encoding is added to retain the position information:

[0071] t x =embed x (x)+pos x

[0072] t y =embed y (y)+pos y

[0073] Among them, x and y represent the global and local features of the input FWBM respectively; pos x and pos y They represent the position encoding of global features and local features respectively; embed(·)· represents the embedding operation.

[0074] Next, layer normalization (LN) is performed on the feature signal to stabilize the training process and speed up convergence:

[0075]

[0076] Among them, layer normalization processes the input x as follows:

[0077]

[0078] Where E(x) and Var(x) represent the expectation and standard deviation of the input x, respectively. ε is added to the denominator for numerical stability, and the default value is 1e -5 ; β is a learnable affine transformation parameter.

[0079] In the multi-head cross attention, global features are converted into query Q, local features are converted into key K) and value V, and the cross attention mechanism is used to aggregate global context information. This process can effectively capture the relationship between distant pixels in the image, allowing the model to have a deeper understanding of global structural information:

[0080]

[0081] in, is the softmax function; d is the normalization parameter; Q x , K y and V y Through the projection matrix W q , W k and W vTransformation yields:

[0082]

[0083] Then, the features are further processed using a feedforward neural network to enhance their expressiveness. The features processed by different branches are fused through element-wise addition operations to further enhance the features of different regions:

[0084]

[0085] Among them, FFN(·) is a feed-forward neural network.

[0086] S2. Collect remote sensing images containing salient targets and construct a data set; use the data set to train the deep learning network to obtain a salient target detection model.

[0087] S201. Build a dataset.

[0088] Two public remote sensing image salient object detection datasets, ORSSD and EORSSD, are used to evaluate the performance of the constructed network. Each dataset contains the original image and its corresponding pixel-level labels. In order to better train the model, the training samples of the dataset are augmented by 8 times.

[0089] ORSSD is the first publicly available remote sensing dataset for salient object detection, containing a total of 800 images, of which 600 images are used for training and 200 images are used for testing. These images are generated by remote sensing satellites and aerial photography systems, mainly collected through Google Earth, and a small amount comes from the existing RSI database. The spatial resolution of each original image is between 0.5 and 2 meters, and the image resolution ranges from 256×256 to 2107×1198. The dataset covers 8 types of salient targets, including airplanes, ships, cars, rivers, ponds, bridges, stadiums, and beaches.

[0090] EORSSD is an extension of ORSSD, with an additional 1,200 images, also from Google Earth. EORRSD contains a total of 2,000 samples, including 1,400 images in the training set and 600 images in the test set. Compared with ORSSD, EORSSD is more challenging in all aspects.

[0091] S202. Training environment settings.

[0092] This example is based on the PyTorch framework and uses a high-performance GPU4060Ti equipped with 16GB of video memory. During the training process, PVTv2 and ResNet50 were selected as feature extraction networks and initialized by loading the corresponding pre-trained weights. The input image size was set to 352×352, Adam was used as the optimizer, batch size was set to 8, epoch was set to 90, and the initial learning rate was set to 1×10 -4 , after 75 epochs, the learning rate is reduced to 1 / 10 of the initial value.

[0093] In order to comprehensively evaluate the performance of the model, a series of evaluation indicators are used to quantitatively compare different methods. Among them, S-measure evaluates the structural similarity between the predicted results and the true labels from two aspects: salient target similarity and background similarity. Its parameter α is set to 0.5 to balance the importance of salient targets and backgrounds. The mean square error (MAE) indicates the absolute error between the predicted results and the true labels, and is one of the commonly used indicators for evaluating prediction accuracy. E-measure evaluates the experimental results based on the difference between the global mean and local pixels, emphasizing the overall characteristics of the image and the matching degree of local details. F-measure comprehensively evaluates the quality of the saliency map by balancing precision and recall. Among these indicators, the smaller the MAE, the better, while the larger the values ​​of other indicators, the better the performance of the model.

[0094] This embodiment defines a hybrid loss in the training phase to back-propagate and update the model parameters. The overall loss consists of the pixel-level loss BCE and the map-level loss IOU, and its formula is:

[0095] L total =L bce (Up(S),GT)+L IOU (σ(Up(S)),GT)

[0096] Among them, s represents the salient target segmentation map, GT represents the true label, Up(·) represents upsampling using bilinear interpolation, and σ(·) is the sigmoid function.

[0097] S3. Input the remote sensing image to be detected into the salient target detection model, extract features of the remote sensing image through convolutional neural network and Transformer, and obtain local features and global features of the remote sensing image; cross-exchange the local features and global features along different dimensions through the cross module to cross-complementarily perceive the local features and global features of the remote sensing image, and obtain cross-local features and cross-global features; learn the contribution of cross-local features and cross-global features to the salient targets in the remote sensing image from multiple aspects through multiple heads in the multi-head cross attention module, and dynamically expand the weights of the cross-local features and cross-global features that respond to the targets in the remote sensing image according to the learning results to obtain balanced features; decode the balanced features through the decoder to obtain the salient targets in the remote sensing image.

[0098] In summary, this embodiment proposes a dual-stream feature fusion perception network (DFSNet). First, DFSNet combines the advantages of Transformer and CNN to establish a dual-branch feature extractor to deeply explore local features in remote sensing images and capture a wide range of dependencies. Then, in order to achieve deep fusion of features, a multi-path complementary perception interaction module (MCIM) is designed to perform multi-path interaction and significant feature alignment between dual branches. Finally, a feature weighted balancing module (FWBM) is used to balance global semantics and local information, refine and enhance features in different regions, and further enhance the perception of significant features.

[0099] In summary, the work is as follows: 1. An end-to-end two-stream feature fusion perception network is developed, which combines the advantages of Transformer and CNN, can model global relationships and capture local fine-grained representations, thereby improving the detection ability of salient objects. 2. In order to make full use of the local bias of CNN and the long-range dependency characteristics of Transformer, realize the multi-level interaction of dual branches and the fine-grained alignment of salient features, a multi-path feature interaction module is proposed. This module uses spatial interaction and channel interaction strategies for low-level features and high-level features respectively to achieve feature complementarity of global information and local details. 3. In addition, in order to prevent the insufficient capture of foreground details due to too many global features or insufficient understanding of the background due to too many local features in the process of dual-branch feature complementation, a feature weighted balancing module (FWBM) is proposed. This module perceives contextual clues, uses feature reweighting strategies to refine and enhance features in different regions, and dynamically adjusts the importance of foreground and background.

[0100] The above is a target detection method for remote sensing images provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding target detection device for remote sensing images, including:

[0101] A model building module is used to build a deep learning network including a sequentially connected encoding module, a multi-head cross-attention module and a decoding module; the encoding module includes a parallel convolutional neural network and a Transformer, and a cross module simultaneously connected to the output end of the convolutional neural network and the output end of the Transformer.

[0102] The model training module is used to collect remote sensing images containing targets to construct a data set; the deep learning network is trained using the data set to obtain a significant target detection model.

[0103] The target detection module is used to input the remote sensing image to be detected into the salient target detection model, extract the features of the remote sensing image through the convolutional neural network and Transformer, and obtain the local features and global features of the remote sensing image; the local features and the global features are cross-interchanged along different dimensions through the cross module to cross-complementarily perceive the local features and the global features of the remote sensing image, and obtain cross-local features and cross-global features; the contribution of the cross-local features and the cross-global features to the targets in the remote sensing image is learned from multiple aspects through the multiple heads in the multi-head cross attention module, and the weights of the cross-local features and the cross-global features that respond to the targets in the remote sensing image are dynamically expanded according to the learning results to obtain balanced features; the balanced features are decoded by the decoder to obtain the salient targets in the remote sensing image.

[0104] For the specific definition of the target detection device for remote sensing images, please refer to the definition of the target detection method for remote sensing images in the above text, which will not be repeated here. Each module in the above-mentioned target detection device for remote sensing images can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0105] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The object detection method of remote sensing image is provided.

[0106] The present invention also provides a computer device structure. At the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The object detection method of remote sensing image is provided.

[0107] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0108] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A method for detecting a target in a remote sensing image, characterized in that: include: Construct a deep learning network consisting of a sequentially connected encoding module, a multi-head cross-attention module, and a decoding module; The encoding module includes a parallel convolutional neural network and a Transformer, and a cross module connected to the output end of the convolutional neural network and the output end of the Transformer at the same time; Collecting remote sensing images containing targets to construct a data set; using the data set to train the deep learning network to obtain a salient target detection model; The remote sensing image to be detected is input into the salient target detection model, and the remote sensing image is feature extracted through the convolutional neural network and Transformer to obtain the local and global features of the remote sensing image; the local features and the global features are cross-interchanged along different dimensions through the cross module to cross-complementarily perceive the local features and the global features of the remote sensing image to obtain cross-local features and cross-global features; the contribution of the cross-local features and the cross-global features to the targets in the remote sensing image is learned from multiple aspects through multiple heads in the multi-head cross attention module, and the weights of the cross-local features and the cross-global features that respond to the targets in the remote sensing image are dynamically expanded according to the learning results to obtain balanced features; The decoder decodes the balanced features to obtain the salient targets in the remote sensing image.

2. The target detection method of remote sensing image according to claim 1, characterized in that: The convolutional neural network comprises: four sequentially connected first residual convolution modules, second residual convolution modules, third residual convolution modules and fourth residual convolution modules; the residual convolution modules are used to extract local features of different sizes in remote sensing images; The Transformer includes: 4 sequentially connected first pyramid vision Transformer modules, second pyramid vision Transformer modules, third pyramid vision Transformer modules and fourth pyramid vision Transformer modules; the pyramid vision Transformer modules are used to extract global features of different sizes in remote sensing images; The cross module includes: a first spatial cross module connected to the output end of the first residual convolution module and the output end of the first pyramid vision Transformer module, a second spatial cross module connected to the output end of the second residual convolution module and the output end of the second pyramid vision Transformer module, a first channel cross module connected to the output end of the third residual convolution module and the output end of the third pyramid vision Transformer module, and a second channel cross module connected to the output end of the fourth residual convolution module and the output end of the fourth pyramid vision Transformer module; the spatial cross module is used to exchange local features and global features along the spatial dimension; the channel cross module is used to exchange local features and global features along the channel dimension.

3. The target detection method of remote sensing image according to claim 2, characterized in that: The first spatial cross module and the second spatial cross module structure respectively perform feature interaction operations on the output of the first residual convolution module and the output of the first pyramid vision Transformer module, and the output of the second residual convolution module and the output of the second pyramid vision Transformer module in the spatial dimension; The first channel cross module and the second channel cross module respectively perform interactive operations on the output of the third residual convolution module and the output of the third pyramid vision Transformer module and the output of the fourth residual convolution module and the output of the fourth pyramid vision Transformer module in the channel dimension, specifically including: Perform channel normalization on the feature map of the input cross module to align the number of channels of local features and global features: in, Represents the global features output by the i-th residual convolution module; represents the local features output by the i-th pyramid vision Transformer module; where i = 1, 2, 3, 4; CBR(·) means performing convolution, batch normalization, and activation function operations in sequence; Perform channel enhancement on local features and global features after channel normalization: in, Represents the global features after channel normalization operation; represents the local features after channel normalization operation; ⊙ represents the multiplication of channel elements; FC represents the fully connected layer; GMP(·) represents the global maximum pooling operation; GAP(·) represents the global average pooling operation; σ(·) represents the Sigmoid activation function; Cross-exchange the local features and global features after channel enhancement: in, Represents the global features after cross-exchange; represents the local features after cross-exchange; SEx(·) represents the spatial interaction operation; CEx(·) represents the channel interaction operation.

4. The target detection method of remote sensing image according to claim 3, characterized in that: A parallel local feature input branch and a global feature input branch are connected between the output end of the cross module and the input end of the multi-head cross attention module; The global feature input branch is used to perform cross-interchange on the global feature after the output of the second channel cross module is Add positional coding and based on the Constructing a query Q in the multi-head cross-attention module; The local feature input branch is used to perform cross-interchange on the local features output by the second channel cross module. Add positional coding and based on the Construct the key K and value V in the multi-head criss-cross attention module.

5. The target detection method of remote sensing image according to claim 2, characterized in that: The decoding module includes a first decoding submodule, a second decoding submodule, a third decoding submodule, and a fourth decoding submodule which are connected in sequence; wherein the input end of the first decoding submodule is connected to the output end of the cross module; the input end of the second decoding submodule is connected to the output end of the first channel cross module; the input ends of the third decoding submodule and the fourth decoding submodule are connected to the output end of the second space cross module; the input ends of the fourth decoding submodule and the fourth decoding submodule are connected to the output end of the first space cross module; and the decoding submodule is used to reconstruct the output of the cross module.

6. A target detection device for remote sensing images, characterized in that: include: A model building module, for building a deep learning network including an encoding module, a multi-head cross attention module and a decoding module connected in sequence; the encoding module includes a parallel convolutional neural network and a Transformer, and a cross module connected to the output end of the convolutional neural network and the output end of the Transformer at the same time; A model training module is used to collect remote sensing images containing targets to construct a data set; the deep learning network is trained using the data set to obtain a salient target detection model; The target detection module is used to input the remote sensing image to be detected into the salient target detection model, extract the features of the remote sensing image through the convolutional neural network and Transformer, and obtain the local features and global features of the remote sensing image; the local features and the global features are cross-interchanged along different dimensions through the cross module to cross-complementarily perceive the local features and the global features of the remote sensing image, and obtain cross-local features and cross-global features; the contribution of the cross-local features and the cross-global features to the targets in the remote sensing image is learned from multiple aspects through multiple heads in the multi-head cross attention module, and the weights of the cross-local features and the cross-global features that respond to the targets in the remote sensing image are dynamically expanded according to the learning results to obtain balanced features; The decoder decodes the balanced features to obtain the salient targets in the remote sensing image.

7. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

8. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of claims 1 to 5 is implemented.

Images