Method, device, equipment and storage medium for target extraction in open space

Abstract

The present application relates to the field of image processing technology, and discloses a method, device, equipment and storage medium for target extraction in open space. The method includes: obtaining a remote sensing image of a target to be extracted, and extracting the target from the remote sensing image using a pre-built target extraction model; the target extraction model includes: an encoder constructed from top to bottom based on an overlapping embedding module and n large selective kernel network modules, and a decoder constructed from top to bottom based on a segmentation head module and n deformable dual-path attention modules; the encoder and the decoder form a U-shaped architecture. The method provided in the present application adopts the U-shaped architecture of the encoder-decoder, the encoder part introduces a lightweight large selective kernel network as the backbone network for feature extraction, and the decoder implements multi-scale learning based on the deformable dual-path attention mechanism, thereby improving the accuracy of target extraction in open space.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for extracting targets in open spaces. Background Art

[0002] The objects in open spaces are complex and diverse, with significant scale differences. Since objects in complex open spaces are highly heterogeneous and mostly have irregular shapes, it poses a huge challenge to target extraction.

[0003] At present, in the field of remote sensing image processing, traditional deep learning methods based on convolutional neural networks (CNN) and transformer architecture have been widely used. However, most of the existing methods focus on extracting features within a fixed range, often unable to take into account both global and local features, and the segmentation accuracy is limited; in addition, in the scenario of high-precision open space, the existing methods are prone to pixel misclassification due to the limited receptive field.

[0004] Therefore, it is urgent to propose a target extraction method suitable for open space extraction tasks, which can flexibly adapt to the complex geometric transformations in open spaces and improve the accuracy of target extraction. Summary of the invention

[0005] To solve the above problems, the present application provides a method, apparatus, device and storage medium for target extraction in open spaces, adopts a U-shaped architecture of encoder-decoder, the encoder part introduces a lightweight large selective kernel network as the backbone network for feature extraction, and the decoder realizes multi-scale learning based on a deformable dual-path attention mechanism, thereby improving the target extraction accuracy in open spaces.

[0006] This application adopts the following technical solutions:

[0007] In a first aspect, a method for extracting an object in an open space is provided, comprising:

[0008] Acquire a remote sensing image of the target to be extracted;

[0009] Use pre-built target extraction models to extract targets from remote sensing images;

[0010] The target extraction model includes: an encoder constructed from top to bottom based on an overlapping embedding module and n large selective kernel network modules, and a decoder constructed from top to bottom based on a segmentation head module and n deformable dual-path attention modules; wherein the n large selective kernel network modules correspond one-to-one to the n deformable dual-path attention modules, and n is an integer greater than or equal to 4;

[0011] The output of the overlapping embedding module is used as the input of the top large selective kernel network module, the output of the top large selective kernel network module is used as the input of the bottom large selective kernel network module, the output of the bottom large selective kernel network module is used as the input of the bottom deformable dual-path attention module, the output of the bottom deformable dual-path attention module and the output of the large selective kernel network module corresponding to the top deformable dual-path attention module are used as the input of the top deformable dual-path attention module through jump connections, and the output of the top deformable dual-path attention module is used as the input of the segmentation head module.

[0012] Optionally, each large selective kernel network module includes: a block embedding submodule and a large selective kernel network submodule arranged in sequence; the large selective kernel network submodule includes: a large kernel selection subblock and a feedforward network subblock of residual connection; the large kernel selection subblock includes: a first fully connected layer, a first GeLU activation function, a large kernel selection layer, and a second fully connected layer arranged in sequence; the feedforward network subblock includes: a third fully connected layer, a deep convolution layer, a second GeLU activation function, and a fourth fully connected layer arranged in sequence;

[0013] Block embedding submodule, used for embedding transformation;

[0014] The large kernel selector block is used to dynamically adjust the receptive field through a large kernel convolution sequence and a spatial selection mechanism;

[0015] Feed-forward network sub-block for channel mixing and feature refinement.

[0016] Optional, large core selection layer for:

[0017] Decomposing the large kernel convolution into multiple depth convolutions with gradually increasing kernel sizes and dilation rates, and sequentially extracting features from the input first original feature map using the multiple depth convolutions to obtain multiple first feature maps;

[0018] Performing dimensionality reduction processing on the multiple first feature maps respectively and splicing them in the channel dimension to generate a second feature map containing multi-scale information;

[0019] Perform average pooling and maximum pooling on the second feature map in the spatial dimension, and use the spatial selection mechanism to generate spatial attention weights;

[0020] According to the spatial attention weights, multiple first feature maps are weighted fused by element-by-element multiplication;

[0021] The fused multiple first feature maps and the first original feature map residual are connected and output.

[0022] Optionally, each deformable dual-path attention module includes: a local specialization submodule, a linear attention submodule and a feature refinement submodule; the local specialization submodule and the linear attention submodule are parallel and both are connected to the feature refinement submodule;

[0023] Local specialization submodule for extracting local features through deformable convolution and channel attention mechanism;

[0024] Linear attention submodule, which is used to extract global features through multi-scale convolution and linear attention mechanism;

[0025] Feature refinement submodule, used to fuse local features and global features through partial convolution and point-by-point convolution.

[0026] Optional, locally specialized submodules for:

[0027] Performing first feature extraction and dimensionality reduction on the second original feature map of the input through a deformable convolution block to obtain a third feature map;

[0028] Performing second feature extraction and dimensionality reduction on the third feature map through a standard convolution block to obtain a fourth feature map;

[0029] Concatenate the third feature map and the fourth feature map in the channel dimension to generate a concatenated feature map;

[0030] After convolution of the concatenated feature map, the channel attention mechanism is used to generate the channel attention map;

[0031] The channel attention map is connected to the residual of the second original feature map and then output.

[0032] Optional, linear attention submodule for:

[0033] The second original feature map of the input is passed through the first linear layer to generate a query vector, a key vector and a value vector;

[0034] Input the query vector, key vector and value vector into three different parallel branches of the aggregation layer respectively, obtain three vector representations processed by the three parallel branches respectively, and integrate the three vector representations;

[0035] The three integrated vector representations are processed using a linear attention mechanism based on ReLU to obtain a fused feature map.

[0036] The fusion feature map is restored to its original dimension through the second linear layer.

[0037] Optional, feature refinement submodule, for:

[0038] Integrate local features and global features into integrated features;

[0039] After applying the convolution operation on part of the channels of the integrated features, the channels are reshaped through a 1×1 convolution layer and nonlinearly activated through a ReLU activation function to generate an attention-weighted feature map for output.

[0040] In a second aspect, a target extraction device for an open space is provided, comprising:

[0041] An image acquisition unit, used for acquiring a remote sensing image of a target to be extracted;

[0042] A target extraction unit, used for extracting targets from remote sensing images using a pre-built target extraction model;

[0043] The target extraction model includes: an encoder constructed from top to bottom based on an overlapping embedding module and n large selective kernel network modules, and a decoder constructed from top to bottom based on a segmentation head module and n deformable dual-path attention modules; wherein the n large selective kernel network modules correspond one-to-one to the n deformable dual-path attention modules, and n is an integer greater than or equal to 4;

[0044] The output of the overlapping embedding module is used as the input of the top large selective kernel network module, the output of the top large selective kernel network module is used as the input of the bottom large selective kernel network module, the output of the bottom large selective kernel network module is used as the input of the bottom deformable dual-path attention module, the output of the bottom deformable dual-path attention module and the output of the large selective kernel network module corresponding to the top deformable dual-path attention module are used as the input of the top deformable dual-path attention module through jump connections, and the output of the top deformable dual-path attention module is used as the input of the segmentation head module.

[0045] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for extracting targets in an open space when executing the computer program.

[0046] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is instructed by a processor, the computer program implements the steps of the above-mentioned target extraction method for open space.

[0047] At least one of the above technical solutions adopted in this application can achieve the following beneficial effects:

[0048] The target extraction model for open space target extraction provided by this application combines a large selective kernel network and deformable dual-path attention, balances performance and operating efficiency, and achieves a wide receptive field and efficient multi-scale learning. The introduction of a lightweight large selective kernel network as the backbone network expands the effectiveness of the large selective kernel network in open space extraction tasks, thereby overcoming the limitations of traditional networks in terms of receptive field. The use of local and global dual-path attention in parallel overcomes the shortcomings of traditional attention mechanisms that are difficult to take into account global and local features. Using the target extraction model proposed in this application to extract targets from remote sensing images can better cope with the irregular shapes of open spaces and improve the accuracy of open space target extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0050] Figure 1 A schematic flow chart of a method for extracting a target in an open space according to an embodiment of the present application is shown;

[0051] Figure 2 A schematic diagram showing the architecture of a target extraction model according to an embodiment of the present application;

[0052] Figure 3 A schematic diagram showing the architecture of a large selective core network module according to an embodiment of the present application;

[0053] Figure 4 A schematic diagram showing the architecture of a large core selection layer according to an embodiment of the present application;

[0054] Figure 5 A schematic diagram showing the architecture of a deformable dual-path attention module according to an embodiment of the present application is shown;

[0055] Figure 6 A schematic diagram showing the architecture of a local specialized submodule according to an embodiment of the present application;

[0056] Figure 7 A schematic diagram showing the architecture of a linear attention submodule according to an embodiment of the present application is shown;

[0057] Figure 8 A schematic diagram showing the architecture of a feature refining submodule according to an embodiment of the present application;

[0058] Fig. 9 A schematic structural diagram of a target extraction device for an open space according to an embodiment of the present application is shown;

[0059] Fig.10 A schematic structural diagram of a computer device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0061] Open spaces are crucial for urban planning. However, due to the high variability and complexity of ground objects, traditional convolutional and attention-based methods face limitations in accurately extracting these spaces. In particular, the significant scale differences between different types of urban spaces make models with limited receptive fields prone to pixel misclassification, resulting in reduced accuracy. To address these challenges, this application proposes a target extraction model for target extraction in remote sensing images of open spaces, aiming to achieve high-precision target extraction in open spaces.

[0062] First, the target extraction model uses a lightweight large selective kernel network (LSKNet) as the backbone network. By dynamically adjusting the receptive field, it can efficiently process the extensive spatial context content of complex objects. Secondly, the target extraction model introduces a deformable dual-path attention module (DDA), which combines the local specialization submodule and the linear attention submodule to achieve fine-grained modeling of local features and global features in parallel. This model overcomes the defect that the traditional attention mechanism cannot take into account local features, and significantly improves the extraction accuracy of urban open spaces.

[0063] Figure 1 FIG. 1 is a flow chart of a method for extracting a target in an open space according to an embodiment of the present application. Figure 1 As shown, the method includes steps S110 to S120:

[0064] Step S110, obtaining a remote sensing image of the target to be extracted;

[0065] Step S120, extracting targets from remote sensing images using a pre-built target extraction model.

[0066] The remote sensing image of the target to be extracted is input into the pre-built target extraction model to achieve target extraction of the remote sensing image.

[0067] Figure 2FIG. 1 is a schematic diagram showing the architecture of a target extraction model proposed in one embodiment of the present application. Figure 2 As shown, the target extraction model includes: an encoder constructed from top to bottom based on an overlapping embedding module and n large selective kernel network modules, and a decoder constructed from top to bottom based on a segmentation head module and n deformable dual-path attention modules; wherein the n large selective kernel network modules correspond one-to-one to the n deformable dual-path attention modules, and n is an integer greater than or equal to 4;

[0068] The output of the overlapping embedding module is used as the input of the top large selective kernel network module, the output of the top large selective kernel network module is used as the input of the bottom large selective kernel network module, the output of the bottom large selective kernel network module is used as the input of the bottom deformable dual-path attention module, the output of the bottom deformable dual-path attention module and the output of the large selective kernel network module corresponding to the top deformable dual-path attention module are used as the input of the top deformable dual-path attention module through jump connections, and the output of the top deformable dual-path attention module is used as the input of the segmentation head module.

[0069] The target extraction module adopts a U-shaped architecture of encoder-decoder.

[0070] The encoder is built based on the overlapping embedding module and the large selective kernel network module. The overlapping embedding module expands the channels of the input remote sensing image. The large selective kernel network module uses the large selective kernel network as the backbone network to perform feature extraction on the input remote sensing image by downsampling.

[0071] The decoder is built on the segmentation head module and the deformable dual-path attention module. The dual paths of the deformable dual-path attention module are parallelized to achieve multi-scale learning. While the decoder is upsampling layer by layer, the jump connection splices the results processed by the deformable dual-path attention module with the results processed by the corresponding large selective kernel network module from the encoder, and then outputs the segmentation result through the segmentation head module.

[0072] Reference Figure 2 As shown, the encoder is constructed based on an overlapping embedding module, a first large selective kernel network module, a second large selective kernel network module, a third large selective kernel network module, and a fourth large selective kernel network module. The decoder is constructed based on a segmentation head module, a first deformable dual-path attention module, a second deformable dual-path attention module, a third deformable dual-path attention module, and a fourth deformable dual-path attention module.

[0073] The overlapping embedding module may include a 3×3 convolutional layer. For the input three-channel remote sensing image, after the channel is expanded by the 3×3 convolutional layer, it is output to the first large selective kernel network module.

[0074] The output of the first large selective nuclear network module is used as the input of the second large selective nuclear network module, the output of the second large selective nuclear network module is used as the input of the third large selective nuclear network module, and the output of the third large selective nuclear network module is used as the input of the fourth large selective nuclear network module.

[0075] The output of the fourth large selective kernel network module serves as the input of the fourth deformable dual-path attention module.

[0076] The output of the fourth deformable dual-path attention module and the output of the third large selective kernel network module are used as the input of the third deformable dual-path attention module through a jump connection, the output of the third deformable dual-path attention module and the output of the second large selective kernel network module are used as the input of the second deformable dual-path attention module through a jump connection, and the output of the second deformable dual-path attention module and the output of the first large selective kernel network module are used as the input of the first deformable dual-path attention module through a jump connection.

[0077] The output of the first deformable dual-path attention module is used as the input of the segmentation head module.

[0078] Each module in the target extraction model has its own corresponding input and output, and the input and output have formats corresponding to the functions of each module. Although "input" and "output" are used here to represent them uniformly, it should be understood that different modules have different inputs and different modules have different outputs. For example: the input of the overlapping embedding module is a remote sensing image, and the output of the overlapping embedding module is a remote sensing image after the expansion channel; the input of the first large selective kernel network module is a remote sensing image after the expansion channel, and the output of the first large selective kernel network module is a feature map after a stage of downsampling feature extraction; the input and output of other modules will not be repeated.

[0079] In the present application, a remote sensing image of a target to be extracted is obtained, and a pre-built target extraction model is used to extract the target from the remote sensing image. The target extraction model includes: an encoder constructed from top to bottom based on an overlapping embedding module and n large selective kernel network modules, and a decoder constructed from top to bottom based on a segmentation head module and n deformable dual-path attention modules; wherein the n large selective kernel network modules correspond one to one to the n deformable dual-path attention modules, and n is an integer greater than or equal to 4; the output of the overlapping embedding module is used as the input of the uppermost large selective kernel network module, the output of the upper large selective kernel network module is used as the input of the lower large selective kernel network module, the output of the lower large selective kernel network module is used as the input of the lowermost deformable dual-path attention module, the output of the lower deformable dual-path attention module, and the output of the large selective kernel network module corresponding to the upper deformable dual-path attention module are used as the input of the upper deformable dual-path attention module through a jump connection, and the output of the uppermost deformable dual-path attention module is used as the input of the segmentation head module.

[0080] The target extraction model provided in this application combines a large selective kernel network and deformable dual-path attention, balances performance and operating efficiency, and achieves a wide receptive field and efficient multi-scale learning. The introduction of a lightweight large selective kernel network as the backbone network expands the effectiveness of the large selective kernel network in open space extraction tasks, thereby overcoming the limitations of traditional networks in terms of receptive field. The use of local and global dual-path attention in parallel overcomes the shortcomings of traditional attention mechanisms that are difficult to take into account global and local features. Using the target extraction model proposed in this application to extract targets from remote sensing images can better cope with the irregular shapes of open spaces and improve the accuracy of open space target extraction.

[0081] The following is a detailed description of the detailed architecture of the target extraction model proposed in this application.

[0082] In some optional embodiments, each large selective kernel network module includes: a block embedding submodule and a large selective kernel network submodule arranged in sequence; the large selective kernel network submodule includes: a large kernel selection subblock and a feedforward network subblock with residual connection; the large kernel selection subblock includes: a first fully connected layer, a first GeLU activation function, a large kernel selection layer, and a second fully connected layer arranged in sequence; the feedforward network subblock includes: a third fully connected layer, a deep convolutional layer, a second GeLU activation function, and a fourth fully connected layer arranged in sequence.

[0083] Block embedding submodule, used for embedding transformation;

[0084] The large kernel selector block is used to dynamically adjust the receptive field through a large kernel convolution sequence and a spatial selection mechanism;

[0085] Feed-forward network sub-block for channel mixing and feature refinement.

[0086] Figure 3 FIG. 1 shows a schematic diagram of the architecture of a large selective core network module proposed in one embodiment of the present application. Figure 3 As shown in the figure, each large selective kernel network module includes a block embedding submodule (Overlap Patch Embed) and a large selective kernel network submodule (LSKNet Block); the large selective kernel network submodule includes a large kernel selection subblock (LK Selection) of residual connection and a feedforward network subblock (FFN), the large kernel selection subblock includes the first normalization layer before it, and the feedforward network subblock includes the second normalization layer before it; the large kernel selection subblock includes: the first fully connected layer, the first GeLU activation function, the large kernel selection layer and the second fully connected layer; the feedforward network subblock includes: the third fully connected layer, the deep convolution layer, the second GeLU activation function and the fourth fully connected layer.

[0087] The input corresponding to the corresponding large selective kernel network module is first embedded through the block embedding submodule, and then output to the large selective kernel network submodule for downsampling feature extraction. The large kernel selection subblock has a spatial selection mechanism and can extract features in different receptive fields. The feedforward network module can further refine the features and perform dimensionality conversion.

[0088] The large kernel selection sub-block constructs a large convolution kernel through a series of deep convolution layers. The key is to obtain different receptive fields by gradually increasing the kernel size and dilation rate of the deep convolution.

[0089] Specifically, the large kernel selection layer is used to: decompose the large kernel convolution into multiple deep convolutions with gradually increasing kernel sizes and expansion rates, use multiple deep convolutions to sequentially extract features from the first original feature map, and obtain multiple first feature maps; perform dimensionality reduction processing on the multiple first feature maps respectively and splice them in the channel dimension to generate a second feature map containing multi-scale information; perform average pooling and maximum pooling on the second feature map in the spatial dimension, and use the spatial selection mechanism to generate spatial attention weights; according to the spatial attention weights, perform weighted fusion of the multiple first feature maps by element-by-element multiplication; residually connect the fused multiple first feature maps and the first original feature map, perform a second convolution operation, and then output.

[0090] Figure 4 FIG. 1 is a schematic diagram showing the architecture of a large core selection layer proposed in an embodiment of the present application. Figure 4 As shown, the structure and function of the large nuclear selection layer are explained.

[0091] Before being processed by the large kernel selection layer, the input first passes through the first fully connected layer for the first convolution operation to obtain the first original feature map.

[0092] The large kernel convolution is decomposed into multiple depth convolutions (such as the first depth convolution and the second depth convolution) with gradually increasing kernel sizes and dilation rates. For example, a depth convolution with a kernel size of 5 and a dilation rate of 1, and a depth convolution with a kernel size of 7 and a dilation rate of 3 have a receptive field of 23. This is the same as the receptive field of a large kernel convolution with a kernel size of 23 and a dilation rate of 1. Multiple depth convolutions are used to sequentially extract features from the first original feature map to obtain multiple first feature maps.

[0093] Each depth convolution can be connected to a 1×1 convolution for dimensionality reduction, and then spliced ​​in the channel dimension to generate a second feature map containing multi-scale information.

[0094] Channel-level average pooling and maximum pooling are applied to extract the spatial relationship of the second feature map. The spatial pooling features are concatenated through the convolution layer to convert the pooling features of the two channels into N spatial attention feature maps. Then the Sigmoid activation function can be applied to each spatial attention feature map to obtain the spatial attention weight corresponding to each deep convolution.

[0095] Multiple first feature maps and spatial attention weights are weighted fused by element-by-element multiplication to obtain attention features.

[0096] The fused multiple first feature maps (i.e., attention features) are residually connected to the first original feature map to achieve the effect of selectively enhancing or suppressing features at specific locations, ensuring that the model focuses on the most important areas.

[0097] After being processed by the large kernel selection layer, the residually connected attention features and the first original feature map are subjected to a second convolution operation through the second fully connected layer to form the output.

[0098] In addition, in order to avoid excessive grid effects, an upper limit can be set for the expansion rate to ensure continuity. Compared with the traditional large kernel convolution construction method, the above method can show higher efficiency in processing spatial information while reducing the number of parameters.

[0099] In some optional embodiments, each deformable dual-path attention module includes: a local specialization submodule, a linear attention submodule and a feature refinement submodule; the local specialization submodule is parallel to the linear attention submodule and both are connected to the feature refinement submodule;

[0100] Local specialization submodule for extracting local features through deformable convolution and channel attention mechanism;

[0101] Linear attention submodule, which is used to extract global features through multi-scale convolution and linear attention mechanism;

[0102] Feature refinement submodule, used to fuse local features and global features through partial convolution and point-by-point convolution.

[0103] Figure 5 FIG. 1 shows a schematic diagram of the architecture of a deformable dual-path attention module proposed in one embodiment of the present application. Figure 5 As shown in Figure 2, the parallel local specialization submodule and linear attention submodule are connected to the feature refinement submodule. The main idea of ​​the deformable dual-path attention module is to capture global features using the linear attention submodule and refine local features based on the local specialization submodule.

[0104] The local specialization submodule aims to enhance the model's ability to extract fine-grained local details and improve the recognition of small-scale targets and complex areas. The local specialization submodule is based on deformable convolution and combines the channel attention mechanism to build a multi-layer convolutional network structure. The deformable convolution flexibly adjusts the receptive field, allowing the model to adaptively capture and strengthen complex local features; the channel attention mechanism further selectively enhances the most discerning feature channels.

[0105] Specifically, the local specialization submodule is used to: perform a first feature extraction and dimensionality reduction on the second original feature map of the input through a deformable convolution block to obtain a third feature map; perform a second feature extraction and dimensionality reduction on the third feature map through a standard convolution block to obtain a fourth feature map; splice the third feature map and the fourth feature map in the channel dimension to generate a spliced ​​feature map; after convolving the spliced ​​feature map, use the channel attention mechanism to generate a channel attention map; and output the channel attention map and the second original feature map after residual connection.

[0106] Figure 6 FIG. 1 shows a schematic diagram of the architecture of a local specialized submodule proposed in one embodiment of the present application. Figure 6 As shown, the structure and function of the local specialized submodule are explained.

[0107] The local specialization submodule performs preliminary feature extraction and dimensionality reduction on the second original feature map of the input through a deformable convolution block (DeConv). The deformable convolution block may include: a first DeformConv2d layer, a first ReLU activation function, and a second DeformConv2d layer. The deformable convolution block can dynamically adjust the sampling position according to the geometric structure of the second original feature map, has the advantage of a flexible receptive field, and is more suitable for complex objects and irregular shapes in remote sensing images.

[0108] The local specialization submodule further refines the features of the third feature map through a standard convolution block (Conv) and performs dimensionality reduction again. The standard convolution block may include: the first Conv3×3 layer, the second ReLU activation function, and the second Conv3×3 layer. After this process, the output from the deformable convolution block (the third feature map) and the output from the standard convolution block (the fourth feature map) are concatenated in the channel dimension to generate a multi-scale feature representation.

[0109] After the concatenated feature map is convolved (1×1 convolution and the third ReLU activation function), the channel attention mechanism is used to generate the channel attention map, and then it is convolved 3×3 and activated by the fourth ReLU. The channel attention mechanism adaptively weights the concatenated feature map dynamically to highlight features with higher discriminative power.

[0110] Finally, the channel attention map and the second original feature map are fused through residual connection. The fused channel attention map and the second original feature map can also be adjusted again using a learnable scale parameter to further optimize the representation of the features.

[0111] The global context perception part is designed in parallel through the linear attention submodule and the local feature submodule. The linear attention submodule introduces a lightweight multi-scale attention mechanism, extracts feature information of different scales through a multi-scale convolutional aggregation layer, and combines the ReLU-based linear attention mechanism to weight and integrate the features, achieving a good balance between execution efficiency and model performance.

[0112] Specifically, the linear attention submodule is used to: generate a query vector, a key vector and a value vector by passing the input second original feature map through the first linear layer; input the query vector, the key vector and the value vector into three different parallel branches of the aggregation layer respectively, obtain three vector representations after processing by the three parallel branches respectively, and integrate the three vector representations; use the ReLU-based linear attention mechanism to process the integrated three vector representations to obtain a fused feature map; restore the dimension of the fused feature map through the second linear layer and then output it.

[0113] Figure 7 FIG. 1 shows a schematic diagram of the architecture of a linear attention submodule proposed in one embodiment of the present application. Figure 7 As shown, the structure and function of the linear attention submodule are explained.

[0114] The input second raw feature map first passes through the first linear layer to generate the query vector (Q), key vector (K), and value vector (V), which are then passed to the aggregation layer.

[0115] The aggregation layer consists of three different parallel branches. Two of the branches construct aggregation modules through convolution kernels of different sizes. Each aggregation module consists of a deep convolution (DWConv) and a 1×1 group convolution (Gconv1×1). For example, the first aggregation module consists of a DWConv3×3 layer and a first Gconv1×1 layer, and the second aggregation module consists of a DWConv5×5 layer and a second Gconv1×1 layer. The deep convolutions of the two branches use convolution kernels of different sizes to capture multi-scale contextual information. The corresponding 1×1 group convolutions aggregate the information captured by the deep convolution in the channel dimension to generate multi-scale Q, K, V representations. The third branch directly passes the unprocessed Q, K, V vectors.

[0116] After integrating the Q, K, and V vector representations processed by the three branches, a linear attention mechanism based on ReLU is applied. Q and K are nonlinearly transformed by the ReLU activation function, and then the transpose of the K vector is matrix multiplied with the Q vector to generate the attention weight matrix. After the attention weight matrix is ​​normalized, it is used to perform weighted summation on the V vector to obtain a feature representation that incorporates global context information, namely the fused feature map.

[0117] The fused feature map is mapped back to the original number of channels through the second linear layer, that is, the dimension is restored and then output. The second linear layer may include: a Conv3×3 layer and a BN layer.

[0118] The ReLU-based linear attention mechanism makes full use of the high computational efficiency of matrix multiplication, which not only significantly expands the global receptive field, but also greatly reduces the computational complexity and hardware execution burden. Compared with the traditional softmax attention mechanism, the ReLU-based linear attention mechanism shows higher efficiency. Under similar computational load, the ReLU-based linear attention mechanism executes significantly faster than the softmax attention mechanism on mobile devices. The design of the above linear attention submodule can significantly improve the inference efficiency while maintaining the model performance.

[0119] At the end of the deformable dual-path attention module, features from both local and global branches are fused and further refined based on a combination of partial convolution and point-wise convolution (1×1 convolution).

[0120] Specifically, the feature refinement submodule is used to: integrate local features and global features into integrated features; apply convolution operation on part of the channels of the integrated features, reshape the channels through a 1×1 convolution layer, perform nonlinear activation through a ReLU activation function, generate an attention-weighted feature map and output it.

[0121] Figure 8 FIG. 1 is a schematic diagram showing the architecture of a feature refining submodule proposed in one embodiment of the present application. Figure 8 As shown, the structure and function of the feature refining submodule are explained.

[0122] The output from the local specialization submodule and the output from the linear attention submodule are integrated to form an integrated feature.

[0123] Partial convolution (PConv) is used to apply 3×3 convolution operations only on a portion of the channels of the integrated features, avoiding redundant calculations and frequent memory accesses.

[0124] After the 1×1 convolution layer, the channel is reshaped and the fifth ReLU activation function is used for nonlinear activation to generate the final attention weighted feature map.

[0125] The target extraction model proposed in this application combines a large selective kernel network module and a deformable dual-path attention module in a U-shaped architecture. The application of large selective kernel networks in open space extraction tasks is realized. The wide context captured by the dynamic receptive field of the large selective kernel network is fully utilized to achieve efficient extraction of different spatial context content. The linear attention module and the local specialization module are used to achieve fine-grained modeling of local features and global features in parallel. The introduction of deformable convolution overcomes the defect that the traditional attention mechanism cannot take into account local features, and significantly improves the extraction accuracy of open spaces.

[0126] Fig. 9 A schematic diagram of the structure of a target extraction device for an open space proposed in one embodiment of the present application is shown. Fig. 9 As shown, the device 900 includes:

[0127] An image acquisition unit 910 is used to acquire a remote sensing image of a target to be extracted;

[0128] The target extraction unit 920 is used to extract the target from the remote sensing image using a pre-built target extraction model;

[0129] The target extraction model includes: an encoder constructed from top to bottom based on an overlapping embedding module and n large selective kernel network modules, and a decoder constructed from top to bottom based on a segmentation head module and n deformable dual-path attention modules; wherein the n large selective kernel network modules correspond one-to-one to the n deformable dual-path attention modules, and n is an integer greater than or equal to 4;

[0130] The output of the overlapping embedding module is used as the input of the top deformable dual-path attention module large selective kernel network module, the output of the top large selective kernel network module is used as the input of the bottom deformable dual-path attention module large selective kernel network module, the output of the bottom deformable dual-path attention module large selective kernel network module is used as the input of the bottom deformable dual-path attention module, the output of the bottom deformable dual-path attention module and the output of the deformable dual-path attention module large selective kernel network module corresponding to the top deformable dual-path attention module are used as the input of the top deformable dual-path attention module deformable dual-path attention module through jump connections, and the output of the top deformable dual-path attention module is used as the input of the deformable dual-path attention module segmentation head module.

[0131] In some optional embodiments, each large selective kernel network module includes: a block embedding submodule and a large selective kernel network submodule arranged in sequence; the large selective kernel network submodule includes: a large kernel selection subblock and a feedforward network subblock of residual connection; the large kernel selection subblock includes: a first fully connected layer, a first GeLU activation function, a large kernel selection layer, and a second fully connected layer arranged in sequence; the feedforward network subblock includes: a third fully connected layer, a deep convolution layer, a second GeLU activation function, and a fourth fully connected layer arranged in sequence; a block embedding submodule for performing embedding conversion; a large kernel selection subblock for dynamically adjusting the receptive field through a large kernel convolution sequence and a spatial selection mechanism; and a feedforward network subblock for performing channel mixing and feature refinement.

[0132] In some optional embodiments, the large kernel selection layer is used to: decompose the large kernel convolution into multiple deep convolutions with gradually increasing kernel sizes and expansion rates, and use multiple deep convolutions to sequentially extract features from the input first original feature map to obtain multiple first feature maps; perform dimensionality reduction processing on the multiple first feature maps respectively and splice them in the channel dimension to generate a second feature map containing multi-scale information; perform average pooling and maximum pooling on the second feature map in the spatial dimension, and use a spatial selection mechanism to generate spatial attention weights; according to the spatial attention weights, perform weighted fusion of the multiple first feature maps by element-by-element multiplication; and connect the fused multiple first feature maps and the first original feature map residual and output them.

[0133] In some optional embodiments, each deformable dual-path attention module includes: a local specialization submodule, a linear attention submodule and a feature refinement submodule; the local specialization submodule is parallel to the linear attention submodule and both are connected to the feature refinement submodule; the local specialization submodule is used to extract local features through deformable convolution and channel attention mechanism; the linear attention submodule is used to extract global features through multi-scale convolution and linear attention mechanism; the feature refinement submodule is used to fuse local features and global features through partial convolution and point-by-point convolution.

[0134] In some optional embodiments, the local specialization submodule is used to: perform a first feature extraction and dimensionality reduction on the second original feature map of the input through a deformable convolution block to obtain a third feature map; perform a second feature extraction and dimensionality reduction on the third feature map through a standard convolution block to obtain a fourth feature map; splice the third feature map and the fourth feature map in the channel dimension to generate a spliced ​​feature map; after convolving the spliced ​​feature map, use the channel attention mechanism to generate a channel attention map; and output the channel attention map and the second original feature map residually connected.

[0135] In some optional embodiments, the linear attention submodule is used to: generate a query vector, a key vector and a value vector by passing the input second original feature map through a first linear layer; input the query vector, the key vector and the value vector into three different parallel branches of the aggregation layer respectively, obtain three vector representations after processing by the three parallel branches respectively, and integrate the three vector representations; use a ReLU-based linear attention mechanism to process the integrated three vector representations to obtain a fused feature map; restore the dimension of the fused feature map through a second linear layer and output it.

[0136] In some optional embodiments, the feature refinement submodule is used to: integrate local features and global features into integrated features; apply a convolution operation on a portion of the channels of the integrated features, reshape the channels through a 1×1 convolution layer, perform nonlinear activation through a ReLU activation function, generate an attention weighted feature map, and then output it.

[0137] It should be noted that the above-mentioned target extraction device 900 for open space can implement the above-mentioned target extraction methods for open space one by one, which will not be described in detail.

[0138] Fig.10 A schematic diagram showing the structure of a computer device according to an embodiment of the present application is shown. Fig.10As shown, the internal structure of the computer device may include a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps of the target extraction method for open space are implemented.

[0139] In one embodiment, the computer device provided by the present application includes a memory and a processor, the memory stores a database and a computer program that can be run on the processor, and the processor implements the following steps when executing the computer program:

[0140] Acquire a remote sensing image of the target to be extracted;

[0141] Use pre-built target extraction models to extract targets from remote sensing images;

[0142] The target extraction model includes: an encoder constructed from top to bottom based on an overlapping embedding module and n large selective kernel network modules, and a decoder constructed from top to bottom based on a segmentation head module and n deformable dual-path attention modules; wherein the n large selective kernel network modules correspond one-to-one to the n deformable dual-path attention modules, and n is an integer greater than or equal to 4;

[0143] The output of the overlapping embedding module is used as the input of the top large selective kernel network module, the output of the top large selective kernel network module is used as the input of the bottom large selective kernel network module, the output of the bottom large selective kernel network module is used as the input of the bottom deformable dual-path attention module, the output of the bottom deformable dual-path attention module and the output of the large selective kernel network module corresponding to the top deformable dual-path attention module are used as the input of the top deformable dual-path attention module through jump connections, and the output of the top deformable dual-path attention module is used as the input of the segmentation head module.

[0144] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0145] Acquire a remote sensing image of the target to be extracted;

[0146] Use pre-built target extraction models to extract targets from remote sensing images;

[0147] The target extraction model includes: an encoder constructed from top to bottom based on an overlapping embedding module and n large selective kernel network modules, and a decoder constructed from top to bottom based on a segmentation head module and n deformable dual-path attention modules; wherein the n large selective kernel network modules correspond one-to-one to the n deformable dual-path attention modules, and n is an integer greater than or equal to 4;

[0148] The output of the overlapping embedding module is used as the input of the top large selective kernel network module, the output of the top large selective kernel network module is used as the input of the bottom large selective kernel network module, the output of the bottom large selective kernel network module is used as the input of the bottom deformable dual-path attention module, the output of the bottom deformable dual-path attention module and the output of the large selective kernel network module corresponding to the top deformable dual-path attention module are used as the input of the top deformable dual-path attention module through jump connections, and the output of the top deformable dual-path attention module is used as the input of the segmentation head module.

[0149] It should be noted that the above functions or steps that can be implemented by the computer device or computer-readable storage medium can refer to the relevant description in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0150] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0151] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0152] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A target extraction method for open space, characterized in that: include: Acquire a remote sensing image of the target to be extracted; Extracting targets from the remote sensing image using a pre-built target extraction model; The target extraction model includes: an encoder constructed from top to bottom based on an overlapping embedding module and n large selective kernel network modules, and a decoder constructed from top to bottom based on a segmentation head module and n deformable dual-path attention modules; wherein the n large selective kernel network modules correspond one-to-one to the n deformable dual-path attention modules, and n is an integer greater than or equal to 4; The output of the overlapping embedding module is used as the input of the uppermost large selective kernel network module, the output of the upper large selective kernel network module is used as the input of the lower large selective kernel network module, the output of the lowermost large selective kernel network module is used as the input of the lowermost deformable dual-path attention module, the output of the lower deformable dual-path attention module and the output of the large selective kernel network module corresponding to the upper deformable dual-path attention module are used as the input of the upper deformable dual-path attention module through jump connections, and the output of the uppermost deformable dual-path attention module is used as the input of the segmentation head module; Each of the deformable dual-path attention modules includes: a local specialization submodule, a linear attention submodule and a feature refinement submodule; the local specialization submodule is parallel to the linear attention submodule and both are connected to the feature refinement submodule; The local specialization submodule is used to extract local features through deformable convolution and channel attention mechanism; The linear attention submodule is used to extract global features through multi-scale convolution and linear attention mechanism; The feature refining submodule is used to fuse the local features and the global features through partial convolution and point-by-point convolution.

2. The target extraction method for open space according to claim 1, characterized in that: Each of the large selective kernel network modules comprises: a block embedding submodule and a large selective kernel network submodule arranged in sequence; the large selective kernel network submodule comprises: a large kernel selection subblock and a feedforward network subblock of residual connection; the large kernel selection subblock comprises: a first fully connected layer, a first GeLU activation function, a large kernel selection layer, and a second fully connected layer arranged in sequence; the feedforward network subblock comprises: a third fully connected layer, a deep convolution layer, a second GeLU activation function, and a fourth fully connected layer arranged in sequence; The block embedding submodule is used to perform embedding conversion; The large kernel selection sub-block is used to dynamically adjust the receptive field through a large kernel convolution sequence and a spatial selection mechanism; The feedforward network sub-block is used for channel mixing and feature refinement.

3. The target extraction method for open space according to claim 2, characterized in that: The large nuclear selection layer is used for: Decomposing the large kernel convolution into a plurality of depth convolutions with gradually increasing kernel sizes and dilation rates, and sequentially extracting features from the input first original feature map using the plurality of depth convolutions to obtain a plurality of first feature maps; Performing dimensionality reduction processing on the plurality of first feature maps respectively and concatenating them in the channel dimension to generate a second feature map containing multi-scale information; Performing average pooling and maximum pooling on the second feature map in the spatial dimension, and generating spatial attention weights using a spatial selection mechanism; According to the spatial attention weight, performing weighted fusion on the plurality of first feature maps by element-by-element multiplication; The fused multiple first feature maps and the first original feature map residual are connected and output.

4. The target extraction method for open space according to claim 1, characterized in that: The local specialized submodule is used to: Performing first feature extraction and dimensionality reduction on the second original feature map of the input through a deformable convolution block to obtain a third feature map; Performing second feature extraction and dimensionality reduction on the third feature map through a standard convolution block to obtain a fourth feature map; Splicing the third feature map and the fourth feature map in the channel dimension to generate a spliced ​​feature map; After convolving the concatenated feature map, generating a channel attention map using the channel attention mechanism; The channel attention map and the second original feature map residual are connected and output.

5. The target extraction method for open space according to claim 1, characterized in that: The linear attention submodule is used to: The second original feature map of the input is passed through the first linear layer to generate a query vector, a key vector and a value vector; Input the query vector, the key vector and the value vector into three different parallel branches of the aggregation layer respectively, obtain three vector representations processed by the three parallel branches respectively, and integrate the three vector representations; Using the linear attention mechanism based on ReLU to process the integrated three vector representations to obtain a fused feature map; The dimension of the fused feature map is restored through the second linear layer and then output.

6. The target extraction method for open space according to claim 1, characterized in that: The feature refining submodule is used to: Integrate the local features and the global features into an integrated feature; After applying the convolution operation on a part of the channels of the integrated features, the channels are reshaped through a 1×1 convolution layer and nonlinearly activated through a ReLU activation function to generate an attention weighted feature map and then output.

7. A target extraction device for open space, characterized in that: The device comprises: An image acquisition unit, used for acquiring a remote sensing image of a target to be extracted; A target extraction unit, used for extracting targets from the remote sensing image using a pre-built target extraction model; The target extraction model includes: an encoder constructed from top to bottom based on an overlapping embedding module and n large selective kernel network modules, and a decoder constructed from top to bottom based on a segmentation head module and n deformable dual-path attention modules; wherein the n large selective kernel network modules correspond one-to-one to the n deformable dual-path attention modules, and n is an integer greater than or equal to 4; The output of the overlapping embedding module is used as the input of the uppermost large selective kernel network module, the output of the upper large selective kernel network module is used as the input of the lower large selective kernel network module, the output of the lowermost large selective kernel network module is used as the input of the lowermost deformable dual-path attention module, the output of the lower deformable dual-path attention module and the output of the large selective kernel network module corresponding to the upper deformable dual-path attention module are used as the input of the upper deformable dual-path attention module through jump connections, and the output of the uppermost deformable dual-path attention module is used as the input of the segmentation head module; Each of the deformable dual-path attention modules includes: a local specialization submodule, a linear attention submodule and a feature refinement submodule; the local specialization submodule is parallel to the linear attention submodule and both are connected to the feature refinement submodule; The local specialization submodule is used to extract local features through deformable convolution and channel attention mechanism; The linear attention submodule is used to extract global features through multi-scale convolution and linear attention mechanism; The feature refining submodule is used to fuse the local features and the global features through partial convolution and point-by-point convolution.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the target extraction method for an open space as claimed in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the target extraction method for an open space as claimed in any one of claims 1 to 6 are implemented.

Images