Remote sensing image feature extraction method based on dynamic mixed gradient convolution

Abstract

The present disclosure provides a remote sensing image ground feature extraction method based on dynamic mixed gradient convolution, which can be applied to the technical field of remote sensing image analysis. The method comprises: inputting a remote sensing image into an encoding network of adaptive boundary extraction, using different convolution groups to extract image features of the remote sensing image, and outputting a first feature map and a second feature map respectively, wherein the number of convolutions in a first convolution group used for outputting the first feature map is less than the number of convolutions in a second convolution group used for outputting the second feature map, and the remote sensing image comprises boundary information of ground feature elements; pyramid pooling is performed on the second feature map to obtain a pooled feature map; the boundary information of the first feature map is encoded and enhanced by using the encoding network to obtain an enhanced feature map; long-distance dependency relationships of the enhanced feature map are captured to obtain an extracted feature map; and the extracted feature map and the pooled feature map are fused to generate a target feature map.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of remote sensing image analysis, and more particularly, to a remote sensing image ground feature extraction method based on dynamic mixed gradient convolution. BACKGROUND

[0002] With the rapid development of remote sensing technology, the ground feature information available from remote sensing images is increasingly rich, and therefore remote sensing image ground feature extraction has a wide range of applications in many fields. The purpose of extracting ground features is to accurately delineate the boundaries of ground objects in remote sensing images. In related technologies, deep learning methods convert low-level features into high-level and abstract features through cascading nonlinear mappings, thereby improving the performance of remote sensing image ground feature extraction.

[0003] In the process of implementing the present disclosure, the inventors have found that in related technologies, the existing deep learning methods have at least the following problems: in the process of extracting ground features from remote sensing images, due to the large size, complex scene, and numerous small targets of remote sensing images, and the presence of shadows and occlusions at the boundaries of objects, the prediction accuracy of ground features is low. SUMMARY

[0004] Therefore, the embodiments of the present disclosure provide a remote sensing image ground feature extraction method and device based on dynamic mixed gradient convolution, an electronic device, a storage medium, and a program product.

[0005] One aspect of the embodiments of the present disclosure provides a remote sensing image ground feature extraction method based on dynamic mixed gradient convolution, including: inputting a remote sensing image into an encoding network for adaptive boundary extraction, extracting image features of the remote sensing image using different convolution groups, and outputting a first feature map and a second feature map, respectively, wherein the number of convolutions in a first convolution group used to output the first feature map is less than the number of convolutions in a second convolution group used to output the second feature map, and the remote sensing image includes boundary information of ground features; pyramid pooling the second feature map to obtain a pooled feature map; encoding and enhancing the boundary information of the first feature map using the encoding network to obtain an enhanced feature map; capturing long-distance dependency relationships of the enhanced feature map to obtain an extracted feature map; and fusing the extracted feature map and the pooled feature map to generate a target feature map.

[0006] According to an embodiment of the present disclosure, in the above inputting the remote sensing image into the adaptive boundary extraction coding network, image features of the remote sensing image are extracted by using different convolution groups, and a first feature map and a second feature map are respectively outputted, comprising: performing down-sampling on the remote sensing image by using a first convolution group of the coding network to obtain the first feature map; performing down-sampling on the first feature map by using a second convolution group of the coding network to obtain the second feature map, wherein the number of convolutions in the second convolution group is greater than the number of convolutions in the first convolution group.

[0007] According to an embodiment of the present disclosure, the above performing down-sampling on the remote sensing image by using the first convolution group of the coding network to obtain the first feature map comprises: performing down-sampling on the remote sensing image by using a first dynamic mixed gradient convolution in the first convolution group to obtain an initial feature map; performing down-sampling on the initial feature map by using a second dynamic mixed gradient convolution in the first convolution group to obtain the first feature map.

[0008] According to an embodiment of the present disclosure, the above performing down-sampling on the remote sensing image by using the first dynamic mixed gradient convolution in the first convolution group to obtain the initial feature map comprises: determining a plurality of convolution weight coefficients of the first dynamic mixed gradient convolution based on the remote sensing image, wherein the first dynamic mixed gradient convolution comprises a normal convolution kernel, a first gradient convolution kernel and a second gradient convolution kernel; performing feature extraction on the remote sensing image by using the normal convolution kernel based on the convolution weight coefficients corresponding to the normal convolution kernel to obtain a normal feature; performing feature extraction on the remote sensing image by using the first gradient convolution kernel based on the convolution weight coefficients corresponding to the first gradient convolution kernel to obtain a first gradient feature; performing feature extraction on the remote sensing image by using the second gradient convolution kernel based on the convolution weight coefficients corresponding to the second gradient convolution kernel to obtain a second gradient feature; and performing feature fusion on the normal feature, the first gradient feature and the second gradient feature to generate the initial feature map.

[0009] According to an embodiment of the present disclosure, the above performing down-sampling on the first feature map by using the second convolution group of the coding network to obtain the second feature map comprises: performing down-sampling on the first feature map by using a first dynamic mixed gradient convolution in the second convolution group to obtain a first convolution map; performing coding enhancement on the first convolution map by using a second dynamic mixed gradient convolution in the second convolution group to obtain a second convolution map; and performing coding enhancement on the second convolution map by using a third dynamic mixed gradient convolution in the second convolution group to obtain the second feature map.

[0010] According to an embodiment of the present disclosure, the method further includes: performing coding enhancement on boundary information of the target feature map by using the coding network to obtain a target enhanced feature map; capturing long-distance dependency of the target enhanced feature map to obtain a target extracted feature map; and outputting the target extracted feature map.

[0011] Another aspect of the embodiments of the present disclosure provides a remote sensing image ground feature extraction device based on dynamic hybrid gradient convolution, comprising: an encoding module configured to input a remote sensing image into an adaptive boundary extraction coding network, extract image features of the remote sensing image by using different convolution groups, and output a first feature map and a second feature map, wherein the number of convolutions in a first convolution group used for outputting the first feature map is less than the number of convolutions in a second convolution group used for outputting the second feature map, and the remote sensing image includes boundary information of a ground feature; a pooling module configured to perform pyramid pooling on the second feature map to obtain a pooled feature map; an enhancement module configured to perform coding enhancement on boundary information of the first feature map by using the coding network to obtain an enhanced feature map; a capturing module configured to capture long-distance dependency of the enhanced feature map to obtain an extracted feature map; and a fusion module configured to perform feature fusion on the extracted feature map and the pooled feature map to generate a target feature map.

[0012] Another aspect of the embodiments of the present disclosure provides an electronic device, comprising: one or more processors; a memory configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0013] Another aspect of the embodiments of the present disclosure provides a computer-readable storage medium, which stores computer-executable instructions, and the instructions are used to implement the method as described above when executed.

[0014] Another aspect of the embodiments of the present disclosure provides a computer program product, which comprises computer-executable instructions, and the instructions are used to implement the method as described above when executed.

[0015] According to an embodiment of the present disclosure, the technical means of inputting a remote sensing image into an adaptive boundary extraction coding network and extracting image features of the remote sensing image by using different convolution groups are adopted, so that the boundary information in the remote sensing image is extracted and enhanced. The second feature map is pyramid pooled to obtain more rich context information. The long-distance dependency of the enhanced feature map is captured to extract the spatial detail information of the boundary information in the remote sensing image. Therefore, at least part of the technical problem of low prediction accuracy of ground feature elements caused by complex scenes of remote sensing images is overcome, and the accuracy of boundary prediction of remote sensing images is improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0017] Figure 1 This illustration schematically shows an exemplary system architecture for applying a method and apparatus for extracting ground features from remotely sensed images based on dynamic hybrid gradient convolution, according to embodiments of the present disclosure.

[0018] Figure 2 A flowchart illustrating a method for extracting ground features from remotely sensed images based on dynamic hybrid gradient convolution according to an embodiment of the present disclosure is shown.

[0019] Figure 3 The illustration shows a schematic diagram of remote sensing image downsampling in a method for extracting land features from remote sensing images based on dynamic hybrid gradient convolution according to an embodiment of the present disclosure;

[0020] Figure 4 The flowchart illustrating the feature extraction of remote sensing images in the method for extracting ground features from remote sensing images based on dynamic hybrid gradient convolution according to embodiments of the present disclosure is shown in the diagram.

[0021] Figure 5 A block diagram of a remote sensing image feature extraction apparatus based on dynamic hybrid gradient convolution according to an embodiment of the present disclosure is shown schematically.

[0022] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing a method for extracting ground features from remotely sensed images based on dynamic hybrid gradient convolution, according to embodiments of the present disclosure. Detailed Implementation

[0023] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein, including technical and scientific terms, have the meanings as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the use of any terms herein should not be interpreted as limiting unless otherwise defined. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

[0026] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should generally be interpreted that the meaning of the expression includes at least one of the items (for example, the items A, B, or C alone, a combination of at least two of the items, etc.). In the case of using expressions similar to "at least one of A, B, or C, etc.", it should generally be interpreted that the meaning of the expression includes at least one of the items (for example, the items A, B, or C alone, a combination of at least two of the items, etc.).

[0027] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (such as including but not limited to user personal information) comply with relevant laws and regulations, necessary security measures are taken, and do not violate public order and good customs.

[0028] With the rapid development of remote sensing technology, it is easy to obtain optical remote sensing images, and the information of ground objects that can be obtained from remote sensing images is becoming more and more abundant. The demand for fine interpretation and application of remote sensing images is increasingly urgent. The purpose of ground feature extraction is to assign a ground feature category label to each pixel of a remote sensing image. It not only identifies what ground objects are in the image, but also accurately outlines the boundaries of the objects. Therefore, optical remote sensing image ground feature extraction has a wide range of applications in many fields. However, the commonly used ground feature extraction method uses manual features, which requires rich prior knowledge and experience, and the feature representation ability is very limited, making it difficult to achieve satisfactory results in complex scenes such as remote sensing images.

[0029] The emergence of deep learning has brought a series of revolutionary progress to the field of image segmentation. Deep learning methods convert low-level features into high-level and abstract features through cascading nonlinear mappings, and such high-level features are not easily obtained in previous manual feature extraction methods. The powerful feature learning ability of deep learning improves the performance of optical remote sensing image ground feature extraction.

[0030] However, the prediction results obtained by the existing deep learning method still have the shortcoming of poor boundary prediction performance. On the one hand, the remote sensing image itself has the characteristics of large size, complex scene, numerous small targets, and shadows and occlusions at the boundaries of objects, which greatly increases the difficulty of boundary extraction of the remote sensing image compared with natural scene pictures. On the other hand, the existing ordinary convolution kernel does not have explicit gradient coding constraints in the initialization process, so it is difficult to focus on the extraction of image gradient information in the training process, thereby affecting the accuracy of boundary prediction.

[0031] In the related art, in order to solve the problem of boundary blur, various strategies have been proposed, which can be roughly divided into the following three categories: skip connection, multi-scale context and additional boundary branch. In the first method, low-level features rich in boundary information are embedded into high-level features through skip connection. The second method is based on the multi-scale context method to expand the receptive field to generate semantic feature maps with strong context relationships under the premise of maintaining high resolution, such as pyramid pooling module (PPM) and multi-hole spatial pyramid pooling. The third method involves an additional boundary branch, which designs multiple weighted edge supervision to emphasize useful boundary information.

[0032] However, the above methods still have the problem that boundary information is difficult to be explicitly coded. How to adaptively model the difference between semantic information and boundary information, and at the same time adaptively enhance the extraction of boundary information while ensuring the extraction of semantic information is also a challenge.

[0033] Therefore, embodiments of the present disclosure provide a remote sensing image feature extraction method based on dynamic mixed gradient convolution, a remote sensing image feature extraction device based on dynamic mixed gradient convolution, an electronic device, a readable storage medium and a computer program product. Wherein the remote sensing image feature extraction method based on dynamic mixed gradient convolution comprises: inputting a remote sensing image into an encoding network for adaptive boundary extraction, using different convolution groups to extract image features of the remote sensing image, and outputting a first feature map and a second feature map, wherein the number of convolutions in a first convolution group used to output the first feature map is less than the number of convolutions in a second convolution group used to output the second feature map, and the remote sensing image includes boundary information of a feature element; pyramid pooling is performed on the second feature map to obtain a pooled feature map; the boundary information of the first feature map is encoded and enhanced by the encoding network to obtain an enhanced feature map; long-distance dependency relationships of the enhanced feature map are captured to obtain an extracted feature map; and the extracted feature map and the pooled feature map are fused to generate a target feature map.

[0034] Figure 1 An exemplary system architecture to which the remote sensing image feature extraction method and device based on dynamic mixed gradient convolution can be applied according to embodiments of the present disclosure is schematically shown. It should be noted that, Figure 1The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0035] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links.

[0036] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0037] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0038] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0039] It should be noted that the method for extracting a remote sensing image ground feature element based on a dynamic mixed gradient convolution provided in the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, the device for extracting a remote sensing image ground feature element based on a dynamic mixed gradient convolution provided in the embodiments of the present disclosure can generally be arranged in the server 105. The method for extracting a remote sensing image ground feature element based on a dynamic mixed gradient convolution provided in the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the terminal device 101, 102, 103 and / or the server 105. Accordingly, the device for extracting a remote sensing image ground feature element based on a dynamic mixed gradient convolution provided in the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the terminal device 101, 102, 103 and / or the server 105. Alternatively, the method for extracting a remote sensing image ground feature element based on a dynamic mixed gradient convolution provided in the embodiments of the present disclosure can also be executed by the terminal device 101, 102, or 103, or by other terminal devices different from the terminal device 101, 102, or 103. Accordingly, the device for extracting a remote sensing image ground feature element based on a dynamic mixed gradient convolution provided in the embodiments of the present disclosure can also be arranged in the terminal device 101, 102, or 103, or in other terminal devices different from the terminal device 101, 102, or 103.

[0040] It should be understood that Figure 1 The number of terminal devices, networks and servers in the system is only illustrative. Any number of terminal devices, networks and servers can be provided according to the needs of implementation.

[0041] Figure 2 A flowchart of the method for extracting a remote sensing image ground feature element based on a dynamic mixed gradient convolution according to the embodiments of the present disclosure is schematically shown

[0042] As shown in Figure 2 The method includes operations S201-S205.

[0043] In operation S201, the remote sensing image is input into an encoding network for adaptive boundary extraction, and the image features of the remote sensing image are extracted using different convolution groups, and a first feature map and a second feature map are output respectively, wherein the number of convolutions in a first convolution group used for outputting the first feature map is less than the number of convolutions in a second convolution group used for outputting the second feature map, and the boundary information of the ground feature element is included in the remote sensing image.

[0044] In operation S202, pyramid pooling is performed on the second feature map to obtain a pooled feature map.

[0045] In operation S203, the boundary information of the first feature map is encoded and enhanced using the encoding network to obtain an enhanced feature map.

[0046] In operation S204, long-distance dependencies of the enhanced feature map are captured to obtain an extracted feature map.

[0047] In operation S205, the extracted feature map is fused with the pooled feature map to generate a target feature map.

[0048] According to an embodiment of the present disclosure, a training set composed of multiple frames of optical remote sensing images is obtained, wherein the boundary information of a ground feature element is included in each frame of remote sensing image, and different semantic categories in the remote sensing image correspond to corresponding dense labels. The semantic categories include sea water, farmland, green forest land, house, and breeding farm, etc. A frame of remote sensing image is selected from the training set and input into an encoding network for adaptive boundary extraction. The encoding network can include a first convolutional group and a second convolutional group. Different convolutional groups are used to extract image features of the remote sensing image, and first and second feature maps are output respectively.

[0049] According to an embodiment of the present disclosure, different convolutional groups in the encoding network can be used to convolve the remote sensing image multiple times to extract boundary information in the enhanced remote sensing image, which corresponds to a sharp gradient change in the remote sensing image. The number of convolutions in the first convolutional group and the number of convolutions in the second convolutional group can be adjusted according to training requirements. For example, the remote sensing image can be convolved five times in total by using the first convolutional group and the second convolutional group in the encoding network. The number of convolutions in the first convolutional group used by the first feature map can be twice, and the number of convolutions in the second convolutional group used by the second feature map can be three times.

[0050] According to an embodiment of the present disclosure, the second feature map is pyramid-pooled, and after pooling, it is introduced into the decoder for upsampling, so that the pooled feature map has rich context information and enhances its feature expression. The first feature map is input into the encoding network for adaptive boundary extraction to encode and enhance the boundary information in the first feature map to obtain an enhanced feature map. The enhanced feature map is input into a CSA module (coordinate attention module) to capture its long-distance dependencies to obtain an extracted feature map. The extracted feature map is fused with the pooled feature map, and then the features in the extracted feature map are introduced into the features of the pooled feature map. That is, the down-sampled feature map of the encoding network is introduced into the up-sampled feature map of the decoder, so that the boundary of the category pixel is more accurately predicted. During the fusion process, the two images can be pixel-registered according to the image features of the extracted feature map and the pooled feature map, the pixel values of the extracted feature map can be filtered and corrected according to the pixel intensity values provided by the pooled feature map, and the extracted feature map can be expanded using the overall image features provided by the pooled feature map, etc., so as to generate a target feature map.

[0051] According to an embodiment of the present disclosure, the remote sensing image is input into an encoding network for adaptive boundary extraction, and different convolution groups are used to extract image features of the remote sensing image, so as to enhance the boundary information in the remote sensing image. The second feature map is pyramid-pooled to obtain more rich context information. The long-distance dependency of the enhanced feature map is captured, and the spatial detail information of the boundary information in the remote sensing image is extracted. Therefore, at least part of the technical problem of low prediction accuracy of ground feature elements due to complex scenes of remote sensing images is overcome, and the accuracy of remote sensing image boundary prediction is improved.

[0052] According to an embodiment of the present disclosure, the remote sensing image is input into an encoding network for adaptive boundary extraction, and different convolution groups are used to extract image features of the remote sensing image, so as to enhance the boundary information in the remote sensing image. The second feature map is pyramid-pooled to obtain more rich context information. The long-distance dependency of the enhanced feature map is captured, and the spatial detail information of the boundary information in the remote sensing image is extracted. Therefore, at least part of the technical problem of low prediction accuracy of ground feature elements due to complex scenes of remote sensing images is overcome, and the accuracy of remote sensing image boundary prediction is improved.

[0053] The first convolution group of the encoding network is used to downsample the remote sensing image to obtain the first feature map, and the second convolution group of the encoding network is used to downsample the first feature map to obtain the second feature map. The number of convolutions in the second convolution group is greater than the number of convolutions in the first convolution group.

[0054] According to an embodiment of the present disclosure, assuming that the resolution of the remote sensing image is 16x16, the first convolution group of the encoding network is used to downsample the remote sensing image to generate a first feature map with a resolution of 4x4. The number of convolutions in the first convolution group is twice. The second convolution group of the encoding network is used to downsample the first feature map to generate a second feature map with a resolution of 2x2. The number of convolutions in the first convolution group is three. The encoding network is used to downsample the remote sensing image, and different convolution groups are used to extract image features of the remote sensing image, which effectively extracts and enhances the boundary information in the remote sensing image.

[0055] Figure 3 An illustrative diagram of down-sampling of a remote sensing image in a remote sensing image ground feature element extraction method based on dynamic mixed gradient convolution according to an embodiment of the present disclosure is shown.

[0056] According to an embodiment of the present disclosure, the first convolution group of the encoding network is used to downsample the remote sensing image to obtain the first feature map, which can include the following operations.

[0057] The first dynamic mixed gradient convolution in the first convolution group is used to downsample the remote sensing image to obtain an initial feature map, and the second dynamic mixed gradient convolution in the first convolution group is used to downsample the initial feature map to obtain the first feature map.

[0058] According to an embodiment of the present disclosure, the network body of the adaptive boundary extraction coding network is selected from a ResNet network (Residual Network). All ordinary convolutions in the ResNet are replaced by dynamic hybrid gradient convolutions. As shown in Figure 3 The first dynamic hybrid gradient convolution and the second dynamic hybrid gradient convolution are included in the first convolution group. The first dynamic hybrid gradient convolution in the first convolution group is used to perform first down-sampling on the remote sensing image with a resolution of 16x16 to generate an initial feature map with a resolution of 8x8. The second dynamic hybrid gradient convolution in the first convolution group is used to perform second down-sampling on the initial feature map with a resolution of 8x8 to generate a first feature map with a resolution of 4x4. By replacing all ordinary convolutions with dynamic hybrid gradient convolutions, the coding network can extract more accurate features of the remote sensing image.

[0059] Figure 4 A flowchart of feature extraction of a remote sensing image in a dynamic hybrid gradient convolution-based remote sensing image feature extraction method according to an embodiment of the present disclosure is schematically shown.

[0060] According to an embodiment of the present disclosure, the first dynamic hybrid gradient convolution in the first convolution group is used to down-sample the remote sensing image to obtain an initial feature map, which can include the following operations.

[0061] Based on the remote sensing image, a plurality of convolution weight coefficients of the first dynamic hybrid gradient convolution are determined, the first dynamic hybrid gradient convolution including an ordinary convolution kernel, a first gradient convolution kernel, and a second gradient convolution kernel. Based on the convolution weight coefficients corresponding to the ordinary convolution kernel, the ordinary convolution kernel is used to extract features of the remote sensing image to obtain ordinary features. Based on the convolution weight coefficients corresponding to the first gradient convolution kernel, the first gradient convolution kernel is used to extract features of the remote sensing image to obtain first gradient features. Based on the convolution weight coefficients corresponding to the second gradient convolution kernel, the second gradient convolution kernel is used to extract features of the remote sensing image to obtain second gradient features. The ordinary features, the first gradient features, and the second gradient features are fused to generate the initial feature map.

[0062] According to an embodiment of the present disclosure, as shown in Figure 4As shown, the dynamic mixed gradient convolution is composed of three parts, i.e., a coefficient prediction module 410, a mixed gradient convolution module 420, and a kernel aggregation module 430. The dynamic mixed gradient convolution is composed of one normal convolution kernel and two gradient convolution kernels. One gradient convolution kernel is obtained by rotating the other gradient convolution kernel by 90 degrees. The normal convolution kernel has no parameter constraint in the convolution process, but the gradient convolution kernel has a parameter constraint in the convolution process. In the convolution process, the gradient convolution kernel has a response of 0 when acting on the inside of the target and a response not equal to 0 when acting on the boundary of the target. For adaptive joint modeling of semantic information and boundary information in remote sensing images, inspired by the traditional mixed spatial enhancement method, the normal convolution kernel and the gradient convolution kernel are combined into a dynamic mixed gradient convolution, which is beneficial to the extraction of a large amount of boundary information of remote sensing images.

[0063] According to an embodiment of the present disclosure, since the boundary information of different remote sensing images is different, the mixed gradient convolution should be able to dynamically adjust the weight ratio of the normal convolution kernel and the gradient according to the content of the remote sensing image. The coefficient prediction module 410 generates convolution weight coefficients for the three convolution kernels of the mixed gradient convolution according to the content of the remote sensing image, and the sum of the three convolution weight coefficients is 1. The normal convolution kernel is aggregated into a new efficient operator, so that the dynamic mixed gradient convolution can adaptively enhance the boundary information in the semantic segmentation information stream, which is beneficial to the extraction of the boundary of the remote sensing image.

[0064] According to an embodiment of the present disclosure, in the mixed gradient convolution module 420, the normal convolution kernel is used to extract features from the remote sensing image to obtain normal features. The first gradient convolution kernel is used to extract features from the remote sensing image to obtain first gradient features. The second gradient convolution kernel is used to extract features from the remote sensing image to obtain second gradient features. The kernel aggregation module 430 fuses the normal features, the first gradient features, and the second gradient features to generate an initial feature map. The kernel aggregation module 430 uses a convolution kernel aggregation method to equivalently replace a feature map aggregation method, thereby speeding up the network inference process.

[0065] According to an embodiment of the present disclosure, the second convolution group of the encoding network is used to downsample the first feature map to obtain the second feature map, which can include the following operations.

[0066] The first dynamic mixed gradient convolution in the second convolution group is used to downsample the first feature map to obtain a first convolution map; the second dynamic mixed gradient convolution in the second convolution group is used to encode and enhance the first convolution map to obtain a second convolution map; and the third dynamic mixed gradient convolution in the second convolution group is used to encode and enhance the second convolution map to obtain the second feature map.

[0067] According to an embodiment of the present disclosure, the first feature map with a resolution of 4x4 is down-sampled by using the first dynamic mixed gradient convolution in the second convolution group to obtain a first convolution map with a resolution of 2x2. The first convolution map is encoded and enhanced by using the second dynamic mixed gradient convolution in the second convolution group to obtain a second convolution map with a resolution of 2x2 and enhanced boundary gradient information. The second convolution map is encoded and enhanced by using the third dynamic mixed gradient convolution in the second convolution group to obtain a second feature map with a resolution of 2x2 and enhanced boundary gradient information. The extraction and enhancement of the boundary information of the remote sensing image by using the dynamic mixed gradient convolution are performed to make the prediction of the pixel boundary of the ground object more accurate.

[0068] According to an embodiment of the present disclosure, the method can further include the following operations.

[0069] The boundary information of the target feature map is encoded and enhanced by using the encoding network to obtain a target enhanced feature map; the long-distance dependency relationship of the target enhanced feature map is captured to obtain a target extraction feature map; and the target extraction feature map is output.

[0070] According to an embodiment of the present disclosure, the target feature map is further encoded and enhanced by using the encoding network to obtain a target enhanced feature map. The target enhanced feature map is input into the CSA module to capture the long-distance dependency relationship thereof to obtain a target extraction feature map. The target feature map is a prediction result of the pixel boundary of the ground object, and is output and used for training of the model to make the prediction model more accurate in predicting the pixel boundary of the ground object.

[0071] Figure 5 A block diagram of a remote sensing image ground object element extraction device based on a dynamic mixed gradient convolution according to an embodiment of the present disclosure is schematically shown.

[0072] As shown in Figure 5 The remote sensing image ground object element extraction device 500 based on the dynamic mixed gradient convolution includes an encoding module 510, a pooling module 520, an enhancement module 530, a capturing module 540, and a fusion module 550.

[0073] The encoding module 510 is configured to input a remote sensing image into an encoding network for adaptive boundary extraction, extract image features of the remote sensing image by using different convolution groups, and output a first feature map and a second feature map respectively, wherein the number of convolutions in a first convolution group used for outputting the first feature map is less than the number of convolutions in a second convolution group used for outputting the second feature map, and the remote sensing image includes boundary information of a ground object element.

[0074] The pooling module 520 is configured to pyramid pool the second feature map to obtain a pooled feature map.

[0075] The enhancement module 530 is configured to encode and enhance boundary information of the first feature map by using the encoding network to obtain an enhanced feature map.

[0076] The capture module 540 is configured to capture long-distance dependencies of the enhanced feature map to obtain an extracted feature map.

[0077] The fusion module 550 is configured to perform feature fusion on the extracted feature map and the pooled feature map to generate a target feature map.

[0078] According to an embodiment of the present disclosure, the remote sensing image is input into an encoding network for adaptive boundary extraction, and different convolution groups are used to extract image features of the remote sensing image, so as to extract and enhance boundary information in the remote sensing image. The second feature map is pyramid-pooled to obtain more rich context information. The long-distance dependencies of the enhanced feature map are captured to extract spatial detail information of the boundary information in the remote sensing image. Therefore, at least part of the technical problem of low prediction accuracy of ground feature elements due to complex scenes of remote sensing images is overcome, and the accuracy of boundary prediction of the remote sensing image is improved.

[0079] According to an embodiment of the present disclosure, the encoding module 510 includes a first downsampling sub-module and a second downsampling sub-module.

[0080] The first downsampling sub-module is configured to downsample the remote sensing image by using a first convolution group of the encoding network to obtain a first feature map.

[0081] The second downsampling sub-module is configured to downsample the first feature map by using a second convolution group of the encoding network to obtain a second feature map, and the number of convolutions in the second convolution group is greater than the number of convolutions in the first convolution group.

[0082] According to an embodiment of the present disclosure, the first downsampling sub-module includes a first downsampling unit and a second downsampling unit.

[0083] The first downsampling unit is configured to downsample the remote sensing image by using a first dynamic mixed gradient convolution in the first convolution group to obtain an initial feature map.

[0084] The second downsampling unit is configured to downsample the initial feature map by using a second dynamic mixed gradient convolution in the first convolution group to obtain the first feature map.

[0085] According to an embodiment of the present disclosure, the first downsampling unit includes a weight determination subunit, a first feature extraction subunit, a second feature extraction subunit, a third feature extraction subunit, and a feature fusion subunit.

[0086] The weight determination subunit is configured to determine a plurality of convolution weight coefficients of the first dynamic mixed gradient convolution based on the remote sensing image, and the first dynamic mixed gradient convolution includes a normal convolution kernel, a first gradient convolution kernel, and a second gradient convolution kernel.

[0087] The first feature extraction subunit is configured to perform feature extraction on the remote sensing image by using the normal convolution kernel based on the convolution weight coefficients corresponding to the normal convolution kernel, to obtain normal features.

[0088] The second feature extraction subunit is configured to perform feature extraction on the remote sensing image by using the first gradient convolution kernel based on the convolution weight coefficients corresponding to the first gradient convolution kernel, to obtain first gradient features.

[0089] The third feature extraction subunit is configured to perform feature extraction on the remote sensing image by using the second gradient convolution kernel based on the convolution weight coefficients corresponding to the second gradient convolution kernel, to obtain second gradient features.

[0090] The feature fusion subunit is configured to perform feature fusion on the normal features, the first gradient features, and the second gradient features, to generate an initial feature map.

[0091] According to an embodiment of the present disclosure, the second down-sampling sub-module includes a third down-sampling unit, a first enhancement unit, and a second enhancement unit.

[0092] The third down-sampling unit is configured to perform down-sampling on the first feature map by using the first dynamic mixed gradient convolution in the second convolution group, to obtain a first convolution map.

[0093] The first enhancement unit is configured to perform coding enhancement on the first convolution map by using the second dynamic mixed gradient convolution in the second convolution group, to obtain a second convolution map.

[0094] The second enhancement unit is configured to perform coding enhancement on the second convolution map by using the third dynamic mixed gradient convolution in the second convolution group, to obtain a second feature map.

[0095] Any of the modules, sub-modules, units, sub-units, or at least part of any of them according to embodiments of the present disclosure can be implemented in one module. Any of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure can be split into multiple modules. Any of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure can be implemented at least in part as a hardware circuit, for example, a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System on Chip, a System on Board, a System in Package, an Application Specific Integrated Circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, or in hardware or firmware, or in any one of software, hardware and firmware, or in a proper combination of any of them. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure can be implemented at least in part as computer program modules, which can perform corresponding functions when the computer program modules are run.

[0096] For example, any of the encoding module 510, the pooling module 520, the enhancement module 530, the capturing module 540 and the fusion module 550 can be combined in one module / unit / sub-unit, or any of them can be split into multiple modules / units / sub-units. Alternatively, at least part of one or more of the modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units, and implemented in one module / unit / sub-unit. According to embodiments of the present disclosure, at least one of the encoding module 510, the pooling module 520, the enhancement module 530, the capturing module 540 and the fusion module 550 can be implemented at least in part as a hardware circuit, for example, a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System on Chip, a System on Board, a System in Package, an Application Specific Integrated Circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, or in hardware or firmware, or in any one of software, hardware and firmware, or in a proper combination of any of them. Alternatively, at least one of the encoding module 510, the pooling module 520, the enhancement module 530, the capturing module 540 and the fusion module 550 can be implemented at least in part as computer program modules, which can perform corresponding functions when the computer program modules are run.

[0097] It should be noted that the remote sensing image ground feature extraction device based on dynamic mixed gradient convolution in the embodiments of the present disclosure corresponds to the remote sensing image ground feature extraction method based on dynamic mixed gradient convolution in the embodiments of the present disclosure, and the description of the remote sensing image ground feature extraction device based on dynamic mixed gradient convolution is specifically referred to the remote sensing image ground feature extraction method based on dynamic mixed gradient convolution, which will not be repeated here.

[0098] Figure 6 A block diagram of an electronic device suitable for implementing the remote sensing image ground feature extraction method based on dynamic mixed gradient convolution according to an embodiment of the present disclosure is schematically shown. Figure 6 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0099] As shown in Figure 6 The electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to programs stored in a Read-Only Memory (ROM) 602 or loaded into a Random Access Memory (RAM) 603 from a storage portion 608. The processor 601 may, for example, include a general-purpose microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a special-purpose microprocessor (such as an Application-Specific Integrated Circuit (ASIC)), etc. The processor 601 can also include an on-board memory for cache use. The processor 601 can include a single processing unit or multiple processing units for performing different actions of the method processes according to the embodiments of the present disclosure.

[0100] In the RAM 603, various programs and data required for the operation of the electronic device 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The processor 601 performs various operations of the method processes according to the embodiments of the present disclosure by executing the programs in the ROM 602 and / or the RAM 603. It should be noted that the programs can also be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 can also perform various operations of the method processes according to the embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0101] According to an embodiment of the present disclosure, the electronic device 600 can further include an input / output (I / O) interface 605 that is also connected to the bus 604. The system 600 can further include one or more of the following components connected to the I / O interface 605: an input part 606 including a keyboard, a mouse, etc.; an output part 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 608 including a hard disk, etc.; and a communication part 609 including a network interface card such as a LAN card, a modem, etc. The communication part 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as necessary. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 610 as necessary, so that a computer program read therefrom is installed in the storage part 608 as necessary.

[0102] According to an embodiment of the present disclosure, the method flow according to the embodiment of the present disclosure can be implemented as a computer software program. For example, the embodiment of the present disclosure includes a computer program product including a computer program carried on a computer-readable storage medium, the computer program containing program codes for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network by the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above-described functions defined in the system implementing the embodiment of the present disclosure are performed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0103] The present disclosure also provides a computer-readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, when the one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0104] According to an embodiment of the present disclosure, the computer readable storage medium can be a non-volatile computer readable storage medium. For example, it can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, a portable compact disc read-only memory (CD-ROM), an optical memory device, a magnetic memory device, or any suitable combination of the foregoing. In this disclosure, a computer readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in connection with an instruction execution system, apparatus, or device.

[0105] For example, according to an embodiment of the present disclosure, the computer readable storage medium can include one or more memories other than the ROM 602 and / or the RAM 603 described above.

[0106] Embodiments of the present disclosure also include a computer program product, which includes a computer program containing program codes for executing the method provided by the embodiments of the present disclosure, and when the computer program product is run on an electronic device, the program codes are used to make the electronic device implement the method for extracting features of remote sensing images based on dynamic hybrid gradient convolution provided by the embodiments of the present disclosure.

[0107] When the computer program is executed by the processor 601, the above-mentioned functions defined in the system / apparatus of the embodiments of the present disclosure are executed. According to an embodiment of the present disclosure, the system, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0108] In one embodiment, the computer program can rely on tangible storage media such as optical, magnetic, etc. storage media. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal on a network medium, and be downloaded and installed through the communication part 609, and / or installed from the detachable medium 611. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any suitable combination of the foregoing.

[0109] According to embodiments of the present disclosure, program code of a computer program provided by embodiments of the present disclosure can be written in any combination of one or more programming languages, and specifically, can be implemented using a high-level procedural and / or object-oriented programming language, and / or an assembly / machine language. Programming languages include, but are not limited to, Java, C++, python, "C" language, or similar programming languages. Program code can execute entirely on a user's computing device, partly on a user device, partly on a remote computing device, or entirely on a remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0110] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0111] Embodiments of the present disclosure have been described. However, these embodiments are merely intended to illustrate the present disclosure, and are not intended to limit the scope of the present disclosure. Although each of the embodiments is described above separately, this does not mean that the measures in each of the embodiments cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art can make various substitutions and modifications, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims

1. A remote sensing image ground feature extraction method based on dynamic mixed gradient convolution, comprising: inputting a remote sensing image into an adaptive boundary extraction encoding network, using different convolution groups to extract image features of the remote sensing image, and outputting first and second feature maps, wherein the number of convolutions in a first convolution group used to output the first feature map is less than the number of convolutions in a second convolution group used to output the second feature map, and the remote sensing image includes boundary information of a ground feature; pyramid pooling the second feature map to obtain a pooled feature map; using the encoding network to encode and enhance the boundary information of the first feature map to obtain an enhanced feature map; capturing long-distance dependencies of the enhanced feature map to obtain an extracted feature map; fusing the extracted feature map and the pooled feature map to generate a target feature map; wherein the inputting a remote sensing image into an adaptive boundary extraction encoding network, using different convolution groups to extract image features of the remote sensing image, and outputting first and second feature maps comprises: using a first convolution group of the encoding network to downsample the remote sensing image to obtain the first feature map; using a second convolution group of the encoding network to downsample the first feature map to obtain the second feature map, the number of convolutions in the second convolution group being greater than the number of convolutions in the first convolution group; wherein the using a second convolution group of the encoding network to downsample the first feature map to obtain the second feature map comprises: using a first dynamic mixed gradient convolution in the second convolution group to downsample the first feature map to obtain a first convolution map; using a second dynamic mixed gradient convolution in the second convolution group to encode and enhance the first convolution map to obtain a second convolution map; using a third dynamic mixed gradient convolution in the second convolution group to encode and enhance the second convolution map to obtain the second feature map.

2. The method of claim 1, wherein, the using a first convolution group of the encoding network to downsample the remote sensing image to obtain the first feature map comprises: using a first dynamic mixed gradient convolution in the first convolution group to downsample the remote sensing image to obtain an initial feature map; using a second dynamic mixed gradient convolution in the first convolution group to downsample the initial feature map to obtain the first feature map.

3. The method of claim 2, wherein, the using a first dynamic mixed gradient convolution in the first convolution group to downsample the remote sensing image to obtain an initial feature map comprises: determining a plurality of convolution weight coefficients of the first dynamic mixed gradient convolution based on the remote sensing image, the first dynamic mixed gradient convolution including a normal convolution kernel, a first gradient convolution kernel, and a second gradient convolution kernel; extracting features of the remote sensing image using the normal convolution kernel based on the convolution weight coefficients corresponding to the normal convolution kernel to obtain normal features; extracting features of the remote sensing image using the first gradient convolution kernel based on the convolution weight coefficients corresponding to the first gradient convolution kernel to obtain first gradient features; perform feature extraction on the remote sensing image by using the second gradient convolution kernel, to obtain a second gradient feature; perform feature fusion on the normal feature, the first gradient feature and the second gradient feature, to generate the initial feature map.

4. The method of claim 1, further comprising: performing coding enhancement on boundary information of the target feature map by using the coding network, to obtain a target enhanced feature map; capture long-distance dependency relationships of the target enhanced feature map, to obtain a target extracted feature map; output the target extracted feature map.

5. A device for extracting ground feature elements of a remote sensing image based on dynamic mixed gradient convolution, comprising: an encoding module configured to input a remote sensing image into a coding network for adaptive boundary extraction, extract image features of the remote sensing image by using different convolution groups, and output a first feature map and a second feature map, wherein a number of convolutions in a first convolution group used for outputting the first feature map is less than a number of convolutions in a second convolution group used for outputting the second feature map, and the remote sensing image comprises boundary information of ground feature elements; a pooling module configured to perform pyramid pooling on the second feature map, to obtain a pooled feature map; an enhancement module configured to perform coding enhancement on boundary information of the first feature map by using the coding network, to obtain an enhanced feature map; a capture module configured to capture long-distance dependency relationships of the enhanced feature map, to obtain an extracted feature map; a fusion module configured to perform feature fusion on the extracted feature map and the pooled feature map, to generate a target feature map; a first downsampling sub-module configured to perform downsampling on the remote sensing image by using a first convolution group of the coding network, to obtain the first feature map; a second downsampling sub-module configured to perform downsampling on the first feature map by using a second convolution group of the coding network, to obtain the second feature map, wherein a number of convolutions in the second convolution group is greater than a number of convolutions in the first convolution group; a third downsampling unit configured to perform downsampling on the first feature map by using a first dynamic mixed gradient convolution in the second convolution group, to obtain a first convolution map; a first enhancement unit configured to perform coding enhancement on the first convolution map by using a second dynamic mixed gradient convolution in the second convolution group, to obtain a second convolution map; a second enhancement unit configured to perform coding enhancement on the second convolution map by using a third dynamic mixed gradient convolution in the second convolution group, to obtain the second feature map.

6. An electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-4.

7. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to implement the method of any one of claims 1-4.

8. A computer program product comprising a computer program which, when executed by a processor, is adapted to implement the method of any one of claims 1 to 4.

Images