Remote sensing image transmission method and device, electronic equipment and storage medium

Abstract

The invention provides a remote sensing image transmission method. The method comprises the following steps: acquiring a remote sensing image; dividing the remote sensing image into a plurality of image blocks with set sizes, and obtaining an image sequence according to the image blocks; respectively inputting each image block contained in the image sequence into a convolutional neural network based on a residual network structure to obtain feature maps of corresponding semantic hierarchies respectively output by a plurality of stages contained in the convolutional neural network; wherein the feature map of each semantic hierarchy corresponds to different resolutions and channel numbers, and the feature map of each semantic hierarchy fuses spatial features of different scales; and carrying out channel coding on a feature map queue formed by the feature maps of the semantic hierarchies so as to transmit a coding result in a channel. According to the invention, only the data having practical significance to the target detection task is transmitted, so that the data transmission amount during remote sensing image transmission is significantly reduced, the bandwidth limitation problem during remote sensing image transmission is solved, and both the remote sensing image transmission efficiency and the detection precision are considered.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of image transmission technology, and in particular, to a remote sensing image transmission method, device, electronic device, and storage medium. Background Art

[0002] Remote sensing images are information about the earth's surface collected by high-altitude platforms such as satellites and drones, and are characterized by high resolution and large coverage. Such images usually contain rich geographic and environmental information, but due to the particularity of the shooting angle, the target objects in the image are often small in size, sparsely distributed, and the background is complex and diverse, resulting in a huge amount of data and extremely high resolution for remote sensing images, which poses great challenges in storage, processing and transmission.

[0003] In some embodiments, the original image data of the remote sensing image is directly transmitted to the ground station or other processing equipment through the wireless communication channel. However, due to the huge amount of data of the remote sensing image, the transmission process needs to occupy a large amount of bandwidth resources, and bandwidth limitation becomes the main bottleneck. In addition, there are interference factors such as Gaussian white noise and multipath fading in the wireless communication channel, and the image data is easily distorted or lost during the transmission process. If data compression and channel coding are used to reduce the amount of transmitted data and enhance the robustness of the transmission, it is difficult to fully extract and transmit semantic information that is of practical significance to the target detection task while ensuring the image quality, resulting in difficulty in balancing transmission efficiency and detection accuracy. Summary of the invention

[0004] The present disclosure provides a remote sensing image transmission method, the method comprising: Acquisition of remote sensing images; Dividing the remote sensing image into a plurality of image blocks of set sizes, and obtaining an image sequence according to the image blocks; Inputting each image block contained in the image sequence into a convolutional neural network based on a residual network structure, respectively, to obtain feature maps of corresponding semantic levels output by multiple stages contained in the convolutional neural network; wherein the feature maps of each semantic level correspond to different resolutions and numbers of channels, and the feature maps of each semantic level fuse spatial features of different scales; The feature map queue formed by the feature maps of each semantic level is channel-coded to transmit the coding result in the channel.

[0005] Optionally, dividing the remote sensing image into a plurality of image blocks of set sizes includes: Generating a sliding window covering the entire image on the remote sensing image; For each sliding window, when the ratio of pixels of the covered remote sensing image to pixels contained in the sliding window reaches a preset threshold, the sliding window is retained; The image block is obtained according to the image area corresponding to the retained sliding window.

[0006] Optionally, the step of inputting each image block contained in the image sequence into a convolutional neural network based on a residual network structure to obtain feature maps of corresponding semantic levels outputted by multiple stages contained in the convolutional neural network comprises: Inputting each image block included in the image sequence into a convolutional neural network based on a residual network structure respectively, and in each stage included in the convolutional neural network, converting the image block input to the stage into a patch embedding feature with an overlapping area; The patch embedding feature is downsampled and convolved to reduce the resolution of the patch embedding feature and increase the number of channels, thereby generating a feature map of the semantic level corresponding to the stage.

[0007] Optionally, after generating a feature map of a semantic level corresponding to the stage, the method further includes: The generated feature map of the semantic level corresponding to the stage is passed into multiple residual blocks contained in the stage. In each residual block, spatial features of different scales of the feature map are extracted through multiple layers of depth-separable convolution and spatial attention mechanism, so as to obtain spatial features of different scales of the feature map output by the multiple residual blocks; The spatial features of different scales in the feature map are adjusted by the number of channels and up-sampled to obtain a feature map of the corresponding semantic level that integrates spatial features of different scales.

[0008] Optionally, performing channel coding on the feature graph queue formed by the feature graphs of each semantic level to transmit the coding result in the channel includes: The feature map of the corresponding semantic level that integrates the spatial features of different scales is converted into a two-dimensional matrix to obtain a flattened feature map of the corresponding semantic level that integrates the spatial features of different scales; The flattened feature maps of the corresponding semantic levels that fuse the spatial features of different scales outputted at each stage are concatenated in scale order to obtain a feature map queue finally outputted by the convolutional neural network; Channel coding is performed on the feature map queue to transmit the coding result in the channel.

[0009] Optionally, in each residual block, spatial features of different scales of the feature map are extracted through multiple layers of depth-separable convolution and spatial attention mechanism, including: In each residual block, spatial features of different scales of the feature map are extracted through multiple layers of depth-wise separable convolution; Aggregate the extracted spatial features of different scales, and perform average pooling and maximum pooling on the aggregated spatial features, so as to obtain the spatial attention weights corresponding to the spatial features of different scales of the feature map; The spatial attention weight is multiplied element-by-element with the feature map of the semantic level corresponding to the stage to extract the weighted spatial features of the feature map at different scales.

[0010] Optionally, before dividing the remote sensing image into a plurality of image blocks of a set size and obtaining an image sequence according to the image blocks, the method further includes: The resolution of the acquired remote sensing images; If the resolution of the acquired remote sensing image is lower than the preset resolution, zero pixels are added to the edge of the acquired remote sensing image to expand the resolution of the acquired remote sensing image to the preset resolution.

[0011] The present disclosure also provides a remote sensing image transmission device, the device comprising: An acquisition unit, used for acquiring remote sensing images; A division unit, used for dividing the remote sensing image into a plurality of image blocks of a set size, and obtaining an image sequence according to the image blocks; An input unit, used to input each image block contained in the image sequence into a convolutional neural network based on a residual network structure, to obtain feature maps of corresponding semantic levels respectively output by multiple stages contained in the convolutional neural network; wherein the feature maps of each semantic level correspond to different resolutions and numbers of channels, and the feature maps of each semantic level fuse spatial features of different scales; The encoding unit is used to perform channel encoding on the feature map queue formed by the feature maps of each semantic level to transmit the encoding result in the channel.

[0012] The present disclosure also provides an electronic device, comprising a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the above method by calling the machine-readable instructions.

[0013] The present disclosure also provides a machine-readable storage medium, wherein the machine-readable storage medium stores machine-readable instructions, and when the machine-readable instructions are called and executed by a processor, the above method is implemented.

[0014] According to the embodiment of the present disclosure, first, a remote sensing image is acquired, the remote sensing image is divided into a plurality of image blocks of a set size, and an image sequence is obtained according to the image blocks; then, each image block contained in the image sequence is respectively input into a convolutional neural network based on a residual network structure, and feature maps of corresponding semantic levels respectively output by the plurality of stages contained in the convolutional neural network are obtained; wherein, the feature maps of each semantic level correspond to different resolutions and numbers of channels, and the feature maps of each semantic level fuse spatial features of different scales; finally, channel coding is performed on the feature map queue formed by the feature maps of each semantic level to transmit the coding result in the channel. Accordingly, the present disclosure can realize the transmission of only data that is of practical significance to the target detection task, thereby significantly reducing the amount of data transmission during remote sensing image transmission, solving the bandwidth limitation problem in remote sensing image transmission, and at the same time, through multi-scale feature extraction and fusion, ensuring that the semantic information and spatial features required for the target detection task are fully transmitted, improving the accuracy of target detection, and taking into account the transmission efficiency and detection performance of remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0016] Figure 1 is a schematic diagram of the architecture of a remote sensing image transmission system shown in an exemplary embodiment; Figure 2 is a flow chart of a remote sensing image transmission method shown in an exemplary embodiment; Figure 3 is a flow chart of another remote sensing image transmission method shown in an exemplary embodiment; Figure 4 is a hardware structure diagram of an electronic device shown in an exemplary embodiment; Figure 5 It is a block diagram of a remote sensing image transmission device shown in an exemplary embodiment. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0018] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0019] Remote sensing images are information about the earth's surface collected by high-altitude platforms such as satellites and drones, and are characterized by high resolution and large coverage. Such images usually contain rich geographic and environmental information, but due to the particularity of the shooting angle, the target objects in the image are often small in size, sparsely distributed, and the background is complex and diverse, resulting in a huge amount of data and extremely high resolution for remote sensing images, which poses great challenges in storage, processing and transmission.

[0020] Please refer to Figure 1 , Figure 1 FIG. 1 is a schematic diagram of the architecture of a remote sensing image transmission system shown in an exemplary embodiment of the present disclosure. Figure 1 As shown, the remote sensing image transmission system includes a satellite camera device 100, which is usually carried on a satellite and is used to obtain remote sensing images of a target detection object 300 on the ground in real time; and a ground receiving device 200, which is set on the ground and is used to receive remote sensing images transmitted by the satellite camera device 100 in real time. The remote sensing image transmission system can transmit the dynamic image of the target detection object 300 monitored by satellite remote sensing back to the ground in real time by carrying the satellite camera device 100 and the data transmission device on the satellite, and cooperate with the ground receiving device 200 to receive the remote sensing image transmitted back to the ground for subsequent target detection and analysis.

[0021] In some embodiments, the satellite camera transmits the original image data of the remote sensing image directly to the ground station or other processing equipment through the wireless communication channel. However, due to the huge amount of data of the remote sensing image, the transmission process needs to occupy a large amount of bandwidth resources, and the bandwidth limitation becomes the main bottleneck. In addition, there are interference factors such as Gaussian white noise and multipath fading in the wireless communication channel, and the image data is easily distorted or lost during the transmission process. If data compression and channel coding are used to reduce the amount of transmitted data and enhance the robustness of the transmission, it is difficult to fully extract and transmit semantic information that is of practical significance to the target detection task while ensuring the image quality, resulting in difficulty in balancing transmission efficiency and detection accuracy.

[0022] In view of this, this specification aims to propose a technical solution for transmitting multi-scale features of remote sensing images to reduce the amount of data transmission and ensure detection accuracy.

[0023] During implementation, first, a remote sensing image is acquired, the remote sensing image is divided into multiple image blocks of set sizes, and an image sequence is obtained based on the image blocks; then, each image block contained in the image sequence is input into a convolutional neural network based on a residual network structure, and feature maps of corresponding semantic levels output by multiple stages contained in the convolutional neural network are obtained; wherein the feature maps of each semantic level correspond to different resolutions and numbers of channels, and the feature maps of each semantic level fuse spatial features of different scales; finally, channel coding is performed on the feature map queue formed by the feature maps of each semantic level to transmit the coding result in the channel.

[0024] For example, the processor of the satellite camera divides the remote sensing image into multiple image blocks of 1024×1024 (i.e., height H=1024, width W=1024) size to generate an image sequence based on the multiple image blocks. Then, the processor of the satellite camera inputs each image block with a resolution of 1024×1024 (H×W) contained in the image sequence into a convolutional neural network based on a residual network structure. The network contains 4 stages, and each stage extracts features layer by layer through different convolution operations and multi-scale processing modules. Specifically, the resolution of the feature map output in the first stage is 256×256 (H / 4×W / 4) and the number of channels is 64; the resolution of the feature map output in the second stage is 128×128 (H / 8×W / 8) and the number of channels is 128; the resolution of the feature map output in the third stage is 64×64 (H / 16×W / 16) of the image block and the number of channels is 256; the resolution of the feature map output in the fourth stage is 32×32 (H / 32×W / 32) of the image block and the number of channels is 512. The specific values ​​of the number of channels in different stages are determined according to the network design, and gradually increase with the increase of network depth to capture richer semantic information. Moreover, in each stage, the receptive field of feature extraction is dynamically adjusted to capture multi-scale spatial features from local details to global semantics. Finally, after obtaining the feature maps of each semantic level output in different stages, the processor of the satellite camera device splices these feature maps into a queue form, and transmits the feature map queue in the channel after channel coding.

[0025] It can be seen that the present invention can realize the transmission of only data that is of practical significance to the target detection task, thereby significantly reducing the amount of data transmitted during remote sensing image transmission, solving the bandwidth limitation problem in remote sensing image transmission, and at the same time, through multi-scale feature extraction and fusion, ensuring that the semantic information and spatial features required for the target detection task are fully transmitted, thereby improving the accuracy of target detection and taking into account both the remote sensing image transmission efficiency and detection performance.

[0026] The present disclosure is described below through specific embodiments in combination with specific application scenarios.

[0027] Please refer to Figure 2 , Figure 2 The flowchart of a remote sensing image transmission method is shown in an exemplary embodiment. The method can perform the following steps: Step 202: Acquire remote sensing images.

[0028] For example, the processor of the satellite camera device obtains a remote sensing image of a target detection object on the ground captured by the camera of the satellite camera device, and the resolution of the remote sensing image is 4096×4096 (ie, height H=4096, width W=4096).

[0029] Among them, remote sensing images are images of the earth's surface collected by high-altitude platforms such as satellites or drones, and have the characteristics of high resolution and large coverage. The resolution of 4096×4096 means that the height (H) of the remote sensing image is 4096 pixels, the width (W) is 4096 pixels, and the total number of pixels is 4096×4096, which is a high-resolution image. Target detection refers to specific targets that need to be detected in remote sensing images, such as buildings, vehicles, vegetation, etc.

[0030] Step 204: Divide the remote sensing image into a plurality of image blocks of a set size, and obtain an image sequence according to the image blocks.

[0031] For example, a processor of a satellite camera divides a remote sensing image into a plurality of image blocks of a size of 1024×1024 (ie, height H=1024, width W=1024) to generate an image sequence according to the plurality of image blocks.

[0032] The high-resolution remote sensing image is divided into multiple smaller image blocks for easy subsequent processing and transmission. The sequence of multiple image blocks is used to input into the convolutional neural network for feature extraction.

[0033] Step 206: Input each image block contained in the image sequence into a convolutional neural network based on a residual network structure to obtain feature maps of corresponding semantic levels output by multiple stages contained in the convolutional neural network; wherein the feature maps of each semantic level correspond to different resolutions and numbers of channels, and the feature maps of each semantic level fuse spatial features of different scales.

[0034] For example, the processor of the satellite camera device inputs each image block with a resolution of 1024×1024 (H×W) contained in the image sequence into a convolutional neural network based on the residual network structure. The network contains 4 stages, and each stage extracts features layer by layer through different convolution operations and multi-scale processing modules. Specifically, the resolution of the feature map output by the first stage is 256×256 (H / 4×W / 4) and the number of channels is 64; the resolution of the feature map output by the second stage is 128×128 (H / 8×W / 8) and the number of channels is 128; the resolution of the feature map output by the third stage is 64×64 (H / 16×W / 16) of the image block and the number of channels is 256; the resolution of the feature map output by the fourth stage is 32×32 (H / 32×W / 32) of the image block and the number of channels is 512. The specific values ​​of the number of channels at different stages are determined according to the network design, and gradually increase with the increase of network depth to capture richer semantic information. Moreover, in each stage, the receptive field of feature extraction is dynamically adjusted to capture multi-scale spatial features from local details to global semantics.

[0035] Among them, the residual network structure is a deep convolutional neural network structure, which solves the gradient vanishing problem in the deep network by introducing residual connections and improves the feature extraction capability. In this embodiment, the convolutional neural network based on the residual network structure is divided into multiple stages, and each stage gradually extracts the semantic features of different levels of the image block through convolution operations. Semantic features at different levels correspond to different feature map resolutions, that is, the spatial size of the feature map. As the network depth increases, the resolution gradually decreases (such as from resolution 256×256 to resolution 32×32). The number of channels represents the depth of the feature map. As the network depth increases, the number of channels gradually increases to capture richer semantic information. Multi-scale spatial features are multi-scale spatial features from local details (such as edges, textures) to global semantics (such as target shape, contextual information) through multiple layers of deep separable convolution operations on the semantic feature map of each level. Dynamically adjusting the receptive field is to dynamically adjust the receptive field of spatial feature extraction according to the target characteristics through deep separable convolution and spatial attention mechanism, so as to efficiently capture multi-scale spatial features.

[0036] Step 208: Channel coding is performed on the feature map queue formed by the feature maps of each semantic level to transmit the coding result in the channel.

[0037] For example, after obtaining the feature maps of each semantic level outputted at different stages, the processor of the satellite camera device splices these feature maps into a queue form, performs channel encoding on the feature map queue, and transmits it in the channel.

[0038] Among them, the feature map queue is to splice the feature maps of different semantic levels into a queue in order, which is convenient for unified processing and transmission of feature maps. Channel coding is to encode the feature map queue to improve the robustness and efficiency of data transmission. Channel transmission is to transmit the encoded data to the ground receiving device through a wireless channel. The channel transmission process may be interfered by noise, multipath fading, etc. Channel coding can effectively reduce data loss and distortion.

[0039] In one embodiment shown, dividing the remote sensing image into multiple image blocks of set sizes includes: generating a sliding window covering the entire image on the remote sensing image; for each sliding window, retaining the sliding window when the ratio of pixels of the covered remote sensing image to pixels contained in the sliding window reaches a preset threshold; and obtaining the image block based on the image area corresponding to the retained sliding window.

[0040] For example, the resolution of the acquired remote sensing image is × , a sliding window covering the entire image is generated on the acquired remote sensing image, the sliding window size is H×W, and the step size is ( H, W), the coordinates of the upper left corner of the sliding window are ( H, W), where: ∈{0, 1, …, H}, ∈{0, 1, …, ( −W) / W}. For each sliding window, calculate the ratio of the pixels of the intersection with the acquired remote sensing image to the pixels contained in the sliding window itself. If the ratio reaches the preset threshold of 0.9, it means that the sliding window contains enough pixels of the remote sensing image, and the sliding window is retained. Finally, an image block of size H×W is generated according to the image area of ​​the remote sensing image corresponding to the retained sliding window.

[0041] Among them, if the ratio of the pixels at the intersection of the sliding window and the acquired remote sensing image to the pixels contained in the sliding window itself is lower than the threshold, the sliding window will be discarded, thereby effectively reducing the amount of calculation for subsequent extraction of image block features, while ensuring the effective availability of the image block, laying the foundation for subsequent feature extraction and target detection tasks.

[0042] In one embodiment shown, each image block contained in the image sequence is input into a convolutional neural network based on a residual network structure to obtain feature maps of corresponding semantic levels output by multiple stages contained in the convolutional neural network, including: each image block contained in the image sequence is input into a convolutional neural network based on a residual network structure, and in each stage contained in the convolutional neural network, the image blocks input to the stage are converted into patch embedding features with overlapping areas; the patch embedding features are downsampled and convolved to reduce the resolution of the patch embedding features and increase the number of channels, thereby generating a feature map of the semantic level corresponding to the stage.

[0043] For example, the image blocks contained in the image sequence have a resolution of 1024×1024 and a channel number of 3. The image blocks contained in the image sequence are respectively input into a convolutional neural network based on a residual network structure. The convolutional neural network contains multiple stages. Each stage includes an OverlapPatchEmbed layer, which converts the image blocks into patch embedding features with overlapping areas. The patch embedding features are downsampled through convolution operations to reduce the resolution and increase the number of channels. The output is a feature map of the semantic level with a resolution of 256×256 and 64 channels.

[0044] Among them, the convolutional neural network based on the residual network structure is divided into multiple stages, and each stage gradually extracts features of different semantic levels through convolution operations. The height and width of the feature map of each stage are determined by the convolution kernel size, convolution step size and padding size, which can be preset as needed. As the depth of the network increases, the resolution of the feature map of each stage gradually decreases and the number of channels gradually increases, thereby capturing multi-scale spatial features from local details to global semantics.

[0045] In the shallow stage, the resolution of the feature map is high (such as 1024×1024), which means that the spatial size of the feature map is large. The high-resolution feature map retains more spatial detail information and can accurately describe the local structure of the image (such as edges, textures, etc.). At the same time, the number of channels in the shallow stage is small (such as 3), which means that the depth of the feature map is shallow. The small number of channels means that the expressive power of the feature map is limited and it is mainly used to capture basic local features. The convolution kernel in the shallow stage is usually small, the receptive field is small, and it can focus on the local area of ​​the image. Due to the high resolution, the feature map can accurately describe the local details of the image (such as edges, corners, textures, etc.).

[0046] In the deep stage, the feature map has a lower resolution (such as 256×256), which means that the spatial size of the feature map is smaller. The low-resolution feature map loses some spatial detail information, but retains higher-level semantic information. At the same time, the deep stage has a larger number of channels (such as 64), which means that the feature map is deeper. A larger number of channels means that the feature map has stronger expressive power and can capture more complex semantic information. The convolution kernels in the deep stage are usually larger and have a larger receptive field, which can cover a larger area of ​​the image and thus capture global semantic information.

[0047] In this embodiment, the image blocks are converted into patch embedding features with overlapping areas, so that part of the pixel information can be shared between adjacent patches, so that context information can be better captured during the feature extraction process. Context information is crucial for target detection tasks, especially in remote sensing images where the target objects are small in size, sparsely distributed, and have complex backgrounds. If there is no overlap between patches, the target object may be segmented into different patches, making it difficult for the model to capture complete context information. By introducing overlapping areas, the context information of the target object can be captured in multiple patches, improving detection accuracy, reducing boundary information loss, enhancing feature continuity, supporting multi-scale feature extraction, and improving robustness.

[0048] In one embodiment shown, after generating a feature map of the semantic level corresponding to the stage, the method further includes: passing the generated feature map of the semantic level corresponding to the stage into multiple residual blocks contained in the stage, and in each residual block, extracting spatial features of different scales of the feature map through multiple layers of deep separable convolution and spatial attention mechanism, thereby obtaining spatial features of different scales of the feature map output by the multiple residual blocks; adjusting the number of channels and upsampling the spatial features of the feature map at different scales to obtain a feature map of the corresponding semantic level that integrates spatial features of different scales.

[0049] For example, the current stage is the third stage. After the OverlapPatchEmbed layer included in this stage, the feature map of the corresponding semantic level has a resolution of 64×64 and a channel number of 256. The feature map is passed to multiple residual blocks (such as 4 residual blocks) included in this stage. Each residual block extracts spatial features of different scales of the feature map through multiple layers (such as two layers) of depth-separable convolution and spatial attention mechanism. Among them, the first layer of convolution kernel size is 5×5, and the expansion rate is 1, which can capture local details. The second layer of convolution kernel size is 7×7, and the expansion rate is 3, which can capture a wider range of contextual information. In each residual block, the spatial attention mechanism is also used to dynamically adjust the receptive field of feature extraction. After the spatial features of different scales are extracted, the number of channels of the spatial features of different scales output by multiple residual blocks are adjusted (such as using 1×1 convolution to unify the number of channels to 256). The low-resolution feature map is upsampled to make its resolution consistent with the high-resolution feature. Figure 1 The upsampled feature map is added element by element to the high-resolution feature map to obtain the feature map of the corresponding semantic level that integrates the spatial features of different scales in the third stage.

[0050] Among them, the residual block is a deep convolutional neural network structure, which solves the gradient vanishing problem in the deep network by introducing residual connections and improves the feature extraction capability. Depthwise separable convolution is an efficient convolution operation that decomposes the standard convolution into depthwise convolution and pointwise convolution to reduce the amount of calculation and the number of parameters. In this embodiment, multiple layers of depthwise separable convolution are used to extract spatial features of different scales. The spatial attention mechanism is an attention mechanism that dynamically adjusts the receptive field of feature extraction by generating attention weights to enhance the expression of important feature areas. Upsampling is an image processing technique that increases the resolution of a low-resolution feature map to the same resolution as a high-resolution feature map through interpolation methods. Figure 1 To.

[0051] In this embodiment, multi-layer deep separable convolution and spatial attention mechanism are used in the residual block to extract spatial features of different scales in the feature map, and multi-scale feature fusion is achieved through channel adjustment and upsampling to generate a feature map that contains both local details and global semantic information.

[0052] In some implementations, the final step of each stage can also stabilize the data distribution through a normalization layer to obtain a feature map of the corresponding semantic level with stable data distribution and fusion of spatial features of different scales. This method can effectively support multi-scale target detection tasks of remote sensing images, while avoiding the transmission of complete remote sensing images and minimizing the amount of data transmission, while improving the accuracy and efficiency of target detection.

[0053] In one embodiment shown, the feature map queue formed by the feature maps of each semantic level is channel encoded to transmit the encoding result in the channel, including: converting the feature map of the corresponding semantic level that integrates spatial features of different scales into a two-dimensional matrix to obtain a flattened feature map of the corresponding semantic level that integrates spatial features of different scales; splicing the flattened feature maps of the corresponding semantic level that integrate spatial features of different scales output by each stage in scale order to obtain a feature map queue finally output by the convolutional neural network; channel encoding the feature map queue to transmit the encoding result in the channel.

[0054] For example, a convolutional neural network contains four stages. The feature map output by the first stage has a resolution of 256×256 and 64 channels, which is flattened into a two-dimensional matrix of 64×65536 (64 channels, 256×256=65536 pixels); the feature map output by the second stage has a resolution of 128×128 and 128 channels, which is flattened into a two-dimensional matrix of 128×16384 (128 channels, 128×128=16384 pixels); the feature map output by the third stage has a resolution of 64×64 and 256 channels, which is flattened into a two-dimensional matrix of 256×4096 (256 channels, 64×64=4096 pixels); the feature map output by the fourth stage has a resolution of 32×32 and 512 channels, which is flattened into a two-dimensional matrix of 512×1024 (512 channels, 32×32=1024 pixels). Next, the two-dimensional matrices flattened at each stage are concatenated in scale order (from high resolution to low resolution). For example, the two-dimensional matrices of 64×65536, 128×16384, 256×4096, and 512×1024 are concatenated in order to obtain the final feature map queue containing semantic information at each level. Finally, the concatenated feature map queue is channel coded to transmit the coded data to the receiving end through the wireless channel.

[0055] Among them, feature map flattening is to convert the three-dimensional feature map (height H×width W×number of channels C) into a two-dimensional matrix (number of channels C×H×W resolution) for subsequent processing and transmission. Channel coding is a technology used to improve the robustness of data transmission. Wireless channel transmission is to transmit the encoded data to the receiving end through the wireless channel. This process may be interfered by noise, multipath fading, etc. In this embodiment, channel coding can effectively reduce data loss and distortion, thereby efficiently transmitting multi-scale feature information in the channel.

[0056] In one embodiment shown, in each residual block, spatial features of different scales of the feature map are extracted through multiple layers of deep separable convolution and spatial attention mechanism, including: in each residual block, spatial features of different scales of the feature map are extracted through multiple layers of deep separable convolution; the extracted spatial features of different scales are aggregated, and the aggregated spatial features are average pooled and maximum pooled respectively, so as to obtain spatial attention weights corresponding to the spatial features of different scales of the feature map; the spatial attention weights are element-wise multiplied with the feature map of the semantic level corresponding to the stage to extract the weighted spatial features of the feature map at different scales.

[0057] For example, the input feature map of the current residual block has a resolution of 64×64 and a channel number of 256. Two layers of depthwise separable convolution are used to extract the spatial features of the feature map. The convolution kernel size of the first layer of depthwise separable convolution is 5×5, and the dilation rate is 1, which can capture the local details of the feature map. The convolution kernel size of the second layer of depthwise separable convolution is 7×7, and the dilation rate is 3, which can capture a wider range of contextual information of the feature map. The feature maps output by the two layers of convolution are concatenated to generate aggregated spatial features, and then the aggregated spatial features are average pooled and max pooled respectively. The results of average pooling and max pooling are concatenated, and processed through convolution operations and Sigmoid activation functions to generate the final spatial attention weights. The generated spatial attention weights are element-wise multiplied with the input feature map to obtain the weighted spatial features of the feature map at different scales.

[0058] Among them, different areas in the feature map contribute differently to the target detection task. Some areas (such as target edges, textures, etc.) may contain more useful information, while other areas (such as background) may contain less useful information. Directly using the original feature map may cause the convolutional neural network to pay insufficient attention to important areas, thereby affecting the detection accuracy. Average pooling is to calculate the average value of each area of ​​the feature map to capture global information. Maximum pooling is to calculate the maximum value of each area of ​​the feature map to capture significant features. The final spatial attention weight is generated through convolution operation (used to capture local spatial information in the feature map and generate attention weights related to local areas) and Sigmoid activation function processing (used to map the output of the convolution operation to the range of [0, 1] to generate normalized attention weights) and multiply it element by element with the input feature map. This is to dynamically adjust the receptive field of feature extraction, enhance the expression of important feature areas, and suppress the expression of unimportant feature areas, thereby improving the model's feature extraction ability and target detection accuracy.

[0059] In some embodiments, each residual block may also include an MLP (Multi-Layer Perceptron), which usually appears as a submodule in the residual block. It enhances the expressiveness and generalization capabilities of the network model through operations such as fully connected layers, activation functions, and DropPath, thereby improving the effect of feature extraction. MLP combined with the spatial attention mechanism can effectively support multi-scale target detection tasks in remote sensing images.

[0060] In one embodiment shown, before dividing the remote sensing image into multiple image blocks of set sizes and obtaining an image sequence based on the image blocks, the method also includes: collecting the resolution of the acquired remote sensing image; if the resolution of the acquired remote sensing image is lower than a preset resolution, adding zero pixels to the edge of the acquired remote sensing image to expand the resolution of the acquired remote sensing image to the preset resolution.

[0061] For example, if the resolution of the remote sensing image acquired by the processor of the satellite camera device is 800×800, which is lower than the preset resolution of 1024×1024, 112 zero pixels are added to the upper, lower, left, and right edges of the image (because (1024-800) / 2=112). The image resolution after addition is 1024×1024, which meets the preset resolution requirement.

[0062] Among them, if the resolution of the acquired remote sensing image does not reach the preset value, such as the resolution of the acquired remote sensing image does not reach 1024×1024, it is necessary to supplement the acquired remote sensing image with 0 pixels to expand the resolution of the remote sensing image to the preset value of 1024×1024 to ensure the resolution of the input image is consistent. Because if the resolution of the input image is insufficient, the resolution of the feature map will not meet the preset parameters, affecting the multi-scale feature extraction. In this way, the problem of size mismatch can be avoided in the subsequent operation of extracting multi-level features from remote sensing images in the convolutional neural network.

[0063] After the satellite camera device 100 performs channel coding on the feature map queue formed by the feature maps of each semantic level to transmit the coding result in the channel, there is a corresponding process of receiving the feature map queue and target parsing.

[0064] like Figure 1 As shown, the ground receiving device 200 receives the encoding result transmitted by the satellite camera device 100, and restores the encoding result to obtain feature maps of each semantic level; then, a number of candidate areas are generated in the feature maps of each semantic level, and each candidate area is classified and regressed into a bounding box to obtain a processed candidate box for target detection; finally, the features of the candidate box for target detection are extracted, and based on the extracted features, the prediction result of the candidate box for target detection is calculated, and the prediction result includes the category information and corresponding confidence of the target, such as the confidence corresponding to a car in a candidate box in the remote sensing image is 0.03, and the confidence corresponding to a building is 0.87.

[0065] In order to help those skilled in the art better understand this method, Figure 3 A complete description of this method is given.

[0066] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of another remote sensing image transmission method according to an exemplary embodiment. Figure 3As shown in the figure, the satellite camera device and the ground receiving device jointly complete the remote sensing image processing, transmission and target detection tasks. The satellite camera device first obtains the collected remote sensing image. If the pixel value of the remote sensing image does not reach the preset pixel value, the remote sensing image is padded with zero pixels; the remote sensing image after zero padding is divided into image blocks, and an image sequence is formed based on the image blocks. The image blocks contained in the image sequence are respectively input into the backbone network (convolutional neural network based on residual network structure) to obtain feature maps of different semantic levels, and extract spatial feature maps of different scales from the feature maps of each semantic level. The feature pyramid network is a network structure for multi-scale feature extraction, which is usually used in combination with the backbone network. It fuses spatial feature maps of different resolutions through top-down and lateral connections to generate multi-scale spatial feature representations, so that the feature maps of each semantic level fuse spatial features of different scales. Then, each feature map is flattened and spliced ​​to obtain a feature map queue, and channel coding and real-to-complex conversion are performed to convert it into a complex signal to meet the transmission requirements of the wireless channel, thereby facilitating transmission in the physical channel. After receiving the coding results transmitted in the physical channel, the ground receiving device performs complex to real number conversion and channel decoding to restore the feature maps of different semantic levels. The feature maps of different semantic levels are then passed into the RPN (Region Proposal Network) and the detection head, and candidate regions are generated in each feature map and classified and bounding box regressed. Finally, the category information and corresponding confidence of the target are output to complete the target detection task.

[0067] Corresponding to the above-mentioned embodiment of the remote sensing image transmission method, this specification also provides an embodiment of a remote sensing image transmission device.

[0068] See also Figure 4 , Figure 4 It is a hardware structure diagram of an electronic device shown in an exemplary embodiment. At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408 and a non-volatile memory 410, and of course may also include other required hardware. One or more embodiments of this specification can be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0069] See also Figure 5 , Figure 5FIG. 5 is a block diagram of a remote sensing image transmission device according to an exemplary embodiment. The remote sensing image transmission device 500 can be applied to Figure 4 In the electronic device shown in the figure, the technical solution of this specification is implemented. The device includes: An acquisition unit 502 is used to acquire remote sensing images; A division unit 504 is used to divide the remote sensing image into a plurality of image blocks of a set size, and obtain an image sequence according to the image blocks; An input unit 506 is used to input each image block contained in the image sequence into a convolutional neural network based on a residual network structure, so as to obtain feature maps of corresponding semantic levels output by multiple stages contained in the convolutional neural network; wherein the feature maps of each semantic level correspond to different resolutions and numbers of channels, and the feature maps of each semantic level integrate spatial features of different scales; The encoding unit 508 is used to perform channel encoding on the feature map queue formed by the feature maps of each semantic level to transmit the encoding result in the channel.

[0070] In this embodiment, the division unit includes: A first generating subunit is used to generate a sliding window covering the entire image on the remote sensing image; A second generating subunit is used for retaining each sliding window when the ratio of pixels of the covered remote sensing image to pixels included in the sliding window reaches a preset threshold; The second generating subunit is further used to obtain the image block according to the image area corresponding to the retained sliding window.

[0071] In this embodiment, the input unit includes: A first input subunit is used to input each image block included in the image sequence into a convolutional neural network based on a residual network structure, and in each stage included in the convolutional neural network, convert the image block input to the stage into a patch embedding feature with an overlapping area; A generating subunit is used to downsample and convolve the patch embedding feature so that the resolution of the patch embedding feature is reduced and the number of channels is increased, thereby generating a feature map of the semantic level corresponding to the stage.

[0072] In this embodiment, the device further includes: An input unit is used to input the generated feature map of the semantic level corresponding to the stage into multiple residual blocks contained in the stage, and in each residual block, spatial features of different scales of the feature map are extracted through multi-layer depth-separable convolution and spatial attention mechanism, so as to obtain spatial features of different scales of the feature map output by the multiple residual blocks; The fusion unit is used to adjust the number of channels and upsample the spatial features of different scales of the feature map to obtain a feature map of the corresponding semantic level that fuses the spatial features of different scales.

[0073] In this embodiment, the encoding unit includes: A conversion subunit, used for converting a feature map of a corresponding semantic level that integrates spatial features of different scales into a two-dimensional matrix, to obtain a flattened feature map of a corresponding semantic level that integrates spatial features of different scales; A splicing subunit, used for splicing the flattened feature maps of the corresponding semantic levels that fuse the spatial features of different scales outputted at each stage in scale order, to obtain a feature map queue finally outputted by the convolutional neural network; The encoding subunit is used to perform channel encoding on the feature map queue to transmit the encoding result in the channel.

[0074] In this embodiment, the input unit includes: A first extraction subunit is used to extract spatial features of different scales of the feature map through multiple layers of depth-separable convolution in each residual block; The aggregation subunit is used to aggregate the extracted spatial features of different scales, and perform average pooling and maximum pooling on the aggregated spatial features, so as to obtain the spatial attention weights corresponding to the spatial features of different scales of the feature map; The second extraction subunit is used to multiply the spatial attention weight and the feature map of the semantic level corresponding to the stage element by element to extract the weighted spatial features of the feature map at different scales.

[0075] In this embodiment, the device further includes: An acquisition unit, used for acquiring the resolution of the acquired remote sensing image; The expansion unit is used to add zero pixels to the edge of the acquired remote sensing image if the resolution of the acquired remote sensing image is lower than the preset resolution, so as to expand the resolution of the acquired remote sensing image to the preset resolution.

[0076] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0077] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Ordinary technicians in this field can understand and implement it without paying creative work.

[0078] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0079] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0080] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0081] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0082] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0083] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0084] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0085] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "one", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0086] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0087] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A remote sensing image transmission method, characterized in that: The method comprises: Acquisition of remote sensing images; Dividing the remote sensing image into a plurality of image blocks of set sizes, and obtaining an image sequence according to the image blocks; Inputting each image block contained in the image sequence into a convolutional neural network based on a residual network structure, respectively, to obtain feature maps of corresponding semantic levels output by multiple stages contained in the convolutional neural network; wherein the feature maps of each semantic level correspond to different resolutions and numbers of channels, and the feature maps of each semantic level fuse spatial features of different scales; The feature map queue formed by the feature maps of each semantic level is channel-coded to transmit the coding result in the channel.

2. The method according to claim 1, characterized in that The step of dividing the remote sensing image into a plurality of image blocks of set sizes comprises: Generating a sliding window covering the entire image on the remote sensing image; For each sliding window, when the ratio of pixels of the covered remote sensing image to pixels contained in the sliding window reaches a preset threshold, the sliding window is retained; The image block is obtained according to the image area corresponding to the retained sliding window.

3. The method according to claim 1, characterized in that The step of inputting each image block contained in the image sequence into a convolutional neural network based on a residual network structure to obtain feature maps of corresponding semantic levels outputted by multiple stages contained in the convolutional neural network comprises: Inputting each image block included in the image sequence into a convolutional neural network based on a residual network structure respectively, and in each stage included in the convolutional neural network, converting the image block input to the stage into a patch embedding feature with an overlapping area; The patch embedding feature is downsampled and convolved to reduce the resolution of the patch embedding feature and increase the number of channels, thereby generating a feature map of the semantic level corresponding to the stage.

4. The method according to claim 3, characterized in that After generating a feature map of a semantic level corresponding to the stage, the method further includes: The generated feature map of the semantic level corresponding to the stage is passed into multiple residual blocks contained in the stage. In each residual block, spatial features of different scales of the feature map are extracted through multiple layers of depth-separable convolution and spatial attention mechanism, so as to obtain spatial features of different scales of the feature map output by the multiple residual blocks; The spatial features of different scales in the feature map are adjusted by the number of channels and up-sampled to obtain a feature map of the corresponding semantic level that integrates spatial features of different scales.

5. The method according to claim 4, characterized in that The channel coding of the feature graph queue formed by the feature graphs of each semantic level to transmit the coding result in the channel includes: The feature map of the corresponding semantic level that integrates the spatial features of different scales is converted into a two-dimensional matrix to obtain a flattened feature map of the corresponding semantic level that integrates the spatial features of different scales; The flattened feature maps of the corresponding semantic levels that fuse the spatial features of different scales outputted at each stage are concatenated in scale order to obtain a feature map queue finally outputted by the convolutional neural network; Channel coding is performed on the feature map queue to transmit the coding result in the channel.

6. The method according to claim 4, characterized in that In each residual block, multiple layers of depth-separable convolution and spatial attention mechanism are used to extract spatial features of different scales of the feature map, including: In each residual block, spatial features of different scales of the feature map are extracted through multiple layers of depth-wise separable convolution; Aggregate the extracted spatial features of different scales, and perform average pooling and maximum pooling on the aggregated spatial features, so as to obtain the spatial attention weights corresponding to the spatial features of different scales of the feature map; The spatial attention weight is multiplied element-by-element with the feature map of the semantic level corresponding to the stage to extract the weighted spatial features of the feature map at different scales.

7. The method according to claim 1, characterized in that Before dividing the remote sensing image into a plurality of image blocks of set sizes and obtaining an image sequence according to the image blocks, the method further includes: The resolution of the acquired remote sensing images; If the resolution of the acquired remote sensing image is lower than the preset resolution, zero pixels are added to the edge of the acquired remote sensing image to expand the resolution of the acquired remote sensing image to the preset resolution.

8. A remote sensing image transmission device, characterized in that: The device comprises: An acquisition unit, used for acquiring remote sensing images; A division unit, used for dividing the remote sensing image into a plurality of image blocks of a set size, and obtaining an image sequence according to the image blocks; An input unit, used to input each image block contained in the image sequence into a convolutional neural network based on a residual network structure, to obtain feature maps of corresponding semantic levels respectively output by multiple stages contained in the convolutional neural network; wherein the feature maps of each semantic level correspond to different resolutions and numbers of channels, and the feature maps of each semantic level fuse spatial features of different scales; The encoding unit is used to perform channel encoding on the feature map queue formed by the feature maps of each semantic level to transmit the encoding result in the channel.

9. An electronic device, characterized in that: It includes a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the method according to any one of claims 1 to 7 by calling the machine-readable instructions.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-readable instructions, and when the machine-readable instructions are called and executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images