Remote sensing image lightweight coding method based on convolutional neural network and self-attention mechanism
By combining convolutional neural network and self-attention mechanism in remote sensing image detection and segmentation algorithm, lightweight encoding method is designed, which solves the problem of low computational efficiency of Transformer encoder, and realizes efficient remote sensing image processing and analysis on edge devices. .
Patent Information
- Application Number
- CN202411024755.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-15
AI Technical Summary
In existing remote sensing image detection and segmentation algorithms, the image encoder based on Transformer is too inefficient to deploy on devices with limited computing resources, especially on edge devices in real time.
Using a lightweight encoding method based on convolutional neural network and self-attention mechanism, the deep-separable convolution and lightweight convolution module extracts local features in the shallow layer of the network, and combines the self-attention mechanism to capture global information in the deep layer, and designs ConViT model, reduces the amount of model parameters and calculation overhead.
It improves the computing efficiency of ViT on edge devices, realizes fast and accurate remote sensing image detection and segmentation, is suitable for environments with limited computing resources, and improves real-time processing capabilities and data transmission bandwidth efficiency.
Smart Images

Figure CN118968299A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of remote sensing image detection and segmentation, and more specifically, relates to a remote sensing image lightweight encoding method based on a convolutional neural network and a self-attention mechanism. Background Art
[0002] Remote sensing image detection and segmentation are key tasks in the current field of remote sensing image understanding, and play an important role in environmental monitoring, urban planning, and disaster warning. In the field of remote sensing image target detection, classic detection algorithms can be divided into two categories: single-stage algorithms and two-stage algorithms. Two-stage algorithms such as Fast R-CNN and Faster R-CNN achieve high detection accuracy through two stages: region recommendation and classification discrimination. Classic segmentation algorithms include FCN, U-Net, and Mask R-CNN. However, with the development of deep learning technology, Transformer-based algorithms have shown their superiority in the field of computer vision. Due to its powerful feature extraction and context learning capabilities, it has gradually become the mainstream algorithm in the field of image detection and segmentation. For example, the target detection algorithm DETR and the image segmentation algorithm MaskFormer, both of which use the Transformer network as the backbone network and have achieved performance indicators that exceed those of convolutional neural networks. However, due to the high computational complexity of the Transformer network, its training and inference require more computing resources, making it difficult to run in real time on edge devices. Compared with convolutional neural networks, the Transformer network often requires more data to achieve better results. Its application in the field of remote sensing images, which has high data acquisition and annotation costs, is often limited.
[0003] In these Transformer-based object detection and image segmentation algorithms, the part that occupies the most computational and parameter resources is often the backbone network of the algorithm. In DETR and MaskFormer, VisionTransformer is used as the backbone network. The main role of the backbone network in image processing tasks is to encode the image and obtain image feature maps of different levels or scales. Therefore, in such tasks, the backbone network is also called an image encoder. In large-scale image detection and segmentation tasks, Vision Transformer (ViT) has a good effect as an image encoder and can obtain very good image representation, especially when the model capacity is large enough and the training is sufficient. However, as the number of model parameters increases, although it can bring higher accuracy, it also brings problems such as increased computational cost, increased memory usage, longer training time, and decreased inference speed. This is particularly prominent in edge computing, embedded systems, and mobile devices, which are usually subject to strict hardware resource constraints.
[0004] In order to be able to train and deploy such advanced Transformer-based detection and segmentation networks on devices with limited computing resources, it is urgent to study and design image encoders that are more lightweight and have powerful image representation capabilities. Summary of the invention
[0005] In view of the defects of the prior art and the need for improvement, the present invention provides a lightweight encoding method for remote sensing images based on convolutional neural networks and self-attention mechanism, which aims to solve the problem that the most advanced image encoder ViT in the current field of remote sensing image detection and segmentation has low computational efficiency and is difficult to deploy on devices with limited computing resources.
[0006] To achieve the above object, according to one aspect of the present invention, a remote sensing image lightweight encoding method based on a convolutional neural network and a self-attention mechanism is provided, comprising:
[0007] S1. Process the input image using depthwise separable convolution to obtain a first feature image whose size is reduced relative to the input image; use a lightweight convolution module to extract features from the current first feature image to obtain a second feature image; the first input image is a remote sensing image;
[0008] S2, using the current second feature image as the input image and repeatedly executing S1 until the size of the final second feature image satisfies: the first feature embedding obtained after block embedding extraction meets the input size of the first self-attention module; the first feature embedding is processed by the first self-attention module to obtain a feature embedding containing global information, the feature embedding is restored to a feature image form and the image size is increased to the same size as the final second feature image, to obtain a third feature image;
[0009] S3, adding the third feature image and the final second feature image, and using the added image as the input image to repeat S1 until the size of the final second feature image satisfies: the second feature embedding obtained after block embedding extraction meets the input size of the second self-attention module; the second feature embedding is processed by the second self-attention module to obtain a feature embedding containing global information; the feature embedding is restored to the feature image form to complete the lightweight encoding of the remote sensing image.
[0010] Furthermore, the depthwise separable convolution reduces the image size to 1 / 2 of its current input image. When the size of the original input image is H×W, then:
[0011] S1 is repeated twice in S2, and the size of the second feature image finally obtained is Using the first embedded module, The second feature image of size 2 is embedded in the patch with a patch size of 2 and obtained The feature map of size is flattened to obtain The first feature embedding of size, The first feature embedding of size matches the input size of the first self-attention module; The first feature embedding of size is input into the first self-attention module;
[0012] In S3, S1 is repeated once, and the size of the second feature image finally obtained is Using the second embedded module, The feature map of size is embedded and extracted with a patch size of 1, and the The feature map of size is flattened to obtain The second feature embedding of size, The second feature embedding of size matches the input size of the second self-attention module; The second feature embedding of the size is input into the second self-attention module.
[0013] Further, a transposed convolution block is used to perform the restoration operation described in S2; and a 3×3DS convolution is used to perform the restoration operation described in S3.
[0014] Furthermore, the lightweight convolution module is a MobileNetV3 module.
[0015] Furthermore, the depthwise separable convolution is a 3×3 depthwise separable convolution; and the lightweight convolution module is composed of a plurality of cascaded lightweight convolution units.
[0016] The present invention also provides a remote sensing image target segmentation method, comprising:
[0017] The remote sensing image to be measured is used as the original input image, and the remote sensing image lightweight encoding method as described above is adopted to obtain the feature image of the remote sensing image to be measured;
[0018] Target segmentation is performed based on the feature image.
[0019] The present invention also provides a remote sensing image target detection method, comprising:
[0020] The remote sensing image to be measured is used as the original input image, and the remote sensing image lightweight encoding method as described above is adopted to obtain the feature image of the remote sensing image to be measured;
[0021] Target detection is performed based on the feature image.
[0022] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the remote sensing image lightweight encoding method as described above when executing the computer program.
[0023] The present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the remote sensing image lightweight encoding method as described above, the remote sensing image target segmentation method as described above, and / or the remote sensing image target detection method as described above.
[0024] The present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the remote sensing image lightweight encoding method as described above, the remote sensing image target segmentation method as described above, and / or the remote sensing image target detection method as described above.
[0025] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:
[0026] (1) The present invention provides a remote sensing image lightweight encoding method based on a convolutional neural network and a self-attention mechanism, which can be used to form an encoder or encoding model. Through ingenious structural design, a lightweight convolution module is introduced in the shallow layer of the network, and a pure convolutional neural network is applied, which is good at extracting local features and detailed texture information. At the same time, the lightweight characteristics of the convolutional neural network are used to reduce the number of parameters and computational overhead of the model, and the lightweight characteristics and local feature extraction capabilities of the convolutional neural network are fully utilized, so that the output feature map is suitable for intensive prediction tasks such as image detection and segmentation, and can converge faster with less data; the self-attention mechanism network is applied in the deep layer of the network, so that while keeping the network parameters and computation lightweight, the global and contextual information in the input image can be captured, and the salient areas in the image can be focused on, which can be applied in various detection and segmentation networks, thereby helping to achieve fast and accurate remote sensing image detection and segmentation. Therefore, the overall parameter amount and computational amount of the present invention are small, which can well meet the requirements of performance and computational overhead, improve the computational efficiency of ViT on edge devices, and is suitable for deployment on edge devices. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 A schematic diagram of a remote sensing image lightweight encoding method based on a convolutional neural network and a self-attention mechanism provided in an embodiment of the present invention;
[0028] Figure 2 A schematic diagram of a remote sensing image lightweight encoding structure based on a convolutional neural network and a self-attention mechanism provided by an embodiment of the present invention;
[0029] Figure 3 A diagram of a lightweight convolution module structure provided by an embodiment of the present invention;
[0030] Figure 4 A depth-separable convolution composition graph provided by an embodiment of the present invention;
[0031] Figure 5 A structural diagram of the SE attention mechanism provided by an embodiment of the present invention;
[0032] Figure 6 Schematic diagram of the self-attention mechanism provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0033] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0034] Embodiment 1
[0035] A lightweight encoding method for remote sensing images based on convolutional neural networks and self-attention mechanism, such as Figure 1 As shown, including:
[0036] S1. Process the input image using depthwise separable convolution to obtain a first feature image whose size is reduced relative to the input image; use a lightweight convolution module to extract features from the current first feature image to obtain a second feature image; the first input image is a remote sensing image;
[0037] S2, using the current second feature image as the input image and repeatedly executing S1 until the size of the final second feature image satisfies: the first feature embedding obtained after block embedding extraction meets the input size of the first self-attention module; the first feature embedding is processed by the first self-attention module to obtain a feature embedding containing global information, the feature embedding is restored to a feature image form and the image size is increased to the same size as the final second feature image, to obtain a third feature image;
[0038] S3, adding the third feature image and the final second feature image, and using the added image as the input image to repeat S1 until the size of the final second feature image satisfies: the second feature embedding obtained after block embedding extraction meets the input size of the second self-attention module; the second feature embedding is processed by the second self-attention module to obtain a feature embedding containing global information; the feature embedding is restored to the feature image form to complete the lightweight encoding of the remote sensing image.
[0039] In order to train and deploy such advanced Transformer-based detection and segmentation networks on devices with limited computing resources, it is necessary to study and design image encoders that are more lightweight and have powerful image representation capabilities. In order to improve the computing efficiency of ViT on edge devices, there are currently three main ideas:
[0040] (1) Combine the convolutional layer with ViT and design a hybrid structure network.
[0041] This approach aims to combine the advantages of convolutional neural networks (CNNs) and ViT to design a more efficient and lightweight hybrid model. For example, CoAtNet inserts convolutional layers between ViT blocks to achieve cross-scale feature interaction and fusion. These hybrid models reduce computational complexity through convolution operations while maintaining the global modeling capabilities of ViT. This approach still applies Transformer-based self-attention operations on high-resolution feature maps, and its structure and operation speed still have room for optimization.
[0042] (2) Introduce more efficient attention operations to improve computational efficiency.
[0043] The standard self-attention operation has a quadratic computational complexity, which limits the application of ViT under large-scale input. To address this problem, some works have explored linear complexity attention variants. For example, Linformer decomposes the self-attention matrix into two low-rank matrices through low-rank approximation, reducing the computational complexity to the linear level. Performer uses orthogonal random features to convert attention calculations into inner product operations, which also achieves linear complexity. These efficient attention operations can significantly reduce the computational overhead of ViT and improve its applicability in resource-constrained environments. The drawback of this method is that a purely Transformer-based network structure still requires more data to fit better.
[0044] (3) Design a pure convolutional model based on the structure of ViT.
[0045] In addition to directly modifying the ViT structure, some works have also tried to refer to the design ideas of ViT and build pure convolutional models. This type of method aims to inherit some excellent features of ViT, such as global modeling capabilities and dynamic attention mechanisms, while using convolution operations to improve computational efficiency. For example, ConvNeXt was inspired by ViT and designed a novel deep convolutional network architecture. It uses techniques such as large-size convolution kernels, depthwise convolution, and attention average pooling to achieve a good balance between accuracy and efficiency. Similarly, ConvMixer draws on ViT's patch embedding and tokenmixing operations to design a simple and effective convolutional model. This method aims to bring existing convolutional neural networks closer to the design architecture of the Transformer network, but in terms of global context learning capabilities, it is still weaker than the Transformer network structure based on the self-attention mechanism.
[0046] This embodiment further optimizes the model structure to address the weaknesses of the current algorithm, and proposes an image encoder that combines a convolutional neural network and a Transformer network for remote sensing image detection and segmentation tasks. Through clever structural design, a pure convolutional neural network is applied in the shallow layer of the network, and a Transformer network is applied in the deep layer of the network, so that while keeping the network parameters and calculations lightweight, it can capture the global and contextual information in the input image, focus on the salient areas in the image, and can be applied in various detection and segmentation networks, thereby helping to achieve fast and accurate remote sensing image detection and segmentation.
[0047] That is, this embodiment is a lightweight image encoder ConViT (Convolution Vision Transformer) that mixes convolutional layers and self-attention layers, which is suitable for remote sensing image detection and segmentation tasks. Specifically, the lightweight convolution module is introduced to make full use of the lightweight characteristics and local feature extraction capabilities of convolutional neural networks, and combine the global context learning capabilities of the self-attention mechanism. This well meets the requirements of performance and computing overhead, and improves the computing efficiency of ViT on edge devices.
[0048] This embodiment is used to perform image coding tasks. A coding model or encoder can be constructed according to the method of this embodiment. Before application, the coding model or encoder needs to be trained. Because the coding model or encoder can be used as a part of the entire detection and segmentation model, the training of the encoder or coding model can be achieved by training the entire detection and segmentation model.
[0049] A training data set, D train ={(x i ,y1),···,(xN ,y N )}, where x i ∈R H×W×c represents an image, H×W is the image size, and c is the number of channels in the image. The goal is to use the designed lightweight image coding method to quickly and accurately capture global context and macroscopic structure information while extracting local features. Finally, feature maps of different scales are obtained.
[0050] As a preferred implementation, the depthwise separable convolution reduces the image size to 1 / 2 of its current input image. When the size of the original input image is H×W, then:
[0051] S1 is repeated twice in S2, and the size of the second feature image finally obtained is Using the first embedded module, The second feature image of size 2 is embedded in the patch with a patch size of 2 and obtained The feature map of size is flattened to obtain The first feature embedding of size, The first feature embedding of size matches the input size of the first self-attention module; The first feature embedding of size is input into the first self-attention module;
[0052] In S3, S1 is repeated once, and the size of the second feature image finally obtained is Using the second embedded module, The feature map of size is extracted by block embedding with Patch Size (block size) of 1, and the The feature map of size is flattened to obtain The second feature embedding of size, The second feature embedding of size matches the input size of the second self-attention module; The second feature embedding of the size is input into the second self-attention module.
[0053] like Figure 2 As shown, specifically:
[0054] (1) Using depthwise separable convolution, we change the number of channels of the input 3×H×W original remote sensing image and reduce the size of the feature map to obtain Feature image; The feature image is input into N1 cascaded lightweight convolution modules to extract features and obtain a new Feature image (C1 is the number of channels c).
[0055] The feature map enters N1 cascaded lightweight convolution modules, for example, Figure 3As shown in the figure, specifically: use 1×1 convolution to expand the number of image channels, use 3×3 DW convolution to extract features, and reduce the computational complexity and model parameter quantity. Use the SE module to adaptively adjust the importance between channels, improve the network's learning effect on key features and the overall feature representation ability, and obtain a feature map; use 1×1 convolution to reduce the number of feature map channels to the original input feature map channel number. After N1 times of the above process, the new Feature map.
[0056] (2) Use depth-wise separable convolution to change the new The number of channels of the feature image is increased and the feature map size is reduced to obtain Feature image; The feature image is input into N2 cascaded lightweight convolution modules to extract features and obtain a new Feature map;
[0057] The feature map enters N2 lightweight convolution modules, for example, Figure 3 As shown in the figure, specifically: 1×1 convolution is used to expand the number of image channels, 3×3 DW convolution is used to reduce the computational complexity and model parameters and extract features. SE module is used to adaptively adjust the importance between channels to improve the feature representation capability, and 1×1 convolution is used to reduce the number of feature map channels; after N2 times of the above process, a new Feature map.
[0058] (3) Use depth-wise separable convolution to change the new The number of channels of the feature map is increased and the size of the feature map is reduced to obtain Will The feature image is input into N3 cascaded lightweight convolution modules to extract features and obtain a new feature map; adopt block embedding module (its function is to change the dimension of features to adapt them to the input of self-attention module and to reduce the number of features and make them lightweight) to the new The feature map is embedded in the block and extracted to obtain Feature map, and flatten it to get feature embedding The feature embedding is input into L1 cascaded self-attention modules to obtain a global information Feature embedding; including global information The feature embedding input is transposed into the convolution block to obtain the final Feature map;
[0059] New The feature map enters the Patch Embedding module with a Patch Size of 2, which divides the input feature map into small blocks (also called "patches"), reducing redundant calculations while retaining important feature information. Feature map.
[0060] (4) To avoid the problem of gradient vanishing, skip connections are introduced to add the input feature map of the block embedding module and the feature map transformed by the block embedding module, self-attention module, and transposed convolution module before inputting the deep separable convolution to maintain the strength of the gradient, accelerate the training process, and improve the training stability. Feature maps and final After adding the feature maps, the input depth can be separated by convolution, and we get feature map; The feature map is input into N4 cascaded lightweight convolution modules to obtain a new Feature map;
[0061] The feature map enters the lightweight convolution module, for example, Figure 3 As shown in the figure, specifically: use 1×1 convolution to expand the number of image channels, use 3×3 DW convolution to reduce computational complexity and extract features. Use SE module to adaptively adjust the importance between channels to improve feature representation capability, and use 1×1 convolution to reduce the number of feature map channels. The feature map enters L2 cascaded self-attention modules (Transformer layers): global context relationship modeling is performed on the feature map, and feature representations containing rich global information are extracted to update the Feature map.
[0062] Using block embedding modules, new The feature map is divided into small blocks to obtain a new feature map; the new The feature map is input into L2 cascaded self-attention modules to obtain a global information That is, the feature map enters the Patch Embedding module with a Patch Size of 1, which divides the input feature map into small blocks (also called "patches"), reducing redundant calculations while retaining important feature information. Feature map.
[0063] Will The feature map is input into a 3×3DS convolution, and shape recovery is performed without changing the size and number of channels of the feature map (i.e., converting the features from the form of a sequence to the form of a feature map).
[0064] In the above scheme, using a transposed convolution block to perform the recovery operation described in S2 can be used as a preferred method, or other methods can be selected; using a 3×3DS convolution to perform the recovery operation described in S3 can be used as a preferred method, or other methods can be selected.
[0065] like Figure 2 As shown in the figure, the encoder processing flow is divided into 4 stages, of which the first two stages are pure convolution modules, and the last two stages are a mixture of convolution modules and self-attention modules. The pure convolution modules of the first two stages are composed of convolution (depth-separable convolution) and lightweight convolution modules; stage3 is composed of 3×3DS convolution and lightweight convolution modules, block embedding modules, self-attention modules and transposed convolutions; stage4 is composed of 3×3DS convolution and lightweight convolution modules, block embedding modules, self-attention modules and 3×3DS convolutions. The following will introduce in turn: 3×3DS convolution, lightweight convolution module, block embedding module, self-attention module and transposed convolution. Among them:
[0066] 1. 3×3DS convolution
[0067] like Figure 4 As shown in the figure, the depthwise separable convolution (DS) combines the depthwise convolution (DW) and pointwise convolution (PW) for feature extraction. Compared with the traditional convolution operation, the depthwise separable convolution has a lower number of parameters and computational cost, so it is often used in lightweight networks. The depthwise separable convolution is mainly divided into two processes: the depthwise convolution is used to extract spatial features, and the pointwise convolution is used to extract channel features. In the feature dimension, the depthwise separable convolution groups the convolution operations, performs independent depthwise convolution on each channel, and then integrates all channels through 1x1 convolution. Specifically, the depthwise separable convolution first performs depthwise convolution on each channel, and then merges all channels into the output feature map through pointwise convolution, thereby reducing the amount of calculation and improving the calculation efficiency.
[0068] (1) Deep convolution
[0069] Deep convolution means that the depth (channel) dimension remains unchanged while changing H / W. It performs independent convolution operations on each channel of the input feature map, that is, each input channel corresponds to a convolution kernel. In this way, different feature channels of the input can be processed without introducing parameter sharing, thereby increasing the expressive power of the model. Deep convolution uses fewer parameters for feature extraction, which helps reduce the risk of overfitting and can reduce the amount of calculation and memory usage.
[0070] (2) Point-wise convolution
[0071] Point-by-point convolution means that the W / H dimension remains unchanged, but the channel is changed. According to the depthwise convolution, the number of input feature map channels = the number of convolution kernels = the number of output feature maps, which will result in too few output feature maps (or too few channels of the output feature map, which can be regarded as 1 output feature map and 3 channels), which may affect the validity of the information. In this case, point-by-point convolution is needed.
[0072] In the present invention, the role of 3×3 DS convolution is mainly to reduce the size of the feature map and change the number of channels of the feature map. The image can be effectively reduced in size by using 3×3 depthwise separable convolution. The specific steps are as follows:
[0073] S1. Extract features from the input image through 3×3 depthwise convolution. Apply a 3×3 filter to each channel of the input image, reducing the size of the feature map while keeping the number of feature map channels unchanged.
[0074] S2. Use a 1×1 convolution kernel to perform a convolution operation on the feature map obtained by depth convolution, and control the dimension of the feature by adjusting the number of output channels.
[0075] S3. By increasing the stride parameter, the size of the output feature map is reduced.
[0076] Through the above steps, 3x3 depthwise separable convolution is used to reduce the size of the image while changing the number of feature map channels, thereby reducing the computational cost and the number of parameters while ensuring the model effect.
[0077] 2. Lightweight convolution module
[0078] The lightweight convolution module can better extract features. By summing the weights of the feature parameters in the channel, it deepens the network while ensuring the lightweight of the network. Regarding the design of the lightweight convolution module, the MobileNetV3 module is preferably used, and its structure is as follows Figure 3As shown in the figure. First, the number of channels is expanded through 1×1 convolution (also called pointwise convolution), and then 3×3 depth-separable convolution is used to reduce the number of parameters and calculations for feature maps. The importance between channels is adaptively adjusted through the SE attention mechanism to improve the feature representation capability. The SE module is added to the MobileNetV3 module to improve the feature utilization and expression capability of the model. Compared with the traditional ReLU activation function, MobileNetV3 uses h-Swish, which can accelerate convergence in deep networks and improve the operating efficiency and accuracy on mobile devices.
[0079] Point-by-point convolution and depth-wise separable convolution have been introduced in detail in the previous section. The following will mainly introduce the SE attention mechanism. The structure of the SE attention mechanism is as follows Figure 5 As shown, the structure mainly includes the following steps:
[0080] (1) Global Information Embedding (Squeeze)
[0081] F sq That is, use global average pooling of channels. In order to solve the problem of using channel dependency, choose to compress the global spatial information into a channel descriptor, that is, use global average pooling of channels to directly compress the H×W×C feature map U containing global information into a 1×1×C feature vector Z. The channel features of C feature maps are compressed into a value, so that the generated channel-level statistics Z contain contextual information, alleviating the problem of channel dependency. The definition is as follows:
[0082]
[0083] where z c is the c-th element of z.
[0084] (2) Adaptive Recalibration (Excitation)
[0085] In order to utilize the information gathered in the compression operation, the next step is to fully capture channel dependencies through the Excitation operation. A fully connected layer network (usually including one or more fully connected layers and activation functions) is used to model global statistical information and learn the excitation weights for each channel. For this purpose, a gate mechanism consisting of two fully connected layers is adopted. The first fully connected layer compresses C channels into C / r channels to reduce the amount of calculation, and then passes through a RELU nonlinear activation layer. The second fully connected layer restores the number of channels back to C channels, and then obtains the weight s through Sigmoid activation. The dimension of the final s is 1×1×C, which is used to characterize the weights of the C feature maps in the feature map U. r refers to the compression ratio.
[0086] s=F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z))
[0087] (3) Re-weighting (Scale)
[0088] Finally, the Scale operation is performed to weight the attention weights obtained above to the features of each channel, and then apply them to the original feature map to weight the features of each channel and highlight the important features. Implementation method: Each feature map in the feature map U is multiplied by the corresponding weight to obtain the final output of the SE module.
[0089]
[0090] 3. Block Embedding Module
[0091] The Patch Embedding module divides the input image into many patches and converts each patch into a low-dimensional vector representation, which is used as the input for subsequent tasks. The Patch Embedding module consists of a convolutional layer with a kernel size equal to the patch size and a step size equal to the patch size. Assuming that the original feature map size is H×W, through PatchEmbedding, the feature map size is reduced to Since the size of the feature map is relatively small in the third and fourth stages, they are For the initial image size of 1024×1024, the feature map size here is 128×128 and 64×64, so a relatively small Patch Size can be set, and the Patch Size is set to 2 and 1 respectively, so that a 64×64 feature map can be obtained. The purpose of Patch Embedding is to reduce computational complexity and improve the efficiency of feature extraction. In traditional convolution operations, adjacent pixels usually overlap a lot, and PatchEmbedding divides the image into blocks, which can reduce redundant calculations while retaining important feature information.
[0092] 4. Self-Attention Module
[0093] The self-attention mechanism is called Scaled Dot-Product Attention. Figure 6 For each attention unit, the Transformer model learns three weight matrices: query weight W Q , key weight W K Sum weight W V For the input vector I, we get the query vector Q = W Q I, key vector K = WK I, value vector V = W V I. The query vector and the key vector are then matrix multiplied (MatMul) to calculate the attention weight, and the attention weight matrix is divided by the square root of the key vector dimension Scaling is performed to stabilize the gradient during training, and then a softmax function is used to normalize the weights. Finally, the attention matrix obtained by the softmax operation is multiplied by the value vector V to obtain the transformed output vector O. The calculation formula is as follows:
[0094]
[0095] A group (W Q , W K , W V ) matrix becomes an attention head. In practical applications, there are multiple attention heads in each layer of Transformer, and each attention head is used to learn a different attention mode. The multi-head attention mechanism is expressed as follows:
[0096] MultiHeadAttention(Q,K,V)=Concat i∈{heads} (Attention(Q i , K i , V i ))W O Among them, Concat represents the merge operation, W O Represents the projection matrix of a multi-head attention head.
[0097] In the encoder layer of Transformer, the attention mechanism used is the self-attention mechanism, while in the decoder layer, the cross-attention mechanism is used. The difference between them is that in the self-attention mechanism, {Q, K} is generated by the same input vector, while in the cross-attention mechanism, {Q, K} is generated by different input vectors. For V, always keep the same input vector as K. In the decoder, when calculating the attention matrix, it is necessary to mask the attention connection between certain position vectors, which can be done by adding a mask matrix M, as shown below:
[0098]
[0099] In the matrix M, the positions where the attention connection needs to be cut off are set to negative infinity, and the other positions are set to 0.
[0100] In the field of computer vision, the Transformer is adapted to computer vision tasks by decomposing the input image into a series of image blocks, for example, a 1024×1024 image can be divided into 16×16 image blocks, each of which is 64×64 in size, and converting each image block into a vector and treating it as a token in the standard Transformer.
[0101] 5. Transposed Convolution
[0102] After the image has been through multiple layers of convolution operations, the size of the output image will become very small. In order to restore the image to its original size for further calculations, it is necessary to map the image from a small resolution to a large resolution through upsampling. Transpose convolution is a type of convolution used for upsampling. Transpose convolution generates a higher resolution feature map by transposing the convolution kernel weight matrix and applying the transposed kernel to the lower resolution feature map.
[0103] This embodiment is based on artificial priors that match the visual detection and segmentation tasks, introduces convolutional layers in the feature extraction scenario, and based on the local connection processing of the convolutional layer, makes the output feature map suitable for intensive prediction tasks such as image detection and segmentation, and can converge faster with less data. It combines the characteristics of convolutional layers and self-attention layers: the first two stages use pure convolution modules, which are good at extracting local features and detailed texture information, while taking advantage of the lightweight characteristics of convolutional neural networks to reduce the number of parameters and computational overhead of the model. In the last two stages, the self-attention module is introduced to capture global context and macro-structure information. In general, a lightweight convolution module is used, and the self-attention module is applied to low-resolution feature maps. The overall network parameters and computational complexity are small, which is suitable for deployment on edge devices.
[0104] Embodiment 2
[0105] A remote sensing image target segmentation method, comprising:
[0106] The remote sensing image to be measured is used as the original input image, and the remote sensing image lightweight encoding method as described above is adopted to obtain the feature image of the remote sensing image to be measured;
[0107] Target segmentation is performed based on the feature image.
[0108] The relevant technical solution is the same as that in Embodiment 1 and will not be described in detail here.
[0109] Embodiment 3
[0110] A remote sensing image target detection method, comprising:
[0111] The remote sensing image to be measured is used as the original input image, and the remote sensing image lightweight encoding method as described above is adopted to obtain the feature image of the remote sensing image to be measured;
[0112] Target detection is performed based on the feature image.
[0113] The image encoder proposed in the present invention, which combines a convolutional neural network and a Transformer network for remote sensing image detection and segmentation tasks, greatly reduces the consumption of computing resources while ensuring the accuracy of remote sensing image detection and segmentation, and broadens the application possibilities of advanced remote sensing image detection and segmentation technology in unmanned systems and edge computing environments. It is particularly suitable for edge computing platforms with high real-time requirements and limited computing resources.
[0114] Embodiment 4
[0115] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the remote sensing image lightweight encoding method as described above when executing the computer program.
[0116] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The memory may be used to store computer programs and / or modules, and the processor may perform various functions of the electronic device by running or executing computer programs and / or modules stored in the memory, and calling data stored in the memory.
[0117] The relevant technical solution is the same as that in Embodiment 1 and will not be described in detail here.
[0118] Embodiment 5
[0119] A computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the remote sensing image lightweight encoding method as described above, the remote sensing image target segmentation method as described above, and / or the remote sensing image target detection method as described above.
[0120] Specifically, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, an internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0121] The relevant technical solution is the same as that in Embodiment 1 and will not be described in detail here.
[0122] Embodiment 6
[0123] A computer program product, comprising a computer program or an instruction, characterized in that when the computer program or the instruction is executed by a processor, the remote sensing image lightweight encoding method as described above, the remote sensing image target segmentation method as described above and / or the remote sensing image target detection method as described above are implemented.
[0124] The relevant technical solution is the same as that in Embodiment 1 and will not be described in detail here.
[0125] In general, the core purpose of the present invention is to solve the problem of computing resource limitations encountered when deploying deep learning models on edge devices, especially in the field of remote sensing image processing. In the prior art, traditional deep learning models, especially models based on attention mechanisms such as visual transformers (ViT), often have large parameter amounts and computing requirements, which makes them difficult to run on edge devices with limited computing power, resulting in a decrease in real-time performance and energy efficiency. In order to meet this challenge, the present invention adopts the following key technical means: 1. Lightweight convolution module: In the shallow stage of the neural network, lightweight convolution modules are applied, such as depthwise separable convolution (Depthwise Separable Convolution) or similar designs. These modules can significantly reduce the number of parameters and computational overhead of the model by reducing redundant calculations, especially cross-channel calculations. This makes the model more efficient when running on edge devices and can converge quickly, especially when the amount of data is limited. This method allows the model to capture the detailed texture of the image at a low computational cost, which is very suitable for tasks such as image detection and segmentation that require intensive prediction. 2. Integration of self-attention mechanism: In the deep layer of the network, the self-attention mechanism is introduced, which helps the model capture the global information and contextual associations of the input image and focus on the key areas in the image. The self-attention mechanism allows the model to take into account the information of the entire image when processing the features of each position, so that long-distance dependencies can be captured. This is particularly important in remote sensing image analysis, because remote sensing images often contain spatial information across a large range, and the self-attention mechanism can effectively handle these global correlations without adding too much parameter burden. The self-attention mechanism can enhance the model's understanding of the overall structure of the image while keeping the model lightweight, which is crucial to improving the accuracy of detection and segmentation.
[0126] The remote sensing image lightweight encoding method based on convolutional neural network and self-attention mechanism proposed in the present invention can not only solve the problem of limited computing resources of edge devices, but also achieve the following effects: (1) Improved real-time processing capability: The lightweight convolution module reduces the amount of calculation and the number of parameters, so that the model can complete the reasoning process with lower latency. The self-attention mechanism can quickly process global information through a parallel computing mechanism, further accelerating the response speed of the model. These technologies work together to enable the model to achieve real-time image processing and analysis on edge devices. (2) Optimization of data transmission bandwidth: The feature maps generated by the lightweight convolution module usually take up less storage space, which means that when performing distributed computing or transmitting intermediate results to the cloud for further processing, the amount of data transmission required will be greatly reduced. This not only saves bandwidth, but also reduces the delay that may occur during the transmission process.
[0127] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A remote sensing image lightweight encoding method based on convolutional neural network and self-attention mechanism, characterized in that: include: S1. Processing the input image using depthwise separable convolution to obtain a first feature image whose size is reduced relative to the input image; A lightweight convolution module is used to extract features from the current first feature image to obtain a second feature image; the first input image is a remote sensing image; S2, using the current second feature image as the input image and repeatedly executing S1 until the size of the final second feature image satisfies: the first feature embedding obtained after block embedding extraction meets the input size of the first self-attention module; the first feature embedding is processed by the first self-attention module to obtain a feature embedding containing global information, the feature embedding is restored to a feature image form and the image size is increased to the same size as the final second feature image, to obtain a third feature image; S3, adding the third feature image and the final second feature image, and using the added image as the input image to repeat S1 until the size of the final second feature image satisfies: the second feature embedding obtained after block embedding extraction meets the input size of the second self-attention module; the second feature embedding is processed by the second self-attention module to obtain a feature embedding containing global information; the feature embedding is restored to the feature image form to complete the lightweight encoding of the remote sensing image.
2. The remote sensing image lightweight encoding method according to claim 1, characterized in that: The depthwise separable convolution reduces the image size to 1 / 2 of its current input image. When the size of the original input image is H×W, then: S1 is repeated twice in S2, and the size of the second feature image finally obtained is Using the first embedded module, The second feature image of size 2 is embedded in the patch with a patch size of 2 and obtained The feature map of size is flattened to obtain The first feature embedding of size, The first feature embedding of size matches the input size of the first self-attention module; The first feature embedding of size is input into the first self-attention module; In S3, S1 is repeated once, and the size of the second feature image finally obtained is Using the second embedded module, The feature map of size is embedded and extracted with a patch size of 1, and the The feature map of size is flattened to obtain The second feature embedding of size, The second feature embedding of size matches the input size of the second self-attention module; The second feature embedding of the size is input into the second self-attention module.
3. The remote sensing image lightweight encoding method according to claim 1, characterized in that: A transposed convolution block is used to perform the restoration operation described in S2; a 3×3 depthwise separable convolution is used to perform the restoration operation described in S3.
4. The remote sensing image lightweight encoding method according to claim 1, characterized in that: The lightweight convolution module is a MobileNetV3 module.
5. The remote sensing image lightweight encoding method according to claim 1, characterized in that: The depthwise separable convolution in S1 is a 3×3 depthwise separable convolution; the lightweight convolution module is composed of a plurality of cascaded lightweight convolution units.
6. A remote sensing image target segmentation method, characterized in that: include: Taking the remote sensing image to be measured as the original input image, and adopting the remote sensing image lightweight encoding method according to any one of claims 1 to 5 to obtain a feature image of the remote sensing image to be measured; Target segmentation is performed based on the feature image.
7. A remote sensing image target detection method, characterized in that: include: Taking the remote sensing image to be measured as the original input image, and adopting the remote sensing image lightweight encoding method according to any one of claims 1 to 5 to obtain a feature image of the remote sensing image to be measured; Target detection is performed based on the feature image.
8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the remote sensing image lightweight encoding method according to any one of claims 1 to 5 are implemented.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed by the processor, the device where the storage medium is located is controlled to execute the remote sensing image lightweight encoding method as described in any one of claims 1 to 5, the remote sensing image target segmentation method as described in claim 6 and / or the remote sensing image target detection method as described in claim 7.
10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by the processor, it implements the remote sensing image lightweight encoding method as described in any one of claims 1 to 5, the remote sensing image target segmentation method as described in claim 6 and / or the remote sensing image target detection method as described in claim 7.
Citation Information
Cited By
Method and system for detecting maturity of tomatoes
CN120219858A
Ship energy efficiency optimization method based on industrial data analysis
CN122366149A
