Remote sensing image feature extraction method, device and equipment and storage medium

Abstract

The application discloses a remote sensing image ground feature extraction method and device, equipment and storage medium, relates to the technical field of image feature extraction. The specific implementation scheme comprises the following steps: acquiring a first training set and a second training set; constructing an initial ground feature extraction network, training the initial ground feature extraction network according to the first training set and the second training set, and obtaining a target ground feature extraction model; and extracting ground features from an input remote sensing image through the target ground feature extraction model. The present application can solve the problem that the visual large model in the prior art is difficult to deploy on the unmanned aerial vehicle platform for ground feature extraction, improve the model precision, reduce the calculation resource requirement of the model, and improve the applicability to the unmanned aerial vehicle platform.

Description

Technical Field

[0001] This application relates to the field of image feature extraction technology, and in particular to a method, apparatus, device and storage medium for extracting features of ground objects from remote sensing images. Background Technology

[0002] As an aerial platform, unmanned aerial vehicles (UAVs) can be integrated with high-precision sensors and computing platforms in industries such as remote sensing and surveying to achieve efficient aerial data acquisition and processing, offering advantages that traditional manned aircraft cannot match. UAV platforms can perform remote sensing image retrieval based on ground feature characteristics in aerial remote sensing images, ensuring rapid and accurate positioning over large areas during real-time flight. Furthermore, with the development of deep learning, large-scale visual models, represented by the Transformer, have demonstrated performance advantages in ground feature extraction.

[0003] However, the stringent accuracy and real-time requirements of drones during flight make it difficult to deploy large-scale, computationally intensive visual models with a large number of parameters on drone platforms for ground feature extraction. For example, while the global attention mechanism of Transformer can effectively capture multi-scale semantic information in remote sensing images, drone platforms struggle to meet the massive computational resource requirements, thus failing to meet the millisecond-level response requirements for high-frequency decision-making on drone platforms.

[0004] Therefore, there is an urgent need for a lightweight method for extracting ground features from remote sensing images that is applicable to UAV platforms. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for extracting ground features from remote sensing images. This solves the problem in the prior art that large visual models are difficult to deploy on UAV platforms for ground feature extraction. It improves model accuracy while reducing the computational resource requirements of the model and enhances its applicability to UAV platforms.

[0006] In a first aspect, embodiments of this application provide a method for extracting ground feature characteristics from remote sensing images, including:

[0007] Obtain a first training set and a second training set. The first training set includes multiple sample groups, each containing multiple first remote sensing images and multiple second remote sensing images. Both the first and second remote sensing images are cropped from the same original remote sensing image, with the second remote sensing image being larger than the first remote sensing image. The second training set includes multiple sample pairs, each containing a third remote sensing image and a fourth remote sensing image. The fourth remote sensing image is the third remote sensing image after semantic masking of the target land cover type. Construct an initial land cover feature extraction network and train it using the first and second training sets to obtain a target land cover feature extraction model. Extract land cover features from the input remote sensing images using the target land cover feature extraction model. The process involves obtaining ground feature features. The initial ground feature extraction network includes a convolution pre-processing module, a patch embedding module, a label encoding module, an encoder module, and a feature mapping module. The remote sensing image is input into the convolution pre-processing module for feature extraction and dimensionality reduction to reduce the number of model parameters, resulting in the target convolutional features. These target convolutional features are then input into the patch embedding module for block segmentation and flattening to obtain the first intermediate features. The first intermediate features are then input into the label encoding module for category labeling and location encoding, followed by random deactivation to obtain the second intermediate features. The second intermediate features are then input into the encoder module for feature extraction to obtain the third intermediate features. Finally, the third intermediate features are input into the feature mapping module for layer normalization, classification token extraction, and feature mapping to obtain the ground feature features.

[0008] Furthermore, the encoder module includes multiple encoder units connected in sequence. The encoder unit is used to perform layer normalization processing on the features of the input encoder unit and then input them into the multi-head attention layer to obtain the first feature to be processed. The first feature to be processed is randomly deactivated and then added to the features of the input encoder unit to obtain the second feature to be processed. The second feature to be processed is layer normalized and then input into the MLP (Multilayer Perceptron) unit to obtain the third feature to be processed. The third feature to be processed is randomly deactivated and then added to the second feature to obtain the output feature of the encoder unit and output it.

[0009] Furthermore, the pre-convolution module is used to input the input remote sensing image into the first convolutional layer to obtain the first convolutional feature. After batch normalization and activation function, the first convolutional feature is input into the second convolutional layer to obtain the second convolutional feature. After batch normalization and activation function, the second convolutional feature is input into the third convolutional layer to obtain the third convolutional feature. After batch normalization and activation function, the third convolutional feature is obtained to obtain the target convolutional feature.

[0010] Furthermore, the initial feature extraction network is trained using the first and second training sets to obtain the target feature extraction model, including:

[0011] The first remote sensing image is input into a preset ground feature extraction model and processed through feature centering and activation functions to obtain the first feature. The first remote sensing image is then input into an initial ground feature extraction network and processed through an activation function to obtain the second feature. The second remote sensing image is then input into the initial ground feature extraction network and processed through an activation function to obtain the third feature. Based on the first, second, and third features, and using a first loss function, the parameters of the initial ground feature extraction network are updated until the first loss function converges, resulting in an intermediate ground feature extraction network. The third and fourth remote sensing images are then input into the intermediate ground feature extraction network to obtain the fourth and fifth features, respectively. The fourth and fifth features are then input into a fully connected network to obtain the sixth and seventh features, respectively. Based on the sixth and seventh features, and using a second loss function, the parameters of the intermediate ground feature extraction network are updated until the second loss function converges, resulting in the target ground feature extraction model.

[0012] Furthermore, the first loss function is shown in equation (1):

[0013]

[0014] In equation (1), L distill Let H(·) represent the first loss function, H(·) represent the cross-entropy loss function, N represent the sum of the number of first and second remote sensing images corresponding to the same original remote sensing image, M represent the number of first remote sensing images corresponding to the same original remote sensing image, and NM represent the number of second remote sensing images corresponding to the same original remote sensing image. Represents the i-th first feature. This represents the i-th second feature. This represents the j-th third feature.

[0015] Furthermore, the second loss function is shown in equation (2) below:

[0016]

[0017] In equation (2), L orth Let z represent the second loss function. S′ img The sixth feature is represented by z. S′ mask_img Representing the seventh feature, z S′ img ·z S′ mask_img ||z| represents the inner product of the sixth and seventh features. S′img || represents the norm of the sixth feature, ||z S′ mask_img || represents the norm of the seventh feature.

[0018] Furthermore, the fourth remote sensing image is obtained through an occlusion step, which includes:

[0019] Based on the third remote sensing image and user prompts, an image of the target land cover type is obtained using the SAM (Segment Anything) model; the target land cover type image in the third remote sensing image is occluded to obtain the fourth remote sensing image.

[0020] Secondly, embodiments of this application provide a remote sensing image feature extraction device, comprising:

[0021] The acquisition module is used to acquire a first training set and a second training set. The first training set includes multiple sample groups, each sample group including multiple first remote sensing images and multiple second remote sensing images. The first and second remote sensing images are both cropped from the same original remote sensing image, and the size of the second remote sensing image is larger than the size of the first remote sensing image. The second training set includes multiple sample pairs, each sample pair including a third remote sensing image and a fourth remote sensing image. The fourth remote sensing image is the third remote sensing image after the target land cover type is semantically masked.

[0022] The model building and training module is used to build an initial ground feature extraction network, and train the initial ground feature extraction network based on the first training set and the second training set to obtain the target ground feature extraction model.

[0023] The extraction module is used to extract ground features from the input remote sensing image using a target ground feature extraction model, thereby obtaining ground feature characteristics.

[0024] The initial ground feature extraction network includes a convolution pre-processing module, a patch embedding module, a label encoding module, an encoder module, and a feature mapping module. The remote sensing image is input into the convolution pre-processing module for feature extraction and dimensionality reduction to reduce the number of model parameters, resulting in the target convolutional features. The target convolutional features are then input into the patch embedding module for block-based processing and flattening, yielding the first intermediate features. The first intermediate features are then input into the label encoding module for category labeling and location encoding, followed by random deactivation, resulting in the second intermediate features. The second intermediate features are then input into the encoder module for feature extraction, yielding the third intermediate features. Finally, the third intermediate features are input into the feature mapping module for layer normalization, classification token extraction, and feature mapping, resulting in the ground feature characteristics.

[0025] Thirdly, embodiments of this application provide an apparatus, the apparatus comprising: a processor; a memory for storing processor-executable instructions; and a method for implementing, as described in the first aspect or any possible implementation of the first aspect, when the processor executes the executable instructions.

[0026] Fourthly, embodiments of this application provide a non-volatile computer-readable storage medium, which includes storage for storing a computer program or instructions that, when executed, cause a method as described in the first aspect or any possible implementation of the first aspect to be implemented.

[0027] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0028] This application embodiment constructs an initial ground feature extraction network by acquiring a first training set and a second training set. This network includes a convolutional pre-processing module for preliminary feature extraction and dimensionality reduction, and an encoder module for better extraction of semantic information. The initial ground feature extraction network is trained to obtain a target ground feature extraction network. The target ground feature extraction network is used to extract ground features from the input remote sensing image to obtain ground feature characteristics. This can improve the model accuracy while reducing the computational resource requirements of the model and improve the applicability to UAV platforms. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A flowchart illustrating the remote sensing image feature extraction method provided in this application embodiment;

[0031] Figure 2 A schematic diagram of the network structure of the initial ground feature extraction network provided in this application embodiment;

[0032] Figure 3 This is a schematic diagram of the encoder unit provided in an embodiment of this application;

[0033] Figure 4 Another flowchart illustrating the remote sensing image feature extraction method provided in this application embodiment;

[0034] Figure 5 This is a schematic diagram of the composition of the remote sensing image feature extraction device provided in the embodiments of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0036] The following description of some technologies involved in the embodiments of this application is provided to aid understanding and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, some descriptions of well-known functions and structures are omitted in the following description.

[0037] As an aerial platform, unmanned aerial vehicles (UAVs) can be integrated with high-precision sensors and computing platforms in industries such as remote sensing and surveying to achieve efficient aerial data acquisition and processing, offering advantages that traditional manned aircraft cannot match. UAV platforms can perform remote sensing image retrieval based on ground feature characteristics in aerial remote sensing images, ensuring rapid and accurate positioning over large areas during real-time flight. Furthermore, with the development of deep learning, large-scale visual models, represented by the Transformer, have demonstrated performance advantages in ground feature extraction.

[0038] However, the stringent accuracy and real-time requirements of drones during flight make it difficult to deploy large-scale, computationally intensive visual models with a large number of parameters on drone platforms for ground feature extraction. For example, while the global attention mechanism of Transformer can effectively capture multi-scale semantic information in remote sensing images, drone platforms struggle to meet the massive computational resource requirements, thus failing to meet the millisecond-level response requirements for high-frequency decision-making on drone platforms.

[0039] Therefore, there is an urgent need for a lightweight method for extracting ground features from remote sensing images that is applicable to UAV platforms.

[0040] Against this background, this disclosure provides a method for extracting ground features from remote sensing images, which can solve the problem that large visual models are difficult to deploy on UAV platforms for ground feature extraction in the prior art. It improves the accuracy of the model while reducing the computational resource requirements of the model and improves the applicability to UAV platforms.

[0041] The execution entity of the remote sensing image feature extraction method provided in this disclosure can be a computer or server, or it can be other electronic devices with data processing capabilities; alternatively, the execution entity can be a processor (e.g., a central processing unit, CPU) in the aforementioned electronic device; furthermore, the execution entity can be an application (APP) installed in the aforementioned electronic device that can implement the function of the method; or the execution entity can be a functional module or unit in the aforementioned electronic device that has the function of the method, etc. No limitation is placed on the execution entity of this method.

[0042] The method for extracting ground features from remote sensing images is illustrated below with reference to the accompanying drawings.

[0043] Figure 1 This is a schematic flowchart of the remote sensing image feature extraction method provided in this application embodiment. Wherein, Figure 1 This is merely one execution order shown in the embodiments of this application and does not represent the only execution order of the remote sensing image feature extraction method. Where the final result can be achieved, Figure 1 The steps shown can be performed in parallel or in reverse order. For example... Figure 1 As shown, the method may include S101 to S103.

[0044] S101. Obtain the first training set and the second training set.

[0045] The first training set includes multiple sample groups, each of which includes multiple first remote sensing images and multiple second remote sensing images. The first and second remote sensing images are both cropped from the same original remote sensing image, and the size of the second remote sensing image is larger than that of the first remote sensing image. The second training set includes multiple sample pairs, each of which includes a third remote sensing image and a fourth remote sensing image. The fourth remote sensing image is the third remote sensing image after the target land cover type is masked by semantic masking.

[0046] For example, the first remote sensing image may be a global view image that includes most of the area in the original remote sensing image, and the second remote sensing image may be a local view image that includes a small part of the area in the original remote sensing image.

[0047] For example, taking the original remote sensing images including image A, image B, and image C as an example, images A, B, and C are processed sequentially. Cropping image A yields a global view image A1 encompassing most of image A, a global view image A2 encompassing most of image A, a partial view image a1 encompassing a small portion of image A, a partial view image a2 encompassing a small portion of image A, and a partial view image a3 encompassing a small portion of image A. Cropping image B yields a global view image B1 encompassing most of image B, a global view image B2 encompassing most of image B, a partial view image b1 encompassing a small portion of image B, a partial view image b2 encompassing a small portion of image B, and a partial view image b3 encompassing a small portion of image B. Cropping image C yields a global view image C1 encompassing most of image C, a global view image C2 encompassing most of image C, a partial view image c1 encompassing a small portion of image C, a partial view image c2 encompassing a small portion of image C, and a partial view image c3 encompassing a small portion of image C.

[0048] Therefore, images A1, A2, B1, B2, C1, and C2 can be identified as the first remote sensing images, and images a1, a2, a3, b1, b2, b3, c1, c2, and c3 can be identified as the second remote sensing images; images A1, A2, a1, a2, and a3 can be used as one sample group, images B1, B2, b1, b2, and b3 can be used as one sample group, and images C1, C2, c1, c2, and c3 can be used as one sample group.

[0049] For example, the target feature type may include roads, buildings, etc., without limitation; the fourth remote sensing image may be obtained by manually masking the target feature type in the third remote sensing image through semantic masking, and the semantic mask corresponding to different target feature types is also different.

[0050] For example, the target land cover types corresponding to the fourth remote sensing image in multiple sample pairs can be the same or different.

[0051] For example, the third remote sensing image includes images D1, D2, and D3, and the corresponding fourth remote sensing images are images d1, d2, and d3, respectively. The target feature type corresponding to image d1 can be a road, the target feature type corresponding to image d2 can be a building, and the target feature type corresponding to image d3 can be vegetation.

[0052] S102. Construct an initial ground feature extraction network. Train the initial ground feature extraction network based on the first training set and the second training set to obtain the target ground feature extraction model.

[0053] The initial ground feature extraction network includes a convolution pre-processing module, a patch embedding module, a label encoding module, an encoder module, and a feature mapping module. The remote sensing image is input into the convolution pre-processing module for feature extraction and dimensionality reduction to reduce the number of model parameters, resulting in the target convolutional features. The target convolutional features are then input into the patch embedding module for block-based processing and flattening, yielding the first intermediate features. The first intermediate features are then input into the label encoding module for category labeling and location encoding, followed by random deactivation, resulting in the second intermediate features. The second intermediate features are then input into the encoder module for feature extraction, yielding the third intermediate features. Finally, the third intermediate features are input into the feature mapping module for layer normalization, classification token extraction, and feature mapping, resulting in the ground feature characteristics.

[0054] Specifically, the encoder module includes multiple encoder units connected in sequence. The encoder unit is used to perform layer normalization processing on the features of the input encoder unit and then input them into the multi-head attention layer to obtain the first feature to be processed. The first feature to be processed is randomly deactivated and then added to the features of the input encoder unit to obtain the second feature to be processed. The second feature to be processed is layer normalized and then input into the MLP unit to obtain the third feature to be processed. The third feature to be processed is randomly deactivated and then added to the second feature to obtain the output feature of the encoder unit and output it.

[0055] Specifically, the pre-convolution module is used to input the input remote sensing image into the first convolutional layer to obtain the first convolutional feature. After batch normalization and activation function, the first convolutional feature is input into the second convolutional layer to obtain the second convolutional feature. After batch normalization and activation function, the second convolutional feature is input into the third convolutional layer to obtain the third convolutional feature. After batch normalization and activation function, the target convolutional feature is obtained.

[0056] For example, the data from the first and second training sets can be directly used to train the initial ground feature extraction network using the gradient descent method to obtain the target ground feature extraction model.

[0057] For example, when training the initial ground feature extraction network, cross-entropy loss function and / or orthogonal loss function can be used as loss function, and there are no restrictions on the loss function.

[0058] The following specific example further illustrates the network structure of the initial ground feature extraction network.

[0059] Figure 2 A schematic diagram of the network structure of the initial feature extraction network provided in this embodiment of the application. (Reference) Figure 2 The convolutional pre-module in the initial ground feature extraction network (i.e. Figure 2The Conv2d module contains three convolutional layers, each followed by a batch normalization layer and a ReLU activation function (the specific structure of the convolutional pre-module is not shown). In the convolutional pre-module, the first convolutional layer has a 7×7 kernel, a stride of 2, and padding of 3; the second convolutional layer has a 3×3 kernel, a stride of 2, and padding of 1; and the third convolutional layer has a 3×3 kernel, a stride of 4, and padding of 3. For a remote sensing image with an input size of 224×224×3, a target convolutional feature of size 14×14×384 can be obtained.

[0060] Compared to directly using the original image as input to the entire Transformer network, this significantly reduces the number of visual tokens fed into subsequent Transformer networks, effectively reducing the computational complexity of subsequent self-attention. At the same time, the local receptive field characteristics of convolution help extract low-level features such as texture and edges in remote sensing images, providing more semantically informative input for subsequent Transformers. Compared to the standard ViT (Vision Transformer) network, setting a convolutional pre-network can reduce the total number of parameters by approximately 30%, while improving feature extraction performance.

[0061] The target convolutional feature input is 14×14×384, which is the initial ground feature extraction network's patch embedding module (i.e., ... Figure 2 After performing block processing and flattening on Patch Embedding, a first intermediate feature with a size of 196×384 can be obtained.

[0062] The patch embedding module can convert target convolutional features into serialized patch tokens and flatten the spatial dimensions through the Flatten operation.

[0063] The first intermediate feature input marker encoding module with a size of 196×384 (i.e. Figure 2 The tagging and encoding module performs category tagging and position encoding, and then performs random deactivation. The tagging and encoding module first concatenates the first intermediate feature with a class token of size 1×384 (i.e., Figure 2 The concat algorithm (in the code) yields a feature of size 197×384. Then, the feature of size 197×384 is coupled with a positional encoding of size 197×384 (i.e., ...). Figure 2 The PositionEmbeddings are added together to obtain a feature of size 197×384. Then, the feature of size 197×384 is randomly deactivated (i.e., ...). Figure 2 After Dropout, a second intermediate feature with a size of 197×384 is obtained.

[0064] The second intermediate feature with a size of 197×384 is input into the encoder module (i.e.) Figure 2 The TransformerEncoder in the algorithm extracts features to obtain a third intermediate feature with a size of 197×384.

[0065] The encoder module includes 12 encoder units connected in sequence (i.e., Figure 2 (Encoder Block in the middle). Figure 3 This is a schematic diagram of the encoder unit provided in an embodiment of this application. Figure 3 As shown, each encoder unit performs layer normalization processing on the features of the input encoder unit (i.e., Figure 3 After Layer Norm, input a multi-head attention layer with a head count of 6 (i.e., ...). Figure 2 Multi-Head Attention (in the context of multi-head attention) is used to obtain the first feature to be processed; the first feature to be processed is then randomly deactivated (i.e., ... Figure 3 After Dropout, the features are added to the features of the input encoder unit to obtain the second feature to be processed; the second feature to be processed is then subjected to layer normalization (i.e., Figure 3 After Layer Norm in the MLP unit (i.e. Figure 3 The MLP Block in the process is used to obtain the third feature to be processed; the third feature to be processed is then randomly deactivated (i.e., Figure 3 The Dropout feature is added to the second feature to be processed to obtain the output feature of the encoder unit and then output.

[0066] For example, the MLP unit includes two linear layers. The features input to the MLP unit are activated by an activation function and then randomly deactivated after passing through the first linear layer. After passing through the second linear layer, they are randomly deactivated again to obtain the output features of the MLP unit. The activation function in the MLP unit can be the ReLU activation function or the GELU activation function.

[0067] Through the self-attention mechanism, the model can capture the global relationships between different features, extract semantic information in complex scenes, directly model long-distance dependencies between different land features, and capture complex spatial semantic structures in remote sensing images, such as road networks, building layouts, and other macroscopic surface features.

[0068] The third intermediate feature with a size of 197×384 is input into the feature mapping module (i.e. Figure 2The MLP Head undergoes layer normalization, classification token extraction, and feature mapping. The feature mapping module first performs layer normalization on the third intermediate feature with a size of 197×384 (i.e., ...). Figure 2 The Layer Norm in the algorithm is used to obtain features of size 197×384. These features are then subjected to token extraction and classification processing (i.e.,...). Figure 2 Extract the class token to obtain a feature of size 1×384, and then extract the feature representation corresponding to the class token (i.e., Figure 2 Pre-Logits in the middle), and then through a linear layer (i.e. Figure 2 The Linear array maps the features to the desired output dimension, ultimately yielding ground feature features of size 1×384.

[0069] S103. Extract ground features from the input remote sensing image using the target ground feature extraction model to obtain ground feature characteristics.

[0070] For example, after obtaining the target ground feature extraction model, the remote sensing image for which ground feature extraction is required can be input into the target ground feature extraction model. The target ground feature extraction model identifies and extracts the target ground feature in the remote sensing image to obtain the ground feature corresponding to the remote sensing image.

[0071] This application embodiment constructs an initial ground feature extraction network by acquiring a first training set and a second training set. This network includes a convolutional pre-processing module for preliminary feature extraction and dimensionality reduction, and an encoder module for better extraction of semantic information. The initial ground feature extraction network is trained to obtain a target ground feature extraction network. The target ground feature extraction network is used to extract ground features from the input remote sensing image to obtain ground feature characteristics. This can improve the model accuracy while reducing the computational resource requirements of the model and improve the applicability to UAV platforms.

[0072] Figure 4 This is another schematic flowchart illustrating the remote sensing image feature extraction method provided in this application embodiment. Some possible implementations include... Figure 4 As shown, the initial ground feature extraction network is trained based on the first and second training sets to obtain the target ground feature extraction model, including S401 to S405.

[0073] S401. Input the first remote sensing image into the preset ground feature extraction model and pass it through feature centering and activation function to obtain the first feature; input the first remote sensing image into the initial ground feature extraction network and pass it through activation function to obtain the second feature; input the second remote sensing image into the initial ground feature extraction network and pass it through activation function to obtain the third feature.

[0074] For example, the preset feature extraction model can be a pre-trained large ViT model (e.g., the ViT-Large model) with feature extraction capabilities, and there are no restrictions on the specific structure of the preset feature extraction model.

[0075] For example, the activation functions in S401 can all be softmax functions.

[0076] S402. Based on the first feature, the second feature, and the third feature, and using the first loss function, update the parameters of the initial ground feature extraction network until the first loss function converges to obtain the intermediate ground feature extraction network.

[0077] For example, during training, the model parameters of the preset ground feature extraction model can be frozen, and the parameters are updated only for the initial ground feature extraction network.

[0078] Specifically, the first loss function is shown in equation (1):

[0079]

[0080] In equation (1), L distill Let H(·) represent the first loss function, H(·) represent the cross-entropy loss function, N represent the sum of the number of first and second remote sensing images corresponding to the same original remote sensing image, M represent the number of first remote sensing images corresponding to the same original remote sensing image, and NM represent the number of second remote sensing images corresponding to the same original remote sensing image. Represents the i-th first feature. This represents the i-th second feature. This represents the j-th third feature.

[0081] It is understandable that using the feature distribution output by the preset feature extraction model as the learning target of the initial feature extraction model, and utilizing the cross-entropy loss of the first and second features (both corresponding to the global view), the initial feature extraction network can be forced to approximate the feature distribution of the preset feature extraction model at the overall semantic level, ensuring high-level semantic consistency. Utilizing the cross-entropy loss of the first and third features (corresponding to the global view and the local view), fine-grained feature matching can be used to improve the sensitivity of the initial feature extraction network to local texture features. This hierarchical loss combination allows the resulting intermediate feature extraction network to inherit the strong representational priors of the preset feature extraction model while possessing lightweight architectural characteristics, ultimately achieving a balance between feature extraction accuracy and computational efficiency.

[0082] S403. Input the third and fourth remote sensing images into the intermediate ground feature extraction network to obtain the fourth and fifth features respectively.

[0083] S404. Input the fourth and fifth features into the fully connected network to obtain the sixth and seventh features respectively.

[0084] It is understandable that fully connected networks are used to further project the features obtained from intermediate feature extraction networks.

[0085] S405. Based on the sixth and seventh features, and using the second loss function, update the parameters of the intermediate ground feature extraction network until the second loss function converges to obtain the target ground feature extraction model.

[0086] Specifically, the second loss function is shown in equation (2) below:

[0087]

[0088] In equation (2), L orth Let z represent the second loss function. S′ img The sixth feature, z S′ mask_img Representing the seventh feature, z S′ img ·z S′ mask_img ||z| represents the inner product of the sixth and seventh features. S′ img || represents the norm of the sixth feature, ||z S′ mask_img || represents the norm of the seventh feature.

[0089] It is understandable that by setting a second loss function, the intermediate feature extraction network can focus more on learning the information of the target feature type in the occluded part, that is, the feature of road, building and other types of features. This encourages the intermediate feature extraction network to generate feature space distributions as orthogonal as possible for different versions of the same image, thereby improving the ability to distinguish important semantic features of the occluded part.

[0090] This embodiment trains an initial ground feature extraction network based on the first feature, the second feature, and the third feature, using a first loss function. This results in an intermediate ground feature extraction network that inherits the strong representational prior of the preset ground feature extraction model and has a lightweight architecture. By training the intermediate ground feature extraction network based on the sixth feature and the seventh feature, using a second loss function, a target ground feature extraction model with better semantic information of ground feature features of the target ground feature type and higher ground feature extraction accuracy can be obtained.

[0091] In some possible implementations, the fourth remote sensing image is obtained through an occlusion step, which includes:

[0092] Based on the third remote sensing image and user prompts, images of the target land cover type are obtained using the SAM model; the images of the target land cover type in the third remote sensing image are occluded to obtain the fourth remote sensing image.

[0093] It is understandable that the SAM model is a pre-trained image segmentation model released by Meta in 2023, which can achieve accurate segmentation with a small number of cues (i.e., user prompts).

[0094] For example, the user prompt can be a point prompt or a box prompt. An interactive segmentation tool based on the SAM model can be used, inputting a third remote sensing image and the user prompt, to extract the target land cover type from the image. A high-precision semantic mask can be generated for the target land cover type image, ensuring the accuracy of the mask boundaries and semantic consistency. The third remote sensing image containing this semantic mask is then identified as the fourth remote sensing image. In this way, the fourth remote sensing image can be obtained quickly and accurately.

[0095] In some possible implementations, after obtaining the training set, the remote sensing images in the training set can be normalized, and at least one of the hue, saturation, and brightness of the random remote sensing images can be adjusted. The remote sensing images can also be translated, scaled, rotated, have their aspect ratio changed, horizontally flipped, or vertically flipped. This can accelerate model training, enhance model robustness, and prevent overfitting.

[0096] Verification Experiment

[0097] Comparative Experiment 1: Using different neural network models trained on the same training set to extract ground features, and then using the extracted ground features to retrieve remote sensing images, the accuracy of different neural network models was compared.

[0098] Based on publicly available raw remote sensing images, a first training set consisting of 18,000 remote sensing images and a second training set consisting of 8,000 remote sensing images were constructed. Different neural network models trained on the first and second training sets (including the Dino-v2(s) model, the Dino-v2(b) model, the ResNet-50 model, the ResNet-101 model, and the Ours model (i.e., the target feature extraction model of this application) and the Ours' model (referring to a model with increased parameters based on the target feature extraction model of this application)) were used for feature extraction. Remote sensing image retrieval was then performed based on the extracted features, and the accuracy comparison results are shown in Table 1.

[0099] Table 1. Accuracy Comparison Results of Different Neural Network Models

[0100]

[0101] Wherein, SR@k represents the retrieval success rate, which indicates the proportion of query images that contain at least one correct match in the Top-k retrieval results; mAP represents the mean precision, used to measure model performance.

[0102] Referring to the accuracy comparison results in Table 1, it can be seen that the target feature extraction model of this application outperforms existing mainstream methods in all evaluation metrics. On the one hand, compared with the Dino-v2(s) and Dino-v2(b) models based on the Vision Transformer model, the target feature extraction model of this application has stronger stability in remote sensing image retrieval tasks while significantly reducing the model size, indicating that the target feature extraction model of this application has better feature extraction capabilities, especially in the Top-10 task, further expanding its leading advantage. On the other hand, compared with the ResNet-50 and ResNet-101 models, the target feature extraction model of this application has significantly improved in feature representation ability and retrieval performance. Furthermore, generally speaking, the larger the number of model parameters, the better its ability to learn feature details, thus having a greater performance advantage. However, compared with Ours' model, which has increased the number of parameters, the target feature extraction model of this application has better model performance despite having fewer parameters, indicating that the target feature extraction model of this application has improved performance while having fewer parameters, making it more suitable for UAV platforms.

[0103] Comparative Experiment 2: Using the target feature extraction model of this application trained on different training sets, feature extraction was performed, and remote sensing image retrieval was performed based on the extracted feature extraction. The accuracy of different training sets was compared.

[0104] Based on the first and second training sets used in Comparative Experiment 1, training sets A, B, C, D, E, and F are constructed.

[0105] The difference between training sets A, B, and C is that the target feature types corresponding to the fourth remote sensing image in training set A only include roads (i.e., the occluded area only includes roads). The target feature types corresponding to the fourth remote sensing image in training set B only include buildings (i.e., the occluded area only includes buildings); and the target feature types corresponding to the fourth remote sensing image in training set C include both roads and buildings (i.e., the occluded area includes both roads and buildings).

[0106] The difference between training set D and training set A is that in training set D, the road area is no longer masked by semantic masking in the fourth remote sensing image of training set A. Instead, the brightness and contrast of the road area are randomly adjusted to weaken the road area.

[0107] The difference between training set E and training set B is that in training set E, the fourth remote sensing image of training set B no longer uses semantic masking to occlude the building areas, but only randomly adjusts the brightness and contrast of the building areas to weaken them.

[0108] The difference between training set F and training set C is that in training set F, the fourth remote sensing image of training set C no longer uses semantic masking to occlude road and building areas, but only randomly adjusts the brightness and contrast of the road and building areas to weaken them.

[0109] The brightness can be reduced by 30% from its original value, and the contrast can be reduced by 20% from its original value.

[0110] The accuracy comparison results for different training sets are shown in Table 2.

[0111] Table 2. Accuracy Comparison Results for Different Training Sets

[0112]

[0113] Wherein, SR@k represents the retrieval success rate, which indicates the proportion of query images that contain at least one correct match in the Top-k retrieval results; mAP represents the mean precision, used to measure model performance.

[0114] Referring to the accuracy comparison results in Table 2, it can be seen that training set C performs best across all metrics. Compared to other training sets, occlusion of multi-category semantic regions (road + building) significantly improves the retrieval success rate and ranking quality, indicating that this global semantic occlusion strategy has a significant effect on improving the robustness of the model. Single-category occlusion (such as "road" or "building"), i.e., training sets A and B, performs slightly worse, and the accuracy of training set A is better than that of training set B, suggesting that building-related semantic regions contribute more to the accuracy.

[0115] Training set F performs worse than training set C, but better than training sets D and E. This indicates that the strategy of weakening multi-category semantic regions (roads + buildings) can also effectively improve retrieval accuracy, but its overall effect is not as good as the occlusion strategy. In the single-category weakening strategy, training sets D and E perform similarly, with training set E having a slightly higher SR@k than training set D, but training set E is slightly inferior to training set D in terms of mAP.

[0116] Overall, the strategy of occluding semantic regions outperforms the strategy of weakening semantic regions. This indicates that directly occluding semantic regions can more effectively force the model to focus on other key information, thereby improving retrieval performance. For a single semantic category, occluding buildings is more effective than weakening buildings, while occluding roads is slightly less effective than weakening roads. This suggests that different landmark categories contribute differently to retrieval performance, and the salience of semantic regions related to buildings may be stronger.

[0117] Table 3 shows a comparison of the number of parameters and the runtime per image between the target feature extraction model of this application and mainstream feature extraction models (i.e., the models in Comparison Experiment 1). Referring to Table 3, it can be seen that the target feature extraction model of this application (i.e., the Ours model) significantly reduces the number of parameters while maintaining feature extraction capabilities, demonstrating good lightweight characteristics. Compared with the Dino-v2(s) and Dino-v2(b) models based on Vision Transformer, the target feature extraction model of this application has a smaller feature dimension, reducing storage overhead and computational complexity, while also having faster inference speed, making it suitable for resource-constrained application scenarios. Compared with the ResNet-50 and ResNet-101 models, the target feature extraction model of this application has a greater advantage in runtime, further verifying its effectiveness in improving computational efficiency. In summary, the target feature extraction model of this application achieves a balance between computational cost, storage requirements, and inference efficiency in feature extraction tasks, providing an optimal solution for efficient remote sensing image analysis.

[0118] Table 3 Retrieval time of different feature extraction networks

[0119] Method Parameters Feature Dimension Single image processing time (seconds) Dino-v2(s) 21M 384 0.19 Dino-v2(b) 86M 768 0.4 ResNet-50 25.6M 2048 0.2 ResNet-101 44.5M 2048 0.29 Ours 18.6M 128 0.15 Ours' 39.8M 256 0.27

[0120] While this application provides method operation steps as shown in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in this embodiment is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the method can be executed sequentially according to this embodiment or the accompanying drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0121] like Figure 5 As shown in the illustration, this application also provides a device for extracting ground feature characteristics from remote sensing images. The device includes:

[0122] The acquisition module 501 is used to acquire a first training set and a second training set. The first training set includes multiple sample groups, each sample group including multiple first remote sensing images and multiple second remote sensing images. The first and second remote sensing images are both cropped from the same original remote sensing image, and the size of the second remote sensing image is larger than the size of the first remote sensing image. The second training set includes multiple sample pairs, each sample pair including a third remote sensing image and a fourth remote sensing image. The fourth remote sensing image is the third remote sensing image after the target land cover type is occluded by semantic masking.

[0123] The model building and training module 502 is used to build an initial ground feature extraction network. The initial ground feature extraction network is trained using a first training set and a second training set to obtain a target ground feature extraction model. The initial ground feature extraction network includes a convolution pre-processing module, a patch embedding module, a label encoding module, an encoder module, and a feature mapping module. Remote sensing images are input into the convolution pre-processing module for feature extraction and dimensionality reduction to reduce the number of model parameters, resulting in target convolutional features. The target convolutional features are input into the patch embedding module for block processing and flattening, resulting in first intermediate features. The first intermediate features are input into the label encoding module for category labeling and location encoding, followed by random deactivation, resulting in second intermediate features. The second intermediate features are input into the encoder module for feature extraction, resulting in third intermediate features. The third intermediate features are input into the feature mapping module for layer normalization, classification token extraction, and feature mapping, resulting in ground feature features.

[0124] The extraction module 503 is used to extract ground features from the input remote sensing image through the target ground feature extraction model to obtain ground feature features.

[0125] The beneficial effects and specific implementation methods of this device embodiment can be referred to the foregoing method embodiment, and will not be repeated here.

[0126] Some modules in the apparatus described in this application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0127] The apparatus or module described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. For ease of description, the above apparatus is described by dividing it into various modules according to their functions. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0128] The methods, apparatus, or modules described in this application can be implemented in a computer-readable program code manner. The controller can be implemented in any suitable manner, such as a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of a memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code manner, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included within it for implementing various functions can also be considered as structures within the hardware component. Alternatively, the device used to implement various functions can be viewed as either a software module that implements the method or a structure within a hardware component.

[0129] This application also provides an apparatus, the apparatus comprising: a processor; a memory for storing processor-executable instructions; wherein, when the processor executes the executable instructions, it implements the method described in this application.

[0130] This application also provides a non-volatile computer-readable storage medium storing a computer program or instructions thereon, which, when executed, enables the method described in this application embodiment to be implemented.

[0131] Furthermore, in the various embodiments of the present invention, each functional module can be integrated into a processing module, or each module can exist independently, or two or more modules can be integrated into a single module.

[0132] The aforementioned storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), cache, hard disk drive (HDD), or memory card. The memory can be used to store computer program instructions.

[0133] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, or it can be embodied in the process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0134] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0135] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of this application.

Claims

1. A method for extracting ground feature characteristics from remote sensing images, characterized in that, include: Obtain the first and second training sets; The first training set includes multiple sample groups, each sample group including multiple first remote sensing images and multiple second remote sensing images. The first and second remote sensing images are both cropped from the same original remote sensing image, and the size of the second remote sensing image is larger than the size of the first remote sensing image. The second training set includes multiple sample pairs, each sample pair including a third remote sensing image and a fourth remote sensing image. The fourth remote sensing image is the third remote sensing image after the target land cover type is semantically masked. An initial ground feature extraction network is constructed, and the initial ground feature extraction network is trained based on the first training set and the second training set to obtain the target ground feature extraction model; The target feature extraction model is used to extract features from the input remote sensing image to obtain the feature characteristics. The initial ground feature extraction network includes a convolution pre-module, a patch embedding module, a label encoding module, an encoder module, and a feature mapping module. The remote sensing image is input into the convolution pre-processing module for feature extraction and dimensionality reduction to reduce the number of model parameters and obtain the target convolutional feature. The target convolutional feature is input into the patch embedding module for block processing and flattening to obtain the first intermediate feature. The first intermediate feature is input into the label encoding module for category labeling and location encoding, and random deactivation processing to obtain the second intermediate feature. The second intermediate feature is input into the encoder module for feature extraction to obtain the third intermediate feature. The third intermediate feature is input into the feature mapping module for layer normalization, extraction of classification tokens, and feature mapping to obtain the ground feature.

2. The method according to claim 1, characterized in that, The encoder module includes multiple encoder units connected in sequence; The encoder unit is used to perform layer normalization processing on the features input to the encoder unit and then input them into the multi-head attention layer to obtain a first feature to be processed. The first feature to be processed is randomly deactivated and then added to the features input to the encoder unit to obtain a second feature to be processed. The second feature to be processed is then subjected to layer normalization processing and input into the MLP unit to obtain a third feature to be processed. The third feature to be processed is then randomly deactivated and then added to the second feature to obtain the output feature of the encoder unit and output it.

3. The method according to claim 1, characterized in that, The pre-convolution module is used to input the input remote sensing image into the first convolutional layer to obtain the first convolutional feature. After batch normalization and activation function, the first convolutional feature is input into the second convolutional layer to obtain the second convolutional feature. After batch normalization and activation function, the second convolutional feature is input into the third convolutional layer to obtain the third convolutional feature. After batch normalization and activation function, the third convolutional feature is obtained as the target convolutional feature.

4. The method according to claim 1, characterized in that, The step of training the initial ground feature extraction network based on the first training set and the second training set to obtain the target ground feature extraction model includes: The first remote sensing image is input into a preset ground feature extraction model and processed by feature centering and activation functions to obtain a first feature; the first remote sensing image is input into the initial ground feature extraction network and processed by activation functions to obtain a second feature; the second remote sensing image is input into the initial ground feature extraction network and processed by activation functions to obtain a third feature. Based on the first feature, the second feature, and the third feature, the parameters of the initial ground feature extraction network are updated according to the first loss function until the first loss function converges, thus obtaining the intermediate ground feature extraction network. The third and fourth remote sensing images are respectively input into the intermediate ground feature extraction network to obtain the fourth and fifth features respectively; The fourth and fifth features are input into a fully connected network to obtain the sixth and seventh features, respectively. Based on the sixth and seventh features, the intermediate ground feature extraction network is updated with parameters according to the second loss function until the second loss function converges, thus obtaining the target ground feature extraction model.

5. The method according to claim 4, characterized in that, The first loss function is shown in equation (1): In equation (1), L distill Let H(·) represent the first loss function, H(·) represent the cross-entropy loss function, N represent the sum of the number of first and second remote sensing images corresponding to the same original remote sensing image, M represent the number of first remote sensing images corresponding to the same original remote sensing image, and NM represent the number of second remote sensing images corresponding to the same original remote sensing image. Represents the i-th first feature. This represents the i-th second feature. This represents the j-th third feature.

6. The method according to claim 4, characterized in that, The second loss function is shown in equation (2) below: In equation (2), L orth Let z represent the second loss function. S′ img The sixth feature, z S′ mask_img Representing the seventh feature, z S′ img ·z S′ mask_img ||z| represents the inner product of the sixth and seventh features. S′ img || represents the norm of the sixth feature, ||z S′ mask_img || represents the norm of the seventh feature.

7. The method according to claim 1, characterized in that, The fourth remote sensing image is obtained through an occlusion step, which includes: Based on the third remote sensing image and user prompts, and using the SAM model, an image of the target land cover type is obtained; The target land cover type in the third remote sensing image is occluded to obtain the fourth remote sensing image.

8. A device for extracting ground feature characteristics from remote sensing images, characterized in that, include: The acquisition module is used to acquire a first training set and a second training set; wherein, the first training set includes multiple sample groups, each sample group includes multiple first remote sensing images and multiple second remote sensing images, the first remote sensing images and the second remote sensing images are both cropped from the same original remote sensing image, and the size of the second remote sensing image is larger than the size of the first remote sensing image; the second training set includes multiple sample pairs, each sample pair includes a third remote sensing image and a fourth remote sensing image, the fourth remote sensing image being a third remote sensing image after semantic masking of the target ground feature type; The model building and training module is used to build an initial ground feature extraction network, and train the initial ground feature extraction network according to the first training set and the second training set to obtain the target ground feature extraction model. The extraction module is used to extract ground features from the input remote sensing image using the target ground feature extraction model to obtain ground feature characteristics; The initial ground feature extraction network includes a convolution pre-processing module, a patch embedding module, a label encoding module, an encoder module, and a feature mapping module. The remote sensing image is input into the convolution pre-processing module for feature extraction and dimensionality reduction to reduce the number of model parameters, resulting in target convolutional features. The target convolutional features are then input into the patch embedding module for block segmentation and flattening to obtain first intermediate features. The first intermediate features are input into the label encoding module for category labeling and location encoding, followed by random deactivation to obtain second intermediate features. The second intermediate features are then input into the encoder module for feature extraction to obtain third intermediate features. Finally, the third intermediate features are input into the feature mapping module for layer normalization, classification token extraction, and feature mapping to obtain ground feature features.

9. An apparatus for performing a method for extracting ground feature characteristics from remotely sensed images, characterized in that, include: processor; Memory used to store processor-executable instructions; When the processor executes the executable instructions, it implements the method as described in any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium, characterized in that, Includes storage of computer programs or instructions that, when executed, cause the method as described in any one of claims 1 to 7 to be implemented.

Images