Remote sensing image segmentation method and device

Abstract

The present invention provides a remote sensing image segmentation method and device, belonging to the field of image processing technology, the method comprising: extracting a feature map of an input image using an image encoder; obtaining a similar feature map of the feature map and each reference sample feature in a pre-created reference sample library; selecting a target similar feature map with the largest similarity peak from all similar feature maps, and inputting the target similar feature map into a prompt embedding generation network to generate a prompt embedding; the prompt embedding generation network is a pre-built convolutional neural network; inputting the prompt embedding and the input image into an image decoder, and outputting a segmentation result. The remote sensing image segmentation method and device provided by the present invention enhances the adaptability to data in a specific field, and also provides a high degree of automated processing flow, which can automatically generate prompt embedding without prior prompts.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a remote sensing image segmentation method and device. Background Art

[0002] At present, the remote sensing image segmentation task of water conservancy scenes faces many challenges, such as complex and diverse scene targets and limited annotated data. Existing methods are difficult to take into account both segmentation accuracy and generalization at the same time. Visual basic large models represented by Segment Anything Model (SAM) have made significant progress in the field of image segmentation, but as a category-free segmentation method that requires prior hints, it is mainly oriented to interactive segmentation tasks and has weak adaptability to automatic segmentation in specific fields. Therefore, there is a need for an automated segmentation method that can efficiently adapt to water conservancy scenes with limited annotated data. Summary of the invention

[0003] The present invention provides a remote sensing image segmentation method and device, which are used to solve the problems of weak data feature adaptability and low automation level of large visual models in water conservancy scene remote sensing image segmentation tasks in the prior art.

[0004] In a first aspect, the present invention provides a remote sensing image segmentation method, comprising: extracting a feature map of an input image using an image encoder; obtaining a similarity feature map between the feature map and each reference sample feature in a pre-created reference sample library; wherein the reference sample feature is a feature vector of a pre-set foreground pixel, and the similarity feature map is used to characterize the similarity between the feature vector of the input image at each pixel position and the reference sample feature; selecting a target similarity feature map with a maximum similarity peak from all similarity feature maps, and inputting the target similarity feature map into a prompt embedding generation network to generate a prompt embedding; the prompt embedding generation network is a pre-constructed convolutional neural network; inputting the prompt embedding and the input image into an image decoder, and outputting a segmentation result.

[0005] According to the remote sensing image segmentation method provided by the present invention, the step of constructing a reference sample library includes: obtaining a target feature map of each sample image based on the feature map of each sample image extracted by the image encoder in combination with the corresponding true mask; performing cluster analysis on the feature vector of each foreground pixel in all target feature maps; and using the feature vectors representing each cluster category as reference sample features to construct the reference sample library.

[0006] According to the remote sensing image segmentation method provided by the present invention, the prompt embedding generation network includes: an RPN layer, a RoIPooling layer and a Prompt Head layer; the RPN layer is used to generate a candidate box from the target similarity feature map to mark the target area of ​​interest in the target similarity feature map; the RoI Pooling layer is used to extract a feature map of a fixed size from each target area; the Prompt Head layer is used to generate a prompt embedding for input image segmentation based on the feature map.

[0007] According to the remote sensing image segmentation method provided by the present invention, a cluster analysis is performed on the feature vector of each foreground pixel in all target feature maps, including: calculating the cosine similarity of each pair of feature vectors in all feature vectors; determining other feature vectors whose cosine similarity with the current feature vector is greater than or equal to a similarity threshold; when the number of other feature vectors is greater than or equal to a preset number, merging the current feature vector and the other feature vectors into the same cluster to cluster the feature vectors.

[0008] According to the remote sensing image segmentation method provided by the present invention, the feature vectors representing the respective cluster categories are the feature vectors closest to the cluster centers of the respective cluster categories.

[0009] According to the remote sensing image segmentation method provided by the present invention, the image encoder is a SAM encoder that is fine-tuned based on a low-rank adaptive technology; and the image decoder is a SAM decoder.

[0010] According to the remote sensing image segmentation method provided by the present invention, the similarity between the feature vector at each pixel position and the reference sample feature is the cosine similarity.

[0011] In a second aspect, the present invention further provides a remote sensing image segmentation device, comprising:

[0012] A first processing module, configured to extract a feature map of an input image using an image encoder;

[0013] The second processing module is used to obtain a similarity feature map between the feature map and each reference sample feature in a pre-created reference sample library; wherein the reference sample feature is a pre-set feature vector of a foreground pixel, and the similarity feature map is used to characterize the similarity between the feature vector of the input image at each pixel position and the reference sample feature;

[0014] A third processing module is used to select a target similarity feature map with the largest similarity peak from all similarity feature maps, and input the target similarity feature map into a prompt embedding generation network to generate a prompt embedding; the prompt embedding generation network is a pre-built convolutional neural network;

[0015] The fourth processing module is used to input the hint embedding and the input image into an image decoder and output a segmentation result.

[0016] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the remote sensing image segmentation methods described above are implemented.

[0017] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the remote sensing image segmentation methods described above.

[0018] The remote sensing image segmentation method and device provided by the present invention obtain a variety of similar feature maps based on reference sample features in a pre-established reference sample library, and determine the similar feature map with the highest similarity peak value (i.e., the target similar feature map) for subsequent use, and combine effective prompt embedding generation and a sophisticated decoding process to enhance the model's adaptability to data in specific fields, and also provide a high degree of automated processing flow, which can automatically generate prompt embedding without the need for prior prompts.

[0019] The present invention designs a method for constructing a reference sample library based on representative samples. The construction of this sample library not only enhances the adaptability of the segmentation model in unknown scenarios, but also provides support for the subsequent training and reasoning process of the prompt embedding generation network. Compared with the traditional sample selection method, it improves the diversity of samples and the coverage of feature distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is one of the flow charts of the remote sensing image segmentation method provided by the present invention;

[0022] Figure 2 This is the second flow chart of the remote sensing image segmentation method provided by the present invention;

[0023] Figure 3 This is the third flow chart of the remote sensing image segmentation method provided by the present invention;

[0024] Figure 4 is a schematic diagram of a similar characteristic diagram provided by the present invention;

[0025] Figure 5 It is a structural schematic diagram of the remote sensing image segmentation device provided by the present invention;

[0026] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] It should be noted that in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be a connection between two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0029] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0030] Combine the following Figure 1-Figure 6The remote sensing image segmentation method and device provided by the embodiment of the present invention are described. It should be noted that the present invention can be applied to the segmentation of remote sensing images in water conservancy scenes, but is not limited to the above single application scene and image type.

[0031] Figure 1 It is one of the flowcharts of the remote sensing image segmentation method provided by the present invention. Figure 2 This is the second flow chart of the remote sensing image segmentation method provided by the present invention. Figure 3 This is the third flow chart of the remote sensing image segmentation method provided by the present invention. Figures 1 to 3 The remote sensing image segmentation method provided in the present invention is described.

[0032] The remote sensing image segmentation method provided by the present invention includes but is not limited to the following steps:

[0033] Step 101: Use an image encoder to extract a feature map of an input image.

[0034] The image encoder in the present invention may be a SAM encoder that is fine-tuned based on a low-rank adaptation technique (Low-Rank Adaptation, LoRA).

[0035] LoRA is a method for fine-tuning by introducing low-rank matrix decomposition into model parameters. The core idea is to locally adjust the model through a low-rank matrix without changing the original model parameters, so as to achieve the purpose of quickly adapting to new tasks. SAM's image encoder is based on the Vision Transformer (ViT) architecture and includes multiple self-attention modules. The present invention integrates LoRA technology in the query and value parts of the self-attention module (Attention Module) of the image encoder, because they are responsible for extracting correlations and calculating outputs from input features in the self-attention mechanism, which is crucial for understanding the spatial relationships and target details in remote sensing images, and can significantly improve the model's adaptability to complex water conservancy scenes. The specific implementation can be divided into the following steps:

[0036] (1) Freeze core weights: keep the query weight matrix Sum weight matrix This is because these weights have learned rich common features during the large-scale pre-training process, which can maintain the stability and versatility of the model.

[0037] (2) Introducing low-rank matrix for local adjustment: Through the low-rank matrix , and , , calculate the adjustment matrix and , and superimposed onto the original weight matrix, the calculation of query and value can be expressed as:

[0038]

[0039]

[0040] in, Represents the input matrix.

[0041] (3) Finally, the attention after LoRA adjustment is calculated as:

[0042]

[0043] in, K represents the key matrix; Indicates the dimension of the key vector, which is used to prevent the inner product value from being too large and causing the gradient to disappear; The function is used to normalize the attention weights into a probability distribution.

[0044] LoRA adds a set of low-rank matrices to the original self-attention layer, forming a supplementary adjustment to the original weight matrix. This adjustment is achieved through the product of low-rank matrices, fine-tuning the original weights without the need for large-scale parameter updates. After fine-tuning training on the water conservancy remote sensing image dataset, the image encoder showed a stronger expressive ability for the data distribution. The experimental results are shown in Table 1, showing the performance comparison of the model before fine-tuning (Baseline) and after fine-tuning (Finetune) under prior prompts.

[0045] Table 1 Comparison results of the models before and after fine-tuning under prior hints

[0046]

[0047] The intersection-over-union ratio ( IoU ), the harmonic mean of precision and recall ( F 1 value), precision ( Precision ) and recall ( Recall ) as the evaluation index, the results show that the fine-tuned model has shown significant improvements in most indicators; although Recall A slight decrease, but combined Precision and F 1, the fine-tuned model still performs better. This trade-off shows that the model pays more attention to improving precision while maintaining a high recall rate, making the segmentation results more accurate and controllable.

[0048] Step 102: Obtain similar feature maps of the feature map and each reference sample feature in a pre-created reference sample library.

[0049] The reference sample feature is a feature vector of a preset foreground pixel of the sample image, and the similarity feature map is used to characterize the similarity between the feature vector of each pixel position of the input image and the reference sample feature. The similarity mentioned here can be represented by cosine similarity.

[0050] The following is a brief description of how to calculate the similarity feature map between the feature map of the input image and any reference sample feature.

[0051] First, use the fine-tuned Image Encoder to encode the input image Perform feature extraction to obtain its feature map; then, calculate the cosine similarity between the feature vector at each pixel position in the feature map and any reference sample feature to obtain a similar feature map of the entire feature map. Figure 4 As shown, Figure 4 It is a schematic diagram of a similar characteristic diagram provided by the present invention.

[0052] Step 103: Select a target similarity feature map with the largest similarity peak from all similarity feature maps, and input the target similarity feature map into a prompt embedding generation network to generate a prompt embedding. The prompt embedding generation network is a pre-built convolutional neural network.

[0053] Since there may be large feature differences between similar targets in water conservancy scenes, the present invention selects samples that best match the target area in the input image from multiple reference sample features to ensure that the generated similarity feature map can accurately reflect the position and shape of the target object.

[0054] Assume that the reference sample library has m reference sample features, then the similarity feature map generated in step 102 has m The step 103 of the present invention is to m A method for determining a target similar feature map from similar feature maps.

[0055] The similarity peak refers to the maximum similarity value in a similar feature graph.

[0056] Step 104: Input the hint embedding and the input image into an image decoder, and output a segmentation result.

[0057] For each hint embedding, it is input into the image decoder (which can be a SAM decoder). In the present invention, the SAM decoder uses pre-trained weights to ensure its strong performance in general segmentation tasks, and fine-tunes it to better align with the hint embedding of the specific task. The SAM decoder segments the corresponding area in the input image according to its position information and features, and generates a mask of the target area:

[0058]

[0059] in, Indicates i The mask of the target region, Represents the SAM decoder, Indicates i Tips embedded, Represents the input image.

[0060] Because each The corresponding mask Only part of the target area in the image can be segmented, so all generated masks need to be merged to obtain the final segmentation result. :

[0061]

[0062] in, N is the number of all masks, express N The union of all masks.

[0063] In order to optimize both the image segmentation accuracy and classification accuracy, this paper combines Dice loss and cross entropy loss to design a hybrid loss function DiceCE Loss:

[0064] ;

[0065] Among them, the cross entropy loss ( ) is used to optimize the classification accuracy of each pixel; Dice loss ( Dice Loss ) is used to measure the overlap area between the predicted mask and the true mask:

[0066]

[0067] in, It represents the area where the predicted mask overlaps with the true mask, that is, the number of pixels in the intersection of the two. represents the area of ​​the predicted mask, that is, the total number of pixels predicted by the model as the target area; Represents the area of ​​the true mask, that is, the total number of pixels in the actual target area.

[0068] By combining these two loss functions, the present invention can better balance its performance in different aspects, especially has better robustness for the segmentation of imbalanced data sets and small objects.

[0069] The remote sensing image segmentation method provided by the present invention obtains multiple similar feature maps based on reference sample features in a pre-established reference sample library, and determines the similar feature map with the highest similarity peak value (i.e., the target similar feature map) for subsequent use, and combines effective prompt embedding generation and a sophisticated decoding process to enhance the model's adaptability to data in specific fields, and also provides a high degree of automated processing flow, which can automatically generate prompt embedding without the need for prior prompts.

[0070] Based on the content of the above embodiment, as an optional embodiment, the remote sensing image segmentation method provided by the present invention includes the steps of constructing a reference sample library:

[0071] (1) Based on the feature map of each sample image extracted by the image encoder and the corresponding true mask, the target feature map of each sample image is obtained.

[0072] For each sample image , combined with its true mask , extract the corresponding target feature map:

[0073] ;

[0074] in, represents the target feature map, Represents the image encoder for the sample image The feature map generated by feature extraction is Represents Hadamard multiplication.

[0075] Among them, the target feature map is a The characteristic matrix of is the number of foreground pixels, l is the dimension of the feature vector of the foreground pixel. The value of is determined based on the specific structure of the image encoder. It can be 256.

[0076] The feature vectors of the foreground pixels of all sample images are combined into a large feature matrix of size ,in are the number of foreground pixels in each image, is the total number of sample images.

[0077] (2) Perform cluster analysis on the feature vector of each foreground pixel in all target feature maps.

[0078] In order to reduce the redundancy in the sample library and ensure the representativeness of the samples, the present invention uses a clustering algorithm to perform cluster analysis on all the feature vectors in the above large feature matrix. The purpose of clustering is to classify feature vectors with similar features into the same cluster, and then select the most representative reference sample features from them.

[0079] The present invention can use a density clustering algorithm (DBSCAN), a kmeans clustering algorithm and other clustering algorithms to cluster the above feature vectors.

[0080] (3) The feature vectors representing the respective clustering categories are used as reference sample features to construct the reference sample library.

[0081] The feature vector representing each cluster category is the feature vector closest to the cluster center of the respective cluster category.

[0082] Based on the content of the above embodiment, as an optional embodiment, the remote sensing image segmentation method provided by the present invention performs cluster analysis on the feature vector of each foreground pixel in all target feature maps, including but not limited to the following steps:

[0083] (1) Calculate the cosine similarity of each pair of feature vectors in all feature vectors; the calculation of cosine similarity belongs to the prior art and will not be described in detail here.

[0084] (2) Determine other feature vectors whose cosine similarity with the current feature vector is greater than or equal to the similarity threshold.

[0085] Specifically, a similarity threshold (eg, 0.7) is preset, indicating that if the cosine similarity between two feature vectors is greater than or equal to the similarity threshold, the two feature vectors are considered to be in each other's neighborhood.

[0086] (3) When the number of other feature vectors is greater than or equal to a preset number, the current feature vector and other feature vectors are merged into the same cluster to perform feature vector clustering.

[0087] Set a preset number, which is an integer value representing the minimum number of other feature vectors whose cosine similarity with the current feature vector is greater than or equal to the similarity threshold.

[0088] The above process is repeated until all the eigenvectors that meet the conditions are assigned to a certain cluster. The present invention determines the neighbor relationship through cosine similarity. This strategy in the present invention is particularly suitable for processing high-dimensional data with complex structures (i.e., the eigenvectors in the present invention), which can improve the accuracy of clustering.

[0089] Based on the contents of the above embodiments, as an optional embodiment, the remote sensing image segmentation method provided by the present invention, the prompt embedding generation network includes: an RPN layer, a RoI Pooling layer and a Prompt Head layer; the RPN (region proposal network) layer is used to generate a candidate box from the target similarity feature map to mark the target area of ​​interest in the target similarity feature map; the RoI Pooling (region of interest pooling) layer is used to extract a feature map of a fixed size from each target area; the Prompt Head (prompt head) layer is used to generate a prompt embedding for input image segmentation according to the feature map.

[0090] As a category-independent segmentation method that requires prior cues, SAM is not suitable for fully automatic image understanding tasks. On the other hand, the output of SAM depends on the number, location and type of prior cues. To this end, this paper proposes a cue learning framework based on reference sample guidance, which aims to guide the model to automatically generate cue embeddings compatible with the SAM decoder by using a small number of samples as references, so as to achieve a more efficient segmentation process.

[0091] Specifically, the present invention uses a region proposal network (RPN) to generate candidate boxes from the similarity feature map to identify the target area. The RPN layer performs sliding window detection through a set of anchor points, and uses the foreground / background classifier to filter out the most likely target area to generate a set of candidate boxes:

[0092]

[0093] in, represents a set of candidate boxes, Indicates i Candidate boxes, Represents the similarity feature map The operation of extracting candidate boxes.

[0094] RPN is a widely used module, usually used in target detection tasks, and weights can be extracted from pre-trained models; the present invention fine-tunes RPN, and its loss function includes classification loss and positioning loss. The classification loss is calculated by the binary cross entropy loss function to calculate the foreground / background classification loss:

[0095]

[0096] in, represents the classification loss; represents the binary cross entropy loss function, Represents the predicted foreground / background probability, that is, the probability value that RPN predicts for each candidate box as the foreground (target area); Represents the true foreground / background label, that is, the actual category label of each candidate box, with foreground being 1 and background being 0.

[0097] The positioning loss is used to adjust the position of the candidate box so that it is as close to the target area as possible. The Huber loss function is commonly used:

[0098]

[0099] in, represents the positioning loss; represents the Huber loss function, Represents the predicted candidate box parameters, that is, the bounding box coordinates predicted by RPN for each candidate box; Represents the true candidate box parameters, that is, the bounding box coordinates of the true target area corresponding to each candidate box.

[0100] Comprehensive classification loss and positioning loss, total loss of RPN for:

[0101] .

[0102] After obtaining the candidate boxes, the present invention performs RoI Pooling operation on the target areas corresponding to these candidate boxes to extract fixed-size feature maps. In order to prevent the loss of position information during RoI Pooling, the present invention integrates sine-cosine position encoding (PE) into the target similarity feature map to ensure that the feature representation of the candidate area can retain spatial information:

[0103]

[0104] in, Represents a fixed-size feature map; Represents the RoI Pooling operation, that is, according to the candidate box , from the input feature map Extract fixed-size feature maps from . represents the target similarity feature map, Represents sine and cosine position encoding.

[0105] The main function of Prompt Head is to generate prompt embeddings from the visual features extracted by RoI Pooling (i.e., fixed-size feature maps). These prompt embeddings will be directly used to guide the segmentation operation of the SAM decoder. The structure of Prompt Head can be regarded as a multi-layer perceptron (MLP), which processes the visual features through a series of fully connected layers. Process it and generate the corresponding prompt embedding:

[0106]

[0107] in, Indicates i Tips embedded, ( ) represents the generation of initial cue embedding through visual features, Represents high-frequency information generated based on the initial cue embedding.

[0108] The present invention generates high-frequency information through a sine function and adds it to the prompt embedding to compensate for the network's deficiencies in high-frequency feature learning, thereby improving segmentation accuracy and detail processing capabilities.

[0109] In summary, the present invention innovatively improves the adaptability and automation of the visual large model in water conservancy remote sensing image segmentation by combining the LoRA fine-tuning technology with the small sample prompt learning strategy, and also has the following beneficial effects:

[0110] (1) The present invention uses LoRA technology for lightweight fine-tuning, successfully improving the segmentation performance of the model without significantly increasing the complexity of the model.

[0111] (2) The present invention constructs a reference sample library to enhance generalization ability. The establishment of the reference sample library ensures that the model can effectively utilize diverse sample features during the reasoning process, thereby better coping with invisible or small-scale represented data.

[0112] (3) The present invention effectively generates a hint embedding that is compatible with the SAM decoder through the target similarity feature map and the RPN network (RPN layer). The hint generation framework guided by reference samples improves the automation level of the segmentation task, reduces the dependence on manual hints, and improves the segmentation accuracy. This shows that the reference sample learning strategy better captures the characteristics of diverse targets in water conservancy scenarios, thereby achieving more accurate segmentation.

[0113] Figure 5 : is a schematic diagram of the structure of the remote sensing image segmentation device provided by the present invention, comprising:

[0114] A first processing module 510 is used to extract a feature map of an input image using an image encoder;

[0115] The second processing module 520 is used to obtain a similarity feature map between the feature map and each reference sample feature in a pre-created reference sample library; wherein the reference sample feature is a pre-set feature vector of a foreground pixel, and the similarity feature map is used to characterize the similarity between the feature vector of the input image at each pixel position and the reference sample feature;

[0116] A third processing module 530 is used to select a target similarity feature map with the largest similarity peak from all similarity feature maps, and input the target similarity feature map into a prompt embedding generation network to generate a prompt embedding; the prompt embedding generation network is a pre-built convolutional neural network;

[0117] The fourth processing module 540 is used to input the hint embedding and the input image into an image decoder, and output a segmentation result.

[0118] It should be noted that the remote sensing image segmentation device provided in the embodiment of the present invention can execute the remote sensing image segmentation method described in any of the above embodiments during specific operation, which will not be described in detail in this embodiment.

[0119] Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the remote sensing image segmentation method, which includes: extracting a feature map of an input image using an image encoder; obtaining a similar feature map between the feature map and each reference sample feature in a pre-created reference sample library; wherein the reference sample feature is a feature vector of a pre-set foreground pixel, and the similarity feature map is used to characterize the similarity between the feature vector of the input image at each pixel position and the reference sample feature; selecting a target similarity feature map with the largest similarity peak from all similarity feature maps, and inputting the target similarity feature map into a prompt embedding generation network to generate a prompt embedding; the prompt embedding generation network is a pre-constructed convolutional neural network;

[0120] The hint embedding and the input image are input to an image decoder, and a segmentation result is output.

[0121] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0122] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the remote sensing image segmentation method provided in the above embodiments.

[0123] On the other hand, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the remote sensing image segmentation method provided in the above embodiments.

[0124] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A remote sensing image segmentation method, characterized in that: include: Extracting a feature map of an input image using an image encoder; Obtaining a similarity feature map between the feature map and each reference sample feature in a pre-created reference sample library; wherein the reference sample feature is a feature vector of a pre-set foreground pixel of a sample image, and the similarity feature map is used to characterize the similarity between the feature vector of the input image at each pixel position and the reference sample feature; Selecting a target similarity feature map with the largest similarity peak from all similarity feature maps, and inputting the target similarity feature map into a prompt embedding generation network to generate a prompt embedding; the prompt embedding generation network is a pre-built convolutional neural network; Inputting the hint embedding and the input image into an image decoder, and outputting a segmentation result; The steps of building a reference sample library include: According to the feature map of each sample image extracted by the image encoder, combined with the corresponding true mask, a target feature map of each sample image is obtained; Perform cluster analysis on the feature vector of each foreground pixel in all target feature maps; The feature vectors representing the respective cluster categories are used as reference sample features to construct the reference sample library.

2. The remote sensing image segmentation method according to claim 1, characterized in that: The prompt embedding generation network includes: an RPN layer, a RoI Pooling layer, and a Prompt Head layer; The RPN layer is used to generate a candidate box from the target similarity feature map to mark the target area of ​​interest in the target similarity feature map; The RoI Pooling layer is used to extract a fixed-size feature map from each target region; The Prompt Head layer is used to generate a prompt embedding for input image segmentation based on the feature map.

3. The remote sensing image segmentation method according to claim 1, characterized in that: Perform cluster analysis on the feature vector of each foreground pixel in all target feature maps, including: Calculate the cosine similarity of each pair of feature vectors in all feature vectors; Determine other feature vectors whose cosine similarity with the current feature vector is greater than or equal to the similarity threshold; When the number of other feature vectors is greater than or equal to a preset number, the current feature vector and the other feature vectors are merged into the same cluster to perform feature vector clustering.

4. The remote sensing image segmentation method according to claim 1, characterized in that: The feature vector representing each cluster category is the feature vector closest to the cluster center of the respective cluster category.

5. The remote sensing image segmentation method according to claim 1, characterized in that: The image encoder is a SAM encoder that is fine-tuned based on a low-rank adaptive technology; and the image decoder is a SAM decoder.

6. The remote sensing image segmentation method according to claim 1, characterized in that: The similarity between the feature vector at each pixel position and the reference sample feature is the cosine similarity.

7. A remote sensing image segmentation device, characterized in that: include: A first processing module, configured to extract a feature map of an input image using an image encoder; The second processing module is used to obtain a similarity feature map between the feature map and each reference sample feature in a pre-created reference sample library; wherein the reference sample feature is a pre-set feature vector of a foreground pixel, and the similarity feature map is used to characterize the similarity between the feature vector of the input image at each pixel position and the reference sample feature; A third processing module is used to select a target similarity feature map with the largest similarity peak from all similarity feature maps, and input the target similarity feature map into a prompt embedding generation network to generate a prompt embedding; the prompt embedding generation network is a pre-built convolutional neural network; a fourth processing module, configured to input the hint embedding and the input image into an image decoder, and output a segmentation result; The steps of building a reference sample library include: According to the feature map of each sample image extracted by the image encoder, combined with the corresponding true mask, a target feature map of each sample image is obtained; Perform cluster analysis on the feature vector of each foreground pixel in all target feature maps; The feature vectors representing the respective cluster categories are used as reference sample features to construct the reference sample library.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the remote sensing image segmentation method according to any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the remote sensing image segmentation method according to any one of claims 1 to 6 are implemented.

Images