A method and system for extracting farmland boundaries based on visual-linguistic alignment

Abstract

This invention relates to a method and system for extracting farmland boundaries based on visual-language alignment. The method includes acquiring and preprocessing remote sensing images to obtain remote sensing image patches, which are then divided into training and testing sets. Natural language text annotation is performed on the preprocessed image patch set to generate scene description text. A hierarchical prompting system is then constructed, and a remote sensing image-text pair dataset is generated. A remote sensing-adaptive visual-language alignment network is constructed and pre-trained using contrastive learning. A farmland boundary extraction network and its overall loss function are constructed and trained. The acquired remote sensing image data and natural language prompting text are input into the trained farmland boundary extraction network to obtain the farmland boundary extraction results. This invention features high computational efficiency and fast inference speed, making it suitable for batch processing of large-scale remote sensing data. It achieves adaptive adjustment of prompts, supports zero-shot and few-shot learning, and improves the continuity and integrity of farmland boundaries.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a method and system for extracting farmland boundaries based on visual-linguistic alignment. Background Technology

[0002] Traditional methods for extracting farmland information from remote sensing images rely primarily on manual visual interpretation and category identification, supplemented by field surveys. While offering high accuracy, these methods suffer from long implementation cycles, high labor costs, and slow information updates, failing to meet the demands of precise monitoring. In recent years, deep learning-based semantic segmentation methods have achieved pixel-by-pixel classification through end-to-end learning, demonstrating higher classification accuracy and stronger feature integrity compared to traditional machine learning algorithms. However, these segmentation models primarily originate from computer vision and are designed for natural images with simple backgrounds and singular targets. Unlike natural images, remote sensing images encompass a wider range of scenes with more complex background information. Furthermore, farmland exhibits high visual similarity to surrounding features, making direct application of segmentation networks to remote sensing images prone to misclassification.

[0003] Existing methods for extracting farmland boundaries have the following shortcomings: First, they rely solely on visual features and cannot understand the semantic meaning of concepts such as "farmland," "paddy field," and "dry land," making it difficult to utilize prior human knowledge and descriptive information about farmland types, resulting in insufficient classification accuracy in complex scenarios. Second, they have weak generalization ability with zero or few samples, requiring extensive retraining with labeled data in new regions or for new crop types, leading to long deployment cycles and high costs. Third, they lack targeted learning mechanisms for farmland boundary areas, resulting in a significant decrease in extraction accuracy in areas with blurred boundaries between adjacent plots.

[0004] Recent studies have attempted to apply visual-language models to farmland segmentation, using segmentation tokens generated by multimodal large language models to drive the segmentation model. However, this approach suffers from the following drawbacks: First, it relies on large language models for inference, resulting in a large number of parameters, high computational resource consumption, and slow inference speed, which is not conducive to large-scale remote sensing data processing. Second, the text descriptions use a fixed template filling method, lacking an adaptive prompt learning mechanism, and cannot dynamically adjust the prompt information based on the visual content of the input image. Third, it does not design specific loss constraints for farmland boundary areas, resulting in limited boundary extraction accuracy. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and system for extracting farmland boundaries based on visual-language alignment, which addresses the shortcomings of the prior art.

[0006] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A method for extracting farmland boundaries based on visual-linguistic alignment, comprising the following steps:

[0007] Acquire remote sensing images, preprocess the remote sensing images to obtain fixed-size three-channel visible light remote sensing image patches, and divide them into training set and test set;

[0008] Natural language text annotation is performed on the entire set of preprocessed image patches. Scene description text is generated based on the scene content and annotation information of each image patch. A hierarchical prompting system is then constructed, and a remote sensing image-text pair dataset is generated. The hierarchical prompting system includes task-level prompts. Category-level prompts and instance-level hints ;

[0009] A remote sensing-adapted visual-language alignment network is constructed. The remote sensing image-text pair dataset is used to perform contrastive learning pre-training on the remote sensing-adapted visual-language alignment network. InfoNCE loss is used as the pre-training loss function to obtain the pre-trained visual-language alignment network.

[0010] Constructing a farmland boundary extraction network and its total loss function Using the total loss function The farmland boundary extraction network is trained to obtain the trained farmland boundary extraction network;

[0011] The collected remote sensing image data and natural language prompts are input into the trained farmland boundary extraction network to obtain the farmland boundary extraction results.

[0012] The beneficial effects of this invention are as follows: By constructing a remote sensing-adaptive visual-language alignment network, this invention establishes a pixel-level correspondence between remote sensing images and natural language, enabling the model to possess semantic understanding capabilities. Visual-language fusion can be achieved without a large language model, resulting in high computational efficiency and fast inference speed, making it suitable for batch processing of large-scale remote sensing data. Through a three-layer hierarchical prompting learning architecture, task knowledge, category knowledge, and instance features are encoded into a unified prompting system. Instance-level prompts are dynamically generated by the model based on the visual content of the input image, achieving adaptive adjustment of prompts and supporting zero-shot and few-shot learning. Furthermore, the boundary-weighted loss function enhances the model's learning of boundary regions, improving the continuity and integrity of farmland boundaries.

[0013] Based on the above technical solution, the present invention can be further improved as follows:

[0014] Further: The preprocessing of the remote sensing image to obtain a fixed-size three-channel visible light remote sensing image patch specifically includes the following steps:

[0015] Atmospheric correction and georegistration are performed on the remote sensing images to obtain accurate spatial location information;

[0016] The processed remote sensing image is cropped to obtain a set of image patches;

[0017] The image blocks in the image block set are labeled pixel by pixel to obtain a set of labeled masks;

[0018] The image blocks in the image block set and the corresponding labeled image blocks in the labeled mask set are synchronously divided according to a preset ratio to obtain the training set and the test set.

[0019] Further: The construction of the remote sensing-adapted visual-language alignment network, and the pre-training of the remote sensing image-text pair dataset for contrastive learning of the remote sensing-adapted visual-language alignment network, specifically includes the following steps:

[0020] Construct a remote sensing-adapted vision-language alignment network, the remote sensing-adapted vision-language alignment network including a remote sensing-adapted visual encoder. and text encoder ;

[0021] Based on the Vision Transformer architecture of the contrastive learning vision-language pre-trained model, remote sensing image patches from the aforementioned remote sensing image-text pair dataset are input into the remote sensing adaptation visual encoder. Visual features are obtained, and the scene description text in the remote sensing image-text pair dataset is input into the text encoder. Text features are obtained;

[0022] Based on the aforementioned visual and textual features, and using InfoNCE loss as the pre-training loss function, the remote sensing adaptive visual encoder is trained. and text encoder Pre-training is performed to determine the parameters of the pre-trained visual-language alignment network.

[0023] Furthermore, the constructed farmland boundary extraction network includes a hierarchical prompting learning module, a cross-modal attention fusion module, and a boundary-aware segmentation module.

[0024] The remote sensing adapted visual encoder Dense feature maps with different spatial resolutions are output at different network depths and then fused using a Feature Pyramid Network (FPN) to obtain multi-scale fused features. ;

[0025] The hierarchical prompting learning module is based on the remote sensing adaptive visual encoder. and text encoder The hierarchical prompting system is used for hierarchical prompting learning to obtain hierarchical prompting features. ;

[0026] The cross-modal attention fusion module includes a pixel-level visual-language similarity calculation layer and a segmentation decoder. The pixel-level visual-language similarity calculation layer fuses features at multiple scales. Flattened into a sequence form as query Q, with hierarchical hint features. Given keys K and values ​​V, cross-modal attention is computed using a multi-head attention mechanism, and the attention-weighted features are then fused with the original multi-scale features. Residual connections and LayerNorm normalization are performed to obtain language-enhanced visual feature maps. ;

[0027] The boundary-aware segmentation module processes the language-enhanced visual feature map. The feature vector of each pixel position (h, w) Perform L2 normalization on the text feature vector of each cultivated land category c. Calculate dot product similarity after L2 normalization and the dot product similarity The classification score corresponding to this pixel location is calculated using the following formula:

[0028]

[0029] in, Temperature coefficient;

[0030] The segmentation decoder upsamples the dense feature map, so that the resolution of the dense feature map is gradually restored to the resolution of the input remote sensing image.

[0031] Further: The hierarchical prompting system is used for hierarchical prompting learning to obtain hierarchical prompting features. Specifically, the steps include the following:

[0032] Task-level prompts in the hierarchical prompting system via text encoder Encode the basic text features to obtain them, and then combine these basic text features with a learnable task cue vector that is randomly initialized and optimized through backpropagation during training. By concatenating the features, we obtain the task-level cue features. ;

[0033] The descriptive text for each farmland category k is processed by a text encoder. After encoding, the farmland text features are obtained, and these farmland text features are then matched with the randomly initialized, learnable cue vectors corresponding to the category, which are optimized through backpropagation during training. By concatenating the features, we obtain category-level cue features. ;

[0034] Remote sensing adapted to visual encoder Output global feature vector As input, instance-level cue vectors are dynamically generated using a multilayer perceptron (MLP). Among them, global feature vectors Visual encoders adapted for remote sensing The multi-scale feature extraction backbone network outputs all feature sequences, which are then subjected to global average pooling along the sequence dimension.

[0035] The task-level prompt feature Category-level prompt features and instance-level hint vectors By concatenating the sequences, we obtain the complete hierarchical cue features. .

[0036] Further: The total loss function of the constructed farmland boundary extraction network include:

[0037] Calculate the segmentation loss using cross-entropy loss (CE) and Dice loss. The calculation formula is as follows:

[0038]

[0039] in, To segment the prediction results, For the pixel-level annotation mask in the annotation mask set;

[0040] Calculate the average visual features of the foreground pixels for each farmland category c. Corresponding text features The contrast loss between them serves as the pixel-to-text alignment loss. The calculation formula is as follows:

[0041]

[0042] in, Average visual features of foreground pixels Corresponding text features Dot product similarity between them Temperature coefficient;

[0043] Apply a weight α to pixels in the boundary regions of the segmentation result and a weight 1 to pixels in the non-boundary regions, and calculate the boundary-weighted loss. The calculation formula is:

[0044]

[0045] Where w(p) = α when pixel p belongs to the boundary region, otherwise w(p) = 1;

[0046] According to the segmentation loss Pixel-text alignment loss and boundary weighted loss Construct the total loss function :

[0047]

[0048] in, , , These are the hyperparameter weights.

[0049] Further: the use of the total loss function Training the farmland boundary extraction network to obtain the trained farmland boundary extraction network specifically includes the following steps:

[0050] Learning phase prompts:

[0051] Freeze the visual encoder of the pre-trained vision-language alignment network and text encoder All parameters are used to train only the task cue vectors in the hierarchical cue learning module. Learnable hint vectors In addition, the parameters of the multilayer perceptron (MLP), the parameters of the cross-modal attention fusion module, and the parameters of the segmentation decoder in the boundary-aware segmentation module;

[0052] End-to-end fine-tuning stage:

[0053] Initialize the network parameters after the prompting learning phase is completed, and unfreeze the visual encoder of the pre-trained visual-language alignment network. and text encoder All parameters of the pre-trained visual-language alignment network are trained using a hierarchical learning rate. and text encoder learning rate Learning rates for the sub-hierarchical cueing learning module, the cross-modal attention fusion module, and the boundary-aware segmentation module .

[0054] Further: The step of inputting the collected remote sensing image data and natural language prompt text into the trained farmland boundary extraction network to obtain the farmland boundary extraction result specifically includes the following steps:

[0055] The collected remote sensing image data is preprocessed to obtain standardized remote sensing image blocks of fixed size;

[0056] Natural language prompt text is constructed based on the extraction task, and the standardized remote sensing image patch and natural language prompt text are input into the trained farmland boundary extraction network.

[0057] The system sequentially extracts multi-scale visual features through a visual encoder, generates hierarchical cueing features through a hierarchical cueing learning module, outputs language-enhanced visual features through a cross-modal attention fusion module, and outputs pixel-level segmentation results through a boundary-aware segmentation module, thus obtaining the farmland boundary extraction result.

[0058] This invention also provides a system for extracting farmland boundaries based on visual-language alignment, comprising the following modules:

[0059] The preprocessing module is used to acquire remote sensing images, preprocess the remote sensing images to obtain fixed-size three-channel visible light remote sensing image blocks, and divide them into training set and test set.

[0060] The annotation construction module is used to perform natural language text annotation on the entire set of preprocessed image patches. It generates scene description text based on the scene content and annotation information of each image patch, then constructs a hierarchical prompting system and generates a remote sensing image-text pair dataset. The hierarchical prompting system includes task-level prompts. Category-level prompts and instance-level hints ;

[0061] The remote sensing adaptation visual-language alignment module is used to construct the remote sensing adaptation visual-language alignment network. The remote sensing adaptation visual-language alignment network is pre-trained by contrastive learning using the remote sensing image-text pair dataset. The InfoNCE loss is used as the pre-training loss function to obtain the pre-trained visual-language alignment network.

[0062] The farmland boundary extraction module is used to construct the farmland boundary extraction network and its total loss function. Using the total loss function The farmland boundary extraction network is trained to obtain the trained farmland boundary extraction network;

[0063] The farmland boundary extraction module is also used to input the collected remote sensing image data and natural language prompt text into the trained farmland boundary extraction network to obtain farmland boundary extraction results.

[0064] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned method for extracting farmland boundaries based on vision-language alignment.

[0065] The present invention also provides a farmland boundary extraction device based on vision-language alignment, including a communication interface, a memory, a communication bus and a processor, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0066] The memory is used to store computer programs;

[0067] When the processor executes the program stored in the memory, it implements the steps of the method for extracting farmland boundaries based on vision-language alignment. Attached Figure Description

[0068] Figure 1 This is a flowchart illustrating a method for extracting farmland boundaries based on visual-language alignment according to an embodiment of the present invention.

[0069] Figure 2 This is a schematic diagram of a farmland boundary extraction system based on visual-language alignment according to an embodiment of the present invention. Detailed Implementation

[0070] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0071] like Figure 1 As shown, a method for extracting farmland boundaries based on vision-language alignment includes the following steps:

[0072] S1: Acquire remote sensing images, preprocess the remote sensing images to obtain fixed-size three-channel visible light remote sensing image blocks, and divide them into training set and test set;

[0073] Here, the remote sensing imagery is a three-channel visible light imagery, including satellite remote sensing imagery and / or UAV aerial imagery. Satellite remote sensing imagery includes, but is not limited to, Gaofen series satellite imagery and domestic commercial satellite imagery. UAV aerial imagery includes, but is not limited to, visible light imagery, and the remote sensing imagery data is a three-channel visible light imagery.

[0074] S2: Perform natural language text annotation on the entire set of preprocessed image patches. Generate scene description text based on the scene content and annotation information of each image patch. Then, construct a hierarchical prompting system and generate a remote sensing image-text pair dataset. The hierarchical prompting system includes task-level prompts. Category-level prompts and instance-level hints ;

[0075] S3: Construct a remote sensing-adapted visual-language alignment network. Use the remote sensing image-text pair dataset to perform contrastive learning pre-training on the remote sensing-adapted visual-language alignment network. Use InfoNCE loss as the pre-training loss function to obtain the pre-trained visual-language alignment network.

[0076] S4: Construct the farmland boundary extraction network and its total loss function Using the total loss function The farmland boundary extraction network is trained to obtain the trained farmland boundary extraction network;

[0077] S5: Input the collected remote sensing image data and natural language prompt text into the trained farmland boundary extraction network to obtain the farmland boundary extraction results.

[0078] In one or more embodiments of the present invention, step S1, which involves preprocessing the remote sensing image to obtain a fixed-size three-channel visible light remote sensing image block, specifically includes the following steps:

[0079] S11: Perform atmospheric correction and georegistration processing on the remote sensing image to obtain accurate spatial location information;

[0080] S12: Cropping the processed remote sensing image to obtain a set of image patches;

[0081] Here, atmospheric correction and georegistration are first performed on the remote sensing image to obtain accurate spatial location information. Then, the pixel values ​​of the remote sensing image after atmospheric correction and georegistration are normalized, and the pixel values ​​of the three channels are normalized to the range of [0, 1] and standardized. Then, the image is cropped into a set of 256×256 pixel image blocks D to obtain a standardized image tensor with dimensions of [3, H, W], where 3 is the number of visible light band channels, and H and W are the spatial height and width.

[0082] S13: Perform pixel-by-pixel annotation on the image blocks in the image block set to obtain an annotation mask set. The annotation content includes pixel value annotation masks and farmland type labels.

[0083] S14: Simultaneously divide the image blocks in the image block set and the corresponding labeled image blocks in the labeled mask set according to a preset ratio to obtain the training set and the test set.

[0084] The image patch set D and its corresponding labeled mask set M are synchronously divided in an 8:2 ratio to obtain the training set and the test set, ensuring that each training / test sample contains an image patch and its corresponding labeled mask.

[0085] In one or more embodiments of the present invention, in step S2, natural language text annotation is performed on the preprocessed set D of all image patches to construct a hierarchical prompting system, which includes task-level prompts. Category-level prompts and instance-level hints ;

[0086] Task-level prompts This provides a general description for defining the task of farmland boundary extraction, applicable to all image patches in the dataset, and can remain unchanged or be finely tuned in different application scenarios. An example is "Identifying and segmenting the boundaries of agricultural farmland patches from remote sensing imagery".

[0087] Category-level hints Based on the farmland types appearing in the dataset, text describing the semantic features of each category is written. Examples include: paddy field type: "Rice-growing areas with clear field ridges, green during the growing season and yellowish-brown after harvest"; dryland type: "Farmland planted with dryland crops such as wheat and corn, with textures varying according to crop type and growth stage"; orchard type: "Agricultural land planted with fruit trees, with visible regularly arranged tree textures." Category-level prompts are constructed based on the farmland type annotations in the dataset and can be expanded by users according to specific application needs.

[0088] Instance-level hints It is dynamically generated by the model during the inference phase based on the visual features of the input image, without the need for manual annotation. The generation process will be explained in the subsequent hierarchical prompting learning module.

[0089] Simultaneously, based on the scene content and annotation information of each image patch, corresponding scene description text is generated and paired with the image patch to form a remote sensing image-text pair dataset, which is used for pre-training of contrastive learning in subsequent steps.

[0090] In one or more embodiments of the present invention, step S3, which involves constructing a remote sensing-adapted visual-language alignment network and performing contrastive learning pre-training on the remote sensing image-text pair dataset, specifically includes the following steps:

[0091] S31: Construct a remote sensing adapted vision-language alignment network, wherein the remote sensing adapted vision-language alignment network includes a remote sensing adapted vision encoder. and text encoder ;

[0092] Here, the remote sensing adaptation visual encoder... It includes: an image adaptation layer for patching and linearly projecting the input three-channel visible light remote sensing image, wherein the weights of the patch embedding layer for the three channels are initialized by the pre-trained weights of the vision-language pre-trained model; and a multi-scale feature extraction backbone network for outputting at least two layers of dense feature maps with different spatial resolutions at different network depths. , ,..., The features are then fused using a Feature Pyramid Network (FPN) to obtain multi-scale fused features. Simultaneously, global average pooling is performed on the complete feature sequence output by the backbone network in the spatial dimension to obtain the global feature vector. .

[0093] S32: Based on the Vision Transformer architecture of the contrastive learning vision-language pre-trained model, the remote sensing image patches in the aforementioned remote sensing image-text pair dataset are input into the remote sensing adaptation visual encoder. Visual features are obtained, and the scene description text in the remote sensing image-text pair dataset is input into the text encoder. Text features are obtained;

[0094] S33: Based on the aforementioned visual and textual features, using InfoNCE loss as the pre-trained loss function, the remote sensing adaptive visual encoder is... and text encoder Pre-training is performed to determine the parameters of the pre-trained visual-language alignment network.

[0095] In one or more embodiments of the present invention, step S4, the construction of the farmland boundary extraction network includes a hierarchical cue learning module, a cross-modal attention fusion module, and a boundary-aware segmentation module:

[0096] S41: The remote sensing adaptation visual encoder Dense feature maps with different spatial resolutions are output at different network depths and then fused using a Feature Pyramid Network (FPN) to obtain multi-scale fused features. ;

[0097] S42: The hierarchical prompting learning module is based on a remote sensing adaptive visual encoder. and text encoder The hierarchical prompting system is used for hierarchical prompting learning to obtain hierarchical prompting features. ;

[0098] S43: The cross-modal attention fusion module fuses features at multiple scales. Flattened into a sequence form as query Q, with hierarchical hint features. Given keys K and values ​​V, cross-modal attention is computed using a multi-head attention mechanism, and the attention-weighted features are then fused with the original multi-scale features. Residual connections and LayerNorm normalization are performed to obtain language-enhanced visual feature maps. ;

[0099] In this embodiment, the cross-modal attention fusion module includes multiple layers of cross-modal attention blocks; in this embodiment, it is set to four layers. The calculation process for each layer is as follows: multi-scale feature fusion. Flattened into a sequence form as query Q, with hierarchical hint features. Given keys K and values ​​V, cross-modal attention is calculated using a multi-head attention mechanism, with the following formula:

[0100]

[0101] Here, the number of heads to focus on is 8;

[0102] Then, the attention-weighted features are combined with the original multi-scale fused features. Residual connections and LayerNorm normalization are performed to obtain language-enhanced visual feature maps. After passing through multiple layers of cross-modal attention, the output is a language-enhanced visual feature map. The dimensions are the same as the input visual features.

[0103] Through cross-modal attention fusion, visual features are enhanced under the guidance of language cues, resulting in higher activation responses in visual regions semantically related to the type of farmland described in the cues, thereby achieving conditional guidance of vision by language.

[0104] S44: The boundary-aware segmentation module includes a pixel-level visual-language similarity calculation layer and a segmentation decoder. The pixel-level visual-language similarity calculation layer calculates the visual feature map of the language enhancement. The feature vector of each pixel position (h, w) Perform L2 normalization on the text feature vector of each cultivated land category c. Calculate dot product similarity after L2 normalization and the dot product similarity The classification score corresponding to this pixel location is calculated using the following formula:

[0105]

[0106] in, Here, the temperature coefficient is... Take 0.07;

[0107] The segmentation decoder upsamples the dense feature map, so that the resolution of the dense feature map is gradually restored to the resolution of the input remote sensing image.

[0108] This calculation method enables the model to naturally support zero-shot classification, and only requires a textual description of the new cultivated land category to achieve the identification of unseen categories.

[0109] Meanwhile, the segmentation decoder in the boundary-aware segmentation module gradually restores the resolution of the feature map (i.e., the multi-channel two-dimensional spatial matrix output after feature extraction from the input image by the intermediate layers of the neural network) to the resolution of the input image. In this embodiment, the segmentation decoder adopts a four-layer upsampling structure, with each layer performing a 2x upsampling. The last layer uses a 1×1 convolution to output the classification score. For the binary classification task of cultivated land / non-cultivated land, the number of output channels is 2; for multi-class tasks, the number of output channels is equal to the number of classes plus 1.

[0110] In one or more embodiments of the present invention, the hierarchical prompting system is used to perform hierarchical prompting learning to obtain hierarchical prompting features. Specifically, the steps include the following:

[0111] S411: Add task-level prompts from the hierarchical prompting system. via text encoder Encode the basic text features to obtain them, and then combine these basic text features with a learnable task cue vector that is randomly initialized and optimized through backpropagation during training. By concatenating the features, we obtain the task-level cue features. ;

[0112] Here, the learnable task cue vectors It is randomly initialized with a length of 16. It is automatically optimized through backpropagation during training and the length can be adjusted according to the specific implementation scenario.

[0113] S412: Transform the descriptive text of each cultivated land category k into a text encoder. After encoding, the farmland text features are obtained, and these farmland text features are then matched with the randomly initialized, learnable cue vectors corresponding to the category, which are optimized through backpropagation during training. By concatenating the features, we obtain category-level cue features. ;

[0114] When identifying K types of arable land, there are K sets of category-level cue features. In this embodiment, the learnable vector length is 8, which can be adjusted according to the specific implementation scenario.

[0115] S413: Remote sensing-adapted visual encoder Output global feature vector As input, instance-level cue vectors are dynamically generated using a multilayer perceptron (MLP). Among them, global feature vectors Visual encoders adapted for remote sensing The multi-scale feature extraction backbone network outputs all feature sequences, which are then subjected to global average pooling along the sequence dimension.

[0116] Instance-level hint vectors It dynamically changes according to different input images, adaptively encoding the visual feature information of land parcels in the current image without manual setting. In this embodiment, the hidden layer dimension of the fully connected network is 512, and the length of the output cue vector is 16, which can be adjusted according to the specific implementation scenario.

[0117] S414: The task-level prompt feature is... Category-level prompt features and instance-level hint vectors By concatenating the sequences, we obtain the complete hierarchical cue features. Dimension [B, [, D], where B is the batch size. To indicate the total length of the sequence, D represents the feature dimension.

[0118] In one or more embodiments of the present invention, the total loss function of the farmland boundary extraction network is constructed. include:

[0119] S45: Calculate the segmentation loss using cross-entropy loss (CE) and Dice loss. This is used to measure the overall deviation between the segmentation prediction result and the corresponding pixel-level labeled mask. The calculation formula is as follows:

[0120]

[0121] in, To segment the prediction results, For the pixel-level annotation mask in the annotation mask set;

[0122] S46: Calculate the average visual feature of the foreground pixels for each farmland category c. Corresponding text features The contrast loss between them serves as the pixel-to-text alignment loss. The calculation formula is as follows:

[0123]

[0124] in, Average visual features of foreground pixels Corresponding text features Dot product similarity between them The temperature coefficient is used; this loss ensures that the visual features of pixels of the same category are aligned with the text features of the corresponding category in the embedding space.

[0125] S47: Apply a weight α to the pixels in the boundary region of the segmentation result and a weight 1 to the pixels in the non-boundary region, and calculate the boundary weighted loss. The calculation formula is:

[0126]

[0127] Where w(p) is the weight of pixel p, w(p) = α when the pixel belongs to the boundary region, and w(p) = 1 otherwise. To predict the probability, the boundary region is extracted by performing morphological dilation and erosion operations on the corresponding pixel-level annotation masks in the annotation mask set. In this embodiment, α is taken as an empirical value of 5, which can be adjusted according to the specific implementation scenario.

[0128] S48: Based on segmentation loss Pixel-text alignment loss and boundary weighted loss Construct the total loss function :

[0129]

[0130] in, , , These are the hyperparameter weights, as shown in this embodiment. Take 1.0, Take 0.2, A value of 0.5 can be obtained adaptively through grid search or empirical values.

[0131] In one or more embodiments of the present invention, the use of the total loss function Training the farmland boundary extraction network to obtain the trained farmland boundary extraction network specifically includes the following steps:

[0132] Learning phase prompts:

[0133] Freeze the visual encoder of the pre-trained vision-language alignment network and text encoder All parameters are used to train only the task cue vectors in the hierarchical cue learning module. Learnable hint vectors In addition, the parameters of the multilayer perceptron (MLP), the parameters of the cross-modal attention fusion module, and the parameters of the segmentation decoder in the boundary-aware segmentation module;

[0134] Here, the learning rate is 5e-4, and the training lasts for 30 epochs.

[0135] End-to-end fine-tuning stage:

[0136] Initialize the network parameters after the prompting learning phase is completed, and unfreeze the visual encoder of the pre-trained visual-language alignment network. and text encoder All parameters of the pre-trained visual-language alignment network are trained using a hierarchical learning rate. and text encoder learning rate Learning rates for the sub-hierarchical cueing learning module, the cross-modal attention fusion module, and the boundary-aware segmentation module Here, the learning rate The learning rate is 1e-5. The training time is 1e-4, with 20 epochs. The optimizer used is AdamW, with a weight decay of 0.01, and the learning rate strategy is linear warm-up plus cosine decay. The batch size is 16. The above training parameters are empirical values ​​for this embodiment and can be adjusted according to specific implementation scenarios.

[0137] In one or more embodiments of the present invention, the step of inputting the collected remote sensing image data and natural language prompt text into the trained farmland boundary extraction network to obtain the farmland boundary extraction result specifically includes the following steps:

[0138] S51: Preprocess the acquired remote sensing image data to obtain standardized remote sensing image blocks of fixed size;

[0139] S52: Construct natural language prompt text based on the extraction task, and input the standardized remote sensing image patch and natural language prompt text into the trained farmland boundary extraction network;

[0140] S53: The visual encoder extracts multi-scale visual features, the hierarchical cueing learning module generates hierarchical cueing features, the cross-modal attention fusion module outputs language-enhanced visual features, and the boundary-aware segmentation module outputs pixel-level segmentation results to obtain the farmland boundary extraction results.

[0141] In practice, after the segmentation is completed, the segmentation results need to be post-processed, including: small patch removal, hole filling, boundary smoothing and vectorization, to obtain the final farmland boundary vector data.

[0142] like Figure 2 As shown, the present invention also provides a farmland boundary extraction system based on vision-language alignment, comprising the following modules:

[0143] The preprocessing module is used to acquire remote sensing images, preprocess the remote sensing images to obtain fixed-size three-channel visible light remote sensing image blocks, and divide them into training set and test set.

[0144] The annotation construction module is used to perform natural language text annotation on the entire set of preprocessed image patches. It generates scene description text based on the scene content and annotation information of each image patch, then constructs a hierarchical prompting system and generates a remote sensing image-text pair dataset. The hierarchical prompting system includes task-level prompts. Category-level prompts and instance-level hints ;

[0145] The remote sensing adaptation visual-language alignment module is used to construct the remote sensing adaptation visual-language alignment network. The remote sensing adaptation visual-language alignment network is pre-trained by contrastive learning using the remote sensing image-text pair dataset. The InfoNCE loss is used as the pre-training loss function to obtain the pre-trained visual-language alignment network.

[0146] The farmland boundary extraction module is used to construct the farmland boundary extraction network and its total loss function. Using the total loss function The farmland boundary extraction network is trained to obtain the trained farmland boundary extraction network;

[0147] The farmland boundary extraction module is also used to input the collected remote sensing image data and natural language prompt text into the trained farmland boundary extraction network to obtain farmland boundary extraction results.

[0148] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned method for extracting farmland boundaries based on vision-language alignment.

[0149] The present invention also provides a farmland boundary extraction device based on vision-language alignment, including a communication interface, a memory, a communication bus and a processor, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0150] The memory is used to store computer programs;

[0151] When the processor executes the program stored in the memory, it implements the steps of the method for extracting farmland boundaries based on vision-language alignment.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for extracting farmland boundaries based on visual-linguistic alignment, characterized in that, Includes the following steps: Acquire remote sensing images, preprocess the remote sensing images to obtain fixed-size three-channel visible light remote sensing image patches, and divide them into training set and test set; Natural language text annotation is performed on the entire set of preprocessed image patches. Scene description text is generated based on the scene content and annotation information of each image patch. A hierarchical prompting system is then constructed, and a remote sensing image-text pair dataset is generated. The hierarchical prompting system includes task-level prompts. Category-level prompts and instance-level hints The construction of the hierarchical prompting system includes: adding task-level prompts to the hierarchical prompting system. via text encoder Encode the basic text features to obtain them, and then combine these basic text features with a learnable task cue vector that is randomly initialized and optimized through backpropagation during training. By concatenating the features, we obtain the task-level cue features. The descriptive text for each farmland category k is processed by a text encoder. After encoding, the farmland text features are obtained, and these farmland text features are then matched with the randomly initialized, learnable cue vectors corresponding to the category, which are optimized through backpropagation during training. By concatenating the features, we obtain category-level cue features. ; Using remote sensing to adapt the visual encoder Output global feature vector As input, instance-level cue vectors are dynamically generated using a multilayer perceptron (MLP). ; A remote sensing-adapted visual-language alignment network is constructed. The remote sensing image-text pair dataset is used to perform contrastive learning pre-training on the remote sensing-adapted visual-language alignment network. InfoNCE loss is used as the pre-training loss function to obtain the pre-trained visual-language alignment network. Constructing a farmland boundary extraction network and its total loss function Using the total loss function The farmland boundary extraction network is trained to obtain the trained farmland boundary extraction network; The collected remote sensing image data and natural language prompts are input into the trained farmland boundary extraction network to obtain farmland boundary extraction results.

2. The method for extracting farmland boundaries based on visual-language alignment according to claim 1, characterized in that, The preprocessing of the remote sensing image to obtain a fixed-size three-channel visible light remote sensing image patch specifically includes the following steps: Atmospheric correction and georegistration are performed on the remote sensing images to obtain accurate spatial location information; The processed remote sensing image is cropped to obtain a set of image patches; The image blocks in the image block set are labeled pixel by pixel to obtain a set of labeled masks; The image blocks in the image block set and the corresponding labeled image blocks in the labeled mask set are synchronously divided according to a preset ratio to obtain the training set and the test set.

3. The method for extracting farmland boundaries based on visual-language alignment according to claim 1, characterized in that, The construction of the remote sensing-adapted visual-language alignment network, and the pre-training of the remote sensing image-text pair dataset for contrastive learning of the remote sensing-adapted visual-language alignment network, specifically includes the following steps: Construct a remote sensing-adapted vision-language alignment network, the remote sensing-adapted vision-language alignment network including a remote sensing-adapted visual encoder. and text encoder ; Based on the Vision Transformer architecture of the contrastive learning vision-language pre-trained model, remote sensing image patches from the aforementioned remote sensing image-text pair dataset are input into the remote sensing adaptation visual encoder. Visual features are obtained, and the scene description text in the remote sensing image-text pair dataset is input into the text encoder. Text features are obtained; Based on the aforementioned visual and textual features, and using InfoNCE loss as the pre-training loss function, the remote sensing adaptive visual encoder is trained. and text encoder Perform pre-training to determine the parameters of the pre-trained vision-language alignment network.

4. The method for extracting farmland boundaries based on visual-language alignment according to claim 3, characterized in that, The constructed farmland boundary extraction network includes a hierarchical prompting learning module, a cross-modal attention fusion module, and a boundary-aware segmentation module. The remote sensing adapted visual encoder Dense feature maps with different spatial resolutions are output at different network depths and then fused using a Feature Pyramid Network (FPN) to obtain multi-scale fused features. ; The hierarchical prompting learning module is based on the remote sensing adaptive visual encoder. and text encoder The hierarchical prompting system is used for hierarchical prompting learning to obtain hierarchical prompting features. ; The cross-modal attention fusion module fuses features at multiple scales. Flattened into a sequence form as query Q, with hierarchical hint features. Given keys K and values ​​V, cross-modal attention is computed using a multi-head attention mechanism, and the attention-weighted features are then fused with the original multi-scale features. Residual connections and LayerNorm normalization are performed to obtain language-enhanced visual feature maps. ; The boundary-aware segmentation module includes a pixel-level visual-language similarity calculation layer and a segmentation decoder. The pixel-level visual-language similarity calculation layer calculates the visual feature map of the language enhancement. The feature vector of each pixel position (h, w) Perform L2 normalization on the text feature vector of each cultivated land category c. Calculate dot product similarity after L2 normalization and the dot product similarity The classification score corresponding to this pixel location is calculated using the following formula: ； in, Temperature coefficient; The segmentation decoder upsamples the dense feature map, so that the resolution of the dense feature map is gradually restored to the resolution of the input remote sensing image.

5. The method for extracting farmland boundaries based on visual-language alignment according to claim 4, characterized in that, The hierarchical prompting system is used to perform hierarchical prompting learning to obtain hierarchical prompting features. Specifically, the steps include the following: The task-level prompt feature Category-level prompt features and instance-level hint vectors By concatenating the sequences, we obtain the complete hierarchical cue features. ; Wherein, the global feature vector Visual encoders adapted for remote sensing The multi-scale feature extraction backbone network outputs all feature sequences, which are then subjected to global average pooling along the sequence dimension.

6. The method for extracting farmland boundaries based on visual-language alignment according to claim 4, characterized in that, The total loss function of the farmland boundary extraction network The construction includes: Calculate the segmentation loss using cross-entropy loss (CE) and Dice loss. The calculation formula is as follows: ； in, To segment the prediction results, For the pixel-level annotation mask in the annotation mask set; Calculate the average visual features of the foreground pixels for each farmland category c. Corresponding text features The contrast loss between them serves as the pixel-to-text alignment loss. The calculation formula is as follows: ； in, Average visual features of foreground pixels Corresponding text features Dot product similarity between them Temperature coefficient; Apply a weight α to pixels in the boundary regions of the segmentation result and a weight 1 to pixels in the non-boundary regions, and calculate the boundary-weighted loss. The calculation formula is: ； Where w(p) = α when pixel p belongs to the boundary region, otherwise w(p) = 1; According to the segmentation loss Pixel-text alignment loss and boundary weighted loss Construct the total loss function : ； in, , , These are the hyperparameter weights.

7. The method for extracting farmland boundaries based on visual-language alignment according to claim 6, characterized in that, The use of the total loss function Training the farmland boundary extraction network to obtain the trained farmland boundary extraction network specifically includes the following steps: Learning phase prompts: Freeze the visual encoder of the pre-trained vision-language alignment network and text encoder All parameters are used to train only the task cue vectors in the hierarchical cue learning module. Learnable hint vectors In addition, the parameters of the multilayer perceptron (MLP), the parameters of the cross-modal attention fusion module, and the parameters of the segmentation decoder in the boundary-aware segmentation module; End-to-end fine-tuning stage: Initialize the network parameters after the prompting learning phase is completed, and unfreeze the visual encoder of the pre-trained visual-language alignment network. and text encoder All parameters of the pre-trained visual-language alignment network are trained using a hierarchical learning rate. and text encoder learning rate Learning rates for the sub-hierarchical cueing learning module, the cross-modal attention fusion module, and the boundary-aware segmentation module .

8. A system for extracting farmland boundaries based on vision-language alignment, characterized in that, Includes the following modules: The preprocessing module is used to acquire remote sensing images, preprocess the remote sensing images to obtain fixed-size three-channel visible light remote sensing image blocks, and divide them into training set and test set. The annotation construction module is used to perform natural language text annotation on the entire set of preprocessed image patches. It generates scene description text based on the scene content and annotation information of each image patch, then constructs a hierarchical prompting system and generates a remote sensing image-text pair dataset. The hierarchical prompting system includes task-level prompts. Category-level prompts and instance-level hints The construction of the hierarchical prompting system includes: adding task-level prompts to the hierarchical prompting system. via text encoder Encode the basic text features to obtain them, and then combine these basic text features with a learnable task cue vector that is randomly initialized and optimized through backpropagation during training. By concatenating the features, we obtain the task-level cue features. The descriptive text for each farmland category k is processed by a text encoder. After encoding, the farmland text features are obtained, and these farmland text features are then matched with the randomly initialized, learnable cue vectors corresponding to the category, which are optimized through backpropagation during training. By concatenating the features, we obtain category-level cue features. ; Using remote sensing to adapt the visual encoder Output global feature vector As input, instance-level cue vectors are dynamically generated using a multilayer perceptron (MLP). ; The remote sensing adaptation visual-language alignment module is used to construct the remote sensing adaptation visual-language alignment network. The remote sensing adaptation visual-language alignment network is pre-trained by contrastive learning using the remote sensing image-text pair dataset. The InfoNCE loss is used as the pre-training loss function to obtain the pre-trained visual-language alignment network. The farmland boundary extraction module is used to construct the farmland boundary extraction network and its total loss function. Using the total loss function The farmland boundary extraction network is trained to obtain the trained farmland boundary extraction network; The farmland boundary extraction module is also used to input the collected remote sensing image data and natural language prompt text into the trained farmland boundary extraction network to obtain farmland boundary extraction results.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the method for extracting farmland boundaries based on visual-language alignment as described in any one of claims 1-7.

10. A farmland boundary extraction device based on vision-language alignment, characterized in that: It includes a communication interface, a memory, a communication bus, and a processor, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the steps of the method for extracting farmland boundaries based on visual-language alignment as described in any one of claims 1-7.

Images