Remote sensing semantic segmentation reasoning acceleration method based on fine-grained vision-language model

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a remote sensing semantic segmentation reasoning acceleration method based on a fine-grained vision-language model, which comprises the following steps of: firstly, performing fine-grained image-text contrast learning by constructing an image-level and region-level fine-grained image-text data set; training a double-tower vision-language model with an image encoder and a text encoder; the fine-grained vision-language model weight obtained through pre-training is migrated to a two-stage reasoning acceleration framework to serve as a target filter; calculating a similarity score between the image features of a series of small image blocks in the large-size remote sensing image test sample and the text features of the target category through a fine-grained vision-language model, and setting a threshold value to screen out the small image blocks with the target category; and inputting the screened image blocks into a remote sensing semantic segmentation model to obtain a final segmentation result. According to the method, the sparse target region of the large-size remote sensing image is positioned by using the fine-grained vision-language model, the region where enough and correct sparse targets are located is efficiently screened out, and the time cost of direct reasoning is reduced, so that the reasoning efficiency is improved.

Description

Technical Field

[0001] The invention relates to a large-scale remote sensing image reasoning acceleration method, in particular to a large-scale remote sensing image reasoning acceleration method of a vision-language model, and belongs to the field of computer technology. Background Art

[0002] Remote sensing semantic segmentation is a core task in the field of remote sensing image processing. Its main purpose is to accurately extract and mark the spatial location information of different objects or targets from remote sensing images. This technology plays a vital role in many application fields, including urban planning, environmental monitoring, agricultural management, natural resource management, and natural disaster prediction and response. Through remote sensing semantic segmentation technology, researchers and practitioners can extract valuable information from a wide range of geographical areas to support decision-making.

[0003] Remote sensing images are usually large-scale and high-resolution, which makes existing remote sensing semantic segmentation methods face huge computational complexity and storage requirements when processing these images. Specifically, processing such large-size images requires not only high-performance computing equipment, but also efficient algorithms to reduce the computational burden. In order to meet these challenges, current research usually adopts the strategy of cropping large-scale and high-resolution remote sensing images. By cutting the entire image into smaller areas for processing, the consumption of computing and storage resources can be significantly reduced, and the training and reasoning efficiency of the model can be improved.

[0004] However, remote sensing images usually cover a wide geographical area, which may contain a variety of complex objects and targets, such as lakes, forests, buildings, roads, vehicles, etc. In practical applications, the distribution of these targets is often sparse, that is, only a small part of the entire image contains these targets, and most of the other areas may just be empty backgrounds or irrelevant content. This sparsity brings additional challenges when processing ultra-large-scale high-resolution remote sensing images, especially images with a resolution of 10000×10000.

[0005] Language-image pre-training has made significant progress in the past few years. In particular, contrast-language image pre-training has demonstrated very impressive zero-shot transfer and generalization capabilities. A series of works have used vision-language pre-training models for downstream tasks such as open vocabulary detection and segmentation, video recognition, etc. Therefore, facing the application scenario of sparse targets in large-scale high-resolution remote sensing images, target area screening based on vision-language models can effectively implement more efficient segmentation reasoning strategies and reduce the computational overhead of irrelevant areas. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a remote sensing semantic segmentation reasoning acceleration method based on a vision-language model, which provides a more accurate solution to the problem of slow reasoning speed of large-size remote sensing images.

[0007] The present invention adopts the following technical solutions to solve the above technical problems:

[0008] The remote sensing semantic segmentation reasoning acceleration method based on the vision-language model includes the following steps:

[0009] Step 1: Build a fine-grained remote sensing image text dataset;

[0010] Step 2: Perform fine-grained image-text contrastive learning and train a dual-tower vision-language model with an image encoder and a text encoder.

[0011] Step 3: Migrate the fine-grained visual-language model pre-trained in step 2 to the two-stage reasoning acceleration framework as a target filter;

[0012] Step 4: Crop the test sample image into a series of small image blocks with overlapping parts, input them into the image encoder of the fine-grained visual-language model in sequence, and input the expected target category into its text encoder;

[0013] Step 5: Set the threshold score to filter out small image blocks containing the target category;

[0014] Step 6: Input the image blocks filtered in step 5 into the remote sensing semantic segmentation model to obtain the segmentation result of the test sample image.

[0015] As a preferred embodiment of the present invention, the specific process of step 1 is as follows:

[0016] 1.1 Given N(1≤N≤10 4 ) samples of the dataset Where X i Represents the i-th sample image, and uses a multimodal large language model to generate a i Generate corresponding global image text description Use information retrieval and language generation models to generate corresponding global image labels Where 1≤n≤100; In addition, the sample image X is calculated using the image-text comparison pre-training model i With the corresponding text description Similarity scores are used to filter out high-quality global image text descriptions.

[0017] 1.2 For region-level image text annotation, a high-quality object detection model is used to generate a text annotation for each sample image X.i Generate candidate bounding boxes for local objects and the corresponding class labels Screen off The confidence score of the candidate boxes is lower than σ, and non-maximum suppression (NMS) is applied to merge the remaining boxes. The multimodal large language model is used to select the candidate target boxes after screening. Generate the corresponding local image text description

[0018] As a preferred embodiment of the present invention, the specific process of step 2 is as follows:

[0019] 2.1 Constructing a fine-grained contrastive learning framework. Using an image encoder E based on a convolutional neural network image For each sample image X i Extracting global image features Will and the corresponding local region candidate bounding box Input to the RoIAlign layer to generate local area image features Where 1≤n≤100. Use a transformer-based text encoder E text A global image text description and the global image label Extracting global text features and global label features At the same time, the local image text description and local image labels Extract local text features and local label features

[0020] 2.2 Calculation of global image-level loss L global and the local area level loss L region .L global and L region Both are image-text contrast loss and image-label loss The specific loss function is defined as follows:

[0021]

[0022] Where N is the batch size, sim represents the similarity function, τ is the learnable temperature parameter, and v image Indicated by E image The extracted image features, t caption and t tag Respectively represented by E text Extracted text description features and label features.

[0023] 2.3 Using the sum of global image-level loss and local region-level loss L total =L global +L region Training step 2.1 The image encoder and text encoder in the fine-grained contrastive learning framework are trained until the neural network model converges.

[0024] As a preferred embodiment of the present invention, the specific process described in step 3 is:

[0025] 3.1 Construct a two-stage reasoning acceleration framework, which mainly consists of a target region filter and a segmentation reasoning model. The target region filter is composed of a visual-language model, including an image encoder E based on a convolutional neural network. image and a transformer-based text encoder E text The segmentation inference model can be any remote sensing semantic segmentation model. In the first stage, the test image samples and target category text are input into the target region filter for region screening, which reduces the number of inferences of the second stage segmentation model, thereby achieving inference acceleration of large-scale remote sensing images.

[0026] 3.2 The image encoder E pre-trained in step 2 image and text encoder E text The weights are transferred to the target filter in the two-stage inference acceleration framework for regional target screening.

[0027] As a preferred solution of the present invention, the specific process of step 4 is:

[0028] 4.1 Given N(1≤N≤10 4 ) samples of large-scale remote sensing image dataset Where X i Represents the i-th sample image. Each test sample image X i Cut into K (1≤K≤2 6 ) small image blocks, denoted as Will The image encoder E is input to the visual-language model in step 3 image Get K small image block features, denoted as Input the desired target category text T into the text encoder E of the visual-language model text Extract text features t.

[0029] 4.2 Calculate K small image block features The similarity score with the target category text feature t, denoted as s i ={s1,s2,…,s k}.

[0030] As a preferred solution of the present invention, the specific process of step 5 is:

[0031] Set the similarity score threshold θ to filter out the s in step 4.2 i ={s1,s2,…,s k} is greater than the similarity score of θ, and we get M(1≤M≤2 6 ) small image blocks that need to be segmented and inferred, denoted as

[0032] As a preferred solution of the present invention, the specific process of step 6 is:

[0033] The M small image blocks obtained by step 5 are input into the two-stage segmentation reasoning model to infer the corresponding mask image, which is recorded as Finally, the M masks are merged and restored to the sample image X i Original size, get the mask image Y of the entire large-size remote sensing image i .

[0034] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0035] 1. The present invention proposes a fine-grained contrastive learning framework and constructs image-level and region-level remote sensing image text datasets, so that the visual-language model focuses on fine-grained specific areas or pixel-level details, thereby improving the visual-language model's ability to locate sparse small targets in large-size, high-resolution remote sensing images.

[0036] 2. The present invention proposes a two-stage reasoning acceleration framework, which uses a fine-grained visual-language model to perform refined target area screening for sparse small targets in large-scale high-resolution remote sensing images, thereby reducing the reasoning time cost and improving the accuracy of the reasoning results. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is an algorithm flow chart of the remote sensing semantic segmentation reasoning acceleration method based on the vision-language model of the present invention.

[0038] Figure 2 Schematic diagram of the fine-grained contrastive learning framework of the present invention.

[0039] Figure 3 It is a schematic diagram of a two-stage reasoning acceleration framework of the remote sensing semantic segmentation reasoning acceleration method based on a fine-grained vision-language model of the present invention. DETAILED DESCRIPTION

[0040] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0041] With the development of remote sensing technology, more and more high-resolution remote sensing images are available. Semantic segmentation of remote sensing images is one of the important tasks in the field of remote sensing, and semantic segmentation algorithms based on deep learning have made significant progress. Existing segmentation models generally traverse the entire image to infer the mask of the target category. However, high-resolution optical remote sensing images are usually large in size (for example, 10000×10000), and the time cost of inference is very high when segmenting a single target category.

[0042] Vision-language models benefit from large-scale web crawling data and show impressive performance gains for tasks that require global alignment, such as classification and retrieval. However, they tend to perform relatively poorly when faced with dense tasks that require more complex understanding at the region or pixel level. This discrepancy stems from the fact that mainstream vision-language models emphasize capturing the overall relationship between text and images, rather than delving into fine-grained specific regions or pixel-level details.

[0043] To solve the above problems, the present invention proposes a remote sensing semantic segmentation reasoning acceleration method based on a fine-grained vision-language model. The fine-grained vision-language model is used to locate the target category, and the areas where sufficient and correct sparse targets are located (including the target category) are screened out, thereby reducing the time cost of direct reasoning and improving the reasoning efficiency.

[0044] Combination Figure 1 As shown, the remote sensing semantic segmentation reasoning acceleration method based on the fine-grained visual-language model of the present invention comprises the following steps:

[0045] Step 1: Build a fine-grained remote sensing image text dataset;

[0046] Step 2: Perform fine-grained image-text contrastive learning and train a dual-tower vision-language model with an image encoder and a text encoder.

[0047] Step 3: Migrate the fine-grained visual-language model pre-trained in step 2 to the two-stage reasoning acceleration framework as a target filter;

[0048] Step 4: Crop the test sample image into a series of small image blocks with overlapping parts, input them into the image encoder of the fine-grained visual-language model in sequence, and input the expected target category into its text encoder;

[0049] Step 5: Set the threshold score to filter out small image blocks containing the target category;

[0050] Step 6: Input the image blocks filtered in step 5 into the remote sensing semantic segmentation model to obtain the segmentation result of the test sample image.

[0051] The specific process of step 1 is as follows:

[0052] 1.1 Given N(1≤N≤10 4 ) samples of the dataset Where X i Represents the i-th sample image, and uses a multimodal large language model to generate a i Generate the corresponding global image text description Use information retrieval and language generation models to generate corresponding global image labels Where 1≤n≤100; In addition, the sample image X is calculated using the image-text comparison pre-training model i With the corresponding text description Similarity scores are used to filter out high-quality global image text descriptions.

[0053] 1.2 For region-level image text annotation, a high-quality object detection model is used to generate a text annotation for each sample image X. i Generate candidate bounding boxes for local objects and the corresponding class labels Screen off The confidence score of the candidate boxes is lower than σ, and non-maximum suppression (NMS) is applied to merge the remaining boxes. The multimodal large language model is used to select the candidate target boxes after screening. Generate the corresponding local image text description

[0054] The specific process of step 2 is as follows:

[0055] 2.1 Constructing a fine-grained contrastive learning framework. Using an image encoder E based on a convolutional neural network image For each sample image X i Extracting global image features Will and the corresponding local region candidate bounding box Input to the RoIAlign layer to generate local area image features Where 1≤n≤100. Use a transformer-based text encoder E text A global image text description and the global image label Extracting global text features and global label features At the same time, the local image text description and local image labels Extract local text features and local label features

[0056] 2.2 Calculation of global image-level loss L global and the local area level loss L region .L global and L region Both are image-text contrast loss and image-label loss The specific loss function is defined as follows:

[0057]

[0058]

[0059] Where N is the batch size, sim represents the similarity function, τ is the learnable temperature parameter, and v image Indicated by E image The extracted image features, t caption and t tag Respectively represented by E text Extracted text description features and label features.

[0060] 2.3 Using the sum of global image-level loss and local region-level loss L total =L global +L region Training step 2.1 The image encoder and text encoder in the fine-grained contrastive learning framework are trained until the neural network model converges.

[0061] The specific process of step 3 is as follows:

[0062] 3.1 Construct a two-stage reasoning acceleration framework, which mainly consists of a target region filter and a segmentation reasoning model. The target region filter is composed of a visual-language model, including an image encoder E based on a convolutional neural network. image and a transformer-based text encoder E text The segmentation inference model can be any remote sensing semantic segmentation model. In the first stage, the test image samples and target category text are input into the target region filter for region screening, which reduces the number of inferences of the second stage segmentation model, thereby achieving inference acceleration of large-scale remote sensing images.

[0063] 3.2 The image encoder E pre-trained in step 2 image and text encoder Etext The weights are transferred to the target filter in the two-stage inference acceleration framework for regional target screening.

[0064] The specific process of step 4 is as follows:

[0065] 4.1 Given N(1≤N≤10 4 ) samples of large-scale remote sensing image dataset Where X i Represents the i-th sample image. Each test sample image X i Cut into K (1≤K≤2 6 ) small image blocks, denoted as Will The image encoder E is input to the visual-language model in step 3 image Get K small image block features, denoted as Input the desired target category text T into the text encoder E of the visual-language model text Extract text features y.

[0066] 4.2 Calculate K small image block features The similarity score with the target category text feature t, denoted as s i ={s1,s2,…,s k}.

[0067] The specific process of step 5 is as follows:

[0068] Set the similarity score threshold θ to filter out the s in step 4.2 i ={s1,s2,…,s k} is greater than the similarity score of θ, and we get M(1≤M≤2 6 ) small image blocks that need to be segmented and inferred, denoted as

[0069] The specific process of step 6 is as follows:

[0070] The M small image blocks obtained by step 5 are input into the two-stage segmentation reasoning model to infer the corresponding mask image, which is recorded as Finally, the M masks are merged and restored to the sample image X i Original size, get the mask image Y of the entire large-size remote sensing image i .

[0071] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A remote sensing semantic segmentation reasoning acceleration method based on a fine-grained visual-language model, characterized in that: The steps include: Step 1: Build a fine-grained remote sensing image text dataset; Step 2: Perform fine-grained image-text contrastive learning and train a dual-tower vision-language model with an image encoder and a text encoder. Step 3: Migrate the fine-grained visual-language model pre-trained in step 2 to the two-stage reasoning acceleration framework as a target filter; Step 4: Crop the test sample image into a series of small image blocks with overlapping parts, input them into the image encoder of the fine-grained visual-language model in sequence, and input the expected target category into its text encoder; Step 5: Set the threshold score to filter out small image blocks containing the target category; Step 6: Input the image blocks filtered in step 5 into the remote sensing semantic segmentation model to obtain the segmentation result of the test sample image.

2. The remote sensing semantic segmentation reasoning acceleration method based on fine-grained visual-language model according to claim 1 is characterized in that: The specific process of step 1 is as follows: 1.1 Given a dataset of N samples Where X i Represents the i-th sample image, 1≤N≤10 4 , using a multimodal large language model for each sample image X i Generate corresponding global image text description Use information retrieval and language generation models to generate corresponding global image labels Where 1≤n≤100; In addition, the sample image X is calculated using the image-text comparison pre-training model i With the corresponding text description Similarity scores are used to filter out high-quality global image text descriptions. 1.2 For region-level image text annotation, a high-quality object detection model is used to generate a text annotation for each sample image X. i Generate candidate bounding boxes for local objects and the corresponding class labels Screen off The confidence score of the candidate boxes is lower than σ, and the remaining boxes are merged by non-maximum suppression. The multimodal large language model is used to select the candidate target boxes after screening. Generate the corresponding local image text description 3. The remote sensing semantic segmentation reasoning acceleration method based on fine-grained visual-language model according to claim 1 is characterized in that: The specific process of step 2 is: 2.1 Constructing a fine-grained contrastive learning framework. Using an image encoder E based on a convolutional neural network image For each sample image X i Extracting global image features Will and the corresponding local region candidate bounding box Input to the RoIAlign layer to generate local area image features Where 1≤n≤100. Use a transformer-based text encoder E text A global image text description and the global image label Extracting global text features and global label features At the same time, the local image text description and local image labels Extract local text features and local label features 2.2 Calculation of global image-level loss L global and the local area level loss L region .L global and L region Both are image-text contrast loss and image-label loss The specific loss function is defined as follows: Where N is the batch size, sim represents the similarity function, τ is the learnable temperature parameter, and v image Indicated by E image The extracted image features, t caption and t tag Respectively represented by E text Extracted text description features and label features; 2.3 Using the sum of global image-level loss and local region-level loss L total =L global +L region Training step 2.1 The image encoder and text encoder in the fine-grained contrastive learning framework are trained until the neural network model converges.

4. The remote sensing semantic segmentation reasoning acceleration method based on fine-grained visual-language model according to claim 1 is characterized in that: The specific process of step 3 is as follows: 3.1 Construct a two-stage reasoning acceleration framework, which mainly consists of a target region filter and a segmentation reasoning model. The target region filter is composed of a visual-language model, including an image encoder E based on a convolutional neural network. image and a transformer-based text encoder E text The segmentation inference model can be any remote sensing semantic segmentation model. In the first stage, the test image samples and target category text are input into the target region filter for region screening, which reduces the number of inferences of the second stage segmentation model, thereby achieving inference acceleration of large-scale remote sensing images. 3.2 The image encoder E pre-trained in step 2 image and text encoder E text The weights are transferred to the target filter in the two-stage inference acceleration framework for regional target screening.

5. The remote sensing semantic segmentation reasoning acceleration method based on fine-grained visual-language model according to claim 1 is characterized in that: The specific process of step 4 is as follows: 4.1 Given N(1≤N≤10 4 ) samples of large-scale remote sensing image dataset Where X i Represents the i-th sample image. Each test sample image X i Cut into K (1≤K≤2 6 ) small image blocks, denoted as Will The image encoder E is input to the visual-language model in step 3 image Get K small image block features, denoted as Input the desired target category text T into the text encoder E of the visual-language model text Extract text features t; 4.2 Calculate K small image block features The similarity score with the target category text feature t, denoted as s i ={s1,s2,…,s k }.

6. The remote sensing semantic segmentation reasoning acceleration method based on fine-grained visual-language model according to claim 1, characterized in that: The specific process of step 5 is as follows: Set the similarity score threshold θ to filter out the s in step 4.2 i ={s1,s2,…,s k } is greater than the similarity score of θ, and we get M(1≤M≤2 6 ) small image blocks that need to be segmented and inferred, denoted as 7. The remote sensing semantic segmentation reasoning acceleration method based on fine-grained visual-language model according to claim 1 is characterized in that: The specific process of step 6 is as follows: The M small image blocks obtained by step 5 are input into the two-stage segmentation reasoning model to infer the corresponding mask image, which is recorded as Finally, the M masks are merged and restored to the sample image X i Original size, get the mask image Y of the entire large-size remote sensing image i .

Images