A multi-modal interactive extraction method based on satellite remote sensing images, a terminal device and a storage medium

Abstract

The application relates to a multi-modal interactive extraction method based on satellite remote sensing images, a terminal device and a storage medium, and the method comprises the following steps: selecting a satellite remote sensing image to be extracted; extracting a full segmentation vector element; calculating a full segmentation object Image Embedding; and interactively extracting a satellite remote sensing ground object target according to user input prompts. The application can efficiently and interactively extract a satellite remote sensing image to be extracted, can extract any visible ground object in the satellite remote sensing image without being limited by a prompt mode, and can output an extracted vector result. The application can support user input of various prompts (point, line, frame position prompt, text prompt, similar ground object prompt), interactively extracts a satellite remote sensing ground object target, and a brand-new labeling mode can greatly improve ground object labeling efficiency. Related technologies can also be used in daily project applications to reduce project application labor costs.

Description

Technical Field

[0001] This invention relates to the field of satellite remote sensing image technology, and in particular to a multimodal interactive extraction method, terminal device and storage medium based on satellite remote sensing images. Background Technology

[0002] With the development of Earth observation technology, the resolution and visual quality of satellite remote sensing image data are rapidly improving. Consequently, the amount of satellite remote sensing data is increasing dramatically. Extracting useful information from this massive amount of data has become a crucial issue. Traditional satellite remote sensing data processing methods typically rely on a single data source and fixed extraction algorithms, which often limits the extraction results to the quality of the data source and the effectiveness of the algorithm.

[0003] In the field of satellite remote sensing, multimodal interactive extraction is a technique used to interactively extract and identify corresponding target objects from satellite remote sensing image data of various satellite sources and resolutions. This technique fully leverages human recognition capabilities and computer processing power, ensuring both accuracy and efficiency in ground feature extraction from satellite remote sensing images. This method avoids the problem of automatic computer interpretation failing to meet accuracy requirements, while also solving the problem of low efficiency and high manpower requirements associated with manual visual interpretation. However, the extraction quality of multimodal interactive extraction is closely related to the base model. In the past, limitations imposed by the generalization and extraction capabilities of deep learning models have constrained the performance of multimodal interactive extraction.

[0004] In recent years, deep learning technology has developed rapidly, resulting in various models adapted to multimodal interactive extraction algorithms. For example, the Segment Anything Model supports flexible prompts, including point, box, or language cues, which users can use to guide the model in specific segmentation tasks. It can be applied to image extraction tasks in a wide range of scenarios. The CLIP model, a multimodal model combining a visual Transformer and a regularized Transformer, is used for matching images and text. It solves image classification and other related tasks by learning the shared latent space between images and text. Its superior performance in zero-shot scenarios makes it widely applicable in many fields. Combining these technologies makes efficient and high-quality multimodal interactive extraction of satellite remote sensing imagery a feasible solution. Summary of the Invention

[0005] The present invention aims to solve the technical problems in the prior art by providing a multimodal interactive extraction method, terminal device and storage medium based on satellite remote sensing images.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] A multimodal interactive extraction method based on satellite remote sensing imagery includes the following steps:

[0008] Step 1: Select the satellite remote sensing image to be extracted;

[0009] Step 2: Extract fully segmented vector features;

[0010] Step 3: Calculate the image embedding of the fully segmented object;

[0011] Step 4: Based on user input prompts, interactively extract satellite remote sensing ground targets.

[0012] In the above technical solution, step 1 specifically involves: selecting a satellite remote sensing image with dimensions of C×H×W;

[0013] Where C is the number of satellite remote sensing image bands, H is the altitude of the satellite remote sensing image, and W is the width of the satellite remote sensing image; the size of the fishing net is H. p ×W p H p W represents the height of the sliced ​​fishing net. p This refers to the width of the sliced ​​fishing net.

[0014] In the above technical solution, step 2 specifically involves: based on C×H×W and H p ×W p By setting a certain overlap rate, the entire satellite remote sensing image is pre-segmented using the SAM model, and then fused, vector topology logic error correction, and multi-part to single-part conversion operations are performed to obtain a fully segmented vector feature mask.

[0015] In the above technical solution, step 3 specifically involves: using the CLIP model to calculate the low-dimensional vectorized semantic Images Embedding of the satellite remote sensing raster matrix under the mask based on the obtained fully segmented vector feature mask.

[0016] In the above technical solution, step 4 specifically includes:

[0017] When a user first draws a point shape: the first point shape is treated as a foreground cue point. Based on the current point feature coordinates, a Point Prompt and a satellite remote sensing raster matrix of a specified size are generated around it. The size of this matrix is ​​used as input parameters for SAM inference to obtain a nondeterministic preview vector feature Polygon. preview If the result is satisfactory, the drawing process ends directly, obtaining the result along with the Polygon. previewThe same final extracted vector features are used; if the result is unsatisfactory, foreground and background cue points are added, and the polygon is updated interactively. preview Continue until the result is satisfactory;

[0018] When a user draws an irregular line, all the points forming the line are thinned out to full foreground prompt raster points, and vector features are directly generated.

[0019] When the user draws a prompt box, generate a Box Prompt based on the top left and bottom right corners of the box.

[0020] When the user inputs a text prompt: After inputting the text prompt, the processed text data is converted into a low-dimensional vectorized semantic image embedding, and the full segmentation vectorized semantic image embedding and I are calculated. t The cosine similarity is used to select the highest-ranking batch of vectorized masks as vector elements for generation;

[0021] When a user uses similar feature hints: after selecting a mask of outlined vector features, the satellite remote sensing raster matrix under the mask is converted into a low-dimensional vectorized semantic image embedding, and the fully segmented vectorized semantic images embedding and I are calculated. t The cosine similarity is used to select the highest-ranking batch of vectorized masks as vector elements for generation.

[0022] In the above technical solution, in step 4, when the user first draws a point shape, the first point shape is regarded as a foreground cue point. Based on the current point feature coordinates, the size of the Point's Prompt and the surrounding satellite remote sensing raster matrix of a specified size is generated as: C×H p ×W p .

[0023] In the above technical solution, in step 4, when the user first draws the point shape, a non-deterministic preview vector element Polygon is obtained. preview When the result is unsatisfactory, add foreground and background cue points:

[0024] Each append operation temporarily generates a Multi Point Prompt and a surrounding satellite remote sensing raster matrix of a specified size, with dimensions C×H. p ×W p All are used as input parameters for SAM inference to update the preview vector.

[0025] A terminal device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the above-described multimodal interactive extraction method based on satellite remote sensing images.

[0026] A storage medium is a non-volatile computer-readable storage medium storing at least one computer program that, when executed by a processor, implements the aforementioned multimodal interactive extraction method based on satellite remote sensing images.

[0027] The present invention has the following beneficial effects:

[0028] The present invention provides a multimodal artificial intelligence interactive extraction method based on satellite remote sensing, which efficiently extracts ground features from satellite remote sensing images. It can extract any visible ground features in satellite remote sensing images without being limited by prompting methods, and can output the extracted vector results.

[0029] The multimodal interactive extraction method based on satellite remote sensing in this invention supports user input of various prompts (point, line, and box location prompts, text prompts, and similar ground feature prompts) to interactively extract satellite remote sensing ground feature targets. The new annotation method can greatly improve the efficiency of ground feature annotation, and the related technology can also be used in daily project applications to reduce the manpower cost of project applications. Attached Figure Description

[0030] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0031] Figure 1 This is a flowchart illustrating the multimodal interactive extraction method based on satellite remote sensing imagery of the present invention.

[0032] Figure 2 This is a schematic diagram illustrating the use of a single-point cueing method to extract satellite remote sensing images of water bodies.

[0033] Figure 3 This is a schematic diagram illustrating the extraction of satellite remote sensing images of buildings using a single-point cueing method.

[0034] Figure 4 This is a schematic diagram illustrating the use of a multi-point cueing method to extract satellite remote sensing images of forest land.

[0035] Figure 5 This is a schematic diagram illustrating the use of line hinting to extract satellite remote sensing images of large bodies of water.

[0036] Figure 6 This is a schematic diagram illustrating the use of the box hint method to extract satellite remote sensing imagery of a building.

[0037] Figure 7 A schematic diagram of satellite remote sensing images used to extract multiple farmland plots using similar feature cues.

[0038] Figure 8 This is a schematic diagram illustrating how to extract multiple water targets from a satellite remote sensing image using text prompts. Detailed Implementation

[0039] This invention mainly constructs a multimodal interactive extraction method based on satellite remote sensing, and based on this method, develops an interactive ground feature extraction terminal device and storage medium for satellite remote sensing images.

[0040] The multimodal interactive extraction method based on satellite remote sensing of the present invention can stably and quickly extract the ground targets required by the user from satellite remote sensing images interactively.

[0041] The multimodal interactive extraction method based on satellite remote sensing in this invention supports user input of various prompts (point, line, and box location prompts, text prompts, and similar ground feature prompts) to interactively extract satellite remote sensing ground feature targets. The new annotation method can greatly improve the efficiency of ground feature annotation, and the related technology can also be used in daily project applications to reduce the manpower cost of project applications.

[0042] like Figure 2-8 As shown, the interactive feature extraction method for satellite remote sensing images consists of a core interactive point, line, and frame location prompt, text prompt, and similar feature prompt vector generation algorithm, as well as numerous map drawing tool algorithms (such as reshaping features, cutting features, etc.) to assist the algorithm. In case the algorithm is not applicable in extreme situations, there is also a manual feature extraction method to make up for it, and the extraction results can be fine-tuned to a suitable vector shape.

[0043] For drawing point, line, and box position prompts: After the user draws, the algorithm first classifies the prompts into the following categories: ① Single-point prompt (clicking only one foreground prompt point); ② Multi-point prompt (including clicking multiple foreground and background prompt points, or drawing a line, which is considered as multiple foreground prompt points); ③ Box prompt (the coordinates of the upper left and lower right corners of the drawn box). After the judgment, the algorithm calculates the corresponding ground feature mask vector elements for the target. If the user is not satisfied, position prompts can be added to the existing prompts for interactive real-time extraction until the extraction result is satisfactory.

[0044] For text prompts: After the user enters the text description they want to extract in the text box (e.g., the white building in the image), the algorithm first standardizes the text description, excluding non-standard punctuation and unknown symbols. Then, it takes the text prompt as input and uses the Chinese CLIP model adapted to satellite remote sensing data to calculate the Text Embedding, i.e., low-dimensional vectorized semantics. It also calculates the batch of mask vector elements with the highest semantic similarity among all masks pre-calculated in the user's current satellite remote sensing image. If the user is not satisfied, the mask can be regenerated or the threshold, confidence level, and other parameters can be modified to recalculate more suitable mask vector elements until the extraction effect is satisfactory.

[0045] For similar feature hints: After the user selects the mask vector to be extracted from the drawn point, line, or box position hints, or draws a mask vector themselves, the algorithm first takes the satellite remote sensing raster matrix under the mask as input, uses the Chinese CLIP model adapted in the satellite remote sensing data to calculate the Image Embedding, i.e., low-dimensional vectorized semantics, and calculates the batch of mask vector elements with the highest semantic similarity among all masks in the user's current satellite remote sensing image. If the user is not satisfied, consistent with the text hint, a more suitable mask vector element can be recalculated until the extraction effect is satisfactory.

[0046] The English words, English abbreviations, and letter variables involved in the specific embodiments of this invention include:

[0047] CLIP: Contrastive Language-Image vs. Training Model;

[0048] Text Embedding: Text word vectors;

[0049] Box: A horizontal rectangle;

[0050] Image Embedding: Image vector;

[0051] Point Prompt: A point prompt;

[0052] Polygon preview Non-deterministic preview vector features;

[0053] Multi-Point Prompt: Multiple point prompts.

[0054] The present invention will now be described in detail with reference to the accompanying drawings.

[0055] The flowchart of the multimodal interactive extraction method based on satellite remote sensing imagery of the present invention is as follows: Figure 1 As shown, the method specifically includes the following steps:

[0056] Step 1: Select the satellite remote sensing image to be extracted:

[0057] Users will select satellite remote sensing images with dimensions of C×H×W;

[0058] Where C is the number of satellite remote sensing image bands, H is the altitude of the satellite remote sensing image, and W is the width of the satellite remote sensing image; the user will also determine the fishing net size H. p ×W p H p W represents the height of the sliced ​​fishing net. p This refers to the width of the sliced ​​fishing net.

[0059] Step 2: Extract fully segmented vector features:

[0060] Based on C×H×W and H p ×W p By setting a certain overlap rate (usually 10%), the entire satellite remote sensing image is pre-segmented using the SAM model, and then fused, vector topology logic error correction, and multi-part to single-part conversion operations are performed to obtain a fully segmented vector feature mask.

[0061] Step 3: Calculate the fully segmented object Image Embedding:

[0062] Based on the fully segmented vector feature mask obtained above, the CLIP model is used to calculate the low-dimensional vectorized semantic Images Embedding of the satellite remote sensing raster matrix under the mask, i.e., I1, I2, ..., I n , and are stored in running memory for later use.

[0063] Step 4 (Branch 1): User draws prompts for point, line, and box positions.

[0064] When a user first draws a point shape, the algorithm treats the first point shape as a foreground cue point. Based on the current point feature coordinates, it generates a Point Prompt: p(x,y) and a satellite remote sensing raster matrix of a specified size (usually the size of a fishnet slice) around it, with a size of C×H. p ×W p All are used as input parameters for SAM inference to obtain nondeterministic preview vector features (Polygons). preview Users can use Polygon preview Determine the subsequent operation steps: ① If the result is satisfactory, stop drawing directly and obtain the Polygon. preview The same final extracted vector features; ② If the result is unsatisfactory, the user can add foreground and background cue points to interactively update the Polygon. preview Each append operation will temporarily generate a Multi Point. Surrounding a satellite remote sensing raster matrix of a specified size, with dimensions C×H p ×W p All of these are used as input parameters for SAM inference to update the preview vector until the result satisfies the user.

[0065] When a user draws irregular lines, the algorithm will thin out all the points forming the lines into a full foreground cue bitmap, i.e. Directly generate vector features;

[0066] When a user draws a tooltip, a Box Prompt will be generated based on the top left and bottom right corners of the box.

[0067] Step 4 (Branch 2): User input text prompt

[0068] After the user inputs text suggestions, the algorithm converts the processed text data into a low-dimensional vectorized semantic ImageEmbedding, i.e., I... t After that, the fully segmented vectorized semantic image embedding is calculated, i.e., I1, I2, ..., I n with I t The cosine similarity is used to select the highest-ranking batch of vectorized masks as vector elements for generation.

[0069] Step 4 (Branch 3): Users use similar feature suggestions

[0070] After the user selects the outlined vector feature mask, the algorithm converts the satellite remote sensing raster matrix under the mask into a low-dimensional vectorized semantic image embedding, i.e., I... i After that, similar to branch 2, the highest batch of cosine similarity vectorized masks are calculated as vector elements for generation.

[0071] The present invention also provides a terminal device that can implement the aforementioned multimodal interactive extraction method. The terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the various steps of the aforementioned multimodal interactive extraction method embodiments and achieves the same technical effect. To avoid repetition, these steps will not be repeated here. The terminal device of the present invention is an electronic computer.

[0072] This invention also provides a storage medium, which is a non-volatile computer-readable storage medium storing at least one computer program. When the computer program is executed by a processor, it implements the various steps of the above-described multimodal interactive extraction method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0073] The present invention provides a multimodal artificial intelligence interactive extraction method based on satellite remote sensing, which efficiently extracts ground features from satellite remote sensing images. It can extract any visible ground features in satellite remote sensing images without being limited by prompting methods, and can output the extracted vector results.

[0074] The multimodal interactive extraction method based on satellite remote sensing in this invention supports user input of various prompts (point, line, and box location prompts, text prompts, and similar ground feature prompts) to interactively extract satellite remote sensing ground feature targets. The new annotation method can greatly improve the efficiency of ground feature annotation, and the related technology can also be used in daily project applications to reduce the manpower cost of project applications.

[0075] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A multimodal interactive extraction method based on satellite remote sensing imagery, characterized in that, Includes the following steps: Step 1: Select the satellite remote sensing image to be extracted; Step 2: Extract fully segmented vector features; Step 3: Calculate the image embedding of the fully segmented object; Step 4: Based on user input prompts, interactively extract satellite remote sensing ground targets; Step 1 specifically involves selecting the size as follows: Satellite remote sensing images; in, Number of satellite remote sensing image bands The altitude of the satellite remote sensing image. The width of the satellite remote sensing image; the size of the fishing net is... ,in The height of the sliced ​​fishing net, The width of the sliced ​​fishing net; Step 2 specifically involves: according to and By setting a certain overlap rate, the entire satellite remote sensing image is pre-segmented using the SAM model, and then fused, vector topology logic error correction, and multi-part to single-part conversion operations are performed to obtain a fully segmented vector feature mask. Step 3 specifically involves: using the CLIP model to calculate the low-dimensional vectorized semantic Images Embedding of the satellite remote sensing raster matrix under the mask, based on the obtained fully segmented vector feature mask. Step 4 is as follows: When a user first draws a point shape: the first point shape is treated as a foreground cue point. Based on the current point feature coordinates, a Point Prompt and a satellite remote sensing raster matrix of a specified size are generated around it. The size of this matrix is ​​used as input parameters for SAM inference to obtain nondeterministic preview vector features. If the result is satisfactory, the drawing process can be stopped directly, and the result will be obtained. The same final extracted vector features are used; if the result is unsatisfactory, foreground and background cue points are added, and the process is updated interactively. Continue until the result is satisfactory; When a user draws an irregular line, all the points forming the line are thinned out to full foreground prompt raster points, and vector features are directly generated. When the user draws a prompt box, generate a Box Prompt based on the top left and bottom right corners of the box. When the user inputs a text prompt: After inputting the text prompt, the processed text data is converted into a low-dimensional vectorized semantic image embedding, and the full segmentation vectorized semantic image embedding is calculated. The cosine similarity is used to select the highest-ranking batch of vectorized masks as vector elements for generation; When a user uses similar feature hints: after selecting a mask of outlined vector features, the satellite remote sensing raster matrix under the mask is converted into a low-dimensional vectorized semantic image embedding, and the fully segmented vectorized semantic images embedding is calculated. The cosine similarity is used to select the highest-ranking batch of vectorized masks as vector elements for generation.

2. The multimodal interactive extraction method based on satellite remote sensing imagery according to claim 1, characterized in that, In step 4, when the user first draws a point shape, the first point shape is regarded as a foreground cue point. Based on the current point feature coordinates, the size of the Point's Prompt and the surrounding satellite remote sensing raster matrix of a specified size are generated as follows: .

3. The multimodal interactive extraction method based on satellite remote sensing imagery according to claim 1, characterized in that, In step 4, when the user first draws a point shape, a nondeterministic preview vector feature is obtained. When the result is unsatisfactory, add foreground and background cue points: Each append operation temporarily generates a Multi Point Prompt and a surrounding satellite remote sensing raster matrix of a specified size. All are used as input parameters for SAM inference to update the preview vector.

4. A terminal device, the terminal device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the multimodal interactive extraction method based on satellite remote sensing imagery as described in any one of claims 1-3.

5. A storage medium, the storage medium being a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing at least one computer program, which, when executed by a processor, implements the multimodal interactive extraction method based on satellite remote sensing imagery as described in any one of claims 1-3.

Images