Remote sensing image segmentation method, system, device and medium based on visual large model

Abstract

The present invention provides a remote sensing image segmentation method, system, device and medium based on a visual large model, which belongs to the technical field of image segmentation. The scheme is based on existing geographic spatial information data products, does not require additional manual prompt input or production of manual labeled data, and does not require additional model or module training. It can automatically generate corresponding point prompts and frame prompts to achieve the wide application of SAM in the automatic segmentation task of remote sensing images, thereby improving the practicability of SAM in the remote sensing field.

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation, and in particular relates to a remote sensing image segmentation method, system, equipment and medium based on a large visual model. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In image segmentation based on large visual models, the accuracy of prompt input is a factor that affects the accuracy of image segmentation. Existing technologies usually manually annotate downstream tasks or rely on large-scale closed or restricted data sets for training, so that the model can understand and generate images related to given text prompts. This results in the existing methods often having difficulty in providing effective and accurate prompts when faced with specific types of data, such as geospatial data, which limits the application performance of large visual models in other specific scenarios.

[0004] Existing studies have shown that the performance of SAM (Segment Anything Model) (a commonly used large visual model) in remote sensing image segmentation tasks can approach or even surpass the performance of visual models trained on specific task datasets to a certain extent. However, SAM's performance in remote sensing image segmentation tasks is still insufficient, mainly because SAM relies heavily on prior manual prompt guidance. Due to differences in the quality of remote sensing images themselves (which may be caused by differences in resolution, sensor type, time phase, etc.) and differences in the type, location, and number of input prompt projects, the segmentation results will be unstable. In order to achieve ideal segmentation results, it is usually necessary to refine the manual prompts, which will cost huge manpower and time costs for different remote sensing segmentation tasks, limiting the widespread application of SAM in the remote sensing field. Summary of the invention

[0005] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a remote sensing image segmentation method, system, device and medium based on a visual large model. The scheme is based on existing geospatial information data products and does not require additional manual prompt input or production of manually labeled data, nor does it require additional model or module training. It can automatically generate corresponding point prompts, box prompts and mask prompts to achieve the wide application of SAM in the automatic segmentation task of remote sensing images, thereby improving the practicability of SAM in the remote sensing field.

[0006] According to a first aspect of an embodiment of the present invention, a remote sensing image segmentation method based on a visual large model is provided, comprising:

[0007] Obtain the remote sensing image to be segmented and its corresponding geospatial information data, and perform corresponding preprocessing;

[0008] For the remote sensing image to be segmented, based on the vector data of the target object in the corresponding geographic spatial information data, a plurality of sampling points are obtained by sequentially performing rasterization processing and data point sampling processing; and point prompts of the remote sensing image are generated based on the sampling points;

[0009] For the remote sensing image to be segmented, based on the key geographic coordinate data of the target object in the corresponding geographic spatial information data, a target object bounding box is constructed, and the bounding box is used as a frame prompt of the remote sensing image;

[0010] For the remote sensing image to be segmented, based on the vector data of the target object in the corresponding geographic spatial information data, it is processed by rasterization and expansion algorithm in sequence to obtain rasterized labels as mask prompts;

[0011] The obtained point prompt, frame prompt and mask prompt are used as prompt information of the remote sensing image to be segmented;

[0012] Based on the prompt information and the remote sensing image to be segmented, the segmentation result of the remote sensing image is obtained by using the pre-trained SAM visual large model.

[0013] Furthermore, the generation of the point prompt is specifically as follows: based on the vector data of the target object in the geographic spatial information data corresponding to the remote sensing image, a binary image is obtained through rasterization processing; based on the binary image, a number of sampling points are obtained through grid sampling; and based on the corresponding position pixels of the sampling points on the remote sensing image, point prompts are generated.

[0014] Furthermore, the grid sampling method is specifically as follows: vector data of the target object is rasterized to obtain corresponding grid labels; the grid labels are divided based on a grid of a preset size; for each grid, discrete points that fall within the grid and are the target to be segmented are randomly sampled to obtain a sampled point set.

[0015] Furthermore, the generation of the frame prompt is specifically as follows: based on the key geographic coordinate data of the target object in the corresponding geographic spatial information data of the remote sensing image, the corresponding image coordinates in the remote sensing image are obtained through coordinate conversion; and the frame prompt in the remote sensing image is formed based on the obtained image coordinates.

[0016] Furthermore, a frame prompt in the remote sensing image is formed based on the obtained image coordinates, and the following optimization process is performed on the formed frame prompt: the frame prompt is used as an initial prompt frame, wherein the coordinates of the upper left corner point of the initial prompt frame are used as the starting point, and the coordinates of the lower right corner point are used as the end point; the horizontal distance and the vertical distance between the starting point and the end point are calculated respectively, and a preset value is determined as the short side length of the optimized frame by determining the shortest side of the initial prompt frame; the long side length of the optimized frame is calculated according to the aspect ratio of the initial prompt frame; an offset value is preset for the long side length of the optimized frame to correct the quality deviation of the vector data, and multiple sub-prompt frames covering the entire target object area are generated without exceeding the original image range; and the multiple sub-prompt frames are used as frame prompts for the target object in the remote sensing image.

[0017] Furthermore, based on the prompt information and the remote sensing image to be segmented, the pre-trained SAM visual large model is used to obtain the segmentation result of the remote sensing image, specifically: the binary images corresponding to the point prompt, box prompt and mask prompt are added pixel by pixel to obtain a fused prompt, and the fused prompt is used as the input prompt information of the SAM visual large model to obtain the segmentation result of the remote sensing image.

[0018] Furthermore, the key geographic coordinate data includes the coordinates of the starting point and the ending point of the road.

[0019] According to a second aspect of an embodiment of the present invention, a remote sensing image segmentation system based on a visual large model is provided, comprising:

[0020] A data acquisition unit, which is used to acquire the remote sensing image to be segmented and its corresponding geospatial information data, and perform corresponding preprocessing;

[0021] A point prompt generating unit is used for obtaining a plurality of sampling points by sequentially performing rasterization processing and data point sampling processing on the remote sensing image to be segmented based on the vector data of the target object in the corresponding geographic spatial information data; and generating point prompts of the remote sensing image based on the sampling points;

[0022] A frame prompt generating unit, which is used to construct a target object bounding box for the remote sensing image to be segmented based on the key geographic coordinate data of the target object in the corresponding geographic spatial information data, and use the bounding box as a frame prompt of the remote sensing image;

[0023] A mask hint generating unit is used for obtaining a rasterized label as a mask hint by sequentially performing rasterization processing and dilation algorithm processing on the remote sensing image to be segmented based on the vector data of the target object in the corresponding geographic spatial information data;

[0024] A prompt information generating unit, which is used to obtain the point prompt, the frame prompt and the mask prompt as prompt information of the remote sensing image to be segmented;

[0025] The image segmentation unit is used to obtain the segmentation result of the remote sensing image based on the prompt information and the remote sensing image to be segmented by using the pre-trained SAM visual large model.

[0026] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising a memory, a processor and a computer program stored and running on the memory, wherein when the processor executes the program, the remote sensing image segmentation method based on a large visual model is implemented.

[0027] According to a fourth aspect of an embodiment of the present invention, there is provided a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the remote sensing image segmentation method based on a large visual model.

[0028] One or more of the above technical solutions have the following beneficial effects:

[0029] The present invention provides a remote sensing image segmentation method, system, device and medium based on a visual large model. The scheme is based on existing geospatial information data products, and does not require additional manual prompt input or manual marking data production, nor does it require additional model or module training. It can automatically generate corresponding point prompts, frame prompts and mask prompts to achieve the wide application of SAM in the automatic segmentation task of remote sensing images, and improve the practicality of SAM in the field of remote sensing. In the generation of frame prompts, the scheme of the present invention takes into account that the roads are crisscrossed on the image and have slender shapes and different lengths. If each road is directly selected using the maximum and minimum values ​​of the starting point and the end point, each frame will have a large amount of redundant information, and may even cover the entire image range. At the same time, there may be overlapping information between frames, which is not conducive to the accurate generation of frame prompt engineering; based on this problem, the present invention uses a number of sub-prompt frames to replace the prompt frame generated using the maximum and minimum values ​​of the starting point and the end point, effectively removing the redundant information in the prompt frame. The advantages of the additional aspects of the present invention will be partially given in the following description, and some will become obvious from the following description, or will be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0031] Figure 1 Schematic diagram of a SAM-seg implementation method of remote sensing image segmentation based on the SAM model described in an embodiment of the present invention;

[0032] Figure 2It is a schematic diagram of a SAM-cls implementation method of remote sensing image segmentation based on the SAM model described in an embodiment of the present invention;

[0033] Figure 3 Schematic diagram of the SAM-det implementation method of remote sensing image segmentation based on the SAM model described in an embodiment of the present invention;

[0034] Figure 4 It is a schematic diagram of a SAM-RSPrompter implementation method of remote sensing image segmentation based on the SAM model described in an embodiment of the present invention;

[0035] Figure 5 A schematic diagram showing comparison of road segmentation results for different point prompt input quantities described in an embodiment of the present invention;

[0036] Figure 6 The road segmentation results of different point sampling methods described in the embodiments of the present invention;

[0037] Figure 7 It is a schematic diagram of the initial frame prompt optimization process described in an embodiment of the present invention;

[0038] Figure 8 This is a flow chart of an image prompt generation method for a large visual model segmentation task described in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0040] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0041] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0042] Embodiment 1

[0043] The purpose of this embodiment is to provide an image segmentation method based on a large visual model.

[0044] With the rapid development of the field of artificial intelligence, basic models have benefited from the substantial expansion of data and model scale. Basic models have been successfully used for zero / few-shot learning, and can complete different downstream tasks without a large amount of task-specific data or model parameter updates. In the field of vision, there are already basic visual models that can demonstrate strong zero-shot generalization performance on different downstream tasks through natural language descriptions or visual cues. The large visual model SAM (SegmentAnything Model) proposed by META specifies the object or content of image segmentation through cues in the form of visual input points, boxes, or masks, enabling the model to perform category-independent segmentation tasks. SAM has been applied to a variety of fields such as medicine, video tracking, and remote sensing.

[0045] Existing studies have shown that the performance of SAM in remote sensing image segmentation tasks can, to a certain extent, approach or even surpass the performance of visual models trained on specific task datasets. However, SAM's performance in remote sensing image segmentation tasks is still insufficient, mainly because SAM relies heavily on prior manual prompt guidance. Due to differences in the quality of remote sensing images themselves (which may be caused by differences in resolution, sensor type, time phase, etc.) and differences in the type, location, and number of input prompt projects, the segmentation results will be unstable. In order to achieve ideal segmentation results, manual prompts usually need to be refined. For different remote sensing segmentation tasks, this will cost huge manpower and time costs, limiting the widespread application of SAM in the remote sensing field.

[0046] In order to realize the automatic remote sensing image segmentation capability of SAM, there are currently four feasible solutions: Figures 1 to 4 As shown:

[0047] Figure 1 The method in (i.e. SAM-seg) is a network framework that combines the image encoder in SAM and the external decoder. Input to frozen SAM video encoder , and after multi-scale feature enhancer After processing to obtain multi-scale features, input into a custom trained network decoder , to perform the segmentation task and obtain the segmentation result In order to achieve the segmentation task of a specific target object, it is necessary to train an unfrozen external decoder .

[0048] Figure 2 The method in (i.e. SAM-cls) is based on SAM segmentation and uses a classifier to assign labels to the segmentation results. Image encoder input to SAM Generate Features , evenly distribute points on the image ( x i , y i ) as input point prompt engineering, through the prompt encoder Generate sparse coded features . and Input to the mask decoder , the segmentation result of SAM can be achieved Finally, the segmentation result Input to an external classifier that needs to be trained Assign labels to each mask in the segmentation result .

[0049] Figure 3 The method in (i.e., SAM-det) is to use an object detector to obtain the bounding box of the target object as a prompt input to SAM. Input to video encoder and object detectors , the target detector needs to be trained under a specific task model to achieve target detection of specific objects and generate the corresponding bounding box The box hint engineering as input to the SAM model. Similarly, the hint encoder Generate sparse coded features and Input to the mask decoder , to obtain the segmentation result of the target object .

[0050] Figure 4 The approach in SAM-RSPrompter is similar to SAM-det, but the difference is that it directly generates prompt embeddings instead of the prompt itself. Image encoder input to SAM Generate Features , and through a feature enhancer Processing to obtain multi-scale features Then through a task-specific training prompter Generate sparse coded features , and finally passes through an external decoder Generate segmentation results In order to meet the segmentation results of a specific task, the prompter and decoder need to be trained, and the frozen encoder does not participate in the training.

[0051] However, the four existing feasible solutions all require specific standard datasets for module training (such as Figure 1 External decoder, Figure 2 Classifier, Figure 3 Object detectors, Figure 4 The external decoder and prompter represent the modules that need to be activated for parameter training). When applied to different tasks, it is necessary to produce or collect corresponding training datasets according to specific standards, which greatly limits the application of large visual models to remote sensing segmentation tasks with different requirements.

[0052] Combined with the above solutions, it can be found that the existing SAM-based remote sensing image segmentation method still has the following problems:

[0053] SAM is able to perform category-independent segmentation tasks, but due to the complex background interference and lack of clearly defined object edges in remote sensing image scenes, it is difficult for SAM to achieve complete segmentation of remote sensing image targets, and its results are heavily dependent on the type, location, and number of prompts. In order to achieve ideal segmentation results, fine manual labeling of prompts (points or bounding boxes, etc.) is usually required. For different remote sensing segmentation tasks, this will cost huge manpower and time costs.

[0054] The above four solutions based on SAM automated remote sensing segmentation tasks still require a large amount of data sets for modular training. SAM-seg, SAM-cls, SAM-det, and SAM-RSPrompter require a certain amount of training data to be trained to obtain decoder module parameters, classifier module parameters, target detection module parameters, and prompt engineering module parameters; and because the functions and goals of different modules are different, the requirements and data formats of the corresponding data sets are also different. This also increases the requirements for training data sets and the labor cost of making corresponding data sets.

[0055] Remote sensing image interpretation is closely related to geospatial information. Remote sensing is one of the main means of obtaining geospatial information, and geospatial information can also provide positioning and geographic reference for remote sensing image processing and interpretation. The remote sensing segmentation application based on SAM and the above four solutions are still in the processing of computer vision, lacking the processing and application of geospatial information data.

[0056] In order to solve the above problems, the solution described in this embodiment uses geographic spatial information data, considers the particularity of linear vector data, and mines its geographic spatial information (such as open source OSM vector data), without any manual prompt input, without labeling data and making specific data sets for network module training, and can automatically generate corresponding point prompts and box prompts as prompt inputs of SAM to achieve remote sensing image segmentation. Specifically, Figure 8 As shown, a remote sensing image segmentation method based on a visual large model includes:

[0057] Step 1: Obtain the remote sensing image to be segmented and its corresponding geospatial information data, and perform corresponding preprocessing;

[0058] Specifically, the step 1 specifically performs the following processing:

[0059] First, remote sensing images and vector data are collected according to task requirements. Remote sensing image preprocessing usually goes through radiation calibration, atmospheric correction, orthorectification, image registration, image fusion, cropping and stitching and other preprocessing processes.

[0060] Since the data publishing platform preprocesses the original remote sensing data to a certain extent before publishing different levels of data, when downloading or receiving image data, users need to take corresponding preprocessing steps according to the actual data situation, and consider the actual area range, and need to crop or splice the image. Secondly, when preprocessing vector data, according to the area range and coordinate system of the preprocessed remote sensing image, use ENVI or ArcGIS professional software to convert the coordinate system and crop the area of ​​the vector data, or use the arcpy library of the Python computer language for batch processing.

[0061] For geographic spatial information data, vector data can be obtained based on OSM open source data, or through non-open source download methods. Here, the vector data of OSM road centerlines in a certain area is taken as an example. Historical OSM data records of any area can be downloaded for free at the official website address.

[0062] According to the coordinate system and regional scope of the image, ENVI or ArcGIS professional software is used to perform coordinate system conversion and regional clipping on OSM road vector data.

[0063] Finally, the collection and preprocessing of remote sensing images and geospatial information data (such as OSM road vector data) are completed.

[0064] Step 2: For the remote sensing image to be segmented, based on the vector data of the target object in the corresponding geographic spatial information data, sequentially undergo rasterization processing and data point sampling processing to obtain a number of sampling points; and generate point prompts of the remote sensing image based on the sampling points;

[0065] In a specific implementation, the generation of the point prompt is specifically as follows: based on the vector data of the target object in the geographic spatial information data corresponding to the remote sensing image, a binary image is obtained through rasterization processing; based on the binary image, a number of sampling points are obtained through grid sampling; based on the corresponding position pixels of the sampling points on the remote sensing image, point prompt generation is realized; wherein, in the grid sampling, the vector data of the target object is rasterized to obtain the corresponding grid label; based on the grid of a preset size, the grid label is divided; for each grid, random sampling is performed from the discrete points that fall within the grid and are the targets to be segmented, to obtain a sampled point set.

[0066] Specifically, the step 2 includes the following processing (the following is explained using a road object as an example):

[0067] 1) Rasterization of vector road data: Use ArcGIS professional software or Python code to convert the vector data of the road centerline into raster labels (0 / 1 binary images) while maintaining the size of the original image.

[0068] 2) Sampling points. The number and location of the point prompt project will affect the results of SAM segmentation (such as Figure 5 and Figure 6 As shown in FIG. 1 ), this embodiment finds through experimental comparison of random sampling, interval sampling, and grid sampling (as shown in FIG. Figure 6 As shown in Figure 2), the points sampled by the grid method are more evenly distributed, making the extraction effect more accurate. Therefore, the scheme described in this embodiment will set an appropriate grid size for specific objects and image resolution. Figure 6 As shown in the figure, the image resolution is 0.8m. When extracting roads, it is set to 10 A regular grid of 10 pixels is used to indicate the point of the project, with pixels labeled as roads and falling on the grid points.

[0069] 3) Point prompt generation. Multiple points after grid sampling are generated as The SAM model is input in the form of .

[0070] Specifically, first, read the rasterization tag to construct a set of discrete points of the road centerline; by setting the size of the regular grid, such as , indicating that each The pixel size is a grid; randomly sample a point from the discrete points that fall within the grid and are roads, and finally obtain the point set after grid sampling .

[0071] Step 3: For the remote sensing image to be segmented, based on the key geographic coordinate data of the target object in its corresponding geographic spatial information data, a target object bounding box is constructed, and the bounding box is used as a frame prompt of the remote sensing image;

[0072] In a specific implementation, the generation of the frame prompt is specifically as follows: based on the key geographic coordinate data of the target object in the corresponding geographic spatial information data of the remote sensing image, the corresponding image coordinates in the remote sensing image are obtained through coordinate conversion; and the frame prompt in the remote sensing image is formed based on the obtained image coordinates.

[0073] In one or more embodiments, a frame prompt in a remote sensing image is formed based on the obtained image coordinates, and the following optimization process is performed on the formed frame prompt: the frame prompt is used as an initial prompt frame, wherein the coordinates of the upper left corner point of the initial prompt frame are used as the starting point, and the coordinates of the lower right corner point are used as the end point; the horizontal distance and the vertical distance between the starting point and the end point are calculated respectively, and a preset value is determined as the short side length of the optimized frame by determining the shortest side of the initial prompt frame; the long side length of the optimized frame is calculated according to the aspect ratio of the initial prompt frame; an offset value is preset for the long side length of the optimized frame to correct the quality deviation of the vector data, and multiple sub-prompt frames covering the entire target object area are generated without exceeding the original image range; and the multiple sub-prompt frames are used as frame prompts for the target object in the remote sensing image.

[0074] Specifically, step 3 specifically performs the following processing:

[0075] GeoJOSN data format conversion. Extract the key geographic coordinate information of the target object. For road objects, the key geographic coordinate information is the coordinates of the starting point and the end point of each road. First, convert the road vector data .shp format into a .geojson format file through ArcGIS professional software or computer language code such as Python. GeoJSON is a format for encoding geographic data structures, supporting the geometry and attribute information of multi-feature objects such as points, lines, and surfaces. The GeoJSON data of OSM road data contains information such as the number, type, and geographic coordinates of the starting point, some intersections, and the end point of each road.

[0076] Coordinate acquisition and conversion of key points. For linear objects, such as roads, the key geographic coordinate information is the starting and ending coordinates of each road. Use Python computer language to read each linear road in GeoJOSN file and the corresponding starting point geographic coordinates ) and the geographic coordinates of the destination ,in, lon represents longitude, lat Represents latitude. Then, use Python language and GDAL library to convert the starting point and end point Convert geographic coordinates to image coordinates ), ,in,( x,y ) are the position coordinates in the image.

[0077] Box hint generation. The box hint input to the SAM model is in the form of (upper left corner point coordinates, lower right corner point coordinates). By obtaining the image coordinates of the key points (road start and end points), an initial bounding box can be used to cover the entire road, and the initial bounding box is expressed as ,in, Indicates taking the minimum value, Indicates taking the maximum value. Since linear roads are crisscrossed on the image and have slender shapes and different lengths, if the initial bounding box is used, there will be a lot of redundant information. Even the initial bounding box generated by one road may cover the entire image range. At the same time, there may be overlaps between the initial bounding boxes generated by multiple roads, which is not conducive to the accurate generation of the box prompt project. Therefore, in order to improve the accuracy of the box prompt generation, the following optimization scheme will be used to split the initial bounding box into several small boxes, specifically:

[0078] Figure 7 The road formed by starting point A and ending point B on a certain image is shown. According to the lateral distance between points AB The vertical distance , determine the shortest side of the prompt box (in Figure 7 For example, , with the short side being the vertical side, that is, the width of the prompt box is set to a fixed value s). The number of all prompt boxes covering the road (n+1) consists of n boxes with a vertical side length of s and one box with a vertical side length of Under the condition that the initial bounding box and several optimized bounding boxes have the same aspect ratio, the length of the other side can be calculated according to the corresponding ratio. In order to reduce the possible deviation of linear data quality, an offset value is added to the long border without exceeding the image range. . Figure 7 All the red dashed boxes in the example are optimized box tips for a road, and each box is represented by the coordinates of the upper left corner and the lower right corner. Finally, the optimized box tips will be generated. Input into the SAM model. Among them, The image coordinates of the upper left corner of the optimization box. The image coordinates of the lower right corner of the optimization box. Figure 7 In the example, For example, the value input to the SAM model can be expressed as .

[0079] Step 4: For the remote sensing image to be segmented, based on the vector data of the target object in the corresponding geospatial information data, it is processed by rasterization and expansion algorithm in sequence to obtain a rasterized label as a mask prompt;

[0080] Specifically, the mask prompt is generated as follows:

[0081] Rasterization of vector road data: Use ArcGIS professional software or Python code to convert linear vector data into raster labels (0 / 1 binary images) while maintaining the size of the original image.

[0082] Widen the width of linear data. Through the expansion algorithm, rasterized linear binary labels are converted into binary labels with a certain width. Taking the road extraction task as an example, if the vector data is the center line of the road, the rasterized linear data is used as the center and expanded to both sides to generate a road label result with a certain width.

[0083] Mask hint generation. The mask hints input to the SAM model are in the form of rasterized labels.

[0084] Step 5: Use the obtained point prompts, box prompts and mask prompts as prompt information of the remote sensing image to be segmented;

[0085] Step 6: Based on the prompt information and the remote sensing image to be segmented, the pre-trained SAM visual model is used to obtain the segmentation result of the remote sensing image; specifically, the binary images corresponding to the point prompt, box prompt and mask prompt are added pixel by pixel to obtain a fused prompt, and the fused prompt is used as the input prompt information of the SAM visual model to obtain the segmentation result of the remote sensing image.

[0086] In one or more embodiments, the obtained point prompts, box prompts, and mask prompts can be used as labels for sample images in a training set of a specific remote sensing image segmentation task for a visual large model to improve the training effect and efficiency of the visual large model.

[0087] In the specific implementation, according to different prompting engineering forms, specific forms of data are input to the SAM model. For point prompting engineering, the input form is a set of image coordinates of sampling points. ,in, ; For the box prompt project, the input form is the image coordinate set of the upper left corner and lower right corner of the optimization box ,in, and They are the image coordinates of the upper left corner and the lower right corner respectively; for mask prompt projects, the input form is a rasterized binary label.

[0088] In one or more embodiments, corresponding to the above-mentioned remote sensing image segmentation method based on a visual large model, a remote sensing image segmentation system based on a visual large model is provided, including:

[0089] A data acquisition unit, which is used to acquire the remote sensing image to be segmented and its corresponding geospatial information data, and perform corresponding preprocessing;

[0090] A point prompt generating unit is used for obtaining a plurality of sampling points by sequentially performing rasterization processing and data point sampling processing on the remote sensing image to be segmented based on the vector data of the target object in the corresponding geographic spatial information data; and generating point prompts of the remote sensing image based on the sampling points;

[0091] A frame prompt generating unit, which is used to construct a target object bounding box for the remote sensing image to be segmented based on the key geographic coordinate data of the target object in the corresponding geographic spatial information data, and use the bounding box as a frame prompt of the remote sensing image;

[0092] A mask hint generating unit is used for obtaining a rasterized label as a mask hint by sequentially performing rasterization processing and dilation algorithm processing on the remote sensing image to be segmented based on the vector data of the target object in the corresponding geographic spatial information data;

[0093] A prompt information generating unit, which is used to obtain the point prompt, the frame prompt and the mask prompt as prompt information of the remote sensing image to be segmented;

[0094] The image segmentation unit is used to obtain the segmentation result of the remote sensing image based on the prompt information and the remote sensing image to be segmented by using the pre-trained SAM visual large model.

[0095] It should be noted here that each module in this embodiment corresponds to each step in Example 1 one by one, and the specific implementation process is the same, which will not be repeated here.

[0096] In further embodiments, there is also provided:

[0097] An electronic device includes a memory and a processor and computer instructions stored in the memory and executed on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the above embodiment. For the sake of brevity, no further description is given here.

[0098] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method described in the above embodiment is completed.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A remote sensing image segmentation method based on a visual large model, characterized in that: include: Obtain the remote sensing image to be segmented and its corresponding geospatial information data, and perform corresponding preprocessing; For the remote sensing image to be segmented, based on the vector data of the target object in the corresponding geographic spatial information data, a plurality of sampling points are obtained by sequentially performing rasterization processing and data point sampling processing; and point prompts of the remote sensing image are generated based on the sampling points; The point prompt is generated by: obtaining a binary image through rasterization based on the vector data of the target object in the geographic spatial information data corresponding to the remote sensing image; obtaining a number of sampling points through grid sampling based on the binary image; and generating the point prompt based on the corresponding position pixels of the sampling points on the remote sensing image; The grid sampling method is specifically as follows: rasterizing the vector data of the target object to obtain corresponding grid labels; dividing the grid labels based on a grid of a preset size; for each grid, randomly sampling from discrete points that fall within the grid and are the target to be segmented to obtain a sampled point set; For the remote sensing image to be segmented, based on the key geographic coordinate data of the target object in the corresponding geographic spatial information data, a target object bounding box is constructed, and the bounding box is used as a frame prompt of the remote sensing image; The generation of the frame prompt is specifically as follows: based on the key geographic coordinate data of the target object in the corresponding geographic spatial information data of the remote sensing image, the corresponding image coordinates in the remote sensing image are obtained through coordinate conversion; and the frame prompt in the remote sensing image is formed based on the obtained image coordinates; The specific processing process of generating the box prompt is: GeoJOSN data format conversion; coordinate acquisition and conversion of key points; box prompt generation; For the remote sensing image to be segmented, based on the vector data of the target object in the corresponding geospatial information data, it is processed by rasterization and expansion algorithm in sequence to obtain rasterized labels as mask prompts; The obtained point prompt, frame prompt and mask prompt are used as prompt information of the remote sensing image to be segmented; Based on the prompt information and the remote sensing image to be segmented, the segmentation result of the remote sensing image is obtained by using the pre-trained SAM visual large model.

2. A remote sensing image segmentation method based on a visual large model as claimed in claim 1, characterized in that: A frame prompt in a remote sensing image is formed based on the obtained image coordinates, and the following optimization process is performed on the formed frame prompt: the frame prompt is used as an initial prompt frame, wherein the coordinates of the upper left corner point of the initial prompt frame are used as the starting point, and the coordinates of the lower right corner point are used as the end point; the horizontal distance and the vertical distance between the starting point and the end point are calculated respectively, and a preset value is determined as the short side length of the optimized frame by determining the shortest side of the initial prompt frame; the long side length of the optimized frame is calculated according to the aspect ratio of the initial prompt frame; an offset value is preset for the long side length of the optimized frame to correct the quality deviation of the vector data, and multiple sub-prompt frames covering the entire target object area are generated without exceeding the original image range; and the multiple sub-prompt frames are used as frame prompts for the target object in the remote sensing image.

3. A remote sensing image segmentation method based on a visual large model as claimed in claim 1, characterized in that: Based on the prompt information and the remote sensing image to be segmented, the pre-trained SAM visual large model is used to obtain the segmentation result of the remote sensing image, specifically: the binary images corresponding to the point prompt, box prompt and mask prompt are added pixel by pixel to obtain a fused prompt, and the fused prompt is used as the input prompt information of the SAM visual large model to obtain the segmentation result of the remote sensing image.

4. The remote sensing image segmentation method based on a visual large model as claimed in claim 1, characterized in that: The key geographic coordinate data includes the starting point and end point coordinates of the road.

5. A remote sensing image segmentation system based on a large visual model, characterized in that: include: A data acquisition unit, which is used to acquire the remote sensing image to be segmented and its corresponding geospatial information data, and perform corresponding preprocessing; A point prompt generating unit is used for obtaining a plurality of sampling points by sequentially performing rasterization processing and data point sampling processing on the remote sensing image to be segmented based on the vector data of the target object in the corresponding geographic spatial information data; and generating point prompts of the remote sensing image based on the sampling points; The point prompt is generated by: obtaining a binary image through rasterization based on the vector data of the target object in the geographic spatial information data corresponding to the remote sensing image; obtaining a number of sampling points through grid sampling based on the binary image; and generating the point prompt based on the corresponding position pixels of the sampling points on the remote sensing image; The grid sampling method is specifically as follows: rasterizing the vector data of the target object to obtain corresponding grid labels; dividing the grid labels based on a grid of a preset size; for each grid, randomly sampling from discrete points that fall within the grid and are the target to be segmented to obtain a sampled point set; A frame prompt generating unit is used to construct a target object bounding box based on key geographic coordinate data of the target object in the corresponding geographic spatial information data of the remote sensing image to be segmented, and use the bounding box as a frame prompt of the remote sensing image; The generation of the frame prompt is specifically as follows: based on the key geographic coordinate data of the target object in the corresponding geographic spatial information data of the remote sensing image, the corresponding image coordinates in the remote sensing image are obtained through coordinate conversion; and the frame prompt in the remote sensing image is formed based on the obtained image coordinates; The specific processing process of generating the box prompt is: GeoJOSN data format conversion; coordinate acquisition and conversion of key points; box prompt generation; A mask hint generating unit is used for sequentially performing rasterization processing and dilation algorithm processing on the remote sensing image to be segmented based on the vector data of the target object in the corresponding geographic spatial information data, so as to obtain a rasterized label as a mask hint; A prompt information generating unit, which is used to obtain the point prompt, the frame prompt and the mask prompt as prompt information of the remote sensing image to be segmented; The image segmentation unit is used to obtain the segmentation result of the remote sensing image based on the prompt information and the remote sensing image to be segmented by using the pre-trained SAM visual large model.

6. An electronic device comprising a memory, a processor and a computer program stored and running on the memory, characterized in that: When the processor executes the program, it implements a remote sensing image segmentation method based on a visual large model as described in any one of claims 1-4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a remote sensing image segmentation method based on a visual large model as described in any one of claims 1 to 4 is implemented.

Images