Semantic segmentation method and system for remote sensing image multi-level mask classification optimization

Abstract

The present invention relates to the field of image processing, and discloses a semantic segmentation method for multi-level mask classification optimization of remote sensing images, comprising the following steps: acquiring remote sensing images and performing preprocessing to obtain multiple cropped sub-blocks; using the mask generation function of the SAM network to extract the potential target area of ​​each cropped sub-block into multiple independent masks; calculating the overlapping coverage of the masks by constructing a spatial index; merging the masks according to the overlapping coverage; calculating the spatial adjacency and category consistency between masks, and performing secondary merging of masks belonging to the same semantic category and adjacent to each other to complete the segmentation of the remote sensing image. The present invention innovatively combines mask generation, preliminary merging, semantic classification and secondary merging to form a multi-stage segmentation process, in which each stage is independent of each other and promotes each other, thereby improving the semantic consistency of the segmentation results and the integrity of object recognition.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a semantic segmentation method and system for optimizing multi-level mask classification of remote sensing images. Background Art

[0002] With the rapid development of remote sensing satellite technology and drone technology, the acquisition of remote sensing images has become increasingly convenient and efficient. Remote sensing images not only play an important role in the fields of geographic information systems (GIS), urban planning, environmental protection, etc., but also show great application potential in disaster monitoring, agricultural management, military reconnaissance, etc. Through remote sensing images, we can obtain large-scale, high-resolution surface information, which is of great significance for scientific research and practical applications. However, how to quickly and accurately extract useful information from massive remote sensing images is still a very challenging task.

[0003] Traditional remote sensing image processing methods mainly include pixel-based methods and region-based methods. The pixel-based method classifies by analyzing the spectral characteristics of each pixel. This method is simple and intuitive, but it is easily affected by noise and cannot effectively handle complex ground structures. The region-based method divides the image into several regions, and then extracts and classifies features for each region. This method improves the accuracy of segmentation and classification to a certain extent, but the computational complexity is high, and it depends on manually set parameters, and its adaptability and robustness are poor.

[0004] In recent years, the development of deep learning technology has brought new opportunities for remote sensing image processing. Deep learning models such as convolutional neural networks (CNNs) have achieved remarkable results in tasks such as image classification, object detection, and semantic segmentation. For example, models such as U-Net and Mask R-CNN have performed well in medical image segmentation and natural scene segmentation, but these models still face some challenges when processing remote sensing images. First, remote sensing images have large scale variations and many types of objects, and the model needs to have strong generalization capabilities. Secondly, high-quality annotated data is expensive and time-consuming to obtain, which limits the training and optimization of the model. In addition, the existing deep learning models consume huge computing resources when processing large-scale data, and it is difficult to meet the requirements of real-time and high efficiency. Summary of the invention

[0005] The main purpose of the present invention is to solve the technical problem in the prior art that the scale of remote sensing images varies greatly and the types of objects are many, resulting in poor model generalization ability. The semantic segmentation method of remote sensing images with multi-level mask classification optimization includes the following steps:

[0006] Acquire remote sensing images and perform preprocessing to obtain multiple cropped sub-blocks;

[0007] Using the mask generation function of the SAM network, the potential target area of ​​each cropped sub-block is extracted into multiple independent masks;

[0008] By building a spatial index, the overlap coverage of the mask is calculated;

[0009] Merge masks based on overlapping coverage;

[0010] The spatial adjacency and category consistency between masks are calculated, and the adjacent masks belonging to the same semantic category are merged twice to complete the segmentation of the remote sensing image.

[0011] As a preferred technical solution, remote sensing images are acquired and preprocessed to obtain multiple cropped sub-blocks, including:

[0012] Standardize the resolution of remote sensing images;

[0013] Extract RGB bands from remote sensing images;

[0014] Crop the remote sensing image into sub-blocks of fixed size;

[0015] For the part of the remote sensing image whose edge is less than the cropping size, black filling is used to expand the image area.

[0016] As a preferred technical solution, by constructing a spatial index, the overlap coverage of the mask is calculated, including:

[0017] Based on the quadtree structure, the two-dimensional space of the remote sensing image is recursively divided into four sub-areas, with one node in each area;

[0018] Assign each mask to the corresponding node according to its spatial position to ensure that the mask distribution of the same node is regional and local;

[0019] Initialize a quadtree root node to represent the entire image space;

[0020] Traverse the mask set, use the bounding rectangle of each mask as the key field of the index, and recursively divide it into the leaf nodes of the quadtree: if the bounding rectangle of the mask is completely within a sub-region, it is assigned to the corresponding sub-node; if the mask spans multiple sub-regions, it is stored in the current node to avoid information loss;

[0021] If the spatial area of ​​the child node is smaller than the preset minimum mask area, the division will stop;

[0022] Overlap coverage calculation, the overlapping area coverage formula is:

[0023]

[0024] in:

[0025] Area(A∩B): The intersection area of ​​masks A and B;

[0026] min(Area(A),Area(B)): The area of ​​the smaller of the two masks.

[0027] As a preferred technical solution, the masks are merged according to the overlapping coverage, including:

[0028] Starting from the root node of the quadtree, merge the stored masks; recursively process all levels from the root node to the leaf node;

[0029] If the current node is a leaf node, perform the merge operation directly in the node; if the current node is not a leaf node, recursively call the merge operation of the child node;

[0030] The merging logic is:

[0031] Input data: mask set m1, m2, ..., m in leaf nodes k ;

[0032] Processing flow:

[0033] A. Traverse the mask pairs in each layer of nodes:

[0034] For each pair of masks (m i ,m j ), calculate its overlapping coverage ratio OverLap Ratio;

[0035] Determine whether the merging conditions are met:

[0036] If the Overlap Ratio is greater than the overlap coverage threshold, the masks are merged, and the bounding rectangle of the merged new mask is the union of the two masks; the merged area is the sum of the areas of the two masks minus the area of ​​the intersection.

[0037] B. Update the mask in the node:

[0038] Delete the original mask m i and m j , insert a new merge mask.

[0039] C. Repeat this operation until all mask pairs have been processed.

[0040] As a preferred technical solution, the method for determining category consistency includes:

[0041] Build a semantic classification model;

[0042] The semantic category output by the semantic classification model is used as an additional attribute of the mask and updated to the mask data structure after the preliminary merger. Each mask records the following information: mask area, bounding rectangle, area, and semantic category;

[0043] As a preferred technical solution, the method for determining spatial adjacency includes:

[0044] If the intersection of the two masks is not empty, the masks are directly determined to be adjacent, and the distance between the masks is defined as 0, and no further calculation is required; if there is no intersection, the shortest distance between the boundary points is calculated. The specific steps are:

[0045] Enclosing rectangle filter:

[0046] First, determine whether the bounding rectangles of the two masks intersect;

[0047] If they intersect, the shortest distance between the boundary points is directly calculated;

[0048] If they do not intersect, proceed to the next step of judgment;

[0049] Determine whether the closest distance between two bounding rectangles is less than the threshold d thresh :

[0050] If it is less than the threshold, the closest distance of the boundary point is calculated; the calculation formula for the closest distance between two masks is:

[0051]

[0052] B1, B2 are the boundary point sets of the two masks respectively; ||p1-p2|| is the Euclidean distance between the boundary points;

[0053] If it is greater than the threshold, it is determined to be non-adjacent and no further calculation is required.

[0054] Candidate point optimization:

[0055] When calculating the closest distance to a boundary point, the candidate point set is filtered by combining the bounding rectangle, and the distance is calculated only for potential adjacent mask boundary points. If the closest distance to the boundary point is less than d thresh If they are adjacent, they are not adjacent.

[0056] As a preferred technical solution, masks belonging to the same semantic category and adjacent to each other are merged twice to complete the segmentation of remote sensing images, including:

[0057] The input mask set is a spatial index data structure constructed in the preliminary mask merging, where each mask contains the following attributes: mask region, bounding rectangle, area and semantic category;

[0058] Hierarchical recursive processing

[0059] Based on the spatial index, starting from the root node, the following steps are performed on the mask set of each node: determine whether it is a leaf node. If it is a leaf node, directly merge the masks in the node; if it is not a leaf node, recursively call the child node to process the masks in the child node and across regions;

[0060] Leaf node mask merging

[0061] For the mask set in the leaf node, execute according to the following process:

[0062] Traverse the mask pairs: for each pair of masks M in the node i ,M j , semantic consistency and spatial proximity determination are performed in sequence;

[0063] Perform merging: If the two masks meet the merging conditions, generate a new mask M new , replace the original mask pair M i ,M j ;

[0064] Update the mask set: insert the newly generated mask into the node and delete the merged mask;

[0065] Parent node and child node mask merge

[0066] For the cross-region mask stored in the parent node, the following process is used:

[0067] Cross-region mask determination: Calculate the overlap coverage or the closest distance between the parent node mask and the child node mask. If the conditions are met, perform the merge.

[0068] Update the mask set of the parent node: delete the original mask to be merged and insert the new mask into the parent node;

[0069] Termination Condition

[0070] When all masks in a node do not satisfy the merging condition, the recursion stops.

[0071] The second aspect of the present invention provides a semantic segmentation system for remote sensing image multi-level mask classification optimization, comprising:

[0072] A cropping unit acquires and preprocesses remote sensing images to obtain multiple cropped sub-blocks;

[0073] The mask generation unit uses the mask generation function of the SAM network to extract the potential target area of ​​each cropped sub-block into multiple independent masks;

[0074] An overlap coverage calculation unit calculates the overlap coverage of the mask by constructing a spatial index;

[0075] A first merging unit merges the masks according to the overlapping coverage ratio;

[0076] The second merging unit calculates the spatial adjacency and category consistency between masks, and merges the adjacent masks belonging to the same semantic category for a second time to complete the segmentation of the remote sensing image.

[0077] The third aspect of the present invention provides an electronic device, comprising: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected via lines; the at least one processor calls the instructions in the memory so that the electronic device executes the above-mentioned semantic segmentation method for optimizing multi-level mask classification of remote sensing images based on spatial indexing.

[0078] The fourth aspect of the present invention provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, the computer executes the semantic segmentation method for multi-level mask classification optimization of remote sensing images based on spatial index as described above.

[0079] The present invention has the following beneficial effects:

[0080] 1. Integrated optimization of multi-stage segmentation process. The present invention innovatively combines mask generation, preliminary merging, semantic classification and secondary merging to form a multi-stage segmentation process. Each stage is independent of each other and promotes each other. The preliminary merging optimizes the mask distribution and provides high-quality input for semantic classification. The secondary merging driven by the classification results introduces the semantic information of the target object into the segmentation result, thereby improving the semantic consistency of the segmentation result and the integrity of object recognition.

[0081] 2. Significantly reduce the dependence on annotated data. Deep learning semantic segmentation models usually require a large amount of high-quality mask annotated data for training. In the field of remote sensing images, annotation is costly and time-consuming. This invention introduces a multi-stage segmentation optimization process, uses a general segmentation model to generate an initial mask and performs multi-level optimization, which greatly reduces the demand for large-scale annotated data, reduces training costs and shortens the development cycle.

[0082] 3. Efficiently process a large amount of mask data. For a large amount of mask data generated by the general segmentation model, the present invention uses spatial indexing and recursive optimization algorithms, combined with overlapping coverage, the closest distance of boundary points, and semantic consistency rules, to achieve efficient processing of mask merging. The optimized process significantly reduces the computational complexity of mask merging and improves processing efficiency.

[0083] 4. Adapting to the scale changes and diversity of objects in remote sensing images, the segmentation method of the present invention is based on spatial indexing and semantic classification optimization, and can effectively adapt to complex scenes with large scale changes and many types of objects in remote sensing images. Without the need for a large amount of labeled data, the accuracy and robustness of the segmentation are guaranteed, making the present invention more economical and practical in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 It is a typical remote sensing image.

[0085] Figure 2 The algorithm flow from the original image input to the initial mask merging;

[0086] Figure 3 The process of optimizing the algorithm for mask secondary merging under semantic classification constraints. DETAILED DESCRIPTION

[0087] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0088] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1-3 The first embodiment of the semantic segmentation method of multi-level mask classification optimization of remote sensing images based on spatial index in the embodiment of the present invention includes:

[0089] Acquire remote sensing images and perform preprocessing to obtain multiple cropped sub-blocks;

[0090] Using the mask generation function of the SAM network, the potential target area of ​​each cropped sub-block is extracted into multiple independent masks;

[0091] By building a spatial index, the overlap coverage of the mask is calculated;

[0092] Merge masks based on overlapping coverage;

[0093] The spatial adjacency and category consistency between masks are calculated, and the adjacent masks belonging to the same semantic category are merged twice to complete the segmentation of the remote sensing image.

[0094] The details are as follows:

[0095] 1. Remote sensing image input and preprocessing

[0096] To adapt to the subsequent semantic segmentation process, the present invention first standardizes the original remote sensing image data to generate RGB image data that meets the input requirements of the semantic segmentation model. The preprocessing process includes but is not limited to standardization of spatial resolution and fixed-size image cropping.

[0097] 1. Data source and formatting

[0098] The present invention is applicable to remote sensing image data from various sources, which are usually stored in standardized formats (such as GeoTIFF, Shapefile, etc.) and contain multiple spectral bands. The present invention only extracts the RGB band and ignores the non-visible light band to ensure the consistency and compatibility of the input data.

[0099] First, the present invention uses remote sensing image processing tools to load images and extract target bands. To ensure the consistency of images in subsequent segmentation model training and reasoning, the present invention uses a resampling algorithm to standardize the resolution of images. The specific method includes: adjusting the image resolution to a resolution consistent with the model training data (such as 0.5 meters / pixel); applying a resampling algorithm (such as bilinear interpolation or nearest neighbor interpolation algorithm) to generate pixel values ​​and construct an image with consistent resolution.

[0100] 2. Cutting

[0101] After completing the image resolution standardization and RGB band extraction, the present invention crops the image into fixed-size sub-blocks to adapt to the input size of the subsequent model. The specific steps are as follows:

[0102] Crop size setting: Select an appropriate crop size based on the task requirements and spatial resolution of the remote sensing image. The present invention uses a crop sub-block size of 2048×1024 pixels to maintain consistency during model training and inference.

[0103] Sliding window cropping method: Use a sliding window to crop the image. The size of the sliding window is the same as the cropping size, and the step length of each slide is 80% of the cropping size in width and height (that is, there is a 20% overlap between adjacent cropping blocks). The advantages of this method include: retaining the ground object information near the cropping boundary, reducing the target loss caused by block segmentation; adapting to the continuity of ground object distribution and improving segmentation accuracy.

[0104] Processing of edge areas: For the portion of the image edge that is not large enough for cropping, the present invention uses black filling to expand the image area to ensure that all sub-blocks are of the same size and meet the model input requirements.

[0105] Through the above input and preprocessing steps, the present invention successfully realizes the standardization and cropping of multi-source remote sensing image data. The output fixed-size sub-blocks provide high-quality input data for the subsequent segmentation model, while minimizing the loss of ground object information in the cropping process.

[0106] (II) Mask generation based on SAM network

[0107] After completing the input and preprocessing of the remote sensing image, the present invention uses Segment Anything Model (SAM) to generate masks for the cropped remote sensing image sub-blocks. SAM is a general segmentation model that can generate high-quality segmentation masks under different input prompts. The present invention uses the mask generation function of the SAM network to extract the potential target area of ​​each cropped sub-block into multiple independent masks, providing data support for subsequent mask classification optimization.

[0108] In the process of mask generation, the present invention ensures the quality and adaptability of the segmentation results by reasonably setting key parameters. The number of sampling points is a key parameter. The image needs to be sampled more densely to ensure comprehensive coverage of multi-target scenes and effectively reduce the problem of missed detection. For the quality score threshold of the generated mask, in order to ensure that the generated mask can cover all potential targets in the image, the present invention appropriately reduces the threshold of the mask quality score, thereby improving the ability to capture boundary targets and complex scene targets. The minimum area of ​​the mask is an important parameter in the mask generation process. The absolute size of the mask area is closely related to the spatial resolution of the remote sensing image. The present invention sets the minimum mask area to 50 to 100 pixels (i.e., about 12.5 to 25 square meters) for images with a spatial resolution of 0.5 meters / pixel. This setting can effectively extract isolated vegetation, small buildings and other small targets, and can also be flexibly adjusted according to task requirements. Through reasonable parameter configuration, high-quality ground object masks can be quickly generated in complex remote sensing scenes. This method makes full use of the segmentation capability of the SAM model and improves its adaptability to small targets and boundary targets through parameter optimization, providing support for achieving high efficiency and accuracy in remote sensing image segmentation.

[0109] 3. Preliminary mask merging driven by spatial index and overlap coverage

[0110] The masks generated based on the SAM network are characterized by large number, complex distribution and lack of semantic information (such as roads, water bodies, vegetation, etc.), which makes them difficult to be used directly for classification or further processing. In response to this problem, the present invention proposes a preliminary mask merging method based on spatial index and overlapping coverage, which effectively optimizes the number and distribution structure of masks. The present invention recursively divides the two-dimensional space of the remote sensing image into multiple sub-regions by constructing a spatial index, quickly locates the mask areas that may overlap, and combines the overlapping coverage as a judgment criterion to perform efficient mask merging operations. While reducing the computational complexity, this method optimizes the spatial distribution of the mask, providing a structured, high-quality data foundation for subsequent semantic classification and processing.

[0111] 1. Spatial index construction

[0112] (1) Introduction of Quadtree

[0113] A. The spatial index is based on a quadtree structure, which recursively divides the two-dimensional space of the remote sensing image into four sub-areas, with one node in each area;

[0114] B. Assign each mask to the corresponding node according to its spatial position (i.e., the circumscribed rectangle) to ensure that the mask distribution of the same node is regional and local.

[0115] (2) Construction process

[0116] A. Initialize a quadtree root node to represent the entire image space;

[0117] B. Traverse the mask set, use the bounding rectangle of each mask as the key field of the index, and recursively divide it into the leaf nodes of the quadtree: if the bounding rectangle of the mask is completely within a sub-region, it is assigned to the corresponding child node; if the mask spans multiple sub-regions, it is stored in the current node to avoid information loss.

[0118] C. If the spatial area of ​​the child node is smaller than the preset minimum mask area, stop dividing

[0119] (3) Calculation of overlapping coverage

[0120] A. Overlap Area Coverage Formula

[0121]

[0122] in:

[0123] Area(A∩B): The intersection area of ​​masks A and B;

[0124] min(Area(A),Area(B)): The area of ​​the smaller of the two masks.

[0125] B. Threshold Setting

[0126] The present invention suggests setting the threshold of the overlap coverage ratio to CoverageThresh=0.7 to ensure that the smaller mask is merged when most of it is covered.

[0127] (4) Preliminary mask merging process

[0128] A. Initial call

[0129] Starting from the root node of the quadtree, merge the stored masks; recursively process all levels from the root node to the leaf node.

[0130] B. Determine whether it is a leaf node

[0131] If the current node is a leaf node, the merge operation is performed directly in the node; if the current node is not a leaf node, the merge operation of the child node is recursively called.

[0132] C. Merge Logic

[0133] Input data: mask set m1, m2, ..., m in leaf nodes k .

[0134] Processing flow:

[0135] a. Traverse the mask pairs in each layer of nodes:

[0136] For each pair of masks (m i ,m j ), calculate its overlapping coverage ratio OverLap Ratio;

[0137] b. Determine whether the merging conditions are met:

[0138] If Overlap Ratio>CoverageThresh, the mask is merged and the bounding rectangle of the new mask is the union of the two. The area after the merger is the sum of the areas of the two minus the area of ​​the intersection.

[0139] c. Update the mask in the node:

[0140] Delete the original mask m i and m j , insert a new merge mask.

[0141] d. Repeat this operation until all mask pairs have been processed.

[0142] D. Special processing of parent nodes and child nodes

[0143] The masks stored in the parent node are masks that span multiple sub-regions. These masks may intersect with the masks in the child nodes. The overlapping coverage of the masks of the parent node and the child node is calculated respectively, and the merging operation is performed. If the mask in the parent node is merged with the mask in the child node, the original mask is deleted and the new mask is inserted into the corresponding parent node.

[0144] (IV) Mask secondary merging under semantic classification constraints

[0145] The present invention proposes a mask secondary merging method based on semantic classification constraints. On the basis of the initial mask merging, the deep learning classification model is used to semantically classify the masks, and the masks belonging to the same semantic category and adjacent are secondary merged through spatial adjacency and category consistency rules. This method uses a semantic classification model (such as YOLOv11) to semantically predict the mask, introduces object category information into the mask, and optimizes the mask distribution using adjacency and category constraints. This method effectively improves the spatial integrity of the mask and the accuracy of object recognition, and provides high-precision support for the semantic segmentation results of remote sensing images. Technical solution steps:

[0146] 1. Mask semantic classification

[0147] (1) Training and preparation of semantic classification model

[0148] A. Data Category Definition

[0149] The present invention defines training data categories according to the requirements of remote sensing image tasks, including common landforms such as roads, water bodies, buildings, vegetation, and farmland, etc. The category settings can be flexibly expanded or adjusted according to specific application scenarios to meet different analysis objectives.

[0150] B. Data Preprocessing

[0151] The training data of the classification model is generated based on the preliminary merged mask. To ensure the uniformity and representativeness of the input data, the present invention uses the following rules to generate the classification interest area:

[0152] a. Expansion rule: Expand the mask boundary outward by 30% to form a region of interest containing the target features, ensuring that the boundary target information is not lost.

[0153] b. Cropping rule: crop the image block with the bounding rectangle of the area of ​​interest, and fill the non-interest area with a black background to reduce interference to the classification model.

[0154] The final generated standardized image patches are used as input to the classification model to improve data quality and model performance.

[0155] C. Selection of training samples

[0156] In order to improve the generalization ability of the semantic classification model, representative samples are selected for annotation and training. The number of samples selected is usually about 0.5% of the total number of masks to ensure efficient training while covering diverse features.

[0157] D. Model Training

[0158] The labeled training samples are used in combination with data enhancement techniques (rotation, scaling, and flipping) to train the semantic classification model and optimize its classification performance.

[0159] (2) Application of semantic classification model

[0160] A. Input data

[0161] For each mask after preliminary merging, the image block of the region of interest is cropped according to the preprocessing rules (the non-interest region within the bounding rectangle is filled with a black background). The cropped standardized image block is used as the input of the semantic classification model.

[0162] B. Classification Prediction

[0163] The cropped image blocks are input into the trained semantic classification model, and the model outputs the semantic category of the mask (such as road, water body, etc.). The classification results are used for subsequent mask merging optimization.

[0164] (3) Semantic category update

[0165] A. Classification result storage

[0166] The semantic category output by the semantic classification model is used as an additional attribute of the mask and updated to the mask data structure after the preliminary merger. Each mask records the following information: mask area, bounding rectangle, area, and semantic category.

[0167] B. Subsequent use of classification results

[0168] The updated mask dataset will serve as the input basis for subsequent adjacency judgment and secondary mask merging under semantic classification constraints to ensure the semantic consistency of the segmentation results and the integrity of object recognition.

[0169] 2. Mask adjacency judgment

[0170] (1) Boundary extraction of complex mask shapes

[0171] In order to accurately describe the complex mask shape, the present invention uses a boundary point extraction algorithm (such as findContours) to extract the outer contour point set of the mask area to form a boundary point set B. The boundary point set can accurately reflect the spatial characteristics of the mask and provide a reliable geometric basis for adjacency determination.

[0172] (2) Rapid detection of overlap

[0173] In order to improve the efficiency of adjacency determination, the present invention gives priority to determining the overlap between masks. If the intersection of two masks is not empty, the masks are directly determined to be adjacent, and the distance between the masks is defined as 0, without further calculation. If there is no intersection, the next step is to calculate the closest distance between the boundary points.

[0174] (3) Calculation of the shortest distance between boundary points

[0175] For non-overlapping masks, the proximity of the masks is determined by the closest distance of the boundary points. The closest distance between two masks is defined as:

[0176]

[0177] B1 and B2 are the boundary point sets of the two masks respectively; ||p1-p2|| is the Euclidean distance between the boundary points.

[0178] (4) Efficient distance calculation method

[0179] In order to further optimize the computational efficiency of complex mask shapes, the present invention adopts the following optimization scheme:

[0180] A. Filtering of bounding rectangles:

[0181] a. First determine whether the bounding rectangles of the two masks intersect;

[0182] i. If they intersect, directly calculate the shortest distance between the boundary points;

[0183] ii. If they do not intersect, proceed to the next step.

[0184] b. Determine whether the closest distance between two circumscribed rectangles is less than the threshold d thresh :

[0185] i. If it is less than the threshold, calculate the closest distance to the boundary point;

[0186] ii. If it is greater than the threshold, it is determined to be non-adjacent and no further calculation is required.

[0187] B. Candidate point optimization:

[0188] When calculating the closest distance to a boundary point, the candidate point set is filtered by combining the bounding rectangle, and the distance is calculated only for potential adjacent mask boundary points. If the closest distance to the boundary point is less than d thresh

[0189] If they are adjacent, they are not adjacent.

[0190] (5) Adjacency Determination Rules

[0191] Based on the above steps, and considering the overlap of masks and the distance calculation method, the adjacency rule of the present invention is as follows: overlap is given priority, if the masks overlap, they are directly judged as adjacent; distance calculation, if there is no overlap, it is judged by the above distance calculation method.

[0192] 3. Mask secondary merging

[0193] (1) Merger rules

[0194] A. Semantic Classification Consistency

[0195] Only masks with the same semantic category are allowed to be merged. The semantic consistency is determined by the category label C output by the semantic classification model. i and C j Judgement: C i =C j .

[0196] B. Spatial Proximity

[0197] On the basis of the consistency of semantic classification, the adjacency between masks is further judged and determined according to the adjacency judgment rules.

[0198] C. Merge Logic

[0199] For the mask M that satisfies the semantic consistency and adjacency conditions i and M j , merge to generate a new mask M new : Mask areas are merged, bounding rectangles are updated, and area is updated.

[0200] (2) Merging steps

[0201] A. Initialization

[0202] The input mask set is a spatial index data structure constructed in the preliminary mask merging, where each mask contains the following attributes: mask region, bounding rectangle, area and semantic category.

[0203] B. Hierarchical recursive processing

[0204] Starting from the root node based on the spatial index, the following steps are performed on the mask set of each node: determine whether it is a leaf node. If it is a leaf node, directly merge the masks within the node; if it is not a leaf node, recursively call the child node to process the masks within the child node and across regions.

[0205] C. Leaf node mask merging

[0206] For the mask set in the leaf node, execute according to the following process:

[0207] a. Traverse the mask pairs: for each pair of masks M in the node i,M j , semantic consistency and spatial proximity are determined in sequence.

[0208] b. Perform merging: If the two masks meet the merging conditions, generate a new mask M new , replace the original mask pair M i ,M j .

[0209] c. Update the mask set: insert the newly generated mask into the node and delete the merged mask.

[0210] D. Parent node and child node mask merging

[0211] For the cross-region mask stored in the parent node, the following process is used:

[0212] a. Cross-region mask determination: Calculate the overlap coverage or the closest distance between the parent node mask and the child node mask. If the conditions are met, perform the merge;

[0213] b. Update the mask set of the parent node: delete the original mask that was merged and insert the new mask into the parent node.

[0214] E. Termination Conditions

[0215] When all masks in a node do not satisfy the merging condition, the recursion stops.

[0216] Through the above steps, the present invention realizes efficient processing of secondary mask merging, especially adding semantic information to the initially merged masks, which significantly improves the semantic consistency and spatial integrity of the segmentation results.

[0217] (V) Output of Remote Sensing Image Semantic Segmentation Results

[0218] 1. Output data structure design

[0219] (1) Data structure of segmentation results

[0220] The remote sensing image semantic segmentation result of the present invention includes the following information: mask area, binary image representation of the mask, describing the area of ​​the target object; circumscribed rectangle, circumscribed rectangle coordinates of the mask, convenient for quickly locating the target area; semantic category, semantic label predicted by the classification model, describing the category information of the target object (such as road, water body, etc.); area information, pixel area of ​​the mask, convenient for counting the size of the target object; spatial index information, recording the storage position of the mask in the quadtree result, convenient for subsequent retrieval and analysis.

[0221] (2) Data format:

[0222] The semantic segmentation results of the present invention can be exported in multiple formats to adapt to different subsequent analysis tasks and software tools, such as GeoJson format, binary image format, vector format Shapefile and tabular data CSV, recording the information of each mask.

[0223] 2. Output visualization

[0224] The segmentation results are visualized in the form of an overlay map combined with the original remote sensing image, which intuitively displays the target area and its semantic classification results. Different color codes are used for masks of different semantic categories. In addition, it also supports comparative display of segmentation results and original images, and can filter specific categories for display to improve the readability of the segmentation results.

[0225] 3. Data storage and export

[0226] The present invention supports storing the segmentation results as a structured directory, which includes the following contents: original image file, which stores the input image; segmentation result file, which stores the segmentation results such as mask area and semantic information;

[0227] Statistical information file, recording the target object’s category, quantity, area and other attribute information.

[0228] Customized export interface, providing standardized API, customizable export path, format and category filtering: export GeoJSON, Shapefile and other files for use in GIS system; export segmentation overlay visualization images; export segmentation statistical data CSV file.

[0229] Through the semantic segmentation result output of the present invention, it is possible to achieve efficient display and storage of results while meeting the requirements of various analysis tasks, providing strong support for the semantic segmentation application of remote sensing images.

[0230] The above describes the semantic segmentation method of multi-level mask classification optimization of remote sensing images based on spatial index in the embodiment of the present invention. The following describes the semantic segmentation device of multi-level mask classification optimization of remote sensing images based on spatial index in the embodiment of the present invention:

[0231] An embodiment of the present invention also provides an electronic device, which may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) (for example, one or more processors) and memories, and one or more storage media for storing applications or data (for example, one or more mass storage devices). Among them, the memory and the storage medium may be short-term storage or permanent storage. The program stored in the storage medium may include one or more modules, each module may include a series of instruction operations in the electronic device. Furthermore, the processor may be configured to communicate with the storage medium and execute a series of instruction operations in the storage medium on the electronic device.

[0232] The electronic device may also include one or more power supplies, one or more wired or wireless network interfaces, one or more input and output interfaces, and / or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that the electronic device structure in this embodiment does not constitute a limitation on the electronic device, and may include more or fewer components, or combine certain components, or arrange components differently.

[0233] The structure of an electronic device provided by an embodiment of the present invention may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) (for example, one or more processors) and memories, and one or more storage media for storing applications or data (for example, one or more mass storage devices). Among them, the memories and storage media may be short-term storage or permanent storage. The program stored in the storage medium may include one or more modules, each module may include a series of instruction operations in the electronic device. Furthermore, the processor may be configured to communicate with the storage medium and execute a series of instruction operations in the storage medium on the electronic device.

[0234] The electronic device may also include one or more power supplies, one or more wired or wireless network interfaces, one or more input and output interfaces, and / or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that the electronic device structure does not constitute a limitation on the electronic device, and may include more or less components than the aforementioned, or combine certain components, or arrange the components differently.

[0235] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computer, the computer executes the steps of the aforementioned method.

[0236] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0237] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0238] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semantic segmentation method for remote sensing image multi-level mask classification optimization, characterized in that: The following steps are involved: Acquire remote sensing images and perform preprocessing to obtain multiple cropped sub-blocks; Using the mask generation function of the SAM network, the potential target area of ​​each cropped sub-block is extracted into multiple independent masks; By building a spatial index, the overlap coverage of the mask is calculated; Merge masks based on overlapping coverage; The spatial adjacency and category consistency between masks are calculated, and the adjacent masks belonging to the same semantic category are merged twice to complete the segmentation of the remote sensing image.

2. The semantic segmentation method for remote sensing image multi-level mask classification optimization according to claim 1 is characterized in that: The remote sensing image is acquired and preprocessed to obtain a plurality of cropped sub-blocks, including: Standardize the resolution of remote sensing images; Extract RGB bands from remote sensing images; Crop the remote sensing image into sub-blocks of fixed size; For the part of the remote sensing image whose edge is less than the cropping size, black filling is used to expand the image area.

3. The semantic segmentation method for remote sensing image multi-level mask classification optimization according to claim 1, characterized in that: The calculation of the overlap coverage of the mask by constructing a spatial index includes: Based on the quadtree structure, the two-dimensional space of the remote sensing image is recursively divided into four sub-areas, with one node in each area; Assign each mask to the corresponding node according to its spatial position to ensure that the mask distribution of the same node is regional and local; Initialize a quadtree root node to represent the entire image space; Traverse the mask set, use the bounding rectangle of each mask as the key field of the index, and recursively divide it into the leaf nodes of the quadtree: if the bounding rectangle of the mask is completely within a sub-region, it is assigned to the corresponding sub-node; if the mask spans multiple sub-regions, it is stored in the current node to avoid information loss; If the spatial area of ​​the child node is smaller than the preset minimum mask area, the division will stop; Overlap coverage calculation, the overlapping area coverage formula is: in: Area(A∩B): The intersection area of ​​masks A and B; min(Area(A),Area(B)): The area of ​​the smaller of the two masks.

4. The semantic segmentation method for remote sensing image multi-level mask classification optimization according to claim 1, characterized in that: The merging of masks according to the overlapping coverage ratio comprises: Starting from the root node of the quadtree, merge the stored masks; recursively process all levels from the root node to the leaf node; If the current node is a leaf node, perform the merge operation directly in the node; if the current node is not a leaf node, recursively call the merge operation of the child node; The merging logic is: Input data: mask set m1, m2, ..., m in leaf nodes k ; Processing flow: A. Traverse the mask pairs in each layer of nodes: For each pair of masks (m i ,m j ), calculate its overlapping coverage ratio OverLap Ratio; Determine whether the merging conditions are met: If the Overlap Ratio is greater than the threshold of the overlap coverage, the masks are merged, and the bounding rectangle of the merged new mask is the union of the two masks; the merged area is the sum of the areas of the two masks minus the area of ​​the intersection. B. Update the mask in the node: Delete the original mask m i and m j , insert a new merge mask. C. Repeat this operation until all mask pairs have been processed.

5. The semantic segmentation method for remote sensing image multi-level mask classification optimization according to claim 1, characterized in that: The method for determining the category consistency includes: Build a semantic classification model; The semantic category output by the semantic classification model is used as an additional attribute of the mask and updated to the mask data structure after the preliminary merger. Each mask records the following information: mask area, bounding rectangle, area, and semantic category.

6. The semantic segmentation method for remote sensing image multi-level mask classification optimization according to claim 1, characterized in that: The method for determining spatial adjacency includes: If the intersection of the two masks is not empty, the masks are directly determined to be adjacent, and the distance between the masks is defined as 0, and no further calculation is required; if there is no intersection, the shortest distance between the boundary points is calculated. The specific steps are: Enclosing rectangle filter: First, determine whether the bounding rectangles of the two masks intersect; If they intersect, the shortest distance between the boundary points is directly calculated; If they do not intersect, proceed to the next step of judgment; Determine whether the closest distance between two bounding rectangles is less than the threshold d thresh : If it is less than the threshold, the closest distance of the boundary point is calculated; the calculation formula for the closest distance between two masks is: B1, B2 are the boundary point sets of the two masks respectively; ||p1-p2|| is the Euclidean distance between the boundary points; If it is greater than the threshold, it is determined to be non-adjacent and no further calculation is required. Candidate point optimization: When calculating the closest distance to a boundary point, the candidate point set is filtered by combining the bounding rectangle, and the distance is calculated only for potential adjacent mask boundary points. If the closest distance to the boundary point is less than d thresh If they are adjacent, they are not adjacent.

7. The semantic segmentation method for remote sensing image multi-level mask classification optimization according to claim 1, characterized in that: The secondary merging of adjacent masks belonging to the same semantic category to complete the segmentation of the remote sensing image includes: The input mask set is a spatial index data structure constructed in the preliminary mask merging, where each mask contains the following attributes: mask region, bounding rectangle, area and semantic category; Hierarchical recursive processing Based on the spatial index, starting from the root node, the following steps are performed on the mask set of each node: determine whether it is a leaf node. If it is a leaf node, directly merge the masks in the node; if it is not a leaf node, recursively call the child node to process the masks in the child node and across regions; Leaf node mask merging For the mask set in the leaf node, execute according to the following process: Traverse the mask pairs: for each pair of masks M in the node i ,M j , semantic consistency and spatial proximity determination are performed in sequence; Perform merging: If the two masks meet the merging conditions, generate a new mask M new , replace the original mask pair M i ,M j ; Update the mask set: insert the newly generated mask into the node and delete the merged mask; Parent node and child node mask merge For the cross-region mask stored in the parent node, the following process is used: Cross-region mask determination: Calculate the overlap coverage or the closest distance between the parent node mask and the child node mask. If the conditions are met, perform the merge. Update the mask set of the parent node: delete the original mask to be merged and insert the new mask into the parent node; Termination Condition When all masks in a node do not satisfy the merging condition, the recursion stops.

8. A semantic segmentation system for remote sensing image multi-level mask classification optimization, characterized in that: The system comprises: A cropping unit acquires and preprocesses remote sensing images to obtain multiple cropped sub-blocks; The mask generation unit uses the mask generation function of the SAM network to extract the potential target area of ​​each cropped sub-block into multiple independent masks; An overlap coverage calculation unit calculates the overlap coverage of the mask by constructing a spatial index; A first merging unit merges the masks according to the overlapping coverage ratio; The second merging unit calculates the spatial adjacency and category consistency between masks, and merges the adjacent masks belonging to the same semantic category for a second time to complete the segmentation of the remote sensing image.

9. An electronic device, comprising a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the electronic device executes each step of the semantic segmentation method for multi-level mask classification optimization of remote sensing images based on spatial index as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the various steps of the semantic segmentation method for multi-level mask classification optimization of remote sensing images based on spatial indexing are implemented as described in any one of claims 1 to 7.

Images