Method and system for optimizing and fusion of semantic segmentation of land objects based on remote sensing images
By combining the semantic segmentation model and the SAM model, the segmentation of remote sensing image objects is optimized, which solves the problems of inaccurate segmentation results and insufficient stability in traditional methods and achieves high-precision segmentation of remote sensing image objects.
Patent Information
- Application Number
- CN202510067209.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-15
- Publication Date
- 2025-09-30
- Estimated Expiration
- 2045-01-15
AI Technical Summary
Traditional machine learning lacks stability and reliability in the segmentation of objects in remote sensing images, and has difficulty accurately processing the details of complex objects, especially irregular buildings or winding rivers, resulting in blurred segmentation results and inaccurate boundaries. It also lacks automation and intelligence, making it difficult to meet high-precision segmentation requirements.
The semantic segmentation model is used to extract the ground feature of remote sensing image data, and the point prompt recognition optimization is performed through the SAM model. The preliminary ground feature segmentation results and the ground feature recognition optimization results are combined with weight fusion to improve the segmentation accuracy.
The accuracy of object category segmentation has been significantly improved, especially in the segmentation boundary area. It can handle complex object scenes stably and reliably, and improve the accuracy and detail expression of the segmentation results.
Smart Images
Figure CN119919666B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of remote sensing image object segmentation, and in particular to a method and system for optimizing the semantic segmentation and fusion of objects based on remote sensing images. Background Art
[0002] Traditional machine learning for remote sensing imagery requires manual design and feature extraction, which not only requires extensive expertise and experience but is also subject to significant subjective influence, resulting in instability and reliability. Traditional machine learning often struggles with handling detailed object contours, such as complex shapes, irregular buildings, or winding rivers. This can lead to blurred contours and inaccurate boundaries in the segmentation results, compromising their accuracy and practicality. Inherent limitations of traditional machine learning network architectures make it difficult to fully capture fine features in complex scenes, resulting in discrepancies between segmentation results and actual object contours. Traditional machine learning lacks automation and intelligence, requiring manual intervention in the feature extraction and classification process, which increases workload and limits its widespread application. Traditional methods are less adaptable to complex backgrounds and variable objects. They also struggle to accurately distinguish between different objects with similar features, resulting in low segmentation accuracy and failing to meet the high-precision segmentation requirements of practical applications. Summary of the Invention
[0003] The purpose of the present invention is to solve the technical problems pointed out by the background technology, and to provide a semantic segmentation optimization fusion method and system for land objects based on remote sensing images. First, the semantic segmentation model is used to extract and identify the land object features in the remote sensing image data to obtain a preliminary land object segmentation result, and then the SAM model is used to use point prompt recognition on the preliminary land object segmentation vector map to obtain the land object recognition optimization result. Finally, the preliminary land object segmentation result is fused with the land object recognition optimization result, which significantly improves the land object category segmentation accuracy.
[0004] The purpose of the present invention is achieved through the following technical solutions:
[0005] A method for semantic segmentation and optimization fusion of land objects based on remote sensing images, the method comprising:
[0006] S1. Collect remote sensing image data of the study area, build a semantic segmentation model to extract the ground feature in the remote sensing image data and output the preliminary ground feature segmentation results, and convert the preliminary ground feature segmentation results into preliminary ground feature segmentation vector patches;
[0007] S2. Construct a SAM model and train it using point sample label data. The trained SAM model uses point prompt recognition to identify the preliminary feature segmentation vector map to obtain the feature recognition optimization result.
[0008] S3. The preliminary object segmentation result and the object recognition optimization result are fused according to the weights to obtain the final object segmentation result.
[0009] In order to better realize the present invention, the semantic segmentation model is constructed according to the U-Net network structure. The semantic segmentation model includes an encoder and a decoder. The encoder includes a convolution layer, an activation function and a pooling layer. The encoder is used to extract multi-level feature of the ground object step by step; the decoder includes an upsampling layer, a skip connection and a convolution layer. The decoder is used to upsample step by step and skip-connect the feature of the corresponding level of the encoder; the semantic segmentation model uses the output layer to output the preliminary feature segmentation result. The preliminary feature segmentation result is single-band raster data. The numerical value corresponding to each pixel of the preliminary feature segmentation result represents the feature category.
[0010] Preferably, the semantic segmentation model is trained according to the following method: a remote sensing image sample database is constructed, and the remote sensing image sample database is constructed according to the association between remote sensing image samples and ground object category labels, and the ground object category labels include buildings, roads, water bodies, vegetation, bare soil and impervious surfaces; the remote sensing image data is preprocessed before being input into the semantic segmentation model, and the preprocessing includes data normalization and standardization.
[0011] Preferably, the preliminary feature segmentation result is converted into a vector format automatically or manually using a raster-to-vector tool in geographic information processing software ArcGIS or QGIS to obtain a preliminary feature segmentation vector map.
[0012] Preferably, the point sample label data are positive sample points and negative sample points extracted from the preliminary object segmentation vector map according to the target object category. The positive sample points represent points belonging to the target object, and the negative sample points represent points not belonging to the target object.
[0013] Preferably, the SAM model selects the target object category boundary and points around the boundary in the preliminary object segmentation vector map and uses point prompts to identify and optimize the boundary contour.
[0014] Preferably, the fusion expression of the final object segmentation result is as follows:
[0015] H i =W1×X i +W2×Y i , where H i represents the ground feature category after pixel i is fused, X i represents the preliminary segmentation result of pixel i, W1 represents the weight of the preliminary segmentation result, Y i represents the optimization result of ground object recognition of pixel i, and W2 represents the weight of the optimization result of ground object recognition.
[0016] A land object semantic segmentation optimization and fusion system based on remote sensing images includes a data acquisition module, a semantic segmentation model, a SAM model and a fusion processing module. The data acquisition module is used to collect remote sensing image data of a study area; the semantic segmentation model is used to extract and output land object features in the remote sensing image data, and convert preliminary land object segmentation results into preliminary land object segmentation vector patches; the SAM model is trained using point sample label data, and the trained SAM model uses point prompt recognition on the preliminary land object segmentation vector patches to obtain land object recognition optimization results; the fusion processing module fuses the preliminary land object segmentation results and the land object recognition optimization results according to weights to obtain a final land object segmentation result.
[0017] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0018] (1) The present invention first extracts and identifies the features of objects in remote sensing image data through a semantic segmentation model to obtain a preliminary object segmentation result, then converts the preliminary object segmentation result into a preliminary object segmentation vector patch, and uses the SAM model to perform point prompt recognition on the preliminary object segmentation vector patch to obtain an object recognition optimization result. Finally, the preliminary object segmentation result is integrated with the object recognition optimization result, which significantly improves the accuracy of object category segmentation.
[0019] (2) The present invention integrates the preliminary object segmentation results of semantic segmentation and the object recognition optimization results of SAM model point prompt recognition, which can obtain more accurate segmentation results, especially the segmentation accuracy in the segmentation boundary area is significantly improved; it can cope with complex object scenes and has the advantages of stable and reliable object segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 This is a flow chart of the method for optimizing the semantic segmentation and fusion of features of the present invention;
[0021] Figure 2 The final segmentation result of the water body target object in the embodiment is listed in the boundary contour segmentation performance effect diagram;
[0022] Figure 3 This is a principle structural block diagram of the ground feature semantic segmentation optimization and fusion system of the present invention. DETAILED DESCRIPTION
[0023] Below in conjunction with embodiment, the present invention is described in further detail:
[0024] Example
[0025] like Figure 1 As shown, a semantic segmentation optimization fusion method for land objects based on remote sensing images includes:
[0026] S1. Collect remote sensing image data of the study area, build a semantic segmentation model to extract the features of the objects in the remote sensing image data and output preliminary object segmentation results. The preferred semantic segmentation model of the present invention is constructed according to the U-Net network structure. The semantic segmentation model includes an encoder and a decoder. The encoder includes a convolution layer, an activation function and a pooling layer. The encoder is used to extract multi-level object features step by step (through a series of convolution and pooling operations, the feature information of the high-resolution remote sensing image is gradually extracted); the decoder includes an upsampling layer, a jump connection and a convolution layer. The decoder is used to upsample and jump-connect the corresponding level of object features of the encoder step by step (in this process, the feature information of different levels of the encoder is fused); the semantic segmentation model uses the output layer to output preliminary object segmentation results. The preliminary object segmentation results are single-band raster data, and the numerical value corresponding to each pixel of the preliminary object segmentation results represents the object category. Then the present invention converts the preliminary object segmentation results into preliminary object segmentation vector patches. In some embodiments, the preliminary feature segmentation results are automatically or manually converted into vector format using the raster-to-vector tool in the geographic information processing software ArcGIS or QGIS to obtain preliminary feature segmentation vector patches; taking the geographic information processing software ArcGIS as an example, appropriate parameters are set through the raster-to-vector tool, such as the vectorization tolerance, smoothness, etc. (for example: setting the vectorization tolerance to 3 and the smoothness to 4), and the raster data is converted into vector format to obtain preliminary feature segmentation vector patches.
[0027] In some embodiments, the semantic segmentation model is trained as follows: a remote sensing image sample database is constructed, and the remote sensing image sample database is constructed by associating remote sensing image samples with object category labels, and the object category labels include buildings, roads, water bodies, vegetation, bare soil, and impervious surfaces, etc. (the object category labels are constructed based on the specific conditions of the study area. For example, if the study area has many water bodies, the labels can be further specifically classified; if it involves a mining area or an ecological protection area, vegetation, bare soil, impervious surfaces, etc. can be further specifically classified.) Before the remote sensing image data is input into the semantic segmentation model, the remote sensing image data is preprocessed before being input into the semantic segmentation model. The preprocessing includes data normalization and standardization, and also includes remote sensing image size adjustment.
[0028] The semantic segmentation model of the present invention (such as U-Net) follows its own network structure and training parameters when performing initial semantic segmentation; during the training process, the model learns the characteristic patterns of different ground objects through a large amount of remote sensing image data, and these characteristic patterns are encoded in the parameters of the model. When the remote sensing image to be processed is input, the encoder part of the model starts working and extracts the feature information of different levels of the image through a series of convolution operations; the convolution operation uses different convolution kernels to slide on the image to extract local features in the image, such as edges, textures, etc. The pooling operation is used to reduce the resolution of the feature map, reduce the amount of data, and retain the main feature information. As the encoder processes layer by layer, the feature information of the image is gradually abstracted and compressed to form feature representations at different levels. The decoder part uses these low-resolution feature representations as a basis and restores them to high-resolution segmentation results through deconvolution and upsampling operations. In this process, the decoder will fuse the feature information of the encoder at different levels to improve the accuracy and detail expression of the segmentation result. Finally, the model classifies each pixel according to the learned feature patterns, divides it into the corresponding ground object category, and generates a preliminary semantic segmentation result; this result reflects the model's preliminary understanding and classification of the ground objects in the remote sensing image.
[0029] S2. Construct a SAM model and use point sample label data for training. The trained SAM model uses point prompt recognition on the preliminary feature segmentation vector map to obtain feature recognition optimization results. The point sample label data of the present invention can be constructed according to the feature category (the point sample can be constructed based on the historical point data of the study area). The present invention can also be constructed based on point data that determines the feature category in the preliminary feature segmentation vector map (for example, it can be a point that has determined the feature category. Taking the feature category of water bodies as an example, the point at the center of the water body; or a point where the water body and vegetation intersect and are clear. This type of point is very helpful for the segmentation accuracy of water bodies and vegetation). The SAM model point prompt is expressed in the form of pixel coordinates, which is a tuple containing point coordinates and point labels; the coordinate part is usually a torch.Tensor object of shape (batch_size, num_points, 2), where batch_size represents the batch size of the input (in Figure 2 In the study area shown in the figure, batch_size is set to 1), which determines the number of samples that can be processed at one time. In practical applications, it can be adjusted according to the hardware resources and the amount of data. num_points is the number of each input point (in Figure 2For the study area shown, num_points is set to 40. This number affects the model's understanding of the image and the segmentation performance, and needs to be optimized based on the specific situation. The last two represent the x and y coordinates of each point, which accurately represent its position in the image. The point label is a torch.Tensor of shape (batch_size, num_points), which is used to distinguish different points, such as positive and negative samples. Positive points represent points that belong to the target object, while negative points represent points that do not belong to the target object.
[0030] In some embodiments, the point sample label data are positive sample points and negative sample points extracted from the preliminary object segmentation vector map according to the target object category. The positive sample points represent points belonging to the target object, and the negative sample points represent points that do not belong to the target object. For example, if the target object category is water body, the corresponding positive sample points and negative sample points are constructed according to the water body, so as to achieve accurate identification and segmentation of the target object category as water body.
[0031] Positive sample point generation: Use PyQGIS to use the "Generate Random Points Inside Polygons" tool and take the preliminary feature segmentation vector map as input data. The tool will generate random points inside each polygon according to certain rules. When generating random points, you can set some parameters, such as the number and distribution of random points, to meet different needs. The generated random points are used as positive sample points, and their labels are set to 1, indicating that these points belong to the target feature.
[0032] Generation of negative sample points: Use the "Buffer" tool in PyQGIS to generate an external buffer for the preliminary feature segmentation vector patch. The distance parameter of the buffer needs to be adjusted according to the actual situation. Generally speaking, if the distance is too small, it may not contain enough negative sample points, and if the distance is too large, it may introduce too much irrelevant information. After the buffer is generated, use the "Generate Random Points Inside Polygon" tool to generate random points inside the buffer. Finally, use the "Select by Location" tool to filter out random points within the buffer and outside the initial vector patch as negative sample points, and set their labels to -1, indicating that these points do not belong to the target feature.
[0033] By using the above-mentioned point prompts, the SAM model can make full use of this prior information and combine it with the original features of the image to conduct a more in-depth analysis of the image. It can re-identify the boundary details of the objects that were ignored or misjudged in the initial segmentation, and optimize the outline of the objects by re-evaluating and adjusting the boundaries of the objects in the image, thereby improving the accuracy and precision of the segmentation results.
[0034] In some embodiments, the SAM model selects the boundary of the target object category and the points around the boundary in the preliminary object segmentation vector map and uses point prompts to identify and optimize the boundary contour. In addition to the above-mentioned general point recognition optimization of the preliminary object segmentation vector map using point prompts, the present application also focuses on the boundary of the target object category and the points around the boundary of the target object category using point prompts, which can better optimize the boundary contour of the target object.
[0035] The SAM model of the present invention has powerful image understanding and segmentation capabilities and can perform more precise contour extraction and optimization of objects in an image based on input image information and prompts. It receives and processes preliminary semantic segmentation results in the form of point prompts as prior information for the SAM model, and combines them with the original features of the image to perform a more in-depth analysis of the image. The SAM model re-evaluates and adjusts the boundaries of objects in the image through its internal neural network structure. It can identify boundary pixels that are significantly different from the surrounding area and correct the classification of these pixels, thereby optimizing the contours of the objects.
[0036] S3. The preliminary object segmentation result and the object recognition optimization result are fused according to the weight to obtain the final object segmentation result. The preliminary object segmentation result and the object recognition optimization result can be fused pixel by pixel by weight using the image fusion method. The object recognition optimization result and the preliminary object segmentation result are fused according to the fusion strategy; in the fusion process, the two results are weighted and summed according to the characteristics of different object categories and image areas to obtain the final object segmentation result. Figure 2 As shown, the present invention selects a research area with more water bodies, and obtains the preliminary segmentation result of water bodies as target objects according to the present invention ( Figure 2 The present invention obtains the final ground object segmentation result ( Figure 2 Through intuitive image comparison, the method of the present invention has significant advantages in optimizing the contours of land objects. The final land object segmentation result of the present invention has been significantly optimized and improved in terms of contour accuracy and details, and the river range and boundaries are presented more accurately.
[0037] In some embodiments, the fusion expression of the final ground feature segmentation result of the present invention is as follows:
[0038] H i =W1×X i +W2×Y i , where H i represents the ground feature category after pixel i is fused, X i represents the preliminary segmentation result of pixel i, W1 represents the weight of the preliminary segmentation result, Y irepresents the optimized feature recognition result for pixel i, and W2 represents the weight of the optimized feature recognition result. If pixel i is a complex building or coastline, the weight of the optimized SAM model result is greater because the optimized SAM model result often more accurately captures its boundary details. For such features, the weight of the optimized SAM model feature recognition result can be set to 0.7 or 0.8, and the weight of the preliminary feature segmentation result can be set to 0.3 or 0.2 to highlight the SAM model's advantage in optimizing feature outlines. If pixel i is a relatively simple, regular feature, such as a large area of farmland or a square, the initial segmentation result may already have good accuracy. In this case, the weight of the optimized SAM model feature recognition result can be appropriately reduced. For example, for such features, the weight of the optimized SAM model feature recognition result can be set to 0.5, and the weight of the preliminary feature segmentation result can be set to 0.5 to retain more classification information in the preliminary feature segmentation result. The present invention can dynamically adjust the fusion weight according to the characteristics of different objects, which can not only give full play to the advantage of the SAM model in optimizing the object contour, but also effectively retain the classification information in the initial segmentation result, making the final fusion result more accurate and reliable.
[0039] like Figure 3 As shown, a semantic segmentation optimization and fusion system for land objects based on remote sensing images includes a data acquisition module, a semantic segmentation model, a SAM model, and a fusion processing module. The data acquisition module is used to collect remote sensing image data of the study area. The semantic segmentation model is used to extract and output land object features in the remote sensing image data, and convert the preliminary land object segmentation results into preliminary land object segmentation vector patches. The SAM model is trained using point sample label data. The trained SAM model uses point prompt recognition on the preliminary land object segmentation vector patches to obtain land object recognition optimization results. The fusion processing module fuses the preliminary land object segmentation results with the land object recognition optimization results according to the weights to obtain the final land object segmentation results. Compared with using only the preliminary land object segmentation results, the final land object segmentation results obtained by the present invention are more effective in optimizing land object contours and improving overall segmentation accuracy. In terms of the mean intersection over union (mIoU) index, the present invention has a significant improvement, with the mean intersection over union (mIoU) index increasing from 0.86 to 0.95.
[0040] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for semantic segmentation and optimization fusion of land objects based on remote sensing images, characterized by: The methods include: S1. Collect remote sensing image data of the study area, build a semantic segmentation model to extract ground feature features in the remote sensing image data and output preliminary ground feature segmentation results, and convert the preliminary ground feature segmentation results into preliminary ground feature segmentation vector patches; the semantic segmentation model is trained according to the following method: build a remote sensing image sample database, the remote sensing image sample database is built by associating remote sensing image samples with ground feature category labels, where the ground feature category labels include buildings, roads, water bodies, vegetation, bare soil, and impervious surfaces; S2. Construct a SAM model and train it using point sample label data. The point sample label data is positive sample points and negative sample points extracted from the preliminary object segmentation vector map according to the target object category. Positive sample points represent points belonging to the target object, and negative sample points represent points not belonging to the target object. The trained SAM model uses point prompt recognition on the preliminary object segmentation vector map to obtain an object recognition optimization result. The SAM model selects the boundary of the target object category and points around the boundary in the preliminary object segmentation vector map and uses point prompt recognition to optimize the boundary contour. S3. The preliminary object segmentation result and the object recognition optimization result are fused according to the weights to obtain the final object segmentation result.
2. The method for semantic segmentation and optimization fusion of land objects based on remote sensing images according to claim 1, characterized in that: The semantic segmentation model is constructed according to the U-Net network structure. The semantic segmentation model includes an encoder and a decoder. The encoder includes a convolutional layer, an activation function, and a pooling layer. The encoder is used to extract multi-level features of objects step by step. The decoder includes an upsampling layer, a skip connection, and a convolutional layer. The decoder is used to upsample and skip the features of the corresponding layers of the encoder. The semantic segmentation model uses the output layer to output the preliminary feature segmentation results. The preliminary feature segmentation results are single-band raster data. The numerical value corresponding to each pixel in the preliminary feature segmentation results represents the feature category.
3. The method for optimizing semantic segmentation and fusion of land objects based on remote sensing images according to claim 1 or 2, characterized in that: The remote sensing image data is preprocessed before being input into the semantic segmentation model, and the preprocessing includes data normalization and standardization.
4. The method for semantic segmentation and optimization fusion of land objects based on remote sensing images according to claim 1, characterized in that: The preliminary feature segmentation results are converted into vector format automatically or manually using raster-to-vector tools in geographic information processing software ArcGIS or QGIS to obtain preliminary feature segmentation vector patches.
5. The method for semantic segmentation and optimization fusion of land objects based on remote sensing images according to claim 1, characterized in that: The fusion expression of the final ground object segmentation result is as follows: ,in represents the ground feature category after pixel i is fused, represents the preliminary ground object segmentation result of pixel i, represents the weight of the preliminary ground feature segmentation result, represents the optimization result of ground feature recognition of pixel i, Indicates the weight of the feature recognition optimization result.
6. A system for optimizing and fusion of semantic segmentation of land objects based on remote sensing images that implements the method for optimizing and fusion of semantic segmentation of land objects according to claim 1, characterized in that: The system comprises a data acquisition module, a semantic segmentation model, a SAM model, and a fusion processing module. The data acquisition module is used to collect remote sensing image data of the study area. The semantic segmentation model is used to extract and output ground feature features from the remote sensing image data, and convert preliminary ground feature segmentation results into preliminary ground feature segmentation vector patches. The SAM model is trained using point sample label data. The trained SAM model uses point cue recognition on preliminary ground feature segmentation vector patches to obtain optimized ground feature recognition results. The fusion processing module fuses the preliminary object segmentation result and the object recognition optimization result according to the weight to obtain the final object segmentation result.