Ground feature semantic segmentation optimization fusion method and system based on remote sensing image

Abstract

The invention relates to the field of remote sensing image ground object segmentation, and discloses a ground object semantic segmentation optimization fusion method and system based on a remote sensing image, and the method comprises the steps: S1, collecting remote sensing image data of a research region, constructing a semantic segmentation model, extracting ground object features in the remote sensing image data, and outputting a preliminary ground object segmentation result; converting the preliminary ground feature segmentation result into a preliminary ground feature segmentation vector pattern spot; s2, an SAM model is constructed, point sample label data is used for training, and the trained SAM model carries out point prompt identification on the initial ground feature segmentation vector pattern spots to obtain a ground feature identification optimization result; and S3, fusing the preliminary ground feature segmentation result and the ground feature recognition optimization result according to the weight to obtain a final ground feature segmentation result. According to the method, the preliminary surface feature segmentation node with semantic segmentation and the surface feature recognition optimization result of SAM model point prompt recognition are fused, so that a more accurate segmentation result can be obtained, and particularly, the segmentation precision of the segmentation boundary region is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image ground object segmentation, and in particular to a ground object semantic segmentation optimization fusion method and system based on remote sensing images. Background Art

[0002] Traditional machine learning for remote sensing images requires manual design and feature extraction, which not only requires a lot of professional knowledge and experience, but also the effect of feature extraction is greatly affected by subjective factors, resulting in insufficient stability and reliability of the results. Traditional machine learning often has deficiencies in dealing with the details of the contours of objects, such as complex shapes or irregular buildings or winding rivers, which are prone to blurred contours and inaccurate boundaries in the segmentation results, which affects the accuracy and practicality of the segmentation results. Due to the inherent limitations of the network structure of traditional machine learning, it is difficult to fully capture the fine features of objects in complex scenes, resulting in deviations between the segmentation results and the actual contours of objects. Traditional machine learning lacks the characteristics of automation and intelligence, and requires manual participation in the feature extraction and classification process, which not only increases the workload, but also limits the wide application of the method. Traditional methods have poor adaptability when facing complex backgrounds and variable objects; for different objects with similar features, traditional methods are difficult to accurately distinguish, resulting in low segmentation accuracy and cannot meet the needs of high-precision segmentation in practical applications. Summary of the invention

[0003] The purpose of the present invention is to solve the technical problems pointed out by the background technology, and to provide a method and system for optimizing the fusion of semantic segmentation of objects based on remote sensing images. First, the semantic segmentation model is used to extract and identify the features of objects in remote sensing image data to obtain preliminary object segmentation results, and then the SAM model is used to use point prompt recognition on the preliminary object segmentation vector map to obtain the object recognition optimization result. Finally, the preliminary object segmentation result is fused with the object recognition optimization result, which significantly improves the accuracy of object category segmentation.

[0004] The purpose of the present invention is achieved through the following technical solutions:

[0005] A method for optimizing and fusion of semantic segmentation of ground objects based on remote sensing images, the method comprising:

[0006] S1. Collect remote sensing image data of the study area, build a semantic segmentation model to extract the features of objects in the remote sensing image data and output preliminary object segmentation results, and convert the preliminary object segmentation results into preliminary object segmentation vector patches;

[0007] S2, constructing a SAM model and using point sample label data for training, the trained SAM model uses point prompt recognition for the preliminary object segmentation vector map to obtain the object recognition optimization result;

[0008] S3. The preliminary object segmentation result and the object recognition optimization result are fused according to the weights to obtain the final object segmentation result.

[0009] In order to better realize the present invention, the semantic segmentation model is constructed according to the U-Net network structure. The semantic segmentation model includes an encoder and a decoder. The encoder includes a convolution layer, an activation function and a pooling layer. The encoder is used to extract multi-level features of land objects step by step; the decoder includes an upsampling layer, a jump connection and a convolution layer. The decoder is used to upsample step by step and jump-connect the corresponding level of land object features of the encoder; the semantic segmentation model uses an output layer to output a preliminary land object segmentation result. The preliminary land object segmentation result is a single-band raster data, and the numerical value corresponding to each pixel of the preliminary land object segmentation result represents the land object category.

[0010] Preferably, the semantic segmentation model is trained according to the following method: a remote sensing image sample database is constructed, and the remote sensing image sample database is constructed according to the association between remote sensing image samples and ground object category labels, and the ground object category labels include buildings, roads, water bodies, vegetation, bare soil and impermeable surfaces; the remote sensing image data is preprocessed before being input into the semantic segmentation model, and the preprocessing includes data normalization and standardization.

[0011] Preferably, the preliminary feature segmentation result is converted into a vector format automatically or manually using a raster-to-vector tool in geographic information processing software ArcGIS or QGIS to obtain a preliminary feature segmentation vector map.

[0012] Preferably, the point sample label data are positive sample points and negative sample points extracted from the preliminary object segmentation vector map according to the target object category, the positive sample points represent points belonging to the target object, and the negative sample points represent points not belonging to the target object.

[0013] Preferably, the SAM model selects the target object category boundary and points around the boundary in the preliminary object segmentation vector map and uses point prompts to identify and optimize the boundary contour.

[0014] Preferably, the fusion expression of the final object segmentation result is as follows:

[0015] H i =W1×X i +W2×Y i , where H i represents the ground feature category after pixel i is fused, X i represents the preliminary segmentation result of pixel i, W1 represents the weight of the preliminary segmentation result, Y i represents the optimization result of ground object recognition of pixel i, and W2 represents the weight of the optimization result of ground object recognition.

[0016] A land object semantic segmentation optimization fusion system based on remote sensing images comprises a data acquisition module, a semantic segmentation model, a SAM model and a fusion processing module, wherein the data acquisition module is used to acquire remote sensing image data of a study area; the semantic segmentation model is used to extract and output land object features in remote sensing image data, and convert a preliminary land object segmentation result into a preliminary land object segmentation vector map; the SAM model is trained using point sample label data, and the trained SAM model uses point prompt recognition on the preliminary land object segmentation vector map to obtain a land object recognition optimization result; the fusion processing module fuses the preliminary land object segmentation result with the land object recognition optimization result according to the weight to obtain a final land object segmentation result.

[0017] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0018] (1) The present invention first extracts and identifies the features of objects in remote sensing image data through a semantic segmentation model to obtain a preliminary object segmentation result, then converts the preliminary object segmentation result into a preliminary object segmentation vector patch, and uses the SAM model to perform point prompt recognition on the preliminary object segmentation vector patch to obtain an object recognition optimization result. Finally, the preliminary object segmentation result is integrated with the object recognition optimization result, thereby significantly improving the accuracy of object category segmentation.

[0019] (2) The present invention integrates the preliminary object segmentation results of semantic segmentation and the object recognition optimization results of SAM model point prompt recognition, which can obtain more accurate segmentation results, especially the segmentation accuracy in the segmentation boundary area is significantly improved; it can cope with complex object scenes and has the advantages of stable and reliable object segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A method flow chart of the method for optimizing the fusion of semantic segmentation of ground objects of the present invention;

[0021] Figure 2 The final object segmentation result of the water body target object in the embodiment is listed in the boundary contour segmentation performance effect diagram;

[0022] Figure 3 This is a principle structural block diagram of the ground feature semantic segmentation optimization and fusion system of the present invention. DETAILED DESCRIPTION

[0023] The present invention is further described in detail below in conjunction with embodiments:

[0024] Example

[0025] like Figure 1 As shown, a semantic segmentation optimization fusion method of ground objects based on remote sensing images, the method includes:

[0026] S1. Collect remote sensing image data of the study area, build a semantic segmentation model to extract the features of objects in the remote sensing image data and output preliminary object segmentation results. The preferred semantic segmentation model of the present invention is constructed according to the U-Net network structure. The semantic segmentation model includes an encoder and a decoder. The encoder includes a convolution layer, an activation function and a pooling layer. The encoder is used to extract multi-level object features step by step (through a series of convolution and pooling operations, the feature information of high-resolution remote sensing images is gradually extracted); the decoder includes an upsampling layer, a jump connection and a convolution layer. The decoder is used to upsample and jump-connect the corresponding level of the encoder features step by step (in this process, the feature information of different levels of the encoder is fused); the semantic segmentation model uses the output layer to output the preliminary object segmentation result. The preliminary object segmentation result is a single-band raster data, and the corresponding value of each pixel in the preliminary object segmentation result represents the object category. Then the present invention converts the preliminary object segmentation result into a preliminary object segmentation vector map. In some embodiments, the preliminary feature segmentation results are automatically or manually converted into vector format using the raster-to-vector tool in geographic information processing software ArcGIS or QGIS to obtain preliminary feature segmentation vector patches; taking geographic information processing software ArcGIS as an example, appropriate parameters are set through the raster-to-vector tool, such as vectorization tolerance, smoothness, etc. (for example: setting the vectorization tolerance to 3 and the smoothness to 4), and the raster data is converted into vector format to obtain preliminary feature segmentation vector patches.

[0027] In some embodiments, the semantic segmentation model is trained as follows: a remote sensing image sample database is constructed, and the remote sensing image sample database is constructed by associating remote sensing image samples and land object category labels, and the land object category labels include buildings, roads, water bodies, vegetation, bare soil, and impervious surfaces, etc. (the land object category labels are constructed according to the specific conditions of the study area. For example, if there are many water bodies in the study area, the labels can be further classified; if it involves a mining area or an ecological protection area, vegetation, bare soil, impervious surfaces, etc. can be further classified. Before the remote sensing image data is input into the semantic segmentation model, the remote sensing image data is preprocessed before being input into the semantic segmentation model. The preprocessing includes data normalization and standardization, and also includes remote sensing image size adjustment.

[0028] The semantic segmentation model (such as U-Net) of the present invention follows its own network structure and training parameters when performing initial semantic segmentation; during the training process, the model learns the characteristic patterns of different objects through a large amount of remote sensing image data, and these characteristic patterns are encoded in the parameters of the model. When the remote sensing image to be processed is input, the encoder part of the model starts to work, and extracts the characteristic information of different levels of the image through a series of convolution operations; the convolution operation uses different convolution kernels to slide on the image to extract local features in the image, such as edges, textures, etc. The pooling operation is used to reduce the resolution of the feature map, reduce the amount of data, and retain the main feature information. As the encoder processes layer by layer, the feature information of the image is gradually abstracted and compressed to form feature representations at different levels. The decoder part is based on these low-resolution feature representations and restores them to high-resolution segmentation results through deconvolution and upsampling operations. In this process, the decoder will fuse the feature information of the encoder at different levels to improve the accuracy and detail expression of the segmentation result. Finally, the model classifies each pixel according to the learned feature patterns, divides it into the corresponding object category, and generates a preliminary semantic segmentation result; this result reflects the model's preliminary understanding and classification of the objects in the remote sensing image.

[0029] S2. Construct a SAM model and use the point sample label data for training. The trained SAM model uses point prompt recognition on the preliminary feature segmentation vector map to obtain the feature recognition optimization result. The point sample label data of the present invention can be constructed according to the feature category (the point sample can be constructed based on the historical point data of the study area). The present invention can also be constructed based on the point data of the feature category determined in the preliminary feature segmentation vector map (for example, it can be a point whose feature category has been determined. Taking the feature category of water body as an example, the point at the center of the water body; or the point where the water body and vegetation intersect and are clear. This type of point is very helpful for the segmentation accuracy of water body and vegetation). The SAM model point prompt is expressed in the form of pixel coordinates, which is a tuple containing point coordinates and point labels; the coordinate part is usually a torch.Tensor object with a shape of (batch_size, num_points, 2), where batch_size represents the batch size of the input (in Figure 2 In the study area shown in the figure, batch_size is set to 1), which determines the number of samples that can be processed at one time. In practical applications, it can be adjusted according to the hardware resources and the amount of data; num_points is the number of each input point (in Figure 2The study area shown, num_points is set to 40). This number setting will affect the model's understanding of the image and the segmentation effect, and needs to be optimized according to the specific situation; the last 2 represents the x and y coordinates of each point, which is used to accurately represent the position of the point in the image. The point label part is a torch.Tensor with a shape of (batch_size, num_points), which is used to distinguish different points, such as positive sample points, negative sample points, etc. Positive sample points represent points that belong to the target object, and negative sample points represent points that do not belong to the target object.

[0030] In some embodiments, the point sample label data are positive sample points and negative sample points extracted from the preliminary object segmentation vector map according to the target object category, the positive sample points represent points belonging to the target object, and the negative sample points represent points that do not belong to the target object; for example, if the target object category is water body, the corresponding positive sample points and negative sample points are constructed according to the water body, so as to achieve accurate identification and segmentation of the target object category as water body.

[0031] Positive sample point generation: Use PyQGIS to use the "Generate Random Points Inside Polygons" tool and take the preliminary feature segmentation vector map as input data. The tool will generate random points inside each polygon according to certain rules. When generating random points, you can set some parameters, such as the number and distribution of random points, to meet different needs. The generated random points are used as positive sample points, and their labels are set to 1, indicating that these points belong to the target features.

[0032] Generation of negative sample points: Use the "Buffer" tool in PyQGIS to generate an external buffer for the preliminary feature segmentation vector patch. The distance parameter of the buffer needs to be adjusted according to the actual situation. Generally speaking, if the distance is too small, it may not contain enough negative sample points, and if the distance is too large, it may introduce too much irrelevant information. After the buffer is generated, use the "Generate Random Points Inside Polygon" tool to generate random points inside the buffer. Finally, use the "Select by Location" tool to filter out random points within the buffer and outside the initial vector patch as negative sample points, and set their labels to -1, indicating that these points do not belong to the target feature.

[0033] By using the above-mentioned point prompts, the SAM model can make full use of this prior information and combine it with the original features of the image to conduct a deeper analysis of the image. It can re-identify the boundary details of objects that were ignored or misjudged in the initial segmentation, and optimize the contours of the objects by re-evaluating and adjusting the boundaries of objects in the image, thereby improving the accuracy and precision of the segmentation results.

[0034] In some embodiments, the SAM model selects the boundary of the target object category and the points around the boundary in the preliminary object segmentation vector map and uses point prompt recognition to optimize the boundary contour. In addition to the above-mentioned general point recognition optimization of the preliminary object segmentation vector map using point prompt recognition, the present application also focuses on the boundary of the target object category and the points around the boundary of the target object category using point prompt recognition, which can better optimize the boundary contour of the target object.

[0035] The SAM model of the present invention has powerful image understanding and segmentation capabilities, and can perform more precise contour extraction and optimization of objects in an image according to input image information and prompts; receive preliminary semantic segmentation results and process them in the form of point prompts as prior information of the SAM model, and perform a more in-depth analysis of the image in combination with the original features of the image; the SAM model re-evaluates and adjusts the boundaries of objects in the image through its internal neural network structure; can identify boundary pixels that are significantly different from the surrounding areas, and correct the classification of these pixels, thereby optimizing the contours of the objects.

[0036] S3. The preliminary object segmentation result and the object recognition optimization result are weighted and fused to obtain the final object segmentation result. The preliminary object segmentation result and the object recognition optimization result can be weighted and fused pixel by pixel using the image fusion method. The object recognition optimization result is fused with the preliminary object segmentation result according to the fusion strategy. During the fusion process, the two results are weighted and summed according to the characteristics of different object categories and image regions to obtain the final object segmentation result. Figure 2 As shown, the present invention selects a research area with more water bodies, and obtains the preliminary segmentation result of water bodies as target objects according to the present invention ( Figure 2 The present invention obtains the final ground object segmentation result ( Figure 2 Through intuitive image comparison, the method of the present invention has significant advantages in optimizing the contour of the object. The final object segmentation result of the present invention has been significantly optimized and improved in terms of contour accuracy and details, and the river range and boundary are presented more accurately.

[0037] In some embodiments, the fusion expression of the final object segmentation result of the present invention is as follows:

[0038] H i =W1×X i +W2×Y i , where H i represents the ground feature category after pixel i is fused, X i represents the preliminary segmentation result of pixel i, W1 represents the weight of the preliminary segmentation result, Y iW2 represents the optimization result of the object recognition of pixel i, and W2 represents the weight of the optimization result of the object recognition. If pixel i is a complex building, coastline, etc., since the result after SAM model optimization can often capture its boundary details more accurately, the weight of the SAM model optimization result is greater. For such objects, the weight of the SAM model object recognition optimization result can be set to 0.7 or 0.8, and the weight of the preliminary object segmentation result can be set to 0.3 or 0.2 to highlight the advantages of the SAM model in optimizing the contour of the object. If pixel i is a relatively simple and regular object, such as a large area of ​​farmland, square, etc., the initial segmentation result may have good accuracy. At this time, the weight of the SAM model object recognition optimization result can be appropriately reduced; for example, for such objects, the weight of the SAM model object recognition optimization result can be set to 0.5, and the weight of the preliminary object segmentation result can be set to 0.5 to retain more classification information in the preliminary object segmentation result. The present invention can dynamically adjust the fusion weight according to the characteristics of different objects, which can not only give full play to the advantages of the SAM model in optimizing the contour of objects, but also effectively retain the classification information in the initial segmentation result, making the final fusion result more accurate and reliable.

[0039] like Figure 3 As shown, a semantic segmentation optimization and fusion system for objects based on remote sensing images includes a data acquisition module, a semantic segmentation model, a SAM model and a fusion processing module. The data acquisition module is used to collect remote sensing image data of the study area. The semantic segmentation model is used to extract and output the object features in the remote sensing image data, and convert the preliminary object segmentation results into preliminary object segmentation vector patches. The SAM model is trained using point sample label data. The trained SAM model uses point prompt recognition to obtain the object recognition optimization result for the preliminary object segmentation vector patches. The fusion processing module fuses the preliminary object segmentation result and the object recognition optimization result according to the weight to obtain the final object segmentation result. Compared with using only the preliminary object segmentation result, the final object segmentation result obtained by the fusion of the present invention has obvious effect in optimizing the object contour, and the overall segmentation accuracy is improved; in terms of the mean intersection over union (mIoU) index, the present invention has significant improvement, and the mean intersection over union (mIoU) index is improved from 0.86 to 0.95.

[0040] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for optimizing and fusion semantic segmentation of land objects based on remote sensing images, characterized by: The methods include: S1. Collect remote sensing image data of the study area, build a semantic segmentation model to extract the features of objects in the remote sensing image data and output preliminary object segmentation results, and convert the preliminary object segmentation results into preliminary object segmentation vector patches; S2, constructing a SAM model and using point sample label data for training, the trained SAM model uses point prompt recognition for the preliminary object segmentation vector map to obtain the object recognition optimization result; S3. The preliminary object segmentation result and the object recognition optimization result are fused according to the weights to obtain the final object segmentation result.

2. The method for optimizing and fusion semantic segmentation of land objects based on remote sensing images according to claim 1, characterized in that: The semantic segmentation model is constructed according to the U-Net network structure. The semantic segmentation model includes an encoder and a decoder. The encoder includes a convolution layer, an activation function and a pooling layer. The encoder is used to extract multi-level features of objects step by step. The decoder includes an upsampling layer, a skip connection and a convolution layer. The decoder is used to upsample step by step and skip-connect the features of objects at the corresponding level of the encoder. The semantic segmentation model uses the output layer to output the preliminary object segmentation result. The preliminary object segmentation result is a single-band raster data. The value corresponding to each pixel in the preliminary object segmentation result represents the object category.

3. The method for optimizing and fusion semantic segmentation of land objects based on remote sensing images according to claim 1 or 2, characterized in that: The semantic segmentation model is trained according to the following method: a remote sensing image sample database is constructed, and the remote sensing image sample database is constructed according to the association between remote sensing image samples and ground object category labels, and the ground object category labels include buildings, roads, water bodies, vegetation, bare soil and impermeable surfaces; the remote sensing image data is preprocessed before being input into the semantic segmentation model, and the preprocessing includes data normalization processing and standardization processing.

4. The method for optimizing and fusion semantic segmentation of land objects based on remote sensing images according to claim 1, characterized in that: The preliminary feature segmentation result is converted into a vector format automatically or manually using a raster-to-vector tool in geographic information processing software ArcGIS or QGIS to obtain a preliminary feature segmentation vector patch.

5. The method for optimizing and fusion semantic segmentation of land objects based on remote sensing images according to claim 1, characterized in that: The point sample label data are positive sample points and negative sample points extracted from the preliminary object segmentation vector map according to the target object category. The positive sample points represent points belonging to the target object, and the negative sample points represent points that do not belong to the target object.

6. The method for optimizing and fusion semantic segmentation of land objects based on remote sensing images according to claim 1 or 5, characterized in that: The SAM model selects the target object category boundary and points around the boundary in the preliminary object segmentation vector map patch and uses point prompts to identify and optimize the boundary contour.

7. The method for optimizing and fusion semantic segmentation of land objects based on remote sensing images according to claim 1, characterized in that: The fusion expression of the final object segmentation result is as follows: H i =W1×X i +W2×Y i , where H i represents the ground feature category after pixel i is fused, X i represents the preliminary segmentation result of pixel i, W1 represents the weight of the preliminary segmentation result, Y i represents the optimization result of ground object recognition of pixel i, and W2 represents the weight of the optimization result of ground object recognition.

8. A system for optimizing semantic segmentation and fusion of objects based on remote sensing images, characterized by: It includes a data acquisition module, a semantic segmentation model, a SAM model and a fusion processing module. The data acquisition module is used to collect remote sensing image data of the study area; the semantic segmentation model is used to extract and output the ground object features in the remote sensing image data, and convert the preliminary ground object segmentation results into preliminary ground object segmentation vector patches; the SAM model is trained using point sample label data, and the trained SAM model uses point prompt recognition on the preliminary ground object segmentation vector patches to obtain ground object recognition optimization results; The fusion processing module fuses the preliminary object segmentation result and the object recognition optimization result according to the weight to obtain the final object segmentation result.

Images