Image segmentation method based on improved SAM model

Abstract

The invention discloses an image segmentation method based on an improved SAM model, and relates to the technical field of image segmentation, and the method comprises the following steps: collecting remote sensing image data sets of various different scenes and categories, and carrying out the data annotation and data enhancement preprocessing operation of remote sensing images; training a preset target detection model by using the labeled remote sensing image data set; using the trained target detection model to predict a remote sensing image, identifying a target area and category in the image, and outputting a prediction result; extracting multi-scale features of each target area detected in the prediction result based on ViT, and performing image segmentation based on SAM to obtain a classification result of each target area; and taking the classification result of each target area as a returned detection result. Through the remote sensing image classification method based on the convolutional neural network, the rapid classification task of the remote sensing images is realized, and powerful technical support and data support are provided for the fields of environment monitoring, urban planning, agricultural management and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically, to an image segmentation method based on an improved SAM model. Background Art

[0002] Image segmentation is a core technology in the field of computer vision. Its goal is to divide different objects or regions in an image into different parts for subsequent analysis and understanding. In recent years, with the rapid development of deep learning technology, the performance of image segmentation has been significantly improved. Among them, Segment Anything Model (SAM) has attracted widespread attention with its ability to "detect everything, segment everything, and generate everything".

[0003] Image segmentation is a basic and crucial technology in the field of computer vision and image processing, and its history can be traced back to the early stages of the development of computer graphics. With the continuous advancement of digital imaging technology and the significant improvement of computer processing capabilities, image segmentation technology has gradually become a bridge connecting low-level image processing and high-level image analysis and understanding.

[0004] Traditional image segmentation methods are mainly based on the pixel-level features of the image, such as color, brightness, texture, etc., and divide the image into different regions or objects through certain algorithms or rules. These methods can be roughly divided into several categories: threshold-based segmentation, region-based segmentation, edge-based segmentation, and segmentation based on specific theories (such as segmentation based on graph theory, segmentation based on clustering, etc.).

[0005] Although traditional image segmentation methods have solved the problem of image segmentation to a certain extent, they often rely on prior knowledge of images, parameter selection and manual intervention, and are difficult to handle image segmentation tasks in complex scenes. With the rise of deep learning technology, deep learning-based image segmentation methods have gradually emerged, bringing new breakthroughs to the field of image segmentation with their powerful feature learning capabilities and end-to-end optimization methods. However, traditional image segmentation methods still have important research value and application scenarios, especially in situations where computing resources are limited and real-time requirements are high. Traditional image segmentation methods are mainly divided into two categories: interactive segmentation and automatic segmentation. Interactive segmentation allows users to guide the segmentation process by iteratively refining masks, but this method requires high user participation and time cost. Although automatic segmentation can automatically segment predefined specific object categories, it requires a large amount of manually annotated data to train the model. The SAM model cleverly combines the advantages of these two methods, and can achieve a wide range of segmentation tasks through flexible user prompts (such as clicks, text boxes, etc.), greatly improving the flexibility and efficiency of segmentation.

[0006] However, although the original SAM model is powerful, it still faces some challenges in practical applications. For example, its model structure based on Visual Transformer (ViT) leads to high computational and memory costs, which has certain limitations for real-time applications. To overcome these limitations, researchers have proposed a variety of improvement strategies to reduce the complexity and computational cost of the model while maintaining or even improving segmentation performance.

[0007] Remote sensing images usually contain a large number of complex and diverse objects, which makes them of great value in object recognition, land cover classification and change detection. However, current remote sensing image segmentation and classification methods are difficult to detect and classify these targets efficiently and accurately. At the same time, in complex scenes, traditional segmentation methods may not be able to accurately extract object boundaries, resulting in insufficient accuracy of classification and analysis results.

[0008] The above disclosed technical solutions have at least the following technical problems: in complex scenes, traditional segmentation methods may not be able to accurately extract the boundaries of objects, resulting in insufficient accuracy of classification and analysis results. In view of the above problems, the present invention proposes a solution. Summary of the invention

[0009] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides an image segmentation method based on an improved SAM model, which performs target detection on the input remote sensing image data and performs target segmentation according to the target detection results to solve the problem of insufficient accuracy of classification and analysis results.

[0010] To achieve the above object, the present invention provides the following technical solutions: An image segmentation method based on an improved SAM model comprises the following steps: collecting remote sensing image data sets of various scenes and categories, and performing data annotation and data enhancement preprocessing operations on the remote sensing images; using the annotated remote sensing image data sets to train a preset target detection model; using the trained target detection model to predict the remote sensing image, identifying the target area and category in the image, and outputting the prediction result; extracting multi-scale features based on ViT for each target area detected in the prediction result, and performing image segmentation based on SAM to obtain the classification result of each target area; and using the classification result of each target area as the returned detection result.

[0011] In a preferred embodiment, the collection of remote sensing image data sets of various scenes and categories is specifically: collecting remote sensing image data containing various scenes and categories, scaling the remote sensing images to the model input size and scaling the pixel values ​​to a preset range to generate a remote sensing image data set.

[0012] In a preferred embodiment, the use of a labeled remote sensing image dataset to train a preset target detection model is specifically as follows: the remote sensing image dataset is divided into a training set and a validation set, the training set is passed to the model, and the output of the target's location and category information is obtained; the value of the loss function is calculated based on the output and annotation information of the model, and based on the value of the loss function, the gradient is calculated and back-propagated to the parameters of the model; the model parameters are updated according to the gradient through an optimization algorithm, the loss function is reduced, the learning rate is controlled, and the step size and direction of the model parameters are updated; and the optimal model is obtained and saved based on the validation set to complete subsequent detection tasks.

[0013] In a preferred embodiment, the target area and category in the image are identified and the prediction result is output, specifically: adjusting the input image size and normalizing it; loading a trained target detection model, inputting the preprocessed image into the model for processing, and obtaining the model prediction result; Decode and output the high-confidence prediction category results predicted by the model.

[0014] In a preferred embodiment, the preprocessed image is input into the model for processing, specifically: the convolutional neural network model simultaneously predicts the bounding box and category probability of the object in the image through a single forward propagation; by setting the confidence threshold, the model filters out the predicted category with high confidence, and decodes it to output the accurate location and identification category of the object.

[0015] In a preferred embodiment, the multi-scale features of each target area detected in the prediction result are extracted based on ViT, and the image is segmented based on SAM to obtain the classification result of each target area, specifically: the classification information in the prediction result is used as prompt information, and a multi-scale feature representation is extracted from the input image using a visual Transformer, and the global and local features in the image are captured through the ViT multi-layer self-attention mechanism; the prompt information is combined with the extracted image features, the prompt information is encoded into features through Transformer, and fused with the image features, and the fused feature matrix contains a combination of image features and prompt information features; based on the fused features, SAM generates a corresponding segmentation mask and fuses it through multi-scale features; the multi-scale features are fused through an upsampling operation, the feature maps of different scales are adjusted to the same size, and they are weighted and fused to obtain a high-quality image segmentation result.

[0016] In a preferred embodiment, based on the fused features, SAM generates a corresponding segmentation mask, and through multi-scale feature fusion, specifically: the target detection image and the SAM segmentation image are completely aligned in space; the bounding box coordinates of each target are extracted from the target detection result; based on the bounding box coordinates, a cropping area is created on the SAM segmentation result image; using the image processing library, the area corresponding to each target bounding box is cropped from the SAM segmentation result image to obtain the pixel category label within the target area; all pixels are traversed, and an array is used to track the count of each category, the extracted pixel category labels are counted, and the number of occurrences of each category label is calculated; the confidence of each pixel category is provided by the segmentation model, and the main category of the target area is obtained based on weighted averaging, and the category with higher confidence will have a greater weight when averaging; the main category of the target area is integrated into the final result together with the position and size of the bounding box of the target detection.

[0017] The technical effects and advantages of the image segmentation method based on the improved SAM model of the present invention are as follows: 1. The present invention uses a remote sensing image classification method based on a convolutional neural network, and with the help of a deep learning model with fast image data processing speed, high detection accuracy, and strong data analysis capabilities, it can achieve rapid classification of remote sensing images, providing strong technical support and data support for environmental monitoring, urban planning, agricultural management and other fields. Functions include: (1) Providing the bounding box coordinates of the target and accurately locating the position of the target. (2) Accurately classifying the detected targets. (3) Rapidly detecting targets in real-time video streams or large-scale image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The figure is a structural schematic diagram of an image segmentation method based on an improved SAM model of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Embodiment 1, Figure 1 The present invention provides an image segmentation method based on an improved SAM model, comprising the following steps: S1, collects remote sensing image datasets of various scenes and categories, and performs data annotation and data enhancement preprocessing operations on remote sensing images; Data labeling is an important part of remote sensing image processing, and its purpose is to provide accurate label information for model training. Labeling methods include: Bounding box annotation: Annotate the target object in the image with a rectangular box.

[0021] Polygonal annotation: For targets with irregular shapes, use polygons for precise annotation.

[0022] Semantic segmentation and annotation: Classify each pixel in the image into categories to achieve refined target segmentation.

[0023] Data enhancement is an important means to improve the generalization ability of the model. It generates more training samples by transforming the original data. The data enhancement preprocessing operations of remote sensing images include: Geometric transformation: such as rotation, scaling, translation, flipping, etc., which changes the spatial position relationship of the image.

[0024] Color transformation: adjust the brightness, contrast, saturation, etc. of the image to simulate images under different lighting conditions.

[0025] Noise addition: Add random noise to the image to improve the model’s robustness to noise.

[0026] Blur processing: Use Gaussian blur, median blur and other methods to simulate image quality degradation.

[0027] The remote sensing image is scaled to the model input size and the pixel values ​​are scaled to a certain range, and the specifications are unified to generate a remote sensing image dataset.

[0028] There are several benefits to data annotation and data enhancement preprocessing of remote sensing images: Improve model performance: Data annotation provides accurate training samples for the model, enabling the improved SAM model to learn more accurate feature representations, thereby improving the accuracy of image segmentation. By annotating objects and targets in remote sensing images, the model can learn richer contextual information, which helps achieve more accurate segmentation in complex scenes.

[0029] Reduce manual intervention: Efficient automatic or semi-automatic annotation tools (such as the improved SAM model itself) can significantly reduce the cost and time of manual annotation, making it possible to annotate large-scale data sets. The higher the quality of the labeled data, the stronger the supervisory signal required for model training, thereby reducing the reliance on manual adjustment of model parameters.

[0030] Promote technological innovation: High-quality labeled data is an important basis for training deep learning models. The improved SAM model based on these data can promote technological innovation in remote sensing image analysis, target detection, change monitoring and other fields.

[0031] Enhance model generalization ability: Data enhancement increases the diversity and complexity of the data set, allowing the improved SAM model to learn more varied feature representations, thereby improving the segmentation performance on unseen remote sensing images. For example, through operations such as rotation, scaling, cropping, and flipping, remote sensing images from different perspectives can be simulated, making the model more robust.

[0032] Improve model training efficiency: Data augmentation can generate more training samples, thereby improving model training efficiency without increasing computing resources. Online data augmentation (i.e., generating augmented samples in real time during training) can further reduce storage space requirements.

[0033] Dealing with data imbalance: There is often an imbalance in the number of objects or targets of different categories in remote sensing images. Through data enhancement, the samples of minority categories can be expanded so that the model can treat each category more equally during training.

[0034] In summary, data annotation and data enhancement preprocessing operations on remote sensing images have many benefits for image segmentation methods based on the improved SAM model. These operations not only improve the performance and training efficiency of the model, but also enhance the generalization ability and robustness of the model, providing strong support for technological innovation in the field of remote sensing image analysis. Therefore, in practical applications, the role of data annotation and data enhancement should be emphasized to give full play to the potential of the improved SAM model.

[0035] S2, use the labeled remote sensing image dataset to train the preset target detection model; The method of using the labeled remote sensing image dataset to train the preset target detection model is as follows: The data set is divided into a training set and a validation set, data is selected from the training set, and the data is passed to the model to obtain the output of the location and category information of the target; According to the output and annotation information of the model, the value of the loss function is calculated, and according to the value of the loss function, the gradient is calculated and back-propagated to the parameters of the model; Update model parameters according to gradients through optimization algorithms, reduce loss functions, control learning rates, and update model parameter step sizes and directions; According to the performance of the validation set, save the optimal model to complete subsequent detection tasks.

[0036] In this experimental case, the core of the deployed network structure is the convolutional neural network (CNN) encoding layer. This network is used to extract features from images, and the improved CSPDarknet feature extractor is adopted. The core of CSPDarknet is the cross-stage partial connection (CSP), which can effectively segment feature maps and perform feature fusion, thereby improving the efficiency and expression ability of feature extraction.

[0037] The cross loss entropy formula is as follows: (1) In formula (1): is the label of sample i; For sample The probability of predicting the positive class.

[0038] (2) In formula (2): is the number of categories; is a symbolic function; For the observed sample Belongs to category The predicted probability of .

[0039] S3 uses the trained target detection model to predict the remote sensing image, identify the target area and category in the image, and output the prediction result.

[0040] The target area and category in the recognition image are outputted as a prediction result, specifically: Resize and normalize the input image to meet the input requirements of the model; Load the trained object detection model and input the preprocessed images into the model for inference; Decode the model predictions and output the predicted class with high confidence.

[0041] Using the trained target detection model to predict remote sensing images and identify target areas and categories in images has the following benefits: Zero-shot or few-shot learning: The improved SAM model has the generalization ability of "zero-shot" or "few-shot", which means that it can still perform effective image segmentation and recognition when encountering targets that have never appeared in the training set. This is especially important for unknown or rare targets that may exist in remote sensing images.

[0042] Broad applicability: Since the SAM model already understands the general concept of objects, it is able to cover a wide range of use cases, including various target types in remote sensing images, such as buildings, vehicles, vegetation, etc.

[0043] Fast response: The improved SAM model can generate corresponding segmentation results based on any prompts in a very short time (e.g., within 50 milliseconds), which is crucial for processing large-scale remote sensing image datasets.

[0044] High-precision segmentation: By introducing technologies such as densely connected networks and feature enhancement modules, the improved SAM model can extract image features more accurately and improve the precision and accuracy of segmentation.

[0045] Multiple prompting methods: Users can provide prompts to the model through multiple methods such as clicking, drawing bounding boxes, and entering text, thereby achieving accurate segmentation of specific targets in remote sensing images.

[0046] Interactive segmentation: The improved SAM model supports interactive segmentation. Users can make real-time adjustments and optimizations during the segmentation process to obtain more satisfactory segmentation results.

[0047] Large-scale datasets: The SA-1B dataset released by Meta contains more than 1.1 billion segmentation masks, providing rich training data for the improved SAM model. These datasets not only improve the training effect of the model, but also enhance its robustness in practical applications.

[0048] Iterative optimization: By repeating the “label-update” cycle multiple times, the quality of the model and dataset can be continuously improved, further improving the accuracy and efficiency of remote sensing image target detection.

[0049] Multimodal fusion: The improved SAM model focuses on video and image recognition, and combined with the current rapidly developing large language models (such as the GPT series), it is expected to achieve the integrated development of multimodal AI. This will provide more comprehensive and in-depth information support for target detection and recognition of remote sensing images.

[0050] In summary, the image segmentation system based on the improved SAM model has the advantages of strong generalization ability, efficient image segmentation performance, flexible interaction mode and optimized data set support when processing remote sensing images. These advantages will greatly improve the accuracy and efficiency of remote sensing image target detection and promote the further development of remote sensing image processing technology.

[0051] S4, extracting multi-scale features based on ViT for each target area detected in the prediction result, and performing image segmentation based on SAM to obtain the classification result of each target area; The multi-scale features of each target area detected in the prediction result are extracted by ViT, image segmentation is performed, masks are generated by SAM, and the target area is classified at the pixel level, specifically: The classification information in the prediction results is used as prompt information, and the Vision Transformer (ViT) is used to extract multi-scale feature representations from the input image. ViT can effectively capture the global and local features of the image, provide rich feature representations, and provide a basis for subsequent image segmentation tasks; Combine the prompt information with the extracted image features, use the flexibility of Transformer to encode the prompt information into features, and fuse them with the image features; Based on the fused features, SAM (Segment Anything Model) generates the corresponding segmentation mask and further improves the accuracy and robustness of the segmentation results through multi-scale feature fusion; Multi-scale features are fused through upsampling operations, feature maps of different scales are adjusted to the same size, and they are weightedly fused to generate high-quality image segmentation results.

[0052] In this experimental case, the convolutional neural network model generates bounding boxes and category predictions of objects in the image through a single forward propagation. Then, SAM extracts the features of these areas based on the bounding boxes output by the target detection model. The convolutional neural network and deconvolution layers are used to generate the corresponding segmentation masks, so that the segmentation results can more accurately distinguish different objects or regions in the image.

[0053] ViT can effectively capture the global and local features of an image, specifically: In ViT, the original self-attention mechanism is composed of , and The specific process is as follows: Linear transformation: (3) in, is the vector representation of the image patch, , and is a learnable weight matrix.

[0054] Attention weight calculation: (4) In formula (4), yes The dimension of the vector, The function converts the degree of association between each image patch and other image patches into weights, and then uses these weights to Weighted sum of rows.

[0055] Residual connections and layer normalization: The original image block vector Add it to the attention weighted result, then go through residual connection and layer normalization: (5) Through the above steps, ViT can effectively capture the global and local features of the image, provide rich feature representations, and lay the foundation for subsequent image segmentation tasks.

[0056] The mask generated in S4 is superimposed on the prediction result generated in S3, and the two results are combined to form the final classification segmentation result.

[0057] The results of target detection and SAM segmentation are integrated as follows: Completely align the target detection image and the SAM segmentation image in space; Extract the bounding box coordinates of each object from the object detection results; Create a cropping region on the SAM segmentation result image based on the bounding box coordinates; Use the image processing library to crop the area corresponding to each target bounding box from the SAM segmentation result image and obtain the pixel category label within the target area; Iterate over all pixels and use an array to track the count of each category, count the extracted pixel category labels, and calculate the number of occurrences of each category label; The confidence of each pixel category is provided by the segmentation model, and the main category of the target area is obtained based on weighted average, and the category with higher confidence will have a greater weight in the average; The main category of the object region is integrated into the final result together with the location and size of the bounding box of the object detection.

[0058] In this experimental case, the final output of the method includes target detection results, confidence, image segmentation results, object location and other related information.

[0059] S5, taking the classification result of each target area as the returned detection result.

[0060] The final classification and segmentation results are visualized, including: 1. Use images or image overlays The classification and segmentation results are directly displayed in the form of images, where different categories are represented by different colors or marks.

[0061] If the original image also needs to be displayed, the segmentation result can be superimposed on the original image for comparison and observation.

[0062] Implementation steps: Use image processing libraries (such as OpenCV, PIL / Pillow, etc.) to read the original image and the segmentation result image.

[0063] According to the category label in the segmentation result image, the pixels of each category are replaced with the corresponding color or mark.

[0064] If superimposed display is required, superimpose the processed segmentation result image with the original image.

[0065] Save or display the final visualization results.

[0066] 2. Draw a bounding box or outline For the classification and segmentation results in the object detection task, you can draw a bounding box to mark the detected object and display the segmentation results within the bounding box.

[0067] Optionally, you can draw the outline of the object to show the segmentation result in more detail.

[0068] Implementation steps: Extract the coordinates of bounding boxes or contours from object detection results.

[0069] Draw bounding boxes or contours on the image using plotting libraries such as matplotlib, seaborn, Plotly, etc.

[0070] Fill the bounding box or contour with the corresponding color or mark to indicate the segmentation result.

[0071] Save or display the final visualization results.

[0072] 3. Create a heat map or density map If the classification and segmentation results contain continuous numerical information (such as probability, confidence, etc.), heat maps or density maps can be used to display the distribution of this information.

[0073] Implementation steps: Map the numerical information in the classification and segmentation results to colors and create a color mapping table.

[0074] Use the plotting library to draw a heat map or density map on an image, where different colors represent different values.

[0075] Adjust the range and resolution of the color map as necessary for optimal visualization.

[0076] Save or display the final visualization results.

[0077] 4. Use interactive visualization tools Use interactive visualization tools such as Tableau, Power BI, Plotly Dash, etc. to create interactive visualization charts so that users can freely explore and analyze data.

[0078] Implementation steps: Import the classification and segmentation results into an interactive visualization tool.

[0079] Choose an appropriate chart type (such as scatter plot, heat map, map, etc.) to display the data.

[0080] Configure the chart's style, color, label and other properties to enhance the visualization effect.

[0081] Add interactive features such as filtering, zooming, dragging, etc. so that users can explore the data more deeply.

[0082] Publish or share your visualizations so that other users can access and view them.

[0083] Visualizing the final classification and segmentation results has many significant benefits, which are not limited to data analysis and understanding, but also involve communication, verification, and improvement. Here are some of the main benefits: Intuitive understanding: Visualization makes the results of classification and segmentation intuitive, so that researchers, developers, and non-technical decision makers can quickly understand the performance of the model. By displaying different categories with color coding or different markers, people can easily identify which areas are correctly classified and which areas have misclassification or fuzzy boundaries.

[0084] Error diagnosis: Errors and anomalies are often easy to spot during visualization. For example, misclassified pixels, fuzziness of category boundaries, overlaps between categories, etc. can all be identified through visualization. This helps locate the problem and serves as a basis for subsequent improvements.

[0085] Communication convenience: Visualization results are a powerful communication tool. It allows team members to share and analyze results without having to deeply understand complex algorithms or model parameters. By showing visual results, consensus can be reached faster and collaboration between teams can be promoted.

[0086] Verify model performance: Visualization results provide an intuitive way to verify the performance of the model. By comparing the visualization results with the true labels or expected results, the accuracy, robustness, and generalization ability of the model can be evaluated. This is crucial for the iterative optimization and final selection of the model.

[0087] User feedback: In some applications, such as medical image analysis and autonomous driving, visualization results can also be directly displayed to users. Users can provide feedback based on the visualization results to help improve the performance of the model and user experience. This interactive feedback loop helps to continuously improve the practicality and reliability of the model.

[0088] Display results: Visualization is an important way to display research and project results. Through exquisite visualization charts and images, more attention and interest can be attracted, and the influence and visibility of research results can be improved. At the same time, this also provides strong support for subsequent academic publications and patent applications.

[0089] In summary, visualizing the final classification and segmentation results has many benefits. It is not only an important means of data analysis and understanding, but also a key tool for communication, verification, improvement and presentation of results.

[0090] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0091] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product.

[0092] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0093] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0094] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0095] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image segmentation method based on an improved SAM model, characterized in that: The following steps are involved: Collect remote sensing image data sets of various scenes and categories, and perform data annotation and data enhancement preprocessing operations on remote sensing images; Use the labeled remote sensing image dataset to train the preset target detection model; Use the trained target detection model to predict remote sensing images, identify target areas and categories in the images, and output prediction results; For each target area detected in the prediction result, multi-scale features are extracted based on ViT, and image segmentation is performed based on SAM to obtain the classification result of each target area; The classification result of each target area is used as the returned detection result.

2. The image segmentation method based on the improved SAM model according to claim 1, characterized in that: The collecting of remote sensing image data sets of various scenes and categories specifically includes: collecting remote sensing image data containing various scenes and categories, scaling the remote sensing images to a model input size and scaling the pixel values ​​to a preset range, and generating the remote sensing image data set.

3. The image segmentation method based on the improved SAM model according to claim 2, characterized in that: The method of using the labeled remote sensing image dataset to train the preset target detection model is as follows: The remote sensing image dataset is divided into a training set and a validation set, and the training set is passed to the model to obtain the output of the location and category information of the target; According to the output and annotation information of the model, the value of the loss function is calculated, and according to the value of the loss function, the gradient is calculated and back-propagated to the parameters of the model; Update model parameters according to gradients through optimization algorithms, reduce loss functions, control learning rates, and update model parameter step sizes and directions; Obtain and save the optimal model based on the validation set to complete subsequent detection tasks.

4. The image segmentation method based on the improved SAM model according to claim 3, characterized in that: The target area and category in the recognition image are outputted as a prediction result, specifically: Resize and normalize the input image; Load the trained object detection model, input the preprocessed image into the model for processing, and obtain the model prediction result; Decode and output the high-confidence prediction category results predicted by the model.

5. The image segmentation method based on the improved SAM model according to claim 4, characterized in that: The pre-processed image is input into the model for processing, specifically: The convolutional neural network model simultaneously predicts the bounding box and class probability of objects in the image through a single forward propagation; By setting the confidence threshold, the model selects predicted categories with high confidence and decodes them to output the accurate location and identification category of the object.

6. The image segmentation method based on the improved SAM model according to claim 5, characterized in that: The multi-scale features of each target area detected in the prediction result are extracted based on ViT, and the image is segmented based on SAM to obtain the classification result of each target area, specifically: The classification information in the prediction results is used as prompt information, and the visual Transformer is used to extract multi-scale feature representations from the input image, and the global and local features in the image are captured through the ViT multi-layer self-attention mechanism; Combine the prompt information with the extracted image features, encode the prompt information into features through Transformer, and fuse them with the image features. The fused feature matrix contains the combination of image features and prompt information features. Based on the fused features, SAM generates the corresponding segmentation mask and fuses it through multi-scale features; Multi-scale features are fused through upsampling operations, feature maps of different scales are adjusted to the same size, and they are weightedly fused to obtain high-quality image segmentation results.

7. The image segmentation method based on the improved SAM model according to claim 6, characterized in that: Based on the fused features, SAM generates the corresponding segmentation mask and fuses the multi-scale features, specifically: Completely align the target detection image and the SAM segmentation image in space; Extract the bounding box coordinates of each object from the object detection results; Create a cropping region on the SAM segmentation result image based on the bounding box coordinates; Use the image processing library to crop the area corresponding to each target bounding box from the SAM segmentation result image and obtain the pixel category label within the target area; Iterate over all pixels and use an array to track the count of each category, count the extracted pixel category labels, and calculate the number of occurrences of each category label; The confidence of each pixel category is provided by the segmentation model, and the main category of the target area is obtained based on weighted average, and the category with higher confidence will have a greater weight in the average; The main category of the object region is integrated into the final result together with the location and size of the bounding box of the object detection.

8. The image segmentation method based on the improved SAM model according to claim 7, characterized in that: The specific calculation formula of the loss function is as follows: Where: L is the loss function, is the label of sample i; For sample The probability of predicting the positive class, is the number of categories; is a symbolic function; For the observed sample Belongs to category The predicted probability of .

9. The image segmentation method based on the improved SAM model according to claim 8, characterized in that: The multi-layer self-attention mechanism is specifically: The self-attention mechanism is , and The specific process is as follows: Linear transformation: Attention weight calculation: Residual connections and layer normalization: in, is the vector representation of the image patch, , and is a learnable weight matrix, yes The dimension of the vector, The function converts the degree of association between each image block and other image blocks into weights, and uses these weights to Weighted sum of rows.

Images