Remote sensing image photovoltaic identification method based on SAM model

Abstract

The invention provides a remote sensing image photovoltaic identification method based on an SAM model, and relates to the technical field of computer vision, and the method comprises the steps: collecting and preprocessing a multi-resolution photovoltaic segmentation data set, and enabling the data set to be used for adjusting an image encoder; constructing an SAM segmentation network comprising the image encoder, the prompt encoder and the mask decoder, and inputting interactive click information and the data set into the SAM segmentation network; adding a segmentation output layer into the mask decoder to generate a preliminary segmentation result; and feeding the preliminary segmentation result back to an interaction module for interactive adjustment, and decoding again after adjustment to generate a final segmentation result.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and specifically relates to a photovoltaic recognition method for remote sensing images based on a SAM model. Background Art

[0002] With the development of renewable energy, photovoltaic power generation has become one of the important forms of energy. Photovoltaic power generation systems mainly include distributed photovoltaic and centralized photovoltaic. In order to better supervise and manage these photovoltaic facilities, the application of computer vision technology and remote sensing images has become increasingly important. Through these technologies, photovoltaic equipment can be effectively monitored, maintained, and fault detected.

[0003] At present, most solar photovoltaic segmentation technologies based on remote sensing images use deep learning algorithms based on supervised learning. Such algorithms usually segment photovoltaic facilities in a specific area and are therefore highly dependent on labeled training data sets. This includes multiple steps such as data collection, data cleaning, and data labeling, among which data labeling is particularly time-consuming and requires delicate operations.

[0004] Existing remote sensing photovoltaic segmentation technology is highly dependent on already labeled training data when applied. This means that if you want to apply it in a new region or scenario, you need to collect and label a large amount of data again, which is not only time-consuming but also increases the cost of the project.

[0005] Manually labeling photovoltaic areas for complex remote sensing images is a tedious and error-prone process. In addition, when it comes to large-scale image collections, ensuring the consistency and accuracy of annotations is also a challenge. Even with the support of an experienced annotation team, it takes a long time to complete the annotation of large-scale datasets, which poses a challenge to the rapid deployment and application of models. Summary of the invention

[0006] The present application provides a remote sensing image photovoltaic recognition method based on the SAM model to solve the problems of excessive reliance on labeled data, difficulty in data labeling and long labeling cycle in traditional remote sensing photovoltaic segmentation technology in supervised learning applications.

[0007] The technical solution adopted in this application is:

[0008] The present application embodiment provides a remote sensing image photovoltaic recognition method based on the SAM model, comprising:

[0009] Collect and preprocess a publicly available multi-resolution remote sensing photovoltaic segmentation dataset, and use the dataset to construct pre-trained weights for photovoltaic recognition to optimize the image encoder;

[0010] Constructing a SAM segmentation network including the image encoder, the hint encoder and the mask decoder, and inputting the interactive click information and the dataset into the SAM segmentation network;

[0011] adding a segmentation output layer to the mask decoder to generate a preliminary segmentation result;

[0012] The preliminary segmentation result is fed back to the interactive module for interactive adjustment, and after the adjustment, it is re-decoded to generate the final segmentation result.

[0013] The remote sensing image photovoltaic recognition method based on the SAM model provided in the present application also includes the following additional technical features: the multi-resolution photovoltaic segmentation data set is collected and preprocessed, and the data set is used to construct pre-trained weights for photovoltaic recognition to optimize the image encoder, specifically: multi-resolution photovoltaic segmentation data of any region disclosed is collected to form a multi-resolution photovoltaic segmentation data set;

[0014] Screening the data set to obtain appropriate image data, removing noise from the image data using a bilateral filter and performing standardization processing so that the image data have the same length and width;

[0015] The pre-trained model is fine-tuned using the processed image data, and the original pre-trained weights of the pre-trained model are adjusted to optimize the image encoder.

[0016] According to one embodiment of the present application, the constructing a SAM segmentation network including the image encoder, the hint encoder and the mask decoder, and inputting the interactive click information and the data set into the SAM segmentation network, specifically comprises:

[0017] Through the interactive operation of the interactive module, positive and negative sample pixel points in the data set are obtained to form interactive click information, and the prompt encoder is constructed based on the interactive click information;

[0018] The image data and the interactive click information are transmitted to the image encoder and the prompt encoder to obtain encoding information.

[0019] According to one embodiment of the present application, a segmentation output layer is added to the mask decoder to generate a preliminary segmentation result, specifically: a segmentation output layer is added to the mask decoder to output semantic information, and the semantic information is superimposed with the image data when outputting the mask through a multi-layer perception mechanism to output the preliminary segmentation result.

[0020] According to an embodiment of the present application, the semantic information is superimposed on the image data when outputting the mask through the multi-layer perception mechanism, specifically:

[0021] The image data is transmitted to the image encoder through the interaction module;

[0022] The image data is processed in blocks and embedded, that is, the input image is converted into an embedding vector, and the embedding vector is enhanced by adding absolute position information to obtain an encoded vector; a series of convolution operations and layer normalization are performed through the connection module, and finally the corresponding feature map is output.

[0023] According to an embodiment of the present application, the hint encoder further includes: recording parameters for the input point information, including embedding dimensions and image embedding size, and initializing the position encoding layer and the hint embedding layer;

[0024] The mask decoder receives the channel dimension, model type, number of predicted masks, upsampling activation function, and depth and hidden dimension parameters during the initialization phase, and defines the upsampler and list, as well as the prediction header for predicting the mask quality;

[0025] The mask decoder receives the output image embedding, position encoding, point embedding and mask input embedding of the image encoder, generates a mask and mask quality prediction, outputs corresponding semantic information, and selects a suitable mask slice;

[0026] The mask decoder returns the generated mask, the mask quality prediction, and the prediction results of the semantic information.

[0027] According to an embodiment of the present application, the preliminary segmentation result is fed back to the interactive module for interactive adjustment, and then re-decoded after adjustment to generate the final segmentation result, specifically:

[0028] The positive sample pixels or negative sample pixels in the preliminary results are adjusted through the interactive module to improve the segmentation accuracy;

[0029] re-inputting the adjusted information into the mask decoder for decoding, the mask decoder receiving information from the image encoder and the hint encoder and combining the adjusted information to generate a new segmentation mask;

[0030] The mask decoder outputs updated mask information, indicator information, and semantic information.

[0031] A computer program product containing instructions, when running on a device, enables the device to execute the steps in a method for photovoltaic recognition of remote sensing images based on a SAM model.

[0032] A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps in a remote sensing image photovoltaic identification method based on a SAM model.

[0033] The electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the photovoltaic recognition method of remote sensing images based on the SAM model are implemented.

[0034] Due to the adoption of the above technical solution, the beneficial effects achieved by this application are as follows:

[0035] 1. This application can efficiently segment photovoltaic categories through pre-training mechanism and mask interaction. This method not only greatly reduces the labeling time and improves the prediction efficiency, but also reduces the fault tolerance through fine-tuning, thereby significantly improving the availability and reliability of segmented remote sensing image photovoltaic data.

[0036] 2. This application proposes a method for predicting and displaying real-time segmentation results through interactive image clicks and buttons. Combined with the remote sensing photovoltaic segmentation model to obtain semantic information, after the segmentation is completed, the user can browse the segmentation results of different channels and save them by pressing buttons. This method allows the user to quickly and accurately perform interactive segmentation of photovoltaic information on the image, while fine-tuning and correcting the segmentation results, improving the accuracy of zero-sample photovoltaic segmentation in low-resolution conditions.

[0037] 3. This application can identify distributed photovoltaics and centralized photovoltaics at low resolution, and interactively distinguish between identified targets and similar targets to avoid omissions, which is convenient for the management and planning of photovoltaic areas. This method not only improves the recognition efficiency and reduces the number of recognitions, but also can more efficiently and accurately identify remote sensing photovoltaic data.

[0038] 4. This application focuses more on the segmentation of photovoltaic categories in remote sensing images by fine-tuning the pre-training weights. By constructing a mask interaction module, the efficiency of photovoltaic segmentation is effectively improved. At the same time, the improved mask decoder module can output more accurate semantic information. Compared with the original model, this method performs better in the semantic segmentation task of photovoltaic categories and significantly improves the accuracy and reliability of segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1 A flowchart of a photovoltaic recognition method for remote sensing images based on a SAM model provided in an embodiment of the present application;

[0041] Figure 2 An overall structural diagram of a remote sensing image photovoltaic recognition method based on a SAM model provided in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of a neural network structure based on a codec architecture and an attention mechanism for a photovoltaic recognition method for remote sensing images based on a SAM model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to more clearly illustrate the overall concept of the present application, a detailed description is given below in an illustrative manner in conjunction with the accompanying drawings.

[0044] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited by the specific embodiments disclosed below. It should be noted that the embodiments of the present application and the features in each embodiment may be combined with each other without conflict.

[0045] In the present application, unless otherwise clearly specified and limited, a first feature "above" or "below" a second feature may be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in an appropriate manner in any one or more embodiments or examples.

[0046] like Figure 1 As shown, a photovoltaic recognition method of remote sensing images based on the SAM model includes:

[0047] A multi-resolution photovoltaic segmentation dataset is collected and preprocessed, and the dataset is used to tune the image encoder.

[0048] Specifically, data collection: The public dataset Photovoltaic Panel (PVP) Dataset is used as a pre-training sample, which provides 4640 photovoltaic cell sample images from 13 provinces in China. Each sample consists of a 512×512 pixel image and a corresponding photovoltaic panel label.

[0049] Data preprocessing: Screening: Select data suitable for remote sensing photovoltaic image segmentation from the dataset.

[0050] Denoising: Use a bilateral filter to remove noise from the image, keeping the edges clear while reducing unnecessary interference factors.

[0051] Standardization processing: Standardize the remote sensing image data so that all images have the same aspect ratio (H, W) in order to unify the processing standards.

[0052] Data format consistency: Crop or expand the image to a size of 512×512 to make it have a consistent scale for subsequent model training.

[0053] Construction of pre-trained weights: The construction of pre-trained weights (Wf) is based on the above processed data by fine-tuning the original pre-trained weights of the SAM model. This process helps to enhance the image encoder (Ei)'s ability to extract features of remote sensing photovoltaic images, thereby better adapting to photovoltaic recognition tasks.

[0054] For example, data collection

[0055] Data collection usually involves obtaining suitable remote sensing photovoltaic image data from public data sources. For example, the Photovoltaic Panel (PVP) Dataset can be used as a pre-training sample. This dataset contains sample images of photovoltaic cells from multiple provinces in China, each sample is 512×512 pixels, and is accompanied by a corresponding photovoltaic panel label.

[0056] Data preprocessing

[0057] The preprocessing process aims to ensure the quality of the dataset so that the image encoder can be trained effectively. The specific steps include:

[0058] Filtering: Filter out suitable remote sensing photovoltaic image data. For example, those with poor image quality due to weather conditions or other reasons may be excluded.

[0059] Denoising: Use a bilateral filter to remove noise from the data. A bilateral filter is an image smoothing filter that reduces unwanted interference while keeping edges sharp.

[0060] Standardization: Standardize the remote sensing image data to make them have the same length and width (H, W). The purpose of this is to make all images have a consistent format before entering the model, so as to facilitate model learning.

[0061] Building pre-trained weights

[0062] The construction of the pre-trained weights Wf is based on the above processed data. Specifically, the pre-trained weights are constructed by fine-tuning the original pre-trained weights of SAM. The goal of this step is to improve Ei's ability to extract features of remote sensing photovoltaic images, making it more focused on the photovoltaic category.

[0063] Image encoder tuning

[0064] The adjustment of the image encoder Ei refers to training and fine-tuning the pre-trained model through the public remote sensing photovoltaic dataset (PVP Dataset). In this process, Ei will be optimized to better capture and represent the key features in photovoltaic images. For example, through training and fine-tuning, Ei can learn to recognize the unique texture or color patterns of photovoltaic panels, thereby improving the accuracy of segmentation.

[0065] Practical examples

[0066] Suppose we have a dataset of 512×512 pixel images that contain both PV panels and non-PV areas. First, we filter out images that are clear and have no obvious occlusions, and then use a bilateral filter to remove noise from the image. Next, the image is normalized, for example by subtracting the mean and dividing by the standard deviation. After completing these preprocessing steps, we input the image into the model and begin fine-tuning Ei so that it can more accurately extract the features of the PV panels. In this way, we can gradually improve the performance of the model, making it more accurate in identifying PV panels.

[0067] Furthermore, the model can be verified and evaluated: after fine-tuning the image encoder, a portion of unseen data can be used to test the generalization ability of the model to ensure that the model not only performs well on the training data, but also can handle unknown data well.

[0068] Furthermore, the robustness of the model can be enhanced: by introducing data augmentation techniques such as rotation, scaling, flipping and other transformations, the training data set can be enriched to improve the model's ability to recognize photovoltaic panels under different viewing angles and lighting conditions. This helps to enhance the robustness of the model and make it more reliable in practical applications.

[0069] Furthermore, different feature extraction strategies can be explored: although the current model has optimized the feature extraction capability for remote sensing photovoltaic data, different feature extraction strategies can be further studied, such as combining multimodal information (such as spectral information, texture features, etc.) to obtain better segmentation effects.

[0070] Furthermore, model fusion can be used: you can try to integrate multiple fine-tuned models for learning, and use model fusion technology to improve the overall recognition accuracy. For example, you can use Bagging or Boosting methods to combine multiple weak classifiers to form a stronger classifier.

[0071] Furthermore, real-time performance optimization can be achieved: considering that remote sensing images are often large in size, the real-time processing capability of the model can be optimized without affecting the model performance. For example, the response speed of the model in practical applications can be improved through lightweight model design and hardware acceleration.

[0072] A SAM segmentation network consisting of an image encoder, a hint encoder, and a mask decoder is constructed, and the interactive click information and the dataset are input into the SAM segmentation network.

[0073] Specifically, the image encoder E i :

[0074] Function: The image encoder is responsible for extracting features from the input image.

[0075] Operation: In E i In the above, the image is first divided into blocks and embedded, which is converted into an embedding vector E. Then, the absolute position embedding P is enabled to add the corresponding position information, and finally the encoded vector V is obtained.

[0076] Process: The image is then processed by a series of Block modules including input vectors, output vectors, layer normalization, multi-head attention mechanism, and multi-layer perceptron. If the window size is set, the attention mechanism in the block will perform window division and restoration. Finally, through a series of convolution operations and layer normalization, the corresponding feature map F is output.

[0077] Prompt Encoder (E p ):

[0078] Function: Prompt the encoder to process the i User click information of the module.

[0079] Operation: Ep receives point information and records the embedding dimension and image embedding size IS, initializes the position encoding layer and hint embedding layer.

[0080] Process: Get dense position encoding, input I S The shape is (E d ,H,W) of random position encoding, and the shape is (1,E d ,H,W). Then, fill the point set, increase the coordinates to move it to the center of the pixel, and add an extra point with coordinates [0,0] and label -1 for filling. Calculate the batch size bs based on the point prompt, and initialize the shape to (bs,0,E d ) is sparsely embedded in the empty tensor. If there is a point hint, the point information is converted into an embedding vector, otherwise the point information is converted to 0 and finally concatenated into the sparse embedding.

[0081] Mask decoder (DM ):

[0082] Function: The mask decoder is responsible for decoding the encoded information into the final mask information.

[0083] Operation: D M It takes in parameters like channel dimension, model type, number of prediction masks, upsampling activation function, and depth and hidden dimensions, and defines the prediction header for the upsampler and list and quality of the prediction masks.

[0084] Process: During the forward propagation, D M Receive E i The output image embedding, position encoding, point embedding and mask input embedding are used to generate masks and mask quality predictions, output corresponding semantic information, and select appropriate mask slices. Finally, the generated masks, mask quality predictions and semantic information prediction results are returned.

[0085] Input data:

[0086] Dataset: Based on the public multi-resolution photovoltaic segmentation dataset of any region, the dataset is preprocessed to ensure data quality. Based on the processed dataset, the pre-trained weights (W) for photovoltaic recognition are constructed. f ).

[0087] Interactive Click Information: Through M i The module obtains positive and negative sample pixels in the image data based on M i The module transmits the acquired click information and establishes a prompt encoder (E p ), load the pre-trained mechanism and predictor instance.

[0088] For example, the image encoder (E i )

[0089] Function: The image encoder is responsible for extracting useful information from the input image.

[0090] For example, when a user uploads a remote sensing image containing a photovoltaic panel, the image encoder will first process the image, such as block and embedding operations, to convert the input image into an embedded vector E. Then, the absolute position information is added to form the encoded vector V. Subsequently, the image data is processed by a series of Block modules, each of which contains an input vector, an output vector, layer normalization, a multi-head attention mechanism, and a multi-layer perceptron. When the window size is set, the attention mechanism in the block will perform window division and restoration processing on the image. Finally, the image passes through the neck module, through a series of convolution operations and layer normalization, and outputs the corresponding feature map F.

[0091] Prompt Encoder (E p )

[0092] Function: The prompt encoder converts the user's interactive click information into a form that the model can understand.

[0093] Example: User passes M i In the module interface, use the keyboard and mouse to interact and select the positive sample pixel point P in the image. + (photovoltaic area) and negative sample pixel P - (Non-PV area). E p Receive these point information and record the embedding dimension and image embedding size I S , initialize the position encoding layer and the hint embedding layer. E p Also get the dense position encoding, input I S , and the shape is (E d ,H,W) random position encoding. By increasing the dimension, the shape is (1,E d ,H,W). Next, fill the point set, adjust the coordinates to move to the center of the pixel, add an extra point with coordinates [0,0] and label -1 for filling. Calculate the batch size bs based on the point prompt, and initialize the shape to (bs,0,E d ). If there is a point hint, the point information is converted to an embedding vector; if there is no point hint, the point information is converted to 0 and concatenated into the sparse embedding.

[0094] Mask decoder (D M )

[0095] Function: The mask decoder generates the final segmentation result from the encoded information.

[0096] Example: D M Receive E i The output image embedding, position encoding, point embedding and mask input embedding of D generate mask and mask quality prediction, output corresponding semantic information, and select appropriate mask slices. Finally, D M The generated mask, mask quality prediction, and semantic information prediction results are returned.

[0097] Input Data

[0098] Dataset: The public Photovoltaic Panel (PVP) Dataset is used as a pre-training sample. The dataset provides 4640 photovoltaic cell sample images from 13 provinces in China. Each sample is a 512×512 pixel image with a corresponding photovoltaic panel label.

[0099] Interactive Click Information: Through M iThe module obtains positive and negative sample pixels in the image data and transmits these click information to E i and E p , using the improved D M Decode the encoded information, output preliminary mask information, indicator information and semantic information, and pass the decoded preliminary results back to M i Modules are displayed as images on the user interface.

[0100] Through the above steps, an efficient SAM segmentation network is constructed, which is able to utilize the user's interactive click information to improve the accuracy of photovoltaic panel segmentation. This approach not only reduces the need for a large amount of labeled data, but also improves the flexibility and user-friendliness of the model.

[0101] Furthermore, the model can be optimized and fine-tuned: continue to optimize the model parameters and find the best hyperparameter configuration through more experiments. For specific types of photovoltaic panels, the model can be further fine-tuned to make it better at identifying specific types of photovoltaic panels.

[0102] Furthermore, data enhancement can be performed: data enhancement techniques (such as rotation, scaling, brightness adjustment, etc.) can be used to increase the diversity of the training set and improve the robustness and generalization ability of the model.

[0103] Furthermore, performance evaluation and comparison can be performed: a complete performance evaluation system is developed to compare with other existing photovoltaic identification methods to verify the effectiveness of the method. The performance of the model is comprehensively evaluated, including accuracy, recall, F1 score and other indicators.

[0104] Furthermore, real-time performance can be improved: according to the needs of real-time applications, further optimize the algorithm, reduce processing time, and improve real-time performance. Consider hardware acceleration technology, such as GPU acceleration or specialized AI chips, to speed up model inference.

[0105] Furthermore, multimodal fusion is possible: explore combining other types of data (such as meteorological data, geographic information, etc.) with remote sensing images, and use multimodal data to improve recognition accuracy.

[0106] Furthermore, the user interaction experience can be improved: the user interaction interface can be improved so that users can see the model prediction results more intuitively and can easily make interactive fine-tuning. More interactive elements, such as drag-and-drop selection and multi-touch, are introduced to make user operation more convenient.

[0107] Furthermore, we can also automate annotation tools: develop auxiliary tools to help users annotate data more quickly and reduce the workload of manual annotation. We can use semi-automatic or fully automatic methods, combined with a small amount of manual verification, to improve the efficiency and accuracy of annotation.

[0108] Furthermore, cloud service deployment is possible: deploy the model to the cloud and provide API services so that users can upload images and obtain recognition results through the Internet. Design a user rights management system for the cloud platform to protect data security.

[0109] Furthermore, mobile application development can also be carried out: mobile applications can be developed so that users can directly take photos and perform photovoltaic identification through mobile devices such as mobile phones. The user experience of mobile applications can be optimized to ensure the compatibility and stability of applications on different devices.

[0110] Furthermore, cross-domain transfer learning can be used to study how to transfer models from one environment to another to adapt to changes in different regions or seasons. Research on cross-domain transfer learning can be carried out to improve the applicability of models on different data sets.

[0111] Furthermore, multi-task learning can also be performed: exploring the possibility of multi-task learning, such as performing photovoltaic identification and other related tasks (such as building detection, etc.) at the same time to improve the versatility of the model.

[0112] A segmentation output layer is added to the mask decoder to generate preliminary segmentation results.

[0113] Specifically, in the mask decoder (D M ) to generate preliminary segmentation results is an important step in the photovoltaic recognition method of remote sensing images based on the SAM model. Specifically, the segmentation output layer is added to enable the mask decoder to output more accurate semantic information, which helps to improve the performance of the model in the photovoltaic category semantic segmentation task. The following is a detailed explanation:

[0114] The role of the segmentation output layer:

[0115] The segmentation output layer is mainly used to output semantic information, that is, to classify different objects in the image and generate corresponding mask images. In this case, the main purpose is to generate a mask image of a photovoltaic panel.

[0116] Use of Multilayer Perceptron:

[0117] When generating the segmentation result, the multi-layer perceptron (MLP) will superimpose the output mask with the original image to obtain the final prediction result. This approach allows the model to not only focus on pixel-level classification, but also combine contextual information to make more reasonable segmentation judgments.

[0118] The process of generating preliminary segmentation results:

[0119] When the mask decoder receives E iAfter taking the image embedding output by (image encoder), the position encodings, the point embeddings, and the embeddings of the mask input, it generates masks and mask quality predictions.

[0120] D M It also outputs the corresponding semantic information and selects the appropriate mask slice. M The generated mask, mask quality prediction, and semantic information prediction results are returned.

[0121] These preliminary segmentation results are then passed back to M i The module is then displayed as a graph on the user interface. Users can interactively fine-tune these preliminary results to further optimize the model output.

[0122] By adding a segmentation output layer to the mask decoder, the model is able to generate more accurate segmentation results for photovoltaic panels. This approach not only improves the accuracy and reliability of the segmentation results, but also enhances the semantic understanding ability of the model, making the model perform better when handling photovoltaic category segmentation tasks.

[0123] For example, in the mask decoder (D M ) to generate preliminary segmentation results is an important step in the photovoltaic recognition method of remote sensing images based on the SAM model. The following is a detailed example:

[0124] Step by step guide

[0125] Initialize D M parameter:

[0126] In the initialization phase, D M It receives parameters such as channel dimension, model type, number of prediction masks, upsampling activation function, and depth and hidden dimensions, and defines the upsampler and list, as well as the prediction header for the quality of the prediction mask.

[0127] Forward propagation process:

[0128] D M During the forward propagation process, E is received i The output image embedding of (image encoder), the position encoding, the point embedding and the embedding of the mask input.

[0129] D M Generate masks and mask quality predictions, output corresponding semantic information, and select appropriate mask slices.

[0130] Finally, D M The generated mask, mask quality prediction, and semantic information prediction results are returned.

[0131] Specific implementation examples

[0132] Suppose we are processing a remote sensing image containing photovoltaic panels and want to segment them using the SAM model.

[0133] Input data:

[0134] A 512×512 pixel remote sensing image is input to the image encoder E i In, E i Process the image and output the image embedding.

[0135] At the same time, through M i The module user marks several positive sample points (P + , indicating photovoltaic area) and negative sample points (P - , indicating non-PV areas), the information of these points is input as prompts to the prompt encoder E p middle.

[0136] Processing process:

[0137] E p The user-marked point information is converted into a prompt vector and passed to D M .

[0138] D M Received from E i Image embedding and E p A prompt vector is generated through a series of processing steps to generate a preliminary segmentation result.

[0139] In D M In , a segmentation output layer is added, which superimposes the output mask with the original image through a multi-layer perceptron (MLP) to generate the prediction result.

[0140] This process also includes generating mask quality predictions to facilitate subsequent optimization.

[0141] Generates the following result:

[0142] D M The final output is a preliminary segmentation result, including the mask map of the photovoltaic panel, mask quality prediction, and related semantic information.

[0143] These preliminary results were sent back to M i blocks and displayed as images on the user interface.

[0144] User Interaction:

[0145] Users can interactively fine-tune based on preliminary results and further optimize model output by adjusting positive and negative sample points or other parameters.

[0146] The fine-tuned information is passed to D againM Re-decode to generate the final result, and display it in the form of an image on the interface.

[0147] By adding a segmentation output layer to the mask decoder, the model can generate more accurate preliminary segmentation results, thereby improving the accuracy and reliability of the entire recognition system.

[0148] Furthermore, model optimization and performance improvement can be achieved: Hyperparameter adjustment: The model's hyperparameters can be tuned to find the best model configuration through grid search or Bayesian optimization. Multi-scale input: Image inputs of different scales are introduced to enhance the model's ability to recognize photovoltaic panels of different sizes.

[0149] Furthermore, data enhancement and expansion can be performed: Diversified data enhancement: Use data enhancement techniques (such as rotation, flipping, color jittering, etc.) to increase the diversity and robustness of training data.

[0150] Introduce external data: Collect more remote sensing images from different regions and under different lighting conditions to expand the training data set and improve the generalization ability of the model.

[0151] Furthermore, model fusion and ensemble learning can be performed: Multi-model fusion: Use ensemble learning methods, such as bagging or boosting, to combine the prediction results of multiple SAM models to improve segmentation accuracy. Fusion of other models: Fusion of the SAM model with other high-performance segmentation models (such as U-Net, DeepLab, etc.) to complement each other.

[0152] Furthermore, real-time and resource optimization can be achieved: Model lightweighting: Through model compression technologies such as pruning, quantization, knowledge distillation, etc., the model parameters can be reduced and the operation efficiency can be improved. Hardware acceleration: GPU or dedicated hardware accelerators (such as TPU) are used to accelerate the model reasoning process and improve real-time processing capabilities.

[0153] Furthermore, the user interaction experience can be enhanced: Visualization tools: Develop a more intuitive user interface so that users can view segmentation results in real time and provide immediate feedback. Automated processes: Simplify user operation processes, automate common tasks, reduce human intervention, and improve work efficiency.

[0154] The preliminary segmentation results are fed back to the interactive module for interactive adjustment, and then re-decoded to generate the final segmentation results.

[0155] Specifically, interactive fine-tuning and final result generation

[0156] Preliminary results show:

[0157] The preliminary segmentation results are first displayed in the user interface in the form of an image. Users can see the model's preliminary segmentation of the image, including the mask map of the photovoltaic panel, the mask quality prediction, and related semantic information.

[0158] Interactive Adjustment:

[0159] Users can interactively fine-tune based on preliminary results through M i The module interface adjusts the initial segmentation results. This includes but is not limited to adjusting the positive sample points (P + ) and negative sample points (P - ) or adjust other parameters.

[0160] Users can select pixels in the image and mark them as photovoltaic areas (P + ) or non-PV area (P - ). These operations will generate new click information and feed it back to the system.

[0161] Re-decode:

[0162] The new click information will be entered into E again. i and E p In the example, the improved D M The encoded information is re-decoded to generate a new segmentation result.

[0163] D M The mask information, indicator information, and semantic information will be regenerated based on the adjusted information and passed back to M i module.

[0164] Final results show:

[0165] The final segmentation result will be re-decoded and displayed in the form of an image on the user interface to ensure the accuracy of the recognition result.

[0166] Users can iterate the process of fine-tuning and re-decoding until the segmentation result meets their requirements.

[0167] Through such an interactive adjustment mechanism, users can directly participate in the optimization process of the segmentation results, thereby improving the flexibility and accuracy of the model. This approach not only reduces the need for a large amount of labeled data, but also enhances the adaptability of the model and user satisfaction.

[0168] For example, the initial segmentation results show:

[0169] The user uploaded a remote sensing image containing a photovoltaic panel into the system.

[0170] The SAM segmentation network is composed of an image encoder (E i ), prompt encoder (E p ) and the mask decoder (D M ) processes this image and generates preliminary segmentation results.

[0171] The preliminary segmentation results include the PV panel mask, mask quality prediction, and semantic information, which are displayed in the form of images on the user interface.

[0172] Interactive Adjustment:

[0173] Users observed that some PV panels were not correctly identified or were mis-segmented in the initial segmentation results.

[0174] By M i The module provides an interactive interface where users can use the mouse to select and mark more positive sample points on the image (P + ) and negative sample points (P - For example, a user finds that a certain area should belong to a photovoltaic panel, but it is incorrectly marked as a non-photovoltaic area in the preliminary results. Then the user clicks in this area and marks it as a positive sample point (P + ).

[0175] M i The module transmits the user's newly added click information to the prompt encoder (E p ), and update the hint vector.

[0176] Re-decode:

[0177] The updated hint vector is then fed back into the mask decoder (D M )middle.

[0178] D M According to the new prompt information, the mask information, indicator information and semantic information are regenerated. M The segmentation boundary will be recalculated and adjusted based on the user's newly added click information, so that the segmentation result is closer to the actual position of the photovoltaic panel.

[0179] Generate a new segmentation mask and superimpose it with the original image to generate a new segmentation result.

[0180] Final results show:

[0181] The re-decoded segmentation results are displayed again in the form of images on the user interface.

[0182] Users can observe that the previously mis-segmented parts have been corrected and the outline of the photovoltaic panel is more accurate.

[0183] If the user still thinks the result is not satisfactory, he can continue to make interactive adjustments until he reaches a satisfactory level.

[0184] Through the above steps, users can participate in the correction process of the segmentation results, which improves the accuracy and reliability of the segmentation. This method not only simplifies the reliance on a large amount of labeled data in traditional segmentation methods, but also enhances the flexibility and user-friendliness of the model.

[0185] Furthermore, the interactive adjustment function can be enhanced: Multiple rounds of interaction: Allow users to make multiple rounds of interactive adjustments until a satisfactory segmentation effect is achieved. After each adjustment, the image can be re-decoded to continuously optimize the results. Intelligent suggestions: Combined with the user's historical interactive behavior, intelligent recommendations are made on areas that may need to be adjusted to reduce the user's operational burden.

[0186] Furthermore, the segmentation quality assessment can be optimized: Automatic assessment: Develop automated tools to help users evaluate the quality of segmentation results, such as automatic scoring through indicators such as IoU (Intersection over Union). Visualization tools: Provide more intuitive visualization tools to help users better understand the segmentation results and facilitate user decision-making.

[0187] Furthermore, the model’s adaptive capabilities can be increased: Dynamic learning: enables the model to dynamically adjust weights based on user feedback to further optimize model parameters Online learning: allows the model to continuously learn during user use to improve segmentation accuracy.

[0188] Furthermore, the robustness of the model can be enhanced: Anomaly detection: An anomaly detection mechanism is introduced to automatically identify and prompt users where segmentation errors may exist.

[0189] Furthermore, multimodal fusion can be performed: combining other sensor data (such as infrared images, lidar data, etc.) to improve the accuracy and robustness of segmentation.

[0190] The remote sensing image photovoltaic recognition method based on the SAM model provided in the present application also includes the following additional technical features: collecting and preprocessing a multi-resolution photovoltaic segmentation data set, and using the data set to pre-train an image encoder, specifically: collecting public multi-resolution photovoltaic segmentation data of any region to form a multi-resolution photovoltaic segmentation data set;

[0191] The data set is screened to obtain suitable image data, and the image data is subjected to a bilateral filter to remove noise and perform standardization so that the image data has the same length and width;

[0192] The pre-trained model is trained using the processed image data and the original pre-trained weights of the pre-trained model are adjusted to adjust the image encoder.

[0193] Specifically, data collection

[0194] Collect data: Collect public multi-resolution photovoltaic segmentation data of any region to form a multi-resolution photovoltaic segmentation dataset. These data can be obtained from public datasets such as Photovoltaic Panel (PVP) Dataset. PVP Dataset provides 4640 photovoltaic cell sample images in 13 provinces in China. Each sample is a 512×512 pixel image with a corresponding photovoltaic panel label.

[0195] Data preprocessing

[0196] Data screening: Screening out suitable image data from the collected remote sensing photovoltaic image data. The screening criteria may be factors such as image quality, clarity, and whether it contains sufficient photovoltaic panel information.

[0197] Denoising: Use a bilateral filter to remove noise from the image. The bilateral filter can effectively remove noise from the image while retaining the edge information of the image, which is very important for subsequent segmentation processing.

[0198] Standardization: Standardize the filtered remote sensing image data to make them have the same length and width (H, W). Standardization helps to make all images have consistent sizes for subsequent processing.

[0199] Training and tuning of pre-trained models

[0200] Pre-trained model training: Use the processed image data to train the pre-trained model. The pre-trained model here refers to the image encoder based on the SAM model (E i ), E can be improved through training i Ability to extract features of remote sensing photovoltaic images.

[0201] Adjust pre-training weights: Adjust the original pre-training weights of the pre-trained model. Through fine-tuning, the model can better adapt to the photovoltaic recognition task, thereby improving the accuracy and reliability of segmentation.

[0202] Practical examples

[0203] Assume that we have photovoltaic data from Google Earth Pro, which is divided into images of size 512×512 in order to have a consistent scale. Next, we need to preprocess these images:

[0204] Data screening: Check the quality of each image and exclude those with poor image quality due to weather or other factors.

[0205] Denoising: Use a bilateral filter to remove noise from the image to ensure image clarity.

[0206] Normalization: Make sure all images have the same height and width (H, W), for example 512×512 pixels.

[0207] Training the pre-trained model: The pre-trained model is trained using the processed image dataset, which helps the model better learn the characteristics of the photovoltaic panels.

[0208] Adjust weights: By fine-tuning the pre-trained model, adjust the weights of the model so that it can focus more on the segmentation task of the photovoltaic category.

[0209] Through the above steps, we can not only ensure the quality of the dataset, but also improve the performance of the image encoder through training and weight adjustment of the pre-trained model, so as to obtain higher accuracy and reliability in subsequent segmentation tasks.

[0210] According to one embodiment of the present application, a SAM segmentation network including an image encoder, a prompt encoder and a mask decoder is constructed, and the interactive click information and the data set are input into the SAM segmentation network, specifically:

[0211] Through the interactive operation of the interactive module, the positive and negative sample pixels in the data set are obtained to form interactive click information, and a prompt encoder is constructed based on the interactive click information;

[0212] The image data and the interactive click information are transmitted to the image encoder and the prompt encoder to obtain encoded information.

[0213] Specifically, the interaction module obtains positive and negative sample pixels

[0214] Interaction operations of interactive modules:

[0215] By M i In the module interface, users can select the positive sample pixel point P in the image through keyboard and mouse interaction. + (representing photovoltaic area) and negative sample pixel point P - (denoting non-photovoltaic area).

[0216] The user selects a certain number of pixels on the image, which represent the parts they think are the photovoltaic area (positive samples) and the parts that are not the photovoltaic area (negative samples).

[0217] To form interactive click information:

[0218] According to M i The interactive click information transmitted by the module is used to construct a prompt encoder (EP) to encode this information into a prompt vector.

[0219] The hint encoder encodes the information of positive and negative sample points selected by the user to form hint vectors, which contain the user's understanding of the image content and help guide the model to better segment the photovoltaic area.

[0220] Building a Hint Encoder

[0221] Construct a hint encoder based on interactive click information:

[0222] In M i Load the pre-training mechanism and predictor instance in the module, load the weight W F , and initialize the predictor instance so that it can use the weights W F Make a preliminary forecast.

[0223] This means that the user's choice will be used as part of the model's input for subsequent segmentation tasks.

[0224] Input image data and interactive click information to the segmentation network

[0225] Image data is transferred to the image encoder:

[0226] Image information via M i Module transfer to E i , in E i In the above method, the image is first divided into blocks and embedded, and the input image is converted into an embedding vector E.

[0227] Enable absolute position embedding P, add the corresponding position information, and finally get the encoded vector V.

[0228] The image is then processed by a series of Block modules, each of which contains an input vector, an output vector, layer normalization, a multi-head attention mechanism, and a multi-layer perceptron.

[0229] When the window size is set, the attention mechanism in the block will perform window division and restoration.

[0230] Finally, the neck module goes through a series of convolution operations and layer normalization to output the corresponding feature map F.

[0231] Interactive click information is passed to the prompt encoder:

[0232] Record the parameters for the incoming point information, including the embedding dimension and the image embedding size I s .

[0233] Initialize the position encoding layer and the hint embedding layer.

[0234] Get dense position encoding, call the position encoding layer, input Is , and the shape is (E d ,H,W) random position encoding.

[0235] Perform the dimension-up operation and obtain the shape (1,E d ,H,W) random position encoding.

[0236] To fill the point set, the main step is to adjust the coordinates and increase them by 0.5 to move them to the center of the pixel to obtain a new point set.

[0237] An extra point is added in with coordinates [0,0] and label -1. This extra point is used for filling and expanding the label L.

[0238] Define an embedding vector Ve to represent the placeholder points (points with label -1).

[0239] According to the label, the point embedding increases the weight of the corresponding category (the label is 0 or 1 corresponding to PE[0] or PE[1] respectively).

[0240] Compute the batch size bs according to the dot hint and initialize the sparse embedding with an empty tensor of shape (bs, 0,).

[0241] If there is a point hint, the point information is converted to an embedding vector suitable for the model. If there is no point hint, the point information is converted to , and finally concatenated into a sparse embedding. The sparse embedding is returned as the result.

[0242] Through the above steps, the image encoder and prompt encoder are constructed, and the image data and interactive click information are input into the SAM segmentation network. These steps ensure that the model can make full use of the information provided by the user and improve the accuracy and reliability of segmentation.

[0243] According to one embodiment of the present application, a segmentation output layer is added to the mask decoder to generate a preliminary segmentation result. Specifically, a segmentation output layer is added to the mask decoder to output semantic information, and the semantic information is superimposed on the image data when outputting the mask through a multi-layer perception mechanism to output a preliminary segmentation result.

[0244] Specifically, the segmentation output layer is added:

[0245] A segmentation output layer is added to the mask decoder. The main function of this layer is to output semantic information. Semantic information refers to the classification information of different objects in the image, specifically the location and shape of the photovoltaic panel.

[0246] Application of multi-layer perception mechanism:

[0247] The Multi-Layer Perceptron (MLP) mechanism is used to superimpose semantic information with image data when outputting the mask. This means that when generating the final segmentation mask, it not only relies on the pixel values ​​of the image, but also refers to the model's understanding of the image content, that is, semantic information.

[0248] Superposition process:

[0249] In the process of generating the segmentation mask, D M will receive data from the image encoder (E i ), position encoding, point embedding, and mask input embedding.

[0250] This information is provided by D M A series of internal processing steps finally generate a preliminary segmentation result, which includes the mask map of the photovoltaic panel, the mask quality prediction and related semantic information.

[0251] The mask quality prediction is used to evaluate the reliability of the segmentation results, while the semantic information provides the classification and understanding of the photovoltaic panels in the image.

[0252] Generation of preliminary segmentation results:

[0253] Finally, (D M ) returns the generated preliminary segmentation results, including the mask map of the photovoltaic panel, the mask quality prediction, and the prediction results of the semantic information.

[0254] These preliminary results are passed back to the interaction module (M i ) and displayed in the form of an image on the user interface for the user to make further interactive adjustments.

[0255] Through the above steps, the mask decoder not only generates the segmentation mask of the photovoltaic panel, but also combines semantic information, so that the model can more accurately understand the image content when outputting the preliminary segmentation result, thereby improving the accuracy and reliability of the segmentation. This method not only reduces the need for a large amount of labeled data, but also further optimizes the segmentation results through user interaction.

[0256] According to one embodiment of the present application, semantic information is superimposed on image data when outputting a mask through a multi-layer perception mechanism, specifically:

[0257] The image data is transmitted to the image encoder through the interaction module;

[0258] The image data is processed in blocks and embedded, that is, the input image is converted into an embedded vector, and the embedded vector is enhanced by adding absolute position information to obtain the encoded vector; a series of convolution operations and layer normalization are performed through the connection module, and finally the corresponding feature map is output.

[0259] Specifically, the image data is transmitted to the image encoder through the interaction module

[0260] The interactive module obtains image information:

[0261] Users through M i Interactive operation on the module interface selects certain pixels in the image as positive sample points (P + ) and negative sample points (P - ), these points are used to construct the cue encoder.

[0262] M i The module transmits these interactive click information to the image encoder (E i ) and the prompt encoder (E p ).

[0263] Image data is processed in blocks and embedded

[0264] Image segmentation and embedding:

[0265] Image information via M i Module transfer to E i In E i In the process, the image is first divided into blocks to better process the local features in the image.

[0266] Next, the input image is converted into an embedding vector E, which is an abstract representation of the image features.

[0267] Add absolute position information:

[0268] Enable absolute position P embedding and add the position information to the embedding vector E to obtain the encoded vector V. This step is to enable the model to understand the spatial position relationship of each part of the image.

[0269] A series of convolution operations and layer normalization are performed through the connection module

[0270] Convolution operation and layer normalization:

[0271] The encoded vector V is passed through the connection module (Neck module) to perform a series of convolution operations. The convolution operation can further extract the features of the image and reduce the data dimension.

[0272] Layer Normalization is applied after the convolution operation in order to stabilize the intermediate output distribution during model training and improve the convergence speed and generalization ability of the model.

[0273] After these processing steps, the corresponding feature map F is finally output.

[0274] Comprehensive step-by-step explanation

[0275] Image data input: First, the image data is passed through M i Module transfer to E i The user can interactively click to select positive and negative sample points in the image, and the information of these points is transferred to E i and E p .

[0276] Embedding and location information: In E i In the process, the image is divided into blocks and embedded, which is converted into an embedding vector E. Subsequently, by adding the absolute position information P, the encoded vector V is obtained. This process enhances the model's understanding of image features.

[0277] Feature extraction and normalization: The encoded vector V passes through a series of convolution operations and layer normalization in the Neck module, and finally outputs the feature map F. These feature maps contain high-level features of the image, which is helpful for subsequent segmentation tasks.

[0278] Through the above steps, the image data is effectively processed and encoded to form high-level feature representations, which will be used in subsequent segmentation tasks. This process not only improves the model's ability to extract image features, but also provides rich semantic information for the subsequent mask decoder, thereby improving the accuracy and reliability of segmentation.

[0279] According to an embodiment of the present application, it further includes: the prompt encoder records parameters for the input point information, including embedding dimensions and image embedding size, and initializes the position encoding layer and the prompt embedding layer;

[0280] The mask decoder receives the channel dimension, model type, number of predicted masks, upsampling activation function, and depth and hidden dimension parameters during the initialization phase, and defines the upsampler and list, as well as the prediction header for the predicted mask quality;

[0281] The mask decoder receives the output image embedding, position encoding, point embedding and mask input embedding of the image encoder, generates masks and mask quality predictions, outputs corresponding semantic information, and selects appropriate mask slices;

[0282] The mask decoder returns the generated mask, the mask quality prediction, and the prediction results of the semantic information.

[0283] Specifically, it also includes: the prompt encoder records parameters for the input point information, including embedding dimension and image embedding size, and initializes the position encoding layer and the prompt embedding layer;

[0284] The mask decoder receives the channel dimension, model type, number of predicted masks, upsampling activation function, and depth and hidden dimension parameters during the initialization phase, and defines the upsampler and list, as well as the prediction header for the predicted mask quality;

[0285] The mask decoder receives the output image embedding, position encoding, point embedding and mask input embedding of the image encoder, generates masks and mask quality predictions, outputs corresponding semantic information, and selects appropriate mask slices;

[0286] The mask decoder returns the generated mask, the mask quality prediction, and the prediction results of the semantic information.

[0287] According to an embodiment of the present application, the preliminary segmentation result is fed back to the interactive module for interactive adjustment, and then re-decoded after adjustment to generate the final segmentation result, specifically:

[0288] The positive sample pixels or negative sample pixels in the preliminary results are adjusted through the interactive module to improve the segmentation accuracy;

[0289] The adjusted information is re-input into the mask decoder for decoding. The mask decoder receives information from the image encoder and the hint encoder and combines the adjusted information to generate a new segmentation mask;

[0290] The mask decoder outputs updated mask information, indicator information, and semantic information.

[0291] Specifically, the preliminary results of interaction module adjustment

[0292] Preliminary results show:

[0293] The user interface will display the preliminary segmentation results, including the mask map of the photovoltaic panel, the mask quality prediction, and the semantic information. Users can intuitively see the model's preliminary segmentation of the image.

[0294] Interactive Adjustment:

[0295] Users through M i Module interface, use keyboard and mouse interactive operation to select positive sample pixels (P + ) or negative sample pixels (P - ) to make adjustments.

[0296] Users can choose to add or delete certain points to improve segmentation accuracy. For example, if a region is misclassified as a non-PV region in the preliminary results, users can add positive sample points (P + ); On the contrary, if a certain area is misjudged as a photovoltaic area, the user can add negative sample points (P - ).

[0297] Re-decode to generate new segmentation results

[0298] Re-enter the adjusted information:

[0299] The user-adjusted information is re-input into the mask decoder (D M ). This information includes new positive sample points (P + ) and negative sample points (P - )’s location.

[0300] M i The module transmits this adjusted information to the prompt encoder (E p ), E p This information is then encoded into a hint vector.

[0301] The mask decoder takes in the information and generates a new segmentation mask:

[0302] The mask decoder receives the i ) and the prompt encoder (E p ) information and generates a new segmentation mask in combination with the adjusted information.

[0303] D M It receives the output image embedding of the image encoder, the position encodings, the point embeddings, and the embeddings of the mask input and generates a new mask and mask quality prediction.

[0304] D M The corresponding semantic information is also output and the appropriate mask slice is selected.

[0305] Output updated information:

[0306] The mask decoder returns the generated updated mask information, mask quality prediction, and semantic information prediction results.

[0307] This updated information is passed back to M i module and displayed as an image on the user interface.

[0308] Specific implementation examples

[0309] Suppose the user is using the system to segment a remote sensing image that contains photovoltaic panels.

[0310] Preliminary segmentation results display:

[0311] The preliminary segmentation results are displayed on the user interface, and the user finds that some PV panel areas are mistakenly segmented into non-PV areas.

[0312] Interactive Adjustment:

[0313] Users through M i The module adds positive sample points (P + ) to help the model better identify these areas.

[0314] Re-decode to generate new segmentation results:

[0315] M i The module passes the adjusted information to E p , E p This information is encoded into a hint vector.

[0316] D M Receive from E i and E p and generates a new segmentation mask in combination with the user-adjusted information.

[0317] D M Output updated mask information, mask quality prediction, and semantic information.

[0318] Output updated information:

[0319] The updated segmentation results are displayed again on the user interface, and the user can see that the previously mis-segmented areas have been correctly identified as photovoltaic panels.

[0320] Through the above steps, users can repeatedly perform interactive adjustments and re-decoding until the segmentation results are satisfactory. This method not only improves the accuracy and reliability of segmentation, but also enhances the flexibility of the model and the user's participation.

[0321] Example 2

[0322] Based on the background technology, for remote sensing images with complex scenes or rich details, traditional manual annotation and segmentation methods may not meet the needs, and a large amount of high-quality annotated data is essential for model training in deep learning. However, manually annotating a large amount of data is a time-consuming and laborious task. To address the above problems, this application uses the public data set Photovoltaic Panel (PVP) Dataset as a pre-training sample, and collects 4640 photovoltaic cell sample images from 13 provinces in China provided by the PVP Dataset. Each group of samples consists of a 512×512 pixel image and a corresponding photovoltaic panel label.

[0323] A photovoltaic image recognition method based on SAM remote sensing, such as Figure 1 As shown, the following steps are included:

[0324] S1) Based on the public multi-resolution PV segmentation dataset of any region, the dataset is preprocessed to ensure data quality, and based on the processed dataset, the pre-trained weights (Wf) for PV recognition are constructed.

[0325] In this embodiment, the public data set Photovoltaic Panel (PVP) Dataset is used as a pre-training sample. PVP Dataset provides 4640 photovoltaic cell sample images from 13 provinces in China. Each sample is a 512×512 pixel image with corresponding photovoltaic panel labels to strengthen the pre-training model of photovoltaic recognition and provide a high-quality data basis for subsequent model training. The prediction data comes from photovoltaic data obtained from Google Earth Pro. These data are divided into 512×512 images to make them have a consistent scale. The data preprocessing process includes screening out suitable remote sensing photovoltaic image data, using bilateral filters to remove noise from the data, and standardizing the remote sensing image data to make them have the same length and width (H, W) to fine-tune the original pre-training weights of SAM.

[0326] S2) Build Mask_Interactive(M i ) module, through interactive clicks to obtain positive and negative sample pixels in the image data, based on M i The module transmits the acquired click information and establishes a prompt encoder (E p ), load the pre-trained mechanism and predictor instance;.

[0327] In step S2) by M i The module interface and keyboard and mouse interactive operation select a certain number of positive sample pixels P in the image + and negative sample pixel P - , where P + Represents the photovoltaic area, P - Represents non-photovoltaic area; according to M i The interactive click information transmitted by the module is used to construct the prompt encoder, which encodes this information into a prompt vector. i Load the pre-training mechanism and predictor instance in the module. The specific steps include: loading weights W f , and initialize the predictor instance so that it can use the weights W f Make a preliminary forecast

[0328] S3) builds a SAM segmentation network based on encoder-decoder, and converts the fine-tuned image encoder (Ei ) and the optimized hint encoder (E p ) and the improved mask decoder (D M ) combined with.

[0329] For S3)E i 、E p and D M The specific steps for improvement include:

[0330] The pre-trained model is trained and fine-tuned through the public remote sensing photovoltaic dataset (PVP Dataset) to improve its ability to extract features of remote sensing photovoltaic images;

[0331] b) Optimize E p The transmission method of the prompt information is through M i The module transmits point information. The E p Only keep the point information as E p Input.

[0332] c) In D M A segmentation output layer is added to output semantic information, and the mask is superimposed with the original image through a multi-layer perceptron when outputting the mask to output the prediction result;

[0333] S4) Build Figure 2 The photovoltaic recognition method based on the SAM model in remote sensing images is shown in i The image information and prompt points obtained by the module are transmitted to E i and E p , using the improved D M Decode the encoded information, output preliminary mask information, indicator information and semantic information, and pass the decoded preliminary results back to M i module and displayed as an image on the user interface.

[0334] like Figure 3 As shown, the specific steps for the preliminary recognition result of the image in S4) include:

[0335] a) Image information is transmitted through M i Module transfer to E i , in E i First, the image I is divided into blocks and embedded into f pe , and convert the input image into an embedding vector E. Enable the absolute position P embedding to add the corresponding position information and finally get the encoded vector V. The formula is as follows:

[0336] E=f pe (I), V=E+P

[0337] The image is then processed by a series of Block modules, each of which contains an input vector V i , output vector V i+1 , layer normalization f LN , multi-head attention mechanism f MHA And multi-layer perceptron f MLP , when the window size is set, the attention mechanism in the block will perform window division and restoration. The formula is as follows:

[0338] V i+1 =f MLP (f LN (V i +f MHA (f LN (V i ))))

[0339] In these Block modules, image features are further extracted and processed, and the formula is as follows:

[0340]

[0341] Finally, the neck module passes through a series of convolution operations and layer normalization f LN , output the corresponding feature map F, the formula is as follows:

[0342] F=f neck (f LN (V block ))

[0343] b) E p Record the parameters for the incoming point information, including the embedding dimension and the image embedding size I s , initialize the position encoding layer and the hint embedding layer

[0344] c) Obtain dense positional encoding P d , call the position encoding layer f p , input I s , and the shape is (E d ,H,W) random position encoding formula is:

[0345] P d =f p (I s )

[0346] in

[0347] Then, through the dimension-raising operation, we get a shape of (1,E d ,H,W) random position encoding and return, the formula is:

[0348] P d =P d *Unsqueeze(0)

[0349] in Unsqueeze(0) means expanding the dimension at the 0th dimension

[0350] d) Point set P s To fill, the main step is to adjust the coordinates to P s The coordinates in the pixel are increased by 0.5 to move it to the center of the pixel to obtain a new point set P. n The formula is:

[0351] P n =P s +0.5

[0352] In P n An extra point is added with coordinates [0,0] and label -1. This extra point is used for filling and expanding label L.

[0353] P n =P n ∪{0,0},L n =L∪{-1}

[0354] Perform position encoding. The main step is to call f p Point set P n Perform position encoding.

[0355] e=f p (P n )

[0356] Define an embedding vector V e , used to represent placeholder points (points with a label of -1). j Points with a value of -1 are set to 0.

[0357]

[0358] According to the label, the point embedding increases the weight of the corresponding category (the label is 0 or 1 corresponding to PE[0] or PE[1] respectively).

[0359]

[0360] e) Calculate the batch size bs according to the point prompt, and initialize the shape to (bs, 0, E d ) is a sparse embedding of the empty tensor S e , the formula is:

[0361] S e =(bs,0,E d )

[0362] If there is a point hint, the point information is converted into an embedding vector P suitable for the model e If there is no point prompt, convert the point information into S e , and finally concatenated into the sparse embedding. Return the sparse embedding S e As a result, the formula is:

[0363]

[0364] f) During the initialization phase, D M It receives the channel dimension, model type, number of prediction masks, upsampling activation function, and f MLP The depth and hidden dimensions of the , and define the upsampler and f MLP list, and a prediction head for predicting mask quality. During the forward propagation, D M Receive E i The output image embedding, position encoding, point embedding and mask input embedding of D generate mask and mask quality prediction, output corresponding semantic information, and select appropriate mask slices. Finally, D M The generated mask, mask quality prediction, and semantic information prediction results are returned.

[0365] S5) The user interactively fine-tunes the preliminary results, further optimizes the model output by adjusting the positive and negative sample points or other parameters, re-decodes the fine-tuned information, generates the final mask information, indicator information and semantic information, outputs the final result, and displays it on the interface in the form of an image to ensure the accuracy of the recognition result.

[0366] Example 3

[0367] A computer program product containing instructions, when running on a device, enables the device to execute the steps in a method for photovoltaic recognition of remote sensing images based on a SAM model.

[0368] Example 4

[0369] A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps in a remote sensing image photovoltaic identification method based on a SAM model.

[0370] Example 5

[0371] The electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the photovoltaic recognition method of remote sensing images based on the SAM model are implemented.

[0372] Anything not described in this application can be achieved by adopting or drawing on existing technologies.

[0373] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0374] The above is only the embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A photovoltaic recognition method for remote sensing images based on the SAM model, characterized in that: include: Collect and preprocess a publicly available multi-resolution remote sensing photovoltaic segmentation dataset, and use the dataset to construct pre-trained weights for photovoltaic recognition to optimize the image encoder; Constructing a SAM segmentation network including the image encoder, the hint encoder and the mask decoder, and inputting the interactive click information and the dataset into the SAM segmentation network; adding a segmentation output layer to the mask decoder to generate a preliminary segmentation result; The preliminary segmentation result is fed back to the interactive module for interactive adjustment, and after the adjustment, it is re-decoded to generate the final segmentation result.

2. The method according to claim 1, characterized in that The collecting and preprocessing of multi-resolution photovoltaic segmentation data sets, and using the data sets to construct pre-trained weights for photovoltaic recognition to optimize the image encoder, specifically comprises: collecting public multi-resolution photovoltaic segmentation data of any region to form a multi-resolution photovoltaic segmentation data set; Screening the data set to obtain appropriate image data, removing noise from the image data using a bilateral filter and performing standardization processing so that the image data have the same length and width; The pre-trained model is fine-tuned using the processed image data, and the original pre-trained weights of the pre-trained model are adjusted to optimize the image encoder.

3. The method according to claim 2, characterized in that The constructing of a SAM segmentation network including the image encoder, the hint encoder and the mask decoder, and inputting the interactive click information and the data set into the SAM segmentation network, specifically: Through the interactive operation of the interactive module, positive and negative sample pixel points in the data set are obtained to form interactive click information, and the prompt encoder is constructed based on the interactive click information; The image data and the interactive click information are transmitted to the image encoder and the prompt encoder to obtain encoding information.

4. The method according to claim 3, characterized in that The step of adding a segmentation output layer to the mask decoder to generate a preliminary segmentation result is specifically as follows: adding a segmentation output layer to the mask decoder to output semantic information, and superimposing the semantic information with the image data when outputting the mask through a multi-layer perception mechanism to output the preliminary segmentation result.

5. The method according to claim 4, characterized in that The semantic information is superimposed on the image data when outputting the mask through the multi-layer perception mechanism, specifically: The image data is transmitted to the image encoder through the interaction module; The image data is processed in blocks and embedded, that is, the input image is converted into an embedding vector, and the embedding vector is enhanced by adding absolute position information to obtain an encoded vector; a series of convolution operations and layer normalization are performed through the connection module, and finally the corresponding feature map is output.

6. The method according to claim 5, characterized in that Also includes: The hint encoder records parameters for the incoming point information, including embedding dimension and image embedding size, and initializes the position encoding layer and the hint embedding layer; The mask decoder receives the channel dimension, model type, number of predicted masks, upsampling activation function, and depth and hidden dimension parameters during the initialization phase, and defines the upsampler and list, as well as the prediction header for predicting the mask quality; The mask decoder receives the output image embedding, position encoding, point embedding and mask input embedding of the image encoder, generates a mask and mask quality prediction, outputs corresponding semantic information, and selects a suitable mask slice; The mask decoder returns the generated mask, the mask quality prediction, and the prediction results of the semantic information.

7. The method according to claim 4, characterized in that The preliminary segmentation result is fed back to the interactive module for interactive adjustment, and then re-decoded to generate the final segmentation result after adjustment, specifically: The positive sample pixels or negative sample pixels in the preliminary results are adjusted through the interactive module to improve the segmentation accuracy; re-inputting the adjusted information into the mask decoder for decoding, the mask decoder receiving information from the image encoder and the hint encoder and combining the adjusted information to generate a new segmentation mask; The mask decoder outputs updated mask information, indicator information, and semantic information.

8. A computer program product comprising instructions, which, when executed on a device, is characterized in that: The device is enabled to execute the steps in the remote sensing image photovoltaic identification method based on the SAM model as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the photovoltaic identification method of remote sensing images based on the SAM model as described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the remote sensing image photovoltaic identification method based on the SAM model as described in any one of claims 1 to 7 are implemented.

Images