Single tree detection method and system based on pre-training large model and ladder side tuning strategy

Abstract

The invention discloses a single tree detection method and system based on a pre-trained large model and a ladder side tuning strategy, and relates to the technical field of forestry remote sensing intelligent monitoring, and the method comprises the steps: obtaining a remote sensing image, preprocessing the remote sensing image, and constructing a reference sample data set; using the pre-trained large model as a global feature encoder, constructing a bypass model as a local feature encoder based on a ladder side tuning strategy, and using the encoder to extract and fuse global and local features of the preprocessed remote sensing image stage by stage to obtain multi-level features; a segmentation threshold and a canopy processing strategy are preset for different scenes; constructing a double-flow feature decoder, processing the multi-level features according to a segmentation threshold value and a canopy processing strategy, and further processing the multi-level features by adopting a full convolutional network to obtain a tree canopy segmentation result; and outputting individual tree detection information in the target area. According to the method, the detection adaptability to complex scenes is improved, the dependence on large-scale annotation data is reduced, the single tree detection cost is reduced, and the efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of forestry remote sensing intelligent monitoring, and more specifically, to a single tree detection method and system based on a pre-trained large model and a ladder-side tuning strategy. Background Art

[0002] Urban trees play an important role in carbon sequestration, climate change mitigation, and improvement of the living environment. As urbanization continues to accelerate, the demand for accurate monitoring of urban suburban forests is increasing. Traditional manual survey methods are costly and time-consuming, and it is difficult to achieve high-precision single-tree level monitoring on a large scale. The rise of high-resolution remote sensing images has provided a new means for urban tree detection, but due to the complex canopy morphology and uneven distribution of trees in urban environments, as well as interference factors such as buildings, roads, and water bodies, the automatic detection of single trees using traditional algorithms is still challenging.

[0003] In recent years, the development of deep learning technology has made significant progress in feature extraction and target detection of remote sensing images. Traditional remote sensing processing methods rely on manually designed feature extraction strategies, which are often limited to fixed rules and assumptions, and show poor adaptability when facing complex and changing environments. Existing studies have shown that deep learning models can automatically learn multi-level features from a large amount of training data, reducing the dependence on manually designed features. In urban tree monitoring, deep learning can effectively deal with the complex morphology and uneven distribution of tree canopies through an adaptive feature learning mechanism, and has shown superior performance than traditional methods in automated detection and high-precision extraction. It has shown potential to outperform traditional methods in forestry remote sensing applications. Deep learning, especially models based on deep convolutional networks, can effectively separate targets and noise in complex backgrounds with its powerful feature learning ability and nonlinear expression ability, thereby achieving accurate detection of trees.

[0004] At the same time, the emergence of large visual models has provided a broader space for innovation in remote sensing image analysis. Such large models are pre-trained on ultra-large-scale data sets. This pre-training process enables the model to learn shared features from different fields and tasks, and to provide stable and reliable performance in a variety of scenarios. It has strong versatility and generalization capabilities, and can achieve cross-domain migration and rapid adaptation under conditions of few or even zero samples, reducing dependence on manual annotation and specific scene data. As a representative achievement of large visual models, the Segment Anything Model (SAM) is not only applicable to natural images, but also can effectively cope with the remote sensing field, providing innovative solutions for instance segmentation tasks such as urban tree detection, ecological monitoring, and resource management in high-resolution remote sensing images.

[0005] Nevertheless, deep learning and large-model single-tree extraction methods still have the following problems that need to be properly addressed, including:

[0006] (1) Deep learning methods usually rely heavily on large-scale labeled datasets and lack the ability to generalize across scenarios;

[0007] (2) When the large model is directly applied to a specific single-wood extraction task, the model output does not contain specific categories, and the model structure still needs to be effectively fine-tuned to meet the application requirements of specific fields;

[0008] (3) There are significant differences in tree characteristics in different scenarios (such as cities, suburban forests, and mixed areas), which requires targeted tuning strategies and scenario adaptation methods. Summary of the invention

[0009] The purpose of the present invention is to overcome the defects in the prior art and provide a single tree detection method and system based on a pre-trained large model and a ladder-side tuning strategy.

[0010] To achieve the above object, the technical solution of the present invention is as follows:

[0011] In a first aspect, the present invention provides a single tree detection method based on a pre-trained large model and a ladder-side tuning strategy, comprising the following steps:

[0012] Acquire remote sensing images and perform preprocessing, and construct a reference sample data set based on the remote sensing images;

[0013] The pre-trained large model is used as a global feature encoder, and a side branch model is constructed based on a ladder-side tuning strategy and used as a local feature encoder. The global feature encoder and the local feature encoder are used to extract the global features and local features of the pre-processed remote sensing image stage by stage and fuse them stage by stage to obtain features at multiple levels;

[0014] Preset different segmentation thresholds and canopy processing strategies for different scenarios;

[0015] Constructing a dual-stream feature decoder, using the dual-stream feature decoder to process the features of the multiple levels according to the segmentation threshold and canopy processing strategy, and then further processing using a fully convolutional network to obtain a tree canopy segmentation result;

[0016] The single tree detection information in the target area is output according to the tree canopy segmentation result.

[0017] Preferably, the pre-trained large model includes a network model of a Vision Transformer structure trained by a masked autoencoder (MAE), and the parameters of the pre-trained large model remain unchanged during the single tree detection process;

[0018] Preferably, the local feature encoder comprises a multi-scale convolutional neural network structure, wherein the multi-scale convolutional neural network structure comprises convolutional layers with convolutional kernels of relatively smaller sizes and convolutional layers with convolutional kernels of relatively larger sizes.

[0019] Preferably, in the multi-scale convolutional neural network structure, the convolution layer with the relatively small-sized convolution kernel is used to extract the local structure, edge and texture information of the trees, and the convolution layer with the relatively large-sized convolution kernel is used to extract background information including the overall shape of the tree crown and the relationship between neighboring trees or other objects.

[0020] Preferably, the fusion process is specifically to perform weighted fusion of the feature map of the global feature and the feature map of the local feature through learnable gating unit parameters, and the weighted fusion process is represented by the following expression:

[0021] X sam =SAM_encoder(x)∈R H×W×C

[0022] X add =Additional_encoder(x)∈R H×W×C

[0023] X`=α·X sam +(1-α)·X add

[0024] Among them, X sam is the feature map of the global feature obtained by the SAM_encoder encoder, i.e. the global feature encoder, X add is the feature map of local features obtained by the Additional_encoder encoder, i.e., the local feature encoder. X` is the feature map obtained by fusion using the learnable gating unit parameters. H×W×C represents the dimensions of the feature map, which are height, width, and number of channels, respectively. α is the learnable gating unit parameter, which is a variable coefficient and represents the weight in weighted fusion. The learnable gating unit parameter α is trained and optimized by back propagation according to the reference sample data set during the training process of the side branch model, so as to achieve a dynamic balance between the global and local information in the features of multiple levels.

[0025] Preferably, in the process of extracting global features and local features stage by stage and fusing them stage by stage, a blocking operation is first performed so that the input blocks remain of consistent size. Feature extraction is performed at each stage in the process of extraction and fusion, and downsampling is performed in the convolution module of the local feature encoder. The extracted global features and local features are fused at each stage and the fused features are input into the next stage for fusion, thereby obtaining features at multiple levels.

[0026] Preferably, the using the dual-stream feature decoder to process the features of the multiple levels and then further processing them using a fully convolutional network specifically includes:

[0027] The dual-stream feature decoder upsamples and restores the features of the multiple levels step by step. The dual-stream feature decoder also includes a feature pyramid network and a region proposal network. The feature pyramid network fuses the features of the multiple levels that have been upsampled and restored step by step to obtain fused features. The region proposal network generates candidate region suggestions based on the fused features to determine the potential single tree position. The fully convolutional network performs fine segmentation on the candidate regions to ultimately obtain a tree canopy segmentation result.

[0028] Preferably, constructing a reference sample data set according to the remote sensing image comprises:

[0029] Annotate the remote sensing image data of some areas and mark the tree canopy range as a reference sample dataset;

[0030] The annotated images and corresponding labels are cropped into fixed-size tiles, part of which is used as a training set to train the local feature encoder and gating parameter α, and the other part is used as a validation set for performance evaluation and parameter tuning.

[0031] Preferably, different segmentation thresholds and canopy processing strategies are used for different scenes, specifically including:

[0032] In urban scenes, the segmentation threshold is increased to reduce misjudgments caused by building shadows and water bodies, and multiple small canopy segments are fused;

[0033] In suburban forest scenes, the segmentation threshold is lowered to capture more potential tree crowns, and the large canopy is appropriately split into small canopy areas to more accurately identify densely overlapping trees;

[0034] In mixed scenes, a universal segmentation threshold and a standard canopy processing strategy are used to ensure that the results remain highly stable in a multi-object environment.

[0035] In a second aspect, the present invention provides a single tree detection system based on a pre-trained large model and a ladder-side tuning strategy, which is used to apply the single tree detection method based on a pre-trained large model and a ladder-side tuning strategy described in the above technical solution, and the system includes:

[0036] A data acquisition and processing module is used to acquire remote sensing images and perform preprocessing, and to construct a reference sample data set based on the remote sensing images;

[0037] A multi-level feature generation module is used to use the pre-trained large model as a global feature encoder, build a side branch model based on the ladder-side tuning strategy and use it as a local feature encoder, use the global feature encoder and the local feature encoder to extract the global features and local features of the pre-processed remote sensing image stage by stage and fuse them stage by stage to obtain features at multiple levels;

[0038] The scene adaptation module is used to preset different segmentation thresholds and canopy processing strategies for different scenes;

[0039] A canopy segmentation result generation module is used to construct a dual-stream feature decoder, and according to the segmentation threshold and canopy processing strategy, the dual-stream feature decoder is used to process the features of the multiple levels, and then a full convolutional network is used for further processing to obtain a tree canopy segmentation result;

[0040] The single tree detection information output module is used to output the single tree detection information in the target area according to the tree canopy segmentation result.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention improves cross-scenario adaptability by using a pre-trained large model to extract global features, extracts local features based on a ladder-side tuning strategy, and uses a learnable gating unit to achieve dynamic fusion of global and local features, thereby enhancing the expression of complex tree crown information and formulating specific adaptation strategies for different scenarios. This significantly improves the detection adaptability to complex scenes, reduces dependence on large-scale labeled data, reduces the cost of single tree detection, and improves efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a schematic diagram of the steps of a single tree detection method based on a pre-trained large model and a ladder-side tuning strategy according to Example 1 of the present application;

[0044] Figure 2 This is a schematic diagram of a global feature encoder according to Embodiment 1 of the present application;

[0045] Figure 3 This is a schematic diagram of a local feature encoder according to Embodiment 1 of the present application;

[0046] Figure 4 This is a schematic diagram of extracting global features and local features stage by stage and fusing them stage by stage in Example 1 of the present application;

[0047] Figure 5 This is a schematic diagram of a single tree before and after detection in Example 1 of the present application. DETAILED DESCRIPTION

[0048] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0049] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0050] Example 1

[0051] See also Figure 1 , Embodiment 1 of the present application provides a single tree detection method based on a pre-trained large model and a ladder-side tuning strategy, comprising the following steps S1-S5:

[0052] S1: Acquire and preprocess remote sensing images, and construct a reference sample data set based on the remote sensing images.

[0053] In Example 1 of the present application, a high-resolution optical remote sensing image of the area to be detected is used as a data source, wherein the spatial resolution of the high-resolution remote sensing image is about 0.1m-0.15m, including three bands of red, green and blue. The image is preprocessed as necessary to meet the subsequent model input requirements, and the preprocessing may include radiation correction, geometric correction, cropping and enhancement.

[0054] The remote sensing image data of some areas are manually or semi-automatically annotated, and the tree canopy range is marked as a reference sample dataset; the annotated images and corresponding labels are cropped into tiles of fixed size, one part of which is used as a training set to train the local feature encoder and gating parameter α, and the other part is used as a validation set for performance evaluation and parameter tuning.

[0055] S2: Use the pre-trained large model as the global feature encoder, build a side branch model based on the ladder-side tuning strategy and use it as the local feature encoder, use the global feature encoder and the local feature encoder to extract the global features and local features of the preprocessed remote sensing image stage by stage and fuse them stage by stage to obtain features at multiple levels.

[0056] The pre-trained large model used in the present invention is a network model of a VIT (Vision Transformer) structure trained by Masked Autoencoders (MAE). In this embodiment 1, the preferred pre-trained large model is an arbitrary segmentation model (Segment Anything Model, SAM). The arbitrary segmentation large model SAM consists of three components: an image encoder, a prompt encoder, and a mask decoder. The image encoder uses a pre-trained masked autoencoder visual transformation network to extract image features. The prompt encoder can process four types of inputs: points, boxes, text, and masks, which are embedded into the model in different ways. The mask decoder is responsible for effectively combining the features of the image and the prompt, and interacting through a cross-attention mechanism, one as a query and the other as a key and value vector. Finally, the features are upsampled by transposed convolution. The mask decoder can generate multiple results based on the provided prompts. Please refer to Figure 2 , Figure 2 The schematic diagram of the global feature encoder of Example 1 of the present application, the present invention uses the SAM image encoder with unchanged parameters frozen to extract the global features of the remote sensing image. In Example 1 of the present application, the ladder-side tuning strategy (Ladder-Side Tuning, LST) is adopted to introduce a convolutional neural network to construct a side branch model as a local feature encoder to extract local features related to the crown. The strategy uses a lightweight network architecture different from the main network as a downstream guide branch to extract local features required for downstream tasks. LST aims to enhance feature representation by combining and working together with different network paths, which refine and supplement the extracted features together. Specifically, the main network (i.e., SAM as a global feature encoder) focuses on obtaining global features, while being supplemented by a downstream guide branch (i.e., a side branch model as a local feature encoder) that specifically extracts more detailed local features. This multi-branch structure allows hierarchical representation, and features at different levels are continuously integrated, similar to a ladder-like configuration. The downstream guide branch helps to tune features laterally, thereby maintaining a balance between global and local feature extraction. The method is able to extract local features of image segmentation instances using a deep convolutional network in a local perception unit while capturing global features. In this part, the convolutional neural network is used as the guiding model for downstream tasks, mainly outputting multi-scale local features to assist the global feature encoder in processing images. During the training process, the parameters of the pre-trained SAM model remain unchanged, while the LST-based side branch model uses the training set in the reference sample dataset for training optimization through back propagation to enhance feature extraction and achieve a dynamic balance between global and local information in features at multiple levels.

[0057] SAM has shown high generalization ability in a variety of scenarios and can effectively segment almost all previously unseen targets, but its performance in specific areas (such as downstream tasks such as accurately identifying individual trees in remote sensing images) still needs to be improved. To overcome this limitation, Example 1 of the present application introduces a convolutional neural network to construct a branch model as a local feature encoder to enhance the model's feature extraction and representation capabilities in specific areas, thereby improving the performance of the corresponding tasks. Please refer to Figure 3 , Figure 3 Schematic diagram of the local feature encoder of Example 1 of the present application. The local feature encoder is composed of a multi-scale convolutional neural network structure, which includes a convolution layer of a relatively small-sized convolution kernel and a convolution layer of a relatively large-sized convolution kernel. The multi-scale convolutional neural network structure in Example 1 includes a convolution layer of a 3*3 convolution kernel and a convolution layer of a 7*7 convolution kernel. The specific functions are as follows:

[0058] (1) Use smaller convolution kernels in the 3×3 convolution layer to focus on fine features of the image, such as the edges and textures of trees, which is crucial for extracting tree crowns.

[0059] (2) Use larger convolution kernels in the 7*7 convolution layer to capture features within a wider field of view, such as the overall shape of the crown and the relationship between neighboring trees. Large-sized convolution kernels can extract more background information, which is especially important for distinguishing trees from complex backgrounds and helps identify the location of trees in the environment.

[0060] (3) Integrate the features extracted by the 3*3 convolutional layer and the 7*7 convolutional layer into spatial prior features for subsequent feature fusion and analysis

[0061] See also Figure 4 , Figure 4Schematic diagram of extracting global features and local features stage by stage and fusing them stage by stage in Example 1 of the present application. Example 1 of the present application deeply fuses global features and local features in each different encoding stage by dividing the global feature encoder into four stages, each stage corresponding to a local feature encoder. Example 1 of the present application inputs global features and local features into a learnable gating unit, optimizes the learnable gating unit parameter α through back propagation, and realizes dynamic weighted fusion of global and local features. The function of α is to dynamically adjust the contribution of each network feature in the composite feature representation. This adaptive weight parameter is optimized according to the contribution of the feature map to the overall task performance during the training phase. The use of a gating unit allows β to vary between 0 and 1, where α=1 represents features that are completely dependent on SAM, and β=0 represents features extracted by the convolutional neural network CNN that is completely dependent on the local encoder. Throughout the training process, α is refined through back propagation, so that the global and local information in the features of multiple levels are dynamically balanced to ensure that the feature integration of the two sources is optimally balanced, reflecting their relevance and effectiveness in affecting the model output. The specific process of weighted fusion is to perform weighted fusion of the feature map of global features and the feature map of local features through learnable gated unit parameters. The weighted fusion process is expressed by the following expression:

[0062] X sam =SAM_encoder(x)∈R H×W×C

[0063] X add =Additional_encoder(x)∈R H×W×C

[0064] X`=α·X sam +(1-α)·X add

[0065] Among them, X sam is the feature map of the global feature obtained by the SAM_encoder encoder, i.e. the global feature encoder, X add is the feature map of local features obtained by the Additional_encoder encoder, i.e., the local feature encoder. X` is the feature map obtained by fusion using the learnable gating unit parameters. H×W×C represents the dimensions of the feature map, which are height, width, and number of channels, respectively. α is the learnable gating unit parameter, which is a variable coefficient and represents the weight in weighted fusion. The learnable gating unit parameter α is trained and optimized by back propagation according to the reference sample data set during the training process of the side branch model, so as to achieve a dynamic balance between the global and local information in the features of multiple levels.

[0066] In the example of the present invention, the above-mentioned learnable gating unit and cross-attention mechanism are used to promote the feature interaction between the global feature encoder and the local feature encoder. In the process of extracting global features and local features stage by stage and fusing them stage by stage, the image is first divided into blocks (Patch Partition) to make the input blocks consistent. Each encoding stage contains a different number of Transformer Blocks for extracting global features and different convolutional modules (Convolutional Block) for extracting local features. Feature extraction is performed in each stage and downsampling is performed in the convolutional module to align the scale of the Transformer Block input. After the first encoding stage, the PatchMerging module between each stage will fuse the extracted global features and local features and input the fused features into the next stage through the cross-attention mechanism. That is, the feature map of the global features input in each stage is a feature map combined with the global feature map and the local feature map of the previous stage. The global features and local features are extracted and fused stage by stage. In this process, multiple levels of features from shallow to deep are obtained. The cross-attention mechanism effectively integrates spatial prior information into features, thereby enhancing the understanding of spatial relationships, which not only improves the accuracy of feature extraction, but also enhances adaptability and flexibility, making it suitable for processing complex and changeable image data.

[0067] S3: Preset different segmentation thresholds and canopy processing strategies for different scenarios.

[0068] In Example 1 of the present application, a customized scene adaptation strategy is implemented to dynamically adjust the detection process so that the final output detection results can adapt to different urban scenes. These adjustments mainly include determining the segmentation threshold and adopting different canopy coverage processing strategies.

[0069] S4: construct a dual-stream feature decoder, use the dual-stream feature decoder to process the features of the multiple levels according to the segmentation threshold and canopy processing strategy, and then use a fully convolutional network for further processing to obtain a tree canopy segmentation result.

[0070] A dual-stream feature decoder is constructed and the features of multiple levels obtained in step S2 are input into the dual-stream feature decoder to perform instance-level segmentation and positioning of single tree targets, thereby generating high-precision single tree detection results. The dual-stream feature decoder processes the feature maps of multiple levels of features, and then outputs the candidate box of instance segmentation, i.e., the tree canopy segmentation result. The dual-stream feature decoder is based on the Feature Pyramid Network (FPN) and the Region Proposal Network (RPN). FPN is a network structure for target detection. It realizes the fusion and utilization of multi-scale features by constructing a feature pyramid, which can improve the detection accuracy. After the features of multiple levels obtained in step S2 are input into the dual-stream feature decoder, the dual-stream feature decoder performs step-by-step upsampling and restoration of the features of the multiple levels, and uses the feature pyramid network FPN to perform feature fusion of the features of the multiple levels to obtain fused features. Then, the dual-stream feature image decoder performs feature screening and inversion based on the fused features, and then the fused features after feature screening and inversion are input into the region proposal network RPN to generate candidate region proposals for single trees to determine the potential single tree position. After the dual-stream feature decoder outputs the candidate region proposal, a fully convolutional network (FCN) is used to further refine the candidate region for accurate bounding box prediction, and the candidate region is finely segmented to finally obtain the tree canopy segmentation result.

[0071] S5: Outputting single tree detection information in the target area according to the tree canopy segmentation result.

[0072] See also Figure 5 , Figure 5 This is a schematic diagram of single tree detection before and after the embodiment 1 of the present application. According to the tree canopy segmentation result generated in step S4, the single tree detection information in the target area is output, that is, the single tree distribution and quantity information in the target area is output.

[0073] Example 2

[0074] Example 2 of the present application is based on Example 1, and further explains the different segmentation thresholds and canopy processing strategies preset for different scenes in Example 1, as follows:

[0075] The scenes are divided into three different types: suburban forest, urban and mixed areas. The implementation of the scene adaptation strategy in Example 2 of the present application is based on the unique characteristics of urban and suburban forest scenes, and aims to optimize the detection process to adapt to different scene characteristics. The region proposal network RPN can effectively identify potential target areas in the image and assign a confidence score to these areas. The segmentation threshold is the preset confidence score, which is considered valid if it is greater than or equal to the confidence score.

[0076] In urban areas, the segmentation threshold is set at a high level to effectively distinguish trees from other man-made elements, because urban scenes have diverse and complex background elements, such as buildings, roads, and other man-made structures, which may have similar colors and shapes as trees. In contrast, a lower threshold is more suitable for suburban forest areas. Due to the high density of trees, smaller canopies and understory vegetation need to be included to ensure complete suburban forest coverage. In addition, when canopy overlap occurs, different strategies will be adopted according to different scenarios. In suburban forest areas, a large canopy is replaced by multiple smaller canopies, which helps to more accurately represent the structure of individual trees and more accurately analyze and promote ecological relationships between trees. In urban areas, multiple small canopies are merged into a larger canopy to reduce redundancy and improve consistency, because urban trees are usually pruned and the canopy shape is more regular. In mixed areas, the distribution of elements is complex and diverse. A moderate universal segmentation threshold and a standardized canopy processing scheme (i.e., the default parameters of the model or network are used without modification) are adopted to avoid the increase in complexity and uncertainty due to over- or under-correction, thereby reducing the overall computational efficiency, adaptability, and accuracy. For the environment where multiple artificial or natural non-tree features such as buildings, roads and water bodies coexist, i.e. cities, the segmentation threshold is increased to reduce false positives, and multiple smaller tree crown fragments are fused to make the results more consistent with the laws and characteristics of urban tree distribution. For the situation where the tree crowns are densely overlapped and the boundaries are blurred in the forest, the segmentation threshold is lowered to capture more potential tree crowns, and the results are refined by disassembling the large canopy into multiple smaller canopies, so as to more accurately characterize the single tree pattern in the high-density vegetation environment. For mixed areas with multiple land cover types, moderate segmentation thresholds and standardized canopy processing schemes are used to balance the influence of different types of object features and avoid over- or under-correction, so as to maintain high detection accuracy under the influence of multiple factors. Based on the statistical analysis and accuracy evaluation of the detection results in different scenarios, the regional threshold and canopy processing parameters are dynamically adjusted to achieve better cross-regional and cross-temporal and spatial single tree detection performance.

[0077] The other technical details and steps of this embodiment 2 are the same as those of embodiment 1 and will not be repeated here.

[0078] Example 3

[0079] Embodiment 3 of the present application provides a single tree detection system based on a pre-trained large model and a ladder-side tuning strategy on the basis of Embodiment 1 and Embodiment 2, which is used to apply the single tree detection method based on a pre-trained large model and a ladder-side tuning strategy described in Embodiment 1, and the system includes:

[0080] A data acquisition and processing module is used to acquire remote sensing images and perform preprocessing, and to construct a reference sample data set based on the remote sensing images;

[0081] A multi-level feature generation module is used to use the pre-trained large model as a global feature encoder, build a side branch model based on the ladder-side tuning strategy and use it as a local feature encoder, use the global feature encoder and the local feature encoder to extract the global features and local features of the pre-processed remote sensing image stage by stage and fuse them stage by stage to obtain features at multiple levels;

[0082] The scene adaptation module is used to preset different segmentation thresholds and canopy processing strategies for different scenes;

[0083] A canopy segmentation result generation module is used to construct a dual-stream feature decoder, and according to the segmentation threshold and canopy processing strategy, the dual-stream feature decoder is used to process the features of the multiple levels, and then a full convolutional network is used for further processing to obtain a tree canopy segmentation result;

[0084] The single tree detection information output module is used to output the single tree detection information in the target area according to the tree canopy segmentation result.

[0085] The other technical details and steps of this embodiment 3 are the same as those of embodiment 1 or embodiment 2 and will not be repeated here.

[0086] In summary, the present invention combines the global feature encoder from SAM, keeps its parameters unchanged, and adds a local feature encoder based on the LST strategy and convolutional neural network, which effectively utilizes the pre-trained segmentation capability of SAM, while allowing the local feature encoder based on the LST strategy to update its parameters to adapt to the task of single tree recognition. In order to determine the importance of the features extracted by the two networks, a learnable gating unit is introduced when fusing these extracted feature maps to evaluate the importance of different feature maps, construct a dual-stream feature decoder, and segment the crown of a single tree according to a preset scene adaptation strategy, and output single tree detection information. Compared with other classification methods, the method proposed in this scheme effectively improves the overall accuracy of single tree segmentation, indicating that this combined method is an effective solution for single tree segmentation in different urban contexts.

Claims

1. A single tree detection method based on a pre-trained large model and ladder-side tuning strategy, characterized in that: The following steps are involved: Acquire remote sensing images and perform preprocessing, and construct a reference sample data set based on the remote sensing images; The pre-trained large model is used as a global feature encoder, and a side branch model is constructed based on a ladder-side tuning strategy and used as a local feature encoder. The global feature encoder and the local feature encoder are used to extract the global features and local features of the pre-processed remote sensing image stage by stage and fuse them stage by stage to obtain features at multiple levels; Preset different segmentation thresholds and canopy processing strategies for different scenarios; Constructing a dual-stream feature decoder, using the dual-stream feature decoder to process the features of the multiple levels according to the segmentation threshold and canopy processing strategy, and then further processing using a fully convolutional network to obtain a tree canopy segmentation result; The single tree detection information in the target area is output according to the tree canopy segmentation result.

2. The single tree detection method based on the pre-trained large model and ladder-side tuning strategy according to claim 1 is characterized in that: The pre-trained large model includes a network model of a Vision Transformer structure trained by a masked autoencoder MAE, and the parameters of the pre-trained large model remain unchanged during the single tree detection process.

3. The single tree detection method based on a pre-trained large model and ladder-side tuning strategy according to any one of claim 1, characterized in that: The local feature encoder includes a multi-scale convolutional neural network structure, which includes convolutional layers with relatively small-sized convolutional kernels and convolutional layers with relatively large-sized convolutional kernels.

4. The single tree detection method based on the pre-trained large model and ladder-side tuning strategy according to claim 3 is characterized in that: In the multi-scale convolutional neural network structure, the convolution layer with relatively small-sized convolution kernels is used to extract the local structure, edge and texture information of the trees, and the convolution layer with relatively large-sized convolution kernels is used to extract background information including the overall shape of the tree crown and the relationship between neighboring trees or other objects.

5. The single tree detection method based on the pre-trained large model and ladder-side tuning strategy according to claim 1 is characterized in that: The fusion process is specifically to perform weighted fusion of the feature map of the global feature and the feature map of the local feature through the learnable gated unit parameters. The weighted fusion process is represented by the following expression: X sam =SAM_encoder(x)∈R H×W×C X add =Additioanl_encoder(x)∈R H×W×C X`=α·X sam +(1-a)·X add Among them, X sam is the feature map of the global feature obtained by the SAM_encoder encoder, i.e. the global feature encoder, X add is the feature map of local features obtained by the Additional_encoder encoder, i.e., the local feature encoder. X` is the feature map obtained by fusion using the learnable gating unit parameters. H×W×C represents the dimensions of the feature map, which are height, width, and number of channels, respectively. α is the learnable gating unit parameter, which is a variable coefficient and represents the weight in weighted fusion. The learnable gating unit parameter α is trained and optimized by back propagation according to the reference sample data set during the training process of the side branch model, so as to achieve a dynamic balance between the global and local information in the features of multiple levels.

6. The single tree detection method based on the pre-trained large model and ladder-side tuning strategy according to claim 5 is characterized in that: In the process of extracting global features and local features at each stage and fusing them at each stage, a block operation is first performed to keep the input blocks of the same size. Feature extraction is performed at each stage and downsampling is performed in the convolution module of the local feature encoder. At each stage, the extracted global features and local features are fused and the fused features are input into the next stage for fusion, thereby obtaining features at multiple levels.

7. The single tree detection method based on the pre-trained large model and ladder-side tuning strategy according to claim 6 is characterized in that: The using the dual-stream feature decoder to process the features of the multiple levels and then using a full convolutional network for further processing specifically includes: The dual-stream feature decoder upsamples and restores the features of the multiple levels step by step. The dual-stream feature decoder also includes a feature pyramid network and a region proposal network. The feature pyramid network fuses the features of the multiple levels that have been upsampled and restored step by step to obtain fused features. The region proposal network generates candidate region suggestions based on the fused features to determine the potential single tree position. The fully convolutional network performs fine segmentation on the candidate regions to ultimately obtain a tree canopy segmentation result.

8. The single tree detection method based on the pre-trained large model and ladder-side tuning strategy according to claim 1 is characterized in that: The constructing a reference sample data set according to the remote sensing image comprises: Annotate the remote sensing image data of some areas and mark the tree canopy range as a reference sample dataset; The annotated images and corresponding labels are cropped into fixed-size tiles, part of which is used as a training set to train the local feature encoder and gating parameter α, and the other part is used as a validation set for performance evaluation and parameter tuning.

9. The single tree detection method based on a pre-trained large model and ladder-side tuning strategy according to any one of claims 1 to 8, characterized in that: The different segmentation thresholds and canopy processing strategies are used for different scenes, specifically including: In urban scenes, the segmentation threshold is increased to reduce misjudgments caused by building shadows and water bodies, and multiple small canopy segments are fused; In suburban forest scenes, the segmentation threshold is lowered to capture more potential tree crowns, and the large canopy is appropriately split into small canopy areas to more accurately identify densely overlapping trees; In mixed scenes, a universal segmentation threshold and a standard canopy processing strategy are used to ensure that the results remain highly stable in a multi-object environment.

10. A single tree detection system based on a pre-trained large model and a ladder-side tuning strategy, used to apply the single tree detection method based on a pre-trained large model and a ladder-side tuning strategy according to any one of claims 1 to 9, characterized in that: The system comprises: A data acquisition and processing module is used to acquire remote sensing images and perform preprocessing, and to construct a reference sample data set based on the remote sensing images; A multi-level feature generation module is used to use the pre-trained large model as a global feature encoder, build a side branch model based on the ladder-side tuning strategy and use it as a local feature encoder, use the global feature encoder and the local feature encoder to extract the global features and local features of the pre-processed remote sensing image stage by stage and fuse them stage by stage to obtain features at multiple levels; The scene adaptation module is used to preset different segmentation thresholds and canopy processing strategies for different scenes; A canopy segmentation result generation module is used to construct a dual-stream feature decoder, and according to the segmentation threshold and canopy processing strategy, the dual-stream feature decoder is used to process the features of the multiple levels, and then a full convolutional network is used for further processing to obtain a tree canopy segmentation result; The single tree detection information output module is used to output the single tree detection information in the target area according to the tree canopy segmentation result.

Images