A vegetation remote sensing image processing method and system based on semi-supervised deep learning
Through semi-supervised deep learning model and feature pyramid technology, the problems of insufficient data volume in the density distribution estimation of vegetation remote sensing images and limited generalization capabilities are solved, the accuracy and adaptability of the density distribution map are improved, and single-timber segmentation is realized and counting, reducing labeling costs and improving processing speed.
Patent Information
- Application Number
- CN202410354602.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-26
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-03-26
AI Technical Summary
The existing vegetation remote sensing image density distribution estimation technology has insufficient data volume, limited model generalization ability, slow processing speed, dependence on remote sensing image preprocessing, high dependence on ground reality, and difficulty in dealing with vegetation complexity and multi-scale problems.
Using a semi-supervised deep learning model, by constructing a feature pyramid, extracting correlation features, performing feature regression and optimization, combining multi-layer CA attention mechanism and pre-adaptation network, loss functions are designed to improve the accuracy and adaptability of density distribution maps.
Effectively extract complex vegetation texture features, improve the accuracy of density distribution map prediction, realize single-wood segmentation and counting, reduce labeling costs, and enhance the generalization ability and real-time nature of the model.
Smart Images

Figure CN118155075B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a vegetation remote sensing image processing method and system based on semi-supervised deep learning. Background Art
[0002] At present, the density distribution estimation technology of vegetation remote sensing images still has some shortcomings and deficiencies, mainly including the following aspects:
[0003] Limited data volume: Deep learning technology has high requirements for data volume, but in practical applications, it is still difficult to obtain large-scale and diverse remote sensing image data. Due to the high cost of obtaining labeled data, the lack of sufficient training data will affect the performance of deep learning models.
[0004] Limited model generalization ability: Existing deep learning models are often trained on specific data sets, and their generalization ability to new remote sensing image data is limited, and there may be overfitting problems. Therefore, it is necessary to further improve the generalization ability of the model to adapt to a wider range of remote sensing image application scenarios.
[0005] Slow processing speed: Deep learning models usually require long training time and high computing resources, which makes the model processing speed slow in practical applications. In order to improve the real-time performance and efficiency of the model, it is necessary to optimize the model structure and algorithm to reduce the computational complexity of the model.
[0006] Dependence on remote sensing image preprocessing: The quality and features of remote sensing images have a crucial impact on the density distribution estimation results. Therefore, before deep learning model training, remote sensing images need to be preprocessed, such as denoising, normalization, enhancement, etc. These preprocessing processes will introduce some additional errors and uncertainties, affecting the accuracy of the model.
[0007] Dependence on ground truth: In order to obtain more accurate density distribution estimation results, it is usually necessary to pair with ground truth data, but the acquisition cost of ground truth is high and there are certain errors and uncertainties. Therefore, how to effectively use ground data and integrate it into the model is also a problem that needs to be solved.
[0008] Limitations on vegetation complexity: Traditional methods are mainly based on hand-designed feature extractors, which are difficult to capture higher-level features of vegetation density distribution. Although deep learning technology can automatically learn higher-level feature representations from data, it still has certain limitations and cannot handle the complexity of vegetation density distribution.
[0009] Difficulty in dealing with the multi-scale problem of remote sensing images: Remote sensing images usually have features at multiple scales, such as details and overall features, while deep learning models can usually only process information at a specific scale. Therefore, when processing remote sensing images, it is necessary to consider how to effectively combine information at multiple scales and perform effective migration and fusion between different scales. Summary of the invention
[0010] In view of the shortcomings of the existing technology, the present invention integrates the advantages of traditional methods and proposes a new semi-supervised deep learning model to estimate the density distribution of vegetation remote sensing images and link the density information with its implicit contour information, providing higher performance in the field of crown segmentation and counting of single trees.
[0011] To achieve the above object, the present invention provides a vegetation remote sensing image processing method based on semi-supervised deep learning, the steps comprising:
[0012] Collecting original image data of vegetation to obtain a data set, wherein the original image data is in the form of remote sensing images;
[0013] Extracting features from the data set to obtain correlation features;
[0014] Performing a feature regression operation on the correlation feature to obtain an initial density distribution map;
[0015] The initial density distribution map is optimized to obtain a final density distribution map, thereby completing the processing of the vegetation remote sensing image. Preferably, the step of obtaining the correlation feature includes:
[0016] Based on the data set, construct a feature pyramid;
[0017] The correlation feature is obtained based on the feature pyramid and the coordinates of the original image data.
[0018] Preferably, the method for obtaining the initial density distribution map includes: performing three upsampling operations on the correlation features, and mapping the correlation features to the original image data size.
[0019] Preferably, the method for obtaining the final density distribution map includes: fine-tuning the parameters of the initial density distribution map through a pre-adaptive network, and the specific steps include: designing a loss function so that the density of the initial density distribution map area is close to 1, and then applying a perturbation loss to constrain the Gaussian distribution to obtain the final density distribution map.
[0020] Preferably, the method for constructing the feature pyramid includes: using ResNet101FPN pre-trained on ImageNet, screening vegetation areas of interest through a multi-layer CA attention mechanism, and constructing the feature pyramid.
[0021] The present invention also provides a vegetation remote sensing image processing system based on semi-supervised deep learning, the system is used to implement the above method, including: an acquisition module, an extraction module, a regression module and an optimization module;
[0022] The acquisition module is used to acquire original image data of vegetation to obtain a data set, wherein the original image data is in the form of remote sensing images;
[0023] The extraction module is used to extract features from the data set to obtain correlation features;
[0024] The regression module is used to perform a feature regression operation on the correlation feature to obtain an initial density distribution map;
[0025] The optimization module is used to optimize the initial density distribution map to obtain a final density distribution map and complete the processing of the vegetation remote sensing image.
[0026] Preferably, the workflow of the extraction module includes:
[0027] Based on the data set, construct a feature pyramid;
[0028] The correlation feature is obtained based on the feature pyramid and the coordinates of the original image data.
[0029] Preferably, the workflow of the regression module includes: performing three upsampling operations on the correlation features, and mapping the correlation features to the original image data size.
[0030] Compared with the prior art, the present invention has the following beneficial effects:
[0031] The present invention can effectively extract complex vegetation texture features and improve the accuracy of density distribution map prediction by capturing vegetation contour information. It also predicts the vegetation density distribution of the original input image through the density map, thereby achieving the segmentation and counting of single trees, solving the problems of low accuracy, poor adaptability and high labeling cost of traditional methods. In addition, the present invention can also perform lightweight fine-tuning on parameters to make the model more adaptable to task scenarios in complex vegetation growth environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0033] Figure 1 A schematic diagram of a method flow chart of an embodiment of the present invention;
[0034] Figure 2 A schematic diagram of a model structure of an embodiment of the present invention;
[0035] Figure 3 Schematic diagram of model prediction according to an embodiment of the present invention. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0038] Before the description, let's first introduce the meaning of the words that appear in the drawings in the specification: Input: Input global feature layer Fi and sample frame coordinates B
[0039] Coord: coordinates
[0040] ROI-Pooling: ROI-pooling operation
[0041] Mapping: Mapping operation
[0042] Multiscale: Multi-scale transformation
[0043] Example features layers:
[0044] Correlation operation:
[0045] Correlated feature map:
[0046] Mapping and Correlation: Mapping and correlation operations
[0047] Adaption and Regression: Adaptation and regression operations
[0048] Input image:
[0049] Backbone: backbone network
[0050] Cablock: CA block
[0051] Example boxes: Example box B
[0052] Mapping
[0053] ROI pooling: ROI-pooling operation
[0054] Correlation: association operation
[0055] Regression: regression operation
[0056] Predict: Prediction density map
[0057] Adaptation: Adaptation operation
[0058] Embodiment 1
[0059] like Figure 1 As shown, it is a schematic diagram of the method flow of an embodiment, and the steps include:
[0060] S1. Collect original image data of vegetation to obtain a data set.
[0061] Drones are used to collect original image data; drones can achieve low-altitude, high-resolution, high-speed, and precise sampling, providing scientific and accurate data support for this embodiment. The production of the data set is based on drone sampling images, and detailed manual annotation is performed to ensure the accuracy and standardization of the data, so that the model can be learned reasonably and efficiently. It should be noted that in this embodiment, the original image data collected are all in the form of remote sensing images.
[0062] S2. Extract features from the data set to obtain correlation features.
[0063] First, use the ResNet101FPN pre-trained on imagenet to screen the vegetation area of interest through the multi-layer CA attention mechanism to build a feature pyramid (as the skeleton model of this embodiment). Specifically, imagenet is an open source model website. This embodiment uses the ResNet101FPN model trained on the website and adds a multi-layer CA attention mechanism to it to build a feature pyramid:
[0064]
[0065] Among them, M i Indicates that multi-layer features are obtained through the ResNet101FPN backbone network; Represents the obtained multi-factor layer set processing the original input image x network through ResNet101; Represents the FPN operation; i represents the i-th layer feature map.
[0066] Afterwards, the correlation features are obtained based on the coordinates of the feature pyramid and the original image data.
[0067] The feature pyramid is combined with the coordinates of the original image data to perform feature mapping to obtain local features. The local features are combined together through the Pooling layer as a sliding convolution layer, and the sliding convolution operation is performed on the original image data to obtain correlation features. The specific steps include:
[0068] On the basis of the backbone network, the Mapping and Correlation (MAC) module is introduced to extract relevant features between samples and images, which enhances the robustness and relevance of the extracted features. At the same time, the Multi-layer Coordinate Attention (MLCA) module can effectively capture the contour information of vegetation and improve the accuracy of density distribution map prediction. Specifically:
[0069] Combining the ResNet101FPN layer with MLCA can better preserve the spatial-channel feature information at different stages. This facilitates the adjustment of the subsequent MAC module and highlights the area of interest. This further reduces the impact of background noise in vegetation remote sensing images. Apply coordinate attention (CA) to the output map of each ResNet101FPN layer as the intrinsic attention module of this layer:
[0070] F i =MLCA(M i )
[0071] Among them, F i Represents the global feature layer.
[0072] This module is lightweight and takes into account the integration of direction-related channel information and location information, which enhances the model's ability to recognize cross-regions.
[0073] S3. Perform feature regression operation on the correlation features to obtain an initial density distribution map.
[0074] The obtained correlation features are upsampled three times to map the features to the original image size and obtain the initial density distribution map. The specific steps of the upsampling operation include:
[0075] As the core of obtaining relevant features, its operation can be divided into two steps. The first step involves inputting the global feature layer F i Map it with the example box coordinates B to obtain the local example feature layer. The purpose of this operation is to extract features from a specific region of interest corresponding to a single tree. This extracted region serves as a template for subsequent feature extraction, which helps the network understand the goal of the task.
[0076] The second step involves using E i As a convolution layer, the global feature layer is convolved to obtain the relevant feature layer and obtain the initial density distribution map. The convolution operation inherently involves capturing the relevant features C i By utilizing the information of single tree templates of different scales as local features in the convolutional layer, the process of extracting features from the original global features can be enhanced. Finally, these related feature layers are uniformly aggregated and concatenated together. This feature fusion captures the relationship between local and global features and can better reflect the density distribution of vegetation.
[0077] E i =GMP(Map(F i ,B))×S j
[0078] C i =Conv(F i ,E i )
[0079] C=CON(GMP(C i , C i-1 ))
[0080] Among them, Map(F i , B) represents the global feature layer F based on the example box coordinate B i Mapping. Represents pooling different example features into a unified size; GMP represents pooling different example features into a unified size; S j represents the scaling factor selected according to the requirements. In this embodiment, 0.8, 0.9, 1.0, 1.1, and 1.2 are selected.
[0081] Repeat the above upsampling steps three times to obtain the initial density distribution map.
[0082] S4. Optimize the initial density distribution map to obtain a final density distribution map.
[0083] In order to process vegetation remote sensing images of different regions and sizes, a pre-adaptation network is used to optimize the initial density distribution map, obtain the final density distribution map, and complete the processing of vegetation remote sensing images; specifically, the Adaptation (AP) module is introduced and the loss function is designed. This pre-adaptation is mainly used to allow the network to learn the characteristics of the image before making predictions. Its purpose is to enable the network to better understand the representation size of the density distribution in the image and determine the area where the crown appears at this scale.
[0084] In summary, from S2 to S4, this embodiment completes the construction of the density prediction model, and the model structure is as follows: Figure 2 As shown in Figure 3 shown.
[0085] This embodiment designs four loss functions, which can optimize the initial density distribution map according to different situations, including: One-Trend loss, Mae loss, Dice loss and Perturbation loss. The training model can comprehensively consider various performance indicators and optimization goals, so that the model can be optimized more comprehensively during the training process. Such a training method can improve the adaptability and robustness of the model, so that it has better generalization ability and stability in practical applications.
[0086] One-Trend loss function: The main purpose is to make the density of the sample box area close to 1. This is done to roughly delineate the area of the crown, and then the perturbation loss is applied to constrain the Gaussian distribution. The overall goal is to obtain more accurate prediction results. The number of steps of this module is elastic and controllable, which is used to fine-tune the network parameters of the regression module. The flexibility of the number of steps allows the performance of the regression module to be adjusted to enhance performance. The specific formula of the One-Trend loss function Y is:
[0087]
[0088] Among them, Adap represents the AP module; Represents the initial density distribution map; flag indicates whether to use the Ap module for parameter fine-tuning.
[0089] Dice loss: The main purpose is to constrain the shape difference between the predicted density map Y and the actual density map G in scenarios where there is a strong imbalance between positive and negative samples. This constraint enables the network to better adapt to the complex texture and morphology of tree remote sensing images in vegetation datasets. Since the input of the Dice function is required to be between 0 and 1, it is necessary to first perform Sigmoid binarization on Y and G, and then use the threshold Filter the area with a value greater than it and assign it a value of 1 to obtain the binary results G′, Y′. The formula is as follows:
[0090]
[0091]
[0092]
[0093] Among them, L dice Represents the Dice loss function.
[0094] MAE loss: is an indicator that measures the difference between the predicted value and the actual observed value. It calculates the average of the absolute value of the prediction error of each individual and the prediction density map Y i and the actual density map G i The difference operation is performed on n pixels, Lmae Is the loss value. The formula is as follows:
[0095]
[0096] Therefore, the loss function of the training phase can be defined as follows:
[0097] L train =α1L mae +α2L dice
[0098] Among them, the two weight coefficients are: α1=0.1, α2=0.00005.
[0099] Perturbation loss: Due to the predicted density map sample area Y bi In fact, it is the result of upsampling the relevant feature map group by 8 times. In essence, the relevant response map of the sample feature to the original image should show a Gaussian distribution at the sample position for density estimation. This is similar to using a geometrically adaptive Gaussian kernel when obtaining a density map. Therefore, given a Gaussian density window G h×w , the formula is as follows:
[0100]
[0101] Therefore, the loss function of the test correction phase is defined as follows:
[0102] L Ap =β1L one-trend +β2L Per
[0103] Among them, the two weight coefficients are: β1 = 10 -8 , β2=10 -4 , in order to ensure the stability of the modified model.
[0104] Embodiment 2
[0105] The present invention also provides a vegetation remote sensing image processing system based on semi-supervised deep learning, comprising: an acquisition module, an extraction module, a regression module and an optimization module; the acquisition module is used to acquire original image data of vegetation to obtain a data set, wherein the original image data is in the form of a remote sensing image; the extraction module is used to perform feature extraction on the data set to obtain correlation features; the regression module is used to perform feature regression operations on the correlation features to obtain an initial density distribution map; the optimization module is used to optimize the initial density distribution map to obtain a final density distribution map, thereby completing the processing of the vegetation remote sensing image.
[0106] The workflow of the extraction module includes: constructing a feature pyramid based on the data set; obtaining correlation features based on the coordinates of the feature pyramid and the original image data. The workflow of the regression module includes: performing three upsampling operations on the correlation features and mapping the correlation features to the original image data size.
[0107] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.
Claims
1. A vegetation remote sensing image processing method based on semi-supervised deep learning, It is characterized in that the steps include: Collecting original image data of vegetation to obtain a data set, wherein the original image data is in the form of remote sensing images; Extracting features from the data set to obtain correlation features; The step of obtaining the correlation feature includes: constructing a feature pyramid based on the data set; obtaining the correlation feature based on the coordinates of the feature pyramid and the original image data; performing feature mapping on the feature pyramid combined with the coordinates of the original image data to obtain local features, combining the local features together as a sliding convolution layer through a Pooling layer, and performing a sliding convolution operation on the original image data to obtain the correlation feature. The specific steps include: introducing a Mapping and Correlation module on the basis of a backbone network to extract relevant features between samples and images and enhance the robustness and correlation of the extracted features; at the same time, using a Multi-layer Coordinate Attention block to capture the contour information of vegetation and improve the accuracy of density distribution map prediction. Specifically: combining the ResNet101FPN layer with the MLCA to retain the spatial-channel feature information of different stages, adjusting the subsequent MAC module, highlighting the area of interest, and reducing the impact of background noise in vegetation remote sensing images; applying coordinate attention to the output image of each ResNet101FPN layer as the inherent attention module of the layer; The correlation feature is subjected to a feature regression operation to obtain an initial density distribution map; the method for obtaining the initial density distribution map comprises: performing three upsampling operations on the correlation feature to map the correlation feature to the original image data size; as the core of obtaining the correlation feature, the operation can be divided into two steps; the first step involves inputting the global feature layer F i Map it with the example box coordinates B to obtain the local example feature layer; the purpose of this operation is to extract features from a specific region of interest corresponding to a single tree. This extracted region is used as a template for subsequent feature extraction for the goal of the network understanding task; the second step involves using E i As a convolution layer, the global feature layer is convolved to obtain the relevant feature layer and the initial density distribution map; the convolution operation inherently involves capturing the relevant features C i ; By using the single tree template information of different scales as local features in the convolutional layer, the process of extracting features from the original global features is enhanced; finally, these related feature layers are uniformly aggregated and connected in series. This feature fusion captures the relationship between local and global features and can better reflect the density distribution of vegetation; Optimize the initial density distribution map to obtain a final density distribution map, and complete the processing of the vegetation remote sensing image; the method for obtaining the final density distribution map includes: fine-tuning the parameters of the initial density distribution map through a pre-adaptive network, and the specific steps include: designing a loss function so that the density of the initial density distribution map area is close to 1, and then applying a perturbation loss to constrain the Gaussian distribution to obtain the final density distribution map.
2. The vegetation remote sensing image processing method based on semi-supervised deep learning according to claim 1, It is characterized in that The method for constructing the feature pyramid includes: using ResNet101FPN pre-trained on ImageNet, screening the vegetation area of interest through a multi-layer CA attention mechanism, and constructing the feature pyramid.
3. A vegetation remote sensing image processing system based on semi-supervised deep learning, the system being used to implement the method described in any one of claims 1-2, It is characterized in that include: Acquisition module, extraction module, regression module and optimization module; The acquisition module is used to acquire original image data of vegetation to obtain a data set, wherein the original image data is in the form of remote sensing images; The extraction module is used to extract features from the data set to obtain correlation features; The workflow of the extraction module includes: constructing a feature pyramid based on the data set; obtaining the correlation feature based on the coordinates of the feature pyramid and the original image data; performing feature mapping on the feature pyramid combined with the coordinates of the original image data to obtain local features, combining the local features together as a sliding convolution layer through a Pooling layer, and performing a sliding convolution operation on the original image data to obtain correlation features. The specific steps include: introducing a Mapping and Correlation module on the basis of a backbone network to extract correlation features between samples and images and enhance the robustness and correlation of the extracted features; at the same time, using a Multi-layerCoordinateAttention block to capture the contour information of vegetation and improve the accuracy of density distribution map prediction. Specifically: combining the ResNet101FPN layer with the MLCA to retain the spatial-channel feature information of different stages, adjusting the subsequent MAC module, highlighting the area of interest, and reducing the impact of background noise in vegetation remote sensing images; applying coordinate attention to the output image of each ResNet101FPN layer as the inherent attention module of the layer; The regression module is used to perform feature regression operation on the correlation feature to obtain an initial density distribution map; the workflow of the regression module includes: performing three upsampling operations on the correlation feature to map the correlation feature to the original image data size; the method for obtaining the initial density distribution map includes: performing three upsampling operations on the correlation feature to map the correlation feature to the original image data size; as the core of obtaining the correlation feature, its operation can be divided into two steps; the first step involves inputting the global feature layer F i Map it with the example box coordinates B to obtain the local example feature layer; the purpose of this operation is to extract features from a specific region of interest corresponding to a single tree. This extracted region is used as a template for subsequent feature extraction for the goal of the network understanding task; the second step involves using E i As a convolution layer, the global feature layer is convolved to obtain the relevant feature layer and the initial density distribution map; the convolution operation inherently involves capturing the relevant features C i ; By using the single tree template information of different scales as local features in the convolutional layer, the process of extracting features from the original global features is enhanced; finally, these related feature layers are uniformly aggregated and connected in series. This feature fusion captures the relationship between local and global features and can better reflect the density distribution of vegetation; The optimization module is used to optimize the initial density distribution map to obtain the final density distribution map and complete the processing of the vegetation remote sensing image; specifically, through the pre-adaptation network, the parameters of the initial density distribution map are fine-tuned, and the specific steps include: designing a loss function so that the density of the initial density distribution map area is close to 1, and then applying perturbation loss to constrain the Gaussian distribution to obtain the final density distribution map.
