Forestry health information monitoring method based on multi-source heterogeneous remote sensing image feature fusion

Abstract

The invention discloses a forestry health information monitoring method based on multi-source heterogeneous remote sensing image feature fusion. According to the method, an unmanned aerial vehicle carrying a high-resolution multispectral camera and a laser radar is used for carrying out aerial photography on a target sample plot, remote sensing images are obtained, and then the images are processed to generate optical orthographic images and laser radar point cloud data; performing two-dimensional planarization on the point cloud data to generate a digital surface model (DSM), extracting multispectral data of the optical image, and performing feature fusion on the DSM and the multispectral data; and dividing a training set and a verification set based on the fused data, and training and optimizing a deep learning model. And finally, utilizing the trained model to classify withered canopies, healthy canopies and other categories in the remote sensing image to realize accurate monitoring of forestry health conditions. According to the method, through fusion of spectrum and structural features, the recognition precision of withered and healthy canopies is improved, and the limitation of single data feature application is overcome.

Description

Technical Field

[0001] This invention relates to the field of forestry health information monitoring technology, and in particular to a method for monitoring forestry health information based on the fusion of features from multi-source heterogeneous remote sensing images. Background Technology

[0002] Farmland shelterbelts are ecological protection systems composed of specific tree species with clearly defined structures and functions. As important agricultural infrastructure, they play an irreplaceable role in improving the regional ecological environment, regulating local climate, and ensuring stable farmland yields. Therefore, timely and accurate monitoring of the health status of farmland shelterbelts is a core prerequisite for carrying out forestry resource management, ecological protection, and agricultural production assurance.

[0003] In current forestry health survey practices, traditional ground surveys are the primary means of identifying forest health status. This method relies on professional technicians going deep into the field to obtain health information through on-site inspections, sample collection, and manual evaluation. However, its inherent limitations are significant: on the one hand, the survey process requires substantial manpower and time costs, resulting in low efficiency; on the other hand, limited by manpower and time, the survey scope often only covers a limited number of samples, making it difficult to achieve comprehensive monitoring of large areas, and manual evaluation is easily influenced by subjective factors, making it difficult to guarantee data objectivity and consistency. With the development of remote sensing technology, UAV aerial remote sensing, with its advantages of high spatial resolution, strong timeliness, and flexible operation, provides a new technical approach for forestry health monitoring. It can quickly acquire remote sensing data of target areas, objectively reflecting forest structural characteristics and the occurrence and spread of pests and diseases, effectively making up for the shortcomings of traditional ground surveys in terms of efficiency and coverage.

[0004] Among them, the application of multi-source heterogeneous remote sensing data has further expanded the depth and breadth of forestry monitoring. Multispectral technology can capture spectral information in the red band (620nm-750nm), infrared band (700nm-800nm), and near-infrared band (750nm-1300nm). This type of data is sensitive to the chlorophyll content of plants and contains high-quality characteristics that can distinguish the health status of green plants. LiDAR technology measures by emitting beam pulses and can accurately obtain structural parameters such as forest height and diameter at breast height. The point cloud data it generates has a strong correlation with tree canopy information and can effectively invert the vertical structural characteristics of trees.

[0005] While multi-source heterogeneous remote sensing data has shown significant potential in forestry monitoring, current technologies have not fully realized its comprehensive value. In current practice, the application of UAV remote sensing data is often limited to single data sources: when using only multispectral data, although canopy spectral reflectance characteristics can be obtained, it lacks the ability to capture information on the vertical structure of trees, making it difficult to accurately distinguish between health degradation caused by structural damage and spectral changes caused by other environmental factors; when relying solely on lidar data, although forest structural parameters can be accurately retrieved, it cannot identify physiological health abnormalities in trees (such as chlorophyll loss, early-stage pest and disease infections, etc.) through spectral features. This limitation in the application of single data features results in low accuracy in identifying forest canopy health information, making it difficult to meet the actual needs of large-scale, precise forestry health monitoring.

[0006] Meanwhile, breakthroughs in deep learning models in image processing and feature extraction have provided technical support for the efficient utilization of multi-source data. By using deep learning models to fuse and analyze features from multi-source heterogeneous remote sensing data, it is hoped that the advantages of spectral and structural features can be integrated to achieve a multi-dimensional interpretation of forestry health information. Therefore, how to effectively integrate heterogeneous multispectral and lidar data carried by UAVs, and combine this with deep learning technology to overcome the monitoring bottlenecks of single data sources, and construct an efficient, accurate, and large-scale forestry health information monitoring method, has become a pressing technical problem to be solved in the fields of remote sensing image processing and forestry monitoring.

[0007] For example, invention application No. 202510247121.4 discloses an intelligent extraction method for pine wilt diseased trees based on UAV hyperspectral imagery. This method improves the efficiency and accuracy of identifying diseased trees, and the severity rating and disease spread trend prediction enhance the foresight of forestry management, improving the efficiency and accuracy of forest health monitoring. However, this method also has shortcomings: it lacks depth in data fusion, focusing only on hyperspectral imagery data and failing to fully integrate forest structure information obtained from lidar data. This results in a lack of structural dimension consideration in the assessment of forest health status, making it difficult to accurately identify health problems caused by structural damage.

[0008] Therefore, there is a need for a forestry health information monitoring method based on the fusion of features from multi-source heterogeneous remote sensing images, so as to achieve more comprehensive, accurate and efficient monitoring of forestry health information. Summary of the Invention

[0009] To address the aforementioned problems, the present invention aims to provide a forestry health information monitoring method based on the fusion of features from multi-source heterogeneous remote sensing images, thereby solving the bottleneck problem in forestry health monitoring caused by the limitations of applying single data features, which results in low accuracy of forest canopy health information identification.

[0010] This invention provides a method for monitoring forestry health information based on the fusion of features from multi-source heterogeneous remote sensing images.

[0011] First aspect: A method for monitoring forestry health information based on the fusion of features from multi-source heterogeneous remote sensing images, including:

[0012] S1. Use a drone equipped with a high-resolution multispectral camera and lidar to take aerial photos of the target sample plot and obtain remote sensing images;

[0013] S2. Process remote sensing images to generate optical orthophotos and lidar point cloud data of the target site;

[0014] S3. Convert the two-dimensional planarization of the lidar point cloud data into a digital surface model (DSM), generate multispectral data of the optical orthophoto image, and perform feature fusion between the DSM and the multispectral data.

[0015] S4. Divide the training set and validation set based on the feature fusion data, and train and optimize the deep learning model based on the training set and validation set;

[0016] S5. Based on the trained and optimized deep learning model, evaluate the classification accuracy of the deep learning model, classify withered canopies, healthy canopies and other objects in remote sensing images, and realize the monitoring of forestry health status.

[0017] In one embodiment of the present invention, when the UAV takes aerial photos of the target sample plot, it uses a zigzag flight to acquire orthophoto images.

[0018] In one embodiment of the present invention, the multispectral data of the optical orthophoto image includes RGB images in the red, green and blue bands and MSI images in the red, green, blue, near-infrared, infrared 1 and infrared 2 bands.

[0019] In one embodiment of the present invention, the feature fusion of DSM and multispectral data includes:

[0020] The DSM was fused into the fourth band of the RGB image and the seventh band of the MSI image, respectively, to obtain DSM+RGB and DSM+MSI images with high expressive information and different gradient spectral features.

[0021] In one embodiment of the present invention, the deep learning model uses the U-Net model for classification, and simultaneously uses random forest, support vector machine and maximum likelihood classifier for comparative monitoring and classification.

[0022] In one embodiment of the present invention, when the feature fusion data is divided into training and validation sets, five types of land surfaces—damaged canopy, healthy canopy, bare soil, weeds, and crops—are classified and defined as regions of interest with equal area, uniform distribution, and random location. The regions of interest are then divided into training and validation sets in a 7:3 ratio.

[0023] In one embodiment of the present invention, the deep learning model uses Kappa coefficients, confusion matrix, overall accuracy, producer accuracy, and user accuracy to evaluate classification accuracy.

[0024] In one embodiment of the present invention, the Kappa coefficient is used to evaluate classification accuracy, and the formula is as follows:

[0025]

[0026] Where Po is the proportion of correct simulations, Pc is the expected proportion of correct simulations under random conditions, and Pp is the proportion of correct simulations under ideal classification conditions.

[0027] In one embodiment of the present invention, classification accuracy is evaluated using producer accuracy and user accuracy, as shown in the formula:

[0028]

[0029]

[0030]

[0031] Where OA represents overall accuracy, PA represents producer accuracy, UA represents user accuracy, TP represents true positive, TN represents true negative, FP represents false positive, and FN represents false negative.

[0032] Second aspect: An electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, performs the steps of the method provided in the first aspect.

[0033] Third aspect: A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in the first aspect.

[0034] The beneficial effects of this invention are:

[0035] 1. This invention effectively integrates the advantages of spectral and structural features by fusing digital surface models (DSM) generated from lidar point cloud data with multispectral data, significantly improving the identification accuracy of withered and healthy canopies and solving the problem of low identification accuracy caused by the limitation of applying single data features.

[0036] 2. This invention employs a multi-gradient feature fusion method to combine RGB images with near-infrared, infrared spectral features, and elevation features. This eliminates the need to calculate complex tree indices and perform feature selection, enabling accurate and rapid identification of damaged canopies. This provides an efficient and convenient new method for monitoring forestry health information.

[0037] 3. By comparing various classification methods such as random forest, support vector machine, maximum likelihood classifier and deep learning U-Net model, this invention provides a basis for model selection in different application scenarios, and enhances the applicability and flexibility of the method. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the forestry health information monitoring method of the present invention;

[0039] Figure 2 This is a flowchart illustrating the principle of the forestry health information monitoring method of the present invention.

[0040] Figure 3 This is a schematic diagram of RGB image acquisition by the UAV according to the present invention;

[0041] Figure 4 This is a schematic diagram of the DSM model and MSI image of the present invention;

[0042] Figure 5 This is a schematic diagram of the U-Net model structure of the present invention;

[0043] Figure 6 This is a schematic diagram illustrating the classification and recognition of different models in this invention;

[0044] Figure 7 This is a schematic diagram of the confusion matrix accuracy analysis of the present invention;

[0045] Figure 8 This is a schematic diagram of the electronic device structure of the present invention. Detailed Implementation

[0046] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0047] Existing forestry health information monitoring methods suffer from limitations such as single monitoring means and limited data sources. They also have weak early warning capabilities for forest pests and diseases and cannot provide timely and effective decision support for forestry management departments.

[0048] Example 1:

[0049] To address the aforementioned issues, this embodiment provides a method for monitoring forestry health information based on the fusion of features from multi-source heterogeneous remote sensing images. Figure 1 This is a schematic diagram of the process of the present invention. Figure 2 This is a schematic diagram illustrating the principle and flow of the present invention. The method includes:

[0050] S1. Use a drone equipped with a high-resolution multispectral camera and lidar to take aerial photos of the target sample plot and obtain remote sensing images.

[0051] Drones, with their high spatial resolution and timely delivery, provide assistance for the rapid and objective determination of forest structure, acquisition of data on the occurrence and spread of pests and diseases, and large-scale, precise forestry surveys.

[0052] When forest trees are attacked by pests and diseases, the transport of water and nutrients is hindered, causing the external color to change from green to red or gray, and the spectral reflectance also changes.

[0053] Chlorophyll plays a crucial role in plant life activities and states, serving as an indicator of plant growth and health. In multispectral data, red, near-infrared, and infrared bands are closely related to chlorophyll content and properties. When trees are affected by pests, diseases, or other damage, the spectral response and biochemical properties of their leaves are altered.

[0054] By using a multispectral camera and lidar sensor mounted on a drone to acquire orthophoto remote sensing data through a zigzag flight, this data acquisition method can comprehensively and efficiently cover the target plot. During the zigzag flight, the two sensors work together to collect data on the target plot from different dimensions.

[0055] S2. Process remote sensing images to generate optical orthophotos and lidar point cloud data of the target site;

[0056] Multispectral data of optical orthophotos include RGB images in the red, green, and blue bands, and MSI images in the red, green, blue, near-infrared, infrared 1, and infrared 2 bands.

[0057] like Figure 3 As shown, RGB images consist of only three bands: red, green, and blue. In forestry remote sensing, the near-infrared (805-875 nm), infrared band 1 (710-730 nm), and infrared band 2 (735-765 nm) bands of a multispectral sensor are more sensitive to chlorophyll. To distinguish the differences in classification results between RGB images and fused images with added spectral gradients (near-infrared and infrared bands), this embodiment uses different bands of a multispectral sensor to generate RGB images (red, green, blue) and MSI images (red, green, blue, near-infrared, infrared 1, infrared 2).

[0058] S3. Convert the two-dimensional planarization of the lidar point cloud data into a digital surface model (DSM), generate multispectral data of the optical orthophoto image, and perform feature fusion between the DSM and the multispectral data.

[0059] To further explore the correlation between canopy identification and ground disturbance features, additional feature variables were added, highlighting the significant height difference between tree canopies and ground disturbance objects. A digital surface model (DSM) generated from LiDAR point cloud data, capable of more finely representing object height variations, was used. Figure 4 As shown, DSM is used for multispectral data feature fusion, aiming to enhance high expressiveness in order to better capture the spatial features of trees.

[0060] In this embodiment, digital surface model data is fused into the fourth band of the RGB image and the seventh band of the MSI image, respectively, to obtain DSM+RGB and DSM+MSI images with highly expressive information and different gradient spectral characteristics.

[0061] Furthermore, after creating the feature fusion image, a low-pass filter is applied to the RGB image, MSI image, DSM+RGB image, and DSM+MSI image to reduce noise. The filtered image is then used for classification.

[0062] Using the aforementioned data, the multispectral data acquired by UAV aerial photography contains high-quality information that can be used to distinguish the health status of trees. LiDAR can accurately measure forest structural characteristics and has a strong correlation with canopy information. Combining these two types of data helps to obtain horizontal canopy reflectivity and enhances vertical structural information for forest monitoring.

[0063] S4. Divide the data into training and validation sets based on feature fusion, and train and optimize the deep learning model based on the training and validation sets.

[0064] Deep learning models can use the U-Net model for classification, while random forest, support vector machine, and maximum likelihood classifier are used for comparative classification. The U-Net model structure is as follows: Figure 5 As shown.

[0065] U-Net is based on a symmetric encoder-decoder structure, where the encoder progressively downsamples (pools) the input image to capture the image's contextual information.

[0066] like Figure 5As shown, given an input image of size 572x572x3, it first undergoes two 3x3 convolution operations to obtain a 568x568x64 feature map. Then, 2x2 max pooling is performed for downsampling. After pooling, the feature map size is halved, and the number of channels doubles, resulting in a 284x284x64 feature map. The next convolutional block then begins, with the number of channels increasing to 128. The feature map becomes smaller and smaller, while the number of channels increases, gradually transforming the extracted features from local and detailed to globally abstract.

[0067] The bridging layer is used to connect the encoder and decoder. After several downsampling steps, the feature map at the bottom layer is 32x32x512 in size. After passing through two 3x3 convolutional layers, it becomes 28x28x1024.

[0068] The decoder progressively upsamples the feature map to restore the spatial dimensions and details of the image, achieving precise localization. First, it performs a 2x upsampling, doubling the feature map size and halving the number of channels. The bottom feature map is then upsampled to obtain a size of 56x56x512.

[0069] Skip connections are a core component of U-Net. Upsampled feature maps are concatenated with corresponding feature maps from intermediate layers in the encoder path. For example, a 56x56x512 feature map in the decoder is concatenated with a feature map of the same size in the encoder. Skip connections directly pass the detailed and texture information extracted by the encoder to the decoder, combining it with the abstract and semantic information obtained from the upsampled data. This effectively compensates for the spatial information lost during downsampling, allowing the decoder to generate more accurate segmentation boundaries while restoring the original dimensions.

[0070] The merged feature map will then undergo two more 3x3 convolution operations to fuse the information brought by the skip connections. After several upsampling, skip connection and convolution operations, the feature map size will gradually recover.

[0071] Finally, a 1x1 convolutional layer is used to map the number of channels to the desired number of categories, resulting in a 388x388x1 image, where the value of each pixel represents the probability that the pixel belongs to the target category.

[0072] The training was performed using four feature fusion images with 3, 4, 6, and 7 bands respectively.

[0073] When dividing the training and validation sets using feature fusion data, five types of land surfaces—damaged canopy, healthy canopy, bare soil, weeds, and crops—are classified as Regions of Interest (ROIs) with equal area (e.g., 0.04 square meters each), uniform distribution, and random location. The ROIs are then divided into training and validation sets in a 7:3 ratio to reduce spatial correlation between the training and validation sets, and each classification model is trained.

[0074] To test the classification performance of multispectral images (MSI), 100 regions of interest were selected for training and 40 regions of interest were selected for validation. During the accuracy validation process, the classification results were divided into three categories: withered canopy, healthy canopy and others.

[0075] S5. Based on the trained and optimized deep learning model, the withered canopy, healthy canopy and other objects in the remote sensing image are classified to realize the monitoring of forestry health status, and the accuracy of the classification results is analyzed in detail.

[0076] By calculating the confusion matrix for each category, evaluation indicators such as producer accuracy, user accuracy, overall accuracy, and Kappa coefficient for each classification model are obtained to accurately assess the accuracy and reliability of the classification.

[0077] The Kappa coefficient is used to evaluate classification accuracy, and the formula is:

[0078]

[0079] Among them, P o To accurately simulate the scale, P c P represents the expected correct simulation scale under random conditions. p This represents the correct simulation ratio under ideal classification conditions, and its value is 100%.

[0080] If two images are identical, the Kappa coefficient is equal to 1; if the Kappa coefficient is greater than or equal to 0.75, it indicates good consistency and high simulation accuracy; if the Kappa coefficient is less than or equal to 0.4, it indicates poor consistency, large differences between the two images, and low simulation accuracy.

[0081] The Kappa coefficient can effectively reflect the classification accuracy of a model, but its value may be affected by the large number of samples of a certain class in the overall data, and may not be able to fully represent the situation of each class.

[0082] Therefore, this embodiment selects a confusion matrix to assist in accuracy verification, such as Figure 7 As shown, the confusion matrix statistically classifies the model's classification results, listing the number of correct and incorrect classifications of the actual and predicted values.

[0083] Producer precision (PA) is the ratio of the number of samples correctly identified by the classifier for a specific class to the total number of samples in that class. It measures the accuracy of the classifier in classifying or distinguishing a particular class; high producer precision indicates that the classifier can identify that class well. User precision (UA) is the ratio of the number of samples correctly identified by the classifier for a specific class to the total number of samples that the classifier classifies into that class. It evaluates the classifier's ability to correctly identify user samples; higher user precision indicates more reliable classification results. Overall precision (OA) is the ratio of the number of samples correctly classified by the classifier to the total number of samples in all classification results. It comprehensively reflects the accuracy of the entire classification process and is an important indicator of the overall performance of the classifier. Higher overall precision indicates that the classifier performs better overall in classifying samples of all classes and can more accurately classify samples into the correct categories. The formula is expressed as:

[0084]

[0085]

[0086]

[0087] Where OA represents overall accuracy, PA represents producer accuracy, UA represents user accuracy, TP represents true positive, TN represents true negative, FP represents false positive, and FN represents false negative.

[0088] Using the method of this embodiment, such as Figure 6 The classification results are shown. Weeds mainly grow on the edges of shelterbelts and along field ridges. In the RGB image classification results, healthy canopies were misclassified as weeds, and withered canopies were misclassified with bare ground. However, MSI images, which incorporate near-infrared and infrared spectral information, enhance the spectral reflectance characteristics of different green plants, bare ground, and withered branches, significantly improving the identification of healthy and damaged canopies.

[0089] Under natural conditions, the canopy usually extends more than 2 meters above the ground. Due to the elevation difference between the canopy and the ground surface, the DSM+RGB image, which incorporates elevation information, significantly improves the extraction effect of the healthy canopy compared to the simple RGB image.

[0090] The classification results of DSM+MSI images show that by adding feature factors, the misclassification problem caused by relying solely on MSI and RGB images can be overcome to a certain extent, and the recognition accuracy can be further improved.

[0091] Comparing traditional classifiers and the deep learning model U-Net, it can be found that the U-Net model has a certain degree of aggregation. Its ability to characterize the details of object edges is weaker than that of traditional classifiers. However, traditional classifiers produce small and discrete dirty data patches in different land categories, which is significantly reduced in the classification results of the U-Net model.

[0092] The canopy classification accuracy based on different classification models using fused images is as follows: Figure 7 As shown, the Random Forest (RF) model based on DSM+MSI images achieved the best recognition performance for healthy canopies, with a producer accuracy of 91.15%. The U-Net model followed closely behind with a producer accuracy of 91.01%. The Support Vector Machine (SVM) model based on RGB images had the lowest recognition accuracy at 76.63%.

[0093] In the identification of withered canopies, Random Forest (RF) based on MSI images achieved the highest accuracy with a producer precision of 96.93%, followed by Random Forest (RF) based on DSM+MSI images with a producer precision of 95.91%, slightly lower than that of MSI images by 0.72%. The lowest producer precision was achieved by Maximum Likelihood Classification (MLC) based on DSM+RGB images, at 78.82%.

[0094] Although MSI images achieved high accuracy in identifying withered canopies, their accuracy in identifying producers of healthy canopies was 5.03% lower than that of DSM+MSI images. Overall, DSM+MSI images performed best in identifying both healthy and withered canopies in shelterbelts, while RGB images showed the worst overall recognition performance.

[0095] This embodiment utilizes the method to fuse image features from different spectral feature data and lidar-derived data. Combined with ground survey data, four models for identifying the distribution of damaged canopies in farmland shelterbelts are constructed based on traditional classifiers and the U-Net model. By incorporating lidar elevation features into the multispectral images, not only is chlorophyll-sensitive spectral information enriched, but the height parameter information between the canopy and surface disturbances is also enhanced, effectively distinguishing disturbances and improving the accuracy of identifying the health status and damage condition of the farmland shelterbelt canopy.

[0096] By employing a feature fusion method, combining RGB images with near-infrared, infrared spectral features, and elevation features, the condition of the protective forest canopy is classified, avoiding the problem of low accuracy in identifying canopy health information based on single data features. Not only is damaged canopy identified, but the results show excellent performance, with the highest producer accuracy reaching 96.93% for withered canopies and 91.15% for healthy canopies.

[0097] By fusing features at different gradients, the MSI image, obtained by adding infrared and near-infrared band information to an RGB image, shows a greater improvement in classification accuracy than the RGB+DSM image, which only adds elevation information. The recognition effect is optimal when both are fused simultaneously. This provides a new approach for monitoring forest health information: by simply acquiring spectral reflectance data and combining it with a DSM derived from lidar, accurate and rapid identification of damaged canopies can be achieved without calculating complex tree indices or performing feature selection.

[0098] By comparing three classic classifiers—random forest, support vector machine, and maximum likelihood classification—with the U-Net model, it was confirmed that the random forest model achieves higher classification accuracy when features are sufficient. In the random forest model, after incorporating near-infrared and infrared spectral information into the RGB image and then adding DSM, the Kappa coefficients increased by 0.1039 and 0.1365, respectively. The U-Net model also achieves good results with fewer features. However, compared with traditional classifiers, the increase in its Kappa coefficient after adding spectral and DSM information is relatively low.

[0099] The present invention also provides an electronic device, Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 8 As shown, the electronic device may include a processor, a communications interface, memory, and a communication bus, wherein the processor, communications interface, and memory communicate with each other via the communication bus. The processor can invoke logical instructions from the memory, for example, to execute the following method:

[0100] S1. Use a drone equipped with a high-resolution multispectral camera and lidar to take aerial photos of the target sample plot and obtain remote sensing images;

[0101] S2. Process remote sensing images to generate optical orthophotos and lidar point cloud data of the target site;

[0102] S3. Convert the two-dimensional planarization of the lidar point cloud data into a digital surface model (DSM), generate multispectral data of the optical orthophoto image, and perform feature fusion between the DSM and the multispectral data.

[0103] S4. Divide the training set and validation set based on the feature fusion data, and train and optimize the deep learning model based on the training set and validation set;

[0104] S5. Based on the trained and optimized deep learning model, evaluate the classification accuracy of the deep learning model, classify withered canopies, healthy canopies and other objects in remote sensing images, and realize the monitoring of forestry health status.

[0105] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments, including, for example:

[0107] S1. Use a drone equipped with a high-resolution multispectral camera and lidar to take aerial photos of the target sample plot and obtain remote sensing images;

[0108] S2. Process remote sensing images to generate optical orthophotos and lidar point cloud data of the target site;

[0109] S3. Convert the two-dimensional planarization of the lidar point cloud data into a digital surface model (DSM), generate multispectral data of the optical orthophoto image, and perform feature fusion between the DSM and the multispectral data.

[0110] S4. Divide the training set and validation set based on the feature fusion data, and train and optimize the deep learning model based on the training set and validation set;

[0111] S5. Based on the trained and optimized deep learning model, evaluate the classification accuracy of the deep learning model, classify withered canopies, healthy canopies and other objects in remote sensing images, and realize the monitoring of forestry health status.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring forestry health information based on feature fusion of multi-source heterogeneous remote sensing images, characterized in that, include: S1. Use a drone equipped with a high-resolution multispectral camera and lidar to take aerial photos of the target sample plot and obtain remote sensing images; S2. Process remote sensing images to generate optical orthophotos and lidar point cloud data of the target site; S3. Convert the two-dimensional planarization of the lidar point cloud data into a digital surface model (DSM), generate multispectral data of the optical orthophoto image, and perform feature fusion between the DSM and the multispectral data. S4. Divide the training set and validation set based on the feature fusion data, and train and optimize the deep learning model based on the training set and validation set; S5. Based on the trained and optimized deep learning model, evaluate the classification accuracy of the deep learning model, classify withered canopies, healthy canopies and other objects in remote sensing images, and realize the monitoring of forestry health status.

2. The method according to claim 1, characterized in that, When the UAV takes aerial photos of the target sample plot, it uses a zigzag flight pattern to acquire orthophoto images.

3. The method according to claim 1, characterized in that, The multispectral data of the optical orthophoto image includes RGB images in the red, green, and blue bands, and MSI images in the red, green, blue, near-infrared, infrared 1, and infrared 2 bands.

4. The method according to claim 3, characterized in that, The feature fusion of DSM and multispectral data includes: The DSM was fused into the fourth band of the RGB image and the seventh band of the MSI image, respectively, to obtain DSM+RGB and DSM+MSI images with high expressive information and different gradient spectral features.

5. The method according to claim 1, characterized in that, The deep learning model uses the U-Net model for classification, and also employs random forest, support vector machine and maximum likelihood classifier for comparative monitoring and classification.

6. The method according to claim 1, characterized in that, When dividing the feature fusion data into training and validation sets, five types of land surfaces—damaged canopy, healthy canopy, bare soil, weeds, and crops—are classified and defined as regions of interest with equal area, uniform distribution, and random location. These regions of interest are then divided into training and validation sets in a 7:3 ratio.

7. The method according to claim 1, characterized in that, The deep learning model uses Kappa coefficients, confusion matrix, overall accuracy, producer accuracy, and user accuracy to evaluate classification accuracy.

8. The method according to claim 7, characterized in that, The Kappa coefficient is used to evaluate classification accuracy, and the formula is: Among them, P o To accurately simulate the scale, P c P represents the expected correct simulation scale under random conditions. p This represents the correct simulation ratio under ideal classification conditions.

9. The method according to claim 7, characterized in that, Classification accuracy is evaluated using producer accuracy and user accuracy, using the following formula: Where OA represents overall accuracy, PA represents producer accuracy, UA represents user accuracy, TP represents true positive, TN represents true negative, FP represents false positive, and FN represents false negative.

Images