Forest litter moisture content inversion method and system based on multispectral images
Through the combination of multispectral image inversion method and deep learning model, the problem of high cost and insufficient accuracy of forest litter moisture content prediction is solved, and efficient and accurate prediction of litter moisture content is achieved, which is suitable for forest fire risk assessment.
Patent Information
- Application Number
- CN202411314688.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-09-20
AI Technical Summary
The existing neural network-based forest litter moisture content prediction method is costly and has insufficient prediction accuracy, which cannot effectively solve the microscopic and local litter moisture content measurement needs in forest fire risk assessment.
Using the inversion method based on multispectral images, a two-stage training strategy is designed through deep learning models combining multi-band images and exponential images. The combination of unsupervised pre-training and supervised training is used to reduce dependence on label data and improve the model. robustness and generalization.
It significantly improves the accuracy and robustness of forest litter moisture content prediction, reduces data acquisition costs, and improves the generalization ability and practical application value of the model.
Smart Images

Figure CN118840664B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a forest litter moisture content inversion method and system based on multispectral images, and corresponding forest litter moisture content detection equipment. Background Art
[0002] Litter generally refers to the metabolic products produced by plants in nature during their growth and development. They are produced by the above-ground parts of plants and returned to the ground. As a source of material and energy for degraders, litter plays a role in maintaining the sustained stability of forest ecosystem functions. In general forest ecosystems, forest litter includes fallen branches, fallen trees, dead standing trees, fallen leaves, fallen bark, dead herbs, dead tree roots, fallen nutritional and reproductive organs, animal remains, and their alienated metabolites. Because it is rich in combustible organic matter, surface litter is also one of the most likely risk factors for forest fires. Among them, the moisture content of litter directly affects the difficulty and behavior of forest fires. Therefore, the fire department and meteorological department will regularly observe the moisture content of litter in different areas of the forest.
[0003] The drying method is the "gold standard" for measuring the moisture content of litter. The drying method requires collecting litter samples, weighing the mass of the wet samples, and then obtaining the mass of the dry samples through long-term drying. The moisture content of the litter is then calculated based on the water loss during the drying process. The drying method has high measurement accuracy, but because this method relies on manual operation, has high efficiency and low cost, and the energy supply in the wild is limited, this method is not suitable for measuring the moisture content of litter at multiple observation points over a large range.
[0004] The moisture content of forest litter can also be measured by the "wood stick method", the standing wave method, and methods based on meteorological data and numerical models, etc. Among them, the wood stick method uses standardized wood sticks as litter representatives, such as Figure 1 As shown in the figure, the moisture content of the wood stick is obtained by measuring the impedance of the wood stick, and then extended to the moisture content of other litter. The wood stick method only considers the litter gaining water (precipitation) or losing water (sunshine, wind, etc.) due to meteorological conditions, and does not consider the type of litter, especially the case that it contains initial moisture. The standing wave method uses Figure 2 The principle shown is to indirectly measure the moisture content by measuring the dielectric constant of the material. In the measured materials with different dielectric constants, when high-frequency electromagnetic waves propagate along the transmission line, due to the mismatch between the impedance at the probe and the transmission line impedance, standing waves are formed on the transmission line, which in turn causes changes in the voltage amplitude at both ends of the transmission line. The measurement range of litter moisture content based on the standing wave principle is small, the ground equipment is easily damaged, and the maintenance cost is high.
[0005] The purpose of predicting litter moisture content based on meteorological data is to establish a corresponding relationship between meteorological factors and litter moisture content. This type of method usually considers factors such as season, vegetation type, rainfall, light conditions, wind speed, atmospheric temperature and humidity, soil temperature and humidity, and deduce the moisture content of litter. The most typical example is the three moisture codes in the Canadian Fire Weather Indicator System (FWI): the fine fuel moisture code (FFMC), the litter moisture code (DMC), and the drought code (DC). The three moisture codes characterize the moisture conditions of different types of litter and are indexed representations of litter moisture. The mass moisture content can be obtained through simple mapping. This method relies on macroscopic measurements of meteorology and satellites and cannot obtain microscopic, local, and accurate litter moisture content.
[0006] The method of numerical model refers to the artificial design of various indexes related to moisture content, the use of expert knowledge to establish a mathematical model between multi-band images and moisture content, and then the use of numerical models to predict the moisture content of litter based on multi-band images. For example: Watson et al. proposed to estimate soil moisture by combining the daily variation of surface temperature obtained by thermal infrared bands with thermal models. Sandholt et al. found that there are soil moisture contours in the LST / NDVI feature space, and based on this, proposed the Temperature Vegetation Drought Index (TVDI) that reflects the dry and wet conditions of the region. Fensholt et al. proposed the Shortwave Infrared Water Stress Index (SIWSI) considering that leaf moisture has a high absorption rate in the shortwave infrared band. Yao et al. proposed the Shortwave Infrared Soil Moisture Index (SIMI) based on the 6th and 7th bands of MODIS. The above methods all use artificially designed indexes to deduce moisture content. Since the establishment of mathematical models relies on expert experience, and the improvement of the prediction accuracy of moisture content also requires massive sample data, the design of this method is difficult and the moisture content measurement accuracy is insufficient.
[0007] In addition, with the rise of artificial intelligence technology, some technicians have also proposed building machine learning / deep learning models, using litter image data with known moisture content information and training the model, so as to achieve the prediction of litter moisture content by the machine based on the image data. However, directly using natural images as input to predict litter moisture content requires high model fitting ability and requires the design of a neural network model with a more complex structure and more parameters, which makes the training of the network model difficult and requires a high demand for labeled data. The economic and time costs of obtaining real data on the moisture content of litter samples are extremely high, which will greatly increase the difficulty of implementing the solution and reduce the practical value of the litter moisture content prediction method based on neural networks. Summary of the invention
[0008] In order to solve the problems of high implementation cost and insufficient prediction accuracy in existing forest litter moisture content prediction schemes based on neural networks, the present invention provides a forest litter moisture content inversion method and system based on multispectral images, and corresponding forest litter moisture content detection equipment.
[0009] The present invention is implemented by the following technical solutions:
[0010] A method for inverting the moisture content of forest litter based on multispectral images comprises the following steps:
[0011] S1: Image forest litter in different areas in multiple bands, preprocess the obtained multi-band images, obtain a large number of sample image groups in different fields of view, and then form a pre-training set.
[0012] The generation method of each sample image group is as follows:
[0013] S11: Acquire multiple forest litter images of different bands in the same field of view, and perform exponential expansion on the images of different bands to obtain multiple exponential images of the same size.
[0014] S12: Multiple forest litter images of different bands and multiple index images in the same field of view are concatenated in the channel dimension to obtain a multispectral image I .
[0015] S13: Identify each multispectral image I The part containing forest litter is segmented according to the specified size to obtain U interrelated sub-images p i Construct a sample image group P: P={ p 1 ,…, p U}.
[0016] S2: Collect forest litter from several sample image groups in the pre-training set and measure them by drying method; the obtained moisture content data is used as label information of each sample image group; and then a training set containing multiple labeled sample image groups is obtained.
[0017] S3: Construct a moisture content prediction model, which is used to output corresponding moisture content prediction results according to the input sample image group.
[0018] The moisture content prediction model provided by the present invention includes a backbone network and a regression module. The backbone network adopts an image recognition model based on deep learning and is used to predict the moisture content of the input multispectral image. p Generate the corresponding feature vector fThe regression module is used to generate multiple feature vectors based on the output of the backbone network. f i Generate a moisture content prediction result.
[0019] S4: Choose any distance loss L Dist As the loss function, the pre-training set is used to perform unsupervised pre-training on the backbone network in the moisture content prediction model, so that the backbone network can predict the moisture content of each sub-image in the same sample image group P. p i The generated feature vectors f i The distance is close.
[0020] S5: retain the model parameters of the backbone network in the pre-training phase, including the distance loss L Dist and label similarity loss L reg The combined loss L As the loss function, the training set is used to train and test the moisture content prediction model, and the model parameters of the network model whose prediction accuracy meets the requirements are retained.
[0021] S6: Obtain multispectral images of forest litter in a specified area I , and split it into U The sub-images are then input into the moisture content prediction model trained in the previous step to obtain the moisture content data of forest litter in the specified area.
[0022] As a further improvement of the present invention, in step S11, the obtained multi-band image includes any multiple of the red band image R, the green band image G, the blue band image B, the near infrared band image NIR and the far infrared thermal imaging band image THM.
[0023] Each index image is generated using any of the multiple index methods including the moisture vegetation index MVI, enhanced vegetation index EVI, normalized difference water index NDWI, and normalized green-blue difference index NGBDI. The expressions of each index method are as follows:
[0024] .
[0025] As a further improvement of the present invention, in step S12, the multispectral image is recorded as I : I ∈ R H×W×(M+N) .
[0026] in, H and W Represents the length and width of the image,M Indicates the number of bands of a multi-band image; N Indicates the number of categories of the index image.
[0027] Each sub-image in the sample image set P is denoted by p i : .
[0028] Where U represents the number of sub-images segmented from each multispectral image.
[0029] As a further improvement of the present invention, in step S3, the backbone network adopts a CNN network with a Transformer or ResNet-18 structure.
[0030] Furthermore, the CNN network in the present invention adopts a ResNet-18 structure, which includes 17 convolutional layers in sequence and is finally connected to a global pooling layer.
[0031] As a further improvement of the present invention, in step S3, the regression module sequentially comprises a fusion layer, a fully connected layer, an activation layer, and a fully connected layer. The fusion layer is implemented in any of the following ways:
[0032] a. Concatenate the feature vectors extracted by the backbone network into a new fused feature vector with U times the original dimension.
[0033] b. Use average pooling or maximum pooling to take the average or maximum value of U feature vectors to obtain a new fused feature vector.
[0034] c. Use bilinear pooling and its variants to process the original feature vector to obtain a new fused feature vector.
[0035] As a further improvement of the present invention, in step S4, the distance loss in the pre-training stage L Dist Using Euclidean distance loss L EuclideanDist or Consin distance loss L consinDist , and the calculation formulas for the two are as follows:
[0036] ,
[0037] In the above formula, f i and f j Respectively represent any two sub-images in the sample image group P p i and p j The feature vector obtained after being processed by the backbone network.
[0038] As a further improvement of the present invention, in step S5, the expression of the joint loss L is: .
[0039] in, and They are the weights of distance loss and label similarity loss in the joint loss respectively; L reg The mean error loss MSE or mean absolute error loss MAE is used. The calculation formulas for the two are as follows:
[0040] ,
[0041] In the above formula, N represents the number of labeled sample image groups in the training set. Indicates the training set i The true label value of the sample image group, The model predicts the i The moisture content of the sample image groups.
[0042] The present invention also includes a forest litter moisture content prediction system based on multispectral images, which includes: a preprocessing module and a numerical prediction module.
[0043] The preprocessing module includes an image acquisition unit, an index expansion unit, an image synthesis unit, an image recognition unit and an image segmentation unit. The image acquisition unit is used to acquire a multi-band image of forest litter in a specified area under the same field of view. The index expansion unit is used to generate a variety of index images based on the multi-band image. The image synthesis unit is used to connect the multi-band image with each index image in series by channel to obtain a multispectral image. The image recognition unit is used to identify the area containing forest litter according to the image of the RGB channel in the multi-band image. The image segmentation unit is used to segment the multispectral image according to the target area identified by the image recognition unit to obtain multiple sub-images.
[0044] The numerical prediction module uses the moisture content prediction model trained in the aforementioned forest litter moisture content inversion method based on multispectral images. The numerical prediction module is used to predict the moisture content of forest litter in a specified area based on multiple input sub-images.
[0045] The present invention also includes a forest litter moisture content detection device, which includes an imaging device and a computer device.
[0046] The imaging device is used to take images of the target area in different bands under the same field of view by using multi-band imaging technology.
[0047] The computer device runs the aforementioned forest litter moisture content prediction system based on multispectral images. The computer device is used to first synthesize a multispectral image based on the multi-band image collected by the imaging device, and then perform image segmentation on the multispectral image to obtain multiple sub-images corresponding to the forest litter area; then each sub-image is input into a trained moisture content prediction model, and finally the moisture content prediction model outputs the moisture content of the forest litter in the target area.
[0048] The technical solution provided by the present invention has the following beneficial effects:
[0049] The present invention provides a method for predicting the moisture content of forest litter based on image data using a machine learning algorithm. In order to overcome the shortcomings of traditional methods in terms of accuracy, robustness and generalization, the present invention combines the original multispectral image with the artificial index image to design a multispectral image data containing richer feature information. The application of this data can significantly promote the convergence of the network model and improve the model accuracy.
[0050] The present invention also designs a two-stage training strategy to address the high cost of obtaining label information on the moisture content in forest litter images. During the pre-training process, based on the core assumption that "the moisture content of litter in different areas should be consistent in the same image taken by a multispectral camera", an unsupervised pre-training process is designed, and then the model parameters of the backbone network are pre-trained and adjusted through massive unlabeled data. Then, a small amount of sample data with supervised information is used to train the entire network model.
[0051] The solution provided by the present invention not only takes advantage of the deep learning adaptive feature learning, but also refers to expert knowledge. Expert knowledge is used to reduce the nonlinear fitting pressure of the model and reduce the demand for data. Massive unlabeled data participates in the training to ensure the robustness and generalization of the model. In addition, the multi-image joint decision-making used in the actual process of the solution of the present invention can further improve the prediction accuracy of the moisture content output by the network model in actual application. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:
[0053] Figure 1 It is the wooden stick used in measuring moisture content by the representative method introduced in the background technology.
[0054] Figure 2 This is the schematic diagram of the equipment for measuring moisture content using the standing wave method introduced in the background technology.
[0055] Figure 3This is a schematic diagram of the forest litter moisture content inversion method based on multispectral images provided in Example 1 of the present invention.
[0056] Figure 4 This is a flow chart of the forest litter moisture content inversion method based on multispectral images provided in Example 1 of the present invention.
[0057] Figure 5 This is a system architecture diagram of the forest litter moisture content prediction system based on multispectral images provided in Example 2 of the present invention.
[0058] Figure 6 This is a system topology diagram of the forest litter moisture content detection device provided in Example 3 of the present invention.
[0059] Figure 7 Images of RGB channels, far-infrared and near-infrared bands acquired for a specified area, and the synthesized multispectral image are displayed.
[0060] Figure 8 This is an image of the error distribution between the predicted results of the network model and the true value in the simulation experiment. DETAILED DESCRIPTION
[0061] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0062] Example 1
[0063] This embodiment provides a method for inverting the moisture content of forest litter based on multispectral images. This method uses an image recognition method based on a deep neural network to predict the moisture content of forest litter in a target area. In view of the accuracy and cost defects of such methods in the existing solutions in the field of forest litter moisture content prediction, this embodiment provides a solution that mainly makes the following improvements: Figure 3 Two improvements are shown:
[0064] 1. Adjust the type of sample images used to train the neural network model.
[0065] The technicians in this case believe that: the existing schemes usually use natural light imaging pictures as sample data, and the characteristic information related to moisture content in such data is relatively limited, which limits the improvement of the prediction performance of the network model. In the scheme of this embodiment, the technicians first introduced the multi-band images in the numerical prediction method into the scheme, then selected several indexes related to moisture content, and synthesized a variety of extended "index images" using the multi-band images, and finally fused the index images and multi-band images by channel, and finally used the obtained new multispectral images with richer multi-dimensional characteristic information as sample images for training the neural network model.
[0066] Second, improve the training strategy of the network model and reduce the scheme’s dependence on supervisory information in sample images.
[0067] The technicians in this case believe that: from a statistical perspective, in the field of view of the same multispectral image, forest litter in different areas is usually located in a smaller area, and the moisture content of forest litter in this area can be considered to be at a relatively close level. Therefore, the present invention divides the training process of the network model into two stages. In the first stage, different areas in each synthesized multispectral image are divided into multiple sub-images, and these sub-images are used to perform unsupervised pre-training on the neural network model, so that the feature information learned by the model from different sub-images is sufficiently close. In the latter stage, sample images with supervised information are prepared, and the pre-trained neural network is subjected to conventional training.
[0068] In the new training strategy provided in this embodiment, the sample images in the first stage do not contain supervisory information, so the image acquisition cost is extremely low, and the technicians expand the scale of sample data used in this stage. In the latter stage, the model has learned the multi-dimensional feature information implicit in the image, so only a smaller scale of labeled sample images is needed for training to achieve model convergence. Based on this two-stage training strategy, this embodiment can ensure the prediction accuracy of the network model finally trained on the basis of using only a limited number of labeled sample images.
[0069] In addition, this embodiment scheme also makes some optimizations to the architecture of the network model, etc., so as to achieve lightweight and local deployment of the scheme while maintaining the prediction accuracy and generalization of the scheme, and improve the ability of online real-time prediction of forest litter moisture content.
[0070] Specifically, Figure 4 As shown, the forest litter moisture content inversion method based on multispectral images provided in this embodiment includes the following steps:
[0071] S1: Image forest litter in different areas in multiple bands, preprocess the obtained multi-band images, obtain a large number of sample image groups in different fields of view, and then form a pre-training set. The generation method of each sample image group is as follows:
[0072] S11: Acquire multiple forest litter images of different bands in the same field of view, and perform exponential expansion on the images of different bands to obtain multiple exponential images of the same size.
[0073] In the scheme of this embodiment, the obtained multi-band image includes a red band image R, a green band image G, a blue band image B, a near-infrared band image NIR, and a far-infrared thermal imaging band image THM. Among them, the red band image R, the green band image G, and the blue band image B can be obtained by natural light imaging technology. Conventional CCD cameras or CMOS cameras can be used to obtain images of the R, G, and B bands, and the data of these bands can be used to synthesize conventional color images. NIR imaging is an infrared imaging technology that mainly records infrared radiation reflected by objects, with a wavelength range of 700~1350nm. It uses a near-infrared light source (such as natural light or an artificial infrared light source) to record infrared images of objects. The principle of THM imaging technology is based on the characteristics of terahertz waves. Terahertz waves are located in the far-infrared region of the electromagnetic spectrum and have the advantages of strong penetration and small absorption coefficient for non-polar substances.
[0074] It should be noted that: the present embodiment uses the imaging data of the above five different bands in the actual application scheme. If the technical conditions and economic costs allow, imaging data in other bands can be added in other embodiments. In addition, the present embodiment can use the imaging data of the above multiple different bands at the same time. In other embodiments, the types of multi-band images used can also be flexibly adjusted as needed.
[0075] This embodiment can further synthesize the required "index image" through multi-band images. The index image is a new type of data obtained by combining images of different bands according to the spectral characteristics of different targets. For example, the vegetation index is a new type of data formed by combining satellite data of visible light and near-infrared bands according to the spectral characteristics of vegetation. The vegetation index is a simple, effective and empirical measure of the surface vegetation status, and is therefore widely used in global and regional land cover, vegetation classification and environmental change monitoring based on satellite images.
[0076] This embodiment selects multiple related indexes according to the specific scenario of forest litter moisture content detection, including: Moisture Vegetation Index MVI, Enhanced Vegetation Index EVI, Normalized Difference Water Index NDWI, Normalized Green-Blue Difference Index NGBDI: The expressions of each index method are as follows:
[0077] .
[0078] Among them, the moisture vegetation index MVI and the enhanced vegetation index EVI are synthesized from images of the red band R and the infrared band NIR; the normalized difference water index NDWI is synthesized from images of the green band G and the infrared band NIR; and the normalized green-blue difference index NGBDI is synthesized from images of the green band G and the blue band B.
[0079] S12: Multiple forest litter images of different bands and multiple index images in the same field of view are concatenated in the channel dimension to obtain a multispectral image I .
[0080] In particular, in the solution of this embodiment, in order to obtain the required multispectral image, the original multi-band image should be kept consistent in the field of view of data acquisition, and the size of the obtained image should be the same. Specifically, the image of each band is recorded as I A : I A ∈ R H×W ,in, H and W represents the length and width of the image; the number of channels C of the original image of each band is 1. The exponential image is synthesized by images of different bands through matrix processing, and the size of the resulting image remains unchanged. The synthesized exponential image is recorded as: I B : I B ∈ R H×W , where the number of channels C of each index image is also 1. Finally, after the multi-band image and multiple index images are connected in series by channel, the obtained multispectral image is recorded as I : I ∈ R H×W×(M+N) .in, M Indicates the number of bands of a multi-band image; N Indicates the number of categories of the index image.
[0081] Similarly, as with multi-band images, in practical applications, technicians can also flexibly choose the type and number of index images to be used according to their needs.
[0082] S13: Identify each multispectral image I The part containing forest litter is segmented according to the specified size to obtain U interrelated sub-images p i Construct a sample image group P: P={p 1 ,…, p U Each sub-image in the sample image group P is denoted by p i : . Where U represents the number of sub-images segmented from each multispectral image.
[0083] In this embodiment, the original multispectral image is segmented for two purposes: one is to obtain multiple sub-sample images from the same sample image, so that the implicit constraint of consistent "water content" between sub-sample images can be used to achieve unsupervised pre-training of the network model. The second is to remove background information in the field of view of the sample image that is not related to forest litter, so as to avoid the interference information contained in the original multispectral image from having a negative impact on the training of the model.
[0084] It should be additionally explained that: in order to adapt to the input of the network model, the number and size of sub-images segmented from different original multispectral images in this embodiment should be consistent. Among them, for the same multispectral image, the sizes of the multiple sub-images segmented therefrom may be consistent or inconsistent. For example, assuming that the size of an original multispectral image is 1024×1024, and the number of segmented sub-images is 5, the sizes of the sub-images may be different, namely 128×128, 256×256, 124×124, 68×68, 56×56, or the size of all sub-images is 224×224. Of course, in order to facilitate the preprocessing of large-scale data, all sub-images segmented from multispectral images can adopt a unified size. In this embodiment, the size of each sub-image is 224×224.
[0085] In addition, when performing image segmentation, in order to ensure that the area of forest litter contained in each sub-image accounts for a large enough proportion, overlap is allowed between the sub-images. The image segmentation process of this embodiment changes the size of the multispectral image, and the number of channels of each segmented sub-image remains consistent with the number of channels of the original multispectral image.
[0086] Finally, the pre-training set obtained in this embodiment includes a plurality of sample image groups P, each of which includes a plurality of sub-images derived from the same multispectral image. p i , the number and size of each sub-image in each sample image group remain consistent, and the content of each sub-image corresponds to the area of forest litter.
[0087] S2: Collect forest litter from several sample image groups in the pre-training set and measure them by drying method; the obtained moisture content data is used as label information of each sample image group, thereby obtaining a training set containing multiple labeled sample image groups.
[0088] In this embodiment, the amount of sample data in the pre-training set is relatively large, and most of the images in the pre-training set have not actually been measured for moisture content, so the cost of acquiring this type of sample data is relatively low. Technicians can use a multi-band camera to collect and pre-process images in different areas. The samples in the training set are randomly selected from the pre-training set. Unlike the pre-training set, after collecting the multi-band images in the training set, technicians also need to sample the forest litter contained therein. The sampled materials are sent to an oven for drying, and then the actual moisture content of the forest litter is measured. The moisture content of the forest litter Usually measured by mass, the calculation formula is as follows:
[0089] ;
[0090] in, m 1 is the original quality of forest litter; m 2 The quality of forest litter after drying; m 1 - m 2 It is the mass of water contained in forest litter.
[0091] Since the moisture content label information of each sample image group needs to be obtained through field sampling and drying measurement, which is a time-consuming and labor-intensive process and will incur high economic costs, the number of samples in the training set designed in this embodiment is significantly lower than the sample data in the pre-training set.
[0092] S3: Construct a moisture content prediction model, which is used to output the corresponding moisture content prediction result according to the input sample image group.
[0093] The moisture content prediction model provided in this embodiment includes a backbone network and a regression module. The backbone network adopts an image recognition model based on deep learning and is used to predict the moisture content of the input multispectral image. p Generate the corresponding feature vector f The regression module is used to generate multiple feature vectors based on the output of the backbone network. f i Generate a moisture content prediction result.
[0094] Specifically, the backbone network can adopt the Convolutional Neural Networks (CNN) commonly used in the field of image recognition. For example, this embodiment adopts the CNN network with the classic ResNet-18 structure as the backbone network. In other embodiments, the backbone network can also adopt other image recognition models based on deep learning, such as the most popular Transformer architecture.
[0095] Specifically, the original network architecture of the ResNet-18 structure used in the backbone network of this embodiment includes 17 convolutional layers, and finally a global pooling layer and a fully connected layer are connected. The last fully connected layer is deleted in the actual solution of this embodiment, that is, features are extracted using 17 convolutional layers and a global pooling layer in the backbone network.
[0096] In this embodiment, the regression module includes a fusion layer, a fully connected layer, an activation layer, and a fully connected layer in sequence. The fusion layer is implemented in any of the following ways:
[0097] a. Concatenate the feature vectors extracted by the backbone network into a new fused feature vector with U times the original dimension.
[0098] b. Use average pooling or maximum pooling to take the average or maximum value of U feature vectors to obtain a new fused feature vector.
[0099] c. Use bilinear pooling and its variants to process the original feature vector to obtain a new fused feature vector.
[0100] Specifically, the details of the implementation method of the fusion layer are shown in Table 1:
[0101] Table 1: Feature description of the fusion layer in the regression module:
[0102]
[0103] In this embodiment, considering that the input sample image group contains 5 sub-images, in order to realize the synchronous extraction of feature information in the 5 sub-images, the moisture content prediction model can use 5 backbone networks with shared weights to process the 5 input sub-images respectively. Then the extracted 5 feature vectors are respectively input into the regression module.
[0104] S4: Choose any distance loss L Dist As the loss function, the pre-training set is used to perform unsupervised pre-training on the backbone network in the moisture content prediction model, so that the backbone network can predict the moisture content of each sub-image in the same sample image group P. p i The generated feature vectorsf i The distance is close.
[0105] In this embodiment, the training goal of the pre-training stage is to make the feature vectors extracted by the backbone network from the same group of sub-images close enough. Based on this goal, the loss function of the pre-training stage adopts the distance loss L Dist , specifically, the commonly used Euclidean distance loss can be used L EuclideanDist or Consin distance loss L consinDist , the calculation formulas of the two are as follows:
[0106] ,
[0107] In the above formula, f i and f j Respectively represent any two sub-images in the sample image group P p i and p j The feature vector obtained after being processed by the backbone network.
[0108] In the pre-training process of this embodiment, a large number of unlabeled sample image groups can be used to perform unsupervised training on the backbone network, and then in the iterative process, the backbone network can accurately distinguish the feature information contained in the sample image through continuous learning. After the pre-training and parameter adjustment of the backbone network, multiple feature vectors with close distances can be extracted from multiple sub-images of the same source, that is, the pre-training goal that the moisture content of litter in different sub-images under the same field of view should be consistent is met.
[0109] S5: retain the model parameters of the backbone network in the pre-training phase, including the distance loss L Dist and label similarity loss L reg The combined loss L As the loss function, the training set is used to train and test the moisture content prediction model, and the model parameters of the network model whose prediction accuracy meets the requirements are retained.
[0110] In this embodiment, the pre-trained backbone network is combined with the regression module for joint training, so that the moisture content prediction model can output the predicted results of the moisture content of the forest litter contained therein according to the input multiple sub-images. Specifically, the training objectives of this embodiment in the training stage include two: one is to make the distance between the feature vectors extracted from the same group of sub-images by the backbone network as close as possible, and the other is to make the moisture content prediction results normalized by the regression module consistent with the data in the real moisture content label. Based on this training goal, the expression of the joint loss L in the training stage set in this embodiment is:
[0111] .
[0112] in, and They are the weights of distance loss and label similarity loss in the joint loss respectively; L reg The mean error loss MSE or mean absolute error loss MAE is used. The calculation formulas for the two are as follows:
[0113] ,
[0114] In the above formula, N represents the number of labeled sample image groups in the training set, Indicates the training set i The true label value of the sample image group, The model predicts the i The moisture content of the sample image groups.
[0115] In the actual solution of this embodiment, considering that the goal of close feature vector distance and similar water content prediction results in the model prediction process is equally important, the weights of the two losses are and The average value is 1.
[0116] S6: Obtain multispectral images of forest litter in a specified area I , and split it into U The sub-images are then input into the moisture content prediction model trained in the previous step to obtain the moisture content data of forest litter in the specified area.
[0117] Example 2
[0118] Based on the scheme of Example 1, this embodiment further provides a forest litter moisture content prediction system based on multispectral images, such as Figure 5 As shown, it includes: a preprocessing module and a numerical prediction module.
[0119] The preprocessing module includes an image acquisition unit, an index expansion unit, an image synthesis unit, an image recognition unit and an image segmentation unit. The image acquisition unit is used to acquire a multi-band image of forest litter in a specified area under the same field of view. The index expansion unit is used to generate a variety of index images based on the multi-band image. The image synthesis unit is used to connect the multi-band image with each index image in series by channel to obtain a multispectral image. The image recognition unit is used to identify the area containing forest litter according to the image of the RGB channel in the multi-band image. The image segmentation unit is used to segment the multispectral image according to the target area identified by the image recognition unit to obtain multiple sub-images.
[0120] The numerical prediction module adopts the moisture content prediction model trained in the forest litter moisture content inversion method based on multispectral images in Example 1. The numerical prediction module is used to predict the moisture content of forest litter in a specified area based on multiple input sub-images.
[0121] Example 3
[0122] Based on Examples 1 and 2, this example further provides a forest litter moisture content detection device. The forest litter moisture content detection device is an actual product solution for implementing the method in Example 1 and the hardware architecture in Example 2. Specifically, Figure 6 As shown, it includes imaging equipment and computer equipment.
[0123] In this embodiment, the imaging device is used to use multi-band imaging technology to capture images of different bands of the target area in the same field of view. As needed, the imaging device of this embodiment adopts a multi-vision imaging system, in which an RGB imaging module, a NIR imaging module, a THM imaging module, and the like can be integrated at the same time.
[0124] The forest litter moisture prediction system based on multispectral images of Example 2 is run in the computer device. The computer device in this embodiment can be an embedded device integrated into the back end of the imaging device. An independent device independent of the imaging device can also be used. Specifically, the computer device in this embodiment can be an intelligent terminal, a tablet computer, a laptop computer, a desktop computer, a rack server, a blade server, a tower server or a cabinet server (including an independent server, or a server cluster composed of multiple servers) that can execute programs.
[0125] The computer device indicated in this embodiment includes at least but is not limited to: a memory and a processor that can be connected to each other through a system bus. Among them, the memory (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory can be an internal storage unit of a computer device, such as a hard disk or a memory of the computer device. In other embodiments, the memory can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is generally used to store an operating system and various application software installed on the computer device. In addition, the memory can also be used to temporarily store various types of data that have been output or are to be output.
[0126] The processor may be a central processing unit (CPU), a graphics processing unit (GPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor is generally used to control the overall operation of a computer device. In this embodiment, the processor is used to run program codes stored in a memory or process data.
[0127] In this embodiment, the computer device is used to first synthesize a multispectral image based on the multi-band image collected by the imaging device, and then perform image segmentation on the multispectral image to obtain multiple sub-images corresponding to the forest litter area; then each sub-image is input into a trained moisture content prediction model, and finally the moisture content prediction model outputs the moisture content of the forest litter in the target area.
[0128] Verification experiment
[0129] In order to verify the effectiveness of the forest litter moisture content inversion method based on multispectral images provided by the present invention, technicians carried out relevant verification experiments.
[0130] 1. Dataset
[0131] The technicians deployed fixed multispectral cameras at multiple observation sites. The cameras were installed on poles to obtain images of the ground below. Finally, 1.2 million multi-band images were collected using these multispectral cameras. The image data obtained includes five bands of imaging: red band image R, green band image G, blue band image B, near-infrared band image NIR, and far-infrared thermal imaging band image THM. Among them, the imaging band of the near-infrared image is 850 nanometers-1200 nanometers; the imaging band of the far-infrared imaging is 8 microns-14 microns.
[0132] Figure 7 The images of RGB channels, far-infrared and near-infrared bands acquired in a specified area are shown separately, as well as an example of a multispectral image synthesized using image data from different bands.
[0133] The technicians also sampled the forest litter in the corresponding shooting area in the image, and dried the samples at 120 degrees Celsius for 5 hours using drying equipment. They compared the water loss before and after drying to obtain the true moisture content of the 1,000 samples.
[0134] Then, when collecting data, the technicians manually marked five areas with relatively dense forest litter. The algorithm automatically cropped the images of each area to obtain multispectral images of five local areas, and used them as sub-images of the corresponding sample images.
[0135] 2. Model and training
[0136] In this experiment, the CNN network with ResNet-18 structure is used as the backbone network. The five parallel feature extraction channels formed by the backbone network are followed by a regression module. In the experiment, the specific architecture of the backbone network is shown in the following table:
[0137] Table 2: Backbone network structure design
[0138]
[0139] In addition, in this experiment, the feature vector output by the GlobalAveragePooling layer is directly used in the pre-training stage to calculate the distance loss for optimizing the backbone network. In the training stage, the regression module is connected after GlobalAveragePooling. The fusion layer in the regression module uses feature concatenation to perform feature fusion, and regression loss is used to optimize the backbone network and regression module.
[0140] In the pre-training stage, the stochastic gradient descent method is used to optimize the backbone network, and 64 image samples are input at a time. The training is performed for 7 cycles on the pre-training dataset of 1.2 million samples, with an initial learning rate of 5×10 -2After the 2nd, 4th, and 6th cycles, the learning rate is multiplied by 0.1.
[0141] In the training phase, the stochastic gradient descent method is used to optimize the backbone network, and 64 image samples are input at a time. The training is performed for 300 cycles on a training dataset of 1000 samples, with an initial learning rate of 1×10 -4 , after the 150th, 200th, and 250th cycles, the learning rate is multiplied by 0.1.
[0142] 3. Test results
[0143] Use the trained model to measure the sample data of the test set and draw the following Figure 8 The measurement error statistics diagram is shown in the figure. In the figure, the horizontal axis is the true value of the moisture content, the vertical axis is the predicted value of the moisture content, and a discrete point represents a test sample.
[0144] exist Figure 8 In the figure, the 45° solid line is the baseline, and the closer the point is to it, the smaller the error; the two 45° dotted lines are 5% error range lines. By observing the data in the figure, it can be seen that the prediction results of the scheme of the present invention are mostly concentrated near the two dotted lines. The prediction accuracy of the scheme of the present invention can be controlled between 5% and 10%, which is relatively high and has high practical value.
[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for inverting the moisture content of forest litter based on multispectral images, characterized in that: It includes the following steps: S1: Image forest litter in different areas at multiple bands, pre-process the obtained multi-band images, obtain a large number of sample image groups at different fields of view, and then form a pre-training set; The method for generating the sample image group is as follows: S11: acquiring multiple forest litter images of different bands in the same field of view, and performing exponential expansion on the images of different bands to obtain multiple exponential images of the same size; S12: Multiple forest litter images of different bands and multiple index images in the same field of view are concatenated in the channel dimension to obtain a multispectral image I ; S13: Identify each multispectral image I The part containing forest litter is segmented according to the specified size to obtain U interrelated sub-images p i Construct a sample image group P: P={ p 1 ,…, p U }; S2: collecting forest litter from a plurality of sample image groups in the pre-training set and measuring them by a drying method, and obtaining moisture content data as label information of each sample image group; thereby obtaining a training set comprising a plurality of labeled sample image groups; S3: construct a moisture content prediction model, which is used to output the corresponding moisture content prediction result according to the input sample image group; The moisture content prediction model includes a backbone network and a regression module; the backbone network adopts an image recognition model based on deep learning and is used to predict the moisture content of the input multispectral image. p Generate the corresponding feature vector f The regression module is used to generate multiple feature vectors according to the output of the backbone network. f i Generate a moisture content prediction result; S4: Choose any distance loss L Dist As the loss function, the backbone network in the moisture content prediction model is unsupervisedly pre-trained using the pre-training set, so that the backbone network can predict the moisture content of each sub-image in the same sample image group P. p i The generated feature vectors f i The distance is close; S5: retain the model parameters of the backbone network in the pre-training phase, including the distance loss L Dist and label similarity loss L reg The combined loss L As a loss function, the water content prediction model is trained and tested using the training set, and the model parameters of the network model whose prediction accuracy meets the requirements are retained; S6: Obtain multispectral images of forest litter in a specified area I , and split it into U The sub-images are then input into the moisture content prediction model trained in the previous step to obtain the moisture content data of forest litter in the specified area.
2. The method for inverting the moisture content of forest litter based on multispectral images according to claim 1, characterized in that: In step S11, the multi-band image obtained includes any multiple of a red band image R, a green band image G, a blue band image B, a near infrared band image NIR, and a far infrared thermal imaging band image THM.
3. The method for inverting the moisture content of forest litter based on multispectral images according to claim 2, characterized in that: In step S11, each index image is generated by using any of the multiple index methods including the moisture vegetation index MVI, the enhanced vegetation index EVI, the normalized difference water index NDWI, and the normalized green-blue difference index NGBDI; the expressions of each index method are as follows: 。 4. The method for inverting the moisture content of forest litter based on multispectral images according to claim 2, characterized in that: In step S12, the multispectral image is recorded as I : I ∈ R H×W×(M+N) ,in, H and W Represents the length and width of the image, M Indicates the number of bands of a multi-band image; N Indicates the number of categories of the index image; Each sub-image in the sample image group P is denoted as p i : , where U represents the number of sub-images segmented from each multispectral image.
5. The method for inverting the moisture content of forest litter based on multispectral images according to claim 2, characterized in that: In step S3, the backbone network adopts a Transformer or CNN network; and / or The CNN network adopts the ResNet-18 structure, which includes 17 convolutional layers in sequence and is finally connected to a global pooling layer.
6. The method for inverting the moisture content of forest litter based on multispectral images according to claim 5, characterized in that: In step S3, the regression module includes a fusion layer, a fully connected layer, an activation layer, and a fully connected layer in sequence; wherein the fusion layer is implemented in any of the following ways: a. Concatenate the feature vectors extracted by the backbone network into a new fused feature vector with U times the original dimension; b. Use average pooling or maximum pooling to take the average or maximum value of U feature vectors to obtain a new fused feature vector; c. Use bilinear pooling and its variants to process the original feature vector to obtain a new fused feature vector.
7. The method for inverting the moisture content of forest litter based on multispectral images according to claim 1, characterized in that: In step S4, the distance loss in the pre-training phase L Dist Using Euclidean distance loss L EuclideanDist or Consin distance loss L consinDist , and the calculation formulas for the two are as follows: , In the above formula, f i and f j Respectively represent any two sub-images in the sample image group P p i and p j The feature vector obtained after being processed by the backbone network.
8. The method for inverting the moisture content of forest litter based on multispectral images according to claim 7, characterized in that: In step S5, the expression of the joint loss L is: ; in, and They are the weights of distance loss and label similarity loss in the joint loss respectively; L reg The mean error loss MSE or mean absolute error loss MAE is used. The calculation formulas for the two are as follows: , In the above formula, N represents the number of labeled sample image groups in the training set. Indicates the training set i The true label value of the sample image group, The model predicts the i The moisture content of the sample image groups.
9. A forest litter moisture content prediction system based on multispectral images, characterized in that: It includes: A preprocessing module, which includes an image acquisition unit, an exponential expansion unit, an image synthesis unit, an image recognition unit and an image segmentation unit; The image acquisition unit is used to acquire a multi-band image of forest litter in a specified area under the same field of view; the index expansion unit is used to generate a plurality of index images according to the multi-band image; the image synthesis unit is used to connect the multi-band image and each index image in series according to channels to obtain a multi-spectral image; the image recognition unit is used to recognize the area containing forest litter according to the image of the RGB channel in the multi-band image; the image segmentation unit is used to perform image segmentation on the multi-spectral image according to the target area recognized by the image recognition unit to obtain a plurality of sub-images; A numerical prediction module, which adopts the moisture content prediction model trained in any one of the forest litter moisture inversion methods based on multispectral images as described in claims 1-8; the numerical prediction module is used to predict the moisture content of forest litter in a specified area based on multiple input sub-images.
10. A device for detecting moisture content of forest litter, comprising: An imaging device, which is used to use multi-band imaging technology to capture images of a target area in different bands in the same field of view; A computer device, in which the forest litter moisture content prediction system based on multispectral images as described in claim 9 is run; the computer device is used to first synthesize a multispectral image based on the multi-band image collected by an imaging device, and then perform image segmentation on the multispectral image to obtain a plurality of sub-images corresponding to the forest litter area; then each sub-image is input into a trained moisture content prediction model, and finally the moisture content prediction model outputs the moisture content of the forest litter in the target area.
