Forage grass identification method based on three-dimensional self-encoding multi-scale feature extraction

Abstract

The invention relates to the technical field of neural networks and image recognition, in particular to a pasture recognition method based on three-dimensional self-encoding multi-scale feature extraction, which comprises the following steps: acquiring a hyperspectral image of a region to be recognized; the hyperspectral image is input into a preset pasture identification model, the pasture coverage type of the to-be-identified area is output, the pasture identification model is obtained through training of a training set, the training set comprises low-altitude unmanned aerial vehicle hyperspectral images containing a plurality of pasture types, and the pasture coverage type of the to-be-identified area is obtained through the low-altitude unmanned aerial vehicle hyperspectral images. The pasture identification model is constructed based on a three-dimensional attention automatic coding mechanism and a multi-scale multi-layer convolutional network. The invention provides a three-dimensional attention automatic coding mechanism and a multi-scale multi-layer convolutional network, and the constructed and trained pasture identification model 3DAMCN can provide a new method for identification monitoring of pasture and research of hyperspectral images.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network and image recognition, and in particular to a forage grass recognition method based on three-dimensional autoencoder multi-scale feature extraction. Background Art

[0002] With the rapid development of remote sensing technology, hyperspectral imaging technology can provide continuous and detailed spectral information by segmenting images through spectral reconstruction, and has become an important tool in the fields of surface type identification, vegetation monitoring and ecological assessment. Compared with traditional imaging technology, hyperspectral imaging technology can capture continuous spectral bands in a wider wavelength range, achieving more detailed identification and evaluation of objects. By mounting the hyperspectral imaging system on the UAV platform, real-time, non-contact, non-destructive testing can be achieved, and areas that traditional technologies cannot detect can be detected, improving the efficiency of data acquisition of the target, and providing data support for efficient and rapid identification of surface object types.

[0003] As an important agricultural resource, grassland forage not only affects the development of livestock industry, but also plays a key role in soil protection and water resource protection. With the development of animal husbandry, the demand for forage has gradually increased, resulting in a shortage of forage and poor quality. By identifying the types of forage, we can understand the characteristics and growth conditions of forage, thereby optimizing the forage planting structure and increasing forage yield. It plays an important role in the cultivation of high-quality varieties and the sustainable use of grassland resources.

[0004] Traditional grassland forage identification methods rely on manual visual inspection and empirical judgment, which is inefficient and difficult to cover large grassland areas. Many varieties of forage have similar appearances and colors and grow in the same area, so it is difficult to identify forage. Most existing studies use two-dimensional images to study forage. The image clarity is relatively high, but the characteristics of the spectral dimension cannot be analyzed, and the recognition efficiency is low for forage with similar textures and colors.

[0005] Hyperspectral imaging technology uses a wide range of bands and high resolution to capture the subtle features and spectral characteristics of plants, and can achieve functions such as accurate vegetation identification, growth monitoring and health assessment, providing strong technical support for agricultural development. Combined with the UAV platform equipped with a hyperspectral imaging system, it can quickly obtain forage images, realize large-scale, high-resolution data collection, and provide more detailed information in the spectral and spatial dimensions, solving the problems of low efficiency and limited coverage of traditional methods. It improves the identification and monitoring efficiency of grassland ecosystems and provides strong support for the scientific management and protection of grassland resources.

[0006] At present, some studies have applied deep learning to the recognition of hyperspectral images of forage from drones, extracted the spectral features and spatial features of images, and realized efficient recognition of objects. Wang constructed a streamlined 2D-CNN model with different feature enhancement modules, and used an improved deep separable convolution method to classify desert grassland plants, overcoming the limitations of traditional grassland survey methods such as low efficiency and insufficient spatial resolution. Lyu used principal component analysis (PCA) and multi-layer perceptron neural network to extract spectral feature parameters and perform spectral dimension reduction, and used multi-endmember spectral mixture analysis and fully constrained least squares hyperspectral unmixing to identify typical vegetation species. Jin proposed a 3D-RNet-O model based on convolutional neural network, optimized the hyperparameter combination on the basis of improving the residual block structure, and effectively solved the problems of low vegetation reflectance faced by hyperspectral remote sensing recognition and classification of grassland degradation indicators. Wei collected grassland samples in the wild, selected sample label data from vegetation map data and nature reserve functional zoning data, extracted spatial features using extended morphological profiles (EMPs), and obtained classification results using random forest algorithm.

[0007] In the study of forage images, some studies use public data sets. In addition to forage, the images also contain crops, buildings and other objects. This type of research improves the recognition accuracy, but has limitations on the research and development of forage. Some studies use near-ground hyperspectral images of forage. This type of data set is shot at a short distance and cannot be used for large-scale recognition and analysis. Some studies also use remote sensing images. The image shooting distance is too far, and even if the resolution is high, it is still impossible to accurately distinguish the types of objects. In response to the above problems, the present invention provides a forage recognition method based on three-dimensional autoencoder multi-scale feature extraction. Summary of the invention

[0008] The purpose of the present invention is to provide a forage recognition method based on three-dimensional autoencoding multi-scale feature extraction, propose a three-dimensional attention automatic encoding mechanism and a multi-scale multi-layer convolutional network, and the constructed and trained 3DAMCN can provide a new method for forage recognition monitoring and hyperspectral image research.

[0009] To achieve the above object, the present invention provides the following solutions:

[0010] A forage grass recognition method based on three-dimensional autoencoder multi-scale feature extraction, comprising:

[0011] Collecting hyperspectral images of the area to be identified;

[0012] The hyperspectral image is input into a preset grass recognition model, and the grass coverage type of the area to be identified is output, wherein the grass recognition model is obtained by training with a training set, and the training set includes low-altitude UAV hyperspectral images containing several grass types. The grass recognition model is constructed based on a three-dimensional attention automatic encoding mechanism and a multi-scale multi-layer convolutional network.

[0013] Optionally, constructing the training set includes:

[0014] Determine the collection area;

[0015] Taking hyperspectral images of forage grass in the collection area by flying a drone at low altitude;

[0016] The hyperspectral image of forage grass is spliced, cropped, pixelated and principal component analyzed to obtain a hyperspectral label image of forage grass and construct the training set.

[0017] Optionally, the forage recognition model includes: a first feature extraction module, a three-dimensional feature weighting module, and a second feature extraction module, wherein the first feature extraction module is used to extract the three-dimensional features of the input image; the three-dimensional feature weighting module is used to perform weighted calculation on the three-dimensional features and then perform self-encoding operation to obtain three-dimensional weighted features; the second feature extraction module is used to perform feature extraction on the three-dimensional weighted features, obtain image features and output the forage category in the input image.

[0018] Optionally, the first feature extraction module adopts a convolutional network, the three-dimensional feature weighting module adopts a three-dimensional attention automatic encoding mechanism, and the second feature extraction module adopts a multi-scale multi-layer convolutional network.

[0019] Optionally, the three-dimensional attention automatic encoding mechanism includes a channel attention unit, a spatial attention unit, and a self-encoding unit, wherein the channel attention unit is used to process space X and space Y in the three-dimensional feature to obtain channel weighted features; the spatial attention unit is used to process spectrum Z in the three-dimensional feature to obtain spatial weighted features; the self-encoding unit is used to encode and then decode the channel weighted features and spatial weighted features to obtain the three-dimensional weighted features.

[0020] Optionally, the multi-scale multi-layer convolution network includes two-dimensional multi-scale multi-layer convolution and three-dimensional multi-scale multi-layer convolution, wherein the two-dimensional multi-scale multi-layer convolution is used to extract the features of space X and space Y in the three-dimensional weighted features; the three-dimensional multi-scale multi-layer convolution is used to extract the features of spectrum Z in the three-dimensional weighted features.

[0021] Optionally, the two-dimensional multi-scale multi-layer convolution and the three-dimensional multi-scale multi-layer convolution are both composed of a number of convolution layers with gradually smaller convolution kernels.

[0022] The beneficial effects of the present invention are:

[0023] The present invention photographed hyperspectral images of forage grass by low-altitude unmanned aerial vehicle on-site, proposed a three-dimensional attention automatic encoding mechanism, calculated the weighted features of the three-dimensional space, and compressed and reconstructed the features, thereby improving the performance of key features. In addition, a multi-scale, multi-layer convolutional network was proposed, which used continuous 2D and 3D convolutions of different sizes to capture spatial and spectral features. The combination of large-size convolution kernels followed by small-size convolution kernels was used to improve the network's ability to search for global and detail features. The constructed and trained 3DAMCN showed excellent performance in recognition tasks, providing a new method for the recognition and monitoring of forage grass and the study of hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A schematic diagram of a three-dimensional autoencoder multi-scale multi-layer convolutional network structure and workflow according to an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of the structure and workflow of a three-dimensional attention automatic encoding mechanism according to an embodiment of the present invention;

[0027] Figure 3 Schematic diagram of the channel attention mechanism structure and workflow of an embodiment of the present invention;

[0028] Figure 4 Schematic diagram of the structure and workflow of the spatial attention mechanism of an embodiment of the present invention;

[0029] Figure 5 Schematic diagram of the structure and workflow of the autoencoder according to an embodiment of the present invention;

[0030] Figure 6 A schematic diagram of a multi-scale, multi-layer convolutional network structure and workflow according to an embodiment of the present invention;

[0031] Figure 7 A schematic diagram of a two-dimensional multi-scale multi-layer convolution structure and workflow of an embodiment of the present invention;

[0032] Figure 8A schematic diagram of a three-dimensional multi-scale multi-layer convolution structure and workflow of an embodiment of the present invention;

[0033] Fig. 9 The accuracy and loss curves of the three-dimensional autoencoder multi-scale multi-layer convolutional network in the LA-UFHI dataset training process of the embodiment of the present invention;

[0034] Fig.10 This is a comparison diagram of the recognition results of the three-dimensional autoencoder multi-scale multi-layer convolutional network and the comparison network on the LA-UFHI dataset according to an embodiment of the present invention, wherein (a) is a real landmark image of the LA-UFHI dataset, (b) is the recognition result of 3D-ResAE on the LA-UFHI dataset, (c) is the recognition result of 3D-MACNet on the LA-UFHI dataset, (d) is the recognition result of DC-CapsNet on the LA-UFHI dataset, (e) is the recognition result of 3D-DenseNet on the LA-UFHI dataset, and (f) is the recognition result of 3DAMCN on the LA-UFHI dataset;

[0035] Fig.11 This is a comparison diagram of the recognition results of the three-dimensional autoencoder multi-scale multi-layer convolutional network and the comparison network on the WHU-Hi-LongKou dataset according to an embodiment of the present invention, wherein (a) is a real landmark image of the WHU-Hi-LongKou dataset, (b) is the recognition result of 3D-ResAE on the WHU-Hi-LongKou dataset, (c) is the recognition result of 3D-MACNet on the WHU-Hi-LongKou dataset, (d) is the recognition result of DC-CapsNet on the WHU-Hi-LongKou dataset, (e) is the recognition result of 3D-DenseNet on the WHU-Hi-LongKou dataset, and (f) is the recognition result of 3DAMCN on the WHU-Hi-LongKou dataset;

[0036] Fig.12This is a comparison diagram of the recognition results of the three-dimensional autoencoder multi-scale multi-layer convolutional network of an embodiment of the present invention and the comparison network on the WHU-Hi-HongHu dataset, wherein (a) is the real landmark image of the WHU-Hi-HongHu dataset, (b) is the recognition result of 3D-ResAE on the WHU-Hi-HongHu dataset, (c) is the recognition result of 3D-MACNet on the WHU-Hi-HongHu dataset, (d) is the recognition result of DC-CapsNet on the WHU-Hi-HongHu dataset, (e) is the recognition result of 3D-DenseNet on the WHU-Hi-HongHu dataset, and (f) is the recognition result of 3DAMCN on the WHU-Hi-HongHu dataset. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] This embodiment provides a forage grass identification method based on three-dimensional autoencoder multi-scale feature extraction. Figure 1 As shown, including:

[0040] Collecting hyperspectral images of the area to be identified;

[0041] The hyperspectral image is input into a preset grass recognition model, and the grass coverage type of the area to be identified is output, wherein the grass recognition model is obtained by training with a training set, and the training set includes low-altitude UAV hyperspectral images containing several grass types. The grass recognition model is constructed based on a three-dimensional attention automatic encoding mechanism and a multi-scale multi-layer convolutional network.

[0042] Furthermore, constructing the training set includes:

[0043] Determine the collection area;

[0044] Taking hyperspectral images of forage grass in the collection area by flying a drone at low altitude;

[0045] The hyperspectral image of forage grass is spliced, cropped, pixelated and principal component analyzed to obtain a hyperspectral label image of forage grass and construct the training set.

[0046] Specifically, this embodiment constructs a data set by collecting low-altitude UAV forage hyperspectral images (LA-UFHI):

[0047] The image was taken at the Experimental Base of the Agricultural-Pastoral Interlaced Area of ​​the Grassland Research Institute of the Chinese Academy of Agricultural Sciences, Hohhot, Inner Mongolia Autonomous Region, China (4034'N, 11145'E). The area is dominated by pasture, crops and trees, covers an area of ​​371.27 hectares, and is 1044 meters above sea level. It is dominated by a temperate continental monsoon climate, with strong winds, drastic changes between cold and hot, uneven seasonal distribution of precipitation, less precipitation, concentrated in summer, an average annual precipitation of 500 mm, and an average annual temperature of about 6°C.

[0048] The size of forage grass is small, and the growing area is often accompanied by weeds and other types of forage grass. In order to accurately distinguish the varieties of forage grass, it is necessary to improve the resolution at a suitable shooting distance. On July 19, 2023, 10:25-11:16, the forage grass images were taken on site at the Experimental Base of the Agricultural and Pastoral Interlaced Area of ​​the Grassland Research Institute of the Chinese Academy of Agricultural Sciences. When the images were taken, the weather was clear, the temperature was about 29. C, and the relative humidity of the air was 38%. The acquisition equipment is the Nano-Hyperspec micro-airborne hyperspectral imager, equipped with the DJI M300 RTK rotor drone. The spectral range of the Nano-Hyperspec micro-airborne hyperspectral imager is 400nm-1000nm, the number of spatial channels is 640, and the number of spectral channels is 270.

[0049] Before shooting, check whether the drone's tripod, arms, propellers and other equipment are normal, and whether the remote control and aircraft are in the correct gear. Then install the gimbal, set the parameters, and test fly at low altitude. Finally, collect images, the flight altitude is 65m, and the flight speed is 5.27m / s. A total of two flight strips of forage images were taken. After splicing the flight strips, the calibration carpet, houses, roads and other areas were cut off to obtain a 700*700*270 size forage hyperspectral image, 700 is the width and height of the image, 270 is the number of bands, and the spatial resolution of the image is 0.065m. The image contains a total of 12 kinds of forage, namely, Chloris virgata, Bromus, Leymus chinensis, Elymus danglingii, Sanguisorba officinalis, Alfalfa, Barley awn, Short-awn Barley Grass, Pennisetum, Elymus dahliae, Sophora alopecuroides and Lotus japonicus.

[0050] This example uses Envi 5.3 to mark the hyperspectral image of grass from a low-altitude drone. 12 types of grass are marked according to the marked position. The pixels of mixed land and grass are enlarged and divided according to the area occupied by the pixel. If the grass ratio exceeds 50%, it is identified as grass, otherwise it is identified as land. This example marks the pixels of the 12 types of grass in detail, confirms the mixed area multiple times, and obtains a hyperspectral label image of grass.

[0051] Furthermore, the forage recognition model includes: a first feature extraction module, a three-dimensional feature weighting module, and a second feature extraction module, wherein the first feature extraction module is used to extract three-dimensional features of the input image; the three-dimensional feature weighting module is used to perform weighted calculation on the three-dimensional features and then perform self-encoding operation to obtain three-dimensional weighted features; the second feature extraction module is used to perform feature extraction on the three-dimensional weighted features, obtain image features and output the forage category in the input image.

[0052] Specifically, the forage recognition model constructed in this embodiment, namely the three-dimensional autoencoder multi-scale multi-layer convolutional network (3DAMCN), is a multi-dimensional, multi-layer, multi-scale feature extraction model used for the recognition and classification research of hyperspectral images. 3DAMCN calculates the feature weights of spatial X, spatial Y and spectral Z dimensions, and increases the degree of attention to important features. By deconstructing and reconstructing the features, the ability to understand features of different dimensions is enhanced. Then 2D and 3D convolution kernels of different sizes are used to extract features. Small convolution kernels are used to capture local detail features, which can better analyze detail features such as edges and textures. Large convolution kernels have a larger receptive field, can find a wider range of information, and help the network find more features that are critical to classification.

[0053] like Figure 1 As shown in the figure, 3DAMCN first uses a convolution layer with 16 1*1 convolution kernels to extract image features, but does not reduce the dimension of the data and retains the size of the input data. Then it calculates the feature weights of the spatial X, spatial Y, and spectral Z dimensions. The weighted features are compressed into low-dimensional features and then reconstructed into high-dimensional data through the decoder, so that the model has more detailed features. Finally, a multi-scale multi-layer convolutional network is used to extract image features.

[0054] Furthermore, the first feature extraction module adopts a convolutional network, the three-dimensional feature weighting module adopts a three-dimensional attention automatic encoding mechanism, and the second feature extraction module adopts a multi-scale multi-layer convolutional network.

[0055] The three-dimensional attention automatic encoding mechanism includes a channel attention unit, a spatial attention unit, and a self-encoding unit, wherein the channel attention unit is used to process the space X and the space Y in the three-dimensional feature to obtain the channel weighted feature; the spatial attention unit is used to process the spectrum Z in the three-dimensional feature to obtain the spatial weighted feature; the self-encoding unit is used to encode and then decode the channel weighted feature and the spatial weighted feature to obtain the three-dimensional weighted feature.

[0056] Specifically, Figure 2As shown in the figure, the three-dimensional attention auto-encoding mechanism (3DAM) calculates the weighted features of three dimensions in parallel, which improves the performance of the features and strengthens the complementarity of information. The channel attention mechanism is used for the space X and Y to select important feature channels and obtain channel weighted features. The spatial attention mechanism is used for the spectrum Z to highlight the key areas of the features and obtain spatial weighted features. This combination allows the network to focus on key information more accurately and increase the attention to key spatial positions. The calculated weighted features are then encoded and decoded. The data is compressed first and then reconstructed to improve the ability to capture important features.

[0057] like Figure 3 As shown in the figure, the channel attention mechanism uses global average pooling and global maximum pooling to represent the feature information of different forages, and learns the relationship between different channels and features through a multi-layer perceptron (MLP). MLP uses two fully connected layers to enable the model to learn more complex features. The two results are added and passed through the sigmoid function to obtain the channel attention weight. The channel attention mechanism dynamically adjusts the responses of different channels, recalculates the importance of the input channel, enhances the feature representation ability of the channel, and helps the network pay more attention to important features.

[0058] The calculation process of the channel attention mechanism is shown in formula (1):

[0059]

[0060] F is the input feature, σ is the sigmoid function, M c is the channel weighted feature.

[0061] like Figure 4 As shown in the figure, the spatial attention mechanism adjusts the feature weight value of the spatial dimension, so that the model pays more attention to the key areas in the input features and learns more features in the spectral dimension. The spatial attention mechanism calculates the average pooling value and the maximum pooling value of the input features. Splicing along the last dimension, the spatial weighted features are generated through the convolution layer and the sigmoid activation function. The spatial attention mechanism makes the model more stable, can adjust the attention of the input features, and reduce the influence of irrelevant information.

[0062] The calculation process of the spatial attention mechanism is shown in formula (2):

[0063]

[0064] Among them, M s is a spatially weighted feature.

[0065] like Figure 3As shown in the figure, 3DAM performs self-encoding operations on the weighted features of spatial X, spatial Y, and spectral Z dimensions. First, the weighted features are mapped to a low-dimensional space and compressed into low-dimensional data to capture spatial and spectral features at different levels and mine more details. Then, the data in the low-dimensional space is mapped back to the original dimension through decoding, the dimension of the data is restored, and the weighted features are reconstructed. The reconstructed data highlights the key information and location, and refines the detailed features. Finally, the reconstructed three-dimensional features are integrated to generate new image weighted features.

[0066] The self-encoding operation enhances the search capability for key features, reduces model cost, improves image quality, and improves data processing efficiency. Figure 5 As shown, the autoencoder transforms the input layer x into the middle hidden layer h, and then reconstructs it into the output layer y. Another expression of the data is performed without losing information. The calculation formulas are shown in formulas (3)-(5). Among them, x is the input data, s is the sigmoid function, W, W' are weights, b, b' are biases, θ is the transformation parameter, and L is the loss function.

[0067] y=fθ(x)=s(Wx+b) (3)

[0068] x'=gθ'(x)=s(W'x+b') (4)

[0069] L(x,x')=L(x,g(f(x))) (5)

[0070] Furthermore, the multi-scale multi-layer convolution network includes two-dimensional multi-scale multi-layer convolution and three-dimensional multi-scale multi-layer convolution, wherein the two-dimensional multi-scale multi-layer convolution is used to extract the features of space X and space Y in the three-dimensional weighted features; the three-dimensional multi-scale multi-layer convolution is used to extract the features of spectrum Z in the three-dimensional weighted features.

[0071] Both two-dimensional multi-scale multi-layer convolution and three-dimensional multi-scale multi-layer convolution are composed of several convolution layers with gradually smaller convolution kernels.

[0072] Specifically, Figure 6 As shown in the figure, the multi-scale multi-layer convolutional network (MMCN) arranges the convolution kernels from large to small to improve the correlation between global features and local features, strengthens the distinction between grasses with similar features such as color and texture, helps to understand the image content, and improves the ability to capture complex features. A complex feature extraction model from large scale to small scale is formed.

[0073] MMCN effectively captures key features by combining large convolution kernels first and small convolution kernels later. The kernel size of the first convolution layer is 7*7. The kernel size of the second convolution layer is 5*5. The kernel size of the third convolution layer is 3*3. The kernel size of the fourth convolution layer is 1*1. A ReLU activation function is added after each convolution layer. This structure enables the network to extract more global features at the beginning and capture a wider range of spectral correlations. The size of the convolution kernel decreases layer by layer, gradually refining the features, discovering more details, and strengthening the ability to mine features such as texture, edges, and shapes. MMCN reduces the risk of overfitting, improves the nonlinear ability of the network, finds more forage features, and improves the ability to recognize different forages.

[0074] Large-size convolution kernels have a larger receptive field, can extract global features, and consider more relevant information. At the same time, large-size convolution kernels can capture more complex and abstract features in the image, and can consider larger spatial areas, thereby capturing the overall structure of the image, providing a broader perspective and deeper insight.

[0075] Small-sized convolution kernels do not change the width and height of the feature map, maintain spatial resolution, enhance the feature representation of local areas through channel transformation, achieve cross-channel feature fusion, and thus extract more representative features. Small-sized convolution kernels have fewer parameters, reduce network complexity, increase nonlinear characteristics, and improve the expressiveness of the model.

[0076] MMCN uses 2D multi-scale multi-layer convolution to extract the features of space X and Y, and uses 3D multi-scale multi-layer convolution to extract the features of spectrum Z. 2D multi-scale multi-layer convolution focuses on the shape, texture and other features of the image. Hierarchical features in the image are extracted through continuous convolution kernels of different sizes.

[0077] like Figure 7 As shown, the calculation formula of 2D convolution is shown in formulas (6)-(8). N is the batch size, C is the number of channels, and H and W are the height and width of the image.

[0078]

[0079]

[0080] like Figure 8As shown in the figure, 3D multi-scale multi-layer convolution extracts the joint features of space and spectrum, and maps the spatial features on the spectrum. Large-size 3D convolution kernels can capture a wider range of spectral correlations, and small-size 3D convolution kernels are more focused on local changes in the spectrum, which helps to understand the spectral information more comprehensively. The multi-layer convolution structure enables each channel to focus on learning spectral features at a specific scale, forming rich and comprehensive spectral features and enhancing the ability to express spectral information.

[0081] The calculation formula of 3D convolution is shown in formulas (9)-(12):

[0082]

[0083] MMCN improves the ability to extract spatial and spectral features. While extracting spectral features through convolution kernels of different scales, it also captures certain spatial context information. Different channels will capture similar features as well as channel-specific features. MMCN improves the robustness of the model through feature overlap and complementarity. The fusion of spectrum and space can more accurately describe the details of hyperspectral images and better solve the problem of spectral mixing.

[0084] Model validation and evaluation:

[0085] (1) Evaluation indicators:

[0086] The experiment is evaluated by 6 indicators. Time is the training time. Overall accuracy (OA) means the number of correctly predicted samples divided by the total number of samples, where TP stands for True Positives, FN stands for False Negatives, FP stands for False Positives, and TN stands for True Negatives.

[0087]

[0088] Precision refers to the proportion of true results in samples predicted to be correct.

[0089]

[0090] Recall refers to the number of correct predictions among samples whose true values ​​are correct.

[0091]

[0092] F1-score is used to evaluate the performance of the model on an imbalanced dataset.

[0093]

[0094] The Kappa coefficient indicates the statistical consistency between the results and the true results, p o Indicates OA, p e is the expected value of the accuracy of random classification, N is the total number of samples, N +i is the total number of samples predicted to be of the i-th class.

[0095]

[0096] (2) Experimental results:

[0097] The WHU-Hi-LongKou and WHU-Hi-HongHu datasets were used as comparative data to evaluate the model effect. The WHU-Hi-LongKou dataset was taken in Longkou Town, Hubei Province, China from 13:49 to 14:37 on July 17, 2018. The study area includes: corn, cotton, sesame, broad-leaf soybean, narrow-leaf soybean and rice. The image size is 550×400 pixels, there are 270 bands from 400nm to 1000nm, and the spatial resolution of the hyperspectral image is about 0.463m. The WHU-Hi-HongHu dataset was collected in Yanchang City, Hubei Province, China from 16:23 to 17:37 on November 20, 2017. The experimental area is a complex agricultural landscape with multiple crops, including Chinese cabbage, Brassica spp. and Brassica spp. The image size is 940 × 475 pixels, there are 270 bands from 400nm to 1000nm, and the spatial resolution of the hyperspectral image is about 0.043m.

[0098] The experimental equipment uses Lenovo Savior Y9000P, Inter(R)Core i7-12700H, 32GB memory, and 64-bit Windows 11 operating system. The tensorflow2.10.0 deep learning framework and the python 3.9 version compiler are used. The batch size is set to 32 and 100 epochs are trained. The learning rate is set to 2e-3 and the Adam optimizer is used. The input data size is 700*700*16, and 70% of the data set is used as the training set.

[0099] Preprocessing can not only reduce the image dimension, but also remove noise and redundant data to improve image quality. Principal Component Analysis (PCA) is used as a preprocessing method. PCA finds the main change directions in the data and projects the data onto these directions to achieve data dimensionality reduction, remove redundant information and noise, and retain the information in the original data to the greatest extent. Three datasets, LA-UFHI, WHU-Hi-LongKou, and WHU-Hi-HongHu, were used to evaluate the performance of 3DAMCN, and four advanced models were compared. The experiment evaluated the effect of each model through 6 indicators.

[0100] The advanced methods compared are: 3D Residual Auto Encoder (3D-ResAE), 3D-MACNet, deep convolutional capsule neural network (DC-CapsNet), and 3D-DenseNet. 3D-ResAE combines 3D convolution and residual, uses deconvolution to reconstruct the original features, solves the gradient vanishing problem, and the model trains data in a semi-supervised manner. 3D-MACNet is retrieved by convolution components, and then uses a three-dimensional multi-scale asymmetric convolutional network module to produce classification results through a fully connected layer (FC). DC-CapsNet uses a 3D extended convolution capsule network to enhance spectral spatial robustness; a decoder network composed of deconvolution layers is used to obtain more spatial relationships and improve recognition accuracy. 3D-DenseNet enhances feature transmission, supports feature reuse, learns spectral spatial features, and improves the recognition accuracy of complex objects.

[0101] (1) Low-altitude UAV forage hyperspectral imagery (LA-UFHI) dataset;

[0102] LA-UFHI consists of 12 types of grass, land and roads, and all land and roads are set as background. The experimental results are shown in Table 1. The OA of 3DAMCN is 99.8%, F1 is 99.2%, and kappa is 99.8%. Other evaluation indicators also achieved very good results. The training process of accuracy and loss is shown in Fig. 9 As shown in the figure, 3DAMCN shows very good robustness and is very stable during training, with only some fluctuations in the early stages of training.

[0103] Table 1

[0104]

[0105] The experimental results show that 3DAMCN has the highest recognition accuracy, the shortest training time, and the best model stability, and can be used to conduct fast and high-precision recognition research on UAV forage hyperspectral images. Among the compared methods, 3D-MACNet has the best experimental results, with an OA of 97.4%, proving that the multi-scale convolution structure can indeed improve recognition efficiency. The recognition accuracy of the other three models all exceeded 90%, and all achieved good experimental results, proving that deep networks can indeed improve the ability to understand images.

[0106] Fig.10 The following are the classification results of five models, among which (a) is the real landmark image of LA-UFHI dataset, (b) is the recognition result of 3D-ResAE on LA-UFHI dataset, (c) is the recognition result of 3D-MACNet on LA-UFHI dataset, (d) is the recognition result of DC-CapsNet on LA-UFHI dataset, (e) is the recognition result of 3D-DenseNet on LA-UFHI dataset, and (f) is the recognition result of 3DAMCN on LA-UFHI dataset. 3DAMCN is closest to the ground truth. The other four methods have obvious recognition error areas in the figure, indicating that some features were missed during the training process, the ability to find details was insufficient, and the unique features of forage grass with similar appearance and texture were not found. However, the four models achieved good results in the recognition of 12 kinds of forage grass, and the recognition results were better than traditional methods and manual recognition, proving that convolutional neural networks can indeed effectively improve the recognition accuracy of forage grass. Experiments have shown that 3DAMCN has a good ability to handle complex features. By combining spatial and spectral dimensions, it has found a large number of key features, strengthened the ability to mine the unique features of each type of forage, found the unique features to distinguish forages with similar shapes and colors, and improved recognition accuracy.

[0107] (2) WHU-Hi-LongKou dataset;

[0108] The complexity of the dataset will also affect the performance of the model. As a dataset with relatively simple types of objects, the LongKou dataset will enhance the model's ability to analyze objects to a certain extent. The experimental results of the LongKou dataset are shown in Table 2. The OA of 3DAMCN is 99.9% and the F1 is 99.8%, showing good recognition performance for all 9 objects.

[0109] Table 2

[0110]

[0111]

[0112] Fig.11The following are the classification results of the five models, where (a) is the real landmark image of the WHU-Hi-LongKou dataset, (b) is the recognition result of 3D-ResAE on the WHU-Hi-LongKou dataset, (c) is the recognition result of 3D-MACNet on the WHU-Hi-LongKou dataset, (d) is the recognition result of DC-CapsNet on the WHU-Hi-LongKou dataset, (e) is the recognition result of 3D-DenseNet on the WHU-Hi-LongKou dataset, and (f) is the recognition result of 3DAMCN on the WHU-Hi-LongKou dataset. As can be seen from the figure, 3DAMCN has the least error area and almost completely restores the classification of real objects. The experimental results of 3D-MACNet are very close to those of 3DAMCN, with only a few pixels misclassified. 3D-ResAE and DC-CapsNet misclassified some areas, resulting in a decrease in F1. However, the number of correctly identified samples is large, so OA has not decreased significantly. With the advantages of deep networks, the experimental results of 3D-DenseNet on the LongKou dataset are also close to the real results, proving that deepening the number of network layers plays an important role in the model.

[0113] (3)WHU-Hi-HongHu dataset;

[0114] The HongHu dataset has many types of objects and is a complex UAV hyperspectral dataset, which has high requirements for the model. The experimental results are shown in Table 3. 3DAMCN also performs well on the HongHu dataset, with an OA of 99.9% and an F1 of 99.2%. The experiment proves that 3DAMCN can also perform well on complex hyperspectral datasets. The experimental results of the other four methods showed a significant decline compared with the previous two datasets. The OA of 3D-ResAE, DC-CapsNet and 3D-DenseNet remained at 90%, and 3D-MACNet remained at 95%.

[0115] Table 3

[0116]

[0117] The classification results of the HongHu dataset are as follows Fig.12As shown in the figure, (a) is the real landmark image of the WHU-Hi-HongHu dataset, (b) is the recognition result of 3D-ResAE on the WHU-Hi-HongHu dataset, (c) is the recognition result of 3D-MACNet on the WHU-Hi-HongHu dataset, (d) is the recognition result of DC-CapsNet on the WHU-Hi-HongHu dataset, (e) is the recognition result of 3D-DenseNet on the WHU-Hi-HongHu dataset, and (f) is the recognition result of 3DAMCN on the WHU-Hi-HongHu dataset. 3DAMCN shows very good results, but some pixels are misclassified. 3D-ResAE has classification errors in multiple areas. The misclassified areas of 3D-MACNet are concentrated on crops with fewer samples, and the results are better in areas with a large number of samples. Noise appears in DC-CapsNet, and obvious classification errors occur in some areas. 3D-DenseNet misclassifies some adjacent plants. Experiments have proved that deep learning is suitable for the detection research of hyperspectral images. It can capture the key features of the image and effectively improve the classification accuracy.

[0118] (4) Comparison results of various parts of the model;

[0119] In order to verify the effect of each part of the model, this embodiment conducted a comparative experiment, combining 2D multi-size multi-layer convolution (2D), 3D multi-size multi-layer convolution (3D), three-dimensional attention mechanism (TAM), and autoencoder (AE) respectively. The experimental results are shown in Table 4. The experimental results show that adding 3D attention mechanism and autoencoder to the network can effectively improve the ability to analyze and understand features, and strengthen the performance of features. 2D multi-size multi-layer convolution + 3D multi-size multi-layer convolution effectively strengthens the connection between spectrum and space, improves the utilization rate of spectral data, integrates features at different levels, strengthens the correlation of data, and improves the overall performance of the model.

[0120] Table 4

[0121]

[0122] The experiment compared four advanced methods. All four models improved the connection between spectrum and space, deepened the number of network layers, and strengthened the ability to discover features. 3D-ResAE deepened the number of network layers, solved the gradient vanishing problem through residual blocks, and strengthened the feature extraction ability. However, 3D-ResAE's ability to cope with complex data sets needs to be improved. On the LongKou data set, the OA is 95.4%, but on the HongHu data set with more types of features, the OA is 91.8%, and on the LA-UFHI data set with complex grassland terrain, the OA is 92.4%. It proves that 3D-ResAE's ability to analyze the features of complex data sets needs to be strengthened.

[0123] The accuracy of 3DAMCN on LA-UFHI is 7.4%, 2.4%, 5.4%, and 3.3% higher than the other four models, respectively. The accuracy on LongKou is 4.5%, 0.7%, 2.3%, and 1.8% higher than the other four models, respectively. The accuracy on HongHu is 8.1%, 4.9%, 9.8%, and 7.7% higher than the other four models, respectively. 3DAMCN has achieved excellent recognition results on all three datasets. For complex datasets such as HongHu, the model performance is not reduced due to complex images. In addition, there are no large areas of recognition errors on the three datasets. Only a few test samples have recognition errors, which reflects the good correspondence between the spatial and spectral features of the model.

[0124] 3DAMCN uses the attention mechanism for all three dimensions, which improves the feature representation ability at the beginning of the network, strengthens the correlation and complementarity of space and spectrum, and strengthens the connection between features. It also encodes and decodes the weighted features of the three dimensions, classifies and reorganizes the features, and further highlights the key features and their locations. The multi-scale multi-layer convolutional network uses a combination of 2D convolution and 3D convolution to strengthen the connection between space and spectrum. Large-size convolution kernels extract global features, and gradually reduce the kernel size in subsequent convolutions, so that the network takes into account both global and local features, strengthens the description of details such as edges and textures, and improves the distinction between different objects.

[0125] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A forage grass identification method based on three-dimensional autoencoder multi-scale feature extraction, characterized in that: include: Collecting hyperspectral images of the area to be identified; The hyperspectral image is input into a preset grass recognition model, and the grass coverage type of the area to be identified is output, wherein the grass recognition model is obtained by training with a training set, and the training set includes low-altitude UAV hyperspectral images containing several grass types. The grass recognition model is constructed based on a three-dimensional attention automatic encoding mechanism and a multi-scale multi-layer convolutional network.

2. The forage grass identification method based on three-dimensional autoencoder multi-scale feature extraction according to claim 1 is characterized in that: Constructing the training set includes: Determine the collection area; Taking hyperspectral images of forage grass in the collection area by flying a drone at low altitude; The hyperspectral image of forage grass is spliced, cropped, pixelated and principal component analyzed to obtain a hyperspectral label image of forage grass and construct the training set.

3. The forage grass identification method based on three-dimensional autoencoder multi-scale feature extraction according to claim 1 is characterized in that: The forage recognition model includes: a first feature extraction module, a three-dimensional feature weighting module, and a second feature extraction module, wherein the first feature extraction module is used to extract the three-dimensional features of the input image; the three-dimensional feature weighting module is used to perform weighted calculation on the three-dimensional features and then perform self-encoding operation to obtain three-dimensional weighted features; the second feature extraction module is used to perform feature extraction on the three-dimensional weighted features, obtain image features and output the forage category in the input image.

4. The forage grass identification method based on three-dimensional autoencoder multi-scale feature extraction according to claim 3 is characterized in that: The first feature extraction module adopts a convolutional network, the three-dimensional feature weighting module adopts a three-dimensional attention automatic encoding mechanism, and the second feature extraction module adopts a multi-scale multi-layer convolutional network.

5. The forage grass identification method based on three-dimensional autoencoder multi-scale feature extraction according to claim 4 is characterized in that: The three-dimensional attention automatic encoding mechanism includes a channel attention unit, a spatial attention unit, and a self-encoding unit, wherein the channel attention unit is used to process the space X and space Y in the three-dimensional feature to obtain the channel weighted feature; the spatial attention unit is used to process the spectrum Z in the three-dimensional feature to obtain the spatial weighted feature; the self-encoding unit is used to encode and then decode the channel weighted feature and the spatial weighted feature to obtain the three-dimensional weighted feature.

6. The forage grass identification method based on three-dimensional autoencoder multi-scale feature extraction according to claim 3 is characterized in that: The multi-scale multi-layer convolution network includes two-dimensional multi-scale multi-layer convolution and three-dimensional multi-scale multi-layer convolution, wherein the two-dimensional multi-scale multi-layer convolution is used to extract the features of space X and space Y in the three-dimensional weighted features; the three-dimensional multi-scale multi-layer convolution is used to extract the features of spectrum Z in the three-dimensional weighted features.

7. The forage grass identification method based on three-dimensional autoencoder multi-scale feature extraction according to claim 6 is characterized in that: The two-dimensional multi-scale multi-layer convolution and the three-dimensional multi-scale multi-layer convolution are both composed of a number of convolution layers with gradually shrinking convolution kernels.

Images