Unmanned aerial vehicle hyperspectral remote sensing image feature extraction and recognition method and system
Through the MBS-TD3DAN method, the forage hyperspectral data set was constructed and the Multi Band Selector and TransDense3D-AttnNet model was used to solve the problems of low grass recognition efficiency and insufficient accuracy, and realize high-precision forage recognition and large-area recognition.
Patent Information
- Application Number
- CN202410826561.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-25
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing forage recognition methods rely on manual recognition efficiency and high cost, high hyperspectral image recognition is difficult, machine learning technology has decreased accuracy in big data scenarios, and deep learning methods are difficult to improve the recognition accuracy of different types of data.
Using the feature extraction and recognition method of drone hyperspectral remote sensing image based on MBS-TD3DAN, the forage hyperspectral data set was constructed, and the dimension reduction and denoising were reduced by Multi Band Selector, and combined with the TransDense3D-AttnNet model was extracted through object, space and spectral dimensions. Features, using self-attention mechanism and dense convolutional layers to achieve high-precision recognition.
It improves the accuracy and efficiency of grass recognition, reduces the recognition cost, solves the problems of low feature extraction accuracy and high computational complexity of hyperspectral image, and realizes accurate identification of large-area grass.
Smart Images

Figure CN118918497A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to but is not limited to the technical field of hyperspectral remote sensing image processing, and in particular relates to a method and system for extracting and recognizing features of unmanned aerial vehicle hyperspectral remote sensing images based on MBS-TD3DAN. Background Art
[0002] As an important branch of remote sensing technology, hyperspectral remote sensing technology is a multi-dimensional information acquisition technology that combines spectral features and imaging technology. Hyperspectral remote sensing images not only contain two-dimensional object information and spatial information, but also integrate spectral information. The image is finely segmented through spectral reconstruction to form continuous and high-resolution spectral reflectance data, so that each pixel corresponds to a complete spectral curve. The traditional ground object detection method is cumbersome, time-consuming and labor-intensive, while the drone-borne hyperspectral remote sensing technology can achieve real-time, non-contact, rapid and non-destructive detection. It can detect areas that traditional remote sensing technology cannot detect, obtain ground object spectra, and provide data support for distinguishing surface ground object types and evaluating component content; it can also obtain data in multiple phases, and improve the accuracy of ground object recognition through the calculation of multiple elements. This "map-spectrum integration" feature makes drone hyperspectral remote sensing technology have high accuracy in distinguishing surface objects, and it has been used in environmental monitoring, precision agriculture, urban planning and other aspects.
[0003] UAV hyperspectral technology has been widely used in agricultural development, especially in the identification and monitoring of grassland forage. Forage is an important part of terrestrial ecosystems. Forage has strong regeneration ability and is rich in various trace elements and vitamins. With the development of grassland ecosystems and animal husbandry, the demand for forage is increasing, and there are problems such as shortage of forage and poor quality. Forage species identification plays an important role in the cultivation of high-quality varieties and the development of forage. By identifying forage, we can understand the characteristics, growth conditions, nutritional value and other information of different types of forage. In this way, targeted improvements can be made, the forage planting structure can be optimized, and the yield and quality of forage can be improved. At present, most forage identification methods still rely on manual identification, which is not only inefficient but also consumes labor costs. The growth area and characteristics of forage determine the difficulty of forage identification. Many varieties of forage have similar appearance, similar color, and similar volume. In addition, multiple types of forage grow in the same area, and they are mixed with weeds, so it is difficult to identify forage. Most existing studies use ordinary images to study forage grasses. The images are relatively clear, but the shooting distance is short and cannot include multiple forage grass varieties.
[0004] Some studies have combined drones with two-dimensional images. Although this has solved the problem of few grass species in the images, two-dimensional images cannot analyze the characteristics of the spectral dimension, and it is difficult to distinguish between grasses with similar textures and colors. Drone hyperspectral images make up for the shortcomings of two-dimensional images. They can not only extract spatial features, but also spectral features, enhance the ability to obtain grass characteristics, and solve the problem of not being able to collect data at a suitable distance. They have effectively improved the image resolution and included a variety of grasses. However, processing the huge amount of data in hyperspectral images has also become a new problem. In addition, the spectral characteristics of different substances are affected by many factors, such as light, temperature, humidity, etc., which will also increase the difficulty of image recognition and lead to reduced accuracy. In existing research on the recognition of hyperspectral images of grass, the main strategies adopted by people are:
[0005] Use public datasets as research objects, such as Indian Pines, Pavia University, Houston2018, WHU-Hi, etc. These datasets have complete and clear labeled data, which greatly facilitates the research of hyperspectral data. However, most datasets are used many times, and most datasets do not meet the required resolution level. Even if some datasets have high resolution, the shooting distance is far and cannot meet actual needs. In addition to grass, there are other objects in this type of image, such as buildings, streets, etc. This type of research only achieves object recognition, but has little significance for the research and development of forage.
[0006] Study on hyperspectral images of near-ground grass. Some studies have taken hyperspectral images of near-ground grass, shortened the shooting distance, improved the resolution and clarity, and collected detailed feature information. However, due to the short shooting distance, it is impossible to include multiple types of grass in one image, and large-scale identification and analysis cannot be carried out. Moreover, there are no corresponding marks for such images, which increases the difficulty of identification.
[0007] Satellite remote sensing images. Satellite remote sensing images are used to identify and classify objects, but the distance is too far. Even if the clarity and resolution are high, it is still impossible to accurately distinguish the types of objects, especially plants. Although large-scale research can be carried out, data labeling and object classification are difficult to solve, which increases the difficulty of object identification and makes it difficult to conduct in-depth research on forage.
[0008] Main technical means. The current research methods are mainly based on machine learning and deep learning. Machine learning technology is mature and the experimental speed is fast, but it is not suitable for scenarios with large amounts of data. When the amount of data is too large, the accuracy drops significantly and the speed also drops significantly. Deep learning is mainly based on various models derived from convolutional neural networks, and the attention mechanism has taken deep learning to a new level. The combination of convolutional neural networks and attention can effectively improve model performance. However, how to improve the recognition accuracy for different types of data is still the focus of research.
[0009] In view of the above analysis, the technical problems that need to be solved urgently in the prior art are:
[0010] (1) At present, most forage identification methods still rely on manual identification, which is not only inefficient but also consumes labor costs.
[0011] (2) In the existing research on the recognition of hyperspectral images of forage, the recognition difficulty is relatively large and cannot meet the actual needs, making it difficult to conduct in-depth research on forage.
[0012] (3) Technical methods based on machine learning are not suitable for scenarios with large amounts of data; how deep learning-based methods can improve recognition accuracy for different types of data remains to be studied. Summary of the invention
[0013] In view of the problems existing in the prior art, the present invention provides a method and system for extracting and recognizing features of unmanned aerial vehicle hyperspectral remote sensing images based on MBS-TD3DAN.
[0014] The present invention is implemented as follows: a method for extracting and identifying features of unmanned aerial vehicle hyperspectral remote sensing images based on MBS-TD3DAN, the method specifically comprising:
[0015] S1: Constructing a UAV forage hyperspectral remote sensing dataset;
[0016] S2: Use Multi Band Selector (MBS) preprocessing method to reduce the image dimension, remove noise, and extract feature bands;
[0017] S3: The TransDense3D-AttnNet (TD3DAN) model is used to mark data weights, extract data features through object, space, and spectrum dimensions, use the self-attention mechanism to transform, map, filter, and perform other operations on the data, and use dense convolutional layers to enhance feature transfer, ultimately achieving high-precision recognition.
[0018] Furthermore, S1 uses three datasets, including the UAV forage hyperspectral remote sensing dataset (UFHI) taken on site, and the UAV hyperspectral datasets WHU-Hi-LongKou and WHU-Hi-HongHu. The UFHI image was taken at the Experimental Base of the Agricultural and Pastoral Interlaced Area of the Grassland Research Institute of the Chinese Academy of Agricultural Sciences, Hohhot, Inner Mongolia Autonomous Region, China (4034'N, 11145'E). The acquisition equipment is Nano-Hyperspec micro airborne hyperspectral imager, equipped with DJI M300RTK rotor drone, using Envi 5.3 Marking UFHI. According to the marked position during shooting, select the area of interest for each type of forage grass for marking. Zoom in on the pixels containing land and forage grass, and divide them according to the area occupied by the pixels. If the proportion of forage grass exceeds 50%, it is identified as forage grass, otherwise it is identified as land. The growth area of forage grass cannot be fixed, so the marked area is not a regular shape. It needs to be marked according to the growth range of forage grass. In this study, land and roads are identified as background areas, and detailed pixel mark is performed for 12 types of forage grass. The intersection area is confirmed multiple times. After multiple markings, the hyperspectral remote sensing label image of forage grass is obtained.
[0019] Further, the S2, MBS calculates the mean of each sample pixel point to obtain the covariance matrix of the image, determines the contribution rate by calculating the ratio of their sum to the sum of all eigenvalues, retains the band with large contribution rate, obtains the characteristic band data with rich information, and saves the band image, which specifically includes:
[0020] (1) Calculate the mean of band data b, where l is the total number of pixels of length b and μ is the mean:
[0021]
[0022] (2) Calculate the covariance matrix σ of b 2 , select the largest eigenvalue S:
[0023]
[0024] (3) Initialize a zero matrix, calculate the eigenvalues and eigenvectors of the matrix, sort the eigenvalues and select the eigenindex, and calculate the contribution rate G of the eigenvalues, where λ i is the i-th eigenvalue, and r is the number of bands to be retained:
[0025]
[0026] Furthermore, the S3, TD3DAN is a multi-channel feature extraction and recognition model proposed in the present invention, including Dense3DBC Block, CBAM module and multi-head self-attention mechanism (MSAM):
[0027] (1)Dense3DBC Block
[0028] Dense3DBC adopts the BN+ReLU+1x1x1Conv+BN+ReLU+3x3x3Conv structure, and θ<1, θ∈(0,1] represents the compression coefficient. The BN+ReLU+1x1Conv structure is added on the basis of Dense Block. Dense3DBC adopts dense connection and reduces the number of denseblocks.
[0029] (2)Convolutional BlockAttention Module(CBAM)
[0030] CBAM calculates the weight value of the feature through the channel and spatial attention mechanism, and dynamically adjusts the weight of the feature map. The channel attention mechanism performs MaxPool and AvgPool operations on the feature F to obtain features of two different dimensions, which are sent to the Multi-Layer Perceptron (MLP) to learn the dependency relationship. Finally, the features output by the MLP are added, and the weights and features are multiplied to obtain the weighted feature Mc of the channel; the spatial attention mechanism performs MaxPool and AvgPool operations on the feature F` output by the channel attention, stacks the two pooled feature maps, and then reduces them to a single channel feature map H×W×1. Finally, the input feature map and the weight are multiplied to obtain the weight Ms of the spatial dimension;
[0031] (3) Multi-head self-attention mechanism (MSAM)
[0032] MSAM performs linear transformation, h is the number of heads, and obtains query ((Q)), key ((K)) and value ((V)) through linear transformation, where W Q ,W K ,W V is the weight matrix:
[0033]
[0034] Then, the scaled dot product attention is calculated for each head, where is the scaling factor:
[0035]
[0036] Then concatenate the outputs of all the heads together, (W i Q ,W i K ,W i V ) is the linear transformation matrix of the ith head, W Ois the output weight matrix:
[0037] MultiHead(Q,K,V)=Concat(head1,....,head h )W o (10)
[0038] head i =Attention(QW i Q ,KW i K ,VW i V ) (11)
[0039] Finally, after the feedforward mechanism MLP, W1, W2 are weight matrices, b1, b2 are biases, x is input data, and f is the activation function:
[0040] MPL=W2f(W1x+b1)+b2 (12)
[0041] Another object of the present invention is to provide a UAV hyperspectral remote sensing image feature extraction and recognition system based on MBS-TD3DAN, the system comprising:
[0042] The UAV forage hyperspectral remote sensing dataset building module is used to collect and process three datasets including UFHI, WHU-Hi-LongKou and WHU-Hi-HongHu;
[0043] Multi-band selector (MBS) preprocessing module, used to reduce the dimension, denoise and extract characteristic bands of images;
[0044] The TransDense3D-AttnNet (TD3DAN) model module is used to extract data features through three dimensions: object, space, and spectrum, and use the self-attention mechanism to transform, map, filter, and other operations on the data to achieve high-precision recognition.
[0045] Furthermore, the UAV forage hyperspectral remote sensing dataset construction module includes:
[0046] The data acquisition equipment is configured as Nano-Hyperspec micro airborne hyperspectral imager, paired with DJI M300RTK rotary wing drone;
[0047] Data labeling software, Envi 5.3 was used to label the UFHI dataset, and the areas of interest for each forage were marked and divided in detail.
[0048] Further, the multi-band selector (MBS) pre-processing module includes:
[0049] The calculation unit is used to calculate the mean and covariance matrix of each sample pixel and select the band with large contribution rate;
[0050] The data storage unit is used to store characteristic band data and its corresponding image.
[0051] Furthermore, the TransDense3D-AttnNet (TD3DAN) model module includes:
[0052] Dense3DBC block, using BN+ReLU+1x1x1Conv+BN+ReLU+3x3x3Conv structure, densely connected feature layers;
[0053] Convolutional Block Attention Module (CBAM), which is used to dynamically adjust the weights of feature maps through channel and spatial attention mechanisms;
[0054] Multi-head self-attention mechanism (MSAM), which is used to linearly transform features and scale dot product attention calculations;
[0055] Feed-forward mechanism, using multi-layer perceptron (MLP) to further process and activate the output features.
[0056] These system claims describe the complete system architecture for implementing feature extraction and recognition of hyperspectral remote sensing images, and clarify the specific functions and composition of each module.
[0057] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:
[0058] First, the present invention constructs a UAV forage hyperspectral remote sensing data set, which provides data support for related research on forage. It solves the problem that there is no high-resolution hyperspectral remote sensing data dedicated to forage at present, and provides help for efficient identification, high-quality seed selection, and cultivation and improvement of forage.
[0059] The present invention proposes a new data preprocessing method MBS. MBS determines the contribution rate through the covariance matrix of the image, improves the significance of important bands, retains bands with large contribution rates, improves the data preprocessing effect, and reduces the data capacity.
[0060] The present invention proposes a new deep learning model TD3DAN. TD3DAN finds important features by analyzing data in spectral and spatial dimensions, and sorts features according to weight values; it captures long-distance dependencies through a self-attention mechanism and integrates global data; it reduces model parameters through dense connections, enhances feature transfer and reuse, improves network learning capabilities, and improves recognition accuracy.
[0061] Second, the technical solution of the present invention solves a technical problem that people have been eager to solve but have never succeeded in solving: the rapid and efficient identification and monitoring of grassland forage has always been one of the important factors restricting the development of forage. If accurate identification cannot be performed, the quality of forage cannot be determined, and thus targeted research cannot be conducted, and forage varieties cannot be cultivated and improved according to local soil and climate environments, which limits the development of high-quality forage varieties. The present invention improves the accuracy of forage identification, saves identification costs and identification time, can accurately identify large areas of forage, and solves the problem of laborious manual identification and poor identification effect.
[0062] Third, the present invention solves the problems of low feature extraction accuracy, high computational complexity and insufficient recognition accuracy in the feature extraction and recognition process of UAV hyperspectral remote sensing images in the prior art. Traditional methods are often susceptible to noise interference when processing high-dimensional hyperspectral data, resulting in poor feature extraction effects. At the same time, there is a lack of effective models to integrate multi-dimensional features, resulting in low recognition accuracy. The present invention introduces the MBS-TD3DAN method and uses a multi-band selector (MBS) for dimensionality reduction and denoising, which significantly improves the accuracy and computational efficiency of feature extraction.
[0063] The significant technical progress of the present invention is that the Multi Band Selector (MBS) preprocessing method is adopted to select the band with large contribution rate by calculating the covariance matrix and eigenvalue, remove redundant information and noise, and retain the characteristic band with rich information. This dimensionality reduction and denoising processing method not only reduces the dimension of data and reduces the computational complexity, but also significantly improves the accuracy of feature extraction, laying a solid foundation for subsequent feature recognition.
[0064] In addition, the present invention proposes the TransDense3D-AttnNet (TD3DAN) model, which extracts data features through three dimensions: object, space, and spectrum, and combines the self-attention mechanism to transform, map, and filter the data, further enhancing the effect of feature extraction. The TD3DAN model uses dense convolutional layers (Dense Block) to strengthen the transmission of features, and dynamically adjusts the weights of feature maps through the convolutional block attention module (CBAM), ensuring the high accuracy and robustness of the model when processing hyperspectral data.
[0065] The present invention has also verified its effectiveness in the feature extraction and recognition of UAV hyperspectral remote sensing images of forage through practical applications. Using the hyperspectral data set taken on the spot, after MBS preprocessing and TD3DAN model recognition processing, a significant recognition effect was achieved. Compared with traditional methods, the present invention has significantly improved feature extraction accuracy, computational efficiency and recognition accuracy, has broad application prospects and practical value, and provides new ideas and methods for the development of UAV hyperspectral remote sensing technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] Figure 1 It is a flow chart of a method for extracting and identifying features of UAV hyperspectral remote sensing images based on MBS-TD3DAN provided by an embodiment of the present invention;
[0067] Figure 2 This is the research area of the UAV forage hyperspectral remote sensing image provided by the embodiment of the present invention;
[0068] Figure 3 The hyperspectral remote sensing image of forage provided by the embodiment of the present invention;
[0069] Figure 4 The hyperspectral remote sensing label image of forage provided by the embodiment of the present invention;
[0070] Figure 5 The WHU-Hi-LongKou dataset provided by the embodiment of the present invention (a) image data; (b) real landmark image;
[0071] Figure 6 The WHU-Hi-HongHu dataset provided by the embodiment of the present invention (a) image data; (b) real landmark image;
[0072] Figure 7 is a TD3DAN model diagram provided by an embodiment of the present invention;
[0073] Figure 8 It is a structural diagram of Dense3DBC provided by an embodiment of the present invention;
[0074] Fig. 9 is a schematic diagram of a channel attention mechanism provided by an embodiment of the present invention;
[0075] Fig.10 is a schematic diagram of a spatial attention mechanism provided by an embodiment of the present invention;
[0076] Fig.11 is a schematic diagram of a Transformer provided by an embodiment of the present invention;
[0077] Fig.12 It is a module diagram of a UAV hyperspectral remote sensing image feature extraction and recognition system based on MBS-TD3DAN provided in an embodiment of the present invention;
[0078] Fig.13 The preprocessing results of the UAV grass hyperspectral remote sensing image provided by the embodiment of the present invention are (a) MBS; (b) PCA; (c) NMF; (d) ICA;
[0079] Fig.14The WHU-Hi-LongKou preprocessing results provided by the embodiment of the present invention are (a) MBS; (b) PCA; (c) NMF; (d) ICA;
[0080] Fig.15 The preprocessing results provided by the embodiment of the present invention are (a) MBS; (b) PCA; (c) NMF; (d) ICA;
[0081] Fig.16 The forage experiment results provided by the embodiments of the present invention are (a) Ground-truth; (b) 3D-ResAE; (c) 3D-MACNet; (d) DC-CapsNet; (e) 3D-DenseNet; (f) TD3DAN;
[0082] Fig.17 The WHU-Hi-LongKou experimental results provided by the embodiments of the present invention are (a) Ground-truth; (b) 3D-ResAE; (c) 3D-MACNet; (d) DC-CapsNet; (e) 3D-DenseNet; (f) TD3DAN;
[0083] Fig.18 These are the WHU-Hi-HongHu experimental results provided by the embodiments of the present invention: (a) Ground-truth; (b) 3D-ResAE; (c) 3D-MACNet; (d) DC-CapsNet; (e) 3D-DenseNet; (f) TD3DAN. DETAILED DESCRIPTION
[0084] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0085] Embodiment 1
[0086] In an experimental base in an agricultural-pastoral transition area, a drone was used to extract and identify features from hyperspectral remote sensing images of forage grass. The goal was to verify the application effect of the MBS-TD3DAN method in actual hyperspectral data.
[0087] 1. Data collection and processing:
[0088] The Nano-Hyperspec micro airborne hyperspectral imager was used with a DJI M300 RTK rotorcraft drone to collect hyperspectral images of forage grass at the experimental base and obtain the UAV forage hyperspectral remote sensing dataset (UFHI).
[0089] Envi 5.3 was used to mark the UFHI dataset. The areas of interest for each type of forage were selected and marked according to the marked positions when shooting, and divided into two categories: forage and land.
[0090] 2. Data preprocessing:
[0091] The Multi Band Selector (MBS) preprocessing method is used to reduce the dimension and denoise the hyperspectral image, calculate the mean and covariance matrix of each sample pixel, and select the characteristic bands with rich information by calculating the contribution rate of the eigenvalue.
[0092] 3. Feature extraction and recognition:
[0093] The TransDense3D-AttnNet (TD3DAN) model is used to extract features from the preprocessed data. The TD3DAN model extracts data features from three dimensions: object, space, and spectrum, and combines the self-attention mechanism and dense convolutional layers to enhance feature transfer.
[0094] After multiple training and iterations, the TD3DAN model was used to perform high-precision recognition of forage grasses in the UFHI dataset and evaluate the recognition effect.
[0095] The experimental results show that the accuracy of feature extraction and recognition of hyperspectral remote sensing images of forage grass is significantly improved by using the MBS-TD3DAN method, and the recognition results are stable and reliable, which effectively verifies the superiority of the method of the present invention.
[0096] Embodiment 2
[0097] In the experimental area of a lake, a drone was used to collect hyperspectral data sets for feature extraction and recognition. The goal was to test the application effect of the MBS-TD3DAN method in different environments.
[0098] 1. Data collection and processing:
[0099] Nano-Hyperspec micro airborne hyperspectral imager was used with DJI M300 RTK rotary wing drone to collect hyperspectral image data of Yangtze University Experimental Area (WHU-Hi-LongKou) and Honghu Lake (WHU-Hi-HongHu).
[0100] The collected data is marked, and the areas of interest are selected for detailed pixel marking and classified into different types of land objects.
[0101] 2. Data preprocessing:
[0102] The Multi Band Selector (MBS) preprocessing method was used to reduce the dimension and denoise the datasets of Yangtze University and Honghu Lake, calculate the mean and covariance matrix of the characteristic bands, and select the characteristic bands with large contribution rates.
[0103] 3. Feature extraction and recognition:
[0104] The TransDense3D-AttnNet (TD3DAN) model is used to extract features from the preprocessed data. By extracting features in three dimensions, namely, object, space, and spectrum, and combining the self-attention mechanism with dense convolutional layers, feature transfer and recognition accuracy are enhanced.
[0105] After multiple training and iterations, the TD3DAN model is used to recognize hyperspectral data in different environments and its recognition effect is evaluated.
[0106] The experimental results show that the MBS-TD3DAN method has high recognition accuracy and stability in different environments. Whether in the Yangtze University experimental area or the Red Lake area, the recognition results are significantly better than the traditional methods, further verifying the wide applicability and superior performance of the invention.
[0107] These two embodiments demonstrate the specific application process and significant effects of the present invention under different experimental environments, and prove the effectiveness and superiority of the MBS-TD3DAN method in feature extraction and recognition of UAV hyperspectral remote sensing images.
[0108] like Figure 1 As shown, an embodiment of the present invention provides a method for extracting and identifying features of a UAV hyperspectral remote sensing image based on MBS-TD3DAN, and the method specifically includes:
[0109] S1: Constructing a UAV forage hyperspectral remote sensing dataset;
[0110] S2: Use Multi Band Selector (MBS) preprocessing method to reduce the image dimension, remove noise, and extract feature bands;
[0111] S3: The TransDense3D-AttnNet (TD3DAN) model is used to mark data weights, extract data features through object, space, and spectrum dimensions, use the self-attention mechanism to transform, map, filter, and perform other operations on the data, and use dense convolutional layers to enhance feature transfer, ultimately achieving high-precision recognition.
[0112] The S1 uses three datasets in total, including a drone forage hyperspectral remote sensing dataset taken on the spot, and drone hyperspectral datasets WHU-Hi-LongKou and WHU-Hi-HongHu.
[0113] 1. UAV Grass Hyperspectral Remote Sensing Image (UFHI)
[0114] (1.1) Study area
[0115] like Figure 2 As shown, the image was taken at the Experimental Base of the Agricultural and Pastoral Interlaced Area, Institute of Grassland, Chinese Academy of Agricultural Sciences, Hohhot, Inner Mongolia Autonomous Region, China (4034'N, 11145'E, Figure 2 The area is dominated by pasture, crops and forests, covers an area of 371.27 hectares, and has an altitude of 1044m. It is dominated by a temperate continental monsoon climate, with strong winds, drastic changes in cold and hot weather, uneven seasonal distribution of precipitation, less precipitation, concentrated in summer, with an average annual precipitation of 500mm and an average annual temperature of about 6℃.
[0116] (2.2) Data collection
[0117] The size of the grass is small, and the growing area is often accompanied by weeds and other types of grass. In order to accurately distinguish the grass varieties, it is necessary to improve the resolution at a suitable shooting distance. UFHI was photographed on site at CAAS from 10:25 to 11:16 on July 19, 2023. When the image was taken, the weather was clear, the temperature was about 29. C, and the relative humidity of the air was 38%. The acquisition equipment was the Nano-Hyperspec micro-airborne hyperspectral imager, equipped with the DJI M300 RTK rotor drone. The spectral range of the Nano-Hyperspec micro-airborne hyperspectral imager is 400nm-1000nm, the number of spatial channels is 640, the number of spectral channels is 270, and the lens focal length is 8mm. The specific parameters are shown in Table 1. The DJI M300 RTK rotor drone uses a customized three-axis gimbal and adopts six positioning and obstacle avoidance technologies. The wind resistance reaches level 7 and the flight time is 55min.
[0118] First, select the take-off and landing locations according to the altitude of the shooting area. Then, design the route according to the shooting range. Before shooting, check whether the drone's tripod, arms, propellers and other equipment are in normal condition, whether the remote control and aircraft are in the correct gear, and whether the main interface and satellite signals are normal. After the inspection, install the gimbal, set the parameters, adjust the lens aperture, and test fly at low altitude. Finally, start taking images with a flight altitude of 65m and a flight speed of 5.27m / s. A total of two flight strips of grass images were taken. After splicing the flight strips, the calibration carpet, houses, roads and other areas were cut out to obtain a 700*700*270 size grass hyperspectral remote sensing image ( Figure 3), 700 is the width and height of the image, 270 is the number of bands, and the spatial resolution of the image is 0.065m. The image contains 12 species of forage grasses, namely Chloris virgata, Bromus japonicus, Leymus chinensis, Elymus nutans Griseb., Sanguisorba officinalis, Alfalfa, Barley Awn, Shortawn Barleygrass, Pennisetum alopecuroides, Elymus dahuricus Turcz., Sophora alopecuroides Linn and Trifolium repens. The number of samples of each forage grass is shown in Table 2. Roads and land are set as background and are not included in the statistical range.
[0119] Table 1 Nano-Hyperspechyperspectral device parameters
[0120] index parameter Spectral range (nm) 400-1000 Number of spatial channels 640 Number of spectral channels 270 Aperture F / 2.5 Spectral sampling rate (nm / pixel) 2.2 Maximum frame rate 350fps Pixel size (μm) 7.4 Flight altitude(m) 65 Lens focal length (mm) 8 FOV(m) 38.48 Field of view (°) 32.98 Spatial resolution (m) 0.065 Flight speed (m / s) 5.27 Strip spacing (m) 30.78
[0121] Table 2 Ground truth categories and their sample numbers of the UFHI dataset
[0122]
[0123] (1.3) Data labeling
[0124] This article uses Envi 5.3 to mark UFHI. According to the marked position during shooting, the area of interest of each type of forage is selected for marking. Zoom in on the pixels containing land and forage, and divide them according to the area occupied by the pixels. If the proportion of forage exceeds 50%, it is identified as forage, otherwise it is identified as land. The growth area of forage cannot be fixed, so the marked area is not a regular shape, and it needs to be marked according to the growth range of forage. In this study, land and roads are identified as background areas, and detailed pixel mark is performed on 12 types of forage. The intersection area is confirmed multiple times. After multiple markings, the hyperspectral remote sensing label image of forage is obtained ( Figure 4 ).
[0125] 2. WHU-Hi Dataset
[0126] (2.1)WHU-Hi-LongKou
[0127] The WHU-Hi-LongKou dataset was captured in Longkou Town, Hubei Province, China from 13:49 to 14:37 on July 17, 2018. The study area includes: corn, cotton, sesame, broad-leaf soybean, narrow-leaf soybean and rice. The image size is 550×400 pixels, there are 270 bands from 400nm to 1000nm, and the spatial resolution of the hyperspectral image is about 0.463m. An overview of the dataset can be found at Figure 5 and Table 3.
[0128] Table 3 Ground truth classes and their respective sample numbers of the WHU-Hi-LongKou dataset
[0129] serial number type Number of samples C1 Corn 34511 C2 Cotton 8374 C3 Sesame 3031 C4 Broad-leaf soybean 63212 C5 Narrow-leaf soybean 4151 C6 Rice 11854 C7 Water 67056 C8 Roadsandhouses 7124 C9 Mixedweed 5229
[0130] (2.2) WHU-Hi-HongHu dataset
[0131] The WHU-Hi-HongHu dataset was collected from 16:23 to 17:37 on November 20, 2017 in Yanchang City, Hubei Province, China. The experimental area is a complex agricultural landscape with multiple crops, including Chinese cabbage, Brassica spp., and Brassica rapa. The image size is 940×475 pixels, with 270 bands from 400nm to 1000nm, and the spatial resolution of the hyperspectral image is about 0.043m. An overview of the dataset is available at Figure 6 and Table 4.
[0132] Table 4 Ground truth classes and their respective sample numbers of the WHU-Hi-HongHu dataset
[0133] serial number type Number of samples serial number type Number of samples C1 Redroof 14041 C12 Brassicachinensis 8954 C2 Road 3512 C13 SmallBrassicachinensis 22507 C3 Baresoil 21821 C14 Lactucasativa 7356 C4 Cotton 163285 C15 Celtuce 1002 C5 Cottonfirewood 6218 C16 Film covered lettuce 7262 C6 Rape 44557 C17 Romainelettuce 3010 C7 Chinese cabbage 24103 C18 Carrot 3217 C8 Pakchoi 4054 C19 Whiteradish 8712 C9 Cabbage 10819 C20 Garlicsprout 3486 C10 Tubermustard 12394 C21 Broadbean 1328 C11 Brassica parachinensis 11015 C22 Tree 4040
[0134] The S2, Multi Band Selector (MBS) can process multi-band image data, calculate the contribution rate of the bands, select the bands containing important information, and enhance the significance of important bands. MBS calculates the mean of each sample pixel to obtain the covariance matrix of the image, determines the contribution rate by calculating the ratio of their sum to the sum of all eigenvalues, retains the bands with large contribution rates, obtains the characteristic band data with rich information, and saves the band image. MBS effectively removes noise, reduces redundant data, reduces data capacity, improves data quality, and speeds up the model. Including:
[0135] (1) Calculate the mean of band data b, where l is the total number of pixels of length b and μ is the mean.
[0136]
[0137] (2) Calculate the covariance matrix σ of b 2 , select the maximum eigenvalue S.
[0138]
[0139] (3) Initialize a zero matrix and calculate the eigenvalues and eigenvectors of the matrix. After sorting the eigenvalues, select the characteristic index and calculate the contribution rate G of the eigenvalue, where λ i is the i-th eigenvalue, and r is the number of bands to be retained.
[0140]
[0141] TransDense3D-AttnNet (TD3DAN) is a multi-channel feature extraction and recognition model proposed in this paper. TD3DAN is a powerful hybrid neural network architecture that can efficiently process hyperspectral remote sensing data. TD3DAN processes data in three dimensions through dense convolutional layers; the weight values of the features are calculated through channel and spatial attention mechanisms, allowing the network to focus on the most important features, improving the efficiency and accuracy of feature extraction; the long-distance dependencies of the data are captured through the multi-head self-attention mechanism, integrating global information.
[0142] TD3DAN( Figure 7 ) Use channel and spatial attention modules to calculate the attention map of the data, emphasize important feature areas, and suppress background information. After attention weighting, enter the Dense3DBC Block to extract deep features. The layers of Dense3DBC are directly connected to each other, which reduces parameters, speeds up the model, effectively transmits information, and emphasizes the feature reuse mechanism. Finally, the model adds a multi-head self-attention mechanism (MSAM). MSAM captures global dependencies, enhances the model's ability to understand complex data, further distinguishes complex backgrounds, and strengthens the analysis of fused grass features, making refined texture features and edge features clearer, and strengthening the ability to distinguish different types of grass. During the training process, the model hyperparameters are adjusted through the backpropagation algorithm and optimizer to optimize the model.
[0143] (1)Dense3DBC Block
[0144] Dense convolutional networks (DenseNet) have made good progress in processing remote sensing images. All layers of DenseNet are interconnected, and each layer can obtain additional input from all previous layers and pass features to all subsequent layers, ensuring the maximum information flow between layers, improving the flow of information and gradients, reducing overfitting problems, achieving feature reuse, and making calculations more efficient. The structure of the Dense layer is Batch Normalization (BN) + ReLU + 3x3Conv. The calculation formula is shown in the formula, where x is the input information and Hl is the composite function to be fitted in the end.
[0145] x l =H l ([x0,x1,...,x l-1 ]) (5)
[0146] Dense3DBC( Figure 8 ) adopts BN+ReLU+1x1x1Conv+BN+ReLU+3x3x3Conv structure, and θ<1, θ∈(0,1] represents the compression coefficient. The BN+ReLU+1x1Conv structure is added on the basis of Dense Block, which effectively reduces the number of feature maps. The three-dimensional convolution layer can extract features in three dimensions. In addition to texture features and shape structure features, it can also extract spectral features and spectral neighborhood features.
[0147] DenseNet uses dense connections to enhance feature extraction capabilities, but too deep a network layer also slows down the model speed and increases the amount of calculation and parameters. Therefore, Dense3DBC also uses dense connections, while reducing the number of dense blocks, reducing the number of network layers, enhancing feature transfer, and reducing parameters.
[0148] (2)Convolutional BlockAttention Module(CBAM)
[0149] CBAM can dynamically adjust the weights of feature maps, pay more attention to important features and spatial locations, improve the model's perception ability, and enhance network performance. Channel attention enhances the feature representation of channels, allowing the model to capture feature information more sensitively. Spatial attention extracts key information at different locations in space, allowing the model to locate features more accurately. Channel attention mechanism ( Fig. 9) The feature F is subjected to MaxPool and AvgPool operations to obtain features of two different dimensions; then it is sent to Multi-Layer Perceptron (MLP) to learn the dependency relationship. Finally, the features output by the MLP are added, and the weights and features are multiplied to obtain the weighted features Mc of the channel. σ is the sigmoid function.
[0150]
[0151] Spatial Attention Mechanism ( Fig.10 ) Perform MaxPool and AvgPool operations on the feature F' output by the channel attention, stack the two feature maps after pooling, and then reduce it to a single channel feature map H×W×1. Finally, multiply the input feature map and the weight to obtain the weight Ms of the spatial dimension.
[0152]
[0153] (3) Multi-head self-attention mechanism (MSAM)
[0154] Multi-head self-attention mechanism MSAM ( Fig.11 ), focusing on the encoded information from different subspaces, using position encoding to capture the position information in the sequence, and enhancing the model's expressiveness. MSAM captures the dependencies between elements in the input sequence, removes the loop structure to achieve parallel computing, and speeds up model training. At the same time, MSAM can pay attention to the differences in input data, thereby effectively fusing global information and helping to process spectral features.
[0155] MSAM performs linear transformation, h is the number of heads, and linear transformation is used to obtain query ((Q)), key ((K)) and value ((V)), where WQ, WK, WV are weight matrices.
[0156]
[0157] Then, the scaled dot product attention is calculated for each head, where is the scaling factor.
[0158]
[0159] Then concatenate the outputs of all heads, (WiQ, WiK, WiV) is the linear transformation matrix of the i-th head, and WO is the output weight matrix.
[0160] MultiHead(Q,K,V)=Concat(head1,....,head h )W o (10)
[0161] headi =Attention(QW i Q ,KW i K ,VW i V ) (11)
[0162] Finally, it passes through the feedforward mechanism MLP. W1, W2 are weight matrices, b1, b2 are biases, x is input data, and f is the activation function.
[0163] MPL=W2f(W1x+b1)+b2 (12)
[0164] LayerNormalization is used after each sublayer of the self-attention mechanism to accelerate training and improve model performance.
[0165] The experiment is evaluated by 6 indicators. Time is the training time. Overall accuracy (OA) means the number of correctly predicted samples divided by the total number of samples, where TP stands for True Positives, FN stands for False Negatives, FP stands for False Positives, and TN stands for True Negatives.
[0166]
[0167] Precision refers to the proportion of true results in samples predicted to be correct.
[0168]
[0169] Recall refers to the number of correct predictions among samples whose true values are correct.
[0170]
[0171] F1-score is used to evaluate the performance of the model on an imbalanced dataset.
[0172]
[0173] The Kappa coefficient indicates the statistical consistency between the result and the true result, po represents OA, pe is the expected value of the accuracy of random classification, N is the total number of samples, and N+i is the total number of samples predicted to be the i-th class.
[0174]
[0175] like Fig.12As shown, an embodiment of the present invention provides a UAV hyperspectral remote sensing image feature extraction and recognition system based on MBS-TD3DAN, which specifically includes:
[0176] The input module is used to collect and produce the UFHI dataset and input the three datasets into the network respectively;
[0177] MBS preprocessing module is used to reduce the dimension of the image, remove noise, and extract characteristic bands;
[0178] TD3DAN feature extraction module is used to extract image features, analyze features and perform recognition.
[0179] The experimental equipment uses Lenovo Savior Y9000P, Inter(R)Core i7-12700H, 32GB memory, and 64-bit Windows 11 operating system. Use tensorflow2.10.0 deep learning framework and python 3.9 version compiler. Set the batch size to 64 and train 100 epochs. The initial learning rate is set to 0.0001, set to 2e-3 after 40 epochs, and set to 0.0005 after 80 epochs. Use Adam optimizer, which has good interpretability for hyperparameters and can adapt the learning rate from two aspects: the mean of the gradient and the mean of the square. The size of the input data is 700*700*16, and 70% of the data set is used as the training set to ensure that the model is trained with a large amount of sample data and sufficient features are extracted.
[0180] In the experiment, three datasets, UFHI, WHU-Hi-LongKou and WHU-Hi-HongHu, were used to evaluate the performance of MBS-TD3DAN. Three popular preprocessing methods were compared: Principal Component Analysis (PCA), Non-negative Matrix Factorization (NMF) and Independent Component Analysis (ICA). Combined with TD3DAN, the differences between the four preprocessing methods were compared through experiments.
[0181] This study compares four advanced models: 3D Residual Auto Encoder (3D-ResAE), 3D-MACNet, deep convolutional capsule neural network (DC-CapsNet), and 3D-DenseNet. These models are used to evaluate the performance of TD3DAN and its ability to process hyperspectral images. 3D-ResAE combines 3D convolution and residual, uses deconvolution to reconstruct the original features, solves the gradient vanishing problem, and trains data in a semi-supervised manner. 3D-MACNet is retrieved by convolution components, and then uses a three-dimensional multi-scale asymmetric convolutional network module to produce classification results through fully connected layers. DC-CapsNet uses a 3D extended convolution capsule network to enhance spectral spatial robustness; a decoder network composed of deconvolution layers is used to obtain more spatial relationships and improve recognition accuracy. 3D-DenseNet enhances feature transmission, supports feature reuse, learns spectral spatial features, and improves the recognition accuracy of complex objects.
[0182] 1. Comparison of preprocessing methods
[0183] 1.1 UAV forage hyperspectral remote sensing image
[0184] First, the preprocessing effects of MBS, PCA, NMF and ICA on UAV forage hyperspectral remote sensing images were compared. Fig.13 As shown in the figure, the image has the best effect after MBS. Each band can show the image, restore the real ground feature category, improve the imaging clarity, and clearly observe the area where the grass is located. This proves that MBS has a good analysis effect on the characteristic bands and has found important characteristic bands.
[0185] The imaging of the other three methods in the last few bands has become blurred and gradually cannot show the outline of the image, indicating that some important features have been lost in the process of dimensionality reduction. Especially after ICA, most bands can no longer be imaged, proving that ICA loses more information, causing the image to become increasingly blurred, and the image outline cannot be seen, and the location of the pasture cannot be distinguished. The first few bands of PCA restore the image outline more clearly, and different areas can be seen, but the subsequent bands are also becoming increasingly blurred and gradually cannot be imaged. In contrast, the preprocessing results of NMF are the best among the three methods. The images presented can roughly distinguish the image outline and see the location of different pastures.
[0186] Next, the images that have been preprocessed are combined with TD3DAN for experiments. The experimental results are shown in Table 5. MBS-TD3DAN has the best experimental results, with OA of 98.5%, F1 of 99.0%, and Kappa of 99.3%. OA is 3.0% higher than PCA, 2.3% higher than NMF, and 3.4% higher than ICA. After the image is processed by MBS, the accuracy and other evaluation indexes of TD3DAN are better than other preprocessing methods. Experiments have shown that MBS can effectively find more representative features from the original hyperspectral data, reduce the dimension of the data, and remove noise and redundant data. Other evaluation parameters prove that the experimental results are highly consistent and accurate.
[0187] Table 5 Comparative experimental results of preprocessing of UAV forage hyperspectral remote sensing images
[0188]
[0189] 1.2WHU-Hi-LongKou
[0190] The preprocessing results of WHU-Hi-LongKou are as follows Fig.14 As shown in the figure, after MBS, the retained bands have higher imaging clarity, highly distinguishing object types and ground feature areas, and finding more detailed information. In contrast, PCA, NMF, and ICA ignore some features, resulting in the latter few bands becoming increasingly blurred and unable to distinguish object categories. Therefore, the LongKou dataset further proves that MBS not only reduces data capacity, but also retains important features and improves data quality.
[0191] The four preprocessed data were trained using TD3DAN. The experimental results are shown in Table 6. MBS-TD3DAN has the best recognition effect on the LongKou dataset, with OA reaching 99.8%, which is 2.3% higher than PCA, 3.1% higher than NMF, and 2.9% higher than ICA. Since the LongKou dataset is relatively simple, the experimental results are not very different, proving that PCA, NMF, and ICA can also effectively process hyperspectral images. MBS has the best experimental evaluation indicators, proving that MBS can extract more important hyperspectral image features and can perform high-precision recognition research.
[0192] Table 6 WHU-Hi-LongKou pretreatment comparison experimental results
[0193]
[0194] 1.3WHU-Hi-HongHu
[0195] From the experimental results ( Fig.15) shows that the preprocessing effect of MBS on the HongHu dataset is still better than the other three methods. The image clarity after MBS is the highest, and it can restore the image well and distinguish different object categories. In contrast, PCA, NMF and ICA have limitations in feature analysis. They do not find enough bands with rich features, making the image blurred in the last few bands and unable to present the outline and details of the image. This further proves the effectiveness of MBS in processing hyperspectral data, which can retain important features and improve data quality.
[0196] TD3DAN was used to train the four pre-processed HongHu data, and the experimental results are shown in Table 7. MBS-TD3DAN still has the highest recognition accuracy on the HongHu dataset, with OA of 97.1%, which is 3.4% higher than PCA, 3.5% higher than NMF, and 5.1% higher than ICA. The experiment once again verified that MBS improves the data processing capability of the model, can remove noise, enhance imaging clarity, and improve the distinction between object types, providing strong support for high-precision recognition research.
[0197] Table 7 WHU-Hi-HongHu pretreatment comparison experimental results
[0198]
[0199] In summary, through preprocessing and recognition experiments on UFHI, WHU-Hi-LongKou and WHU-Hi-HongHu datasets, it is verified that MBS, as a hyperspectral image preprocessing method, can not only reduce data dimension, but also find representative features, enhance imaging clarity, improve the distinction between ground object types, and improve data quality, providing a new method for the processing and analysis of hyperspectral images.
[0200] 2. Comparison of model methods
[0201] 2.1 UAV hyperspectral remote sensing image of forage
[0202] The UFHI dataset consists of 12 types of grasses, land and roads, and all land and roads are set as the background. Five different advanced models are used to experiment on the data preprocessed by MBS. The performance of different methods in processing hyperspectral data is evaluated by the accuracy and model parameters of the grass recognition task. The experimental results are shown in Table 8. The proposed TD3DAN has the best experimental results. The six evaluation indexes are all at the highest level, and OA is 98.5%. Indicators such as F1, kappa, and Time prove that the model has a high degree of consistency and reflects the real results. Experiments have shown that all five models have achieved good results, and OA has reached more than 94%.
[0203] Table 8 UAV forage hyperspectral remote sensing image recognition results
[0204]
[0205]
[0206] exist Fig.16 In the figure, TD3DAN is closest to the groundtruth, with only a few pixels showing recognition errors, which proves that the TD3DAN model proposed in this paper has excellent performance in dealing with complex backgrounds and multiple types of grass, and can be used for high-precision recognition research on grass hyperspectral images. The other four methods all have obvious recognition error areas in the figure, indicating that the feature extraction and analysis capabilities are not enough, and some important features are missed, so that grasses with similar shapes, colors or textures are not completely distinguished, resulting in some recognition error areas. However, the four models have achieved a high level of overall recognition rate for the 12 types of grass, and the recognition results are better than traditional methods and manual recognition methods.
[0207] 2.2WHU-Hi-LongKou
[0208] The complexity of the dataset will also affect the model performance. As a dataset with relatively few types of objects, the LongKou dataset will enhance the model's ability to extract and analyze features of the object types to a certain extent. In Table 9, TD3DAN's OA is 99.8%, showing excellent recognition performance. The recognition results of the 9 types of objects are very good, almost completely restoring the real object situation.
[0209] Table 9 WHU-Hi-LongKou recognition results
[0210]
[0211] Fig.17The recognition results of 5 models on the LongKou dataset are shown. 3D-ResAE and DC-CapsNet made mistakes in identifying a small number of pixels and identified them as other categories, resulting in a decrease in F1. Since the number of correctly identified samples is still relatively large, there is no particularly obvious decrease in OA. With the advantage of deep network layers, the experimental results of 3D-DenseNet on the LongKou dataset can also be close to the real results. A few samples have recognition errors, which proves that the deepening of the network layers plays an important role in the model. The recognition effect of 3D-MACNet is also excellent, almost restoring the real landform, and the recognition error areas are mainly concentrated on Narrow-leaf soybean. The experimental results of TD3DAN also almost restore the real landform, and there are very few recognition errors in the pixels. It proves that the TD3DAN model is highly feasible for the recognition analysis of hyperspectral images and can be used for high-precision recognition research.
[0212] 2.3WHU-Hi-HongHu
[0213] The HongHu dataset has many types of objects and is a complex UAV hyperspectral dataset, which requires high robustness and adaptability of the model. The experimental results are shown in Table 10. TD3DAN performs better on the HongHu dataset, with OA of 97.1%, F1 of 95.6% and kappa coefficient of 97.6%, which reflects its excellent robustness and recognition and detection capabilities, and can also perform well in dealing with more complex hyperspectral datasets.
[0214] Table 10 WHU-Hi-HongHu recognition results
[0215]
[0216]
[0217] The recognition result is as follows Fig.18As shown in the figure, TD3DAN almost restores the real location of the ground objects and can accurately distinguish plants with similar colors and textures. Although the recognition results of the other four models are not as good as TD3DAN, they have also achieved good results in multi-object recognition in complex backgrounds, with an accuracy rate of more than 90%. 3D-ResAE can effectively distinguish most plants, but the recognition rate of some plants is low. 3D-MACNet can almost reach 80% recognition rate for each plant, only Cottonfirewood is about 60%, and the feature analysis ability of some plants needs to be strengthened. Noise appeared in the result map of DC-CapsNet, and recognition errors occurred in many areas. 3D-DenseNet's recognition results for plants in some adjacent areas overlapped, and they were recognized as adjacent species. However, the model has good stability, and the overall recognition rate is more than 90%. Experiments have proved that deep learning is suitable for the recognition and detection research of hyperspectral images. It can analyze the ground object components of the image, extract key features, and refine the feature details. It can be applied to practical high-precision recognition research.
[0218] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.
[0219] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.
Claims
1. A method for extracting and recognizing features of UAV hyperspectral remote sensing images based on MBS-TD3DAN, characterized in that: The method specifically includes: S1: Constructing a UAV forage hyperspectral remote sensing dataset; S2: Use Multi Band Selector (MBS) preprocessing method to reduce the image dimension, remove noise, and extract feature bands; S3: The TransDense3D-AttnNet (TD3DAN) model is used to mark data weights, extract data features through object, space, and spectrum dimensions, use the self-attention mechanism to transform, map, filter, and perform other operations on the data, and use dense convolutional layers to enhance feature transfer, ultimately achieving high-precision recognition.
2. The method for extracting and identifying features of unmanned aerial vehicle hyperspectral remote sensing images based on MBS-TD3DAN as claimed in claim 1, characterized in that: The S1 uses three datasets, including the UAV forage hyperspectral remote sensing dataset (UFHI) taken on site, and the UAV hyperspectral datasets WHU-Hi-LongKou and WHU-Hi-HongHu. The UFHI image was taken at the Experimental Base of the Agricultural and Pastoral Interlaced Area of the Grassland Research Institute of the Chinese Academy of Agricultural Sciences, Hohhot, Inner Mongolia Autonomous Region, China (4034'N, 11145'E). The acquisition equipment is the Nano-Hyperspec micro airborne hyperspectral imager, equipped with the DJI M300RTK rotor drone, using Envi 5.3 Marking UFHI. According to the marked position during shooting, select the area of interest for each type of forage grass for marking. Zoom in on the pixels containing land and forage grass, and divide them according to the area occupied by the pixels. If the proportion of forage grass exceeds 50%, it is identified as forage grass, otherwise it is identified as land. The growth area of forage grass cannot be fixed, so the marked area is not a regular shape. It needs to be marked according to the growth range of forage grass. In this study, land and roads are identified as background areas, and detailed pixel mark is performed for 12 types of forage grass. The intersection area is confirmed multiple times. After multiple markings, the hyperspectral remote sensing label image of forage grass is obtained.
3. The method for extracting and identifying features of unmanned aerial vehicle hyperspectral remote sensing images based on MBS-TD3DAN as claimed in claim 1, characterized in that: S2, MBS calculates the mean of each sample pixel to obtain the covariance matrix of the image, determines the contribution rate by calculating the ratio of their sum to the sum of all eigenvalues, retains the band with large contribution rate, obtains the characteristic band data with rich information, and saves the band image, which specifically includes: (1) Calculate the mean of band data b, where l is the total number of pixels of length b and μ is the mean: (2) Calculate the covariance matrix σ of b 2 , select the largest eigenvalue S: (3) Initialize a zero matrix, calculate the eigenvalues and eigenvectors of the matrix, sort the eigenvalues and select the eigenindex, and calculate the contribution rate G of the eigenvalues, where λ i is the i-th eigenvalue, and r is the number of bands to be retained:
4. The method for extracting and identifying features of unmanned aerial vehicle hyperspectral remote sensing images based on MBS-TD3DAN as claimed in claim 1, characterized in that: The S3, TD3DAN is a multi-channel feature extraction and recognition model proposed by the present invention, comprising: (1)Dense3DBC Block It adopts the BN+ReLU+1x1x1Conv+BN+ReLU+3x3x3Conv structure, and θ<1, θ∈(0,1] represents the compression coefficient. On the basis of Dense Block, the BN+ReLU+1x1Conv structure is added. Dense3DBC adopts dense connection and reduces the number of dense blocks. (2)Convolutional BlockAttention Module(CBAM) The weight value of the feature is calculated through the channel and spatial attention mechanism, and the weight of the feature map is dynamically adjusted. The channel attention mechanism performs MaxPool and AvgPool operations on the feature F to obtain features of two different dimensions, which are sent to the Multi-Layer Perceptron (MLP) to learn the dependency relationship. Finally, the features output by the MLP are added, and the weights and features are multiplied to obtain the weighted feature Mc of the channel; the spatial attention mechanism performs MaxPool and AvgPool operations on the feature F` output by the channel attention, stacks the two feature maps after pooling, and then reduces them to a single channel feature map H×W×1. Finally, the input feature map and the weight are multiplied to obtain the weight Ms of the spatial dimension; (3) Multi-head self-attention mechanism (MSAM) MSAM performs linear transformation, h is the number of heads, and obtains query ((Q)), key ((K)) and value ((V)) through linear transformation, where W Q ,W K ,W V is the weight matrix: Then, the scaled dot product attention is calculated for each head, where is the scaling factor: Then concatenate the outputs of all the heads together, (W i Q ,W i K ,W i V ) is the linear transformation matrix of the ith head, W O is the output weight matrix: MultiHead(Q,K,V)=Concat(head1,....,head h )W o (10) head i =Attention(QW i Q ,KW i K ,VW i V ) (11) Finally, after the feedforward mechanism MLP, W1, W2 are weight matrices, b1, b2 are biases, x is input data, and f is the activation function: MPL=W2f(W1x+b1)+b2 (12) 5. A UAV hyperspectral remote sensing image feature extraction and recognition system based on MBS-TD3DAN as described in claims 1-4, characterized in that: The system specifically includes: The UAV forage hyperspectral remote sensing dataset building module is used to collect and process three datasets including UFHI, WHU-Hi-LongKou and WHU-Hi-HongHu; Multi-band selector (MBS) preprocessing module, used to reduce the dimension, denoise and extract characteristic bands of images; TransDense3D-AttnNet (TD3DAN) model module is used to obtain the image through three aspects: object, space, and spectrum. The data features are extracted from the dimension and the self-attention mechanism is used to transform, map, filter and perform other operations on the data to achieve high-precision recognition.
6. The UAV hyperspectral remote sensing image feature extraction and recognition system based on MBS-TD3DAN according to claim 5 is characterized in that: The UAV forage hyperspectral remote sensing dataset construction module includes: The data acquisition equipment is configured as Nano-Hyperspec micro airborne hyperspectral imager, coupled with DJI M300 RTK rotary wing drone; Data labeling software, Envi 5.3 was used to label the UFHI dataset, and the areas of interest for each forage were marked and divided in detail.
7. The UAV hyperspectral remote sensing image feature extraction and recognition system based on MBS-TD3DAN according to claim 5 is characterized in that: The multi-band selector (MBS) pre-processing module comprises: The calculation unit is used to calculate the mean and covariance matrix of each sample pixel and select the band with large contribution rate; The data storage unit is used to store characteristic band data and its corresponding image.
8. The UAV hyperspectral remote sensing image feature extraction and recognition system based on MBS-TD3DAN according to claim 5 is characterized in that: The TransDense3D-AttnNet (TD3DAN) model module includes: Dense3DBC block, using BN+ReLU+1x1x1Conv+BN+ReLU+3x3x3Conv structure, densely connected feature layers; Convolutional Block Attention Module (CBAM), which is used to dynamically adjust the weights of feature maps through channel and spatial attention mechanisms; Multi-head self-attention mechanism (MSAM), which is used to linearly transform features and scale dot product attention calculations; Feed-forward mechanism, using multi-layer perceptron (MLP) to further process and activate the output features.
Citation Information
Patent Citations
Hyperspectral remote sensing image classification method based on double attention mechanism
CN113011499A
Grassland multi-temporal hyperspectral classification method based on Transform network
CN116597204A
Grassland hyperspectral data classification method
CN117132814A
Pollen image classification method based on convolutional neural network and multi-scale cavity attention fusion
CN117496260A
