Method for constructing a large basic model for low-altitude UAV hyperspectral remote sensing

Abstract

The present invention discloses a method for constructing a basic large model of low-altitude unmanned aerial vehicle hyperspectral remote sensing, including constructing a data set, wherein the sample is a H×W×D hyperspectral sample; constructing a three-dimensional extended MAE model, including an image block layer, a 3D position encoding layer, a dynamic mask layer, a MAE model, and a HED model; constructing a total loss function L; training the three-dimensional extended MAE model to obtain a large spectral model, which is used to reconstruct the hyperspectral sample to obtain a predicted hyperspectral image. The present invention extends the traditional two-dimensional MAE model to three dimensions, combines the dynamic occlusion rate strategy and 3D position encoding to enable the reconstructed image to better express the spectral characteristics of different objects and surfaces, and then combines the HED model to enhance the processing of edge details and provide more accurate physical boundaries. The present invention is suitable for accurate material identification and classification, and is particularly suitable for complex environmental monitoring and resource assessment tasks in the fields of precision agriculture, environmental protection, and disaster management.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a method for constructing a low-altitude unmanned aerial vehicle hyperspectral remote sensing basic large model. Background Art

[0002] Hyperspectral images are unique because of their rich spectral information across multiple bands, each of which can capture different physical and chemical properties. This is because different substances absorb or reflect light in specific bands, and this characteristic can identify and distinguish the composition of surface materials or other objects. For example, vegetation and water bodies have different characteristics of reflecting and absorbing light in different bands, and these characteristics can be used to accurately identify them.

[0003] Hyperspectral imaging technology is becoming increasingly mature in the field of remote sensing through the application of low-altitude drones, especially in earth observation and environmental monitoring. Although traditional hyperspectral imaging methods can capture a large amount of environmental data, they still face efficiency challenges in diverse data processing and information extraction when processing large-scale data sets. This is because current hyperspectral image processing methods only process spatial information and ignore band information. Especially when processing hyperspectral images, some image reconstruction defects may occur, which will cause inaccurate processing of downstream tasks, especially in tasks that require precise processing of band characteristics, such as material composition analysis, vegetation and ecological monitoring, water quality analysis, mineral and geological mapping, and urban planning and management.

[0004] MAE (Masked Autoencoder) is a pre-training method for computer vision. Its operation can be simply summarized as: randomly occlude some image blocks in the image according to a certain ratio, and then reconstruct the pixel values ​​of these occluded parts. The MAE model is based on the ViT architecture. The image processing process is to first divide the image into blocks, then linearly project, classify and mark the image blocks, and then send them to the Transformer encoder for encoding and then to the decoder for decoding. The MAE model has two main features: 1. It adopts an asymmetric encoder-decoder structure, in which the encoder only processes the unoccluded part of the image, and the decoder is responsible for reconstructing the occluded part. 2. Use a higher occlusion rate, which is also called the mask rate or mask ratio.

[0005] However, the MAE model only processes spatial information when processing images. For example, image segmentation is to segment the image in width and height, and position encoding is to only perform position encoding on the spatial information of height and width. Therefore, it only processes spatial information and ignores band information, which results in the inability to utilize band information and inaccurate information extraction, resulting in some image reconstruction defects. In addition, when training the MAE model, the occlusion rate remains unchanged in each training round, which limits the learning ability. Summary of the invention

[0006] The purpose of the present invention is to provide a method for constructing a basic large model of low-altitude UAV hyperspectral remote sensing, which can solve the problems of insufficient utilization of the above-mentioned band information, fixed occlusion rate limiting learning, and inaccurate information extraction.

[0007] In order to achieve the above-mentioned object, the technical solution adopted by the present invention is as follows: a method for constructing a basic large model of low-altitude unmanned aerial vehicle hyperspectral remote sensing, comprising the following steps;

[0008] S1, construct data set X;

[0009] Obtain hyperspectral images of different regions with a size of H×W. Each hyperspectral image contains at least D bands. For each hyperspectral image, select D bands to form a hyperspectral sample with a dimension of H×W×D. All hyperspectral samples constitute a data set X.

[0010] S2, constructing a three-dimensional extended MAE model, including an image segmentation layer, a 3D position encoding layer, a dynamic mask layer, a MAE model, and a HED model;

[0011] The image segmentation layer is used to divide the input hyperspectral samples into N H ×N W ×N D A three-dimensional block of dimension p×q×r, where p, q, and r are the length, width, and number of bands of the three-dimensional block, respectively. H 、N W and N D are the number of segments of the hyperspectral sample in length, width, and number of bands, respectively;

[0012] The 3D position coding layer is used to perform 3D position coding on each 3D block using a PE vector as follows;

[0013] ,

[0014] ,

[0015] in, is an even value of the PE vector, is an odd value of the PE vector, pos is the current 3D block position, 0≤pos≤N H ×N W ×N D -1,d pos is the number of hidden layer dimensions of the MAE model, c is the band c among the D bands, λ c is the central wavelength of band c, f is the hidden layer dimension index, 0≤f≤(N H ×N W ×N D-1) / 2-1;

[0016] The dynamic mask layer is used to occlude the 3D block according to the dynamic occlusion rate m(t), forming the input X of the MAE model input , where the occluded 3D block data is set to zero, and only the 3D position code is retained. m(t) is obtained according to the following formula;

[0017] ,

[0018] In the formula, m min 、m max are the preset lower limit of occlusion rate, the upper limit of occlusion rate, e is the natural index, T is the preset total training rounds, t is the tth training round, 1≤t≤T;

[0019] The MAE model is used for input X input , for X input Reconstruct the occluded blocks in the image to obtain the predicted hyperspectral image;

[0020] The HED model is used to input a hyperspectral sample, perform edge detection on it and generate an edge mask probability map;

[0021] S3, construct the total loss function L, L = L HED +L MAE , where L MAE , L HED They are the losses contributed by the MAE model and the HED model respectively;

[0022] ,

[0023] ,

[0024] In the formula, for the pixel point p with coordinates (x, y) xy ,y c (x,y) is the p in the hyperspectral sample xy The value of band c, p c (x,y) is the predicted hyperspectral image p xy In band c, M(x,y) is the value of p in the edge mask probability map xy The value of HED(x,y) is the value of HED model. xy The probability of predicting a marginal value, thre HED is the preset edge probability threshold, is the indicator function, α is the weight adjustment parameter;

[0025] S4, according to the total training rounds T, the three-dimensional extended MAE model is trained with the dataset X, and L is trained and optimized by the gradient descent method to obtain the spectral large model.

[0026] As a preferred embodiment: the dynamic mask layer constitutes the input of the MAE model according to steps Sa1 to Sa4;

[0027] Sa1, the total number of three-dimensional blocks N that are preset for the input MAE model keep , N keep <N H ×N W ×N D , define the dynamic occlusion rate m(t), m(t)=m min +(m max -m min );

[0028] Sa2, for each 3D block B at position (i, j, k) ijk , sample a random variable R from uniform distribution U(0,1) ijk , calculate B according to the following formula ijk The mask M ijk ;

[0029] ,

[0030] Sa3, if M ijk =1, B ijk Marked as unobstructed block, if M ijk =0, and B ijk Mark it as an occluded block and change B ijk The internal data is set to zero, and only the 3D position code is retained;

[0031] Sa4, count the number of unobstructed blocks N actual , if N actual <N keep , randomly select N from the occluded blocks keep -N actual The occluded blocks are used as padding blocks, and the unoccluded blocks and padding blocks together constitute the input of the MAE model.

[0032] As a preference: N keep =0.75×N H ×N W ×N D .

[0033] As a preference: , , , To round up, m min =25%, m max =75%.

[0034] Preferably, before the hyperspectral samples are input into the image block layer, image preprocessing is performed to perform image enhancement and denoising on the hyperspectral samples.

[0035] When performing the block operation at the image block layer, the present invention blocks the hyperspectral samples in length, width and number of bands to generate three-dimensional blocks of dimensions p×q×r, and improves the two-dimensional image blocks in the prior art into three-dimensional image blocks, each of which contains not only spatial information in height and width, but also r band information. This method can help the model more accurately combine and utilize spatial and spectral information when processing hyperspectral images, especially in applications that require accurate processing of band characteristics, such as material composition analysis, environmental monitoring, etc.

[0036] The present invention proposes a new position coding method, which changes the traditional two-dimensional position coding into three-dimensional 3D position coding. When coding, considering the influence of spatial information and band information on the image, the present invention encodes the image blocks in the spatial direction and the spectral band direction respectively. Considering that in the spectral band dimension, the traditional position coding method contains spatial equidistant information, and considering the relationship between different bands, the equidistant information is not applicable to the wavelength dimension. The wavelength range of the spectral band may be very wide. Directly using the wavelength may cause the encoding value range to be too large, which is not easy to process by the model. Therefore, the present invention uses a logarithmic scale for processing. In , In , it belongs to the encoding of spectral band direction and logarithmic scale.

[0037] The present invention proposes a dynamic occlusion rate solution, which dynamically occludes the image block according to the dynamic occlusion rate m(t) before inputting the encoder of the MAE model. Combined with the formula of m(t), when t=0, Approaching 0, m(t)≈m min When t=T / 2, Approaching 0.5, Based on the dynamic occlusion rate m(t), the present invention can divide the entire model training into three stages. In the initial stage of training, m(t) is close to m min , exposing more input data with low occlusion rate, allowing the MAE model to quickly capture basic features and details in the data, such as edges, textures, and color changes, which are key elements for understanding image content. In the middle of training, when t approaches T / 2, m(t) gradually approaches , which makes the MAE model start to shift from simple visual features to more abstract and complex structures in the data, such as the integration of spectral features and the fusion of multi-band information. In the later stage of training, as t approaches T, m(t) approaches m max ,At this stage, the high occlusion rate forces the model to reconstruct the lost information from ,very limited visible data, which strengthens the model's learning of the intrinsic ,structure and potential patterns of the data, which is very important for the model to enhance ,its generalization ability on unknown data.

[0038] According to the changing m(t) value, the present invention designs a method of adding padding blocks and unobstructed blocks to the MAE model. Since the value of m(t) is different in each round of training, in order to ensure that the number of three-dimensional blocks input to the MAE model is the same, it is necessary to preset the total number of three-dimensional blocks N input to the MAE model as described in step Sa1. keep After blocking, count the number of unblocked blocks N as described in step Sa4 actual , if N actual <N keep , randomly select N from the occluded blocks keep -N actual The occluded blocks are used as padding blocks, and the unoccluded blocks and padding blocks together constitute the input of the MAE model.

[0039] The present invention proposes a method for fusing the HED model-related loss with the MAE model-related loss, performs edge extraction through the HED model, and then gives higher weights to pixels in the edge area.

[0040] Compared with the prior art, the advantages of the present invention are: the present invention is used to reconstruct the hyperspectral images in the earth observation and environmental monitoring scenes processed by the hyperspectral imaging system carried by the low-altitude UAV, and the reconstructed predicted hyperspectral map has the characteristics of detailed spectral feature expression and optimized edge processing. It is specifically reflected in the following points:

[0041] (1) Based on the image segmentation layer and 3D position encoding layer, the data of the encoder in the input MAE model is expanded from two dimensions to three dimensions. This improvement enables the MAE model to more effectively process and analyze hyperspectral data, especially the complex spatial-spectral information from low-altitude drones. Through this three-dimensional input, the MAE model can not only capture the width and height information of the image, but also go deep into the spectral dimension, thereby achieving a more comprehensive understanding and recognition of environmental features. This three-dimensional processing significantly improves the data representation capability, making the model more accurate and efficient when performing tasks such as environmental monitoring, resource management, and disaster response.

[0042] (2) The value of the dynamic occlusion rate m(t) of the present invention increases nonlinearly with the increase of training rounds, so that the MAE model is exposed to a wider range of data in the early stage and quickly captures the global and basic features. As the occlusion rate increases, the model further learns from limited information and effectively reconstructs key details. This strategy significantly reduces the dependence on noisy data features, enhances the stability and reliability of features, and the ability to interpret complex data features, thereby effectively avoiding overfitting and improving learning efficiency.

[0043] (3) The HED model is introduced for edge extraction, giving higher weights to pixels in the edge area. This is because the edge area is crucial to the visual perception and content understanding of the overall image. Increasing the weight of pixels in the edge area can improve the sensitivity to image edges and prediction accuracy. This method is usually more accurate than traditional image processing operator edge extraction. Moreover, this method only changes the loss function in the model training stage to increase the focus on edges, and does not require additional calculation steps in the inference stage, so it does not increase inference time or resource consumption.

[0044] In summary, the present invention can provide more accurate visualization of physical boundaries and transition areas by enhancing the processing of edge details while maintaining the rich information in hyperspectral data. The dynamic occlusion rate strategy and 3D position encoding of the model enable the reconstructed image to better express the spectral characteristics of different objects and surfaces, which is suitable for accurate material identification and classification, and is particularly suitable for complex environmental monitoring and resource assessment tasks. This capability provides the necessary flexibility and accuracy for dealing with various situations in the real world, such as changing environmental conditions or different types of surface materials. Therefore, the present invention can be used in a variety of application fields such as precision agriculture, environmental protection, and disaster management. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the structure of the three-dimensional extended MAE model;

[0046] Figure 2 It is a flow chart of the present invention;

[0047] Figure 3 This is a graph showing the dynamic occlusion rate changing with training rounds. DETAILED DESCRIPTION

[0048] The present invention will be further described below in conjunction with the accompanying drawings.

[0049] Example 1: See Figures 1 to 3 ,A method for constructing a basic large model of low altitude UAV hyperspectral remote sensing includes the following steps;

[0050] S1, construct data set X;

[0051] Obtain hyperspectral images of different regions with a size of H×W. Each hyperspectral image contains at least D bands. For each hyperspectral image, select D bands to form a hyperspectral sample with a dimension of H×W×D. All hyperspectral samples constitute a data set X.

[0052] S2, constructing a three-dimensional extended MAE model, including an image segmentation layer, a 3D position encoding layer, a dynamic mask layer, a MAE model, and a HED model;

[0053] The image segmentation layer is used to divide the input hyperspectral samples into NH ×N W ×N D A three-dimensional block of dimension p×q×r, where p, q, and r are the length, width, and number of bands of the three-dimensional block, respectively. H 、N W and N D are the number of segments of the hyperspectral sample in length, width, and number of bands, respectively;

[0054] The 3D position coding layer is used to perform 3D position coding on each 3D block using a PE vector as follows;

[0055] ,

[0056] ,

[0057] in, is an even value of the PE vector, is an odd value of the PE vector, pos is the current 3D block position, 0≤pos≤N H ×N W ×N D -1,d pos is the number of hidden layer dimensions of the MAE model, c is the band c among the D bands, λ c is the central wavelength of band c, f is the hidden layer dimension index, 0≤f≤(N H ×N W ×N D -1) / 2-1;

[0058] The dynamic mask layer is used to occlude the 3D block according to the dynamic occlusion rate m(t), forming the input X of the MAE model input , where the occluded 3D block data is set to zero, and only the 3D position code is retained. m(t) is obtained according to the following formula;

[0059] ,

[0060] In the formula, m min 、m max are the preset lower limit of occlusion rate, the upper limit of occlusion rate, e is the natural index, T is the preset total training rounds, t is the tth training round, 1≤t≤T;

[0061] The MAE model is used for input X input , for X input Reconstruct the occluded blocks in the image to obtain the predicted hyperspectral image;

[0062] The HED model is used to input a hyperspectral sample, perform edge detection on it and generate an edge mask probability map;

[0063] S3, construct the total loss function L, L = L HED +L MAE , where L MAE , L HED They are the losses contributed by the MAE model and the HED model respectively;

[0064] ,

[0065] ,

[0066] In the formula, for the pixel point p with coordinates (x, y) xy ,y c (x,y) is the p in the hyperspectral sample xy The value of band c, p c (x,y) is the predicted hyperspectral image p xy In band c, M(x,y) is the value of p in the edge mask probability map xy The value of HED(x,y) is the value of HED model. xy The probability of predicting a marginal value, thre HED is the preset edge probability threshold, is the indicator function, α is the weight adjustment parameter;

[0067] S4, according to the total training rounds T, the three-dimensional extended MAE model is trained with the dataset X, and L is trained and optimized by the gradient descent method to obtain the spectral large model.

[0068] In this embodiment, the dynamic mask layer constitutes the input of the MAE model according to steps Sa1 to Sa4;

[0069] Sa1, the total number of three-dimensional blocks N that are preset for the input MAE model keep , N keep <N H ×N W ×N D , define the dynamic occlusion rate m(t), m(t)=m min +(m max -m min );

[0070] Sa2, for each 3D block B at position (i, j, k) ijk , sample a random variable R from uniform distribution U(0,1) ijk , calculate B according to the following formula ijk The mask M ijk ;

[0071] ,

[0072] Sa3, if M ijk =1, Bijk Marked as unobstructed block, if M ijk =0, and B ijk Mark it as an occluded block and change B ijk The internal data is set to zero, and only the 3D position code is retained;

[0073] Sa4, count the number of unobstructed blocks N actual , if N actual <N keep , randomly select N from the occluded blocks keep -N actual The occluded blocks are used as padding blocks, and the unoccluded blocks and padding blocks together constitute the input of the MAE model.

[0074] In addition, N keep =0.75×N H ×N W ×N D . , , , To round up, m min =25%, m max =75%. Before the hyperspectral samples are input into the image block layer, image preprocessing is also performed to enhance and denoise the hyperspectral samples.

[0075] Regarding the hyperspectral samples, predicted hyperspectral images and edge mask probability maps, the dimensions of the hyperspectral samples and predicted hyperspectral images are H×W×D, and the dimension of the edge mask probability map is H×W, so the pixel point p at the coordinate (x, y) xy Corresponding points can be found in hyperspectral samples, predicted hyperspectral images and edge mask probability maps.

[0076] Example 2: See Figures 1 to 3 Based on Example 1, in order to more clearly illustrate the input X sent to the MAE model for each training input , we take the specific data as an example, assuming that the hyperspectral samples are divided into N H ×N W ×N D =8×8×8=512 3D blocks, the total number of 3D blocks of the preset MAE model is N keep =0.75×512=384 blocks.

[0077] If m(t)=25% during a training, then 75% of the data needs to be retained, that is, just 128 3D blocks are blocked, and the number of unblocked blocks N is left. actual =384=N keep , then 384 blocks constitute the input X of the MAE model inputIf m(t)=75% in a training run, then 25% of the data needs to be retained, that is, 512×75%=384 3D blocks are blocked, and the number of unblocked blocks is N. actual =128, then randomly select N from the occluded blocks keep -N actual = 384-128 = 256 occluded blocks are used as padding blocks, and the unoccluded blocks and padding blocks together constitute the input of the MAE model.

[0078] Example 3: See Figures 1 to 3 After obtaining the spectral big model on the basis of Example 1, it can be used to process the hyperspectral image later. For the hyperspectral image to be processed, it is processed into a hyperspectral sample of dimension H×W×D according to step S1 of the present invention, and then sent to the spectral big model for reconstruction to obtain the corresponding predicted hyperspectral image. In the present invention, the hyperspectral image comes from the hyperspectral imaging system carried by the low-altitude unmanned aerial vehicle. The hyperspectral image can be various images collected in earth observation and environmental observation. The processed predicted hyperspectral image contains rich spatial information and band information, and various heads are connected to the back end of the spectral big model to realize various downstream tasks, such as classification, inspection, segmentation, denoising, etc.

[0079] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a low-altitude UAV hyperspectral remote sensing basic large model, characterized by: The steps include: S1, construct data set X; Obtain hyperspectral images of different regions with a size of H×W, each hyperspectral image contains at least D bands, and for each hyperspectral image, select D bands to form a hyperspectral sample of dimension H×W×D. All hyperspectral samples constitute a data set X, where H and W are the height and width of the hyperspectral image, respectively; S2, constructing a three-dimensional extended MAE model, including an image segmentation layer, a 3D position encoding layer, a dynamic mask layer, a MAE model, and a HED model; The image segmentation layer is used to divide the input hyperspectral samples into N H ×N W ×N D A three-dimensional block of dimension p×q×r, where p, q, and r are the length, width, and number of bands of the three-dimensional block, respectively. H 、N W and N D are the number of segments of the hyperspectral sample in length, width, and number of bands, respectively; The 3D position coding layer is used to perform 3D position coding on each 3D block using a PE vector as follows; ， ， in, is an even value of the PE vector, is an odd value of the PE vector, pos is the current 3D block position, 0≤pos≤N H ×N W ×N D -1,d pos is the number of hidden layer dimensions of the MAE model, c is the band c among the D bands, λ c is the central wavelength of band c, f is the hidden layer dimension index, 0≤f≤(N H ×N W ×N D -1) / 2-1; The dynamic mask layer is used to occlude the 3D block according to the dynamic occlusion rate m(t), forming the input X of the MAE model input , where the occluded 3D block data is set to zero, and only the 3D position code is retained. m(t) is obtained according to the following formula; ， In the formula, m min 、m max are the preset lower limit of occlusion rate, the upper limit of occlusion rate, e is the natural index, T is the preset total training rounds, t is the tth training round, 1≤t≤T; The MAE model is used for input X input , for X input Reconstruct the occluded blocks in the image to obtain the predicted hyperspectral image; The HED model is used to input a hyperspectral sample, perform edge detection on it and generate an edge mask probability map; S3, construct the total loss function L, L = L HED +L MAE , where L MAE , L HED They are the losses contributed by the MAE model and the HED model respectively; ， ， In the formula, for the pixel point p with coordinates (x, y) xy ,y c (x,y) is the p in the hyperspectral sample xy The value of band c, p c (x,y) is the predicted hyperspectral image p xy In band c, M(x,y) is the value of p in the edge mask probability map xy The value of HED(x,y) is the value of HED model. xy The probability of predicting a marginal value, thre HED is the preset edge probability threshold, is the indicator function, α is the weight adjustment parameter; S4, according to the total training rounds T, the three-dimensional extended MAE model is trained with the dataset X, and L is trained and optimized by the gradient descent method to obtain the spectral large model.

2. The method for constructing a low-altitude UAV hyperspectral remote sensing basic large model according to claim 1 is characterized in that: The dynamic mask layer forms the input of the MAE model according to steps Sa1~Sa4; Sa1, the total number of three-dimensional blocks N that are preset for the input MAE model keep , N keep <N H ×N W ×N D , define the dynamic occlusion rate m(t), ; Sa2, for each 3D block B at position (i, j, k) ijk , sample a random variable R from uniform distribution U(0,1) ijk , calculate B according to the following formula ijk The mask M ijk ; ， Sa3, if M ijk =1, B ijk Marked as unobstructed block, if M ijk =0, and B ijk Mark it as an occluded block and change B ijk The internal data is set to zero, and only the 3D position code is retained; Sa4, count the number of unobstructed blocks N actual , if N actual <N keep , randomly select N from the occluded blocks keep -N actual The occluded blocks are used as padding blocks, and the unoccluded blocks and padding blocks together constitute the input of the MAE model.

3. The method for constructing a low-altitude unmanned aerial vehicle hyperspectral remote sensing basic large model according to claim 2 is characterized in that: N keep =0.75×N H ×N W ×N D 。 4. The method for constructing a low-altitude unmanned aerial vehicle hyperspectral remote sensing basic large model according to claim 1 is characterized in that: , , , To round up, m min =25%, m max =75%.

5. The method for constructing a low-altitude unmanned aerial vehicle hyperspectral remote sensing basic large model according to claim 1 is characterized in that: Before the hyperspectral samples are input into the image block layer, image preprocessing is also performed to enhance and denoise the hyperspectral samples.

Images